0

我正在使用 html 清洁器从网站获取数据……但我不断收到此错误。

服务器返回 HTTP 响应代码：403 用于 URL：http ://www.groupon.com/browse/chicago?z=skip

我不确定我做错了什么，因为我以前使用过相同的代码并且它工作得很好。有人可以帮助我吗？

代码如下：

public ArrayList ParseGrouponDeals(ArrayList arrayList) {
    try {
        CleanerProperties props = new CleanerProperties();

        props.setTranslateSpecialEntities(true);
        props.setTransResCharsToNCR(true);
        props.setOmitComments(true);

        TagNode root = new HtmlCleaner(props).clean(new URL("http://www.groupon.com/browse/chicago?z=skip"));

        //Get the Wrapper.
        Object[] objects = root.evaluateXPath("//*[@id=\"browse-deals\"]");
        TagNode dealWrapper = (TagNode) objects[0];

        //Get the childs
        TagNode[] todayDeals = dealWrapper.getElementsByAttValue("class", "deal-list-tile grid_5_third", true, true);
        System.out.println("++++ Groupon Deal Today: " + todayDeals.length + " deals");
        for (int i = 0; i < todayDeals.length; i++) {
            String link = String.format("http://www.groupon.com%s", todayDeals[i].findElementByAttValue("class", "deal-permalink", true, true).getAttributeByName("href").toString());
            arrayList.add(link);
        }
        return arrayList;
    } catch (Exception e) {
        System.out.println("Error parsing Groupon:" + e.getMessage());
        e.printStackTrace();
    }
    return null;
}
4

1 回答 1

0

对我来说，添加“用户代理”可以解决问题；像这样使用它：

        final URL urlSB = new URL("http://www.groupon.com/browse/chicago?z=skip");
        final URLConnection urlConnection = urlSB.openConnection();
        urlConnection.addRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0");
        urlConnection.connect();
        final HtmlCleaner cleaner = new HtmlCleaner();
        final CleanerProperties props = cleaner.getProperties();
        props.setNamespacesAware(false);
        final TagNode tagNodeRoot = cleaner.clean(urlConnection.getInputStream());
于 2014-08-26T16:51:43.663 回答