6

嘿，我正在尝试从 twitter 个人资料页面获取 html，但 httpurlconnection 仅返回一小段 html。我的代码

for(int i = 0; i < urls.size(); i++)
{
URL url = new URL(urls.get(i));
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestProperty("User-Agent","Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6");
System.out.println(connection.getResponseCode());
String line;
StringBuilder builder = new StringBuilder();
BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
while((line = reader.readLine()) != null)
{
    builder.append(line);
}
String html = builder.toString();
}

我总是得到 200 作为每次呼叫的响应代码。然而，大约有 1/3 的时间返回整个 html 文档，而另一半只返回前几百行。html 被截断时返回的数量并不总是相同的。

有任何想法吗？谢谢你的帮助！

附加信息：查看标题后，我似乎得到了重复的内容长度标题。第一个是全长，另一个短得多（可能代表我有时会得到的长度）如何处理重复的标题？

4

2 回答 2

2

这对我来说很好，我在之后添加了一个换行符builder.append(line);以使其在控制台中更具可读性，但除此之外它返回了该页面的所有 HTML：

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.ArrayList;
import java.util.List;

public class RetrieveHTML {

    public static void main(String[] args) throws IOException {
        List<String> urls = new ArrayList<String>();
        urls.add("http://stackoverflow.com/questions/3285077/java-httpurlconnection-cutting-off-html");

        for (int i = 0; i < urls.size(); i++) {
            URL url = new URL(urls.get(i));
            HttpURLConnection connection = (HttpURLConnection) url.openConnection();
            connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6");
            System.out.println(connection.getResponseCode());
            String line;
            StringBuilder builder = new StringBuilder();
            BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
            while ((line = reader.readLine()) != null) {
                builder.append(line);
                builder.append("\n"); 
            }
            String html = builder.toString();
            System.out.println("HTML " + html);
        }

    }
}
于 2010-07-19T21:14:01.383 回答
0

查看我的 HTTP 类

https://stackoverflow.com/questions/9349378/java-net-httpurlconnection-returning-your-browsers-cookie-functionality-has-be

基于此 API。随意改变一些东西。

于 2012-03-22T07:51:25.480 回答