0

我有一个页面要解析（抓取），但首先我必须通过控制措施。
我设法计算了一些控制号，当我尝试通过 POST 方法参数传递这些控制号和其他内容时，我认为页面会自行刷新并生成新的控制号，因此我计算的那些控制号没有通过检查，我无法通过访问所需的页面。

首先，我使用 HtmlAgilityPack 方法获取页面并获取这些控制编号的值：

HtmlWeb web = new HtmlWeb();
HtmlDocument mainPage = web.Load(url);
int controlNumber = FindControlNumber();

之后，我尝试通过 POST 方法传递计算出的数字：

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(newUrl);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
string data = @"id=" + id + "&controlNumber=" + controlNumber;
byte[] dataStream = Encoding.UTF8.GetBytes(data);
request.ContentLength = dataStream.Length;
Stream newStream = request.GetRequestStream();
newStream.Write(dataStream, 0, dataStream.Length);
newStream.Close();
HttpWebResponse webResponse = (HttpWebResponse)request.GetResponse();
StreamReader sr = new StreamReader(webResponse.GetResponseStream());
string html = sr.ReadToEnd();

但随后不是导航到所需页面，而是显示初始页面并显示消息“错误的控制号”。

我究竟做错了什么？

4

1 回答 1

0

真的，我唯一需要做的就是设置 cookie 并使用 WebRequest 和 WebResponse 而不是 HPA 的 HtmlWeb。
这是有效的代码： 

        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
        request.Method = "GET";
        request.KeepAlive = true;
        CookieContainer cookies = new CookieContainer(); // instantiate cookie container

        request.CookieContainer = cookies;

        HttpWebResponse response = (HttpWebResponse)request.GetResponse();
        var stream = response.GetResponseStream();

        // Calculate control number...

        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(newUrl);
        request.Method = "POST";
        request.ContentType = "application/x-www-form-urlencoded";
        string data = @"id=" + id + "&controlNumber=" + controlNumber;
        byte[] dataStream = Encoding.UTF8.GetBytes(data);
        request.ContentLength = dataStream.Length;

        request.CookieContainer = cookies;

        Stream newStream = request.GetRequestStream();
        newStream.Write(dataStream, 0, dataStream.Length);
        newStream.Close();
        HttpWebResponse webResponse = (HttpWebResponse)request.GetResponse();
        StreamReader sr = new StreamReader(webResponse.GetResponseStream());
        string html = sr.ReadToEnd();
于 2013-07-30T11:29:57.657 回答