c# - 使用 C# 提取 Web 文档

Question

我正在尝试使用 c# 从网页获取数据

到目前为止，这是我的代码：

WebBrowser wb = new WebBrowser();

wb.Url = new Uri("http://www.microsoft.com");
HtmlDocument doc = wb.Document;

MessageBox.Show(doc.ToString());

不幸的是 wb 仍然为空，并且 Url 属性永远不会被设置。

任何人都可以帮助我吗？

谢谢

Answer 1

我会使用WebClient类而不是 Web 浏览器。Web 浏览器类更适合与 UI 交互，而 WebClient 更适合与页面进行编程交互。这是一些示例代码：

private void sendMessage(JaxtrSmsMessage message)
{
    HttpWebRequest request;
    HttpWebResponse response;
    CookieContainer cookies;
    string url = "http://www.jaxtr.com/user/login.jsp";

    try
    {
        request = (HttpWebRequest)WebRequest.Create(url);
        request.AllowAutoRedirect = true;
        request.CookieContainer = new CookieContainer();
        response = (HttpWebResponse)request.GetResponse();
        if (response.StatusCode == HttpStatusCode.OK)
        {
            StringBuilder sb = new StringBuilder();
            StreamReader reader = new StreamReader(response.GetResponseStream());
            while (!reader.EndOfStream)
            {
                sb.AppendLine(reader.ReadLine());
            }

            //Get the hidden value out of the form.                
            String fp = Regex.Match(sb.ToString(), "\"__fp\"\\svalue=\"(([A-Za-z0-9+/=]){4}){1,19}\"", RegexOptions.None).Value;
            fp = fp.Substring(14);
            fp = fp.Replace("\"", String.Empty);


            cookies = request.CookieContainer;
            //response.Close();
            String requestString = "http://www.jaxtr.com/user/Login.action?tzOffset=6&navigateURL=&refPage=&jaxtrId=" + HttpUtility.UrlEncode(credentials.Username) + "&password=" + HttpUtility.UrlEncode(credentials.Password) + "&Login=Login&_sourcePage=%2Flogin.jsp&__fp="+HttpUtility.UrlEncode(fp);
            request = (HttpWebRequest)WebRequest.Create(requestString);
            request.CookieContainer = cookies; //added by myself

            response = (HttpWebResponse)request.GetResponse();
            Console.WriteLine("Response from login:" + response.StatusCode);

            String messageText = (message.TruncateMessage && message.MessageText.Length > JaxtrSmsMessage.MAX_MESSAGE_LENGTH ? message.MessageText.Substring(JaxtrSmsMessage.MAX_MESSAGE_LENGTH) : message.MessageText);

            String messageURL = "http://www.jaxtr.com/user/sendsms?CountryName=" + HttpUtility.UrlEncode(message.CountryName) + "&phone=" + HttpUtility.UrlEncode(message.DestinationPhoneNumber) + "&message=" + HttpUtility.UrlEncode(messageText) + "&bySMS=" + HttpUtility.UrlEncode(message.BySMS.ToString().ToLower());

            request = (HttpWebRequest)WebRequest.Create(messageURL);
            request.CookieContainer = cookies;
            response = (HttpWebResponse)request.GetResponse();

            Console.WriteLine("Response from send SMS command=" + response.StatusCode);

            StringBuilder output = new StringBuilder();

            using (Stream s = response.GetResponseStream())
            {
                StreamReader sr = new StreamReader(s);
                while (!sr.EndOfStream)
                {
                    output.AppendLine(sr.ReadLine());
                }
            }
            response.Close();
        }
        else
        {
            Console.WriteLine("Client was unable to connect!");
        }
    }
    catch (System.Exception e)
    {
        throw new SMSDeliveryException("Unable to deliver SMS message because "+e.Message, e);
    }
}

Answer 2

首先，您必须使用 URL 路径调用导航方法，并且您必须从 Web 浏览器控件中捕获 Document complete 事件

 webBrowser.DocumentCompleted += new WebBrowserDocumentCompletedEventHandler(webBrowser_DocumentCompleted);
 webBrowser.Navigate("http:\\www.microsoft.com");

然后使用文档完成事件获取加载的网页文档

 void webBrowser_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
    {             
          MessageBox.Show(webBrowser.DocumentText.ToString());
    }

希望这可以帮助...

Answer 3

我认为您忘记启动页面导航。看这里

但是也...

导航是一个异步过程。您需要将 Web 浏览器对象放在类范围内，然后像这样处理导航事件：

    private void webBrowser1_Navigated(object sender, WebBrowserNavigatedEventArgs e)
    {
        HtmlDocument doc = webBrowser1.Document;
        doc = webBrowser1.Document;
    }

您会看到 doc 在处理程序中将不为空。

Answer 4

尝试 的DocumentText属性HtmlDocument，而不是调用ToString()。