c# - 如何在 ABOT C# Web Crawler 中获取 html 输出页面？

Question

我正在尝试在 c# 中使用 ABOT 制作网络爬虫。我搜索了很多示例并添加了 ABOT 网络爬虫。由此我只能得到日志输出而不是 Html 页面输出。我只想得到 html 页面输出。因为 HTML 输出是 HTML 敏捷工具的输入。帮助我从 C# 中的 ABOT 网络爬虫获取 HTML 输出。谢谢。

Answer 1

在快速入门页面上进行了解释

//Create an instance of the crawler and subscribe to the PageCrawlCompleted event
PoliteWebCrawler crawler = new PoliteWebCrawler();
crawler.PageCrawlCompleted += crawler_ProcessPageCrawlCompleted;

//The event handler method
void crawler_ProcessPageCrawlCompleted(object sender, PageCrawlCompletedArgs e)
{
    CrawledPage crawledPage = e.CrawledPage;

    if (crawledPage.WebException != null || crawledPage.HttpWebResponse.StatusCode != HttpStatusCode.OK)
        Console.WriteLine("Crawl of page failed {0}", crawledPage.Uri.AbsoluteUri);
    else
        Console.WriteLine("Crawl of page succeeded {0}", crawledPage.Uri.AbsoluteUri);


    //crawledPage.Content.Text //raw html
    //crawledPage.HtmlDocument //lazy loaded html agility pack object (HtmlAgilityPack.HtmlDocument)
    //crawledPage.CSDocument   //lazy loaded cs query object (CsQuery.Cq)
}

Answer 2

void crawler_ProcessPageCrawlCompleted(object sender, PageCrawlCompletedArgs e)
{
    CrawledPage crawledPage = e.CrawledPage;
    crawledPage.Content.Text // HTML

}

Answer 3

获取 htmlpage 仅使用：

crawledPage.Content

函数内部

`static void crawler_ProcessPageCrawlCompleted(object sender, PageCrawlCompletedArgs e)`

例如：

static void crawler_ProcessPageCrawlCompleted(object sender, PageCrawlCompletedArgs e)
    {
        CrawledPage crawledPage = e.CrawledPage;

        if (crawledPage.WebException != null || crawledPage.HttpWebResponse.StatusCode != HttpStatusCode.OK)
            Console.WriteLine("Crawl of page failed {0}", crawledPage.Uri.AbsoluteUri);
        else
            Console.WriteLine("Crawl of page succeeded {0}", crawledPage.Uri.AbsoluteUri);

        if (string.IsNullOrEmpty(crawledPage.Content.Text))
            Console.WriteLine("Page had no content {0}", crawledPage.Uri.AbsoluteUri);

        var htmlAgilityPackDocument = crawledPage.HtmlDocument; //Html Agility Pack parser
        var angleSharpHtmlDocument = crawledPage.AngleSharpHtmlDocument; 

        //get content

        Console.WriteLine(crawledPage.Content);


    }