c# - Asp.net Request.Browser.Crawler - 动态爬虫列表？

Question

我了解了为什么 Request.Browser.Crawler 在 C# 中总是 False ( http://www.digcode.com/default.aspx?page=ed51cde3-d979-4daf-afae-fa6192562ea9&article=bc3a7a4f-f53e-4f88-8e9c-c9337f6c05a0 ) .

有没有人使用某种方法来动态更新爬虫的列表，所以 Request.Browser.Crawler 真的有用吗？

Answer 1

我对Ocean 的 Browsercaps提供的结果很满意。它支持微软的配置文件没有检测到的爬虫。它甚至会解析出您网​​站上的爬虫版本，而不是我真的需要那种详细程度。

Answer 2

您可以对照Request.UserAgent.

Peter Bromberg 写了一篇关于在 ASP.NET 中编写ASP.NET 请求记录器和爬虫杀手的好文章。

这是他在Logger课堂上使用的方法：

public static bool IsCrawler(HttpRequest request)
{
   // set next line to "bool isCrawler = false; to use this to deny certain bots
   bool isCrawler = request.Browser.Crawler;
   // Microsoft doesn't properly detect several crawlers
   if (!isCrawler)
   {
       // put any additional known crawlers in the Regex below
       // you can also use this list to deny certain bots instead, if desired:
       // just set bool isCrawler = false; for first line in method 
       // and only have the ones you want to deny in the following Regex list
       Regex regEx = new Regex("Slurp|slurp|ask|Ask|Teoma|teoma");
       isCrawler = regEx.Match(request.UserAgent).Success;
   }
   return isCrawler;
}