实际上,我在用户代理中为 Bot 设备获取了 Bot Fraud,这似乎是 comscore 爬虫。
Mozilla/5.0(兼容;近端;+https://www.comscore.com/Web-Crawler)
如何停止从代理 IP 进行机器人访问。
非常感谢
如果你使用 cloudflare,你可以用它来删除它。
(http.user_agent contains "Yandex")
or (http.user_agent contains "muckrack")
or (http.user_agent contains "Qwantify")
or (http.user_agent contains "Sogou")
or (http.user_agent contains "BUbiNG")
or (http.user_agent contains "knowledge")
or (http.user_agent contains "CFNetwork")
or (http.user_agent contains "Scrapy")
or (http.user_agent contains "SemrushBot")
or (http.user_agent contains "AhrefsBot")
or (http.user_agent contains "Baiduspider")
or (http.user_agent contains "python-requests")
or ((http.user_agent contains "crawl")
or (http.user_agent contains "Crawl")
or (http.user_agent contains "bot" and not http.user_agent contains "bingbot" and not http.user_agent contains "Google" and not http.user_agent contains "Twitter")
or (http.user_agent contains "Bot" and not http.user_agent contains "Google")
or (http.user_agent contains "Spider")
or (http.user_agent contains "spider")
and not cf.client.bot)
在 cloudflare 上配置:
检查块: