是否可以让 Selenium 抓取 TLD 并逐步导出找到的任何 404 列表?
我在 Windows 机器上停留了几个小时,想在回到 *nix 的舒适环境之前运行一些测试......
是否可以让 Selenium 抓取 TLD 并逐步导出找到的任何 404 列表?
我在 Windows 机器上停留了几个小时,想在回到 *nix 的舒适环境之前运行一些测试......
我不太了解 Python,也不了解它的任何常用库,但我可能会做这样的事情(使用 C# 代码作为示例,但这个概念应该适用):
// WARNING! Untested code here. May not completely work, and
// is not guaranteed to even compile.
// Assume "driver" is a validly instantiated WebDriver instance
// (browser used is irrelevant). This API is driver.get in Python,
// I think.
driver.Url = "http://my.top.level.domain/";
// Get all the links on the page and loop through them,
// grabbing the href attribute of each link along the way.
// (Python would be driver.find_elements_by_tag_name)
List<string> linkUrls = new List<string>();
ReadOnlyCollection<IWebElement> links = driver.FindElement(By.TagName("a"));
foreach(IWebElement link in links)
{
// Nice side effect of getting the href attribute using GetAttribute()
// is that it returns the full URL, not relative ones.
linkUrls.Add(link.GetAttribute("href"));
}
// Now that we have all of the link hrefs, we can test to
// see if they're valid.
List<string> validUrls = new List<string>();
List<string> invalidUrls = new List<string>();
foreach(string linkUrl in linkUrls)
{
HttpWebRequest request = WebRequest.Create(linkUrl) as HttpWebRequest;
request.Method = "GET";
// For actual .NET code, you'd probably want to wrap this in a
// try-catch, and use a null check, in case GetResponse() throws,
// or returns a type other than HttpWebResponse. For Python, you
// would use whatever HTTP request library is common.
// Note also that this is an extremely naive algorithm for determining
// validity. You could just as easily check for the NotFound (404)
// status code.
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
if (response.StatusCode == HttpStatusCode.OK)
{
validUrls.Add(linkUrl);
}
else
{
invalidUrls.Add(linkUrl);
}
}
foreach(string invalidUrl in invalidUrls)
{
// Here is where you'd log out your invalid URLs
}
此时,您有一个有效和无效 URL 的列表。您可以将这一切包装成一个方法,您可以将 TLD URL 传递到该方法中,并使用每个有效 URL 递归调用它。这里的关键是您没有使用 Selenium 来实际确定链接的有效性。如果您真的在进行递归爬网,您不会希望“单击”链接以导航到下一页。相反,您希望直接导航到页面上的链接。
您可能会采取其他方法,例如通过代理运行所有内容,并以这种方式捕获响应代码。这在一定程度上取决于您期望如何构建您的解决方案。