0

我对 HttpWebRequest 类有疑问。
我正在尝试获取网站的源代码：
http ://www.filmweb.pl/film/Igrzyska+%C5%9Bmierci%3A+Kosog%C5%82os.+Cz%C4%99%C5%9B%C4%87 +1-2014-626983
但我总是收到错误消息：

System.Net.WebException occurred
  HResult=-2146233079
  Message=Too many automatic redirections were attempted.
  Source=System
  StackTrace:
       at System.Net.HttpWebRequest.GetResponse()
       at ProjectName.ClassName.MethodName(String urlAddress)
  InnerException:

那是我的代码：

Uri uri = new Uri(@"http://www.filmweb.pl/film/Igrzyska+%C5%9Bmierci%3A+Kosog%C5%82os.+Cz%C4%99%C5%9B%C4%87+1-2014-626983");
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();

我使用 Fiddler Web Debugger 工具将 Firefox 请求与我的 C# .NET 请求进行比较，但仍然没有答案。

火狐：

GET http://www.filmweb.pl/film/Igrzyska+%C5%9Bmierci%3A+Kosog%C5%82os.+Cz%C4%99%C5%9B%C4%87+1-2014-626983 HTTP/1.1
Host: www.filmweb.pl
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:43.0) Gecko/20100101 Firefox/43.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive


HTTP/1.1 200 OK
Cache-Control: private, no-cache, no-store, max-age=0, must-revalidate, proxy-revalidate
Content-Type: text/html;charset=UTF-8
Content-Language: pl-PL
Transfer-Encoding: chunked
Date: Wed, 07 Oct 2015 13:36:31 GMT
X-Cache: HIT from blade110.non.3dart.com
X-Cache-Hits: 116
Server: Apache

C# .NET：

GET http://www.filmweb.pl/film/Igrzyska+%C5%9Bmierci:+Kosog%C5%82os.+Cz%C4%99%C5%9B%C4%87+1-2014-626983 HTTP/1.1
Host: www.filmweb.pl
Connection: Keep-Alive


HTTP/1.1 301 Moved Permanently
Cache-Control: private, no-cache, no-store, max-age=0, must-revalidate, proxy-revalidate
Content-Type: text/html;charset=UTF-8
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Language: pl-PL
Location: /film/Igrzyska+%C5%9Bmierci%3A+Kosog%C5%82os.+Cz%C4%99%C5%9B%C4%87+1-2014-626983
Content-Length: 0
Accept-Ranges: bytes
Date: Wed, 07 Oct 2015 13:34:51 GMT
X-Cache: MISS from blade712.non.3dart.com
Server: Apache

我已经阅读了其他帖子并通过不同的方式更新了我的代码，例如。

request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.TransferEncoding = "gzip, deflate";
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:43.0) Gecko/20100101 Firefox/43.0";

request.Referer = "http://www.filmweb.pl/film/Igrzyska+%C5%9Bmierci%3A+Kosog%C5%82os.+Cz%C4%99%C5%9B%C4%87+1-2014-626983";
request.KeepAlive = true;
request.AllowAutoRedirect = true;
request.MaximumAutomaticRedirections = 250;
request.Proxy = null;
request.UseDefaultCredentials = true;

CookieContainer cookieContainer = new CookieContainer();
request.CookieContainer = cookieContainer;

但没有任何效果：-/

有人可以帮我解决这个问题吗？

4

1 回答 1

0

在获取深层链接之前，您需要在网站加载时拥有初始 cookie。

以下代码适用于我：

//  cookies
CookieContainer cookieContainer = new CookieContainer();

// make one call to the root of the website
// to get the cookies set
Uri uri = new Uri(@"http://www.filmweb.pl");
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.CookieContainer = cookieContainer;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();

using(var s = response.GetResponseStream())
{
   using(var sr = new StreamReader(s)) 
   {
      // linqpad
      sr.ReadToEnd().Dump(); // to check for errors
   }
}

// we have cookies now
// do the deep link fetch
uri = new Uri(@"http://www.filmweb.pl/film/Igrzyska+%C5%9Bmierci%3A+Kosog%C5%82os.+Cz%C4%99%C5%9B%C4%87+1-2014-626983");
request = (HttpWebRequest)WebRequest.Create(uri);
request.CookieContainer = cookieContainer;
response = (HttpWebResponse)request.GetResponse();

//store the result
using(var f = File.Create("C:\\temp\\pl.txt"))
{
    response.GetResponseStream().CopyTo(f);
}

确保在抓取网站时遵守他们的许可和使用政策。不要做任何超出合理使用或违反任何受版权保护的材料的事情。

于 2015-10-07T14:04:10.320 回答