9

我无法使用 urllib2 打开一个特定的 url。同样的方法适用于其他网站，例如“http://www.google.com”，但不适用于本网站（在浏览器中也可以正常显示）。

我的简单代码：

from BeautifulSoup import BeautifulSoup
import urllib2

url="http://www.experts.scival.com/einstein/"
response=urllib2.urlopen(url)
html=response.read()
soup=BeautifulSoup(html)
print soup

谁能帮我让它工作？

这是我得到的错误：

Traceback (most recent call last):
  File "/Users/jontaotao/Documents/workspace/MedicalSchoolInfo/src/AlbertEinsteinCollegeOfMedicine_SciValExperts/getlink.py", line 12, in <module>
    response=urllib2.urlopen(url);
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 400, in open
    response = meth(req, response)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 513, in http_response
    'http', request, response, code, msg, hdrs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 432, in error
    result = self._call_chain(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 372, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 619, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 400, in open
    response = meth(req, response)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 513, in http_response
    'http', request, response, code, msg, hdrs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 438, in error
    return self._call_chain(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 372, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 521, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 404: Not Found

谢谢

4

3 回答 3

9

我刚试过这个并收到 404 代码和页面返回。

猜测它正在执行用户代理检测，无论是偶然还是故意不向 python urllib 提供内容。

澄清一下urllib，我收到了urlopen返回的带有 404 代码和 HTML 内容的响应对象。提出了urllib2.urlopen一个urllib2.HTTPError例外。

我建议您尝试将您的用户代理设置为看起来像浏览器的东西。这里有一个关于这个的问题：Changing user agent on urllib2.urlopen

于 2012-09-06T14:42:07.203 回答
4

您可以使用try except捕获错误 

try:
    u = urllib2.urlopen(req)
except urllib2.HTTPError, e:
    print e.code
    print e.msg
    return
于 2015-08-14T12:13:57.543 回答
0

嗯...您确定该 URL 有效吗？尝试“http://www.google.com” 我有类似的代码，并且 urllib 没有问题。或者您可以使用 try - except 语句来查看错误的详细信息。当然，MattH 的回答与事实非常相似 :)

于 2012-09-06T20:13:11.707 回答