-1

我有一个脚本（见下文）可以检查（或多或少）网站上的链接位置，它工作正常，但只要链接所在的源 URL 不是 200 响应，它就会退出，我只想让它跳转提前到下一个或返回一些消息“错误”，甚至更好地返回 http 状态代码。我需要一个快速的解决方案，如果有人可以帮助我，那就太棒了:)

URLs.csv= 包含指向某个页面的链接的网站列表
domain.com= 检查链接是否存在的域，如果是，则它大致位于何处。

import csv
from lxml import html

with open('URLs.csv', 'r') as csvfile:
    urls = [row[0] for row in csv.reader(csvfile)]

for url in urls:
    print url

    doc = html.parse(url)
    if doc.xpath('//a[contains(@href,"domain.com")]'):
        for anchor_node in doc.xpath('//a[contains(@href,"finanzen.de")]'):
            if anchor_node.xpath('./ancestor::div[contains(@class, "sidebar")]'):
                print 'Sidebar'
            elif anchor_node.xpath('./parent::div[contains(@class, "widget")]'):
                print 'Sidebar'
            elif anchor_node.xpath('./ancestor::div[contains(@id, "sidebar")]'):
                print 'Sidebar'            
            elif anchor_node.xpath('./ancestor::div[contains(@class, "comment")]'):
                print 'Kommentar'
            elif anchor_node.xpath('./ancestor::div[contains(@id, "comment")]'):
                print 'Kommentar'
            elif anchor_node.xpath('./ancestor::div[contains(@class, "foot")]'):
                print "Footer"
            elif anchor_node.xpath('./ancestor::div[contains(@id, "foot")]'):
                print "Footer"
            elif anchor_node.xpath('./ancestor::div[contains(@class, "post")]'):
                print "Contextual"
            else:
                print 'Unidentified Link'
        else:
            print 'Link is Dead'

Python-Shell

            Python 2.7.4 (default, Apr  6 2013, 19:55:15) [MSC v.1500 64 bit (AMD64)]
            Type "help", "copyright", "credits" or "license" for more information.
            [evaluate Linkidentifizierung.py]
            http://urlnotworking.com/broken.html
            Rückverfolgung (innerste zuletzt):
            File "C:\Program Files (x86)\Wing IDE 101 4.1\src\debug\tserver\_sandbox.py",     line 11, in <module>
            File "C:\Python27\Lib\site-packages\lxml\html\__init__.py", line 735, in parse
        return etree.parse(filename_or_url, parser, base_url=base_url, **kw)
            File "C:\Python27\Lib\site-packages\lxml\etree.pyd", line 3197, in  lxml.etree.parse (src\lxml\lxml.etree.c:64726)
       H‹GH‹ÏÿP0H…ÛtHƒÿu
            File "C:\Python27\Lib\site-packages\lxml\etree.pyd", line 1571, in lxml.etree._parseDocument (src\lxml\lxml.etree.c:92363)
            File "C:\Python27\Lib\site-packages\lxml\etree.pyd", line 1600, in lxml.etree._parseDocumentFromURL (src\lxml\lxml.etree.c:92647)
            File "C:\Python27\Lib\site-packages\lxml\etree.pyd", line 1500, in lxml.etree._parseDocFromFile (src\lxml\lxml.etree.c:91710)
            File "C:\Python27\Lib\site-packages\lxml\etree.pyd", line 1047, in lxml.etree._BaseParser._parseDocFromFile (src\lxml\lxml.etree.c:88610)
            File "C:\Python27\Lib\site-packages\lxml\etree.pyd", line 577, in lxml.etree._ParserContext._handleParseResultDoc (src\lxml\lxml.etree.c:84019)
            File "C:\Python27\Lib\site-packages\lxml\etree.pyd", line 676, in lxml.etree._handleParseResult (src\lxml\lxml.etree.c:85122)
           File "C:\Python27\Lib\site-packages\lxml\etree.pyd", line 614, in lxml.etree._raiseParseError (src\lxml\lxml.etree.c:84417)
    IOError: Error reading file 'http://urlnotworking.com/broken.html':  failed to load HTTP resource
4

1 回答 1

0 
print url

try:
  html.parse(url)
except Exception, e:
  print "something went wrong: %s" % e
  continue

if doc.xpath....

我对 lxml.html 库不够熟悉，无法了解您是否可以更详细地了解 URL 加载失败的原因。（提示——我会使用“请求”库来加载我的 url，然后将结果传递给 lxml 解析器。）

于 2013-06-02T22:38:36.260 回答