0

我有一个工作蜘蛛抓取图像 URL 并将它们放在 scrapy.Item 的 image_urls 字段中。我有一个继承自 ImagesPipeline 的自定义管道。当特定 URL 返回非 200 http 响应代码时（例如 401 错误）。例如，在日志文件中，我发现

WARNING:scrapy.pipelines.files:File (code: 404): Error downloading file from <GET http://a.espncdn.com/combiner/i%3Fimg%3D/i/headshots/tennis/players/full/425.png> referred in <None>
WARNING:scrapy.pipelines.files:File (code: 307): Error downloading file from <GET http://www.fansshare.com/photos/rogerfederer/federer-roger-federer-406468306.jpg> referred in <None>

但是，我无法在函数的自定义图像管道中捕获错误代码404、307item_completed()等：

def item_completed(self, results, item, info):

    image_paths = []
    for download_status, x in results:
        if download_status:
            image_paths.append(x['path'])
            item['images'] = image_paths  # update item image path
            item['result_download_status'] = 1
        else:
            item['result_download_status'] = 0
            #x.printDetailedTraceback()
            logging.info(repr(x)) # x is a twisted failure object

    return item

在files.pymedia_downloaded()的函数内部挖掘 scrapy 源代码，我发现对于非 200 响应代码，会记录一个警告（解释上述 WARNING 行），然后引发 a。FileException

if response.status != 200:
        logger.warning(
            'File (code: %(status)s): Error downloading file from '
            '%(request)s referred in <%(referer)s>',
            {'status': response.status,
             'request': request, 'referer': referer},
            extra={'spider': info.spider}
        )

        raise FileException('download-error')

我还如何访问此响应代码，以便我可以在 item_completed() 函数的管道中处理它？

4

2 回答 2

1

如果您对异步编程和 Twisted 回调和 errbacks 不太熟悉，您很容易与 Scrapy 的媒体管道中链接的所有这些方法混淆，因此在您的案例中，基本想法是覆盖media_downloaded这样的方式来处理这样的非 200 响应（只是快速而肮脏的 PoC）：

class MyPipeline(ImagesPipeline):

    def media_downloaded(self, response, request, info):
        if response.status != 200:
            return {'url': request.url, 'status': response.status}
        super(MyPipeline, self).media_downloaded(response, request, info)

    def item_completed(self, results, item, info):
        image_paths = []
        for download_status, x in results:
            if download_status:
                if not x.get('status', False):
                    # Successful download
                else:
                    # x['status'] contains non-200 response code
于 2017-01-20T10:34:47.337 回答
0

捕获非 200 响应代码的正确方法似乎是继承 media_downloaded 但调用父函数并捕获异常。这是有效的代码：

    def media_downloaded(self, response, request, info):
    try:
        resultdict = super(MyPipeline, self).media_downloaded(response, request, info)
        resultdict['status'] = response.status
        logging.warning('No Exception : {}'.format(response.status))
        return resultdict
    except FileException as exc:
        logging.warning('Caused Exception : {} {}'.format(response.status, str(exc)))
        return {'url': request.url, 'status': response.status}

响应代码可以在 item_completed() 内部处理

def item_completed(self, results, item, info):
    image_paths = []
    for download_status, x in results:
        if x.get('status', True):
            item['result_download_status'] = x['status'] # contains non-200 response code
            if x['status'] == 200:
                image_paths.append(x['path'])
                item['images'] = image_paths  # update item image path
于 2017-01-25T17:04:25.613 回答