0

我在 scrapy 中使用 ProxyMiddleware ，它会抛出不可追踪的错误。这是回溯：

Unhandled error in Deferred:   
[twisted] CRITICAL: Unhandled error in Deferred:
[twisted] CRITICAL:
Traceback (most recent call last):

File "/usr/local/lib/python3.5/dist-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
result = g.send(result)
File "/usr/local/lib/python3.5/dist-packages/scrapy/crawler.py", line 80, in crawl   
self.engine = self._create_engine()
File "/usr/local/lib/python3.5/dist-packages/scrapy/crawler.py", line 105, in _create_engine
return ExecutionEngine(self, lambda _: self.stop())
File "/usr/local/lib/python3.5/dist-packages/scrapy/core/engine.py", line 69, in init
self.downloader = downloader_cls(crawler)
File "/usr/local/lib/python3.5/dist-packages/scrapy/core/downloader/init.py", line 88, in init    
self.middleware = DownloaderMiddlewareManager.from_crawler(crawler)
File "/usr/local/lib/python3.5/dist-packages/scrapy/middleware.py", line 58, in from_crawler
return cls.from_settings(crawler.settings, crawler)    
File "/usr/local/lib/python3.5/dist-packages/scrapy/middleware.py", line 40, in from_settings
mw = mwcls()    
TypeError: init() missing 1 required positional argument: 'arg'

这里是文件：

设置.py

HTTP_PROXY = 'http://127.0.0.1:8123'

DOWNLOADER_MIDDLEWARES = {
#Tor Middleware
'RaidForums.middlewares.ProxyMiddleware': 400
 }

 SPIDER_MIDDLEWARES = {
'RaidForums.middlewares.ProxyMiddleware': 400
 }

middelwares.py

class ProxyMiddleware(object):
      def process_request(self, request, spider):
          request.meta['proxy'] = settings['HTTP_PROXY']
      def __init__(self, arg):
          super(ProxyMiddleware, self).__init__()
          self.arg = arg
4

1 回答 1

0

在您的 settings.py 中，启用scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware比您的自定义（401 或更高）更高的优先级值，因此它比您的自定义中间件更远离引擎。

在您的 middlewares.py 中，您可以使用 访问设置spider.settings[]，settings[]但无法正常工作。我还认为您可以__init__()完全摆脱 ProxyMiddleware。

请参阅scrapinghub 中的此示例，以了解您的距离。

于 2018-10-08T03:05:40.280 回答