0

我在抓取 javascript 网站时遇到了一些问题。我正在使用scrapy-splash和docker将js渲染为html来抓取。

import scrapy
from scrapy_splash import SplashRequest
class MySpider (scrapy.Spider):
       name = 'spd'
       start_urls = ['http://example.com']

       def start_requests (self):
            for url in self.start_urls:
                yield SplashRequest(url, self.parse, endpoint='render.html', args={'wait':0.5},)
       def parse (self, response):
            for href in response.xpath('xpath'):
                   yield {'info': href.xpath('xpath')}

这是我的终端输出的内容：

  2017-05-30 13:20:51 [scrapy.core.engine] INFO: Spider opened
  2017-05-30 13:20:51 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
  2017-05-30 13:20:51 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
  2017-05-30 13:20:51 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://example.com via http://192.168.99.100:8050/render.html> (failed 1 times): Connection was refused by other side: 61: Connection refused.
  2017-05-30 13:20:51 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://example.com via http://192.168.99.100:8050/render.html> (failed 2 times): Connection was refused by other side: 61: Connection refused.
  2017-05-30 13:20:51 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET http://example.com via http://192.168.99.100:8050/render.html> (failed 3 times): Connection was refused by other side: 61: Connection refused.
  2017-05-30 13:20:51 [scrapy.core.scraper] ERROR: Error downloading <GET http://example.com via http://192.168.99.100:8050/render.html>: Connection was refused by other side: 61: Connection refused.
  2017-05-30 13:20:51 [scrapy.core.engine] INFO: Closing spider (finished)
4

1 回答 1

0

以下日志消息表明 Splash docker 容器未运行或未在预期端口上侦听。

DEBUG: Retrying <GET http://example.com via http://192.168.99.100:8050/render.html> (failed 1 times): Connection was refused by other side: 61: Connection refused.
DEBUG: Retrying <GET http://example.com via http://192.168.99.100:8050/render.html> (failed 2 times): Connection was refused by other side: 61: Connection refused.
DEBUG: Gave up retrying <GET http://example.com via http://192.168.99.100:8050/render.html> (failed 3 times): Connection was refused by other side: 61: Connection refused.

要查看 Docker 容器的状态，包括已退出的容器，请尝试运行：

sudo docker ps -a | grep scrapinghub/splash
于 2020-03-31T06:23:14.683 回答