python - CrawlSpider 派生对象没有属性“状态”

Question

我尝试按照http://doc.scrapy.org/en/0.22/topics/jobs.html中的描述使用 spider.state ，但出现错误

MyCrawlSpider has no attribute 'state'

我尝试在CrawlSpider 派生类的init () 函数中使用它。这可能是问题吗？

class MyCrawlSpider(CrawlSpider):
    crawl_start = datetime.utcnow().isoformat()

    def __init__(self, *args, **kwargs):
        super(MyCrawlSpider, self).__init__(*args, **kwargs)

        if self.state.get('crawl_start'):
            crawl_start = self.state.get('crawl_start')
        else:
            self.state["crawl_start"] = crawl_start

我的目标是让 crawl_start 属性始终位于我的爬虫首先启动的 isoformat 日期时间字符串上，与 x 何时恢复开始无关

Answer 1

根据源代码，在信号处理程序中通过扩展state在蜘蛛上设置属性：scrapy.contrib.spiderstate.SpiderState spider_opened()

class SpiderState(object):
    """Store and load spider state during a scraping job"""

    ...

    def spider_closed(self, spider):
        if self.jobdir:
            with open(self.statefn, 'wb') as f:
                pickle.dump(spider.state, f, protocol=2)

    def spider_opened(self, spider):
        if self.jobdir and os.path.exists(self.statefn):
            with open(self.statefn, 'rb') as f:
                spider.state = pickle.load(f)
        else:
            spider.state = {}

信号在__init__()方法执行之后发送 -state蜘蛛实例上还没有属性 - 这就是您收到错误的原因。