python - 如何在解析函数中停止爬虫

Question

我正在使用scrapy来抓取搜索结果。我有一个变量search_page，它告诉我们在哪个页码。

我在parse函数中有那个变量。

现在我想要如果search_page > 500then 爬虫应该停止爬行

我怎样才能做到这一点

def parse(self, response):

    hxs = HtmlXPathSelector(response)
    sites = hxs.select('//div[@class="headline_area"]')
    items = []

    for site in sites[:5]:
        item = StackItem()
        log.msg(' LOOP' +str(ivar)+ '', level=log.ERROR)
        item['title'] ="yoo ma"
        request =  Request("blabla",  callback=self.test1)
        request.meta['item'] = item
        page_number = nextlink.split("&")[-3].split("=")[-1]
        if page_number > 500:
                 STOP
        ivar = ivar + 1
        yield request

Answer 1

https://scrapy.readthedocs.org/en/latest/topics/exceptions.html?highlight=closeSpider

从 scrapy.exceptions 导入 CloseSpider

    if int(page_number) > 500:
       raise CloseSpider('Search Exceeded 500')