0

我需要将一个类的登录 URL 传递给蜘蛛类并对其执行网络抓取。

import quotes as q
import scrapy
from scrapy.crawler import CrawlerProcess
class ValidateURL:

    def checkURL(self,urls):
        try:    
            if(urls):
                for key, value in urls.items():
                    if value['login_details']:
                        self.runScrap(value)                                      

        except:
            return False

    def runScrap(self,data):       
            if data:
               process = CrawlerProcess()
# here I'm passing a URL (mail.google.com)
               process.crawl(q.QuotesSpider, passed_url=data['url'])
               process.start()



# -*- coding: utf-8 -*-
from scrapy import Spider
from scrapy.http import FormRequest
from scrapy.utils.response import open_in_browser
import sys
import logging
from bs4 import BeautifulSoup
# import scrapy
# from scrapy.crawler import CrawlerProcess

logging.basicConfig(filename='app.log',level=logging.INFO)

class QuotesSpider(Spider):
    name = 'quotes'
    # I need to update this with passed variable
    start_urls = ('https://quotes.toscrape.com/login',)





    def parse(self, response):
        pass



    def scrape_pages(self, response):
      pass

我的代码是不言自明的，需要使用传递的参数更新超类变量。我该如何实施？我尝试使用self.passed_url，但只能在函数内部访问，没有得到更新。

4

1 回答 1

1

您需要将传递的参数名称与蜘蛛start_urls属性匹配。

根据文档，如果您不覆盖__init__蜘蛛的方法，则所有传递给蜘蛛类的参数都会映射到蜘蛛属性。因此，为了覆盖该start_urls属性，您需要发送准确的参数名称。

像这样的东西：

    def runScrap(self,data):       
        if data:
            process = CrawlerProcess()
            process.crawl(q.QuotesSpider, start_urls=[data['url']])
            process.start()

希望能帮助到你。

于 2020-04-19T19:33:00.013 回答