-1

我有一个使用scrapy_selenium 的正在运行的解决方案，用于带有javascript 加载的站点。正如您在下面的代码中看到的那样，在使用 parseDetails 生成 detailPage 时使用了 SeleniumRequest -

但是，当我需要在我的主页上准备好 SeleniumRequest（而不仅仅是下面的详细信息页面）时，我该怎么办？

在这种情况下，我该如何使用 SeleniumRequest？

import scrapy
from scrapy_selenium import SeleniumRequest

class ZoosSpider(scrapy.Spider):
  name = 'zoos'
  allowed_domains = ['www.tripadvisor.co.uk']
  start_urls = [
                "https://www.tripadvisor.co.uk/Attractions-g186216-Activities-c53-a_allAttractions.true-United_Kingdom.html"
                ]  
  existList = []  

  def parse(self, response):
    tmpSEC = response.xpath("//section[@data-automation='AppPresentation_SingleFlexCardSection']")
    for elem in tmpSEC:
      link = response.urljoin(elem.xpath(".//a/@href").get())   
      yield SeleniumRequest(
        url=link, 
        wait_time= 10,        
        callback=self.parseDetails)  

  def parseDetails(self, response):
    tmpName = response.xpath("//h1[@data-automation='mainH1']/text()").get()  
    tmpLink = response.xpath("//div[@class='Lvkmj']/a/@href").getall()    
    tmpURL = tmpTelnr = tmpMail = "N/A"

    yield {
      "Name": tmpName,
      "URL": tmpURL,
    }
4

1 回答 1

0

您可以使用自己的函数start_requests()来启动第一个请求。

class ZoosSpider(scrapy.Spider):

    def start_requests(self):
        for link in self.start_urls:
            yield SeleniumRequest(
                url=link, 
                wait_time= 10,        
                callback=self.parse)

请参阅文档中的第一点：Spider

The first requests to perform are obtained by calling the start_requests() method  
which (by default) generates Request for the URLs specified in the start_urls 
and the parse method as callback function for the Requests.
于 2021-12-01T20:42:55.447 回答