2

我正在使用 scrapy 进行带有此 URL 的抓取项目https://www.walmart.ca/en/clothing-shoes-accessories/men/mens-tops/N-2566+11

我尝试使用 url 并在 shell 中打开它，但出现 430 错误，所以我在标题中添加了一些设置，如下所示：

scrapy shell -s COOKIES_ENABLED=1 -s USER_AGENT='Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0' " https://www.walmart.ca/en/clothing-shoes -配件/男士/男士上衣/N-2566+11 "

它得到了“200”页面，但是一旦我使用视图（响应），它就会将我引导到一个页面，上面写着：对不起！您的网络浏览器不接受 cookie。

这是日志的屏幕截图： 在此处输入图像描述

4

4 回答 4

3

你应该有

COOKIES_ENABLED = True

在你的settings.py文件中。

另见

COOKIES_DEBUG = True

要调试 cookie，您将看到每个响应/请求分别传入/传出的 cookie。

于 2017-06-15T12:56:16.460 回答
1

尝试发送所有必需的标头。

headers = {
    'dnt': '1',
    'accept-encoding': 'gzip, deflate, sdch, br',
    'accept-language': 'en-US,en;q=0.8',
    'upgrade-insecure-requests': '1',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36',
    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'cache-control': 'max-age=0',
    'authority': 'www.walmart.ca',
    'cookie': 'JSESSIONID=E227789DA426B03664F0F5C80412C0BB.restapp-108799501-8-112264256; cookieLanguageType=en; deliveryCatchment=2000; marketCatchment=2001; zone=2; originalHttpReferer=; walmart.shippingPostalCode=V5M2G7; defaultNearestStoreId=1015; walmart.csrf=6f635f71ab4ae4479b8e959feb4f3e81d0ac9d91-1497631184063-441217ff1a8e4a311c2f9872; wmt.c=0; userSegment=50-percent; akaau_P1=1497632984~id=bb3add0313e0873cf64b5e0a73e3f5e3; wmt.breakpoint=d; TBV=7; ENV=ak-dal-prod; AMCV_C4C6370453309C960A490D44%40AdobeOrg=793872103%7CMCIDTS%7C17334',
    'referer': 'https://www.walmart.ca/en/clothing-shoes-accessories/men/mens-tops/N-2566+11',
}

yield Request(url = 'https://www.walmart.ca/en/clothing-shoes-accessories/men/mens-tops/N-2566+11', headers=headers)

您可以像这样以您的方式实现，而不是使用start_urls我推荐的start_requests()方法。它易于阅读。

class EasySpider(CrawlSpider): 
    name = 'easy' 

    def start_requests(self):
        headers = {
        'dnt': '1',
        'accept-encoding': 'gzip, deflate, sdch, br',
        'accept-language': 'en-US,en;q=0.8',
        'upgrade-insecure-requests': '1',
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36',
        'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
        'cache-control': 'max-age=0',
        'authority': 'www.walmart.ca',
        'cookie': 'JSESSIONID=E227789DA426B03664F0F5C80412C0BB.restapp-108799501-8-112264256; cookieLanguageType=en; deliveryCatchment=2000; marketCatchment=2001; zone=2; originalHttpReferer=; walmart.shippingPostalCode=V5M2G7; defaultNearestStoreId=1015; walmart.csrf=6f635f71ab4ae4479b8e959feb4f3e81d0ac9d91-1497631184063-441217ff1a8e4a311c2f9872; wmt.c=0; userSegment=50-percent; akaau_P1=1497632984~id=bb3add0313e0873cf64b5e0a73e3f5e3; wmt.breakpoint=d; TBV=7; ENV=ak-dal-prod; AMCV_C4C6370453309C960A490D44%40AdobeOrg=793872103%7CMCIDTS%7C17334',
        'referer': 'https://www.walmart.ca/en/clothing-shoes-accessories/men/mens-tops/N-2566+11',
        }       

        yield Request(url = 'https://www.walmart.ca/en/clothing-shoes-accessories/men/m‌​ens-tops/N-2566+11', callback = self.parse_item, headers = headers)

        def parse_item(self, response): 
            i = CravlingItem() 
            i['title'] = " ".join( response.xpath('//a/text()').extract()).strip() 
            yield i
于 2017-06-16T16:41:15.503 回答
1

如果网页需要点击接受cookies，您可以使用FormRequest.from_response

这是 Google 同意页面的示例

def start_requests(self):
  yield Request(
    "https://google.com/",
    callback=self.parse_consent,
  )

def parse_consent(self, response):
  yield FormRequest.from_response(
    response,
    clickdata={"value": "I agree"},
    callback=self.parse_query,
    dont_filter=True,
  )

def parse_query(self, response):
  for keyword in self.keywords:
    yield Request(
      <google_url_to_parse>,
      callback=<your_callback>,
      dont_filter=True,
    )

请注意，值clickdata可能会根据您的位置/语言而有所不同，您应该将“我同意”更改为正确的值。

于 2021-10-20T10:01:46.673 回答
0

我可以确认该COOKIES_ENABLED设置无助于修复错误。相反，使用以下 googlebotUSER_AGENT使其工作：

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; http://www.google.com/bot.html) Chrome/W.X.Y.Z‡ Safari/537.36

感谢制作此脚本的人，该脚本使用该用户代理发出请求：https ://github.com/juansimon27/scrapy-walmart/blob/master/product_scraping/spiders/spider.py

于 2020-08-30T06:20:04.080 回答