1

我正在尝试从使用 Ajax 的站点获取数据。页面加载，然后 Javascript 请求内容。有关详细信息，请参阅此页面：https ://www.tele2.no/mobiletelefon.aspx

问题是，当我尝试通过调用此 url 来模拟此过程时： https ://www.tele2.no/Services/Webshop/FilterService.svc/ApplyPhoneFilters

我收到 400 响应，告诉我该请求不被允许。这是我的代码：

# -*- coding: utf-8 -*-
import scrapy
import json

class Tele2Spider(scrapy.Spider):
    name = "tele2"
    #allowed_domains = ["tele2.no/mobiltelefon.aspx"]
    start_urls = (
        'https://www.tele2.no/mobiltelefon.aspx/',
    )

    def parse(self, response):
        url = 'https://www.tele2.no/Services/Webshop/FilterService.svc/ApplyPhoneFilters'
        my_data = "{filters: []}"
        req = scrapy.Request( url, method='POST', body=json.dumps(my_data), headers={'X-Requested-With': 'XMLHttpRequest','Content-Type':'application/json'}, callback=self.parser2)
        yield req

    def parser2(self, response):
      print "test"

我是scrapy和python的新手，所以我可能缺少一些明显的东西

4

1 回答 1

3

filters关键问题在于正文中缺少引号：

url = 'https://www.tele2.no/Services/Webshop/FilterService.svc/ApplyPhoneFilters'
req = scrapy.Request(url,
                     method='POST',
                     body='{"filters": []}',
                     headers={'X-Requested-With': 'XMLHttpRequest',
                              'Content-Type': 'application/json; charset=UTF-8'},
                     callback=self.parser2)
yield req

或者，您可以将其定义为字典，然后调用json.dumps()以将其转储为字符串：

params = {"filters": []}
req = scrapy.Request(url,
                     method='POST',
                     body=json.dumps(params),
                     headers={'X-Requested-With': 'XMLHttpRequest',
                              'Content-Type': 'application/json; charset=UTF-8'},
                     callback=self.parser2)

作为证明，这是它在控制台上给我的：

2014-12-30 12:30:38-0500 [tele2] DEBUG: Crawled (200) <GET https://www.tele2.no/mobiltelefon.aspx/> (referer: None) 
2014-12-30 12:30:42-0500 [tele2] DEBUG: Crawled (200) <POST https://www.tele2.no/Services/Webshop/FilterService.svc/ApplyPhoneFilters> (referer: https://www.tele2.no/mobiltelefon.aspx/) 
test
于 2014-12-30T17:31:46.750 回答