我有以下蜘蛛,它几乎应该发布到表单。我似乎无法让它工作。当我通过 Scrapy 执行此操作时,响应永远不会显示。有人能告诉我我哪里出了问题吗?
这是我的蜘蛛代码:
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
import scrapy
from scrapy.http import FormRequest
from scrapy.shell import inspect_response
class RajasthanSpider(scrapy.Spider):
name = "rajasthan"
allowed_domains = ["rajtax.gov.in"]
start_urls = (
'http://www.rajtax.gov.in/',
)
def parse(self, response):
return FormRequest.from_response(
response,
formname='rightMenuForm',
formdata={'dispatch': 'dealerSearch'},
callback=self.dealer_search_page)
def dealer_search_page(self, response):
yield FormRequest.from_response(
response,
formname='dealerSearchForm',
formdata={
"zone": "select",
"dealertype": "VAT",
"dealerSearchBy": "dealername",
"name": "ana"
}, callback=self.process)
def process(self, response):
inspect_response(response, self)
当我dealer_search_page()
用 Splash 替换 my 时:
def dealer_search_page(self, response):
yield FormRequest.from_response(
response,
formname='dealerSearchForm',
formdata={
"zone": "select",
"dealertype": "VAT",
"dealerSearchBy": "dealername",
"name": "ana"
},
callback=self.process,
meta={
'splash': {
'endpoint': 'render.html',
'args': {'wait': 0.5}
}
})
我收到以下警告:
2016-03-14 15:01:29 [scrapy] WARNING: Currently only GET requests are supported by SplashMiddleware; <POST http://rajtax.gov.in:80/vatweb/dealerSearch.do> will be handled without Splash
inspect_response()
并且程序在到达我的process()
函数之前退出。
该错误表明 Splash 尚不支持POST
。将Splash
适用于这个用例还是我应该使用Selenium
?