我想我会在几天内为这个问题苦苦挣扎时,会请求很大的帮助。我尝试了所有可能的方法(据我所知),但仍然没有结果。我做错了什么,但仍然无法弄清楚它是什么。所以感谢每一位愿意去冒险的人。首先要做的事情:我正在尝试使用 POST 方法将信息发布到 delta.com 上的表单 与此网站一样,它很复杂,因为它们包含在会话、cookie 和 Javascript 中,因此可能会出现问题。我正在使用我在 stackoverflow 中找到的代码示例: Using MultipartPostHandler to POST form-data with Python 这是我为增量网页调整的代码。
from scrapy.selector import HtmlXPathSelector
from scrapy.http import FormRequest, Request
from delta.items import DeltaItem
from scrapy.contrib.spiders import CrawlSpider, Rule
class DmozSpider(CrawlSpider):
name = "delta"
allowed_domains = ["http://www.delta.com"]
start_urls = ["http://www.delta.com"]
def start_requests(self, response):
yield FormRequest.from_response(response, formname='flightSearchForm',url="http://www.delta.com/booking/findFlights.do", formdata={'departureCity[0]':'JFK', 'destinationCity[0]':'SFO','departureDate[0]':'07.20.2013','departureDate[1]':'07.28.2013','paxCount':'1'},callback=self.parse1)
def parse1(self, response):
hxs = HtmlXPathSelector(response)
sites = hxs.select('//')
items = []
for site in sites:
item = DeltaItem()
item['title'] = site.select('text()').extract()
item['link'] = site.select('text()').extract()
item['desc'] = site.select('text()').extract()
items.append(item)
return items
当我指示蜘蛛在终端中爬行时,我看到:
scrapy crawl delta -o items.xml -t xml
2013-07-01 13:39:30+0300 [scrapy] INFO: Scrapy 0.16.2 started (bot: delta)
2013-07-01 13:39:30+0300 [scrapy] DEBUG: Enabled extensions: FeedExporter, LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
2013-07-01 13:39:30+0300 [scrapy] DEBUG: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, RedirectMiddleware, CookiesMiddleware, HttpCompressionMiddleware, ChunkedTransferMiddleware, DownloaderStats
2013-07-01 13:39:30+0300 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2013-07-01 13:39:30+0300 [scrapy] DEBUG: Enabled item pipelines:
2013-07-01 13:39:30+0300 [delta] INFO: Spider opened
2013-07-01 13:39:30+0300 [delta] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2013-07-01 13:39:30+0300 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023
2013-07-01 13:39:30+0300 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
2013-07-01 13:39:33+0300 [delta] DEBUG: Crawled (200) <GET http://www.delta.com> (referer: None)
2013-07-01 13:39:33+0300 [delta] INFO: Closing spider (finished)
2013-07-01 13:39:33+0300 [delta] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 219,
'downloader/request_count': 1,
'downloader/request_method_count/GET': 1,
'downloader/response_bytes': 27842,
'downloader/response_count': 1,
'downloader/response_status_count/200': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2013, 7, 1, 10, 39, 33, 159235),
'log_count/DEBUG': 7,
'log_count/INFO': 4,
'response_received_count': 1,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'start_time': datetime.datetime(2013, 7, 1, 10, 39, 30, 734090)}
2013-07-01 13:39:33+0300 [delta] INFO: Spider closed (finished)
如果您与链接中的示例进行比较,即使我使用几乎相同的代码,我也看不到我设法制作了 POST 方法。我什至尝试使用来自 W3schools 的非常简单的 HTML/PHP 表单,我将其放在服务器上,但在那里相同。我从来没有设法创建过 POST。我认为问题很简单,但是因为只有我拥有的 Python 知识是 Scrapy 并且所有 Scrapy 都是我在网上找到的(我有据可查)和示例,但这对我来说仍然不够。因此,如果任何人至少可以展示正确的方式,那将是非常大的帮助。