3

I have a set of Scrapy spiders. They need to be run daily from a desktop application. What is the simplest way (from user's point of view) to install and run it on another windows machine?

4

3 回答 3

1

创建一个作为系统命令运行的脚本（例如run_spider.py ）。scrapy crawl <spider_name>

run_spider.py

from os import system
output_file_name = 'results.csv'
system('scrapy crawl myspider -o ' + output_file_name + ' -t csv')

然后将该脚本提供给 PyInstaller：

pyinstaller run_spider.py
于 2014-02-22T18:21:57.227 回答
0

我猜最简单的方法是用python为他们编写一个脚本......

如果您正在运行 Windows Server，您甚至可以安排您使用的命令（scrapy crawl yourspider）来运行蜘蛛。

于 2013-08-30T14:32:47.577 回答
0

这是将蜘蛛作为独立脚本或可执行文件运行的另一种可能性

import scrapy
from scrapy.crawler import CrawlerProcess

class MySpider(scrapy.Spider):
    # Your spider definition
    ...

process = CrawlerProcess({
    'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
})

process.crawl(MySpider)
process.start() # the script will block here until the crawling is finished

你可以在这里找到更多信息：https ://doc.scrapy.org/en/1.0/topics/practices.html

于 2018-12-01T11:22:53.050 回答