8

在 Scrapy 1.0 之前，我可以非常简单地针对本地文件运行 Scrapy Shell：

$ scrapy shell index.html

升级到1.0.3后开始报错：

$ scrapy shell index.html
2015-10-12 15:32:59 [scrapy] INFO: Scrapy 1.0.3 started (bot: scrapybot)
2015-10-12 15:32:59 [scrapy] INFO: Optional features available: ssl, http11, boto
2015-10-12 15:32:59 [scrapy] INFO: Overridden settings: {'LOGSTATS_INTERVAL': 0}
Traceback (most recent call last):
  File "/Users/user/.virtualenvs/so/bin/scrapy", line 11, in <module>
    sys.exit(execute())
  File "/Users/user/.virtualenvs/so/lib/python2.7/site-packages/scrapy/cmdline.py", line 143, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "/Users/user/.virtualenvs/so/lib/python2.7/site-packages/scrapy/cmdline.py", line 89, in _run_print_help
    func(*a, **kw)
  File "/Users/user/.virtualenvs/so/lib/python2.7/site-packages/scrapy/cmdline.py", line 150, in _run_command
    cmd.run(args, opts)
  File "/Users/user/.virtualenvs/so/lib/python2.7/site-packages/scrapy/commands/shell.py", line 50, in run
    spidercls = spidercls_for_request(spider_loader, Request(url),
  File "/Users/user/.virtualenvs/so/lib/python2.7/site-packages/scrapy/http/request/__init__.py", line 24, in __init__
    self._set_url(url)
  File "/Users/user/.virtualenvs/so/lib/python2.7/site-packages/scrapy/http/request/__init__.py", line 59, in _set_url
    raise ValueError('Missing scheme in request url: %s' % self._url)
ValueError: Missing scheme in request url: index.html

这种行为是有意的还是 Scrapy Shell 中的错误？

作为一种解决方法，我可以在“文件”URL 方案中使用文件的绝对路径：

$ scrapy shell file:////absolute/path/to/index.html

显然，这不那么方便和容易。

4

3 回答 3

7

更新：对于 Scrapy >=1.1，这是一个内置功能，你可以这样做：

scrapy shell file:///path/to/file.html

老答案：

根据针对本地文件运行 scrapy shell 中的讨论，此提交引入了相关更改。为这个问题创建了一个Pull Request以使 Scrapy shell 再次打开本地文件，并且计划成为 Scrapy 1.1 的一部分。

于 2015-11-09T04:19:57.563 回答
2

对于 Scrapy==2.5.1，您可以对本地文件运行 Scrapy Shell，如下所示：

如果文件在同一目录中，请在文件名前使用“./”，如下所示：

scrapy shell ./file.html
于 2022-01-11T20:20:06.070 回答
1

With following configuration

  • MacOS X
  • Scrapy 1.6.0

what worked for me is going with scrapy shell ./index.html with index.html being in the root folder of your scrapy generated project

于 2020-09-08T01:32:32.110 回答