我正在尝试根据 Firefox 和 Safari 抓取 HAL+Json 的链接,它正在返回一个 Scrapy 无法识别的响应对象。
链接是https://catalogue.presto.com.au/ - 这在 Chrome 中打开很好,在浏览器中显示 JSON,但如果我尝试使用 Firefox 或 Safari,它会下载文件。我怀疑 Scrapy 在打开链接时会下载文件,所以它不会抓取它。
有没有人遇到过类似的情况或有解决方案?
通过 Shell 访问
当我尝试使用终端“scrapy shell https://catalogue.presto.com.au ”访问网站时
"2015-03-15 00:15:08+0700 [default] DEBUG: Crawled (200) <GET https://catalogue.presto.com.au>"
然后我尝试查看(响应)并收到此错误:
>>> view(response)
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/Library/Python/2.7/site-packages/scrapy/utils/response.py", line 86, in open_in_browser
response.__class__.__name__)
TypeError: Unsupported response type: Response
运行 Scrapy 对象:
def parse(self, response):
print response.__class__
open_in_browser(response)
2015-03-15 00:23:05+0700 [prestotv2] DEBUG: Crawled (200) <GET
class 'scrapy.http.response.Response' (referer: None) #this line is from "print response.__class__
2015-03-15 00:23:05+0700 [prestotv2] ERROR: Spider error processing <GET https://catalogue.presto.com.au/>
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/base.py", line 1201, in mainLoop
self.runUntilCurrent()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/base.py", line 824, in runUntilCurrent
call.func(*call.args, **call.kw)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/defer.py", line 382, in callback
self._startRunCallbacks(result)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/defer.py", line 490, in _startRunCallbacks
self._runCallbacks()
--- <exception caught here> ---
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/defer.py", line 577, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/Users/nathansu/Documents/Development/Whutstream/scraping/Presto/presto/spiders/TvSpider.py", line 38, in parse
open_in_browser(response)
File "/Library/Python/2.7/site-packages/scrapy/utils/response.py", line 86, in open_in_browser
response.__class__.__name__)
exceptions.TypeError: Unsupported response type: Response