我正在尝试使用以下代码从该站点上抓取产品名称和价格：
class ProductSpider(scrapy.Spider):
name = 'product'
start_urls = ['https://www.bodyenfitshop.nl/']
def parse(self, response):
# follow links to different categories
for href in response.css('ol.nav-primary > li.category-node > a::attr(href)'):
# A category page currently does not list all the items belonging to that category.
# Add "?p=1" to get the list view
href = href.extract() + "?p=1"
print(href)
yield SplashRequest(href, self.parse_category, args={
'wait': 0.5
})
def parse_category(self, response):
for product in response.css('li.item'):
new_name = product.css('div.product-name > a::text').extract_first().strip()
# TODO: find better regex
new_price = float(product.css('span.price::text').re('\d+,\d+')[0].replace(",", "."))
new_url = product.css('div.product-name > a::attr(href)').extract_first().strip()
yield Product(name=new_name, price=new_price, url=new_url)
我怀疑问题出在传递给 SplashRequest 的 href 中，但是当我将它们打印出来时，它们都有完全限定的 URL，如下所示：
https://www.bodyenfitshop.nl/duursport/?p=1
SO上有关此错误的所有其他问题（例如this或this）都可以通过将“https”添加到其 URL 来解决。但我已经有了这些。所以我不知道是什么导致了这些。
这是我得到的错误之一（重复多次）
2017-05-31 22:51:07 [scrapy.core.scraper] ERROR: Error downloading <GET https://www.
bodyenfitshop.nl/afslanken/?p=1 via https://www.bodyenfitshop.nl/afslanken/?p=1>
Traceback (most recent call last):
File "c:\program files (x86)\python\lib\site-packages\twisted\internet\defer.py", line 1301, in _inlineCallbacks
result = g.send(result)
File "c:\program files (x86)\python\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "c:\program files (x86)\python\lib\site-packages\scrapy_splash\middleware.py", line 358, in process_request
priority=request.priority + self.rescheduling_priority_adjust
File "c:\program files (x86)\python\lib\site-packages\scrapy\http\request\__init__.py", line 94, in replace
return cls(*args, **kwargs)
File "c:\program files (x86)\python\lib\site-packages\scrapy_splash\request.py", line 76, in __init__
**kwargs)
File "c:\program files (x86)\python\lib\site-packages\scrapy\http\request\__init__.py", line 25, in __init__
self._set_url(url)
File "c:\program files (x86)\python\lib\site-packages\scrapy\http\request\__init__.py", line 58, in _set_url
raise ValueError('Missing scheme in request url: %s' % self._url)
ValueError: Missing scheme in request url: render.html
render.html 是（我认为）Splash 生成的默认名称。我认为我无法改变这一点。
非常感谢任何帮助或朝正确方向戳！