2

有人可以帮助我使用scrapy请求类提出请求吗

我已经尝试过了，但它不起作用：

from scrapy.selector import HtmlXPathSelector
from scrapy.http.request import Request
url = 'http://www.fetise.com'
a = Request(url)
hxs = HtmlXPathSelector(a)

错误是：

Traceback (most recent call last):
 File "sa.py", line 83, in <module>
 hxs = HtmlXPathSelector(a)
 File "/usr/local/lib/python2.7/dist-packages/scrapy/selector/lxmlsel.py",line 31,in __init__
_root = LxmlDocument(response, self._parser)
 File "/usr/local/lib/python2.7/dist-packages/scrapy/selector/lxmldocument.py",line 27,in __new__
cache[parser] = _factory(response, parser)
File "/usr/local/lib/python2.7/dist-packages/scrapy/selector/lxmldocument.py",line 13, in _factory
 body = response.body_as_unicode().strip().encode('utf8') or '<html/>'AttributeError: 'Request' object has no attribute 'body_as_unicode'`

我知道回调..实际上我首先想从网站上删除 URL，然后将它们用作起始 URL ....

4

2 回答 2

1

请试试这个：

import urllib
from scrapy.selector import HtmlXPathSelector
from pprint import pprint

url = 'http://www.fetise.com'
data = urllib.urlopen(url).read()
hxs = HtmlXPathSelector(text=data)

lista = hxs.select('//ul[@class="categoryMenu"]/li/ul/li/a/@href').extract()

acb = ["http://www.fetise.com/" + i if "http://www.fetise.com/" not in i else i for i in lista] + [u"http://www.fetise.com/sale"]

pprint(acb)

这是输出：

[u'http://www.fetise.com/apparel/shirts',
 u'http://www.fetise.com/apparel/tees',
 u'http://www.fetise.com/apparel/tops-and-tees',
 u'http://www.fetise.com/accessories/belts',
 u'http://www.fetise.com/accessories/cufflinks',
 u'http://www.fetise.com/accessories/jewellery',
 u'http://www.fetise.com/accessories/lighters',
 u'http://www.fetise.com/accessories/others',
 u'http://www.fetise.com/accessories/sunglasses',
 u'http://www.fetise.com/accessories/ties-cufflinks',
 u'http://www.fetise.com/accessories/wallets',
 u'http://www.fetise.com/accessories/watches',
 u'http://www.fetise.com/footwear/boots',
 u'http://www.fetise.com/footwear/casual',
 u'http://www.fetise.com/footwear/flats',
 u'http://www.fetise.com/footwear/heels',
 u'http://www.fetise.com/footwear/loafers',
 u'http://www.fetise.com/footwear/sandals',
 u'http://www.fetise.com/footwear/shoes',
 u'http://www.fetise.com/footwear/slippers',
 u'http://www.fetise.com/footwear/sports',
 u'http://www.fetise.com/innerwear/boxers',
 u'http://www.fetise.com/innerwear/briefs',
 u'http://www.fetise.com/personal-care/deos',
 u'http://www.fetise.com/personal-care/haircare',
 u'http://www.fetise.com/personal-care/perfumes',
 u'http://www.fetise.com/personal-care/personal-care',
 u'http://www.fetise.com/personal-care/shavers',
 u'http://www.fetise.com/apparel/tees/gifts-for-her',
 u'http://www.fetise.com/footwear/sandals/gifts-for-her',
 u'http://www.fetise.com/footwear/shoes/gifts-for-her',
 u'http://www.fetise.com/footwear/heels/gifts-for-her',
 u'http://www.fetise.com/footwear/flats/gifts-for-her',
 u'http://www.fetise.com/footwear/ballerinas/gifts-for-her',
 u'http://www.fetise.com/footwear/loafers/gifts-for-her',
 u'http://www.fetise.com/sale']
于 2013-06-21T16:52:17.850 回答
0

文档建议您需要在请求完成时传递回调。回调将有权访问响应对象：

从文档：

将附加数据传递给回调函数¶ 请求的回调是在下载该请求的响应时将调用的函数。回调函数将使用下载的 Response 对象作为其第一个参数来调用。

例子：

def parse_page1(self, response):
    return Request("http://www.example.com/some_page.html",
                      callback=self.parse_page2)

def parse_page2(self, response):
    # this would log http://www.example.com/some_page.html
    self.log("Visited %s" % response.url)
于 2013-06-21T15:59:37.850 回答