4

我正在尝试使用 response.replace() 替换谷歌搜索结果页面的搜索结果块的响应正文，但我遇到了一些编码问题。

scrapy  shell "http://www.google.de/search?q=Zuckerccc"

>>> srb = hxs.select("//li[@class='g']").extract()
>>> body = '<html><body>' + srb[0] + '</body></html>'    # get only 1st search result block
>>> b = response.replace(body = body)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "scrapy/lib/python2.6/site-packages/scrapy/http/response/text.py", line 54, in replace
    return Response.replace(self, *args, **kwargs)
  File "scrapy/lib/python2.6/site-packages/scrapy/http/response/__init__.py", line 77, in replace
    return cls(*args, **kwargs)
  File "scrapy/lib/python2.6/site-packages/scrapy/http/response/text.py", line 31, in __init__
    super(TextResponse, self).__init__(*args, **kwargs)
  File "scrapy/lib/python2.6/site-packages/scrapy/http/response/__init__.py", line 19, in __init__
    self._set_body(body)
  File "scrapy/lib/python2.6/site-packages/scrapy/http/response/text.py", line 48, in _set_body
    self._body = body.encode(self._encoding)
  File "../local_1/Linux-2.6c2.5-x86_64/Python/Python-147.0-0/lib/python2.6/encodings/cp1252.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_table)
UnicodeEncodeError: 'charmap' codec can't encode character u'\u0131' in position 529: character maps to <undefined>

我也尝试创建自己的回应，

>>> x = HtmlResponse("http://www.google.de/search?q=Zuckerccc", body = body, encoding = response.encoding)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "scrapy/lib/python2.6/site-packages/scrapy/http/response/text.py", line 31, in __init__
    super(TextResponse, self).__init__(*args, **kwargs)
    self._set_body(body)
  File "scrapy/lib/python2.6/site-packages/scrapy/http/response/text.py", line 48, in _set_body
    self._body = body.encode(self._encoding)
  File "../local_1/Linux-2.6c2.5-x86_64/Python/Python-147.0-0/lib/python2.6/encodings/cp1252.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_table)
UnicodeEncodeError: 'charmap' codec can't encode character u'\u0131' in position 529: character maps to <undefined>
  File "scrapy/lib/python2.6/site-packages/scrapy/http/response/__init__.py", line 19, in __init__

此外，当我在 replace() 函数中使用 _body_declared_encoding() 进行编码时，它可以工作。

replace(body = body, encoding = response._body_declared_encoding())

我不明白为什么 response._body_declared_encoding() 和 response.encoding 不同。任何人都可以对此有所了解。

那么，解决这个问题的好方法是什么？

4

2 回答 2

3

我成功地用这些代码行替换了响应正文：

scrapy  shell "http://www.google.de/search?q=Zuckerccc"
>>> google_result = response.xpath('//li[@class="g"]').extract()[0]
>>> body = '<html><body>' + google_result + '</body></html>'
>>> b = response.replace(body = body)
于 2015-06-30T19:28:11.823 回答
1

我查看源代码scrapy.http.response.text，当我们使用 TextResponse 时，我们需要先告诉 self._encoding。所以我们可以这样做：

>>>response._encoding='utf8'
>>>response._set_body("aaaaaa")
>>>response.body
>>>'aaaaaa'
于 2018-01-10T04:49:07.680 回答