1

我有一个可以在本地运行的scrapy 脚本，但是当我将它部署到Scrapinghub 时，它会给出所有错误。调试后，错误来自 Yielding item。

这是我得到的错误。

ERROR   [scrapy.utils.signal] Error caught on signal handler: <bound method ?.item_scraped of <sh_scrapy.extension.HubstorageExtension object at 0x7fd39e6141d0>> Less
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
    result = f(*args, **kw)
  File "/usr/local/lib/python2.7/site-packages/pydispatch/robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "/usr/local/lib/python2.7/site-packages/sh_scrapy/extension.py", line 45, in item_scraped
    item = self.exporter.export_item(item)
  File "/usr/local/lib/python2.7/site-packages/scrapy/exporters.py", line 304, in export_item
    result = dict(self._get_serialized_fields(item))
  File "/usr/local/lib/python2.7/site-packages/scrapy/exporters.py", line 75, in _get_serialized_fields
    value = self.serialize_field(field, field_name, item[field_name])
  File "/usr/local/lib/python2.7/site-packages/scrapy/exporters.py", line 284, in serialize_field
    return serializer(value)
  File "/usr/local/lib/python2.7/site-packages/scrapy/exporters.py", line 290, in _serialize_value
    return dict(self._serialize_dict(value))
  File "/usr/local/lib/python2.7/site-packages/scrapy/exporters.py", line 300, in _serialize_dict
    key = to_bytes(key) if self.binary else key
  File "/usr/local/lib/python2.7/site-packages/scrapy/utils/python.py", line 117, in to_bytes
    'object, got %s' % type(text).__name__)
TypeError: to_bytes must receive a unicode, str or bytes object, got int

它没有指定有问题的字段，但是通过消除过程，我意识到这是代码的这一部分：

  try:
            item["media"] = {}
            media_index = 0

            media_content = response.xpath("//audio/source/@src").extract_first()
            if media_content is not None:
                item["media"][media_index] = {}
                preview = item["media"][media_index]
                preview["Media URL"] = media_content
                preview["Media Type"] = "Audio"
                media_index += 1
        except IndexError:
            print "Index error for media " + item["asset_url"]

我清理了一些部分以使其更容易解决，但基本上这部分是问题所在。它不喜欢项目媒体的东西。

我是 Python 和 Scrapy 的初学者。很抱歉，如果这被证明是愚蠢的基本 Python 错误。任何的想法？

编辑：所以从 ThunderMind 得到答案后，解决方案是简单地为 key 做 str(media_index)

是的，就在这里：

item["media"][media_index] = {}

media_index 是可变的。并且 Keys 不能是可变的。阅读Python dict，了解应该使用什么作为键。

4

1 回答 1

1

是的，就在这里：

item["media"][media_index] = {}

media_index 是可变的。并且 Keys 不能是可变的。阅读Python dict，了解应该使用什么作为键。

于 2019-01-29T11:31:34.700 回答