2

我正在使用Python Twitter 工具下载大量用户的最新 200 条推文。我收到一个仅间歇性发生的 gzip 错误。在看似随机的时间间隔内,循环会因下面的错误堆栈而崩溃。如果我立即重新启动循环并发送同一个用户,下载它就不会出现问题。当推文崩溃时,我查看了推文的标题,似乎与不会导致问题的标题没有任何不同。而且我已经确认,我毫无问题地返回的许多结果也被 gzip 压缩并且未压缩良好。

有没有人以前见过这个问题和/或可以提出修复/解决方法?

这是错误堆栈,值得:

File "/Users/martinlbarron/Dropbox/Learning Python/downloadTimeline.py", line 33, in <module>
    result=utility.downloadTimeline(kwargs,t)
  File "/Users/martinlbarron/Dropbox/Learning Python/utility.py", line 73, in downloadTimeline
    response=t.statuses.user_timeline(**kargs)
  File "/Library/Python/2.7/site-packages/twitter-1.9.0-py2.7.egg/twitter/api.py", line 173, in __call__
    return self._handle_response(req, uri, arg_data)
  File "/Library/Python/2.7/site-packages/twitter-1.9.0-py2.7.egg/twitter/api.py", line 184, in _handle_response
    data = f.read()
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gzip.py", line 245, in read
    self._read(readsize)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gzip.py", line 299, in _read
    self._read_eof()
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gzip.py", line 338, in _read_eof
    hex(self.crc)))
IOError: CRC check failed 0xf4196259 != 0x34967f68L
logout

添加我的代码(温柔点,我是 Python 新手)

我有一个推特名称列表。我在下面的代码中遍历它们,调用我的 twitter 下载函数 (downloadTimeline)。

t = Twitter(
    auth=OAuth("XXX", "XXX",
               "XXX", "XXX"))
for i in range(startRange,endRange):
    #Get the id string for user
    row=newlist[i]
    sc=row[3]
    kwargs = dict(count=200, include_rts=False, include_entities=False, trim_user=True, screen_name=sc)
    result=utility.downloadTimeline(kwargs,t)

在 downloadTimeline 中,我得到了 twitter 响应(响应),然后将其解析为字典

def downloadTimeline(kargs, t):

    #Get timeline
    mylist = list()
    counter=1000
    try:
        response=t.statuses.user_timeline(**kargs)
        counter=response.rate_limit_remaining
        #parse the file out
        if len(response)>0:
            for tweet in response:
                user=tweet['user']
                dict =  {
                    'id_str': cleanLines(tweet['id_str']), 
                    #ommitting the whole list of all the variables I save
                }
                mylist.append(dict)

    except twitter.TwitterError as e:
            print("Fail: %i" % e.e.code)

    return  (mylist, counter)

最后,虽然这显然不是我的代码,但在 Python Twitter 工具框架中,这是一段似乎令人窒息的代码(特别是在 f = gzip.GzipFile(fileobj=buf) 处)

   def _handle_response(self, req, uri, arg_data):
        try:
            handle = urllib_request.urlopen(req)
            if handle.headers['Content-Type'] in ['image/jpeg', 'image/png']:
                return handle
            elif handle.info().get('Content-Encoding') == 'gzip':
                # Handle gzip decompression
                buf = StringIO(handle.read())
                f = gzip.GzipFile(fileobj=buf)
                data = f.read()
            else:
                data = handle.read()

            if "json" == self.format:
                res = json.loads(data.decode('utf8'))
                return wrap_response(res, handle.headers)
            else:
                return wrap_response(
                    data.decode('utf8'), handle.headers)
        except urllib_error.HTTPError as e:
            if (e.code == 304):
                return []
            else:
                raise TwitterHTTPError(e, uri, self.format, arg_data)

事实证明,在 Python Twitter 工具中关闭接受 gzip 标头非常容易。但是当我这样做时,我收到以下错误。我想知道响应是否以某种方式被截断:

  File "/Users/martinlbarron/Dropbox/Learning Python/downloadTimeline.py", line 33, in <module>
    result=utility.downloadTimeline(kwargs,t)
  File "/Users/martinlbarron/Dropbox/Learning Python/utility.py", line 73, in downloadTimeline
    response=t.statuses.user_timeline(**kargs)
  File "/Library/Python/2.7/site-packages/twitter-1.9.0-py2.7.egg/twitter/api.py", line 175, in __call__
    return self._handle_response(req, uri, arg_data)
  File "/Library/Python/2.7/site-packages/twitter-1.9.0-py2.7.egg/twitter/api.py", line 193, in _handle_response
    res = json.loads(handle.read().decode('utf8'))
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
    obj, end = self.scan_once(s, idx)
ValueError: Unterminated string starting at: line 1 column 13699 (char 13699)
logout
4

1 回答 1

2

代替 :

buf = StringIO(handle.read())
f = gzip.GzipFile(fileobj=buf)
data = f.read()

尝试这个:

decomp = zlib.decompressobj(16+zlib.MAX_WBITS)
data = decomp.decompress(handle.read())

不要忘记import zlib

于 2012-12-03T21:27:13.017 回答