我正在使用Python Twitter 工具下载大量用户的最新 200 条推文。我收到一个仅间歇性发生的 gzip 错误。在看似随机的时间间隔内，循环会因下面的错误堆栈而崩溃。如果我立即重新启动循环并发送同一个用户，下载它就不会出现问题。当推文崩溃时，我查看了推文的标题，似乎与不会导致问题的标题没有任何不同。而且我已经确认，我毫无问题地返回的许多结果也被 gzip 压缩并且未压缩良好。
有没有人以前见过这个问题和/或可以提出修复/解决方法？
这是错误堆栈，值得：
File "/Users/martinlbarron/Dropbox/Learning Python/downloadTimeline.py", line 33, in <module>
result=utility.downloadTimeline(kwargs,t)
File "/Users/martinlbarron/Dropbox/Learning Python/utility.py", line 73, in downloadTimeline
response=t.statuses.user_timeline(**kargs)
File "/Library/Python/2.7/site-packages/twitter-1.9.0-py2.7.egg/twitter/api.py", line 173, in __call__
return self._handle_response(req, uri, arg_data)
File "/Library/Python/2.7/site-packages/twitter-1.9.0-py2.7.egg/twitter/api.py", line 184, in _handle_response
data = f.read()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gzip.py", line 245, in read
self._read(readsize)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gzip.py", line 299, in _read
self._read_eof()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gzip.py", line 338, in _read_eof
hex(self.crc)))
IOError: CRC check failed 0xf4196259 != 0x34967f68L
logout
添加我的代码（温柔点，我是 Python 新手）
我有一个推特名称列表。我在下面的代码中遍历它们，调用我的 twitter 下载函数 (downloadTimeline)。
t = Twitter(
auth=OAuth("XXX", "XXX",
"XXX", "XXX"))
for i in range(startRange,endRange):
#Get the id string for user
row=newlist[i]
sc=row[3]
kwargs = dict(count=200, include_rts=False, include_entities=False, trim_user=True, screen_name=sc)
result=utility.downloadTimeline(kwargs,t)
在 downloadTimeline 中，我得到了 twitter 响应（响应），然后将其解析为字典
def downloadTimeline(kargs, t):
#Get timeline
mylist = list()
counter=1000
try:
response=t.statuses.user_timeline(**kargs)
counter=response.rate_limit_remaining
#parse the file out
if len(response)>0:
for tweet in response:
user=tweet['user']
dict = {
'id_str': cleanLines(tweet['id_str']),
#ommitting the whole list of all the variables I save
}
mylist.append(dict)
except twitter.TwitterError as e:
print("Fail: %i" % e.e.code)
return (mylist, counter)
最后，虽然这显然不是我的代码，但在 Python Twitter 工具框架中，这是一段似乎令人窒息的代码（特别是在 f = gzip.GzipFile(fileobj=buf) 处）
def _handle_response(self, req, uri, arg_data):
try:
handle = urllib_request.urlopen(req)
if handle.headers['Content-Type'] in ['image/jpeg', 'image/png']:
return handle
elif handle.info().get('Content-Encoding') == 'gzip':
# Handle gzip decompression
buf = StringIO(handle.read())
f = gzip.GzipFile(fileobj=buf)
data = f.read()
else:
data = handle.read()
if "json" == self.format:
res = json.loads(data.decode('utf8'))
return wrap_response(res, handle.headers)
else:
return wrap_response(
data.decode('utf8'), handle.headers)
except urllib_error.HTTPError as e:
if (e.code == 304):
return []
else:
raise TwitterHTTPError(e, uri, self.format, arg_data)
事实证明，在 Python Twitter 工具中关闭接受 gzip 标头非常容易。但是当我这样做时，我收到以下错误。我想知道响应是否以某种方式被截断：
File "/Users/martinlbarron/Dropbox/Learning Python/downloadTimeline.py", line 33, in <module>
result=utility.downloadTimeline(kwargs,t)
File "/Users/martinlbarron/Dropbox/Learning Python/utility.py", line 73, in downloadTimeline
response=t.statuses.user_timeline(**kargs)
File "/Library/Python/2.7/site-packages/twitter-1.9.0-py2.7.egg/twitter/api.py", line 175, in __call__
return self._handle_response(req, uri, arg_data)
File "/Library/Python/2.7/site-packages/twitter-1.9.0-py2.7.egg/twitter/api.py", line 193, in _handle_response
res = json.loads(handle.read().decode('utf8'))
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 326, in loads
return _default_decoder.decode(s)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Unterminated string starting at: line 1 column 13699 (char 13699)
logout