我一直在从事一个研究项目,涉及从 twitter 获取朋友和关注者信息。我编写的代码适用于像我这样拥有 1k 关注者或朋友的示例帐户。但我在为拥有 1220 万粉丝的唐纳德特朗普账户运行它时遇到了麻烦。在我的程序中查询超出了某个时间速率限制后,我等待 15 分钟,然后再次请求相同的。此过程适用于大约 100 万用户信息,但之后会卡住,或者有时会出现 URL 错误。请提出在处理如此大的数据时可能出现的问题以及解决方案。请在下面找到相同的代码片段 -
def get_info_by_screen_name(t, screen_names):
sn_to_info = {}
while len(screen_names) > 0:
# Process 100 ids at a time...
screen_names_str = ','.join([str(sn) for sn in screen_names[:100]])
screen_names = screen_names[100:]
response = make_twitter_request(t,
getattr(getattr(t, "users"), "lookup"),
screen_name=screen_names_str)
if response is None:
break
if type(response) is dict: # Handle Twitter API quirk
response = [response]
for user_info in response:
sn_to_info[user_info['screen_name']] = user_info
return sn_to_info
t = oauth_login()
friends_ids, followers_ids = get_friends_followers_ids(t,
screen_name="realDonaldTrump",
friends_limit=100,
followers_limit=100)
# Basic usage...
print friends_ids
print followers_ids
# Basic usage...
info = {}
data_test = {}
info.update(get_info_by_id(t, friends_ids))