我尝试下载大约 6000 个 http 地址的 /home.htm 文件。为了速度,我尝试使用 grequests 一次发送它们,但我只得到大约 200 个答案,其中大多数给出了连接被拒绝的错误。当我将地址分成 100 个块然后单独发送每个块时,大约 1200 个地址会回答我(=他们的 /home.htm 下载成功),即使我使用与以前相同的地址。
我在 Ubuntu 16.04 上使用 Python3.6 运行它。
import grequests
import requests
import sys
import os
import resource
# Counts exceptions and prints them
def exceptionh(request, exception):
...
# Yields succesive n-sized chunks
def make_chunks(req, n):
for i in range(0, len(req), n):
yield req[i:i+n]
def run(ipport):
# Make http links
http_links = []
for ip in ipport:
http_links.append('http://' + ip.strip() + '/home.htm')
# changing limit, without it there are too many Errno24 Exceptions
resource.setrlimit(resource.RLIMIT_NOFILE, (131072, 131072))
# Request making
rq = []
ctr = 0
for link in http_links:
rq.append(grequests.get(link, timeout=30, headers={'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.75 Safari/537.36'}, stream=False))
ctr += 1
rq = list(make_chunks(rq, 100))
# Send requests
results = []
for chunk in rq:
results.append(grequests.map(chunk, exception_handler=exceptionh))
# Save .html
for chunk in results:
for response in chunk:
if response is not None:
# write it in html file
正如我上面描述的结果不同。当我以块的形式发送请求时,我得到的结果比我一次发送的要多。这是为什么?有没有更好的方法来解决这个问题?