3

I recently inherited a python project, and I'm working on maintaining it now. Part of the code makes a few hundred thousand requests from a website and saves the results to a database. The code is reusing the same httplib.HTTPConnection object for reach request and then just looping over a 

conn.request("GET",someString,'',headers)

response = conn.getresponse()

section. A few days ago in my logs I saw that one of the requests threw the exception:

[Errno 104] Connection reset by peer

followed by every other conn.request() failing. My first inclination was to just build a new connection for each request, but the perfomance impact of that was profound and horrible. So my question is, how do I fix this, especially since I'm not 100% sure how I can even really test this.

If I just call conn.connect() after an exception, will it correctly reconnect?

I'm looking for advise on how to fix it and possibly how I could test it.

Thanks for your time.

4

1 回答 1

3

我认为您首先需要确定要处理的故障模式。例如，连接是否由于服务器上的临时资源问题而重置，并且快速周转连接将修复它？或者，服务器是否关闭或重新启动，您应该中止您的进程？

假设第一种情况，我认为您的想法是正确的。尝试这样的事情（注意，这不是工作代码 - 这只是逻辑的一个例子）：

while True:
    try:
        conn.request("GET",someString,'',headers)
        response = conn.getresponse()
    except httplib.HTTPException, e:
        conn.connect()
        continue
    break

您可能应该为此添加一些逻辑，以在重复的连接尝试之间暂停并在一定次数的尝试后放弃（这基本上是上面的第二种情况）。

为了对此进行测试，请尝试使用 tcpkill 来重置 TCP 连接：

http://www.gnutoolbox.com/tcpkill-command/

于 2012-11-30T21:07:34.917 回答