1

我想从我的 rabbitmq 获取消息，然后使用 gevent 生成 greenlet 来处理它。我的代码是：

import pika
import gevent.monkey
gevent.monkey.patch_all()
import gevent
from gevent.pool import Pool
from gevent import Timeout
from meliae import scanner
import MySQLdb
import urllib2
from MySQLdb.cursors import SSCursor

db=MySQLdb.connect(host='125.221.225.12',user='root',passwd='young001',charset='utf8',db='delicious',use_unicode=True) 
cur = db.cursor()

p = Pool(300)
success_count = 0
fail_count = 0


connection = pika.BlockingConnection(pika.ConnectionParameters(host='125.221.225.12'))
channel = connection.channel()
channel.queue_declare(queue='url_queue')

#logfile = file("log.txt",'aw',buffering=0)

def insert_into_avail(url):
    cur.execute("insert into avail_urls(url) values (%s)",url)
    db.commit()

def insert_into_fail(url):
    cur.execute("insert into fail_urls(url) values (%s)",url)
    db.commit()


class TooLong(Exception):
    pass

def down(url):
    global success_count
    global fail_count
    print 'the free pool is', p.free_count()
    try:
        with Timeout(30,TooLong):
            url_data = urllib2.urlopen(url)
            if url_data and url_data.getcode() ==200:
                #print 'url is ok'
                insert_into_avail(url)
                success_count = success_count+1
            else:
                print 'the code is ',url_data.getcode()
            #logfile.write(url)
            #print 'the pool is ', len(p)
            print 'success count is', success_count
    except:
        insert_into_fail(url)
        fail_count = fail_count+1
        #print 'the url is down',url
        #print 'the pool is ', len(p)
        print 'the fail_count is', fail_count


for method,properties,body in channel.consume('url_queue'):
    #print 'body is ',body
    channel.basic_ack(method.delivery_tag)
    scanner.dump_all_objects("dump.txt")
    p.spawn(down,body)
    #scanner.dump_all_objects("dump.txt")

    #print 'the pool is ', len(p)
    #logfile.write
p.join()

它工作得很慢，一些输出如下：

the fail_count is 30
the free pool is 295
the fail_count is 31
the free pool is 295
the fail_count is 32
the free pool is 295
the fail_count is 33
the free pool is 295
the fail_count is 34
the free pool is 295
the fail_count is 35
the free pool is 295

池中只有大约 5 个 greenlet，我想在池中获得 300 个 greenlet，因此程序运行速度很快。怎么了？如何调试它？因为rabbitmq，或gevent？或我的代码？谢谢

*更新： *当我评论“scanner.dump_all_objects("dump.txt")”时，它在完整池中运行，可能是因为当我将对象写入文件时，需要时间，所以从rabbitmq获取消息很慢。

但是在我的代码中，似乎存在内存泄漏问题，程序占用了数百M内存，所以我想打印所有对象以查找内存泄漏问题？如何解决它？有更好的方法吗？

4

0 回答 0