python - Network multithread in python

Question

I'm writing a script in Python that will scrape some pages from my web server and put them in a file. I'm using mechanize.Browser() module for this particular task.

However, I've found that creating one single instance of mechanize.Browser() is rather slow. Is there a way I could relatively painlessly use multihreading/multiprocessing (i.e. issue several GET requests at once)?

Answer 1

使用gevent或eventlet获取并发网络 IO。

Answer 2

如果你想要工业级的 Python 网页抓取，请查看scrapy。它使用 Twisted 进行异步通信，速度非常快。每秒能够爬取 50 页并不是一个不切实际的期望。