3

我想从网站上获取很多页面，例如

curl "http://farmsubsidy.org/DE/browse?page=[0000-3603]" -o "de.#1"

但在 python 中获取页面数据，而不是磁盘文件。有人可以发布pycurl代码来做到这一点，
或者如果可能的话，可以快速urllib2（不是一次一个），
或者说“算了吧，curl 更快更健壮”？谢谢

4

6 回答 6

6

所以你有两个问题，让我举一个例子。请注意，pycurl 已经完成了多线程/不是一次一个而没有您的辛勤工作。

#! /usr/bin/env python

import sys, select, time
import pycurl,StringIO

c1 = pycurl.Curl()
c2 = pycurl.Curl()
c3 = pycurl.Curl()
c1.setopt(c1.URL, "http://www.python.org")
c2.setopt(c2.URL, "http://curl.haxx.se")
c3.setopt(c3.URL, "http://slashdot.org")
s1 = StringIO.StringIO()
s2 = StringIO.StringIO()
s3 = StringIO.StringIO()
c1.setopt(c1.WRITEFUNCTION, s1.write)
c2.setopt(c2.WRITEFUNCTION, s2.write)
c3.setopt(c3.WRITEFUNCTION, s3.write)

m = pycurl.CurlMulti()
m.add_handle(c1)
m.add_handle(c2)
m.add_handle(c3)

# Number of seconds to wait for a timeout to happen
SELECT_TIMEOUT = 1.0

# Stir the state machine into action
while 1:
    ret, num_handles = m.perform()
    if ret != pycurl.E_CALL_MULTI_PERFORM:
        break

# Keep going until all the connections have terminated
while num_handles:
    # The select method uses fdset internally to determine which file descriptors
    # to check.
    m.select(SELECT_TIMEOUT)
    while 1:
        ret, num_handles = m.perform()
        if ret != pycurl.E_CALL_MULTI_PERFORM:
            break

# Cleanup
m.remove_handle(c3)
m.remove_handle(c2)
m.remove_handle(c1)
m.close()
c1.close()
c2.close()
c3.close()
print "http://www.python.org is ",s1.getvalue()
print "http://curl.haxx.se is ",s2.getvalue()
print "http://slashdot.org is ",s3.getvalue()

最后，这些代码主要是基于pycurl网站上的一个例子=.=

可能你真的应该阅读文档。人们在上面花费大量时间。

于 2010-08-20T05:23:24.820 回答
3

这是一个基于 urllib2 和线程的解决方案。

import urllib2
from threading import Thread

BASE_URL = 'http://farmsubsidy.org/DE/browse?page='
NUM_RANGE = range(0000, 3603)
THREADS = 2

def main():
    for nums in split_seq(NUM_RANGE, THREADS):
        t = Spider(BASE_URL, nums)
        t.start()

def split_seq(seq, num_pieces):
    start = 0
    for i in xrange(num_pieces):
        stop = start + len(seq[i::num_pieces])
        yield seq[start:stop]
        start = stop

class Spider(Thread):
    def __init__(self, base_url, nums):
        Thread.__init__(self)
        self.base_url = base_url
        self.nums = nums
    def run(self):
        for num in self.nums:
            url = '%s%s' % (self.base_url, num)
            data = urllib2.urlopen(url).read()
            print data

if __name__ == '__main__':
    main()
于 2009-12-24T22:40:54.437 回答
1

您可以将其放入 for 循环内的 bash 脚本中。

但是，您在使用 python 解析每个页面时可能会取得更好的成功。 http://www.securitytube.net/Crawling-the-Web-for-Fun-and-Profit-video.aspx 您将能够获得准确的数据并将其同时保存到数据库中。 http://www.securitytube.net/Storing-Mined-Data-from-the-Web-for-Fun-and-Profit-video.aspx

于 2009-12-24T19:06:19.487 回答
1

如果你想使用 python 爬取一个网站，你应该看看scrapy http://scrapy.org

于 2011-02-06T11:40:45.367 回答
1

使用BeautifulSoup4和请求-

抢头页：

page = Soup(requests.get(url='http://rootpage.htm').text)

创建一个请求数组：

from requests import async

requests = [async.get(url.get('href')) for url in page('a')]
responses = async.map(requests)

[dosomething(response.text) for response in responses]

请求要求 gevent 这样做顺便说一句。

于 2012-05-05T03:47:08.100 回答
1

我可以推荐你使用human_curl的异步模块

看例子：

from urlparse import urljoin 
from datetime import datetime

from human_curl.async import AsyncClient 
from human_curl.utils import stdout_debug

def success_callback(response, **kwargs):
    """This function call when response successed
    """
    print("success callback")
    print(response, response.request)
    print(response.headers)
    print(response.content)
    print(kwargs)

def fail_callback(request, opener, **kwargs):
    """Collect errors
    """
    print("fail callback")
    print(request, opener)
    print(kwargs)

with AsyncClient(success_callback=success_callback,
                 fail_callback=fail_callback) as async_client:
    for x in xrange(10000):
        async_client.get('http://google.com/', params=(("x", str(x)),)
        async_client.get('http://google.com/', params=(("x", str(x)),),
                        success_callback=success_callback, fail_callback=fail_callback)

用法很简单。然后页面成功加载失败的 async_client 调用你回调。您还可以指定并行连接的数量。

于 2012-10-07T19:57:22.317 回答