0

我正在尝试在 python 中制作链接爬虫；我知道Harveman，但这不是我要找的。这是我到目前为止所拥有的：

import httplib, sys

target=sys.argv[1]
subsite=sys.argv[2]
link = "http://"+target+subsite

def spider():
    while 1:
        conn = httplib.HTTPConnection(target)
        conn.request("GET", subsite)
        r2 = conn.getresponse()
        data = r2.read().split('\n')
        for x in data[:]:
            if link in x:
                print x
spider()

但我似乎找不到过滤 x 的方法，所以我可以检索链接。

4

2 回答 2

2

如果你要走这条路，那么你可以从安装开始，requests让bs4生活更轻松 - 并根据以下内容开始你自己的蜘蛛模板：

import requests
from bs4 import BeautifulSoup

page = requests.get('http://www.google.com')
soup = BeautifulSoup(page.text)
# Find all anchor tags that have an href attribute
print [a['href'] for a in soup.find_all('a', {'href': True})]
# ['http://www.google.co.uk/imghp?hl=en&tab=wi', 'http://maps.google.co.uk/maps?hl=en&tab=wl', 'https://play.google.com/?hl=en&tab=w8', 'http://www.youtube.com/?gl=GB&tab=w1', 'http://news.google.co.uk/nwshp?hl=en&tab=wn', 'https://mail.google.com/mail/?tab=wm', 'https://drive.google.com/?tab=wo', 'http://www.google.co.uk/intl/en/options/', 'http://www.google.co.uk/history/optout?hl=en', '/preferences?hl=en', 'https://accounts.google.com/ServiceLogin?hl=en&continue=http://www.google.co.uk/', '/advanced_search?hl=en-GB&authuser=0', '/language_tools?hl=en-GB&authuser=0', 'https://www.google.com/intl/en_uk/chrome/browser/promo/cubeslam/', '/intl/en/ads/', '/services/', 'https://plus.google.com/103583604759580854844', '/intl/en/about.html', 'http://www.google.co.uk/setprefdomain?prefdom=US&sig=0_cYDPGyR7QbF1UxGCXNpHcrj09h4%3D', '/intl/en/policies/']
于 2013-06-17T23:30:52.107 回答
1

我认为会工作

import re
re.findall("href=([^ >]+)",x)
于 2013-06-17T23:20:16.287 回答