0

我用 urllib2 打开了这个链接“ http://www.amazon.com/s?rh=n%3A1 ”，我试图获取下一页链接（href="/s/ref=lp_1_pg_2?rh=n% 3A283155%2Cn%3A%211000%2Cn%3A1&page=2&ie=UTF8&qid=1376769633") 在 html 文本中可用。但是，read() 函数继续将这部分读取为 (href="/s?rh=n%3A1&page=2") 不起作用。有什么方法可以让读取函数正确读取链接？

4

2 回答 2

2

这样做是因为您没有标题。我试过了：

from mechanize import Browser
from bs4 import BeautifulSoup

browser = Browser()

html_page = browser.open("http://www.amazon.com/s?rh=n%3A1")
soup = BeautifulSoup(html_page)
link = soup.find("a", {"title" : "Next Page"})
print link

输出：

<a title="Next Page" id="pagnNextLink" class="pagnNext" href="/s?rh=n%3A1&amp;page=2">
<span id="pagnNextString">Next Page</span>
<span class="srSprite pagnNextArrow"></span>
</a>

然后我添加了标题：

from mechanize import Browser
from bs4 import BeautifulSoup
browser = Browser()

browser.addheaders = [('User-agent', 'Mozilla/5.0\
    (Windows NT 6.2; WOW64) AppleWebKit/537.11 (KHTML, like Gecko)\
    Chrome/23.0.1271.97 Safari/537.11')]

html_page = browser.open("http://www.amazon.com/s?rh=n%3A1")
soup = BeautifulSoup(html_page)
link = soup.find("a", {"title" : "Next Page"})
print link

输出：

<a title="Next Page" id="pagnNextLink" class="pagnNext" href="/s/ref=lp_1_pg_2/177-4872792-4084836?rh=n%3A283155%2Cn%3A%211000%2Cn%3A1&amp;page=2&amp;ie=UTF8&amp;qid=1376771097">
<span id="pagnNextString">Next Page</span>
<span class="srSprite pagnNextArrow"></span>
</a>

所以只需像这样添加标题信息

例子：

from bs4 import BeautifulSoup
import urllib2

req = urllib2.Request("http://www.amazon.com/s?rh=n%3A1")
req.add_header('User-agent', 'Mozilla/5.0\
            (Windows NT 6.2; WOW64) AppleWebKit/537.11 (KHTML, like Gecko)\
            Chrome/23.0.1271.97 Safari/537.11')

html_page = urllib2.urlopen(req)

if html_page.getcode() == 200:
    soup = BeautifulSoup(html_page)
    link = soup.find("a", {"title" : "Next Page"})
    print link['href']

else:
    print "Error loading page"

输出：

/s/ref=lp_1_pg_2/176-2670743-2970243?rh=n%3A283155%2Cn%3A%211000%2Cn%3A1&page=2&ie=UTF8&qid=1376771750
于 2013-08-17T20:27:13.120 回答
0

尝试，

import urllib2
from HTMLParser import HTMLParser

    class MyHTMLParser(HTMLParser):
        href = []
        def handle_starttag(self, tag, attrs):
            if tag == "a":
                for attr in attrs:
                    if attr[0] == "href" and 'page' in attr[1] and 'rh' in attr[1]:
                        self.href.append(attr[1])


    def _get(url):
        response = urllib2.urlopen(url)
        html = response.read()
        parser = MyHTMLParser()
        parser.feed(html.decode('utf-8'))
        href = parser.href
        print href

    _get('http://www.amazon.com/s?rh=n%3A1')
于 2013-08-17T20:19:30.163 回答