1

我想使用 lxml 和 cssselect 从选择下拉框中获取/抓取数据。

到目前为止，我已经尝试了一些事情并在网上搜索了没有运气：

所以以下或类似的代码在我正在抓取的页面上：

<select id="attribute501" class="required-entry super-attribute-select" name="super_attribute[501]">
<option value="">Choose an Option...</option>
<option value="33">5</option>
<option value="31">10</option>
</select>

使用以下 cssselect 语句：

css_sel_price_dd = 'html body.catalog-product-view div.wrapper div.page div.main-container div.main div.col-main div.product-view div.product-essential form div.product-shop select'


pack_size_obj = root.cssselect(css_sel_price_dd)
print 'This strain has a drop down select box for different pack sizes: '
print pack_size_obj
pack_size = pack_size_obj[0].text_content().encode('utf-8').strip()
print 'Pack Size: ' + pack_size

产生以下结果：

This strain has a drop down select box for different pack sizes: 
[<SelectElement 10e77d590 name='super_attribute[501]'>]
Pack Size: Choose an Option...

但到目前为止，我无法从另一个选项访问文本值，即来自 5 和 10

<option value="33">5</option>
<option value="31">10</option>
  • 我已经尝试向 css 路径和各种东西添加选项，但似乎无法猜测语法并且在网上找不到任何东西......

仍然无法正常工作 - 这是带有 url 等的实际代码片段。

import lxml.html
import requests
import csv


user_agent = {'User-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/536.26.17 (KHTML, like Gecko) Version/6.0.2 Safari/536.26.17'}
# Open a session 
s = requests.session()

s.get('http://www.seedsman.com/en/cannabis-seeds', headers = user_agent)

css_sel_price_dd = '.product-shop select option'

open_seed_url = 'http://www.seedsman.com/en/amsterdam-mist-feminised-seeds'
print 'From the following URL: ' + open_seed_url

r = s.get(open_seed_url, headers = user_agent)
r_encoding = r.encoding
print 'encoding type of r:'
print r_encoding

myhtml = r.text # gives us a unicode string work with unicode and encode at the end when have piece of data we want

print myhtml.encode('utf-8')

root = lxml.html.fromstring(myhtml)

pack_size_obj = root.cssselect(css_sel_price_dd)
print pack_size_obj
for opt in pack_size_obj:
    print '{0} : {1}'.format(opt.get('value'), opt.text)

希望有人可以看到出了什么问题？

返回：

From the following URL: http://www.seedsman.com/en/amsterdam-mist-feminised-seeds
encoding type of r: UTF-8
** HTML removed for brevity ** 
[<Element option at 0x1108dcbf0>] 
None : Choose an Option... 
[Finished in 3.9s]
4

2 回答 2

3

您需要添加option到您的选择器，然后您将获得一个选项元素列表。看这个示例片段，只有 html 被简化了一点：

src_html = """
<html>
    <head></head>
    <body>
        <div>
            <select id="attribute501" class="required-entry super-attribute-select" name="super_attribute[501]">
                <option value="">Choose an Option...</option>
                <option value="33">5</option>
                <option value="31">10</option>
            </select>
        </div>
    </body>
</html>
"""
root = html.fromstring(src_html)

css_sel_price_dd = 'html body div select option'
pack_size_obj = root.cssselect(css_sel_price_dd)
for opt in pack_size_obj:
    print '{0} : {1}'.format(opt.get('value'), opt.text)
于 2013-04-30T11:47:18.297 回答
0

您可以使用xpath 

options = root.xpath("//select[@id='attribute501']/option")
option_text = [option.text for option in options]

或者

options = root.xpath("//select[@id='attribute501']/option/text()")
于 2015-01-14T05:11:58.447 回答