我正在使用 scrapy 来提取部分地址,我需要有关 . 这是代码(抱歉,如果这是无效代码,不确定如何正确粘贴到问题中)。
<div class="result">
<h3>
<a href="/provider/service/xxxxx/">service name</a>
</h3>
<p>
"blah blah"
</p>
<strong>Physical Address</strong>
"123 address street, someplace, somewhere"
<br/>
<strong>Postcode</strong>
"xxx"
<br/>
<strong>District/town</strong>
"someplace"
<br/>
<strong>Region</strong>
"someplace bigger"
<br/>
<strong>Phone</strong>
"xx xxx xxxx"
<br/><strong>Fax Number</strong>
"xx xxx xxxx"
<br/>
<!--strong>Email</strong-->
<a href="#" onclick="window.location=('mail'+'to:'+'xxxxx'+''+'@'+'xxxx.xx.xx'+''); return false;">
"xxxxx"
<strong></strong>
"xxxxx.xx.xx"
</a>
<a rel="nofollow" class="printlist-add" href="/provider/print-list/add/xxxx/">Add to print list</a>
</div>
<hr/>
这是我的蜘蛛
from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from test.items import TestItem
class NewSpider(BaseSpider):
name = "my_spider"
download_delay = 2
allowed_domains = ["website.com"]
start_urls = [
"http://website.com/site1"
]
def parse(self, response):
hxs = HtmlXPathSelector(response)
sites = hxs.select('//*[@id="search-results"]/div')
items = []
for site in sites:
item = WebhealthItem()
item['practice'] = site.select('h3/a/text()').extract()
item['url'] = site.select('h3/a/@href').extract()
item['address1'] = site.select('strong[text() = "Physical Address"]/following-sibling::text()[1]')
items.append(item)
return items
该行item['address1'] = site.select('strong[text()="Physical Address"]/following-sibling::text()[1]')
返回一个字符串值[<HtmlXPathSelector xpath='strong[text()="Physical Address"]/following-sibling::text()[1]' data=u'\n\t\t\t 123 address street, someplace, some'>]
。最后几个字符被剪掉。
当我添加.extract()
这些值时,它们会显示在 cmd 中,[u'\n\t\t\t 123 address street, someplace, somewhere']
但它们不会出现在输出表中。
我一直在寻找解决方案,并且尝试过.select('text()').extract()
,但这也不对。
一如既往地非常感谢任何帮助。
附言。关于如何将页面源代码放入此论坛的问题的建议也将不胜感激。谢谢