0

我想知道为什么我的链接提取器没有任何链接。我正在我的链接提取器中扩展CrawlSpider和设置tagsandattrs属性。我正在尝试从网页上的选择框中删除链接（http://www.realclearpolitics.com/epolls/latest_polls/president/），因此我正在寻找具有值属性的选项标签。

在此处查看代码：https ://github.com/jtfairbank/RCP-Poll-Scraper/blob/master/polldata/spiders/pres.py

这是根目录：https ://github.com/jtfairbank/RCP-Poll-Scraper

有任何想法吗？谢谢！

4

1 回答 1

0

您的链接提取器没有获得任何链接，因为您正在搜索带有的<option>标签，并且此页面tags=['option']上没有选择或选项标签。

尝试restrict_xpaths='//td[@class="lp-race"]'：

>>> SgmlLinkExtractor(
...     allow=(r"epolls/2012/president/[a-z]{2}/[a-z]+_romney_vs_obama-[0-9]{4}\.html"),
...     allow_domains=('realclearpolitics.com',),
...     restrict_xpaths='//td[@class="lp-race"]',
... ).extract_links(response)

[Link(url='http://www.realclearpolitics.com/epolls/2012/president/fl/florida_romney_vs_ob...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/oh/ohio_romney_vs_obama...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/va/virginia_romney_vs_o...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/ia/iowa_romney_vs_obama...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/wi/wisconsin_romney_vs_...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/or/oregon_romney_vs_oba...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/mo/missouri_romney_vs_o...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/ca/california_romney_vs...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/md/maryland_romney_vs_o...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/mi/michigan_romney_vs_o...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/co/colorado_romney_vs_o...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/mn/minnesota_romney_vs_...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/wa/washington_romney_vs...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/ct/connecticut_romney_v...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/nv/nevada_romney_vs_oba...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/mt/montana_romney_vs_ob...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/pa/pennsylvania_romney_...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/in/indiana_romney_vs_ob...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/ma/massachusetts_romney...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/ga/georgia_romney_vs_ob...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/id/idaho_romney_vs_obam...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/az/arizona_romney_vs_ob...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/me/maine_romney_vs_obam...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/la/louisiana_romney_vs_...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/hi/hawaii_romney_vs_oba...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/tx/texas_romney_vs_obam...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/ar/arkansas_romney_vs_o...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/ne/nebraska_romney_vs_o...',
于 2012-10-20T18:45:33.190 回答