0

我遇到了4天的这个问题，我陷入了死胡同。我想抓取“ http://www.ledcor.com/careers/search-careers ”。在每个职位列表页面（即http://www.ledcor.com/careers/search-careers?page=2）上，我进入每个职位链接并获取职位名称。到目前为止，我有这个工作。

现在，我正试图让蜘蛛进入下一个工作列表页面（例如从http://www.ledcor.com/careers/search-careers?page=2到http://www.ledcor.com/careers/ search-careers?page=3并抓取所有工作）。我的爬取规则不起作用，我不知道什么是错的，什么是缺失的。请帮忙。

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import HtmlXPathSelector
from craigslist_sample.items import CraigslistSampleItem

class LedcorSpider(CrawlSpider):
    name = "ledcor"
    allowed_domains = ["www.ledcor.com"]
    start_urls = ["http://www.ledcor.com/careers/search-careers"]


    rules = [
        Rule(SgmlLinkExtractor(allow=("http://www.ledcor.com/careers/search-careers\?page=\d",),restrict_xpaths=('//div[@class="pager"]/a',)), follow=True),
        Rule(SgmlLinkExtractor(allow=("http://www.ledcor.com/job\?(.*)",)),callback="parse_items")
    ]

def parse_items(self, response):
    hxs = HtmlXPathSelector(response)
    item = CraigslistSampleItem()
    item['title'] = hxs.select('//h1/text()').extract()[0].encode('utf-8')
    item['link'] = response.url
    return item

这是 Items.py

from scrapy.item import Item, Field

class CraigslistSampleItem(Item):
    title = Field()
    link = Field()
    desc = Field()

这是 Pipelines.py

class CraigslistSamplePipeline(object):
    def process_item(self, item, spider):
        return item

更新：（@blender 建议）它不会爬行

rules = [
    Rule(SgmlLinkExtractor(allow=(r"http://www.ledcor.com/careers/search-careers\?page=\d",),restrict_xpaths=('//div[@class="pager"]/a',)), follow=True),
    Rule(SgmlLinkExtractor(allow=("http://www.ledcor.com/job\?(.*)",)),callback="parse_items")
]
4

3 回答 3

1

你的restrict_xpaths论点是错误的。删除它，它会工作。

$ scrapy shell http://www.ledcor.com/careers/search-careers

In [1]: from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor

In [2]: lx = SgmlLinkExtractor(allow=("http://www.ledcor.com/careers/search-careers\?page=\d",),restrict_xpaths=('//div[@class="pager"]/a',))

In [3]: lx.extract_links(response)
Out[3]: []

In [4]: lx = SgmlLinkExtractor(allow=("http://www.ledcor.com/careers/search-careers\?page=\d",))

In [5]: lx.extract_links(response)
Out[5]: 
[Link(url='http://www.ledcor.com/careers/search-careers?page=1', text=u'', fragment='', nofollow=False),
 Link(url='http://www.ledcor.com/careers/search-careers?page=2', text=u'2', fragment='', nofollow=False),
 Link(url='http://www.ledcor.com/careers/search-careers?page=3', text=u'3', fragment='', nofollow=False),
 Link(url='http://www.ledcor.com/careers/search-careers?page=4', text=u'4', fragment='', nofollow=False),
 Link(url='http://www.ledcor.com/careers/search-careers?page=5', text=u'5', fragment='', nofollow=False),
 Link(url='http://www.ledcor.com/careers/search-careers?page=6', text=u'6', fragment='', nofollow=False),
 Link(url='http://www.ledcor.com/careers/search-careers?page=7', text=u'7', fragment='', nofollow=False),
 Link(url='http://www.ledcor.com/careers/search-careers?page=8', text=u'8', fragment='', nofollow=False),
 Link(url='http://www.ledcor.com/careers/search-careers?page=9', text=u'9', fragment='', nofollow=False),
 Link(url='http://www.ledcor.com/careers/search-careers?page=10', text=u'10', fragment='', nofollow=False)]
于 2013-10-01T03:12:41.343 回答
1

您需要转义问号并为正则表达式使用原始字符串：

r"http://www\.ledcor\.com/careers/search-careers\?page=\d"

否则，它会查找类似...careerspage=2和的 URL ...carrerpage=3。

于 2013-09-30T03:39:28.863 回答
0

尝试这个：

rules = [Rule(SgmlLinkExtractor(), follow=True, callback="parse_items")]

此外，需要进行适当的更改pipeline.py并粘贴管道和项目代码。

于 2013-09-30T06:28:02.547 回答