0

我刚刚开始使用 Scrapy：这是我要抓取的网站示例：

http://www.thefreedictionary.com/same

我的蜘蛛的代码：

from scrapy.spider import BaseSpider
from scrapy.http import Request
from scrapy.selector import HtmlXPathSelector
from dic_crawler.items import DicCrawlerItem

from urlBuilder import *   

class Dic_crawler(BaseSpider):
    name = "dic"
    allowed_domains = ["www.thefreedictionary.com"]
    start_urls = listmaker()[:]
    print start_urls

    def parse(self, response):
        hxs = HtmlXPathSelector(response)
        sites = hxs.select('//*[@id="MainTxt"]/table/tbody')
        print 'SITES:\n',sites


        item = DicCrawlerItem()

        item["meanings"] = sites.select('//*[@id="MainTxt"]/table/tbody/tr/td/div[1]/div[1]/div[1]/text()').extract()

        print item

        return item

listmaker() 返回要废弃的 url 列表。

我的问题是，如果我在 xpath 中选择 until 'tbody'并返回一个空的站点变量，则站点变量为空，而如果我只选择表，我将得到我想要的站点部分。

因此，我无法将单词的含义检索到 item["meanings"] 中，因为tbody之后的部分不会选择超出tbody。

此外，该网站给出了我想提取的多种含义，但我只知道如何提取一种方法。

谢谢

4

1 回答 1

1

这是一个蜘蛛骨架，可以帮助您入门：

from scrapy.spider import BaseSpider
from scrapy.http import Request
from scrapy.selector import HtmlXPathSelector

class Dic_crawler(BaseSpider):
    name = "thefreedictionary"
    allowed_domains = ["www.thefreedictionary.com"]
    start_urls = ['http://www.thefreedictionary.com/shame']

    def parse(self, response):
        hxs = HtmlXPathSelector(response)

        # loop on each "noun" or "verb" or something... section
        for category in hxs.select('id("MainTxt")//div[@class="pseg"]'):

            # this is simply to get what's in the <i> tag
            category_name = u''.join(category.select('./i/text()').extract())
            self.log("category: %s" % category_name)

            # for each category, a term can have multiple definition
            # category from .select() is a selector
            # so you can call .select() on it also,
            # here with a relative XPath expression selecting all definitions
            for definition in category.select('div[@class="ds-list"]'):
                definition_text = u'\n'.join(
                    definition.select('.//text()').extract())
                self.log(" - definition: %s" % definition_text)
于 2013-11-07T09:30:25.323 回答