4

我的主要蜘蛛代码：

    from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from Belray_oil.items import BelrayOilItem

class BelraySpider(BaseSpider):
    name = "Belray_oil"
    allowed_domains = ["mxdirtrider.com/"]
    start_urls = ["http://www.mxdirtrider.com/h-products/bel-ray/2011-02/pr-bel-ray-accessories-lubricant-oil-2-stroke-2t-mineral-engine.htm?ref=search"]

def parse(self, response):
    hxs = HtmlXPathSelector(response)
    name = hxs.select("//div[@id='product-title']/h1/span/text()").extract()
    MSRP = hxs.select("//div[@id='price']/span[1]/text()").extract()
    Sale = hxs.select("//div[@id='price']/span[2]/strong/text()").extract()
    print name, MSRP, Sale

我的物品文件：

    from scrapy.item import Item, Field

class BelrayOilItem(Item):
    name = Field()
    MSRP = Field()
    Sale = Field()

我运行时的终端日志输出：scrapy crawl Belray_oil -o items.csv -t csv

    2013-07-05 18:03:25-0400 [scrapy] INFO: Scrapy 0.14.4 started (bot: Belray_oil)
2013-07-05 18:03:25-0400 [scrapy] DEBUG: Enabled extensions: FeedExporter, LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, MemoryUsage, SpiderState
2013-07-05 18:03:25-0400 [scrapy] DEBUG: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, RedirectMiddleware, CookiesMiddleware, HttpCompressionMiddleware, ChunkedTransferMiddleware, DownloaderStats
2013-07-05 18:03:25-0400 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2013-07-05 18:03:25-0400 [scrapy] DEBUG: Enabled item pipelines: 
2013-07-05 18:03:25-0400 [Belray_oil] INFO: Spider opened
2013-07-05 18:03:25-0400 [Belray_oil] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2013-07-05 18:03:25-0400 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023
2013-07-05 18:03:25-0400 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
2013-07-05 18:03:26-0400 [Belray_oil] DEBUG: Crawled (200) <GET http://www.mxdirtrider.com/h-products/bel-ray/2011-02/pr-bel-ray-accessories-lubricant-oil-2-stroke-2t-mineral-engine.htm?ref=search> (referer: None)
2013-07-05 18:03:26-0400 [Belray_oil] ERROR: Spider error processing <GET http://www.mxdirtrider.com/h-products/bel-ray/2011-02/pr-bel-ray-accessories-lubricant-oil-2-stroke-2t-mineral-engine.htm?ref=search>
    Traceback (most recent call last):
      File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1182, in mainLoop
        self.runUntilCurrent()
      File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 805, in runUntilCurrent
        call.func(*call.args, **call.kw)
      File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 381, in callback
        self._startRunCallbacks(result)
      File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 489, in _startRunCallbacks
        self._runCallbacks()
    --- <exception caught here> ---
      File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 576, in _runCallbacks
        current.result = callback(current.result, *args, **kw)
      File "/usr/lib/python2.7/dist-packages/scrapy/spider.py", line 62, in parse
        raise NotImplementedError
    exceptions.NotImplementedError: 

2013-07-05 18:03:26-0400 [Belray_oil] INFO: Closing spider (finished)
2013-07-05 18:03:26-0400 [Belray_oil] INFO: Dumping spider stats:
    {'downloader/request_bytes': 310,
     'downloader/request_count': 1,
     'downloader/request_method_count/GET': 1,
     'downloader/response_bytes': 13379,
     'downloader/response_count': 1,
     'downloader/response_status_count/200': 1,
     'finish_reason': 'finished',
     'finish_time': datetime.datetime(2013, 7, 5, 22, 3, 26, 204316),
     'scheduler/memory_enqueued': 1,
     'spider_exceptions/NotImplementedError': 1,
     'start_time': datetime.datetime(2013, 7, 5, 22, 3, 25, 970550)}
2013-07-05 18:03:26-0400 [Belray_oil] INFO: Spider closed (finished)
2013-07-05 18:03:26-0400 [scrapy] INFO: Dumping global stats:
    {'memusage/max': 116150272, 'memusage/startup': 116150272}

输出中的 CSV 始终为空，我无法弄清楚到底是什么问题。请小帮助！

4

1 回答 1

4

Item你应该在你的parse方法中返回一个：

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from Belray_oil.items import BelrayOilItem 


class BelraySpider(BaseSpider):
    name = "Belray_oil"
    allowed_domains = ["mxdirtrider.com"]
    start_urls = [
        "http://www.mxdirtrider.com/h-products/bel-ray/2011-02/pr-bel-ray-accessories-lubricant-oil-2-stroke-2t-mineral-engine.htm?ref=search"]

    def parse(self, response):
        hxs = HtmlXPathSelector(response)
        item = BelrayOilItem()
        item['name'] = hxs.select("//div[@id='product-title']/h1/span/text()").extract()
        item['MSRP'] = hxs.select("//div[@id='price']/span[1]/text()").extract()
        item['Sale'] = hxs.select("//div[@id='price']/span[2]/strong/text()").extract()
        return item

然后，items.csv您将拥有：

name,msrp,sale
Bel-Ray 2T Mineral Engine 2-Stroke  ,MSRP $9.75,$8.13

希望有帮助。

于 2013-07-06T18:53:56.220 回答