0

我正在尝试为 scrapy 编写 mysql 管道，但出现“导入错误”： 

File "C:\Python27\lib\site-packages\scrapy-0.14.3-py2.7-win32.egg\scrapy\utils\misc.py", line 39,    
in load_object raise ImportError, "Error loading object '%s': %s" % (path, e)
ImportError: Error loading object 'tutorial.pipelines.MySQLStore': No module named exceptions

管道的代码是：

from scrapy import log
from scrapy.core.exceptions import DropItem
from twisted.enterprise import adbapi

import time
import MySQLdb.cursors

class FilterWords(object):
    """A pipeline for filtering out items which contain certain words in their
    description"""

    # put all words in lowercase
   words_to_filter = ['politics', 'religion']

       def process_item(self, spider, item):
       print spider
       for word in self.words_to_filter:
           if word in unicode(item['description']).lower():
               raise DropItem("Contains forbidden word: %s" % word)
       else:
           return item

class MySQLStore(object):

    def __init__(self):
        # @@@ hardcoded db settings
        # TODO: make settings configurable through settings
        self.dbpool = adbapi.ConnectionPool('',
            db='',
            user='',
            passwd='',
            cursorclass=MySQLdb.cursors.DictCursor,
            charset='utf8',
            use_unicode=True
        )

    def process_item(self, spider, item):
        # run db query in thread pool
        query = self.dbpool.runInteraction(self._conditional_insert, item)
        query.addErrback(self.handle_error)

        return item

    def _conditional_insert(self, tx, item):
        # create record if doesn't exist. 
        # all this block run on it's own thread
        tx.execute("select * from scrapytest where link = %s", (item['link'][0], ))
        result = tx.fetchone()
        if result:
            log.msg("Item already stored in db: %s" % item, level=log.DEBUG)
        else:
            tx.execute(\
                "insert into scrapytest (title, link, desc) "
                "values (%s, %s, %s)",
                (item['title'][0],
                item['link'][0],
                item['desc'][0])
            )
           log.msg("Item stored in db: %s" % item, level=log.DEBUG)

    def handle_error(self, e):
        log.err(e)

我曾尝试使用其他管道（如 Json）并且它们可以工作，但是这里有一些我不明白的导入问题。

4

1 回答 1

2

错误信息非常清楚！

ImportError: ... No module named exceptions

您正在尝试 import exceptions，但没有这样的模块。查看导入该模块的代码：

from scrapy.core.exceptions import DropItem

这应该是：

from scrapy.exceptions import DropItem

没有scrapy.core.exceptions模块。

于 2012-05-04T13:40:17.840 回答