0

我写了一个蜘蛛从网站下载数据，然后点击链接获取详细数据。蜘蛛还使用默认的scrapy图像管道下载图像。到目前为止一切正常。

但是当我第二次[使用另一个搜索词]启动蜘蛛时，图像下载不再起作用。爬行就像应该的那样工作。我没有收到任何错误。

这是蜘蛛：

class DiscoSpider(BaseSpider):

    def __init__(self, query):
        super( BaseSpider, self ).__init__()
        self.name = "discogs"
        self.allowed_domains = ["discogs.com"]
        self.start_urls = [
            "http://www.discogs.com/search?q=%s&type=release" % query
        ]

    # parse all releases for the current search
    def parse(self, response):
        logging.debug('scrapy.parse')
        hxs = HtmlXPathSelector(response)
        li = hxs.select("//div[@id='page_content']/ol/li")

        items = []
        for l in li:
            item = DiscogsItem()

            ...

            # get the link for the callback for the tracklist
            link = l.select("a/@href").extract()[0]
            item['link'] = '' if link == None else link

            # get the img location
            img = l.select("a/img/@src").extract() 
            item['image_urls'] = [None] if img == None else img 

            # get the url for the tracklist callback
            url = urlparse.urljoin('%s%s' % ('http://www.', self.allowed_domains[0]), link)

            # request and callback to get tracklist for release
            item = Request(url, meta={'item':item}, callback=self.parse_tracklist)

            items.append(item)
            yield item        

    # callback to get the tracklist for each release
    def parse_tracklist(self, response):
        item = response.request.meta['item']

        hxs = HtmlXPathSelector(response)
        rows = hxs.select("//div[@class='section_content']/table[@class='playlist mini_playlist']/tr")
        tracklist = []

        for row in rows:
            track = {}

            title = row.select("td[@class='track']/span[@class='track_title']/text()").extract()
            track['title'] = '' if title in [None, '', []] else self.clean_track(title[0])

            ...

            tracklist.append(track)

        item['tracklist'] = tracklist    
        yield item

这是项目：

class DiscogsItem(Item):
    # define the fields for your item here like:
    link = Field()
    artist = Field()
    release = Field()
    label = Field()
    year = Field()
    tracklist = Field()
    image_urls = Field()
    images = Field()
    thumb = Field()

在我的scrapy设置中：

ITEM_PIPELINES = ['scrapy.contrib.pipeline.images.ImagesPipeline']
IMAGES_STORE = '/home/f/work/py/discogs/tmp'
CONCURRENT_REQUESTS = 100
CONCURRENT_REQUESTS_PER_IP = 20
IMAGES_EXPIRES = 0

我在一个单独的过程中从 PyQt-UI 运行蜘蛛，我是 Scrapy / PyQT / StackOverflow 的新手（抱歉格式错误）。

我在一个带有 Python 2.7、PyQt4 和 Scrapy 0.12.0.2546 的 Xubuntu 12.04 机器上。

有谁知道为什么第二个图像下载不起作用？

提前致谢。

4

1 回答 1

0

我现在回答我自己的问题，虽然我真的不知道问题是什么。我所做的是像这样更改蜘蛛的构造函数：

class DiscogsSpider(CrawlSpider):

    name = "discogs"
    allowed_domains = ["discogs.com"]

    def __init__(self, query):
        super( DiscogsSpider, self ).__init__()
        self.start_urls = [
            "http://www.discogs.com/search?q=%s&type=release" % query
        ]

我现在也扩展了 CrawlSpider 而不是 BasSpider。图像下载现在按预期工作。

谢谢。

于 2013-02-26T11:43:34.327 回答