0

I want to extract all the product url from the link "http://www.shopclues.com/diwali-mega-mall/hot-electronics-sale-fs/audio-systems-fs.html" using scrapy in python. Below is the function I'm using to do this:

def parse(self, response):
        print("hello");

        hxs = HtmlXPathSelector(response)
        sites = hxs.select('//div[@id="pagination_contents"]')
        items = []
        i=3
    for site in sites:
            item = DmozItem()
            item['link'] = site.select('div[2]/div['+str(i)+']/a/@href').extract()
            i=int(i)+1;
            print i
            items.append(item)
    return items

The x-path of each product div is: //div[@id="pagination_contents"]/div[2]/div['+str(i)+']/a/@href

But I'm getting only one link and not all the products' url.

4

1 回答 1

1

我认为你的问题是hxs.select('//div[@id="pagination_contents"]')只返回一个结果，然后你只在循环中进行一次迭代。

您可以选择以下所有<div>包含<a>, 的元素并循环这些元素：

sites = hxs.select('//div[@id="pagination_contents"]/div[2]/div[a]')
for site in sites:
    ## This loop will run 33 times in my test.
    ## Access to each link:
    item['link'] = site.select('./a[2]/@href').extract()
于 2013-10-26T14:50:18.843 回答