2

我想从这个网站获取手机的费用 http://www.univercell.in/buy/SMART

我试图测试它所以我使用：scarpy shell http://www.univercell.in/control/AjaxCategoryDe ​​tail?productCategoryId=PRO-SMART&category_id=PRO-SMART&attrName=&min=&max=&sortSearchPrice=&VIEW_INDEX=2&VIEW_SIZE=15&serachupload=&sortupload=

但我无法连接到该站点。由于页面是使用 ajax 加载的，所以我使用 firebug 找到了 start_url。谁能建议我哪里出错了

4

2 回答 2

0

如何编写一个 JavaScript 脚本来执行单击页码时已经执行的操作，然后简单地转储从服务器返回的 XML。我的意思是尝试调用服务器，就好像该站点托管在您的桌面上一样！

当您点击一个数字时调用的 JavaScript 函数就是您要访问的页面的paginateList('numberOfPage')位置。numberOfPage

函数的主体是

function paginateList(viewIndex) {
        var productCategoryId = document.pageSelect.category_id.value;
        var viewSize = document.pageSelect.VIEW_SIZE.value;
        var min = "";
        if(document.pageSelect.min!=null)
            min = document.pageSelect.min.value;
        var max = "";
        if(document.pageSelect.max!=null)
            max = document.pageSelect.max.value;
        var attrName = "";
        if(document.pageSelect.attrName!=null)
        attrName = document.pageSelect.attrName.value;
        if(attrName==""){
         var commaAttr=document.getElementById('commaAttr'); 
          attrName=commaAttr.value;
          }
        var limitView = 'true';
        var sortSearchPrice = "";
        if(document.pageSelect.sortSearchPrice!=null)   
        sortSearchPrice = document.pageSelect.sortSearchPrice.value;
          var url2="/control/AjaxCategoryDetail?productCategoryId="+productCategoryId+"&category_id="+productCategoryId+"&attrName="+attrName+"&min="+min+"&max="+max+"&sortSearchPrice="+sortSearchPrice+"&VIEW_INDEX="+viewIndex+"&VIEW_SIZE="+viewSize+"&serachupload=&sortupload=";
            pleaseWait('Y');
            jQuery.ajax({url: url2,
             data: null,
             type: 'post',
             async: false,
             success: function(data) {
              $('#searchResult').html(data);  
              pleaseWait('N');   
             },
             error: function(data) {
                alert("Error during product searching");
             }
         });

使用这些递归地从每个页面获取数据。

希望能帮助到你！

于 2013-05-28T16:19:14.893 回答
0

这是你的蜘蛛：

from scrapy.item import Item, Field
from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector


class UnivercellItem(Item):
    vendor = Field()
    model = Field()
    price = Field()

BASE_URL = "http://www.univercell.in/control/AjaxCategoryDetail?productCategoryId=PRO-SMART&category_id=PRO-SMART&attrName=&min=&max=&sortSearchPrice=&VIEW_INDEX=%s&VIEW_SIZE=15&serachupload=&sortupload="

class UnivercellSpider(BaseSpider):
    name = "univercell_spider"
    allowed_domains = ["www.univercell.in"]
    start_urls = [BASE_URL % index for index in range(1, 21)]

    def parse(self, response):
        hxs = HtmlXPathSelector(response)
        mobiles = hxs.select("//div[@class='productsummary']")
        print mobiles
        for mobile in mobiles:
            item = UnivercellItem()
            item['vendor'] = mobile.select('.//div[1]/div/text()').extract()[0].strip()
            item['model'] = mobile.select('.//div[3]/div[1]/a/text()').extract()[0].strip()
            item['price'] = mobile.select('.//span[@class="regularPrice"]/span/text()').extract()[0].strip()
            yield item

将其保存到spider.py并通过scrapy runspider spider.py -o output.json. 然后output.json你会看到：

{"model": "T375", "vendor": "LG", "price": "Special Price Click Here"}
{"model": "P725 Optimus 3D Max", "vendor": "LG", "price": "Special Price Click Here"}
{"model": "P705 Optimus L7", "vendor": "LG", "price": "Special Price Click Here"}
{"model": "9320 Curve", "vendor": "Blackberry", "price": "Special Price Click Here"}
{"model": "Xperia Sola", "vendor": "Sony", "price": "Rs.14,500.00"}
{"model": "Xperia U", "vendor": "Sony", "price": "Special Price Click Here"}
{"model": "Lumia 610", "vendor": "Nokia", "price": "Special Price Click Here"}
...

希望有帮助。

于 2013-05-28T19:40:15.003 回答