4

I am trying to scrape book genre information from google.

Much like when you put a calculation/conversion into google, the result displays in a box above the search results. I can scrape the data within this box in the browser (console) quite easily, however when I attempt the same code within casper, the content boxes don't appear anywhere in the code. The only way I can replicate this in browser is if I turn JS off.

I'm not sure why a different format is displayed to CasperJS and my own browser, but is there a way to get them to be the same? Here is the current code I am using, where

links= https://www.google.com/webhp?hl=en&tab=ww#safe=off&hl=en&output=search&sclient=psy-ab&q=The+Love+Affairs+of+a+Bibliomaniac+book+genre&oq=The+Love+Affairs+of+a+Bibliomaniac+book+genre

casper.start();
casper.userAgent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36        (KHTML, like Gecko) ');
casper.thenOpen(links, function() {

casper.waitForSelector('.answer_predicate', function() {
this.echo(this.getHTML('.answer_predicate'));

});



});
casper.run();
}
4

2 回答 2

4

运行以下：

var casper = require('casper').create({
  pageSettings: {
    loadImages: false,
    loadPlugins: false,
    userAgent: 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1588.0 Safari/537.36'
  }
});

links = 'https://www.google.com/webhp?hl=en&tab=ww#safe=off&hl=en&output=search&sclient=psy-ab&q=The+Love+Affairs+of+a+Bibliomaniac+book+genre&oq=The+Love+Affairs+of+a+Bibliomaniac+book+genre'

casper.start();

casper.thenOpen(links, function() {
  this.waitForSelector('.answer_predicate', function() {
    this.echo(this.getHTML('.answer_predicate'));
    this.echo(this.getElementInfo('.answer_predicate').text);
  });
});

casper.run();

给我这个输出：

<span class="kno-a-v">Fiction</span>
Fiction

我的假设是这里的问题与这里发布的问题相同。

于 2013-08-13T20:52:12.383 回答
0

@PAEz: Try this script, is pretty similar to the one hexid created, but is done with phantomjs-only (no casperjs).

Btw, since hexid answered first and I'm doing something pretty similar, I think he is the one that deserves the bounty. 

var page = require('webpage').create();
page.settings.userAgent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.71 Safari/537.36';
page.open('https://www.google.com/webhp?hl=en&tab=ww#bav=on.2,or.r_qf.&cad=b&fp=c210d6fe329544e7&hl=en&q=The+Love+Affairs+of+a+Bibliomaniac+book+genre&safe=off', function() {
    window.setTimeout(function() {
        var genre = page.evaluate(function() {
            return document.getElementsByClassName('answer_predicate')[0].textContent;
        });
        console.log(genre);
    }, 3000);
});

I'm using phantomjs 1.9.1 btw.

于 2013-08-15T23:33:47.567 回答