python - 将 Scrapy 与 CasperJS 相结合

Question

当我使用 Scrapy-Splash 尝试 Splash 时，它不支持像真正的浏览器一样导航。它只呈现 HTML，但是当它单击一个按钮时，它不会呈现它导航到的下一页。

根据我的简单研究，只有 CasperJS 支持使用导航进行网页抓取。但是，是否可以结合 Scrapy 和 CasperJS

Answer 1

You're confusing browser with testing frameworks, Selenium does exactly that but instead of Firefox, use PhantomJS as a headless browser (which is what CasperJS uses).

Click method

# Assume the button has the ID "submit" :)
driver.find_element_by_id("submit").click()

alexce answer on scrapy with selenium

If you need the content of the requests, use something like BrowserMob as a proxy.