javascript - 由于 javascript 变量，无法使用 beautifulsoup 和 urllib 进行抓取

Question

不幸的是，我是 beautifulsoup 和 urllib 的新手，所以我什至可能无法正确询问我需要什么。有一个网站 www.example.com 我需要从该网站中提取一些数据，该网站显示一条随机消息。

问题是用户按下按钮后会显示消息，否则会显示类似“按下按钮以查看消息”的一般消息。

在搜索stackoverflow之后，我意识到可能没有办法通过使用我的浏览器调用这样的url来更改变量.. www.example.com/?showRandomMsg='true'

在某些线程中，我读到也许我可以用 bookmarlets 做到这一点。

无论如何使用带有 beautifulsoup 或 urllib 的书签来访问该网站并使其显示随机消息？

提前致谢！:D

Answer 1

I came back after a long time just to answer quickly my own question..

I found many solutions and tutorials on the web and most of them were suggesting using Selenium and xpath but this method was more complex than I needed..

So I ended up using Selenium ONLY for emulating the Browser (firefox in my case) and grabbing the html after the page was loaded completely.

After that I was still using beautifoulsoup to parse the html code (whihc now would include the javascript data too).