html - Jsoup Delay due to Streaming Website

Question

A question regarding Jsoup: I am building a tool that fetches prices from a website. However, this website has streaming content. If I browse manually, I see the prices of 20 mins ago and have to wait about 3 secs to get the current price. Is there any way I can make some kind of delay in Jsoup to be able to obtain the prices in the streaming section? I am using this code:

conn = Jsoup.connect(link).userAgent("Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.72 Safari/537.36");

conn.timeout(5000);

doc = conn.get();

Answer 1

You can Use a JavaFX WebView with javascript enabled. After waiting the two seconds, you can extract the contents and pass them to JSoup.

(After loading your url into your WebView using the example above)
String text=view.getEngine() executeScript("document.documentElement.outerHTML");
Document doc = Jsoup.parse(html);

Answer 2

正如评论中提到的，该站点很可能使用某种类型的脚本，而 Jsoup 则无法使用。由于 Jsoup 只是获取初始 HTML 响应，并且不执行任何 javascript。

我想给你一些关于现在去哪里的更多指导。在这些情况下，最好的选择是为这些类型的网站转移到另一个平台。您可以迁移到HTMLUnit，它是一个无头浏览器，或Selenium，它可以使用 HTMLUnit 或真正的浏览器，如 Firefox 或 Chrome。如果您认为自己需要超越 HTMLUnit，我会推荐 Selenium，因为与 Selenium 可以支持的消费者浏览器相比，HTMLUnit 有时可能不太稳定。您可以将 Selenium 与 HTMLUnit 驱动程序一起使用，让您可以选择稍后无缝移动到另一个浏览器。