Question

I am parsing a webpage using a BufferedReader wrapped around an InputStream in Java:

HttpURLConnection conn = (HttpURLConnection)url.openConnection(myUrl);
InputStream stream = conn.getInputStream();

The problem is that the numbers I want to get from the page are dynamically generated with Ajax calls, and aren’t available in the stream. Is there some way I can refresh the stream after waiting a bit, or can anyone think of some other way to get the data? The page is here. The numbers I need are the “Dollar Volume” and “Share Volume” in the middle of the page.

Thanks,
Jared

Answer 1

If the numbers are generated with Ajax calls, then your code needs to make those Ajax calls. The only thing akin to “refreshing the stream” is reloading the page, and that’s still not going to have the numbers in if they’re usually loaded by separate web requests by the Javascript on the client.

Just find what requests the Javascript makes, and make the same requests from your own code.

(You should check that the web site in question is happy for you to scrape their data like this, by the way.)