Home/ Questions/Q 8000537
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-04T15:49:43+00:00

I’m not sure if this is possible but I would like to retrieve some

  • 0

I’m not sure if this is possible but I would like to retrieve some data from a web page that uses Javascript to render data. This would be from a linux shell.

What I am able to do now:

  1. http post using curl/lynx/wget to login and get headers from command line

  2. use headers to get into ‘secure’ locations in the webpage on command line

However, the only elements that are rendered on the page are the static html. Most of the info I need are rendered dynamically with js (albeit eventually as a html as well) and don’t show up on a command line browser. I understand the issue is with the lack of a js interpreter.

As such… some workarounds I thought might be possible are:

  1. calling full browsers from command line and somehow passing the info back to stdout. this would mean that I have to be able to POST.

  2. passing the headers (with session info, etc…) i got from curl to one of these full browsers and again dumping the output html back to stdout. it could very be a printscreen function on the window if all else fails.

  3. a pure java solution would be OK too.

Anyone has any experience doing something similar and succeeding?

Thanks!

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    You can use WebDriver to do, just that you need have web browser installed. There are other solution as well such as Selenium and HtmlUnit (without browser but might behave differently).

    You can find example of Selenium project at here.

    WebDriver

    WebDriver is a tool for writing automated tests of websites. It aims
    to mimic the behaviour of a real user, and as such interacts with the
    HTML of the application.

    Selenium

    Selenium automates browsers. That’s it. What you do with that power is
    entirely up to you. Primarily it is for automating web applications
    for testing purposes, but is certainly not limited to just that.
    Boring web-based administration tasks can (and should!) also be
    automated as well.

    HtmlUnit

    HtmlUnit is a “GUI-Less browser for Java programs”. It models HTML
    documents and provides an API that allows you to invoke pages, fill
    out forms, click links, etc… just like you do in your “normal”
    browser.

    I would recommend use WebDriver because it is not required standalone server like Selenium, while for HtmlUnit might suitable if you dont want install browser without worry about Xvfb in headless environment.

    • 0