Question

I need a good page rendering library so that I can extract all links(including anchor text, the underlying hyperlink, absolute position of the link on the page) from a web page.

I have been using the CSSBox library, however, the href attribute is missing from the rendered box model. In other words, I can only obtain anchor text and position of the link using CSSBox alone. I have to take advantage of the anchor text and another html parsing library(e.g.,Jsoup) to get the href attribute(i.e., the de facto URL).

Is there any library that can better achieve my goal?

Answer 1

Recommendation

Consider using Geb:

It brings together the power of WebDriver, the elegance of jQuery
content selection, the robustness of Page Object modelling and the
expressiveness of the Groovy language.

It can be used for scripting, scraping and general automation — or
equally as a functional/web/acceptance testing solution via
integration with testing frameworks such as Spock, JUnit & TestNG.

Requirements

As mentioned, this is only suitable if you are open to the use of Groovy. However, since Groovy integrates so easily with Java, this typically isn’t a problem.

Furthermore, this will require a browser. I’m not sure if this is a deal breaker for you.

Usage

From the docs:

You can obtain the size and location of content on the page. All units
are in pixels. The size is available via the height and width
properties, while the location is available as the x and y properties
which represent the distance from the top left of the page (or parent
frame) to the top left point of the content.

For example:

Browser.drive {
  // Load the page
  go "http://myapp.com"

  // $("a") returns all hyperlinks on the page, similar to jQuery
  $("a").each { a ->
     // Display the required link properties and attributes
     println """
        The link to '${a.@href}' with text '${a.text()}' is at location (${a.x}, ${a.y}),
        with a height of ${a.height}px and a width of ${a.width}px.
     """
  }
}