Question

I am currently using JTidy to parse an HTML document and fetch a collection of all anchor tags in the given HTML document. I then extract the value of each tag’s href attribute to come up with a collection of links on the page.

Unfortunately, these links can be expressed in a few different ways: some absolute (http://www.example.com/page.html), some relative (/page.html, page.html, or ../page.html). Even more, some can just be anchors (#paragraphA). When I visit my page in a browser, it knows automatically how to handle these different href values if I were to click the link, however if I were to follow one of these links retrieved from JTidy using an HTTPClient programatically, I first need to provide a valid URL (so e.g. I would first need to transform /page.html, page.html, and http://www.example.com/page.html to http://www.example.com/page.html).

Is there some built-in functionality, whether in JTidy or elsewhere, that can achieve this for me? Or will I need to create my own rules to transform these different URLs into an absolute URL?

Answer 1

The vanilla URL class might get you most of the way there, assuming you can work out which context to use. Here are some examples:

package grimbo.url;

import java.net.MalformedURLException;
import java.net.URL;

public class TestURL {
    public static void main(String[] args) {
        // context1
        URL c1 = u(null, "http://www.example.com/page.html");
        u(c1, "http://www.example.com/page.html");
        u(c1, "/page.html");
        u(c1, "page.html");
        u(c1, "../page.html");
        u(c1, "#paragraphA");

        System.out.println();

        // context2
        URL c2 = u(null, "http://www.example.com/path/to/page.html");
        u(c2, "http://www.example.com/page.html");
        u(c2, "/page.html");
        u(c2, "page.html");
        u(c2, "../page.html");
        u(c2, "#paragraphA");
    }

    public static URL u(URL context, String url) {
        try {
            URL u = null != context ? new URL(context, url) : new URL(url);
            System.out.println(u);
            return u;
        } catch (MalformedURLException e) {
            e.printStackTrace();
            return null;
        }
    }
}

Results in:

http://www.example.com/page.html
http://www.example.com/page.html
http://www.example.com/page.html
http://www.example.com/page.html
http://www.example.com/../page.html
http://www.example.com/page.html#paragraphA

http://www.example.com/path/to/page.html
http://www.example.com/page.html
http://www.example.com/page.html
http://www.example.com/path/to/page.html
http://www.example.com/path/page.html
http://www.example.com/path/to/page.html#paragraphA

As you can see, there are some results that aren’t what you want. So maybe you try and parse the URL using new URL(value) first, and if that results in a MalformedURLException you could try relative to a context URL.