Question

I am trying to determine if there is a way to check the availability of a potentially large list of urls (> 1000000) without having to send a GET request to every single one.

Is it safe to assume that if http://www.example.com is inaccessible (as in unable to connect to server or the DNS request for the domain fails), or I get a 4XX or 5XX response, then anything from that domain will also be inaccessible (e.g. http://www.example.com/some/path/to/a/resource/named/whatever.jpg)? Would a 302 response (say for whatever.jpg) be enough to invalidate the first assumption? I imagine sub domains should be considered distinct as http://subdomain.example.com and http://www.example.com may not direct to the same ip?

I seem to be able to think of a counter example for each shortcut I come up with. Should I just bite the bullet and send out GET requests to every URL?

Answer 1

Unfortunately, no you cannot infer anything from 4xx or 5xx or any other codes.

Those codes are for individual pages, not for the server. It’s quite possible that one page is down and another is up, or one has a 500 server-side error and another doesn’t.

What you can do is use HEAD instead of GET. That retrieves the MIME header for the page but not the page content. This saves time server-side (because it doesn’t have to render the page) and for yourself (because you don’t have to buffer and then discard content).

Also I suggest you use keep-alive to accelerate responses from the same server. Many HTTP client libraries will do this for you.