Question

I am looking for a way to open many connections to websites in java.

My problem is that the server response times are really slow in many occasions. I therefore thought it would be a good idea to look for some java library that can handle opening many connections and that perhaps uses a handler when the connection/response is ready.

Is there something like (this is pseudo code!!)

getWebsiteContent(URL url) { //does not wait for the response
new AsynchronousConnection(url) {

    requestServedHandler(ConnectionObject) {
        InputStream is = ConnectionObject.getInputStream();

        //here I will process the content of the stream only when it is available
    }
} 
}

I would not mind a handler that gives me the raw HTML.

Answer 1

I’m not aware of such a specialized library, but you can create your own multi-threaded solution. Write your retriever task as a Callable that returns the HTML then submit these callables with the appropriate URLs to an ExecutorService. There are loads of code examples for this pattern on the net, but I could write you one if you ask.
A little more hint: This setup resembles to the quite common producer-consumer pattern. Your tasks could put the site contents into a BlockingQueue and let the processor thread poll in periodically to look for ready data. Either poll with a timeout if you want more liveliness or without, but that would block until data is available.