Question

I’m trying to make a client that can send HTTP requests and receive responses from web servers. I tried using Java’s HttpURLConnection class but it doesn’t give me enough control over what actually gets sent to the server, so I’d like to compose my own HTTP request messages and send them over a Socket. However, reading from the Socket’s InputStream is prohibitively slow for some servers, and I’d like to speed that up if possible. Here’s some code that I used to test how slow the reads were for the socket as compared to the HttpURLConnection:

public static void useURLConnection() throws Exception
{
    URL url = new URL("http://" + hostName + "/");
    HttpURLConnection conn = (HttpURLConnection) url.openConnection();
    InputStream in = conn.getInputStream();
    byte[] buffer = new byte[buffersize];
    long start = System.currentTimeMillis();
    while(in.read(buffer) != -1) { }
    System.out.println(System.currentTimeMillis() - start);
}

public static void useSocket() throws Exception
{
    byte[] request = ("GET / HTTP/1.1\r\nHost: " + hostName + "\r\n\r\n").getBytes();
    Socket socket = new Socket(hostName, 80);
    OutputStream out = socket.getOutputStream();
    InputStream in = socket.getInputStream();
    out.write(request);
    byte[] buffer = new byte[buffersize];
    long start = System.currentTimeMillis();
    while(in.read(buffer) != -1) { }
    System.out.println(System.currentTimeMillis() - start);
}

Both methods run in about the same amount of time for some servers, such as http://www.wikipedia.org, but reading from the socket is much slower — minutes as opposed to milliseconds — for others, such as http://www.google.com. Can someone explain why this is, and perhaps give me some pointers as to what, if anything, I can do to speed up the reads from the socket? Thanks.

Answer 1

So, HTTP/1.1 turns on keepalive by default for client requests. In your socket example, you’re sending HTTP/1.1 as your version string, so you’re implicitly accepting that you can support keepalive, yet you’re completely disregarding it.

Basically, you’re blocking trying to read more from the server, despite the fact that the server is waiting for you to do something (either send another request or close the connection.)

You need to either send a header “Connection: close” or send HTTP/1.0 as your version string.