Question

I want to download a HTTP query with java, but the file I download has an undetermined length when downloading.

I thought this would be quite standard, so I searched and found a code snippet for it: http://snipplr.com/view/33805/

But it has a problem with the contentLength variable. As the length is unknown, I get -1 back. This creates an error. When I omit the entire check about contentLength, that means I always have to use the maximum buffer.

But the problem is that the file is not ready yet. So the flush gets only partially filled, and parts of the file get lost.

If you try downloading a link like http://overpass-api.de/api/interpreter?data=area%5Bname%3D%22Hoogstade%22%5D%3B%28++node%28area%29%3B++%3C%3B%29+%3Bout+meta+qt%3B with that snippet, you’ll notice the error, and when you always download the maximum buffer to omit the error, you end up with a corrupt XML file.

Is there some way to only download the ready part of the file? I would like if this could download big files (up to a few GB).

Answer 1

This should work, i tested it and it works for me:

void downloadFromUrl(URL url, String localFilename) throws IOException {
    InputStream is = null;
    FileOutputStream fos = null;

    try {
        URLConnection urlConn = url.openConnection();//connect

        is = urlConn.getInputStream();               //get connection inputstream
        fos = new FileOutputStream(localFilename);   //open outputstream to local file

        byte[] buffer = new byte[4096];              //declare 4KB buffer
        int len;

        //while we have availble data, continue downloading and storing to local file
        while ((len = is.read(buffer)) > 0) {  
            fos.write(buffer, 0, len);
        }
    } finally {
        try {
            if (is != null) {
                is.close();
            }
        } finally {
            if (fos != null) {
                fos.close();
            }
        }
    }
}

If you want this to run in background, simply call it in a Thread:

Thread download = new Thread(){
    public void run(){
        URL url= new URL("http://overpass-api.de/api/interpreter?data=area%5Bname%3D%22Hoogstade%22%5D%3B%0A%28%0A++node%28area%29%3B%0A++%3C%3B%0A%29+%3B%0Aout+meta+qt%3B");
        String localFilename="mylocalfile"; //needs to be replaced with local file path
        downloadFromUrl(url, localFilename);
    }
};
download.start();//start the thread