Question

I’ve developed a HTTP communication object for the purpose pof downloading files via a GET request.

This works just fine when downloading a text file. However, downloading a compressed file such as zip, gz or tar.gz appears to download the file but the file is not valid.

In the case zip I get a meesage saying it tried to move the pointer before the beginnning of the file.
In the case of .tar.gz the message is Data error in file.tar. File is broken.

In all cases the download links I use do allow a complete and correct download from the URL. Yet, the Java code based download brings the file down but it is not valid.

The code is as follows:

public class HTTPCommunicationGet {

    private URIBuilder sendData;
    private URI target;
    private HttpGet getConnection;

    public HTTPCommunicationGet(String url, TreeMap<String, String> components) {
        super(url, components);
    }

    public HTTPCommunicationGet(String url, String queryString) {
        super(url, queryString);
    }

    protected void defineSendData() throws URISyntaxException, IOException {
        this.sendData = new URIBuilder(new URI(this.getUrl()));
        if (this.getComponents() != null && this.getComponents().size() > 0) {
            for (Map.Entry<String, String> component : this.getComponents().entrySet()) {
                this.sendData.setParameter(component.getKey(), component.getValue());
            }
        }
    }

    protected void retrieveRemoteData() throws IOException, MalformedURLException, URISyntaxException, DataMapHTTPGetException {

        this.target = this.sendData.build();
        this.getConnection = new HttpGet(target);
        HttpResponse response = client.execute(this.getConnection);
        if (response.getStatusLine().toString().toUpperCase().contains("200 OK")) {
            this.setResponse(response.getStatusLine().toString(), "Data Retrieved");
            BufferedReader rd = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));
            String line = "";
            while ((line = rd.readLine()) != null) {
                this.remoteData.append(line);
            }
        } else {
            String message = String.format("%s: Provider connection exception; response returned was not 200 OK", this.target.toASCIIString());
            this.setResponse(response.getStatusLine().toString(), message);
            DataMapHTTPGetException ex = new DataMapHTTPGetException(target.toString(), message);
            throw ex;
        }
    }

    public void downloadFiles(String localFile) throws DataMapConnectionException, FileNotFoundException, IOException, URISyntaxException {
        // check that we have remoteData set
        this.defineSendData();
        this.retrieveRemoteData(); // everything is bubbled up to the controller class that is calling this.

        File localMetaFile = new File(localFile);
        switch (this.archiveMetaFile(localMetaFile)) {
            case -1:
                IOException ex = new IOException(String.format("The file %s could not be moved", localFile));
                throw ex;
            //break;
            case 0:
                infoLog.info(String.format("%s: this file did not already exist", localFile));
                break;
            case 1:
                infoLog.info(String.format("%s: this file was found and successfully archived to the processed directory", localFile));
                break;
        }

        BufferedWriter fileWriter = new BufferedWriter(new FileWriter(localFile));
        fileWriter.write(this.remoteData.toString());
        fileWriter.close();
    }
}

As you can see this is called via downloadFiles after the object has been initialised. I’ve cut out the code that is not needed for this example such as the archiveMetaFile method.

Any pointers on why this is not working for compressed files is much appreciated.

Cheers
Nathan

Answer 1

The problem is likely that you are using a BufferedReader instead of an InputStream. Readers are used for text data and impose a character encoding whereas InputStreams can handle raw binary data.

Try switching to a BufferedInputStream instead. The use of any Reader class will corrupt binary data.