Question

I’m experiencing a strange error with one of our systems that I am at a loss to explain. In our system the backend is generating a large TSV output file which we are then serving across HTTP using the following code:

    BufferedInputStream input = new BufferedInputStream(p.getInputStream(), (int)FileUtils.BYTES_PER_MEGABYTE * 16);
    OutputStream output = resp.getOutputStream();
    byte[] buffer = new byte[(int) (FileUtils.BYTES_PER_KILOBYTE * 8)];
    do
    {
        int read = input.read(buffer);
        if (read <= 0) break;
        output.write(buffer);           
    } while (true);
    input.close();
    output.close();

Then on the client side there is a TSV parser consuming the HTTP response but on very large inputs we start seeing strange artifacts where the parser will report a line as having the wrong number of items and the error message prints the line it will be parsing and that line will be a random chunk of data i.e. not an entire line of the data

My first thought was that the TSV being generated was malformed but I’ve pretty much ruled this out by copying the file directly from the backend system and then running it through three independently written open source TSV parsers (including the one the client code is using) and all of them are capable of parsing the file fine when running on the local file.

For reference the code for the TSV parser we’re using is here

This leads me to two possibilities:

1. The code I’ve shown for copying the file across HTTP is flawed in some way – in which case I’d love for someone to point out what dumb but non-obvious mistake I’ve made!
2. BufferedReader.readLine() which is being used by the consuming parser is not guaranteed to read whole lines? I’d wouldn’t be entirely surprised if this is the case as I’ve been bitten by strange read behavior over slow network streams in .Net so wonder if a similar problem can apply in Java?

Or is there some other explanation I’ve overlooked?

Answer 1

In posting this question I suddenly spotted what the error was (typically!)

The following portion of the code I posted for copying the file is incorrect:

int read = input.read(buffer);
if (read <= 0) break;
output.write(buffer);

It should instead be as follows:

int read = input.read(buffer);
if (read <= 0) break;
output.write(buffer, 0, read);

The problem being that I was always writing the whole buffer to the output stream even when we’d read less from the input than the size of the buffer. This meant that at the end of the file we’d print the last chunk of the data and whatever was left in the rest of the buffer hence the random chunk of data left over!