Question

I’m trying to read all bytes from a web site but I think I don’t get all bytes. I give a high value for bytes array length. I used this method but it always returns an exception.

Here is the code:

DataInputStream dis = new DataInputStream(s2.getInputStream());

byte[] bytes = new byte[900000];

// Read in the bytes
int offset = 0;
int numRead = 0;
while (offset < bytes.length
    && (numRead=dis.read(bytes, offset, bytes.length-offset)) >= 0) {
        offset += numRead;
}

// Ensure all the bytes have been read in
if (offset < bytes.length) {
    throw new IOException("Could not completely read website");
}
out.write(bytes);

Edited Version:

ByteArrayOutputStream bais = new ByteArrayOutputStream();
InputStream is = null;
try {
    is = s2.getInputStream();
    byte[] byteChunk = new byte[4096]; // Or whatever size you want to read in at a time.
    int n;
    while ( (n = is.read(byteChunk)) > 0 ) {
        bais.write(byteChunk, 0, n);
    }
}
catch (IOException e) {
    System.err.printf ("Failed while reading bytes");
    e.printStackTrace ();
    // Perform any other exception handling that's appropriate.
}
finally {
    if (is != null) { is.close(); }
}
byte[] asd = bais.toByteArray();
out.write(asd);

Answer 1

This is the problem:

if (offset < bytes.length)

You’ll only trigger that if the original data is more than 900,000 bytes. If the response is entirely complete in less than that, read() will return -1 correctly to indicate the end of the stream.

You should actually be throwing an exception if offset is equal to bytes.length, as that indicates that you might have truncated data 🙂

It’s not clear where you got the 900,000 value from, mind you…

I would suggest that if you want to stick with the raw stream, you use Guava‘s ByteStreams.toByteArray method to read all the data. Alternatively, you could keep looping round, reading into a smaller buffer, writing into a ByteArrayOutputStream on each iteration.