Question

I’m dealing with the following code that is used to split a large file into a set of smaller files:

FileInputStream input = new FileInputStream(this.fileToSplit);
            BufferedInputStream iBuff = new BufferedInputStream(input);
            int i = 0;

            FileOutputStream output = new FileOutputStream(fileArr[i]);
            BufferedOutputStream oBuff = new BufferedOutputStream(output);

            int buffSize = 8192;
            byte[] buffer = new byte[buffSize];
            while (true) {
                if (iBuff.available() < buffSize) {
                    byte[] newBuff = new byte[iBuff.available()];
                    iBuff.read(newBuff);
                    oBuff.write(newBuff);
                    oBuff.flush();
                    oBuff.close();

                    break;
                }
                int r = iBuff.read(buffer);

                if (fileArr[i].length() >= this.partSize) {
                    oBuff.flush();
                    oBuff.close();
                    ++i;
                    output = new FileOutputStream(fileArr[i]);
                    oBuff = new BufferedOutputStream(output);
                }
                oBuff.write(buffer);
            }

        } catch (Exception e) {
            e.printStackTrace();
        }

This is the weird behavior I’m seeing… when I run this code using a 3GB file, the initial iBuff.available() call returns a value of a approximatley 2,100,000,000 and the code works fine. When I run this code on a 12GB file, the initial iBuff.available() call only returns a value of 200,000,000 (which is smaller than the split file size of 500,000,000 and causes the processing to go awry).

I’m thinking this discrepancy in behvaior has something to do with the fact that this is on 32-bit windows. I’m going to run a couple more tests on a 4.5 GB file and a 3.5 GB file. If the 3.5 file works and the 4.5 one doesn’t, that will further confirm the theory that it’s a 32bit vs 64bit issue since 4GB would then be the threshold.

Answer 1

Well if you read the javadoc it quite clearly states:

Returns the number of bytes that can
be read from this input stream
without blocking (emphasis added by me)

So it’s quite clear that what you want is not what this method offers. So depending on the underlying InputStream you may get problems much earlier (eg a stream over the network with a server that doesn’t return the filesize – you’d have to read the complete file and buffer it just to return the “correct” available() count, which would take a lot of time – what if you only want to read a header?)

So the correct way to handle this is to change your parsing method to be able to handle the file in pieces. Personally I don’t see much reason at all to even use available() here – just calling read() and stopping as soon as read() returns -1 should work fine. Can be made more complicated if you want to assure that every file really contains blockSize byte – just add an internal loop if that scenario is important.

int blockSize = XXX;
byte[] buffer = new byte[blockSize];
int i = 0;
int read = in.read(buffer);
while(read != -1) {
   out[i++].write(buffer, 0, read);
   read = in.read(buffer);
}