Question

I have the following method of copying files:

public static void nioCopy(File source, File destination) {
    FileInputStream fis = null;
    FileOutputStream fos = null;
    FileChannel input = null;
    FileChannel output = null;
    try {
        fis = new FileInputStream(source);
        fos = new FileOutputStream(destination);

         input = fis.getChannel();
         output = fos.getChannel();

        input.transferTo(0, input.size(), output);

    } catch (FileNotFoundException ex) {
        Logger.getLogger(Utilities.class.getName()).log(Level.SEVERE, "Can't find either of input/output files.", ex);
    } catch (IOException ex) {
        Logger.getLogger(Utilities.class.getName()).log(Level.SEVERE, "Can't open either of input/output file for reading/writing", ex);
    } finally {
        try {
            fis.close();
            fos.close();
            input.close();
            output.close();
        } catch (IOException ex) {
            Logger.getLogger(Utilities.class.getName()).log(Level.SEVERE, "Error closing streams", ex);
        }

    }
}

And I’m using to copy file but sometimes I get a silent error or undefined behavior or I just don’t know how to explain it and this is what I get:

Here is my source:

-rw-r--r-- 1 nb9 team92 3.1G 2011-10-13 16:31 6443_6#5_1_6443_6#5_2.fastq.f.fq.gz

And here is the destination:

-rw-r--r-- 1 nb9 team92 2.0G 2011-10-13 16:49 6443_6#5_1_6443_6#5_2.fastq.f.fq.gz

I get no exceptions while this process is being performed and by the looks of it everything should have been successful but then when I start unpacking the file I get:

java.io.EOFException: Unexpected end of ZLIB input stream

Clearly the destination is 1 gig OFF the original mark.

The only peculiarity is that both files are on a VERY busy lustre filesystem is it possible that this is making some funny things?

Answer 1

The fact that is truncated at 2Gb got me suspicious. I searched and it looks like an issue with nio. It could also be that the target filesystem allows max 2Gb files.

From Java NIO: Buffers

At present, buffer sizes are limited to 2GB (the maximum positive number that can be represented in an int. An updated planned for Java 7 will allow large buffers (with the size and indexes held as a long).

Anyway, just to be sure:

• Can you try to copy it on the same filesystem, if you have space?
• Can you try with Apache Commons IO FileUtils.copyFile()? Seems they fixed this issue.
• If you can upgrade, try with Java 7, since is out already