Question

Hi I have some code that uses block

RandomAccessFile file = new RandomAccessFile("some file", "rw");
FileChannel channel = file.getChannel();

// some code
String line = "some data";
ByteBuffer buf = ByteBuffer.wrap(line.getBytes());
channel.write(buf);

channel.close();
file.close();

but the specific of the application is that I have to generate large number of temporary files, more then 4000 in average (used for Hive inserts to the partitioned table).

The problem is that sometimes I catch exception

Failed with exception Too many open files

during the app running.

I wounder if there any way to tell OS that file is closed already and not used anymore, why the

channel.close();
file.close();

does not reduce the number of opened files. Is there any way to do this in Java code?

I have already increased max number of opened files in

#/etc/sysctl.conf:
kern.maxfiles=204800
kern.maxfilesperproc=200000
kern.ipc.somaxconn=8096

Update:
I tried to eliminate the problem, so I parted the code to investigate each part of it (create files, upload to hive, delete files).

Using class ‘File’ or ‘RandomAccessFile’ fails with the exception “Too many open files”.

Finally I used the code:

FileOutputStream s = null;
FileChannel c = null;

try {
    s = new FileOutputStream(filePath);
    c = s.getChannel();
    // do writes
    c.write("some data"); 
    c.force(true);
    s.getFD().sync();

} catch (IOException e) {
    // handle exception
} finally {
    if (c != null)
        c.close();
    if (s != null)
        s.close();
}

And this works with large amounts of files (tested on 20K with 5KB size each). The code itself does not throw exception as previous two classes.
But production code (with hive) still had the exception. And it appears that the hive connection through the JDBC is the reason of it.
I will investigate further.

Answer 1

The amount of open file handles that can be used by the OS is not the same thing as the number of file handles that can be opened by a process. Most unix systems restrict the number of file handles per process. Most likely it something like 1024 file handles for your JVM.

a) You need to set the ulimit in the shell that launches the JVM to some higher value. (Something like ‘ulimit -n 4000’)

b) You should verify that you don’t have any resource leaks that are preventing your files from being ‘finalized’.