Question

I am using memory mapped IO for an index file, but the problem is that I’m not able to resize the file if it is mostly empty.

Somewhere before:

MappedByteBuffer map = raf.getChannel().map(MapMode.READ_WRITE, 0, 1 << 30);
raf.close();
// use map
map.force();
map = null;

Resize:

for (int c = 0; c < 100; c++) {
    RandomAccessFile raf = new RandomAccessFile(indexFile, "rw");
    try {
        raf.setLength(newLen);
        if (c > 0) LOG.warn("used " + c + " iterations to close mapped byte buffer");
        return;
    } catch (Exception e) {
        System.gc();
        Thread.sleep(10);
        System.runFinalization();
        Thread.sleep(10);
    } finally {
        raf.close();
    }
}

When using Windows or Linux 32-bit I often have the unmapping problem, but in the 64 bit Linux production environment everything seems to work without warnings, but the file keeps the original size.

Can anyone explain why this happens and/or how to solve the problem?

Answer 1

Your issue is that you are using unreliable method to close mapped byte buffer (one hundred calls to System.gc() and System.runFinalization() don’t guarantee you anything). Unfortunately there is no reliable method in Java API to do that, but on Sun JVM (and perhaps on some others too) you can use the following code:

public void unmapMmaped(ByteBuffer buffer) {
  if (buffer instanceof sun.nio.ch.DirectBuffer) {
    sun.misc.Cleaner cleaner = ((sun.nio.ch.DirectBuffer) buffer).cleaner();
    cleaner.clean();
  }
}

Of course it is JVM-dependent and you should be ready to fix your code if Sun ever decides to change sun.nio.ch.DirectBuffer or sun.misc.Cleaner in an incompatible manner (but actually I don’t believe this will ever happen).