Question

Reading a file in using a Mapped FileChannel seems to be lightning fast… But I was wondering how they are doing this?

Are they simply just reading in a large (~64kB) buffer and then letting me march through that? Or is there more to it?

I’m just impressed by the speed and want to better understand the algorithm behind it.

Answer 1

memory mapping, maps the file into your memory and Java provides a library to to wrap this so you can access it relatively safely.

It’s benefits include:

• there only one copy in memory, in the OS disk cache and in your application(s) memory.
• you can access random areas of the file without a system call.
• Java does limit how much you can map in. i.e. if your maximum heap is 1 GB and your maximum direct memory is 1 GB, you can still map in 1 TB.

It’s disadvantages include:

• it consumes virtual memory which it doesn’t give back if you re-map or close the files. This is not such a problem if you have a 64-bit JVM, but is very limiting if you have a 32-bit JVM which might only have 1 GB free. It releases virtual memory when the GC runs.
• it reads/writes a minimum of a page at a time. This can be good if you have lots of random access but actually slows sequential access if you are reading from/writing to many files on disk. Appending to many files randomly 4KB at a time can result in highly fragments files which is not idea.
• working with memory mapped files can be more difficult that using a plain DataXxxxStream or BufferedReader/Writer.

I have written a couple of libraries making memory mapped files easier to work with and I would say I would use it when ultra-low latency is critical or you need to read large amounts of memory which you expect to be in disk cache already and you want to make the most of your disk cache.

It’s worth noting that memory mapped doesn’t make your disk sub-system faster and if that is your limiting factor it won’t matter which way you read/write data.