Question

Possible Duplicate:
mmap() vs. reading blocks

I heard (read it on the internet somewhere) that mmap() is faster than sequential IO. Is this correct? If yes then why it is faster?

• mmap() is not reading sequentially.
• mmap() has to fetch from the disk itself same as read() does
• The mapped area is not sequential – so no DMA (?).

So mmap() should actually be slower than read() from a file? Which of my assumptions above are wrong?

Answer 1

I heard (read it on the internet somewhere) that mmap() is faster than sequential IO. Is this correct? If yes then why it is faster?

It can be – there are pros and cons, listed below. When you really have reason to care, always benchmark both.

Quite apart from the actual IO efficiency, there are implications for the way the application code tracks when it needs to do the I/O, and does data processing/generation, that can sometimes impact performance quite dramatically.

1. mmap() is not reading sequentially.
   2) mmap() has to fetch from the disk itself same as read() does
   3) The mapped area is not sequential – so no DMA (?).

So mmap() should actually be slower than read() from a file? Which of my assumptions above are wrong?

1. is wrong… mmap() assigns a region of virtual address space corresponding to file content… whenever a page in that address space is accessed, physical RAM is found to back the virtual addresses and the corresponding disk content is faulted into that RAM. So, the order in which reads are done from the disk matches the order of access. It’s a "lazy" I/O mechanism. If, for example, you needed to index into a huge hash table that was to be read from disk, then mmaping the file and starting to do access means the disk I/O is not done sequentially and may therefore result in longer elapsed time until the entire file is read into memory, but while that’s happening lookups are succeeding and dependent work can be undertaken, and if parts of the file are never actually needed they’re not read (allow for the granularity of disk and memory pages, and that even when using memory mapping many OSes allow you to specify some performance-enhancing / memory-efficiency tips about your planned access patterns so they can proactively read ahead or release memory more aggressively knowing you’re unlikely to return to it).

2. absolutely true

3. "The mapped area is not sequential" is vague. Memory mapped regions are "contiguous" (sequential) in virtual address space. We’ve discussed disk I/O being sequential above. Or, are you thinking of something else? Anyway, while pages are being faulted in, they may indeed be transferred using DMA.

Further, there are other reasons why memory mapping may outperform usual I/O:

• there’s less copying:
  • often OS & library level routines pass data through one or more buffers before it reaches an application-specified buffer, the application then dynamically allocates storage, then copies from the I/O buffer to that storage so the data’s usable after the file reading completes
  • memory mapping allows (but doesn’t force) in-place usage (you can just record a pointer and possibly length)
    • continuing to access data in-place risks increased cache misses and/or swapping later: the file/memory-map could be more verbose than data structures into which it could be parsed, so access patterns on data therein could have more delays to fault in more memory pages
• memory mapping can simplify the application’s parsing job by letting the application treat the entire file content as accessible, rather than worrying about when to read another buffer full
• the application defers more to the OS’s wisdom re number of pages that are in physical RAM at any single point in time, effectively sharing a direct-access disk cache with the application
• as well-wisher comments below, "using memory mapping you typically use less system calls"
• if multiple processes are accessing the same file, they should be able to share the physical backing pages

The are also reasons why mmap may be slower – do read Linus Torvald’s post here which says of mmap:

…page table games along with the fault (and even just TLB miss)
overhead is easily more than the cost of copying a page in a nice
streaming manner…

And from another of his posts:

• quite noticeable setup and teardown costs. And I mean noticeable. It’s things like following the page tables to unmap everything cleanly. It’s the book-keeping for maintaining a list of all the mappings. It’s The TLB flush needed after unmapping stuff.

• page faulting is expensive. That’s how the mapping gets populated, and it’s quite slow.

Linux does have "hugepages" (so one TLB entry per 2MB, instead of per 4kb) and even Transparent Huge Pages, where the OS attempts to use them even if the application code wasn’t written to explicitly utilise them.

FWIW, the last time this arose for me at work, memory mapped input was 80% faster than fread et al for reading binary database records into a proprietary database, on 64 bit Linux with ~170GB files.