Question

I have a file mmap’d read-only/shared, with multiple threads/processes reading the data concurrently. A single writer is allowed to modify the data at any time (using a mutex in a separate shared memory region). Changes are performed using a write() on the underlying file. The overall setup is part of a database that is intended to be transactionally consistent.

A number of arbitrary data pages will be written out in any order, and then fdatasync() is called. Nothing in the file points to these altered pages until a root page is written. The root page is written using a second file descriptor that was opened with O_SYNC, so the write will not return until the root page has been written successfully. All of the pages being written are part of the mmap region, so they will eventually become visible to all of the readers.

The question is – does the final O_SYNC write become visible immediately, as soon as the kernel copies the user buffer into the page cache? Or does it become visible only after the synchronous write completes? I’ve read thru the kernel code a bit but haven’t followed it all the way; it looks to me like the user data is copied immediately to the page cache, and then a write is scheduled, and then it waits for the write to complete. In the meantime, the written data is already present in the page cache and so is immediately visible to the reader processes. This is undesirable because if the physical write actually fails, the transaction must be rolled back, and readers should never be allowed to see anything that was written by an unsuccessful transaction.

Anyone know for certain how O_SYNC writes interact with the page cache? I suppose just to be safe I can wrap accesses to the root page with a mutex, but that adds a layer of overhead that would be better to avoid.

Answer 1

Under the formal POSIX standard, updates to MAP_SHARED regions can appear at any time. The Synchronised I/O definition specifies that the write will only return once the data has landed on physical media, but doesn’t talk about the data seen by other processes.

In practice on Linux, it works as you have described – the page cache is the staging area from where device writes are dispatched, and a MAP_SHARED mapping is a view of the page cache.

As an alternative, you could put a copy of the root page into a shared anonymous region. The reading processes would use that copy, and the writing process would update it after it has synched the root page to disk. You will still need synchronisation though, because you can’t atomically update an entire page.