I’m writing an application that will have to be able to handle many concurrent accesses to it, either by threads as by processes. So no mutex’es or locks should be applied to this.
To make the use of locks go down to a minimum, I’m designing for the file to be “append-only”, so all data is first appended to disk, and then the address pointing to the info it has updated, is changed to refer to the new one. So I will need to implement a small lock system only to change this one int so it refers to the new address.
How is the best way to do it?
I was thinking about maybe putting a flag before the address, that when it’s set, the readers will use a spin lock until it’s released. But I’m afraid that it isn’t at all atomic, is it?
e.g.
- a reader reads the flag, and it is unset
- on the same time, a writer writes the flag and changes the value of the int
- the reader may read an inconsistent value!
I’m looking for locking techniques but all I find is either for thread locking techniques, or to lock an entire file, not fields. Is it not possible to do this? How do append-only databases handle this?
edit:
I was looking at how append-only db’s (couchDB) do it, and it seems they use a thread only to serialize the writes to file. Does that mean it isn’t possible to make them embeddable, like sqlite, without locking the entire file with file system locks?
Thanks!
Cauê
Be careful about the append semantics of your filesystem – it probably doesn’t provide atomic append operations.
One option is to memory map (
mmap) your file as shared, then do atomic memory operations like compare-and-swap on the pointer. Your success will depend on whether your OS has such an operation (Linux, OSX do).A correct (although I’m not sure it is fast) way accomplish what you want is with
rename– it is an atomic file operation on most filesystems. Keep the most up-to-date data in an official file location. To update the data, write your new data to a temporary file, then rename that temporary file to the official location.