I am developing an application which is required to keep a certain time t of data in memory (configurable) and another time t2 worth of data in persistent storage on disk. The reason behind this is so that frequently asked data will be stored in memory and retrieved faster, whilst older, less often user data will be stored and retrievable on disk.
The problem is: I can’t simply write to memory and then copy the entire content of the memory buffer to disk after time t as if the application crashes, the most recent data stored in memory will be lost. So each time new data is received I need to store it simultaneously in memory and on disk.
My question is, is there an efficient way to mirror the buffer in memory to a portion of the disk? I am looking for a more efficient way than writing to memory and then to disk on each data update.
You can memory map the file, both unix and windows system support this (but with different API). After that, you can simply write to that memory location, and do a sync when needed.