Question

I’m working on a Multi Threaded application programmed in C++. I uses some temporary files to pass data between my threads. One thread writes the data to be processed into files in a directory. Another thread scans the directory for work files and reads the files and process them further, then delete those files. I have to use these files , because if my app gets killed when , i have to retain the data which has not been processed yet.

But i hate to be using multiple files. I just want to use a single file. One thread continuously writing to a file and other thread reading the data and deleting the data which has been read.
Like a vessel is filled from top and at bottom i can get and delete the data from vessel. How to do this efficiently in C++ , first is there a way ..?

Answer 1

As was suggested in the comments to your questions using a database like SQLite may be a very good solution.
However if you insist on using a file then this is of course possible.

I did it myself once – created a persistent queue on disk using a file.

Here are the guidelines on how to achieve this:

1. The file should contain a header which point to the next unprocessed record (entry) and to the next available place to write to.
2. If the records have variable length then each record should contain a header which states the record length.
3. You may want to add to each record a flag that indicates whether the record was processed
4. file locking can be used to ensure no one reads from the portion of the file that is being written to
5. Use low level IO – don’t use buffered streams of any kind, use direct write semantics

And here is the schemes for reading and writing (probably with some small logical bugs but you should be able to take it from there):

READER

1. Lock the file header and read it and unlock it back
2. Go to the last record position
3. Read the record header and the record
4. Write the record header back with the processed flag turned on
5. If you are not at the end of file Lock the header and write the new location of the next unprocessed record else write some marking to indicate there are no more records to process
6. Make sure that the next record to write points to the correct place

You may also want the reader to compact the file for you once in a while:

1. Lock the entire file
2. Copy all unprocessed records to the beginning of the file (You may want to keep some logic as not to overwrite your unprocessed records – maybe compact only if processed space is larger than unprocessed space)
3. Update the header
4. Unlock the file

WRITER

1. Lock the header of the file and see where the next record is to be written then unlock it
2. Lock the file from the place to be written to the length of the record
3. Write the record and unlock
4. Lock the header if the unprocessed record mark indicates there are no records to process let it point to the new record unlock the header

Hope this sets you on the write track