Question

I’m writing a program, some kind of database. While I was reading manual of fclose(3) I found that it calls fflush(3) to flush FILE* buffers to disk (actually to OS buffer, but it doesn’t matter right now, we can always call fsync(2)).

Because I’m writing a DB it is obvious that I want to prevent data loss. If there is no disk space and fflush(3) in fclose(3) fails — we will lose our data, because

using FILE* after an error in fclose() will cause undefined behavior

So I thought about explicit use of fflush(3) before fclose(3), warn user about low disk space and recall fflush(3) after a while.

I’ve read the C standard and thought this was a good idea. In practice, after failed fflush the second call would always return 0 (no error), but would actually do nothing. fsync didn’t help me (I thought data might be saved in RAM).

How can I prevent data loss in such a situation? Maybe there are some rules of thumb.

Here is my test code:

#include <stdio.h>
int main()
{
    FILE *a = fopen("/tmp/1", "wb")
    if ( !a )
        perror("fopen");

    if ( fwrite("test", 1, 4, a) != 4 )
        perror("fwrite");  // always OK, cause data is buffered


    while( fflush(a) )  // ...second call will always return 0!
    {
        perror("fflush");  // if there is no disk space, I will get this perror, but ...
    }


    if ( fclose(a) )  // always ok, because calls only close(2)
        perror("fclose"); 

    return 0;
}

Answer 1

The reason the subsequent fflush() operations succeed is that there is no (new) data to write to disk. The first fflush() failed; that is tragic but history. The subsequent fflush() has nothing to do, so it does so successfully.

If you are writing to a database, you have to be careful about each write – not just dealing with problems at the end. Depending on how critical your data is, you may need to go through all sorts of gyrations to deal with problems – there are reasons why DBMS are complex, and failed writes are one of them.

One way of dealing with the problem is to pre-allocate the space for the data. As others have noted, classic Unix file systems allow for sparse files (files where there are empty blocks with no disk space allocated for them), so you actually have to write some data onto each page that you need allocated. Then you only have to worry about ‘disk full’ problems when you extend the space – and you know when you do that and you can deal with that failure carefully.

On Unix-based systems, there are a variety of system calls that can help you synchronize your data on disk, and options to ‘open’ etc. These include the ‘O_DSYNC’ and related values. However, if you are extending a file, they can still cause failures for ‘out of space’, even with the fancy synchronizing options. And when you do run into that failure, you have to wait for space to become available (because you asked the user to tell you when it is available, perhaps), and then try the write again.