Question

Recently I ran into a “fun” problem with the Microsoft implementation of the CRTL. tmpfile places temp files in the root directory and completely ignores the temp file directory. This has issues with users who do not have privileges to the root directory (say, on our cluster). Moreover, using _tempnam would require the application to remember to delete the temporary files, which it is unable to do without a considerable amount of rework.

Therefore I bit the bullet and wrote Win32 versions of all of the IO routines (create_temp, read, write, seek, flush) which call the appropriate method. One thing I’ve noticed is the now abysmal performance of the library.

Results from the test suite:

CRTL:    4:30.05 elapsed
Win32:  11:18.06 elapsed

Stats measured in my routines:
Writes:  3129934 (   44,642,745,008 bytes)
Reads:    935903 (    8,183,423,744 bytes)
Seeks:   2205757 (2,043,782,657,968 bytes traveled)
Flushes:   92442

Example of a CRTL v. Win32 method:

int io_write(FILE_POINTER fp, size_t words, const void *buffer)
{
#if !defined(USE_WIN32_IO)
    {
        size_t words_written = 0;

        /* read the data */
        words_written = fwrite(buffer, sizeof(uint32_t), words, fp);
        if (words_written != words)
        {
            return errno;
        }
    }
#else /* !defined(USE_WIN32_IO) */
    {
        DWORD bytesWritten;

        if (!WriteFile(fp, buffer, words * sizeof(uint32_t), &bytesWritten, NULL)
            || (bytesWritten != words * sizeof(uint32_t)))
        {
            return GetLastError();
        }
    }
#endif /* USE_WIN32_IO */

    return E_SUCCESS;
}

As you can see, they are effectively identical, yet the performance (in release mode) is wildly divergent. Time spent in WriteFile and SetFilePointer dwarf the time spent in fwrite and fseeko, which seems counterintuitive.

Ideas?

UPDATE: perfmon notes that fflush is about 10x cheaper than FlushFileBuffers and fwrite is ~1.1x slower than WriteFile. The net result is a huge performance loss with FlushFileBuffers used in the same manner as fflush. There is no change from FILE_ATTRIBUTE_NORMAL to FILE_FLAG_RANDOM_ACCESS either.

Answer 1

I think it’s probably due to this issue, described on MSDN’s page for FlushFileBuffers:

Due to disk caching interactions
within the system, the
FlushFileBuffers function can be
inefficient when used after every
write to a disk drive device when many
writes are being performed separately.
If an application is performing
multiple writes to disk and also needs
to ensure critical data is written to
persistent media, the application
should use unbuffered I/O instead of
frequently calling FlushFileBuffers.
To open a file for unbuffered I/O,
call the CreateFile function with the
FILE_FLAG_NO_BUFFERING and
FILE_FLAG_WRITE_THROUGH flags. This
prevents the file contents from being
cached and flushes the metadata to
disk with each write. For more
information, see CreateFile.

In general, FlushFileBuffers is an “expensive” operation, since it flushes everything in the write-back cache:

FlushFileBuffers(): This function will flush everything in the write-back cache, as it
does not know what part of the cache belongs to your file. This can take a lot of time,
depending on the cache size and the speed of the media. How necessary is it? There is
a thread which goes through and writes out dirty pages, so it is likely not very
necessary.

I presume that fflush does not flush the entire write-back cache. In that case, it’s much more efficient, but that efficiency comes at the risk of potential data loss. The CRT’s source code for fflush confirms this, since _commit calls FlushFileBuffers:

/* lowio commit to ensure data is written to disk */
if (str->_flag & _IOCOMMIT) {
        return (_commit(_fileno(str)) ? EOF : 0);
}

From the implementation of _commit:

if ( !FlushFileBuffers((HANDLE)_get_osfhandle(filedes)) ) {
        retval = GetLastError();
}