Question

I have a C++ class that writes its data out to a binary std::ofstream. The class is storing the data as a boost:shared_array but I have eliminated this as the problem. The problem is with the call to write() on the ofstream.

The issue is that it seems to leak memory. The system it is running on is CentOS 64bit, GCC 4.1.2.

When looking at top and free when the application is running the executable itself does not continue to consume memory (as backed up by the memory profiler in Netbeans), but the amount of free system memory does decrease over time. What’s more, when the application exits this memory is not reclaimed!

This is a particular issue because the intention is to write out to disk continuously at around 50MB/s for hours on end. However, once we get down to around 90MB of free system memory it seems to “stabilize” and not reduce any further and the application continues to run OK. It does however screw up the system for the other running processes, which is bad, mmkay.

Below is an ever-so slightly simplified version of the class that is causing the grief.

class WritableMessage
{
public:
    WritableMessage();
    void write(std::ofstream* const idxStream, std::ofstream* const dataStream);

private:
    IdxRecord m_idx;
    boost::shared_array<char> m_data;
};

The ofstreams are initialised and destructed elsewhere but essentially they remain open for writing “forever”.

void WritableMessage::write(std::ofstream* const idxStream, std::ofstream* const dataStream)
{
    //Call the IdxRecord to write itself (just a call to idxStream->write())
    m_idx.write(idxStream, dataStream->tellp());

    //This is the main issue, because this data can be up to about 2MB in size
    //for each write.  Commenting out this line removes all issues with the memory leak
    dataStream->write(m_data.get(), m_idx.getMessageSize());

    //I'd expect the flush to clear any buffers that the ofstream maintains,
    //but apparently not
    idxStream->flush();
    dataStream->flush();
}

Answer 1

It looks like you don’t have a problem, and that is just the system cache working as intended.
Linux is very agressive on caching, so it will use almost all your free memory for this, but whenever an application needs more memory, it will release some of that memory and grant it to the application.