Question

Let’s say I’m writing a little library in C — some data structure, say. What should I do if I’m unable to allocate memory?

It might be pretty important, e.g. I need some memory to initialize the data structure in the first place, or I’m inserting a key-value pair and want to wrap it in a little struct. It could also be less critical, for instance something like a pretty_print function that builds up a nice string representation of the contents. However, it’s typically more serious than your average error — there might not be a point in continuing at all. A ton of sample uses of malloc online just straight up exit the program if it returns NULL. I’m guessing a lot of real client code does that too — just pop up some error, or write it to stderr, and abort. (And a lot of real code probably doesn’t check the return value of malloc at all.)

Sometimes it makes sense to return NULL, but not always. Error codes (or just some boolean success value), either as return values or out parameters work fine, but it seems like they can clutter up or hurt the readability of the API (then again, maybe that’s somewhat expected in a language like C?). Another option is to have some sort of internal error state the caller can subsequently query, e.g. with a get_error function, but then you have to be careful about thread safety, and it might be easy to miss; people tend to be lax about checking for errors anyway, and if it’s a separate function altogether they might not know about it, or they might not bother (but then I guess that’s their problem).

(I’ve sometimes seen malloc wrapped in a function that just tries again until memory is available…

void *my_malloc(size_t size)
{
    void *result = NULL;
    while (result == NULL)
        result = malloc(size);
    return result;
}

But that seems kind of silly and maybe dangerous.)

What’s a proper way to handle this?

Answer 1

If allocation fails in a way that prevents forward progress, the only acceptable solution for library code is to back-out whatever allocations and other changes have already been made in the partially-completed operation and return a failure code to the caller. Only the calling application can know what the right way to proceed is. Some examples:

1. A music player might just abort or return to an initial/stopped state and wait for user input again.
2. A word processor might need to store an emergency dump of the current document state to a recovery file then abort.
3. A high-end database server might need to reject and back out the whole transaction and report to the client.

If you follow the oft-advised but backwards idea that your library should just abort the caller on allocations failures, you will either have many programs that determine they cannot use your library for this reason, your users’ of the programs that use your library will be extremely angry when an allocation failure causes their valuable data to be thrown away.

Edit: One objection some of the “abort” camp will raise against my answer is that, on systems with overcommit, even calls to malloc that appeared to succeed may fail when the kernel tries to instantiate physical storage for the virtual memory allocated. This ignores the fact that anyone needing high reliability will have overcommit disabled, as well as the fact that (at least on 32-bit systems) allocation failure is more likely due to virtual address space exhaustion than physical storage exhaustion.