Question

If you use set_new_handler and your handler function is called, is errno guaranteed to be set, the way it is on a return of 0 from malloc? Or is it better to use strerror(ENOMEM)? errno works on Microsoft C++ and GCC, but that still leaves the question of whether it’s guaranteed.

Answer 1

I don’t think errno is good enough for detecting dynamic memory allocation failures. Looking at N3337, specifically 3.7.4.1 Allocation functions:

2 […] Even if the size of the space requested is zero, the request can fail.
If the request succeeds, the value returned shall be a non-null pointer value (4.10) p0 different from any
previously returned value p1, unless that value p1 was subsequently passed to an operator delete. The
effect of dereferencing a pointer returned as a request for zero size is undefined.³⁵

3 An allocation function that fails to allocate storage can invoke the
currently installed new-handler function (18.6.2.3), if any. [ Note: A
program-supplied allocation function can obtain the address of the
currently installed new_handler using the std::get_new_handler
function (18.6.2.4). —end note ] If an allocation function declared
with a non-throwing exception-specification (15.4) fails to allocate
storage, it shall return a null pointer. Any other allocation function
that fails to allocate storage shall indicate failure only by throwing
an exception of a type that would match a handler (15.3) of type
std::bad_alloc (18.6.2.1).

and footnote 35 (this is only a indicative and non-normative):

³⁵⁾ The intent is to have operator new() implementable by calling std::malloc() or std::calloc(), so the rules are substantially
the same. C++ differs from C in requiring a zero request to return a non-null pointer.

Now, heading on to the C standard draft, N1570 and a look at 7.5 Errors <errno.h>:

3 The value of errno in the initial thread is zero at program startup (the initial value of
errno in other threads is an indeterminate value), but is never set to zero by any library
function.202) The value of errno may be set to nonzero by a library function call
whether or not there is an error, provided the use of errno is not documented in the
description of the function in this International Standard.

It appears that errno may be set by a malloc failure but this is not required.

Also 7.22.3 Memory management functions from the N1570 does not specify that malloc or friends are required to set errno.

My suggestion would be to stick to what the standard guarantees and use the exception (std::bad_alloc) thrown by new (i.e. not use the no-throw new).