Question

I’d like to add as new feature to a grown application to start execution of another application using fork (2) and exec (3) (currently execl).

Unfortunately the application uses shared memory, shared semaphores and has a huge number of open file descriptors, which are all duplicated when calling fork. I know that I should close all file descriptors, shared resources etc. before calling execl in the child process, but parts of these are handled by third-party libraries and I have no means of accessing them.

To top it all the application is threaded (using posix threads), but that should be ok as long as there are no asynchronous system calls between fork and exec in the child process (according to http://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-using-them).

The problem I’m facing now is that the execl call seems to somehow corrupt the shared resources, leading to a deadlock in the thread that uses them. This seems only to happen when calling execl as exiting the child process immediatly after fork with _exit (2) does not produce this behaviour.

What is the proper way to deal with shared memory and shared semaphores when spawning a new process with fork and exec?

Answer 1

Most likely your deadlocks arise from not closing your file descriptors in the child.
Before trying anything else try this:

Iterate through all file descriptors in the child /proc/[pid]/fd and close them before calling exec.

If this won’t hep then you have to mark all memory (apart from your stack frame) as MADV_DONTFORK with madvise. You can get the list of all current maps from /proc/[pid]/maps and your local stack frame by comparing the mapping range to any pointer on your local thread stack.