Question

According to the Nginx documentation:

If you need to replace nginx binary
with a new one (when upgrading to a
new version or adding/removing server
modules), you can do it without any
service downtime – no incoming
requests will be lost.

My coworker and I were trying to figure out: how does that work?. We know (we think) that:

• Only one process can be listening on port 80 at a time
• Nginx creates a socket and connects it to port 80
• A parent process and any of its children can all bind to the same socket, which is how Nginx can have multiple worker children responding to requests

We also did some experiments with Nginx, like this:

• Send a kill -USR2 to the current master process
• Repeatedly run ps -ef | grep unicorn to see any unicorn processes, with their own pids and their parent pids
• Observe that the new master process is, at first, a child of the old master process, but when the old master process is gone, the new master process has a ppid of 1.

So apparently the new master process can listen to the same socket as the old one while they’re both running, because at that time, the new master is a child of the old master. But somehow the new master process can then become… um… nobody’s child?

I assume this is standard Unix stuff, but my understanding of processes and ports and sockets is pretty darn fuzzy. Can anybody explain this in better detail? Are any of our assumptions wrong? And is there a book I can read to really grok this stuff?

Answer 1

For specifics: http://www.csc.villanova.edu/~mdamian/Sockets/TcpSockets.htm describes the C library for TCP sockets.

I think the key is that after a process forks while holding a socket file descriptor, the parent and child are both able to call accept() on it.

So here’s the flow. Nginx, started normally:

1. Calls socket() and bind() and listen() to set up a socket, referenced by a file descriptor (integer).
2. Starts a thread that calls accept() on the file descriptor in a loop to handle incoming connections.

Then Nginx forks. The parent keeps running as usual, but the child immediately execs the new binary. exec() wipes out the old program, memory, and running threads, but inherits open file descriptors: see http://linux.die.net/man/2/execve. I suspect the exec() call passes the number of the open file descriptor as a command line parameter.

The child, started as part of an upgrade:

1. Reads the open file descriptor’s number from the command line.
2. Starts a thread that calls accept() on the file descriptor in a loop to handle incoming connections.
3. Tells the parent to drain (stop accept()ing, and finish existing connections), and to die.