Question

There is one very useful gdb “script” called poor man’s profiler. It calls this command:

gdb -ex "set pagination 0" -ex "thread apply all bt" --batch -p $pid

It works well for most linux processes, but don’t work for nginx web server.

Normal output:

# gdb -ex "set pagination 0" -ex "thread apply all bt" -batch -p 5286
Using host libthread_db library "/lib/libthread_db.so.1".
[Thread debugging using libthread_db enabled]
[New Thread 0xb7d996c0 (LWP 5286)]
[New Thread 0xb588ab90 (LWP 5292)]
[New Thread 0xb608bb90 (LWP 5291)]
[New Thread 0xb688cb90 (LWP 5290)]
[New Thread 0xb708db90 (LWP 5289)]
[New Thread 0xb788eb90 (LWP 5288)]
0xffffe410 in __kernel_vsyscall ()

Thread 6 (Thread 0xb788eb90 (LWP 5288)):
#0  0xffffe410 in __kernel_vsyscall ()
#1  0xb7e5d7a6 in epoll_wait () from /lib/libc.so.6
#2  0xb7ef4f3b in epoll_dispatch () from /usr/lib/libevent-1.3b.so.1
#3  0xb7ee963a in event_base_loop () from /usr/lib/libevent-1.3b.so.1
#4  0x08055537 in worker_libevent (arg=0x805f3a0) at thread.c:245
#5  0xb7ed2192 in start_thread () from /lib/libpthread.so.0
#6  0xb7e5d02e in clone () from /lib/libc.so.6

[cut]

Thread 1 (Thread 0xb7d996c0 (LWP 5286)):
#0  0xffffe410 in __kernel_vsyscall ()
#1  0xb7e5d7a6 in epoll_wait () from /lib/libc.so.6
#2  0xb7ef4f3b in epoll_dispatch () from /usr/lib/libevent-1.3b.so.1
#3  0xb7ee963a in event_base_loop () from /usr/lib/libevent-1.3b.so.1
#4  0x0804f439 in main (argc=1, argv=0xbfbaff14) at memcached.c:4681
#0  0xffffe410 in __kernel_vsyscall ()

Nginx output:

# gdb -ex "set pagination 0" -ex "thread apply all bt" -batch -p 6120
Using host libthread_db library "/lib/libthread_db.so.1".
0xffffe410 in __kernel_vsyscall ()

But it works well if “bt” instead of “thread apply all bt” is used:

# gdb -ex "set pagination 0" -ex "bt" -batch -p 6120
Using host libthread_db library "/lib/libthread_db.so.1".
0xffffe410 in __kernel_vsyscall ()
#0  0xffffe410 in __kernel_vsyscall ()
#1  0xb7c83778 in epoll_wait () from /lib/libc.so.6
#2  0x080664c0 in ngx_epoll_process_events (cycle=0x80d14b8, timer=500, flags=1) at src/event/modules/ngx_epoll_module.c:530
#3  0x0805f73b in ngx_process_events_and_timers (cycle=0x80d14b8) at src/event/ngx_event.c:245
#4  0x080652b3 in ngx_worker_process_cycle (cycle=0x80d14b8, data=0x0) at src/os/unix/ngx_process_cycle.c:795
#5  0x08063ba1 in ngx_spawn_process (cycle=0x80d14b8, proc=0x80651fb <ngx_worker_process_cycle>, data=0x0, name=0x80a881d "worker process", respawn=-3) at src/os/unix/ngx_process.c:196
#6  0x080648d2 in ngx_start_worker_processes (cycle=0x80d14b8, n=8, type=-3) at src/os/unix/ngx_process_cycle.c:355
#7  0x0806581f in ngx_master_process_cycle (cycle=0xfffffffc) at src/os/unix/ngx_process_cycle.c:136
#8  0x0804d076 in main (argc=1, argv=0xbfd86544) at src/core/nginx.c:396

Why is that? As far as I know “thread apply all bt” must work even if there is only one execution thread in process.

Update:

Manual gdb connection and issuing “info threads” command.

# gdb -p 17461
GNU gdb 6.4
Copyright 2005 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i586-suse-linux".
Attaching to process 17461
Reading symbols from /export/depo/apache/linux/nginx-0.8.34/sbin/nginx...done.
Using host libthread_db library "/lib/libthread_db.so.1".
Reading symbols from /lib/libcrypt.so.1...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /export/depo/mysql/linux/mysql-proxy-0.7.2/lib/libpcre.so.0...done.
Loaded symbols for /opt/gnu/mysql-proxy/lib/libpcre.so.0
Reading symbols from /usr/lib/libssl.so.0.9.8...done.
Loaded symbols for /usr/lib/libssl.so.0.9.8
Reading symbols from /usr/lib/libcrypto.so.0.9.8...done.
Loaded symbols for /usr/lib/libcrypto.so.0.9.8
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libz.so.1...done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
0xffffe410 in __kernel_vsyscall ()
(gdb) info threads
(gdb) thread 0
Thread ID 0 not known.
(gdb) bt
#0  0xffffe410 in __kernel_vsyscall ()
#1  0xb7d0dd98 in __epoll_wait_nocancel () from /lib/libc.so.6
#2  0x08066e7d in ngx_epoll_process_events (cycle=0x816b708, timer=500, flags=1) at src/event/modules/ngx_epoll_module.c:530
#3  0x0805f4a2 in ngx_process_events_and_timers (cycle=0x816b708) at src/event/ngx_event.c:245
#4  0x08064d83 in ngx_worker_process_cycle (cycle=0x816b708, data=0x0) at src/os/unix/ngx_process_cycle.c:795
#5  0x08063661 in ngx_spawn_process (cycle=0x816b708, proc=0x8064ccb <ngx_worker_process_cycle>, data=0x0, name=0x80a42ed "worker process", respawn=-4) at src/os/unix/ngx_process.c:196
#6  0x080643a1 in ngx_start_worker_processes (cycle=0x816b708, n=8, type=-4) at src/os/unix/ngx_process_cycle.c:355
#7  0x0806595c in ngx_master_process_cycle (cycle=0x816b708) at src/os/unix/ngx_process_cycle.c:249
#8  0x0804cf7f in main (argc=1, argv=0xbfdb1874) at src/core/nginx.c:396

Answer 1

The thread apply all bt does (surprise!) bt for each thread. It also stops as soon as there is any thread where bt results in an error (but that should not happen).

So if you attach to nginx by hand, then do info threads, and then keep repeating thread N and bt, you’ll likely find a thread for which GDB can’t produce a backtrace.

It might be interesting to know exactly what GDB prints for that thread.

You might also want to retry with the current version of GDB (7.2)