Question

While debugging an application using the Apache Zookeeper C runtime library, I run into issues when setting breakpoints with the default all-stop mode in GDB. With the Zookeeper thread being unable to run, the server will timeout the session and thus delete any ephemeral znodes you may have created. Using non-stop mode I can prevent this from occurring, but I lose the convenience of being able to examine the state of any non-Zookeeper thread.

Is there a way in GDB to specify that one (or more) threads will continue running in an application when a breakpoint is hit, but the others will stop running? That way I can examine the state of threads I care about and ignore the state of those I want running in the background.

Edit: This is essentially a duplicate of not stopping all threads in gdb. The solution there to use background commands with non-stop mode essentially solves my problem since I can stop threads and restart them asynchronously whenever I want to, so maybe we should close this one.

Answer 1

With Zookeeper it turns out there is a hack you can perform in order to allow a session to continue while interrupted in gdb. This hack leverages a few properties of Zookeeper and gdb:

• You can have multiple Zookeeper clients with same session ID
• gdb doesn’t stop child processes at parent breakpoints
• You can ignore gdb signals in a child process without affecting the parent

Based on this, the solution becomes spawning a child process which connects to Zookeeper with the same client id as the parent and does nothing else. Zookeeper clients however have the notion of session moving, where every so often the client will switch which server they are connected to. If you have two clients with same session ID, one of them could move leaving the other connected to a server that doesn’t hold their session. To prevent this, the child must only connect to the server the parent is currently connected to. The parent and child thus look like the following:

Parent(zh):
  host = zookeeper_get_connected_host(zh)
  client_id = zoo_client_id(zh)
  if fork == 0
    exec child host client_id
  end

Child(host, client_id):
  ignore SIGINT
  zh = zookeeper_init(host, client_id)
  while(true)
    sleep
  end

I tested this using libzookeeper_mt-0.3.3 and it works as described. Some nastiness starts spewing out of the Zookeeper logs when you do this hack which can be frustrating. If you can’t ignore the logs, you can turn them off as follows:

zoo_set_debug_level((ZooLogLevel)0);

It’s an undocumented way in Zookeeper of disabling logging.