Question

I have a long-running multithreaded program, and I’d like to occasionally like to call a function with Profile.runcall and dump the data to a file.

The hotshot documentation states: Note: The hotshot profiler does not yet work well with threads. It is useful to use an unthreaded script to run the profiler over the code you’re interested in measuring if at all possible.

However, I’ve test having multiple threads profile a function simultaneously, and I’ve encountered no problems. In my case, the functions being profiled are all self-contained and do not spawn any additional threads during execution. And I make sure that every Profile object has its own file so that they don’t clobber each other.

So my guess is that the warning about multiple threads means that I would have problems if the function being profiled spawned additional threads itself. When I profile a test function that does spawn other threads, the spawned threads are not profiled. However, this still doesn’t seem to cause any problems, since the function being profiled seems to work just fine.

Can someone confirm or refute that hotshot is safe to run in a multithreaded program? I don’t want to deploy something that might crash later, even if it seems to pass my initial tests.

Answer 1

For what it’s worth, we have a setting in CubicWeb (which uses multiple threads) to enable hotshot profiling, and so far I’ve never experienced issues when enabling profiling.