Question

Background

I have a multi-component c++ codebase. There is one central component which comprises the primary executable, and there are a number of components which compile to dynamic modules (.so files). The central executable is capable of loading and unloading them at runtime (hotswapping them, if you will).

There is one file, called Scheduler.h, which declares a Scheduler class, which provides synchronous events at specific times or intervals, and a few helper classes which are used to make requests to the scheduler. There is an Event class, which holds timing data, and an abstract action class, with a single pure virtual function, DoEvent. There is also a Scheduler.cpp, which contains definitions for most of the functionality in Scheduler.h (except for the template classes, which are declared and defined in the header file).

An Event owns a pointer to a subclass of action, which is how the functionality of the scheduler is controlled. Scheduler.h provides a few of these subclasses itself.

action is declared like this:

class action
{
    action();
    virtual ~action();
    virtual DoEvent() = 0;
};

FunctionCallAction, a subclass of action is declared and defined like this:

template <class R, class T>
class FunctionCallAction : public action
{
public:
    FunctionCallAction(R (*f)(T), T arg) : argument(arg), callback(f) {}
    ~FunctionCallAction() {}
    void DoEvent() { function(argument); }
private:
    R (*callback)(T);
    T argument;
};

HelloAction, another subclass, is declared like this:

// In Scheduler.h
class HelloAction : public action
{
    ~HelloAction();
    void DoEvent();
};

// in Scheduler.cpp
HelloAction::~HelloAction() {}
void HelloAction::DoEvent() { cout << "Hello world" << endl; }

One of my dynamic libraries, CloneWatch, declared in CloneWatch.h and defined in CloneWatch.cpp, uses this scheduler service. In its constructor, it creates a persistent event, scheduled to run every 300 seconds. In its destructor, it removes this event. When this module is loaded, it obtains a reference to the existing scheduler object. The process of “loading” a module entails using dlopen() to open the library, dlsym() to search for a factory method (aptly named Factory), and to use this factory method to create an instance of some object (the semantics are not relevant). To close the library, the object created by the factory method is deleted, and dlclose() is called to remove the library from the process’ address space.

Loading and unloading libraries at runtime is controlled by a command.

// relevant declarations
const float DB_CLEAN_FREQ = 300;
event_t cleanerevent; // event_t is a typedef to an integral type
void * RunDBCleaner(void *); // static function of CloneWatch
Scheduler& scheduler;

// in constructor:
Event e(DB_CLEAN_FREQ, -1, new FunctionCallAction<void *, void *>(CloneWatch::RunDBCleaner, (void *) this));
cleanerevent = scheduler.AddEvent(e);

// in destructor:
scheduler.RemoveEvent(cleanerevent);

Scheduler::RemoveEvent is lazy. Rather than traversing the entire priority queue of Events, it maintains a set of “cancelled events”. If, during the course of its event processing, it pops from its queue an event with an ID that matches an ID in its set of cancelled events, the event is not run or rescheduled and is immediately cleaned up. The processes of cleaning up an event entails deleting the action object that it owns.

Problem

The problem I’m having is that the program segment faults. The fault occurs inside the Scheduler’s event loop, which looks roughly like this:

while (!eventqueue.empty() && e.Due())
{
    Event e = eventqueue.top();
    eventqueue.pop();
    if (cancelled.find(e.GetID()) != cancelled.end())
    {
        cancelled.erase(e.GetID());
        e.Cancel();
        continue;
    }

    QueueUnlock();
    e.DoEvent();
    QueueLock();

    e.Next();

    if (e.ShouldReschedule()) eventqueue.push(e);
}

The call to e.Cancel deletes the event’s action. The call to e.Next may delete the event’s action (only if the event has expired on its own. In this case, e.ShouldReschedule will return false and the event will be dropped). For testing purposes, I added some print statements to the destructors of the action class and subclasses to see what was going on.

The Kicker

If the event is deleted from e.Next, through expiring, everything proceeds as normal. However, when I unload the module, causing the event to be retired through the cancelled list, the program experiences a segmentation fault as soon as the action’s destructor is called. This happens some time after the module is unloaded, because of the scheduler’s lazy deletion of events

It does not enter any of the destructors, but immediately faults. I’ve tried various combinations of managed and unmanaged deletion of the event’s action, as well as doing it in different places and different ways. I’ve run it through valgrind and gdb, but they both just politely inform me that a segmentation fault occurred, and for the life of me I can’t isolate why (Though I don’t know how to use either one very well).

If I call e.Cancel in the main body of the loop as well, forcing a deletion, and comment out the line that reschedules the event, thus forcing the event to be cancelled as soon as it executes, the fault does not occur.

I’ve also replaced the action with a HelloAction, but this one does not fault. Something very specific about the destructor of FunctionCallAction is where the issue apparently lies. I’ve more or less eliminated semantical error, and I suspect that it’s the result of some obscure behavior of the compiler or dynamic linker. Does anyone see the problem?

Answer 1

This is a behavior of the compiler.

The problem is that FunctionCallAction is defined (not just declared) in its header file. This is a necessary side effect of being a template class, however declaring a regular class with the functionality of a FunctionCallAction<void *, void *> yields the same results IF the class is defined in the header file.

It was a mundane restriction on template classes having unintended side effects in an unusual circumstance.

The reason is that if the definition of a class is in the header file, it gets compiled into each file that uses it. Since I’m using it from the code of my dynamic library, that’s where it’s being compiled. Thus, when the library is unloaded, the code for the destructors, and the entire rest of the class, no longer exists.

I solved this problem by making a FunctionCallAction non-template class and leaving only its declaration in Scheduler.h, and moving its definition to Scheduler.cpp. This way, the functions are provided by the always-loaded core executable instead of individually by the dynamic modules.

The call to the action’s destructor was segment faulting because the destructor itself was no longer part of the address space of the process.