Question

I’m wondering if at all it is possible to sandbox a dynamically linked library via dlopen and friends. The aim is to recover from an error within the library without tearing down the whole application e.g SEGFAULT, etc.

Anyone had any experience in this area?

Answer 1

OK well generally speaking exception handling is highly operating system dependent. I am going to make some assumptions and try to provide some generic guidance. Please know that this is by no means an exhaustive reply, but should serve as a place to start.

I will assume that:

1. For the most part, you are interested in safeguarding against memory leaks.

2. You are not interested in Windows (which is whole-other-ball-of-wax) since you mentioned dlopen (you would have said LoadLibrary otherwise)

3. That you are aware of the nuances of linking against C++ symbols. If you are not read up on it at mini howto on dlopen c++

Generally speaking

There is no general solution to the described problem without involving specialized operating systems that provide data and code segment sand-boxing there are Trusted Systems and specialty operating system kernels that can do this, but i assume that you want to do this on a good old *nix or windows environment.

Compiler stuff further complicates issues (does your C++ compiler generate weak symbols by default? typically it would) This affects how exception handling happens in a try-catch.

Simple operating system exception handling that raises signals (SIGSEGV, SIGFPE etc.):

Under POSIX system supporting sigaction…

Let’s say you want to protect against generic things like bad memory addressing. Trap the SIGSEG using sigaction before dlopening a library (to protect against .init functions) and then also do a signal check before calling a function within the library. Consider using SA_STACK to ensure that your handler jumps into a stack you have good control over, and SA_SIGINFO to ensure that your handler gets info about the source.

A good place to start on this is at the Signal handling on GNU libc manual

Under C++: use wrappers and with try-catch to catch soft exceptions

try {
foo();
}
catch() {
// do something
}

where foo is a weak symbol pointing to a function in your dll see c++ dlopen mini-howto for a lot more examples and details on loading classes etc.

If you have more specific needs, post them, i’ll see if i can provide more info.

Cheers