Question

I would like to store a reference to an object as a weak_ptr. In pure C++, the following works :

#include <iostream>

#include <boost/shared_ptr.hpp>
#include <boost/weak_ptr.hpp>

using namespace std;
using namespace boost;

struct Empty
{
    Empty(){}
};

struct Store
{
    weak_ptr<Empty> value;
    Store(){};

    void setValue(shared_ptr<Empty> v) {
        cout << "storing " << v << endl;
        this->value = weak_ptr<Empty>(v);
        shared_ptr<Empty> v_ok = this->value.lock();
        if (v_ok) {
            cout << "ok, v has been stored" << endl;
        }
    }

    shared_ptr<Empty> getValue() {
        shared_ptr<Empty> p = this->value.lock();
        if (p) {
            cout << "stored value : " << p << endl;
        } else {
            cout << "there's nothing here !" << endl;
        }
        return p;
    }
};

int main()
{
    shared_ptr<Empty> e(new Empty);
    shared_ptr<Store> st(new Store);

    st->setValue(e);
    st->getValue();
    return 0;
}

compiling and running this will give you this :

%> ./a.out 
storing 0x8c6c008
ok, v has been stored
stored value : 0x8c6c008

Now, if I encapsulate that with boost python :

#include <iostream>

#include <boost/shared_ptr.hpp>
#include <boost/python.hpp>
#include <boost/weak_ptr.hpp>

using namespace std;
using namespace boost;
using namespace boost::python;

struct Empty
{
    Empty(){}
};

struct Store
{
    weak_ptr<Empty> value;
    Store(){};

    void setValue(shared_ptr<Empty> v) {
        cout << "storing " << v << endl;
        this->value = weak_ptr<Empty>(v);
        shared_ptr<Empty> v_ok = this->value.lock();
        if (v_ok) {
            cout << "ok, v has been stored" << endl;
        }
    }

    shared_ptr<Empty> getValue() {
        shared_ptr<Empty> p = this->value.lock();
        if (p) {
            cout << "stored value : " << p << endl;
        } else {
            cout << "there's nothing here !" << endl;
        }
        return p;
    }
};

BOOST_PYTHON_MODULE (test)
{
    class_< Empty, shared_ptr<Empty> >("Empty");

    class_< Store, shared_ptr<Store> >("Store")
    .def("get",&Store::getValue)
    .def("set",&Store::setValue);
}

and now a small python script to try it out

from test import *

e = Empty()
st = Store()

st.set(e)
st.get()

… and the result is …

storing 0x9eb2a18
ok, v has been stored
there's nothing here !

so apparently while I’m still in the same method (setValue), there is no problem retrieving a
shared_ptr from Store::value. But as soon as I get out of this context, there’s nothing left !

How can this be ? Is python passing a brand new (and useless) shared_ptr as an argument to setValue, which is then destroyed at the end of the call ? I’m lost here.

Answer 1

This is extremely curious. I’ve ruled out the std vs. boost shared pointer possibility, and played around with a few sanity checks, and as far as I can tell it’s something that boost python is doing to the shared pointer that breaks it.

Tracing object constructors / destructors, the lifetimes of Empty and Store are being managed as you would expect (no copies occur).

An extremely interesting thing is that shared_from_this continues to work, even when weak_ptr<>.lock() doesn’t, and, in fact, a new weak pointer created from a new shared pointer (from shared_from_this) does work.

So this lead me to the thread linked in the comments, it seems like there’s something boost python is doing with the deleter and reference count that is breaking the weak pointer.

Inspecting the shared pointers in the debugger this is what we get:

When we call setValue, this is what the argument looks like:

1: p = (const 'boost::shared_ptr<Empty>' &) @0x7fff5fbfe720: {
  px = 0x100346900, 
  pn = {
    pi_ = 0x100338dd0
  }
}
> p *p.pn.pi_
$5 = (boost::detail::sp_counted_impl_pd<void*,boost::python::converter::shared_ptr_deleter>) {
  <boost::detail::sp_counted_base> = {
    _vptr$sp_counted_base = 0x10061aa30, 
    use_count_ = 2, 
    weak_count_ = 2
  }, 
  members of boost::detail::sp_counted_impl_pd<void*,boost::python::converter::shared_ptr_deleter>: 
  ptr = 0x0, 
  del = {
    owner = {
      m_p = 0x10049db90
    }
  }
}

If we create a shared pointer using shared_from_this on the argument, it looks like this:

1: p = (const 'boost::shared_ptr<Empty>' &) @0x7fff5fbfe5e0: {
  px = 0x100346900, 
  pn = {
    pi_ = 0x1003468e0
  }
}
> p *p.pn.pi_
$4 = (boost::detail::sp_counted_impl_pd<Empty*,boost::detail::sp_ms_deleter<Empty> >) {
  <boost::detail::sp_counted_base> = {
    _vptr$sp_counted_base = 0x10061b170, 
    use_count_ = 2, 
    weak_count_ = 2
  }, 
  members of boost::detail::sp_counted_impl_pd<Empty*,boost::detail::sp_ms_deleter<Empty> >: 
  ptr = 0x0, 
  del = {
    initialized_ = true, 
    storage_ = {
      data_ = "\000i4\000\001\000\000\000?h4\000\001\000\000", 
      align_ = {<No data fields>}
    }
  }
}

There’s one thing to note here: the address of the shared count is different: this is a different shared pointer instance… so we’ve somehow created two different shared pointers to the same address. This is an extremely bad thing, since we’d expect this to shortly result in a double-free.

However, it doesn’t. (I’m quite keen to understand this further, if anyone has any ideas?)

I don’t know why it doesn’t, to be honest (presumably there is something subtle going on here), but in any case, all this does point to a solution: We can use shared_from_this to create a shared pointer, from the value passed, which we can use to create a weak pointer that actually works.

So, to wrap up, here’s the fix:

#include <iostream>
#include <boost/shared_ptr.hpp>
#include <boost/python.hpp>
#include <boost/weak_ptr.hpp>
#include <boost/enable_shared_from_this.hpp>

namespace bp = boost::python;

struct Empty: boost::enable_shared_from_this<Empty>{ };

struct Store
{
    boost::weak_ptr<Empty> value;

    void setValue(boost::shared_ptr<Empty> const& v) {
        value = boost::weak_ptr<Empty>(v->shared_from_this());
        boost::shared_ptr<Empty> v_ok = value.lock();
        if (v_ok) {
            std::cout << "ok, v has been stored" << std::endl;
        }
    }

    boost::shared_ptr<Empty> getValue() {
        boost::shared_ptr<Empty> p = value.lock();
        if (p) {
            std::cout << "stored value : " << p << std::endl;
        } else {
            std::cout << "there's nothing here !" << std::endl;
        }
        return p;
    }
};

BOOST_PYTHON_MODULE (libmylibinterface)
{
    bp::class_< Empty, boost::shared_ptr<Empty> >("Empty",bp::init<>())
        ;

    bp::class_< Store, boost::shared_ptr<Store> >("Store")
        .def("get",&Store::getValue)
        .def("set",&Store::setValue);

}