Question

I am trying to expose C++ container to Python.
I have:

class Container {
    std::auto_ptr<Iterator> __iter__();
};

class Iterator {
    Container & parent;
    Item __next__();
};

class Item {
    Container & parent;
};

Item class internally references data that exists in Container. Iterator which returned Item instance doesn’t have to exist for Item to be usable.

c = Container()
for i in c:
    store = i

print store

In above code I would expect to get Container, Iterator and few Item instances.
When it reaches print statement I would expect Iterator to be already destructed, but Container instance has obviously still exist for store.

Now here comes the problem. I don’t know what CallPolicy to use to achieve that effect:
Defining:

class_<Container>("Container", ...)
  .def("__iter__", &Container::__iter__, return_interal_reference<>() )
;

class_<Iterator>("Iterator", ...)
  .def("next", &Iterator::__next__, what_call_policy_here? )
;

class_<Item>("Item", ...)
  .def("__str__", ... )
;

What should I use in place of what_call_policy_here?

Answer 1

Ok, so after long digging I think I came up with solution that is transparent to exposed type.

Short description

Basically solution was to create CallPolicy that will automatically store reference to parent object (i.e. Container) inside returned child object (i.e. Iterator) as its attribute (I used a private name, but Python is quite liberal in that matter).

Then automatically copy this to all sibling objects (sibling is another child of parent, but one that was created with a call to method of another child, so not directly from parent).

Implementation

This required a bit of fiddling with CallPolicy. I had to create two custom ones:

// This policy is used for methods returning items that require object to be
// kept as long as return thing is alive.
// It stores reference in attribute named Property_::name
template <typename Property_, class BasePolicy_ = boost::python::default_call_policies>
struct store_parent_reference: public BasePolicy_
{
    template <class ArgumentPackage>
    static PyObject* postcall(ArgumentPackage const& args_, PyObject* result)
    {
        result = BasePolicy_::postcall(args_, result);

        PyObject* parent = detail::get_prev< std::size_t(1) >::execute(args_, result);
        PyObject* child = result;

        if( PyObject_SetAttrString( child, Property_::name, parent ) == -1 )
        {
            std::ostringstream err;
            err << "store_parent_reference::postcall could not set attribute `"                    << Property_::name
                << "` on newly allocated object `"
                << extract<std::string>( object( handle<>(borrowed(child))).attr("__str__")() )()
                << "`";
            throw std::runtime_error(err.str());
        }



        return result;
    }
};


// This policy is used for methods returning "sibling" in the meaning both the returned object
// and one that has this method called on require "parent" object to be alive.
//
// It copies reference to "parent" to attribute named ChildProperty_::name
// from "original" object's attribute named SiblingProperty_::name
template <typename ChildProperty_, typename SiblingProperty_ = ChildProperty_, class BasePolicy_ = boost::python::default_call_policies>
struct copy_parent_from_sibling: public BasePolicy_
{
    template <class ArgumentPackage>
    static PyObject* postcall(ArgumentPackage const& args_, PyObject* result)
    {
        result = BasePolicy_::postcall(args_, result);

        PyObject* sibling = detail::get_prev< std::size_t(1) >::execute(args_, result);
        PyObject* new_child = result;

        PyObject* parent = PyObject_GetAttrString( sibling, SiblingProperty_::name );

        if( parent == NULL )
        {
            std::ostringstream err;
            err << "copy_parent_from_sibling::postcall could not get attribute `"
                << SiblingProperty_::name
                << "` from sibling `"
                << extract<std::string>( object( handle<>(borrowed(sibling))).attr("__str__")() )()
                << "` to set up attribute `"
                << ChildProperty_::name
                << "` of returned object which is `"
                << extract<std::string>( object( handle<>(borrowed(new_child))).attr("__str__")() )()
                << "`";
            throw std::runtime_error(err.str());
        }

        if( PyObject_SetAttrString( new_child, ChildProperty_::name, parent ) == -1 )
        {
            std::ostringstream err;
            err << "copy_parent_from_sibling::postcall could not set attribute `"
                << ChildProperty_::name
                << "` on returned object which is `"
                << extract<std::string>( object( handle<>(borrowed(new_child))).attr("__str__")() )()
                << "`";
            throw std::runtime_error(err.str());
        }

        Py_DECREF(parent);

        return result;
    }
};

Usage

Now the usage:

struct ContainerProperty {
    static const char * const name;
};
const char * const ContainerProperty::name = "__container"

class_<Container>("Container", ...)
    .def("__iter__", &Container::__iter__, store_parent_reference< ContainerProperty >() )
;

class_<Iterator>("Iterator", ...)
    .def("next", &Iterator::__next__, copy_parent_from_sibling< ContainerProperty >() )
;

class_<Item>("Item", ...)
;

Note: It is hard to give complete minimal working sample for boost::python stuff, so I might have missed some detail above, but solution seems to be working fine for me (I was tracking destructor calls to check).

Also this is not the only solution. Notice that store_parent_reference is somewhat similar to return_internal_reference with the difference it explicitly needs a place to store data. This is only because copy_parent_from_sibling needs to copy it from somewhere.

Main benefit of this approach is that it does not require original classes to be aware of Python stuff.