[C++-sig] Retaining a useful weak_ptr to a wrapped C++ interface implemented in Python

[Holger Brandsmeier]

Adam,

you probably run into a similar problem that I ran into (not with
boost::shared_ptr but with our custom implementation which strongly
mimics boost::shared_ptr). I wanted to have a class in C++ that is
extended in python, and it should store a weak_ptr to itself.
Unfortunately I didn't solve this yet, I stored a shared_ptr which
means that the object gets never deleted. I have to revisit this at a
later point, but I'll try to explain you what's going on.

You probably have a function like this

void addCallback(share_ptr<> arg) {
callbacks.push_back(weak_ptr<>(arg);
}

Or you immediately take the argument as a weak_ptr, that doesn't
matter. The more important fact is, you created that object in some
C++, you have that object in python and then you call addCallback from
the python context.

So, before the function exits your object `arg` exists at least in three places:
 1) somewhere in C++ where it was created
 2) in the python context
 3) in the context of addCallback

To understand what is going wrong, I need to explain that the shared
pointer in context 1 and 3 are not the same! They share the same
deallocator and the same object, but they are not the same shared
pointer, i.e. their use count and weak_count are not the same. That is
due to the way boost python handles step 2, that magic is in these
three files (in boost/python):

converter/shared_ptr_to_python.hpp
converter/shared_ptr_from_python.hpp
converter/shared_ptr_deleter.hpp

In `shared_ptr_deleter.hpp` there is a certain magic implemented, that
does the following:
Assume you have class `Parent` and a class `Child` derived from it.
Now you can do:
 - create an instance of Child C++ and bring it to python as shared_ptr<Child>
 - pass that instance to C++ (via shared_ptr<Child> or shared_ptr<Parent>)
 - get it later back from C++ but as a shared_ptr<Parent>
 - magic: you can treat that instance as a shared_ptr<Child>
In C++ you would need to do a dynamic cast to get this functionallity,
but because that object has been known to python to be an instance of
Child, boost::python automatically makes it an instance of Child, nice
right?

Unfortunately your (and my) problem are a consequence of this. When
you go from 2->3 boost::python prepares for doing its magic. It
doesn't just return a copy of the shared_ptr from 1), it creates a new
shared_ptr with a special Deallocator object. The use_count at that
moment is 2: one for python 2) and one for addCallback() 3). When the
function addCallback() finishes, the use_count=1 (from python) and
weak_count=1 from 3). Once the python context ends then use_count=0
and weak_count=1, and I believe that is exactly what you observe.

In this case as use_count drops to 0 the boost custom Deallocator gets
called. This is usally not bad, as he just deregisters (decreases the
use cound by 1) in the shared_ptr for context 1.Only if that use_count
in context 1) would drop to 0 the object would get deleted (that's why
you don't observe it to be deleted). The problem is now, that the two
weak/shared ptrs (which still point to a healthy and alive object) are
now disconnected. So when you try to turn the weak_ptr in context3 to
a strong pointer you would get serious problems.

I hope this is a correct explanation of the sitation. To solve this
would need to revist the magic for shared_ptrs in boost::python. I
plan to try and solve it some point later, but I am no regular
developer for boost::python and I can not promise that I will succeed,
nor when.

-Holger

On Tue, Mar 6, 2012 at 03:23, Adam Preble <adam.preble at gmail.com> wrote:
> I am trying to break a smart pointer cycle in my application.  There is an
> interface that can be implemented in both c++ and Python, and it is used for
> messaging callbacks.  I can contain a list of these as shared pointers, but
> it only makes sense for cases where the owner of the container has exclusive
> control of the object in question.  These work fine in both c++ and Python.
>  I decided for the messaging callback list I would switch them to
> weak_ptr's.
>
> I get into trouble if I register a Python implementation of the callback
> with ownership staying within Python.  When it comes time to communicate a
> message back, I see that the weak_ptr has:
>
> use_count = 0
> weak_count = 1
>
> The pointer fails to cast.  Meanwhile, the object is looks very much alive
> in Python.  I have a destructor in the c++ definition interface for
> debugging and I see it never gets called.  Similarly, __del__ doesn't get
> called on the Python implementation, although I am lead to believe I can't
> trust it.  Perhaps most compelling is that I can continue to poke and prod
> at the object in the Python shell well after this failed weak_ptr cast.
>
> I am wondering if there is anything I can do for the cast to succeed.  At
> this point all I can think of is making the owner of the container get a
> specific reference to the blasted thing, which I'd rather not have to do.
>
>
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig