[C++-sig] Manage new object return policy (was Re: new to python; old to C++)

[David Abrahams]

on Wed Nov 12 2008, Paul Melis <paul-AT-pecm.nl> wrote:

> David Abrahams wrote:
>> on Mon Nov 10 2008, Paul Melis <paul-AT-pecm.nl> wrote:
>>
>>   
>>> The FAQ entry on this doesn't really help, so perhaps I'm not using the return
> policy
>>> correctly or missing something else. The full test code (which is actually quite
>>> small) is attached.
>>>     
>>
>> Please reduce it to its absolute minimum.  Remove every single thing you
>> don't need in order to produce the result.  If the problem isn't
>> glaringly obvious to you then, I will help.
>>   
> Well, I tried reducing in several different ways. The bottom line is
> that I can't seem to produce this without using the wrapper class (or
> perhaps I missed the right combination of return policies).

This is a classic example of the FAQ entry.  See below.

> Simplest test case (see below for the rest of the files):
>
> #!/usr/bin/env python
> from boost import *
>  
> class MyCallback(Callback):
>  
>     def __init__(self):
>         Callback.__init__(self)
>  
>     def __call__(self):
>         return Value()

A new C++ Value instance is created here, managed by a Python Value
instance.  When this function returns, its caller's stack frame will own
the only reference to Value.  When this is invoked from C++ as an
override of a C++ virtual function, its caller is a function in
Boost.Python.

>  
> cb = MyCallback()
>  
> r = go(cb)
>
>
> Storing a reference to the Value instance returned in __call__ keeps the
> instance alive, so the dangling pointer is caused by the Value instance
> that is returned getting destructed. But this only happens when the
> instance is passed through go() as its return value. When calling cb
> directly the Value is not destructed (as I would expect). What I don't
> understand is why it is destructed when it *is* passed through go(),
> shouldn't the manage_new_object policy keep it alive?

I hope the other things I've written above and below answer the first
question.  As for the second one, manage_new_object says to manage a new
C++ object's lifetime, and the problem you have is that a Python object
already owns the C++ object, and that object will go away.

I might have a hunch about how to fix some of these problems
using a Python WeakPtr, but unfortunately I don't have any time right
now to work on it.  Best I can do is suggest you open a Trac ticket
referencing this thread.

> Regards,
> Paul
>
> // classes.h
>
> #ifndef CLASSES_H
> #define CLASSES_H
>
> #include <stdlib.h>
>
> class Value
> {
> public:
>     Value()                 { printf("Value::Value()\n"); }
>     Value(const Value& v)   { printf("Value::Value(const Value& v)\n"); }
>     ~Value()                { printf("Value::~Value()\n"); }
> };
>
> class Callback
> {
> public:
>     Callback()                      {}
>     Callback(const Callback& cb)    {}
>     virtual ~Callback()             {}
>     // Returns new Value instance
>     virtual Value* operator()() { return new Value(); }

This one doesn't get executed; it's been overridden by CallbackWrap.

> };
>
> Value *go(Callback *cb)
> {
>     return (*cb)();
> };
>
> #endif
>
>
> // boost.cpp
>
> // g++ -shared -fPIC -O3 -o boost.so -I/usr/include/python2.5 boost.cpp
> -lboost_python -lpython2.5
>
> #include <boost/python.hpp>
> #include "classes.h"
>
> namespace bp = boost::python;
>
> struct CallbackWrap : Callback, bp::wrapper<Callback>
> {
>     virtual Value *operator()()
>     {
>         printf("CallbackWrap::operator()\n");
>
>         if (bp::override call = this->get_override("__call__"))
>         {

A python override has been found here

>             printf("Have override for __call__, calling it\n");
>             return call();

It is invoked in the line above. To return control to you, Boost.Python
will reach into the Python Value object to pull out a pointer to the C++
Value object it manages.  However, in your case, the reference count on
the Python object is 1, and the C++ object will have been deleted by its
owner Python object by the time the library returns.

>         }
>
>         return Callback::operator()();
>     }
>
>     Value *default_call()
>     {
>         printf("CallbackWrap::default_call()\n");
>         return this->Callback::operator()();
>     }
> };
>
>
> BOOST_PYTHON_MODULE(boost)
> {
>     bp::class_< Value >("Value");
>
>     bp::class_< CallbackWrap, boost::noncopyable >("Callback")
>         .def("__call__", &Callback::operator(),
> &CallbackWrap::default_call,
> bp::return_value_policy<bp::manage_new_object>());
>
>     bp::def("go", &go, bp::return_value_policy<bp::manage_new_object>());
> }
>
>
>
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com