[C++-sig] Re: call_method

[David Abrahams]

David Abrahams <dave at boost-consulting.com> writes:

> "Peter Dimov" <pdimov at mmltd.net> writes:
>
>>> I'd like to explore a few refinements with you. For example, it may
>>> be possible to get real shared_ptr interoperability by deriving
>>> py_counted_base from counted_base. Maybe we should ask Peter to
>>> weigh in on this. Peter, Brett's code is attached to this page:
>>> http://aspn.activestate.com/ASPN/Mail/Message/1430235
>>
>> I don't really understand what the code is about ;-) 
>
> Here's the quick skinny: 
>
> - Boost.Python allows you to specify how Python objects will hold the
>   C++ objects they wrap. You can specify that they be held by
>   shared_ptr<T> (or any other smart pointer), in which case the
>   library will generate converters to/from Python for
>   shared_ptr<T>. The to_python converters will simply build a new
>   Python object around the shared_ptr<>.
>
> - If you have virtual functions you want to ``override in Python'',
>   you actually have to hold the T object with a derived class U which
>   overrides the virtual functions to dispatch back to Python. In this
>   case, class U naturally has to have access to the Python object
>   (that's the pyobj_ member you see in Brett's code).
>
> - You can further specify that your C++ object is held by
>   shared_ptr<U>, which is what Brett is doing. That allows you to hold
>   a U object for dispatching, yet still pass shared_ptrs around in
>   your C++ code.
>
> There are several problems with the above arrangement, but the most
> important one is that if you keep the shared_ptr<U> alive longer than
> its corresponding Python object, calls to the Python-overridable
> virtual functions will crash, because they'll try to call through an
> invalid pointer. That's what Brett is trying to address with his crazy
> pointer.
>
>> but I'm leaning towards moving counted_base back to boost::detail
>> and removing the intrusive counting "feature" of shared_ptr.
>>
>> The interoperability/shared_from_this problems have alternate solutions as
>> described in the techniques "document".
>
> Hmm. I guess that rules out that option.
>
>> I'm not sure whether this helps or not, but it's also possible to have a
>> shared_ptr to PyObject, by using Py_DECREF as the deleter (also described in
>> the attached file).
>
> Now /that's/ starting to sound intriguing.  I wonder if it's possible
> to derive our U class from boost::python::objects::instance<>, and
> automatically generate shared_ptr<> converters for it.
>
> Maybe the whole idea of holding a shared_ptr<U> is bogus from the get-go.

OK, I've implemented the first part of this change in the CVS.
The current status is:

* Any Python object wrapping a C++ object of type T can be converted
  to a shared_ptr<U>, where U is T or any of its publicly-accessible
  base classes and U has been exposed with class_<U, ...>

* There is no need to explicitly specify shared_ptr<T> as the holder
  in class_<T, ...>

* The shared_ptr actually manages the lifetime of the Python object,
  so it will never be destroyed before the corresponding C++ object is
  destroyed.

Not implemented yet:

* Automatic conversion of shared_ptr<T> to python (e.g. when used as a
  return type). For that, you still need to pass shared_ptr<T> to
  class_<...> as the holder, and since the old mechanism doesn't take
  advantage of the new shared_ptr deleter introspection feature yet,
  you always get a new Python object.

Open questions:

* There is some cost in code and compilation time associated with
  these automatic conversions for each class_<...> instance. Should
  users really be forced to pay for these conversions, or should they
  be explicitly requested, e.g.

     register_shared_ptr<T>();

  Such a function may be needed anyway, since classes that are exposed
  other than via class_<> (see, e.g. class Simple in
  libs/python/test/m1.cpp) may want to be managed via shared_ptr<>.

* Can we do better for "callback classes" which dispatch virtual
  functions to Python (like BaseWrap at
  http://www.boost.org/libs/python/doc/tutorial/doc/class_virtual_functions.html)
  than to ask users to store their own PyObject* "self" reference?
  Some opportunities we may be missing by not providing a nice base
  class:

  1. The ability to prevent users from copying, or even creating these
     objects - which is dangerous because of the weak PyObject*
     reference.

  2. The ability to relieve users from having to touch the self
     pointer when implementing virtual function overrides (call_method
     could be a member of the base class)

  3. The ability to convert shared_ptrs, references, and pointers to
     these classes into Python objects without creating a new Python
     object - instead we could just use the stored self pointer if we
     can detect derivationj from our special base class. I'm not very
     convinced this is useful for shared_ptr, since deleter
     introspection will allow us to retrieve the PyObject* if the
     object came from Python -- and shared_ptr<BaseWrap> shouldn't
     arise otherwise if we implement #1. However, it might be useful
     for regular pointers and references.

* And now, a radical notion: should we ban the use of shared_ptr<T> as
  an argument to class_<...> ?
  

-- 
                       David Abrahams
   dave at boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution