[C++-sig] Tips for exposing classes with own memory management model

[Jim Bosch]

On 04/18/2013 11:44 AM, Alex Leach wrote:
> Thank you for the quick response!
>
> On Thu, 18 Apr 2013 15:24:09 +0100, Jim Bosch <talljimbo at gmail.com> wrote:
>
>     If you go with writing your own PyTypeObject, you will indeed have a lot more control, but it will greatly limit how much Boost.Python you can use (no class_, for instance, at least), and you'll need to dive deep into the Boost.Python implementation to learn how and when you can use it.  I'd only consider recommending this approach if you wanted to wrap one or two simple classes this way and then use regular Boost.Python for everything else.
>
>
> No class_<..> type would be a problem, as I've already exposed a lot of its derived classes this way.. There's got to be about 100 in total. :-\
>
>     I think the best solution would probably be to use shared_ptr with a custom deleter, as that gives you control over how your objects are allocated while giving Boost.Python an object it knows how to handle extremely well.  One key ingredient of this is that instead of wrapping C++ constructors, you'll want to wrap factory functions that return shared_ptrs.  You can even wrap such functions as Python constructors using make_constructor.
>
> I've already had to do this once, so I've got some experience with the technique, although can't remember exactly why it was needed. Thank you for a viable option, though!
>
>     All that said, my first recommendation would be to try to wrap it (or at least a subset of it) without trying to get the optimal memory performance first, and only fix it if it actually is a performance problem.  You might be surprised at where the time ends up going.
>
>
> :) Lol, yes, I can see how attempting to define a new PyTypeObject could become very time-consuming!
>
> If I were to go this route though (I probably will), is there a Boost Python registry or something where I can register these new types? I just thought to look into list.hpp, as I figure the boost::python::list would probably make use of the PyListType, which could be thought of as a similar PyTypeObject specialisation to one I might like to create.
>
>

You can't really register the types themselves.  All you can do is register custom converters for them, i.e.  You'll need to read the code and comments in the "converter" subdirectories of the Boost.Python source to really learn how to do that, though I think there are some how-tos scattered about the web.

That would be enough to allow you to wrap functions and member functions that take these objects using boost::python::make_function and the like, and you could then add those to your type object using C API calls (PyObject_SetAttr) or their Boost.Python equivalents.  Even so, you're starting down what seems like a really painful road.

> ......
>
>
> Now I've done a little more digging, I think I have an idea of how to do it, which I'll detail now.. Any tips, pointers or advice in Boost::Python'ising it would be appreciated, as I have next to no experience with the Python-C API beyond Boost.
>
> Resources
> ---------
>
> Worth mentioning that the best (simplest) resources I've found to go by, are:
>
>     [1] - For the C-Python side: http://docs.python.org/2/extending/newtypes.html
>     [2] - For Boost-Python side: http://www.boost.org/doc/libs/1_53_0/boost/python/list.hpp
>
>
> Steps to expose:-
> -----------------
>
> a.  - Create mytypeobject.h, a raw C-Python extension type, as explained in [1].
>
> a.i. - Create unit-test, by deriving such a type in Python, making a
>         few instances, deleting some and leaving others to the garbage
>         collector.
> b. Create mytypeobject.hpp, where a type similar to boost::python::list is declared. Register it, with for example:-
>
> // From list.hpp:-
> //
> namespace converter
> {
>    template <>
>    struct object_manager_traits<list>
>        : pytype_object_manager_traits<&PyList_Type,list>
>    {
>    };
> }
>

Doing this sort of thing will allow you to get a Python object that's an instance of your PyTypeObject.  It might be a useful bit of utility code, but it's really not directly what you want, I think, which is to be able to convert between Python instances of your class and C++ instances.

> What else?
> ----------
>
> I very much doubt it will be this simple, as any exposed class_<> would probably still attach a typical PyObject, thereby using PyTypeObject's memory management functions, rather than that of any type object specialisation.
>
>
> Example
> -------
>
> For example, let's suppose PyListObject has some functionality I need in an exposed class, and I need the garbage collector to use PyListType instead of PyTypeObject, when managing its memory.
>
> Is there a way to attach a PyListObject to a class_<> instance, instead of a PyObject? Perhaps an alternative to the class_<> template could be designed to use any arbitrary type, instead of the default PyTypeObject. Something which could be used like this would be cool:-
>
> boost::python::type_object_<PyListType, PyList, [boost::noncopyable]>
>      ("alt_list", "PyListObject wrapped in Boost.",
>       init<>())
>      .def("append", PyList_Append)
>      /// ...
>      ;
>
> How much farther do you think I would need to dig into Boost internals to implement such functionality? Worth it?
>

It's pretty much definitely not worth it, IMO; you'd have to essentially duplicate and rewrite major parts of Boost.Python to support putting a custom PyTypeObject in a class_.  The class_ infrastructure relies very heavily not just on its own PyTypeObject hiearchy, but also on a custom metaclass.  In fact, now that I think about it, you'll probably need to do some of that even if you don't try to use class_ or something like it.  I was originally thinking that maybe you could get away with essentially wrapping your own classes just using the Python C API directly (i.e. following the approach in the "extending and embedding" tutorial in the official Python docs), but then use Boost.Python to wrap all of your functions and handle type conversion.  But even that seems like it's pretty difficult.

So I guess the summary is that I think you may be making a mistake by taking this approach, but I'm sure you'll learn something either way.  You've been warned ;-)


Jim