[C++-sig] Tips for exposing classes with own memory management model

[Alex Leach]

Hi,

Thanks again for the fast response!


On Thu, 18 Apr 2013 17:14:05 +0100, Jim Bosch <jbosch at astro.princeton.edu>
wrote:

>
> You can't really register the types themselves.  All you can do is  
> register custom converters for them, i.e.  You'll need to read the code  
> and comments in the "converter" subdirectories of the Boost.Python  
> source to really learn how to do that, though I think there are some  
> how-tos scattered about the web.

Registered converters do sound another good option; I have seen some nice
how-tos describing their usage. Thanks again for another decent
suggestion. So far, I've found the class_<> template so easy to use that
I've hardly needed to touch converters, nor any form of registry.

>
> That would be enough to allow you to wrap functions and member functions  
> that take these objects using boost::python::make_function and the like,  
> and you could then add those to your type object using C API calls  
> (PyObject_SetAttr) or their Boost.Python equivalents.  Even so, you're  
> starting down what seems like a really painful road.

Yes, I think you're right; it does sound a very long and painful road! But
when I use a generator to iterate through containers holding millions of
instances - which I totally intend to do - I'd like the base classes of
these instances to be as lightweight as possible.

I don't know much (anything) about the implementations, but I've read that
Cython's memoryview's are amazingly efficient at extracting lightweight
instances from C/C++ containers. I think numpy arrays might be similar in
this regard. From what I've been seen today, I imagine there are
specialised PyTypeObject's somewhere in their midsts.

>
> Doing this sort of thing will allow you to get a Python object that's an  
> instance of your PyTypeObject.  It might be a useful bit of utility  
> code, but it's really not directly what you want, I think, which is to  
> be able to convert between Python instances of your class and C++  
> instances.

Yes, I was hoping this memory-managed base class could be used very
similarly to both Python's and Boost's 'object'. Registered converters
seem like a must in this regard.

>
> It's pretty much definitely not worth it, IMO; you'd have to essentially  
> duplicate and rewrite major parts of Boost.Python to support putting a  
> custom PyTypeObject in a class_.  The class_ infrastructure relies very  
> heavily not just on its own PyTypeObject hiearchy, but also on a custom  
> metaclass.

I wouldn't want to repeat anything - I'm familiar with the principles
D.R.Y. and KISS... Inheriting functionality from class_<> and overriding
specific members would be preferable, assuming its constructors doesn't
explicitly use PyTypeObject. Is that impossible, given the current class_
template inheritance chain?

>  In fact, now that I think about it, you'll probably need to do some of  
> that even if you don't try to use class_ or something like it.  I was  
> originally thinking that maybe you could get away with essentially  
> wrapping your own classes just using the Python C API directly (i.e.  
> following the approach in the "extending and embedding" tutorial in the  
> official Python docs), but then use Boost.Python to wrap all of your  
> functions and handle type conversion.  But even that seems like it's  
> pretty difficult.

Yes, that does sound tough! I've struggled to understand the C-Python docs
before, and can't waste too much time atm repeating the process..

>
> So I guess the summary is that I think you may be making a mistake by  
> taking this approach, but I'm sure you'll learn something either way.   
> You've been warned ;-)

Thanks for the warnings, and please excuse my naiivety.

It was late last night when my real problem made me think of defining a
new PyTypeObject, but I should definitely steer clear of any potential
pitfalls and time-thieves atm. I should just get it working, first off...

Actual problem
--------------

My actual problem? (Sorry, it's hard to explain without an inheritance
diagram... So I just made one - see the png attached.)

Yesterday, I witnessed first hand the diamond of death multiple
inheritance problem. I had extended the Base class to the memory-managed
class - lets call these BaseObj and ManagedObj, respectively - with an
additional 'Print(void)' method, to be exposed in Python as
PyBaseObj.__repr__.

ADerived (and about 100 other classes) all inherit from ManagedObj, and I
thought it could be possible to call 'this->Print()', in e.g. PyADerived's
methods. As the libraries I'm using don't use virtual inheritance, I soon
learnt this would be impossible, but I figured I could add
'bases<PyBaseObj, PyManagedObj>' to the 'class_<PyDerivedObj, ...>'
template constructor, and call it from C++ with:
'bp::object(aderived_obj).attr("__repr__")()'.

So I've done that, which seems to work fine, but there do seem to be
problems in the registry when calling certain derived class methods. I've
got the old error:-

Boost.Python.ArgumentError: Python argument types in
       PyBDerived.GetA(PyBDerived)
did not match C++ signature:
       GetA(PyBDerived{lvalue})

The return type (which isn't mentioned in the above error) is a reference
to another class - PyADerived - which also indirectly derives from
ManagedObj.

I can't figure out exactly why this breaks - I've played a lot with
constness, to no avail - but I think the fix might be to expose ManagedObj
directly, and not a class derived from it, which is what I've been doing
fairly repeatedly up until now... BaseObj, however, would have to be
derived, as it has pure-virtual member functions.

It would probably have been much more sensible to write the Print method
as a free function, and attach it to ManagedObj with def(...), rather than
exposing as a derived class member function.

I don't actually need any functionality in BaseObj that I can't call from
a derived class, so I think I should just skip exposing that and attach
the 'Print' method directly to the exposed ManagedObj...

Apologies for the rambling and the lengthening emails!

Thanks again for taking the time to write out your thoughts.
Kind regards,
Alex




-- 
Using Opera's mail client: http://www.opera.com/mail/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: thin_wrappers.png
Type: image/png
Size: 30732 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20130418/ee1c2305/attachment-0001.png>