C wrappers and the proxy dilemma

[Ken Seehart]

Anyone who has wrapped C or C++ libraries has encountered the proxy 
dilemma. 

A long time ago, I naively thought that I could start by deriving my 
high level python class from the c-python type, but this leads to many 
difficult problems because several of the underlying methods return the 
low level objects.  After reading up on the subject, I learned that the 
"correct" solution is to use composition rather than inheritance.  It 
makes sense, but it is nevertheless rather annoying.

An excellent example of this is the wxPython library, which uses 
composition (proxies) and even solves the OOR (original object return) 
problem that is associated with this kind of proxy oriented solution.

Conclusion: There is no really good solution to this kind of thing that 
doesn't get messy.  Wrapping C or C++ in python becomes difficult 
whenever you have methods that return (or take as arguments) instances 
of the objects that you are trying to wrap in high level python.  Any 
solution seems to add more overhead than is desirable (in terms of both 
programmer time and run time).

This problem comes up over and over again, so perhaps it is even worth a 
PEP if a change to the python language would facilitate a more 
convenient solution to this nagging problem.

First, mentally set aside the idea of using composition.  I understand 
the set of problems that it solves, but it also creates a new set of 
problems (such as function call overhead on every method call).  I also 
understand why composition is better than inheritance.  But in order to 
contemplate this proposal, it is necessary to temporarily set aside the 
composition idea.  This proposal involves taking another look at 
something slightly more akin to the inheritance approach that we all 
gave up on before.

Okay, so here is a somewhat radical proposal:

Add a writable  __overload__ attribute to low level python type.  This 
would be assigned a python 'type' instance (or None to have no effect).  
The effect is to place __overload__ at the /beginning /of the __mro__ of 
the low level type, making it possible to directly alter the behavior of 
the base python type without using inheritance.  This means that 
instances of the base type acquire the attributes of our high level 
python class.  It is important that the __overload__ type is checked 
/before /the actual type.

Yeah, it sounds a bit scary at first.  A bit too loose maybe.  So the 
type definition should have to explicitly enable this feature to prevent 
people from doing bad things.  But dwelling too much on the scariness 
seems somewhat non-pythonic.  It is more important that we can do really 
good things than that we prevent ourselves from going out of our way to 
do bad things (as long as it is not too /easy /to do bad things).

So here are some of the benefits:

1. Eliminates the duality between proxy objects and low level objects.

2. Increased speed on methods whose syntaxes are not being altered 
(because no wrapper is needed).

3. Much less work than writing wrappers for every method (even if such a 
task is partially automated).

Usage example:

from foolib import c_foo

class Foo(c_foo):
    """Overload of c_foo"""

    def __new__(typ, *args, **kwargs):
        return c_foo(*args, **kwargs)

    def __init__(self, parrot_state):
        c_foo.__init__(self)
        self.parrot_state = parrot_state

    def myfunc(self):
        return 5 * self.juju(self.parrot_state) # juju method is defined 
in foolib

# This makes all c_foo instances into Foo instances:  Poof!
c_foo.c_footype.__overload__ = Foo



x = Foo('not dead yet') # actually creates a c_foo instance that looks 
like a Foo instance

# c_foo.Add is defined in foolib and takes c_foo instances
x.Add(Foo('strange'))
x.Add(Foo('missing'))

# c_foo.GetChildren is defined in foolib and returns c_foo instances
for y in x.GetChildren():
    print y.myfunc() # Wow, I am magically accessing Foo.myfunc


Yeah I know, that's an infinitely recursive inheritance; but that's 
okay, we just implement the __overload__ feature such that the MRO 
thingy will do the right thing for us.  The semantics will be that Foo 
is derived from c_foo, and all instances of c_foo behave as if the are 
instances of Foo.

I used the term "radical" earlier, to describe this idea.  What I meant 
was that it seems to violate some basic Object Oriented principles.  
However, when evaluating solutions I try not to apply any principles too 
dogmatically.  Theory is only relevant when it has /practical 
/consequences.  So I recommend not quoting theory without also defending 
the relevance of the theory to the case in point.  IMO, anything that 
eliminates thousands of lines of boilerplate code is worth bending 
theory a little.

Is it possible to get the same syntactic/semantic results without 
changing python?

- Ken

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20070622/d4b4b443/attachment.html>