Extending Python: rewriting a single method in C

[Jacek Generowicz]

jepler at inetnebr.com (Jeff Epler) writes:

> On 12 Mar 2001 11:13:51 +0000, Jacek Generowicz
>  <jmg at ecs.soton.ac.uk> wrote:
> >I have an object-oriented numerical code written in Python. It has
> >come to the stage where the speed of the code is seriously impeding
> >progress. Profiling reveals (surprise, surprise) that most of the time
> >is spent in only 2 methods.
> >
> >Is there any way of re-writing just these methods in C ?
> 
> It's easy to write a function in C.  Other posters have mentioned references 
> to documentation about doing this.

As I mentioned in my replies to Paul and Alex (which I inadvertently
sent by mail rather than through the NG) I know that this can be done,
but I specifically want to rewrite single class/instance methods . . .

> However, you cannot make a builtin function be a method of a class/instance.

Bummer.

> Instead, you could write:
> 
> from cmodule import fast_a, fast_b
> 
> class fastKlass(Klass):
> 	def a(self, int1, int2):
> 		fast_a(self, int1, int2)
> 	def b(self, int1, int2):
> 		fast_b(self, int1, int2)
> In fast_a and fast_b, you can get arguments by using the ParseTuple string
> "(Oii)" and then use PyObject_SetAttrString and PyObject_GetAttrString
> to store or retrieve attributes (including, if necessary, instance
> methods) from the instance by their name.

Perhaps it is appropriate to insert here the exchange between Alex and
myself:

JG> AM> JG> Could something along following lines work ?
JG> AM> JG> 
JG> AM> JG> Write a module called my_extensions containing a function called
JG> AM> JG> foo_bar.
JG> AM> JG> 
JG> AM> JG> import my_extensions
JG> AM> JG> 
JG> AM> JG> class foo:
JG> AM> JG>    bar = my_extensions.foo_bar
JG> AM> 
JG> AM> No, this does not work, because only python-function-objects, NOT
JG> AM> builtin-function-objects, are mutated into unbound methods when
JG> AM> they are attributes of a class-object.
JG> 
JG> Yes, I've noticed this after a bit of experimenting.
JG> 
JG> AM>  Note that the name
JG> AM> 'builtin function object' is a bit misleading, as it actually
JG> AM> describes any function _written in C_ (part of an extension).
JG> AM> 
JG> AM> What CAN work is more explicit delegation:
JG> AM> 
JG> AM> class foo:
JG> AM>     def bar(self, whatever, blahblah):
JG> AM>         return my_extensions.foo_bar(self, whatever, blahblah)
JG> 
JG> Yes, but I was trying to avoid the function call overhead
JG> (particularly as one of the functions I need to rewrite is very short
JG> indeed).

> Note that if there are *many* calls to the method, rather than each
> invocation taking a large amount of time,

Bingo, this is exactly the case for one of the methods.

> then this approach is not going to gain you much, since it actually
> *increases* the function-call overhead.

And here is what Alex had to say about this:

AM> > Yes, but I was trying to avoid the function call overhead
AM> > (particularly as one of the functions I need to rewrite is very short
AM> > indeed).
AM> 
AM> No way to do that -- the function call overhead is there even
AM> if you call directly from Python to the C extension. 

But isn't the function call overhead greater for calling python
functions than it is for calling the C extensions ?

AM> If the function/method is called from a few places, you might
AM> manually inline in there, maybe.

Not sure I understand what you mean exactly. Are you suggesting that I
call an extension function wherever I would previously have called and
instance method ?

How do I get at the members of a class in C if that class was defined
in python ?

JE> In fast_a and fast_b, you can get arguments by using the ParseTuple string
JE> "(Oii)" and then use PyObject_SetAttrString and PyObject_GetAttrString
JE> to store or retrieve attributes (including, if necessary, instance
JE> methods) from the instance by their name.

Is there an organized reference where I can find the specifications of
all these functions ?  All I have come across shows example code, but
no full descriptions of what can/must be done.

Maybe some kind soul could show me what the correct code for a minimal
extension type is ?

For example, I tried to write an extension type equivalent to

class foo:
    def __init__ ( self, a ):
	self.a = a
    def show ( self ):
	print a

My various attempts, almost work, but there is always something not
quite right.

Incidentally, why does the xxobject.c file in the Objects directory of the
Python distribution contain no initxxobject function ?

Jacek