[Python-Dev] Need advice, maybe support

[Guido van Rossum]

> In the last months, I made very much progress with Stackless 3.0 .
> Finally, I was able to make much more of Python stackless
> (which means, does not use recursive interpreter calls) than
> I could achive with 1.0 .
> 
> There is one drawback with this, and I need advice:
> Compared to older Python versions, Py 2.2.2 and up uses
> more indirection through C function pointers than ever.
> This blocked my implementation of stackless versions, in
> the first place.
> 
> Then the idea hit me like a blizzard:
> Most problems simply vanish if I add another slot to the
> PyMethodDef structure, which is NULL by default:
> ml_meth_nr is a function pointer with the same semantics
> as ml_meth, but it tries to perform its action without
> doing a recursive call. It tries instead to push a frame
> and to return Py_UnwindToken.
> Doing this change made Stackless crystal clear and simple:
> A C extension not aware of Stackless does what it does
> all the time: call ml_meth.
> Stackless aware C code (like my modified ceval.c code)
> calls the ml_meth_nr slots, instead, which either defaults
> to the ml_meth code, or has a special version which avoids
> recursive interpreter calls.
> I also added a tp_call_nr slot to typeobject, for similar
> reasons.
> 
> While this is just great for me, yielding complete
> source code compatability, it is a slight drawback, since
> almost all extension modules make use of the PyMethodDef
> structure. Therefore, binary compatability of Stackless
> has degraded, dramatically.
> 
> I'm now in some kind of dilemma:
> On the one side, I'm happy with this solution (while I have
> to admit that it is not too inexpensive, but well, all the
> new descriptor objects are also not cheap, but just great),
> on the other hand, simply replacing python22.dll is no longer
> sufficient. You need to re-compile everything, which might
> be a hard thing on Windows (win32 extensions, wxPython).
> Sure, I would stand this, if there is no alternative, I would
> have to supply a complete replacement package of everything.
> 
> Do you (does anybody) have an alternative suggestion how
> to efficiently maintain a "normal" and a "non-recursive"
> version of a method without changing the PyMethodDef struc?
> 
> Alternatively, would it be reasonable to ask the Python core
> developers, if they would accept to augment PyMethodDef and
> PyTypeObject with an extra field (default NULL, no maintenance),
> just for me and Stackless?
> 
> Many thanks for any reply - sincerely -- chris

I don't think we can just add an extra field to PyMethodDef, because
it would break binary incompatibility.  Currently, in most cases, a
3r party extension module compiled for an earlier Python version can
still be used with a later version.  Because PyMethodDef is used as an
array, adding a field to it would break this.

I have less of a problem with extending PyTypeObject, it grows all the
time and the tp_flags bits tell you how large the one you've got is.
(I still have some problems with this, because things that are of no
use to the regular Python core developers tend to either confuse them,
or be broken on a regular basis.)

Maybe you could get away with defining an alternative structure for
PyMethodDef and having a flag in tp_flags say which it is; there are
plenty of unused bits and I don't mind reserving one for you.  Then
you'd have to change all the code that *uses* tp_methods, but there
isn't much of that; in fact, the only place I see is in typeobject.c.

If this doesn't work for you, maybe you could somehow fold the two
implementation functions into one, and put something special in the
argument list to signal that the non-recursive version is wanted?
(Thinking aloud here -- I don't know exactly what the usage pattern of
the nr versions will be.)

--Guido van Rossum (home page: http://www.python.org/~guido/)