[Python-Dev] Python Run-Time System and Extensions

[Daniel Silva]

[Note: I first sent this to Jeremy Hylton through his MIT e-mail address,
but in case he no longer uses that one, I'm resending to python-dev and
his Zope account.]

Hello,

My name is Daniel Silva and I'm working on a Python compiler that
generates PLT Scheme code.=A0 The work is nearly done, except for large
parts of the run-time system and support for python C extensions.

Since PLT's platform is MzScheme, I need to connect the MzScheme
foreign-function interface to the C Python foreign-function interface and
vice-versa.=A0 MzScheme's FFI works with SchemeObject C data structures and
Python's FFI works with PyObject, among others.=A0 We aim for
source compatibility, not binary.=A0 To achieve this, we see two
possibilities: provide our own Python.h and typedef PyObject as another
name for SchemeObject, or marshall SchemeObject structures into PyObject
structures.

If we were to pretend that SchemeObjects are PyObjects, we could have
Scheme do most of the work, but we run into problems with C structure
field access.=A0 Through this method, we can use the existing code of the
Python runtime system that uses selectors -- which I heard you are
responsible for (thank you!) -- and replace the implementation of
selectors like PyString_Get_Size with calls to Scheme equivalents, such as
scheme_string_get_size, which would not break code that uses selectors.

This approach is problematic when we encounter C code that looks like
my_py_obj->some_field.=A0 This obviously would be incompatible, as
SchemeObjects do not have the same fields.=A0 Such a style is used in
various parts of the Python runtime system, and we would have to
re-implement all of those.=A0 That is a bit of a burden, but more worrysome
is the possibility of third-party Python C extensions using this style --
those would not work with our system.

The alternative is to marshall every SchemeObject into the PyObject data
structure described in CPython's own headers.=A0 This method would make it
possible for us to use both the Python runtime system (and automatically
keep up with changes) and third-party extensions.=A0 However, once our
objects are marshalled into PyObjects, any change made to the new target
is not seen by the original SchemeObject, so we lose mutation.=A0 Without
mutation, our interpreter is useless for virtually every Python program.

We are ready to pick an option and run with it.=A0 Do you think one of thos=
e
two holds better hope than the other, or do you see a third alternative?
I am willing to provide the remaining selectors for the CPython project,
or if they already exist, to write the necessary documentation to advocate
their use to those writing extensions.

Regards,

Daniel Silva