[Python-3000] Cross-interpreter FFI for Python 3000?

[Phillip J. Eby]

At 02:45 PM 8/8/2006 -0700, Paul Prescod wrote:
>As you said elsewhere, PyPy can compile an Rpython+rctypes program to a
>C file, just as Pyrex does. So I don't understand why you see them as
>"entirely unrelated".

Disclaimer again: I like and use Pyrex; I even built additional support for 
it into setuptools.  Conversely, I've only used ctypes once and am not sure 
I care for its API.  But as a practical matter, these preferences are 
irrelevant; I will end up learning to use ctypes and liking it, and so will 
everybody else, because ctypes' *dynamic* advantage will clean Pyrex's 
clock at the very moment that extcompiler is as easy to use as Pyrex is now.

To summarize the differences, Pyrex is:

* A *Python-like* language, rather than Python
* Invents new inheritance/import facilities
* Imports various bits of syntax from C, including operators, pointers, etc.
* Inherently tied to the CPython API in its implementation
* Has its own system of "header" files for compile-time import/include
* Generates C code directly from Pyrex
* Cannot be executed by standard Python

PyPy's RPython+rctypes is:

* 100% Python, with certain dynamicity constraints
* Is not tied to any particular back end -- it can be translated to C, LLVM 
code, or even JavaScript if you like, as the type inference, annotation, 
and optimization machinery is backend-independent
* Code can be run in a normal Python interpreter if a ctypes library is 
available

The only relationship I see between the two are some overlap in use cases, 
and the letters "R", "P" and "Y" in the names.  :)  In particular, Pyrex 
cannot be used in the interpreter, and I can't see Guido allowing Pyrex's C 
syntax to infect Python-the-language, so this is likely to be a stable 
barrier keeping Pyrex from having this feature, unless Greg or somebody 
else decides to create a Pyrex interpreter, or perhaps an import hook to 
translates Pyrex source code to Python bytecode that invokes the ctypes 
API.  :)

(Note, by the way, that such an import hook/translator would be equally 
usable in PyPy, instantly making it possible to compile Pyrex to any 
backend supported by PyPy!  I suggest you let that idea sink in for a 
little bit, as it helps to illustrate why making ctypes the standard FFI is 
the One Obvious Way To Do It.)


>There are different syntaxes, but the goals are very similar.

Well, you could say that about Python and Ruby, to name just two.  Syntax 
is important.  :)

But that's also entirely ignoring the wide range of practical issues 
alluded to above, and some more I'll dig into below.


>Pyrex uses optional type declarations (which are planned
>for Python 3000). RPython infers types from rctypes API calls (which
>will also be available in Python 3000).

They're available in Python 2.5, which means code can be written for them 
today.  The dynamic usability of ctypes from interpreted Python means that 
Pyrex will become a historical footnote as soon as the 
RPython+rctypes->CPython translator is practically usable; i.e., when it 
can compete with Pyrex for code generation speed (and speed of generated 
code), installability, documentation, and user community.

At that point, the advantage of being able to debug your C interface using 
the interpreter's ctypes library, and then to compile the code only if/when 
you need to, will be a killer advantage.

IMO, it doesn't make sense to fight that now-inevitable future, either on 
behalf of Pyrex or some imagined "better" alternative; instead, we might as 
well hasten that future's arrival.  We can always provide better syntax for 
ctypes at a later date, the way 'classmethod' and friends arrived in Python 
2.2 but didn't get syntax until 2.4.  If you can't wait that long, write 
that import hook to turn Pyrex source into Python bytecode.  :)


>  Perhaps it would be better if I
>dropped the reference to Rpython and merely talked about "extcompiler"
>there tool which is very parallel to the Pyrex compiler?

I'm at a bit of a loss as to how to explain how very not-useful that 
comparison is.  I would suggest reading up on PyPy architecture and Pyrex 
architecture a bit.  From an end-user perspective you can compare them as 
things that take Python-looking stuff in and spit C code out, but the devil 
is definitely in the details.  See also the lists I gave above.


>You make some good points about Pyrex and ctypes. I'd rather explore the
>design space after I've heard whether this design direction has the
>potential to be fruitful. I infer that you think "yes".

See http://dirtsimple.org/2005/10/children-of-lesser-python.html for what I 
think.  :)

In that article, I highlighted the absence of a standard Python FFI as 
being a stumbling block to the future evolution of the language, but noted 
that PyPy would likely end up with a solution.  The subsequent emergence of 
ctypes as an FFI shared by CPython and PyPy has already solved this 
problem; it is merely a question of recognizing the fact.

As of Python 2.5, anything else is going to have a serious uphill battle to 
fight -- even if it's something like Pyrex, that at least already *exists* 
and has at least *one* part-time maintainer.  A brand-new FFI invented by 
committee and with nobody yet stepping up to implement or maintain it, 
really has no chance at all.

(This is all IMO, of course, but I find it hard to imagine how anything 
else could succeed.)