[IronPython] CPython Extensions from IronPython

[Fuzzyman]

Hello all,

Yesterday I spent some time looking at hosting the CPython interpreter 
from IronPython.

Thanks to Jonathon Hogg for suggesting we try embedding and Seo for 
pointing us to Python.NET. Thanks also to Resolver for letting me spend 
time on this.

Basically the Python.NET has already wrapped CPython as an assembly for 
embedding in .NET applications. It is quite a thin wrapper around the 
embedding API, so you have to acquire and release the GIL around 
everything and you send and receive PyObjects.

I'll write up what I've done and post it, with the code, to my blog - 
and send a heads up to these lists. What I have achieved is only proof 
of concept and far from the final solution. It is not even clear that 
this is the final approach we will take. The next step is for us to 
experiment with replicating the CPython API and recompiling extensions 
with managed C++. This should give us an idea of which path will be most 
fruitful.

The work I have done so far has some serious limitations, but it does 
work, so if you have a desperate need to use CPython extensions from 
IronPython you can probably build on what I have done.

I have passing data-structures in both directions working (ints, floats, 
strings, tuples, lists and dictionaries), with object proxying for 
everything else. The GIL is acquired and released for you around 
everything. I have successfully used matplotlib with Tkinter from 
IronPython with this!

I still have issues with passing booleans and None in and out, plus 
other basic types like complex numbers aren't converted for you (not 
difficult problems I just haven't sorted it out). Additionally data 
structures are copied in and out which is expensive - and you lose 
changes that CPython makes to mutable data structures you pass in. Magic 
methods aren't yet proxied because I am just using '__getattr__' for 
proxying attribute access and '__call__' for proxying calls. A 
'__getattribute__' approach, or implementing the magic methods on the 
proxy objects would get round some of these problems - but not (I think) 
solve type checking issues.

For more efficient work you can leave data as PyObject structures rather 
than copy. One avenue of investigation would be to see if we can make 
these PyObject structures (managed wrappers around the CPython data) 
behave like  their equivalent IronPython objects. This would allow some 
source code to operate unmodified.

Oh - and my approach only works with IronPython 2 because of the 
annoying bug when passing Arrays as arguments in IronPython 1. This can 
be worked around, but targeting IronPython 2 may be better as we could 
look at using the DLR to help with making PyObjects behave like 
IronPython data types.

All the best,

Michael Foord
http://www.manning.com/foord