[pypy-dev] cpyext - compilation, and swig?

[Amaury Forgeot d'Arc]

Hi,

2011/5/13 Dan Stromberg <drsalists at gmail.com>:
>
> When I build a swig-generated C extension module against CPython 2.6, and
> attempt to load it using cpyext, I get:
>
> /usr/local/pypy-1.5/bin/pypy
>>>>> cpyext.load_module('_odirectcmodule.so', 'odirect')
> Traceback (most recent call last):
>   File "<console>", line 1, in <module>
> ImportError: unable to load extension module './_odirectcmodule.so':
> ./_odirectcmodule.so: undefined symbol: PyClass_Type
>>>>>
>
> 1) Is that the right way to load a C extension module using cpyext?

No, you should just import it, like with CPython.

> 2) Do I need to recompile the module for pypy?

Yes, because the memory layout of objects is different, and some
macros are turned into function calls.
And to enforce the difference with CPython, the extension modules are
named differently: mymodule.pypy-15.so

Normally, it's enough to run "pypy setup.py install"

> 3) Are there known issues with cpyext and swig?

Some SWIG extension modules are known to work.
Normally it should compile and work normally,
except if you have custom typemaps and other C
code that uses the CPython API.

> 4) Of swig, Cython, hand-coded C, and ctypes, which is the better bet today
> for reliability with use from PyPy?

Cython won't work with pypy because it plays too much with the CPython
internals.

I don't have an opinion about it, but some of us consider ctypes as the best way
to develop extension modules. ctypes will eventually become much
faster with pypy,
with the JIT compiler emitting directly the machine code to call the function.
In any case, only a pure Python implementation can be sped up by the JIT.
A C function cannot be traced...

At the moment, cpyext is still considered as a (useful) experiment that helps
you leverage existing C extensions, but it will always be slow for different
reasons, the most obvious one is that the garbabe collector used by pypy
moves objects, whereas PyObject* pointers are expected to stay in place...

Also, there are probably other directions that we have not yet explored:
- implement another C API better suited to pypy
- a SWIG backend which emits ctypes code or similar
And probably others...

-- 
Amaury Forgeot d'Arc