[Cython] State of PyPy compatibility wrt. arrays, incl. NumPy

[Stefan Behnel]

Hi,

Yury V. Zaytsev, 09.10.2013 15:01:
> I've been playing with my Cython-generated extension on PyPy, trying to
> load it through CPyExt, and, surprisingly, after a few fixes to PyPy, it
> generally seems to work.

Yes, we invested quite a bit of work into that already. Generally speaking,
my drive for supporting PyPy has substantially slowed down during the last
year or so. The interest on PyPy side in doing even the most obvious
optimisations was close to non-existant, which made it essentially
uninteresting to keep working on it on our side.


> However, it looks like the situation with whatever kind of arrays is
> actually pretty gloomy at the moment.
> 
> First, PyPy supports Python array.arrays, but doesn't replicate Python
> internals, so the extension even fails to load, since in PyPy
> tp_basicsize == 16 and not 56.
> 
> Not sure what to make of this one... for my needs, that would be enough.
> I guess PyPy developers have to be approached to see if it's practical
> to expose CPython structure, or agree on a different API?

The C-internals of the array.array module are private even in CPython, so
I'm not surprised that PyPy doesn't expose them. The normal buffer
interface would generally be enough, though.


> Second, they say that the new-style buffer access is broken and it's not
> clear if they will support it at all & when this is going to happen.

What do you mean by "broken"? Currently broken in PyPy? Would you have a
more complete quote? I couldn't find a discussion on the pypy mailing list
about this on a quick look.


> Third, apparently, there is already some limited support for NumPy C-API
> in NumPyPy, and it seems that there is interest in improving this
> support to make it usable:
> 
> http://docs.scipy.org/doc/numpy/reference/c-api.array.html

Interesting. Cython has legacy support for older NumPy versions through
their C-API instead of the buffer interface. We could enable that for PyPy.
Might need some minor adaptations, but might still be the easiest way to
get it working.


> Finally, GetItem on NumPyPy arrays, which will be very slow, but at
> least should work, rather than not, is also broken. I think this will be
> easiest of all to fix.

How is it broken? Where should it be fixed?


> So my question is, has anyone been playing with PyPy recently? Is there
> anyone interested in making Cython-generated code to work nicely with
> CPyExt? What are the plans w.r.t. the points I mentioned?

No plans, but if it's easy to fix, or if someone wants to invest the time
to work on it, it should be done, obviously.


> I've noticed that PyPy jobs in Jenkins are disabled... not sure whether
> this really means anything or not.

It means that we don't currently have a working installation of a recent
PyPy on our build server. The Linux builds they provide depend on fairly
recent system libraries and I consider building PyPy ourselves way too much
overhead. Also, running the tests took ages because CPyExt is so slow, so
the whole setup became useless after a while.

If you have a server lying around somewhere that would allow you to run
nightly tests of Cython against nightly builds of PyPy, I certainly
wouldn't mind if you set it up for it.

Stefan