Future of Pypy?

[Laura Creighton]

In a message of Sun, 22 Feb 2015 12:45:03 +0000, Dave Farrance writes:

>Maybe there's not enough people like me that have really felt the need for
>the speed.  Or maybe it's simply the accident of the historical
>development path that's set-in-stone an interpreter rather than a JIT.
>Anybody got a useful perspective on this?

I don't understand 'an interpreter rather than a JIT'.  PyPy has a
JIT, that sort of is the whole point.

One problem is that hacking on PyPy itself is hard.  Lots of people
find it too hard, and give up.  (Of course, lots of people give up
on hacking CPython too.  I think that hacking on PyPy is harder than
hacking on CPython, but I am quite biased.)  So this is a barrier to
getting more people to work on it.

Provided your code is in pure python, PyPy already works great for you.
as you found out.  Pure Python libraries aren't a problem, either.

The problem arises when you want to add your fancy graphic library to
what you do, because chances are that fancy library is a C extension.
And there is no magic 'sprinke this dust here' on a C extension that
makes it acceptable to PyPy.  It's a hard job.  The PyPy team has
gone after, and rewritten how to do this I think 5 times now.  Maybe
more.  Every time the goal has been to make it easier for the average
programmer to make an interface, and, of course to not slow things down
too much.  C extensions, in general, make PyPy code run a lot slower because
you cannot get the JIT in there to speed things up, so you may be
stuck with unjitted PyPy performance, even on your python code,
which isn't speedy.  You also find lots of bugs in the C extensions,
which don't get noticed until you, for instance, no longer have a ref
counting GC.

Some of the things aren't really bugs, exactly, just that the person
who wrote the thing knew far, far, too much about how CPython works
and has produced something that had no need or desire to be portable
anywhere else.  The closer the person who wrote the extension
was 'to the metal' .. knowing _exactly_ how CPython does things and how
to squeeze that for the tiniest last drop of performance improvement ....
the harder things are for whoever wants to get it to work with PyPy, which
has a competely different architecture and a whole lot of other assumptions.

And the slower that it will run.  So, if it is a small thing, then the
usual suggestion is to rewrite it in pure python and let the JIT handle
it.  Very, very often the result is faster than the pure C code, but
clearly this isn't something you want to do with a huge graphics
library ...

There is hope that we can take another crack at this problem using
things learned from the Transactional Memory stuff, but nobody is promising
anything yet.  Also Armin has had a new neat thought.

https://mail.python.org/pipermail/pypy-dev/2015-February/013085.html

If we can get rid of the warmup time, then PyPy should be more popular
than it is now.  Lots of people run PyPy once, un-warmed up, see no
big improvement, and decide it's not for them.

But the problem of

'here is a huge chunk of C code, designed to work with Language X.
Now make it work with Language Y (PyPy) which isn't written in C'
can only be simplified to a certain extent.  There comes a point
where _this is just bloody hard_.

Laura