[pypy-dev] PyPy and production use

[Martijn Faassen]

Hey,

I'm changing the title of this thread as this is another topic. It's one 
I care about as an observer of PyPy.

Antonio Cuni wrote:
[snip]
> Finally, a note which is not related with the work plan but to a 
> question that appeared in the logs; someone asked whether we want to 
> "sell" RPython as a stand-alone product.

I think you should. I'll explain why below.

> Last sunday I gave a PyPy talk at Pycon italy and, surprisingly enough, 
> RPython was perceived as the only (or the most relevant) usable product 
> of the pypy project. I tried hardly to explain that rpython is basically 
> only an accident and not very usable, but people are still impressed by 
> the 300000% speedup.
> It would be interesting to know if this perception is world-wide or only 
> limited to the attendants of my talk (maybe because I didn't stress 
> enough the pros of the other pypy goals or the cons of rpython).

I got this impression back at the presentation last year at EuroPython. 
I therefore think this impression is more widespread than just your 
audience in Italy. This is not because you've communicated the state or 
future potential of PyPy wrongly, but because the audience likely has a 
different perspective than you do.

I realize that PyPy is much more than RPython and that RPython is 
somewhat incidental to the project. For people working on production 
projects right now, these other features are less relevant at this point 
in time, however. Let's take PyPy's generated interpreter. If I'm 
currently using the CPython platform (as most Python programmers are), 
choosing the PyPy-C interpreter would mean that I have far less 
extension module support, and a slightly slower interpreter. I'd need 
some pretty unusual use cases currently for the benefits of PyPy-C to 
outweigh its drawbacks.

The PyPy developers themselves of course *do* have unusual use cases. 
They want to experiment with interpreter generation, different language 
features, different platforms, and so on.

I expect this to change over time. Once the PyPy (generated) JIT 
matures, PyPy will become a lot more attractive. If you could then 
somehow figure out a way to use CPython modules with PyPy's interpreter 
you'd get even more people. Right now, unless I really really need some 
unusual language feature in my production project, I don't have a 
motivation to use PyPy's interpreter.

Maturing the PyPy interpreter will take quite a lot of time. You've come 
a long way and it's quite impressive, but at the same time I don't 
expect to be using a PyPy-generated interpreter to develop web 
applications any time soon.

Now RPython. RPython generated modules are usable from CPython. This 
means I can write relatively plain Python code, generate a module, and 
start using this module in my web application today. The benefit is 
vastly increased speed for my module.

Using RPython with CPython is attractive for the same reasons Pyrex is 
attractive, with the benefits that it's more well-engineered than Pyrex. 
Pyrex is currently more mature to use. Using RPython in this way is a 
bit of a struggle still in comparison. This is however tool support that 
I feel can improve relatively quickly.

An open source projects needs mindshare. It needs users. PyPy is an open 
source project, and I'd recommend getting any users you can get. 
Granted, you'll get opportunists like me who are only there for 
production benefits. These people are less interested in language 
experimentation and new features - they're there for the speed 
primarily: the main thing that PyPy, through RPython, can offer to a 
Python programmer today. But if your aim is to provide a platform that 
is suitable for production use, you *need* opportunists like that. They 
can help mature the platform. Open source projects thrive by attracting 
developers that are in it for their own (enlightened) self-interest.

Some of these opportunists might actually help out beyond just RPython. 
They might in fact be interested in such language experimentation. It 
might however not be a priority for them. For instance, I'm interested, 
but I need an excuse to spend more time on PyPy. Production benefits are 
a great excuse. Using RPython extensions with CPython is the shortest 
route to production benefits.

Besides gaining new users, supporting RPython as a product will help 
PyPy in other ways as well. It will get you more module support. New 
users will come in and help you write more extension modules in RPython. 
They'll be doing this initially so they can use them in CPython, but 
it's a nice side effect.

Currently I'm working with Godefroid Chapelle (well, he's been doing the 
most work so far) to try to make an RPython version of a web templating 
engine (Zope's page template engine). We hope we will be able to use 
this technology to speed up the generation of web templates. We'll be 
using this within a CPython server.

We see the potential of PyPy. Godefroid made the observation that a web 
templating engine implements an interpreter. PyPy has technology to 
generate a JIT for arbitrary interpreters in RPython. This means that in 
the future we might be able to have a JIT-ed templating language, 
hopefully boosting performance even more. (besides the huge coolness factor)

Just the potential for future JITting isn't motivating enough to 
experiment with PyPy right now though. The ability to use RPython with 
CPython is the critical bit. This is what gives us the hope we'll get a 
payoff in our production work. The JIT is a nice bonus that might become 
important later.

So if people conclude that RPython is currently the most relevant 
product of the PyPy project for them, it doesn't mean that they haven't 
understood the potential of PyPy in the future. It's because they're 
thinking about how PyPy is relevant to their own work today.

PyPy has always been a unique project: it's a research project and it's 
an open source project aiming for real-world use. It takes ideas from 
ivory towers and applies them to a pragmatic, real-world language like 
Python. I think this balance between pragmatic and research goals is one 
of this project's greatest strengths. I encourage PyPy developers to 
continue to embrace both goals.

Regards,

Martijn