[pypy-dev] Benchmarking PyPy performance on real-world Django app

[Michael Foord]

On 7 October 2011 18:08, Maciej Fijalkowski <fijall at gmail.com> wrote:

> On Fri, Oct 7, 2011 at 7:04 PM, Michael Foord <fuzzyman at gmail.com> wrote:
> >
> >
> > On 7 October 2011 17:50, Maciej Fijalkowski <fijall at gmail.com> wrote:
> >>
> >> On Fri, Oct 7, 2011 at 1:33 PM, Igor Katson <igor.katson at gmail.com>
> wrote:
> >> > When I first started benchmarking one of my Django sites,
> >> > http://trip-travel.ru/ (using postgres driver pypq),
> >> > I was disappointed by the results. PyPy was visually much slower than
> >> > cPython (I just looked at how the site loads in the browser)
> >> >
> >> > But today I got some incredible results, I finally made PyPy to work
> >> > faster
> >> > than cPython, and found out that it got faster after loading the page
> >> > several hundred times with "ab" or "siege"
> >> >
> >> > Here a the results, of mean response time by querying the home page
> with
> >> > apache's "ab" (cPython 2.7.2, Django 1.3, PyPy 1.6.0), served with
> >> > cherrypy
> >> > wsgi server:
> >> >
> >> > After 10 requests (excluding the first request):
> >> > cPython - 163.529 ms
> >> > PyPy - 460.879 ms
> >> >
> >> > 50 request more:
> >> > cPython - 168.539
> >> > PyPy - 249.850
> >> >
> >> > 100 requests more:
> >> > cPython - 166.278 ms
> >> > PyPy - 131.104
> >> >
> >> > 100 requests more:
> >> > cPython - 165.820
> >> > PyPy - 115.446
> >> >
> >> > 300 requests more:
> >> > cPython - 165.543
> >> > PyPy - 107.636
> >> >
> >> > 300 requests more:
> >> > cPython - 166.425
> >> > PyPy - 103.065
> >>
> >> Thanks for doing the benchmarks :)
> >>
> >> >
> >> > As we can see, the JIT needs much time to warm up, but when it does,
> the
> >> > result is pretty noticeable.
> >> > By the way, with psycopg2, the site responds for 155.766 ms in average
> >> > (only
> >> > 10 ms faster), so using PyPy with Django makes much sense for me.
> >> >
> >> > As for now, pypy cannot run with uWSGI, which I use in production, but
> >> > maybe
> >> > i'll switch to PyPy for production deployments if "PyPy + PyPQ + Some
> >> > pure
> >> > python WSGI server" suite will outperform (uWSGI + cPython +
> psycopg2).
> >> > Though, the need to load the page 500 times after each server reload
> is
> >> > not
> >> > comfortable.
> >>
> >> I've heard people using gunicorn. Maybe this is a good try? Loading
> >> the pages is indeed annoying, but you need to load it a couple times
> >> for results not to be noticably slower :) We kind of know that the JIT
> >> warmup time is high, but it's partly a thing to fix and partly an
> >> inherent property of the JIT.
> >>
> >
> >
> > FWIW I shared this internally at Canonical, and whilst people were
> impressed
> > there was some concern that having substantially worse performance for
> the
> > first few hundred requests would a) be a showstopper and b) screw up
> > metrics. The technique of deliberate warming immediately after restart is
> > interesting, but it would be hard to hit all the code paths.
> >
> > I realise this is inherent in the way the jit works - but I thought it
> was a
> > response worth sharing. I also assured them that pre-warm-up performance
> > would continue to improve as pypy improves. ;-)
> >
> > All the best,
> >
> > Michael Foord
> >
>
> It's also true for anything based on JVM and I've *never* seen anyone
> complain about it. The whole concept of a JIT is just "new" to python
> world. Seriously, would actually anyone notice if first 50 requests
> after restart take extra 200ms??? I'm not sure, my computer and my
> internet connection both have hiccups of over 200ms. Note that many
> code paths are common, which means that speed up is also shared.
>

I think you have a point and I've added your response to the internal
discussion.

Michael

-- 

http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20111007/5711599c/attachment.html>