[pypy-issue] [issue1298] performance w/ sqlalchemy test suite

[mike bayer]

New submission from mike bayer <mike_mp at zzzcomputing.com>:

per this HN thread: https://news.ycombinator.com/item?id=4648758 Fijal asked
that I report over here the slowness I'm seeing when running SQLAlchemy's very
large test suite with pypy.   Our tests are normally run on Amazon EC2 small
instances, where SQLA's suite takes about 2 hours 17 minutes to run with pypy +
sqlite - the equivalent suite on cPython takes 17 minutes on that platform, and
runs many more tests that are already skipped on the pypy platform (such as
profiling tests).   

Running a vanilla set of SQLA tests is pretty easy, dependencies are only sqlite
+ nose.    From a checkout here:

    hg clone http://hg.sqlalchemy.org/sqlalchemy

the full set of 4000+ tests can be run like:

    python ./sqla_nose.py -v 

a smaller segment of the tests that can be compared more expediently can be run
like this:

    python ./sqla_nose.py -v -w sql

Results I have running these on both my MacBook as well as on the EC2 instance:

    all tests, OSX:    760 sec (Pypy) vs 211 sec (cPython)  3.6x slower

    -w sql tests, OSX:  24 sec (Pypy) vs 4.5 sec (cPython)  5.3x slower

    EC2 small, all tests:   2 hr 17 minutes (Pypy) vs 17 minutes (cPython)  8x
slower

on my end I can only make fairly ignorant guesses as to why this is, the very
large size of the SQLAlchemy test suite and the large number of new codepaths it
produces I'd imagine gives pypy a lot of work in running the JIT, but other than
that I don't have much insight.  Memory usage remains stable with Pypy, it does
use 99% CPU all the way through as much as it can, however.

----------
messages: 4884
nosy: pypy-issue, zzzeek
priority: performance bug
status: unread
title: performance w/ sqlalchemy test suite

________________________________________
PyPy bug tracker <tracker at bugs.pypy.org>
<https://bugs.pypy.org/issue1298>
________________________________________