[pypy-dev] Pypy Benchmark of Tornado.
Maciej Fijalkowski
fijall at gmail.com
Fri Sep 5 19:01:59 CEST 2014
Hi
Can you please put it all in a blog post (ideally with examples), it's
a lot of useful info.
I'm willing to help
On Fri, Sep 5, 2014 at 10:47 AM, Jolitz Ben - bjolit
<Ben.Jolitz at acxiom.com> wrote:
> I don’t have specific suggestions on Mongo, but I can share what I’ve
> learned in a few months of using PyPy and Tornado.
>
> You want to make use of CFFI in PyPy to accelerate operations that would
> usually be slow in Python, namely encryption and database drivers. But
> always test first to see if you really need to go to C.
>
> /Any/ CPython C-Extensions will torpedo performance. Anything blocking (like
> the majority of DB drivers) will similarly destroy Tornado performance.
>
> Code that is overly dynamic also does lousy on PyPy. If your driver has a
> ton of paths or makes idiot use of threading.Lock, expect to have an uphill
> struggle in optimization.
>
> When in doubt, ask yourself if the algorithm is appropriate.
>
> If you can’t make the Python driver performant and there exists a C-API for
> it, then it is trivial to wrap it with CFFI.
>
> If it doesn’t support nonblocking operations, you can find alternatives. For
> example in MySQLdb, others have found you can add the Connection._fd to the
> IOLoop and use it to do a send_query, read_query. If you still can’t find an
> alternative to a blocking call, you can still mimic nonblocking IO by using
> pthreads, a work queue and a callback pthread. It’s not perfect and theres a
> lot you can optimize, but it can easily allow you to delegate long running
> operations to C.
>
> Another thing I learned was to avoid generating Python-side c callback
> pointers frequently. If C is going to callback into Python with your result
> to a unique Request, you’re going to need to tag it appropriately. I prefer
> to use dictionaries and the callback attribute handed to me by gen.Task and
> pass a unique-enough key to C to callback in with.
>
> As with everything in optimization, profile your code first. If you’re
> losing speed heavily somewhere else, then the above will only serve to
> distract you.
>
> Cheers,
>
> Ben
>
>
> From: Phyo Arkar <phyo.arkarlwin at gmail.com>
> Date: Thursday, September 4, 2014 at 1:55 PM
> To: Ben Jolitz <Ben.Jolitz at acxiom.com>
> Cc: pypy-dev <pypy-dev at python.org>
>
> Subject: Re: [pypy-dev] Pypy Benchmark of Tornado.
>
> Thanks alot Ben,
> Ok , as PyPy is choice of quora and they also use tornado , i might keep
> testing on larget projects.
> How about mongodb performance on PyPy , i heard its slower due to no C
> Extension (no CFFI) for pypy.
> Your suggestion will be very appreciated.
>
>
> On Thu, Sep 4, 2014 at 4:56 AM, Jolitz Ben - bjolit <Ben.Jolitz at acxiom.com>
> wrote:
>>
>> I use Tornado and have found PyPy can yield a 30-50% performance increase
>> for a moderately complex project.
>>
>> Ben
>>
>> From: Phyo Arkar <phyo.arkarlwin at gmail.com>
>> Date: Wednesday, September 3, 2014 at 8:21 AM
>> To: pypy-dev <pypy-dev at python.org>
>> Subject: Re: [pypy-dev] Pypy Benchmark of Tornado.
>>
>> I expect pypy to be faster in those cases but select io is not cpu
>> intensive thing to do so no real benefit using pypy here i guess.
>> On Wed, Sep 3, 2014 at 4:43 PM, Phyo Arkar <phyo.arkarlwin at gmail.com>
>> wrote:
>> >
>> > It just return json document with a few thousand characters (1053 bytes)
>> > $siege -c 400 -t 20s -r 2000 http://localhost:9999/js
>> >
>> > Python 2.7.7:
>> >
>> > Lifting the server siege... done.
>> >
>> > Transactions: 14478 hits
>> > Availability: 100.00 %
>> > Elapsed time: 19.10 secs
>> > Data transferred: 14.54 MB
>> > Response time: 0.01 secs
>> > Transaction rate: 758.01 trans/sec
>> > Throughput: 0.76 MB/sec
>> > Concurrency: 8.91
>> > Successful transactions: 14478
>> > Failed transactions: 0
>> > Longest transaction: 1.08 seconds
>> > Shortest transaction: 0.00
>> >
>> > pypy-2.3.1 stable:
>> >
>> > Transactions: 15149 hits
>> > Availability: 100.00 %
>> > Elapsed time: 19.63 secs
>> > Data transferred: 15.21 MB
>> > Response time: 0.02 secs
>> > Transaction rate: 771.73 trans/sec
>> > Throughput: 0.77 MB/sec
>> > Concurrency: 11.92
>> > Successful transactions: 15149
>> > Failed transactions: 0
>> > Longest transaction: 1.09 seconds
>> > Shortest transaction: 0.00
>> >
>> >
>> >
>> > pypy--c-jit-73283-912dd9df99a8-linux64 (latest nightly build)
>> >
>> > Lifting the server siege... done.
>> >
>> > Transactions: 14361 hits
>> > Availability: 100.00 %
>> > Elapsed time: 19.13 secs
>> > Data transferred: 14.42 MB
>> > Response time: 0.03 secs
>> > Transaction rate: 750.71 trans/sec
>> > Throughput: 0.75 MB/sec
>> > Concurrency: 21.53
>> > Successful transactions: 14361
>> > Failed transactions: 0
>> > Longest transaction: 3.03 seconds
>> > Shortest transaction: 0.00
>> >
>> >
>> >
>> >
>> > Pypy Nightly have some request Randomly get to 3.0 Seconds , normally
>> > those requests (in Cpython) are only ~0.001 to 0.002 sec.
>> >
>>
>> It just return json document with a few thousand characters (1053 bytes)
>> $siege -c 400 -t 20s -r 2000 http://localhost:9999/js
>>
>> Python 2.7.7:
>>
>> Lifting the server siege... done.
>>
>> Transactions: 14478 hits
>> Availability: 100.00 %
>> Elapsed time: 19.10 secs
>> Data transferred: 14.54 MB
>> Response time: 0.01 secs
>> Transaction rate: 758.01 trans/sec
>> Throughput: 0.76 MB/sec
>> Concurrency: 8.91
>> Successful transactions: 14478
>> Failed transactions: 0
>> Longest transaction: 1.08 seconds
>> Shortest transaction: 0.00
>>
>> pypy-2.3.1 stable:
>>
>> Transactions: 15149 hits
>> Availability: 100.00 %
>> Elapsed time: 19.63 secs
>> Data transferred: 15.21 MB
>> Response time: 0.02 secs
>> Transaction rate: 771.73 trans/sec
>> Throughput: 0.77 MB/sec
>> Concurrency: 11.92
>> Successful transactions: 15149
>> Failed transactions: 0
>> Longest transaction: 1.09 seconds
>> Shortest transaction: 0.00
>>
>>
>>
>> pypy--c-jit-73283-912dd9df99a8-linux64 (latest nightly build)
>>
>> Lifting the server siege... done.
>>
>> Transactions: 14361 hits
>> Availability: 100.00 %
>> Elapsed time: 19.13 secs
>> Data transferred: 14.42 MB
>> Response time: 0.03 secs
>> Transaction rate: 750.71 trans/sec
>> Throughput: 0.75 MB/sec
>> Concurrency: 21.53
>> Successful transactions: 14361
>> Failed transactions: 0
>> Longest transaction: 3.03 seconds
>> Shortest transaction: 0.00
>>
>>
>>
>>
>> Pypy Nightly have some request Randomly get to 3.0 Seconds , normally
>> those requests (in Cpython) are only ~0.001 to 0.002 sec.
>>
>>
>> ***************************************************************************
>> The information contained in this communication is confidential, is
>> intended only for the use of the recipient named above, and may be legally
>> privileged.
>>
>> If the reader of this message is not the intended recipient, you are
>> hereby notified that any dissemination, distribution or copying of this
>> communication is strictly prohibited.
>>
>> If you have received this communication in error, please resend this
>> communication to the sender and delete the original message or any copy
>> of it from your computer system.
>>
>> Thank You.
>>
>> ****************************************************************************
>>
>>
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
>>
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
More information about the pypy-dev
mailing list