Overcoming python performance penalty for multicore CPU

[Ryan Kelly]

On Sun, 2010-02-21 at 22:22 +0100, Martin v. Loewis wrote:
> John Nagle wrote:
> >    I know there's a performance penalty for running Python on a
> > multicore CPU, but how bad is it?  I've read the key paper
> > ("www.dabeaz.com/python/GIL.pdf"), of course.  It would be adequate
> > if the GIL just limited Python to running on one CPU at a time,
> > but it's worse than that; there's excessive overhead due to
> > a lame locking implementation.  Running CPU-bound multithreaded
> > code on a dual-core CPU runs HALF AS FAST as on a single-core
> > CPU, according to Beasley.
> 
> I couldn't reproduce these results on Linux. Not sure what "HALF AS
> FAST" is; I suppose it means "it runs TWICE AS LONG" - this is what I
> couldn't reproduce.
> 
> If I run Beazley's program on Linux 2.6.26, on a 4 processor Xeon (3GHz)
> machine, I get 30s for the sequential execution, 40s for the
> multi-threaded case, and 32s for the multi-threaded case when pinning
> the Python process to a single CPU (using taskset(1)).
> 
> So it's 6% overhead for threading, and 25% penalty for multicore CPUs -
> far from the 100% you seem to expect.

It's far from scientific, but I've seen behaviour that's close to a 100%
performance penalty on a dual-core linux system:

   http://www.rfk.id.au/blog/entry/a-gil-adventure-threading2

Short story:  a particular test suite of mine used to run in around 25
seconds, but a bit of ctypes magic to set thread affinity dropped the
running time to under 13 seconds.


  Cheers,

     Ryan

-- 
Ryan Kelly
http://www.rfk.id.au  |  This message is digitally signed. Please visit
ryan at rfk.id.au        |  http://www.rfk.id.au/ramblings/gpg/ for details

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/python-list/attachments/20100222/6205917e/attachment-0001.sig>