Overcoming python performance penalty for multicore CPU

[Ryan Kelly]

On Sun, 2010-02-21 at 23:05 +0100, Martin v. Loewis wrote: 
> > It's far from scientific, but I've seen behaviour that's close to a 100%
> > performance penalty on a dual-core linux system:
> > 
> >    http://www.rfk.id.au/blog/entry/a-gil-adventure-threading2
> > 
> > Short story:  a particular test suite of mine used to run in around 25
> > seconds, but a bit of ctypes magic to set thread affinity dropped the
> > running time to under 13 seconds.
> 
> Indeed, it's not scientific - but with a few more details, you could
> improve it quite a lot: what specific Linux distribution (the posting
> doesn't even say it's Linux), what specific Python version had you been
> using? (less important) what CPUs? If you can: what specific test suite?

I'm on Ubuntu Karmic, Python 2.6.4, an AMD Athlon 7750 dual core.

Unfortunately the test suite is for a proprietary application.  I've
been able to reproduce similar behaviour with an open-source test suite,
using the current trunk of the "pyfilesystem" project:

  http://code.google.com/p/pyfilesystem/


In this project "OSFS" is an object-oriented interface to the local
filesystem.  The test case "TestOSFS.test_cases_in_separate_dirs" runs
three theads, each doing a bunch of IO in a different directory.

Running the tests normally:

    rfk at durian:/storage/software/fs$ nosetests fs/tests/test_fs.py:TestOSFS.test_cases_in_separate_dirs
    .
    ----------------------------------------------------------------------
    Ran 1 test in 9.787s


That's the best result from five runs - I saw it go as high as 12
seconds.  Watching it in top, I see CPU usage at around 150%.

Now using threading2 to set the process cpu affinity at the start of the
test run:

    rfk at durian:/storage/software/fs$ nosetests fs/tests/test_fs.py:TestOSFS.test_cases_in_separate_dirs
    .
    ----------------------------------------------------------------------
    Ran 1 test in 3.792s


Again, best of five.  The variability in times here is much lower - I
never saw it go above 4 seconds.  CPU usage is consistently 100%.



  Cheers,

     Ryan


-- 
Ryan Kelly
http://www.rfk.id.au  |  This message is digitally signed. Please visit
ryan at rfk.id.au        |  http://www.rfk.id.au/ramblings/gpg/ for details


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/python-list/attachments/20100222/a3be32fd/attachment-0001.sig>