Python's biggest compromises

[enoch]

aahz at pythoncraft.com (Aahz) wrote in message news:<bgr96h$a9$1 at panix3.panix.com>...
> In article <ad02da8c.0308060705.7ff1fa4f at posting.google.com>,
> enoch <enoch at gmx.net> wrote:
> >anthony_barker at hotmail.com (Anthony_Barker) wrote in message news:<899f842.0307310555.56134f71 at posting.google.com>...
> >> 
> >> What to you think python largest compromises are?
> >
> >Its non existant SMP scalability.
> 
> Would you care to back up your claim with some actual evidence?
> 
> (Yes, there are issues with Python on SMP machines, but to call Python's
> built-in threading "non-existent SMP scalability" is either a lie or
> revelatory of near-complete ignorance.  

Ok, I confess, the term you cited might be little bit exaggerated. But
there's no need to get personal. I'm surely not a liar (w.r.t. to this
thread, everything else is not a matter of public concern ;) ). The
ignorance part, well, we can talk about that ...

> That doesn't even count the various IPC mechanisms.)

Correct me if I'm wrong, but I don't think any form of IPC is a
measurement of scalability of something like the python interpreter.

Here are some sources which show that I'm not alone with my assessment
that python has deficiencies w.r.t. SMP systems:

http://www.python.org/pycon/papers/deferex/
"""
It is optimal, however, to avoid requiring threads for any part of a
framework. Threading has a significant cost, especially in Python. The
global interpreter lock destroys any performance benefit that
threading may yield on SMP systems, [...]
"""


http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=utf-8&safe=off&selm=bekg96%24mi%241%40panix2.panix.com
(note the author of that post)
"""
>My project will be running on an SMP box and requires scalability.
>However, my test shows that Python threading has very poor
performance
>in terms of  scaling. In fact it doesn't scale at all.

That's true for pure Python code.
"""

I'm aware that you know quite well about these facts, so I'll leave it
at that. But let me just add one more link which maybe you don't know:

http://www.zope.org/Members/glpb/solaris/multiproc 

"""
Well, in worst case, it can actually give you performance UNDER 1X. 
The latency switching the GIL between CPUs comes right off your
ability to do work in a quanta.  If you have a 1 gigahertz machine
capable of doing 12,000 pystones of work, and it takes 50 milliseconds
to switch the GIL(I dont know how long it takes, this is an example)
you would lose 5% of your peak performance for *EACH* GIL switch. 
Setting sys.setchechinterval(240) will still yield the GIL 50 times a
second.  If the GIL actually migrates only 10% of the time its
released, that would 50 * .1  * 5% = 25% performance loss.  The cost
to switch the GIL is going to vary, but will probably range between .1
and .9 time quantas (scheduler time intervals) and a typical time
quanta is 5 to 10ms.
[...]
However, I have directly observed a 30% penalty under MP constraints
when the sys.setcheckinterval value was too low (and there was too
much GIL thrashing).
"""

So, although python is capable of taking advantage of SMP systems
under certain circumstances (I/O bound systems etc. etc.), there are
real world situations where python's performance is _hurt_ by running
on a SMP system.
Btw. I think even IPC might not help you there, because the different
processes might bounce betweeen CPUs, so only processor binding might
help.



I did quite a bit of googling on this problem - several times -
because I'm selling zope solutions. Sometimes, the client wants to run
the solution on an existing SMP system, and worse, the system has to
fulfill some performance requirements. Then I have the problem of
explaining to him that his admins need to undertake some special tasks
in order for zope to be able to exploit the multiple procs in his
system.




Aazh, I'm lurking this newsgroup since approx. 3 years, so I know who
you are. You have participated in nearly any discussion about threads,
I know your slides, and there's no doubt that you have forgotten more
about this subject than I'll never know.