Python's biggest compromises

[Andrew Dalke]

Aahz:
> Would you care to back up your claim with some actual evidence?

Aaaz?  What do you define as SMP scalability?

For me it's that when my program runs on an SMP machine then
I should expect non-trivial performance advantages.  (Excepting
the effects of othe programs on the system.)

This can mean many things.  For example, a FORTRAN compiler
might parallelize array operations even though to the programmer
the FORTAN code is not threaded.  Still, for most people the
idea of "SMP scalability" is the ability to have separate threads
run at the same time on different CPUs.

Which doesn't work well for Python if all the threads are busy
running native code, due to the GIL.

If the threads are usually blocked on I/O then the GIL is
released.  Your own essay on a multithreaded spider shows
you know this well.  And for this case Python does have good
SMP support, so enoch's complaint about its "non existant" is
overstated, but still true in his context of writing a large server
app.

> (Yes, there are issues with Python on SMP machines, but to call Python's
> built-in threading "non-existent SMP scalability" is either a lie or
> revelatory of near-complete ignorance.  That doesn't even count the
> various IPC mechanisms.)

Given that it is a problem (and one I've had to deal with), calling
it an "issue" seems too weak, while dissing enoch this way is way
too strong.

There are IPC mechanisms, but if that's your definitions of SMP
scalability then any language with socket support (or bindings
for PVM/MPI/CORBA/Linda/...) is SMP scalable, which kinda
ruins the usefulness of that phrase.

I would say Python's SMP scalability is poor for Python-bound
threads.  Better would be a language where I can share objects
between free-running SMP threads, as with Greg Stein's
experimental patches years ago.  Not-as-good are ones which
let me pass data through shared memory, as POSH, and worse
are those which assume pickles always work (I use some
extensions with unpicklable objects).  Even worse (for my
development style) are those that require working through some
IDL.

I also am developing a heavy web app and want scalability
on SMP machines, and I also bemoan Python's poor threading
applicability for my case.  I'll work around it with some IPC
mechanism, but it is a workaround.

(OTOH, that means bugs in the extensions won't crash the
server.)

                    Andrew
                    dalke at dalkescientific.com