Using Python for processing of large datasets (convincing managment)

[Alex Martelli]

Thomas Jensen wrote:
        ...
> I've heard that Python threads don't scale (well?) to multiple CPUs ?
> Maybe that's only on Windows?

No, it's true on all platforms: Python threads only use one CPU.  There
is a global interpreter lock (GIL) to ensure only one Python bytecode
is being executed at any given time, except for those which call out to
C-coded parts (e.g. for blocking I/O) which explicitly release the GIL.

With Python, you can exploit multiple CPUs only by multi-*processing* --
and here, it's possible that Windows' multi-processing inefficiencies
may byte you (with Unix-like systems, often multiple processes or
multiple threads in one process have quite comparable performance).


Alex