multithreading concept

[sturlamolden]

On Feb 7, 2:53 am, "S.Mohideen" <m... at blackhole.labs.rootshell.ws>
wrote:

> Python is praised about - me too. But at one instance it fails. It fails to
> behave as a true multi-threaded application. That means utilizing all the
> CPUs parallely in the SMP efficiently stays as a dream for a Python
> Programmer.

This has been discussed to death before. Win32 threads and pthreads
(which is what Python normally uses, depending on the platform) are
designed to stay idle most of the time. They are therefore not a tool
for utilizing the power of multiple CPUs, but rather make certain kind
of programming tasks easier to program (i.e. non-blocking I/O,
responsive UIs). The GIL is not a problem in this context. If threads
stay idle most of the time, the GIL does not harm.

If you want to utilize the computing power of multiple CPUs, you
should use multiple processes instead of threads. On Python this is
mandatory due to the GIL. In any other language it it highly
recommended. The de-factor standard for parallel multiprocessing (MPI)
uses multiple processes, even on SMPs. Anyone with serious intentions
of using multiple processors for parallel computing should use
multiple processes and fast IPC - not multiple threads, shared memory
and synchronization objects - even if the language is plain C. With
multiple threads, a lot of time is wasted doing context switches and
book keeping for the  thread synchronization. In addition, obscure and
often very difficult to identify bugs are introduced.

There are a Python binding for MPI (mpi4py) and a similar pure Python
project (Parallel Python) that will take care of all these details for
you.


> Discussion threads say its due to GIL - global interpreter lock. But nobody
> has mentioned any alternative to that apart from suggestions like "Code it
> in C" and POSH (http://poshmodule.sf.net). Is there any other way we can
> make Python programs really multithreaded in real sense.

As mentioned, use MPI or Parallel Python. MPI is by far the more
mature, but Parallel Python could be easier for a pythoneer.
Multithreading has different use.