Will multithreading make python less popular?

[ma]

Very well written response! Thanks Sturla


On Tue, Feb 17, 2009 at 10:50 AM, sturlamolden <sturlamolden at yahoo.no>wrote:

> On 16 Feb, 10:34, rushen... at gmail.com wrote:
>
> > And the story begins here. As i search on the net,  I have found that
> > because of the natural characteristics of python such as GIL, we are
> > not able to write multi threaded programs. Oooops, in a kind of time
> > with lots of cpu cores and we are not able to write multi threaded
> > programs.
>
> The GIL does not prevent multithreaded programs. If it did, why does
> Python have a "threading" module?
>
> The GIL prevents one use of threads: parallel processing in plain
> Python. You can still do parallel processing using processes. Just
> import "multiprocessing" instead of "threading". The two modules have
> fairly similar APIs. You can still use threads to run tasks in the
> background.
>
> The GIL by the way, is an implementation detail. Nobody likes it very
> much. But it is used for making it easier to extend Python with C
> libraries (Python's raison d'etre). Not all C libraries are thread-
> safe. The GIL is also used to synchronize access to reference counts.
> In fact, Ruby is finally going to get a GIL as well. So it can't be
> that bad.
>
> As for parallel processing and multicore processors:
>
> 1. Even if a Python script can only exploit one core, we are always
> running more than one process on the computer. For some reason this
> obvious fact has to be repeated.
>
> 2. Parallel processing implies "need for speed". We have a 200x speed
> penalty form Python alone. The "need for speed" are better served by
> moving computational bottlenecks to C or Fortran. And in this case,
> the GIL does not prevent us from doing parallel processing. The GIL
> only affects the Python portion of the code.
>
> 3. Threads are not designed to be an abstraction for parallel
> processing. For this they are awkward, tedious, and error prone.
> Current threading APIs were designed for asynchronous tasks. Back in
> the 1990s when multithreading became popular, SMPs were luxury only
> few could afford, and multicore processors were unheard of.
>
> 4. The easy way to do parallel processing is not threads but OpenMP,
> MPI, or HPF. Threads are used internally by OpenMP and HPF, but those
> implementation details are taken care of by the compiler. Parallel
> computers have been used by scientists and engineers for three decades
> now, and threads have never been found a useful abstraction for manual
> coding. Unfortunately, this knowledge has not been passed on from
> physicists and engineers to the majority of computer programmers.
> Today, there is a whole generation of misguided computer scientists
> thinking that threads must be the way to use the new multicore
> processors. Take a lesson from those actually experienced with
> parallel computers and learn OpenMP!
>
> 5. If you still insist on parallel processing with Python threads --
> ignoring what you can do with multiprocessing and native C/Fortran
> extensions -- you can still do that as well. Just compile your script
> with Cython or Pyrex and release the GIL manually. The drawback is
> that you cannot touch any Python objects (only C objects) in your GIL-
> free blocks. But after all, the GIL is used to synchronize reference
> counting, so you would have to synchronize access to the Python
> objects anyway.
>
>
> import threading
>
> def _threadproc():
>     with nogil:
>        # we do not hold the GIL here
>        pass
>     # now we have got the GIL back
>     return
>
> def foobar():
>    t = threading.Thread(target=_threadproc)
>    t.start()
>    t.join()
>
> That's it.
>
>
>
>
>
> Sturla Molden
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20090217/f4bb0a12/attachment-0001.html>