Can python threads take advantage of use dual core ?

[sturlamolden]

On Aug 17, 6:00 pm, nikhilketkar <nikhilket... at gmail.com> wrote:


> What are the implications of the Global Interpreter Lock in Python ?

This is asked every second week or so.

The GIL is similar to the Big Kernel Lock (BKL) use to support SMPs in
older versions of the Linux kernel. The GIL prevents the Python
interpreter to be active in more than one thread at a time.

If you have a single CPU with a single core, the GIL has no
consequence what so ever.

If you have multiple CPUs and/or multiple cores, the GIL has the
consequence that you can forget about SMP scalability using Python
threads. SMP scalability require more fine grained object locking,
which is what the Java and .NET runtimes do, as well as current
versions of the Linux kernel. Note that IronPython and Jython use fine
grained locking instead of a GIL.

> Does this mean that Python threads cannot exploit a dual core
> processor and the only advantage of using threads is in that
> computation and IO-bound operations can continue in parallel ?


The Python standard library releases the GIL in read/write operations
for e.g. files and sockets. This can be combined with threads to allow
multiple IO operations to continue in parallel. The GIL has no or very
little significance for IO-bound scalability in Python.

CPU-bound tasks are a different game. This is where the GIL matter.
You must either release the GIL or use multiple processes for
exploiting multiple processors in an SMP computer with Python. The GIL
can be released explicitely in C extension code. f2py and ctypes can
also release the GIL.

Note that you would NOT use threads for speeding up CPU-bound
operations, even when programming in C or Fortran. Threads are neither
the only nor the preferred way to exploit multicore CPUs for CPU-bound
tasks. Instead of threads, use either an MPI library or OpenMP
compiler pragmas. You can use MPI directly from Python (e.g. mpi4py),
or you can use OpenMP pragmas in C or Fortran code which you call
using ctypes or f2py.

Summary:

Use Python threads if you need to run IO operations in parallel.
Do not use Python threads if you need to run computations in parallel.



Regards,
Sturla Molden