Threading In Python

[Chris Angelico]

On Thu, Dec 12, 2013 at 7:08 PM, marcinmltd <marcinmltd at gmail.com> wrote:
> I'm big fan of multiprocessing module, but recently I started looking at
> threading in Python more closely and got couple of questions I hope You can
> help me with:

The first thing to note is that there are two completely separate concepts here:

1) Python is a language. It is defined by a set of specifications
covering syntax, semantics, etc, etc, etc.

2) CPython, Jython, IronPython, and PyPy are four implementations of
the Python language. (There are others, too, but those are the four
biggest.) They're all a bit different in how they behave.

Most likely, you're talking about CPython, which is the most
commonly-used implementation of Python. I'm going to discuss CPython
here, but be aware that the other Pythons may well (and in some cases
definitely DO) behave differently.

> 1. When I run two or more threads in my python process are they really run
> concurrently on mulicore machine?
>
> 2. Browsing through documentation it looks like python interpreter protects
> its sensitive states by using GIL. Can you guys list situations when this
> happens?

I'll answer these two together. Most likely the two threads will *not*
run concurrently, because CPython uses the Global Interpreter Lock
around all Python code execution. The GIL is released any time there's
going to be a long-running C-level call, especially something that
waits; so if your threads are spending most of their time, say, trying
to read from sockets, then there's no problem and the GIL won't get in
your way. There are a very VERY few CPU-bound operations that release
the GIL (some numpy calculations, I think, but I'm not familiar with
that library), but the general rule of thumb is that whenever the
CPU's busy, you own the GIL.

> 2. What would be general advice from python experts on when to use threadng
> and when switch to multliprocessing in python? Is the decision still
> influenced by how often we need to comunicate between the tasks as it's in
> C\C++?

Use threading when you're waiting for something else, and
multiprocessing when you need to exercise multiple CPU cores. Again,
that's just a rule of thumb, but it's close enough for a base
understanding of the difference.

Since I don't have real-world experience with any Python other than
CPython, I won't speak to those; but do expect that there'll be
differences. (I think, for instance, Jython's threading module is
backed by Java's threads, with all the advantages and disadvantages
that entails.) But the beauty of this mailing list is that someone
else will fill that in :)

ChrisA