Please help with Threading

[Chris Angelico]

On Sun, May 19, 2013 at 10:02 AM, Carlos Nepomuceno
<carlosnepomuceno at outlook.com> wrote:
> I didn't know Python threads aren't preemptive. Seems to be something really old considering the state of the art on parallel execution on multi-cores.
>
> What's the catch on making Python threads preemptive? Are there any ongoing projects to make that?

Preemption isn't really the issue here. On the C level, preemptive vs
cooperative usually means the difference between a stalled thread
locking everyone else out and not doing so. Preemption is done at a
lower level than user code (eg the operating system or the CPU),
meaning that user code can't retain control of the CPU.

With interpreted code eg in CPython, it's easy to implement preemption
in the interpreter. I don't know how it's actually done, but one easy
implementation would be "every N bytecode instructions, context
switch". It's still done at a lower level than user code (N bytecode
instructions might all actually be a single tight loop that the
programmer didn't realize was infinite), but it's not at the OS level.

But none of that has anything to do with multiple core usage. The
problem there is that shared data structures need to be accessed
simultaneously, and in CPython, there's a Global Interpreter Lock to
simplify that; but the consequence of the GIL is that no two threads
can simultaneously execute user-level code. There have been
GIL-removal proposals at various times, but the fact remains that a
global lock makes a huge amount of sense and gives pretty good
performance across the board. There's always multiprocessing when you
need multiple CPU-bound threads; it's an explicit way to separate the
shared data (what gets transferred) from local (what doesn't).

ChrisA