Multi-threading python
Michael Chermside
mcherm at destiny.com
Wed Sep 11 09:45:10 EDT 2002
David Brown wrote:
> I've just been reading a little about the GIL and threading in python, and I
> have a couple of questions that I hope someone can answer.
I'm not an expert, but I'll try.
> Am I right in thinking that threading in Python does not use the underlying
> OS's threads, but effectively implements its own scheduler? Or does it use
> the OS's threads, but uses the GIL to ensure that only one is runable at a
> time?
It uses the OS's threads, and then uses the GIL to ensure that only one
is executing Python bytecodes at a time. However, other threads may be
runable... they could be blocked waiting on I/O, or executing some C
code from a module that releases the GIL before performing a
long-running computation.
> On a dual-processor system, does this mean that only one thread of a Python
> program can run at a time, even when both CPUs are free, so that two
> seperate python interpreter processes are needed to take advantage of the
> two CPUs?
Yes.
> What about when a Python thread is blocked for some other reason, such as
> while waiting for a file read - will that block other threads, or will the
> waiting thread definitely release the GIL and allow other threads to run?
Depends on what it's blocked for. If it's blocked for I/O, then the GIL
is released and the other threads will run. I'm not sure what else it
would block for that might be a problem. But you could certainly CREATE
a problem if you tried... simplest way would be to write C code which
ran for a long time WITHOUT releasing the GIL.
It would be much more difficult to block things through badly written
Python code, since after each bytecode the interpretor can switch to a
different thread.
> If the GIL does block all threads in this manner, are there any plans to
> introduce more fine-grained locking to improve scalability, analagous to
> locking in the Linux kernel?
No. The general consensus is that eliminating the GIL would be quite
difficult and that there's not really all that much to gain. (Most
casual users don't have multi-processor systems; serious users of this
sort probably use Numeric and/or put the tight loops into C anyhow.)
However, if you know someone who's good with this stuff and wants to
volunteer, I'm sure they wouldn't turn down working code (unless it
introduced other problems).
-- Michael Chermside
More information about the Python-list
mailing list