thread/queue bug

[Tim Peters]

[Peter Otten]
> What I believe to be a minimal example:
>
> <freeze.py>
> import Queue
> import threading
> import time
> 
> q = Queue.Queue(4)
> 
> def proc():
>    while True:
>        q.get(1)
>        Queue.Queue()
>        print "YADDA"
> 
> threading.Thread(target=proc).start()
> 
> while True:
>    print "yadda"
>    q.put(None)
>    time.sleep(1)
> </freeze.py>
> 
> <freezemain.py>
> import freeze
> </freezemain.py>

CPython has an internal, reentrant import lock.  When a thread does an
import, it acquires this lock, and the lock remains held until that
import is complete.  As a consequence, no *other* thread can do an
import (it blocks waiting to obtain the internal import lock) until
the original import completes.  So until "import freeze" returns in
the main thread, no other thread can do an import.

Partly for that reason, it's generally a Horrible Idea to start a
thread as a side effect of importing a module.  That's what freeze.py
does, and you get the expected deadlock as a result.  The main thread
is hung waiting for "import freeze" to return, and the spawned thread
is hung at an import in Queue.__init__() waiting for the main thread
to release the import lock.

> Invoking freezemain.py produces
>
> yadda
> yadda
> yadda
> yadda
> yadda
> yadda
> 
> i. e. consistently q.maxsize + 2. One item about to be put, one
> already taken before Queue.Queue(). Deferring execution of the
> module-level code until after the import
>
> <nofreeze.py>
> import Queue
> import threading
> import time
> 
> q = Queue.Queue(4)
> 
> def proc():
>    while True:
>        q.get(1)
>        Queue.Queue()
>        print "YADDA"
> 
> def start():
>    threading.Thread(target=proc).start()
> 
>    while True:
>        print "yadda"
>        q.put(None)
>        time.sleep(1)
> </nofreeze.py>
> 
> <nofreezemain.py>
> import nofreeze
> nofreeze.start()
> </nofreezemain.py>
> 
> and invoking nofreezemain.py produces
> yadda
> YADDA
> yadda
> YADDA
> yadda
> YADDA
> yadda
> YADDA
> 
> apparently ad infinitum.

Right.  Note that this is the same reason threaded tests in Python's
standard regression suite define a 'test_main()' function, called by
the regrtest.py driver after import of the test's module completes. 
It's generally suicidal to start a thread as a side effect of an
import.

...

> Import _is_ a sensitive phase...

It's quite easy to avoid thread problems in imports:  never start a
thread as a side effect of importing, and you'll never get a deadlock
due to importing.

> As an experiment I moved
> 
> try:
>    import thread
> except ImportError:
>    import dummy_thread as thread
> 
> from the Queue.Queue.__init__() method to the module body --
> and now freezemain.py seems to work, too. So that would be an
> easy remedy, but sure there is a reason why that import
> statement is in such an unusual place?

I think you'd have to ask Brett (who did most of the work on
dummy_thread and dummy_threading).  It doesn't really matter, though: 
it's a general truth that starting a thread as a side effect of
importing is a recipe for deadlock, and hacking specific methods and
functions to avoid imports just moves the problem around.  It's not a
goal that anything in the standard Python library cater to bad thread
practice here (the bad thread practice being, again, starting a thread
as a side effect of importing).