[Python-Dev] Fun with 2.3 shutdown

[Tim Peters]

[Armin Rigo]
> The behavior of Python regarding whether threads should continue to
> run after the main one exited has never been too clear.  Wouldn't it
> make sense to try to control this aspect more precisely ?

threading.py already tried to.  Whether or not a native platform thread
outlives the main thread varies across platforms.  A Thread created via
threading.py prevents the main thread from exiting on all platforms so long
as the Thread is running, unless the user calls its setDaemon(True) method
before starting the thread.  Whether a daemon Thread can outlive the main
thread remains platform-dependent.  These platform dependencies exist
because there's no cross-platform way for Python to forcibly end a thread's
life from outside the thread, or to force a thread to continue running after
the main thread exits.  Python *can* wait for a Thread to end on its own in
a cross-platform way, and threading.py keeps the main thread alive until all
non-daemon Threads have stopped running via that portable method.  So it
does what it reasonably *could* do.

> I have also experienced this kind of shutdown glitches.

You mean the uselessly empty "Exception in thread" messages at shutdown?
I've only seen those when running Zope3 tests so far (I count *non*-empty
"Exception in thread" shutdown msgs as a different glitch, though).  I
expect every long-time Python programmer has seen the ever-popular "object
'None' has no attribute so-and-so" kinds of shutdown glitches, and we fix
more of those every release by purging __del__ methods, and methods invoked
by __del__ methods, of references to module globals.  The segfaults at
shutdown I saw later while trying to debug the Zope3 symptom were new to me.

> It definitely needs more thinking, but what about this one: Python
> would only shutdown after all threads are finished;

In the Zope3 case, there are about a dozen Threads still running, and
they'll never finish.  Nine are in an Event.wait() that will never trigger;
the rest are in an unbounded polling loop with a sleep(3) each time around.
They're all daemon Threads, though, so asked Python explicitly not to wait
for them to finish.

> but exiting the main thread would by default send a SystemExit
> exception to all current threads, if that can be done cleanly (oh
> well, this problem again).

I'm afraid that would be a major change in shutdown semantics for non-daemon
Threads, yes?  Right now the main thread will happily wait as long as it
takes for Threads to decide to stop on their own, and I've seen programs
rely on that (fire up a bunch of producer and consumer Threads, then let the
main thread fall off the end of the script).  It might help the Zope3
problem, though, if SystemExit got raised in daemon Threads when the main
thread wanted to exit.  But stuffing a pending exception on a Thread's todo
list is asynchronous, and in the case of the Zope3 Threads sitting in their
sleep(3) polling loops, the SystemExit won't be noticed until the sleep
expires.  Since 3 might actually be bigger than 3 in some apps <wink>,
there's really no limit on how long the main thread might have to wait to be
sure all the daemon Threads noticed their SystemExit.

> Another question, what was the rationale for setting module globals
> to None instead of simply deleting them (and getting the expected
> AttributeErrors both at C and at Python level) ?

I suspect it's because there's no "efficient" way to iterate over a dict and
simultaneously mutate it in a size-changing way; the C PyDict_Next()
protocol allows the caller to change the value associated with existing keys
(like setting them to None), but is unpredictable if the caller adds or
deletes keys during PyDict_Next() iteration.

Well, due to quirks of the current implementation, I suspect it actually
could delete all the keys without harm (the relevant quirk is that the
current implementation never resizes a dict in response to a key deletion;
it can resize only when a new key gets added; resizing can shuffle the keys
around in a new order, which is what makes PyDict_Next unpredictable if the
dict does get resized during iteration).