[Python-Dev] Threading, atexit, and logging

[Tim Peters]

[Martin v. Löwis]
> In bug #1566280 somebody reported that he gets an
> exception where the logging module tries to write
> to closed file descriptor.
>
> Upon investigation, it turns out that the file descriptor
> is closed because the logging atexit handler is invoked.
> This is surprising, as the program is far from exiting at
> this point.

But the main thread is done, right?  None of this appears to make
sense unless we got into Py_Finalize(), and that doesn't happen until
the main thread has nothing left to do.

> Investigating further, I found that the logging atexit
> handler is registered *after* the threading atexit handler,
> so it gets invoked *before* the threading's atexit.

Ya, and that sucks.  Can't recall details now, but it's not the first
time the vagaries of atexit ordering bit a threaded program.  IMO,
`threading` shouldn't use atexit at all.

> Now, threading's atexit is the one that keeps the
> application running, by waiting for all non-daemon threads
> to shut down. As this application does all its work in
> non-daemon threads, it keeps running for quite a while -
> except that the logging module gives errors.
>
> The real problem here is that atexit handlers are
> invoked even though the program logically doesn't exit,
> yet (it's not just that the threading atexit is invoked
> after logging atexit - this could happen to any atexit
> handler that gets registered).
>
> I added a patch to this report which makes the MainThread
> __exitfunc a sys.exitfunc, chaining what is there already.
> This will work fine for atexit (as atexit is still imported
> explicitly to register its sys.exitfunc), but it might break
> if other applications still insist on installing a
> sys.exitfunc.

Well, that's been officially deprecated since 2.4, but who knows?

> What do you think about this approach?

It's expedient :-)  So was using atexit for this to begin with.
Probably "good enough".  I'd rather, e.g., that `threading` stuff an
exit function into a module global, and change Py_Finalize() to look
for that and run it (if present) before invoking call_sys_exitfunc().
That is, break all connection between the core's implementation of
threading and the user-visible `atexit` machinery.

`atexit` is a hack specific to "don't care about order" finalization
functions, and it gets increasingly obscure to try to force it to
respect a specific ordering sometimes (e.g., now you have a patch to
try to fix it by relying on an obscure deprecated feature and hoping
users don't screw with that too -- probably "good enough", but still
sucky).