[Python-Dev] Threading, atexit, and logging

[Tim Peters]

[Martin v. Löwis]
>>> Upon investigation, it turns out that the file descriptor
>>> is closed because the logging atexit handler is invoked.
>>> This is surprising, as the program is far from exiting at
>>> this point.

[Tim Peters]
>> But the main thread is done, right?

[Martin]
> Wrong. main.py (which is the __main__ script in the demo
> code) is done, yes.

Fine, but the main thread /has/ entered Py_Finalize().  That's key
here, and wasn't clear originally.

> However, threading.py has machinery to not terminate the main
> thread as long as there are non-daemon threads.

Right.

...

>> IMO, `threading` shouldn't use atexit at all.

> That is (in a way) my proposal (although I suggest to use
> sys.exitfunc instead).

Same thing to me.  I'd rather thread cleanup, which is part of the
Python core, not rely on any of the user-visible (hence also
user-screwable) "do something at shutdown" gimmicks.  Thread cleanup
is only vaguely related to that concept because "cleanup" here implies
waiting for an arbitrarily long time until all thread threads decide
on their own to quit.  That's not something to be cleaned up /at/
shutdown time, it's waiting (potentially forever!) /for/ shutdown
time, and that mismatch is really the source of the problem.

>> It's expedient :-)  So was using atexit for this to begin with.
>> Probably "good enough".  I'd rather, e.g., that `threading` stuff an
>> exit function into a module global, and change Py_Finalize() to look
>> for that and run it (if present) before invoking call_sys_exitfunc().

> Ok, that's what I'll do then.
>
> Yet another alternative would be to have the "daemonic" thread feature
> in the thread module itself (along with keeping track of a list of
> all running non-daemonic thread).

Sorry, I couldn't follow the intent there.  Not obvious to me how
moving this stuff from `threading` into `thread` would make it
easier(?) for the implementation to wait for non-daemon threads to
finish.