[Python-Dev] Loggers in the stdlib and logging configuration APIs

[Vinay Sajip]

The logging configuration calls fileConfig and dictConfig disable all existing
loggers, and enable only loggers explicitly named in the configuration (and
their children). Although there is a disable_existing_loggers option for each
configuration API, which can be used to prevent disabling of existing loggers,
the default value of this option is True (for backward compatibility) and so
existing ("old") loggers are disabled by default.

This can lead to unexpected behaviour when loggers are used by the standard
library. A recent issue, 

http://bugs.python.org/issue10626

relates to test_concurrent_futures failing, but only when exercised twice with a
test_logging call in between, as in

regrtest.py test_concurrent_futures test_logging test_concurrent_futures

I've fixed test_logging so that it remembers the disabled state of existing
loggers and restores it, and now the above test works fine. However, since
recent changes in logging, we are expecting to use it to e.g. display messages
on stderr when exceptions can't be raised in stdlib code. Thus, I think I may
need to change the way disabling of loggers works, so I wanted to get some
feedback from python-dev both about possible approaches and when to apply them
(With 3.2 so close to release I'm not proposing to do anything precipitate, just
raising the issue for discussion).

Essentially, some or all of the loggers in the stdlib itself should perhaps be
immune to the "disable existing loggers" logic. Otherwise, a fileConfig() or
dictConfig() call with default semantics will prevent a stdlib message (logged
when exceptions can't be raised) from being displayed, unless a user explicitly
names those stdlib loggers in their configuration. This is not practical because
users would need to update their configurations whenever a new logger appeared
in the stdlib. Some possible approaches are:

1. Change the default configuration behaviour so that by default, existing
loggers are NOT disabled. This is of course strongly backwards-incompatible, but
would it make sense to consider it for 3.3? Perhaps we should add a warning?
2. Change the disabling logic so that it never disables stdlib loggers. This
raises the question of how to identify them, and there are several loggers in
the stdlib in 3.2:

     "py.warnings" (used by the logging package when warnings are redirected to
logging),
     "concurrent.futures",
     "http.cookiejar",
     "multiprocessing"

     However, it would be good to have some consistency of naming stdlib loggers
- perhaps using __name__ as is recommended practice for library and application
developers, but with a prefix such as "py." to indicate that it's a part of the
stdlib. This would make the disabling logic change easier as it could just check
the "py." prefix rather than have an internal list of logger names, which would
need more maintenance and be more error-prone.

There are quite possibly other approaches I haven't thought of. I'd be grateful
for your suggestions, as I'd like to minimise the impact on users, while making
it easy to add more logging to the stdlib.

Regards,

Vinay Sajip