Trouble propagating logging configuration

[Peter Otten]

On 01/09/2021 13:48, Loris Bennett wrote:
> "Dieter Maurer" <dieter at handshake.de> writes:
> 
>> Loris Bennett wrote at 2021-8-31 15:25 +0200:
>>> I am having difficulty getting the my logging configuration passed on
>>> to imported modules.
>>>
>>> My initial structure was as follows:
>>>
>>>   $ tree blorp/
>>>   blorp/
>>>   |-- blorp
>>>   |   |-- __init__.py
>>>   |   |-- bar.py
>>>   |   |-- foo.py
>>>   |   `-- main.py
>>>   `-- pyproject.toml
>>>
>>> whereby the logging configuration is done in main.py.
>>>
>>> After thinking about it, I decided maybe the inheritance wasn't working
>>> because main.py is in the same directory as the other files.
>>
>> Should you speak about Python's `logging` module, then
>> the "inheritance" does not depend on the source layout.
>> Instead, it is based on the hierarchy of dotted names.
>> It is completely up to you which dotted names you are using
>> in your `getLogger` calls.
> 
> Yes, but to quote from https://docs.python.org/3.6/howto/logging.html#logging-basic-tutorial:
> 
>    A good convention to use when naming loggers is to use a module-level
>    logger, in each module which uses logging, named as follows:
> 
>      logger = logging.getLogger(__name__)
> 
>    This means that logger names track the package/module hierarchy, and
>    it’s intuitively obvious where events are logged just from the logger
>    name.
> 
> so in this case the source layout is relevant, isn't it?

In most cases you will only add handlers to the root logger. If you want 
special treatment of a specific logger you need to know its name 
including its package, i. e. log_test.first, not first. If you have the 
log_test directory in your path and use

import first

instead of

import log_test.first

then not only the __name__ will be wrong/unexpected, relative imports 
will fail, too.
In other words, this is generally a bad idea.

>> Furthermore, the place of the configuration (and where in the
>> code it is activated) is completely irrelevant for the "inheritance".
> 
> OK, so one issue is that I was getting confused by the *order* in which
> modules are being called.  If I have two modules, 'foo' and 'bar', in
> the same directory, configure the logging just in 'foo' and then call
> 
> 
>    foo.some_method()
>    bar.some_method()
> 
> then both methods will be logged.   If I do
> 
>    bar.some_method()
>    foo.some_method()
> 
> then only the method in 'foo' will be logged.
> 
> However, I still have the following problem.  With the structure
> 
>    $ tree .
>    .
>    |-- log_test
>    |   |-- __init__.py
>    |   |-- first.py
>    |   `-- second.py
>    |-- pyproject.toml
>    |-- README.rst
>    |-- run.py
>    `-- tests
>        |-- __init__.py
>        |-- config
>        `-- test_log_test.py
> 
> I have __name__ variables as follows:
> 
>    __file__: /home/loris/log_test/log_test/first.py, __name__: log_test.first
>    __file__: /home/loris/log_test/log_test/second.py, __name__: log_test.second
>    __file__: ./run.py, __name__: __main__
> 
> If I have
> 
>    [loggers]
>    keys=root,main,log_test
> 
> in my logging configuration and initialise  the logging in run.py with
> 
>    logging.config.fileConfig("/home/loris/log_test/tests/config")
>    logger = logging.getLogger()
> 
> or
> 
>    logging.config.fileConfig("/home/loris/log_test/tests/config")
>    logger = logging.getLogger("log_test")
> 
> then only calls in 'run.py' are logged.
> 
> I can obviously initialise the logging within the subordinate package,
> i.e. in 'log_test/__init__.py', but that seems wrong to me.
> 
> So what is the correct way to initialise logging from a top-level script
> such that logging is activated in all modules requested in the logging
> configuration?
> 
>> For details, read the Python documentation for the `logging` module.
> 
> If they were sufficient, I wouldn't need the newsgroup :-)
> 
> Thanks for the help,

Perhaps you can start with logging.basicConfig() in run.py. If the 
imported modules contain module-level logging messages you have to put 
the import statements after the basicConfig() invocation.

You should see that simple changes like setting the logging level affect 
messages from all loggers. If that's not enough to get the granularity 
you want give your loggers fixed names

# in test_log/first.py
logger = logging.getLogger("test_log.first")

or make sure that __name__ is what you expect

# in test_log/first.py
assert __name__ == "test_log.first"

during the debugging phase.