Logging into single file from multiple modules in python when TimedRotatingFileHandler is used

[Chethan Kumar S]

Hi all,

Need help with below query on python logging module.

I have a main process which makes use of different other modules. And these modules also use other modules. I need to log all the logs into single log file. Due to use of TimedRotatingFileHandler, my log behaves differently after midnight. I got to know why it is so but couldn't get how I can solve it.
Issue was because of serialization in logging when multiple processes are involved.

Below is log_config.py which is used by all other modules to get the logger and log.
import logging
import sys
from logging.handlers import TimedRotatingFileHandler

FORMATTER = logging.Formatter("%(asctime)s — %(name)s — %(message)s")
LOG_FILE = "my_app.log"

def get_file_handler():
   file_handler = TimedRotatingFileHandler(LOG_FILE, when='midnight')
   file_handler.setFormatter(FORMATTER)
   return file_handler

def get_logger(logger_name):
   logger = logging.getLogger(logger_name)
   logger.setLevel(logging.DEBUG) # better to have too much log than not enough
   logger.addHandler(get_file_handler())
   #with this pattern, it's rarely necessary to propagate the error up to parent
   logger.propagate = False
   return logger

All other modules call, 'logging = log_config.get_logger(name)' and use it to log.
I came to know about QueueHandler and QueueListener but not sure how to use them in my code. How can I use these to serialize logs to single file.?

Any help is appreciated.
Thanks,
Chethan