Logging into single file from multiple modules in python when TimedRotatingFileHandler is used

[Peter J. Holzer]

On 2022-06-21 02:04:52 -0700, Chethan Kumar S wrote:
> I have a main process which makes use of different other modules. And
> these modules also use other modules. I need to log all the logs into
> single log file. Due to use of TimedRotatingFileHandler, my log
> behaves differently after midnight. I got to know why it is so but
> couldn't get how I can solve it.
> Issue was because of serialization in logging when multiple processes
                                                     ^^^^^^^^^^^^^^^^^^
> are involved.

I think this is crucial point. Not "multiple modules" (as you wrote in
the subject), but "multiple processes". If each of multiple processes
wants to rotate the logfile itself, it is very likely that they will
work at cross-purposes.

Somebody already suggested using logrotate (or similar external tools).
You can then use WatchedFileHandler
(https://docs.python.org/3/library/logging.handlers.html#watchedfilehandler)
to automatically detect when a logile has been rotated.

Alternatively you can use a central logging service (like syslog) which
handles all that stuff.

        hp

-- 
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | hjp at hjp.at         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://mail.python.org/pipermail/python-list/attachments/20220627/a58c9f6f/attachment.sig>