Improper creating of logger instances or a Memory Leak?

foobar wjshipman at gmail.com
Sat Jun 18 16:02:48 EDT 2011


I've run across a memory leak in a long running process which I can't
determine if its my issue or if its the logger.

The long and short is I'm doing load testing on an application server
which spawns handlers threads which in turn each spawn a single
application thread. A graphic representation would be One Server ->
(to many pairs of) [ Handler <-> Application ].

 Each application thread gets a logger instance in it's init() method
via:

        self.logger = logging.getLogger('ivr-'+str(self.rand))

where self.rand is a suitably large random number to avoid collisions
of the log file's name.  Until the log file gets created I attach am
memory handler

        self.memhandler             =
logging.handlers.MemoryHandler(1000)
        self.memhandler.setLevel(10)
        formatter                       = logging.Formatter('%
(levelname)s %(message)s')
        self.memhandler.setFormatter(formatter)
        self.logger.addHandler(self.memhandler)

when the application thread formally starts with the run() method I
create the log file and terminate the memory handler

        filehandler = logging.FileHandler(logfilename)
        filehandler.setLevel(10)
        formatter = logging.Formatter('%(levelname)s %(message)s')
        filehandler.setFormatter(formatter)

        self.memhandler.setTarget(filehandler)
        self.memhandler.close()
        self.logger.removeHandler(self.memhandler)
        self.logger.addHandler(filehandler)


finally the last statements in the run() method are:

        filehandler.close()
        self.logger.removeHandler(filehandler)
        del self.logger #this was added to try and force a clean up of
the logger instances.

Using the objgraph to look at the objects in memory I find the number
of logger instances equal to the total number of threads to have lived
despite the fact that either a) there are only the standard load
testing number of threads alive, 35 or b) that there no threads
running nor are there any stale waiting for the GC.

>From objgraph a selection of the most prevalent objects in memory are
(this is with the system idle post-run):

list                                         256730
dict                                        128933
Logger                                  128164            # total
application threads executed running load testing.
function                                     2356
wrapper_descriptor                   1028
builtin_function_or_method        702
method_descriptor                     648
tuple                                           643
weakref                                      629
getset_descriptor                       304
type                                            252
set                                              224
member_descriptor                    209
module                                       128
WeakSet                                    102


The only references to self.logger other than those listed are wrapper
methods defined in the application thread to wrap up the log / debug
methods.  Any help or direction would be much appreciated.



More information about the Python-list mailing list