[Python-Dev] RE: logging needs better documentation

[Neal D. Becker]

Vinay Sajip at Red Dove wrote:

> Hi Neal,
> 
> I quite agree that the documentation for the logging package can be
> improved. As others have said in response to your post, you can definitely
> help by indicating more specifically where you find the documentation
> lacking, with a patch if possible. For example, your initial post about
> the length prefix for pickles is documented in the docstring for
> SocketHandler.makePickle():
> 
> "Pickles the record in binary format with a length prefix, and returns it
> ready for transmission across the socket."
> 
> I agree that the documentation does not mention specifically that the
> length is encoded as four bytes, or that it is packed using struct.pack(),
> or exactly how you unpack it. I also agree that more examples would be
> helpful. I will endeavour to improve the situation insofar as time allows.
> Patches and specific suggestions from users, especially new users like
> you, will be a definite help to me.
> 

Thanks.  In this case at least, just an example was all I needed.

Perhaps this example is a little larger than you would want?  It is a system
for collecting simulation results from a cluster and logging onto a server. 
The clients send periodic datagrams with fqdn, pid, argv and a result.  The
server updates a file whose name is .../fqdn:pid.

The parts I had trouble with were:
1) the 4 extra bytes in the pickle
2) the fact that formatter needed to be on the server side, not the client
3) the apparantly undocumented fact that most (but not all!) handlers are in
logging.handlers.

,----[ /disk1/nbecker/shannon2/nonlinear/logserver.py ]
| #!/usr/bin/env python
| 
| import socket
| import logging
| import pickle
| import string
| import os
| 
| def Update (fname, msg):
|     newname = fname+".new"
|     f = open (newname, "w")
|     f.write (str(msg))
|     f.close()
|     oldname = fname+".old"
|     try:
|         os.link (fname, oldname)
|     except:
|         pass
|     os.rename (newname, fname)
|     try:
|         os.unlink (oldname)
|     except:
|         pass
|     
| class MyHandler (logging.Handler):
|     def __init__ (self, path):
|         logging.Handler.__init__ (self)
|         self.path = path
|         try:
|             os.makedirs (path, 0775)
|         except:
|             pass
|         
|     def emit (self, rec):
|         form = self.format (rec)
|         print form
|         els = string.split (form, "|")
|         date = els[0]
|         fqdn = els[2]
|         argv = els[3]
|         pid = els[4]
|         msg = els[5]
|         
|         fname = self.path+'/'+fqdn+':'+pid
|         Update (fname, [argv,msg])
|         
| sock = socket.socket (socket.AF_INET, socket.SOCK_DGRAM)
| sock.bind (("", 8881))
| 
| logger = logging.getLogger()
| hdlr = MyHandler("/tmp/logger")
| formatter = logging.Formatter('%(asctime)s|%(levelname)s|%(message)s')
| hdlr.setFormatter(formatter)
| 
| try:
|     while (True):
|         data, address = sock.recvfrom (8192)
|         rec = logging.makeLogRecord (pickle.loads(data[4:]))
|         hdlr.emit (rec)
|         
| finally:
|     sock.close()
`----
,----[ /disk1/nbecker/shannon2/nonlinear/logclient.py ]
| #!/usr/bin/env python
| 
| import logging
| import logging.handlers
| import sys
| import os
| import socket
| 
| class logclient (object):
|     def __init__ (self):
|         self.logger = logging.getLogger()
|         hdlr = logging.handlers.DatagramHandler ('myhostname.com',
|         8881) self.logger.addHandler(hdlr)
|         self.logger.setLevel(logging.INFO)
| 
|     def __call__(self, msg):
|         self.logger.info ("%s|%s|%s|%s" %
|         (socket.getfqdn(),sys.argv,os.getpid(),msg))
| 
| if (__name__ == "__main__"):
|     c = logclient()
|     c (str (["hello"]))
`----