[Mailman-Developers] marshal.load() and error handling

[Cillian Sharkey]

I'd been getting EOF errors for some time now, both in log/error and from
cron/checkdbs, e.g:

Traceback (most recent call last):
  File "/local/mailman/cron/checkdbs", line 92, in ?
    main()
  File "/local/mailman/cron/checkdbs", line 43, in main
    count = mlist.NumRequestsPending()
  File "/local/mailman/Mailman/ListAdmin.py", line 96, in NumRequestsPending
    self.__opendb()
  File "/local/mailman/Mailman/ListAdmin.py", line 69, in __opendb
    self.__db = marshal.load(fp)
EOFError: EOF read where object expected

So I decided to track it down: the problem was lists with corrupt request.dbs.
I think the solution would be to handle the EOFError exception in ListAdmin.py
in __opendb. I don't know Python, but perhaps something along the lines of:

    def __opendb(self):
        if self.__db is None:
            assert self.Locked() and self.__filename
            try:
                fp = open(self.__filename)
                self.__db = marshal.load(fp)
                fp.close()
            except IOError, e:
                if e.errno <> errno.ENOENT: raise
                self.__db = {}
+           except EOFError, e:
+               self.__db = {}

In addition to this, it may be worthwhile handling other errors (ValueError,
TypeError etc.) which indicate corrupt databases. Some of the other mailman
code which uses marshal.load does this. Perhaps all uses of marshal.load
should be checked for extra error handling?

There was no easy way to find out which request database was corrupt (there
doesn't appear to be any script similar to check_db to do the same for the
request.db file) so I just wrote a script based on check_db to help find
corrupt request.dbs. I think something like this (cleaned up of course) should
be added to the distribution..

bin/check_request_db
------------8<------------
#! /usr/bin/env python

"""Check the raw request.db for a mailing list.

Usage: %(program)s listname
"""

import sys
import os
import string
import marshal

import paths
from Mailman import mm_cfg

program = sys.argv[0]

def testfile(filename):
    try:
        fp = open(filename)
    except IOError, (code, msg):
        print filename, 'cannot be opened:\n\t', msg
        return 1
    else:
        try:
            d = marshal.load(fp)
        except (EOFError, ValueError, TypeError), msg:
            print filename, 'is corrupted:\n\t', msg
            return 1
        else:
            print filename, 'is fine'
            return 0

def main():
    if len(sys.argv) == 2:
        listname = string.lower(sys.argv[1])
    else:
        print __doc__ % globals()
        sys.exit(1)

    listpath = os.path.join(mm_cfg.LIST_DATA_DIR, listname)
    configdb = os.path.join(listpath, 'request.db')

    testfile(configdb)

if __name__ == '__main__':
    main()

------------>8------------

Regards,

--
Cillian

[PS: please CC me on any replies]