[Mailman-Users] Backing up Mailman list installations

[Greg Ward]

On 29 November 2001, Thomas T. Veldhouse said:
> If that is the case, then are these files portable between Python
> implementations and versions?

Yes, with the proviso that the format for Python bytecode (which is also
stored in marshal files -- you see them as .pyc files) usually changes
with each new Python version.  But Mailman doesn't marshal bytecode,
just lists and dictionaries.

Here's proof -- and, at the same time, a handy method for dumping
Mailman config.db files to human-readable form.  I wrote this little
script, called pprint_marshal:

  import sys
  from marshal import load
  from pprint import pprint

  fn = sys.argv[1]
  f = open(fn)
  d = load(f)
  pprint(d)

Then I transferred the ~mailman/lists tree from our mail server to my
development machine.  (Both are running Debian Linux on x86 processors,
so that's not much of a portability challenge.)  I ran the script on a
particular config.db with Python 1.5.2, 2.0, 2.1.1, and 2.2b1; it worked
just fine on all of them.  Eg.

  $ python1.5 pprint_marshal mailman/lists/test/config.db
  $ python2.0 pprint_marshal mailman/lists/test/config.db
  $ python2.1 pprint_marshal mailman/lists/test/config.db
  $ python2.2 pprint_marshal mailman/lists/test/config.db

Except for some minor changes in how Python formats strings, I got the
same result each time.

Then I copied everything to our old mail server, a Sun running Solaris
2.6.  Only Python 1.5 and 1.6 were available there, and again I got the
same results.

Oh yeah, we moved all our Mailman lists from that Sun machine
(big-endian, Solaris) to an x86 Linux machine (little-endian) a few
months ago, and had no problems whatsoever.  It never even occurred to
me that Mailman might use a non-portable format for config.db!  (C'mon,
give the Mailman cabal a little more credit...)

> I had a horrible problem moving a mailman
> 2.0.2 using Python 1.52 on FreeBSD to Mailman 2.0.3 using Python 1.6 on
> Linux (2.2) about one year ago.  The files could NOT be read by mailman.  I
> was forced to add all the users to the new list by hand and thus reset all
> the passwords and settings for each user (most unhappy about that).  It was
> also a bit of a pain to get the archives working again so that users could
> browse them on the web.

That sounds awful -- my sympathies.  I have no idea what could have gone
wrong, but I'm quite confident that config.db files are portable across
architectures and Python versions.

> I am not an expert with them -- I avoid them in favor of true RDBMS.  I was
> under and assumption that they were in platform specific binary files
> (though the file layout maybe the same on each platform, byte ordering could
> make a difference).

Not that this has *anything* to do with Mailman, but: Berkeley DB files
have a byte-order, but are platform-independent.  Using a little-endian
file on a big-endian machine obviously incurs a runtime penalty, and
vice-versa.  See
  http://www.sleepycat.com/docs/ref/am_conf/byteorder.html

        Greg
-- 
Greg Ward - software developer                gward at mems-exchange.org
MEMS Exchange                            http://www.mems-exchange.org