[Mailman-Users] arch python errors, bad marshall data

[Tom Perrine]

>>>>> On Sun, 6 Jan 2002 12:12:27 -0800, Micah Anderson <micah at riseup.net> said:

    Micah> The archives of a particular list went astray, so I tried to run arch
    Micah> to re-pickle them, it went for some time (we have archives for a
    Micah> couple years), up until December of this year, then it puked, anyone
    Micah> know how to fix this??

I have no idea if this is related, but my experience may be
interesting...

I have a mailing list that has been running since 1994.  It has about
20,000 messages in the non-Mailman archives.  I have been importing
and re-importing it into Mailman, as I was playing with some Mailman
options and also correcting corruption in the original archives as
part of migration into Mailman.

The first few times I cat'ed all the archive files together and ran
"arch" on the resulting huge mbox file.  This took overnight, most
likely because the machine was heavily into swap, and trying to
compute long lists of links, etc.

Later, I started "arch"-ing the individual files separately.  This
stayed out of swap and typically runs all the files in less than an
hour!

I guess what I'm getting at is that the "arch" program uses LOTS of
memory when building some of the indexes, and that incremental
"arch"-ing may be more efficient and less likely to exercise
memory-related bugs in Mailman, Python, or the underlying system.

It will also avoid the situations where you hit the pre-process memory
limits, which is what I *suspect* has happened to you.

-- 
Tom E. Perrine (tep at SDSC.EDU) | San Diego Supercomputer Center 
http://www.sdsc.edu/~tep/     | Voice: +1.858.534.5000
"The French are glad to die for love..."  - Moulin Rouge