[Mailman-Developers] (2.0.6) pipermail takes >1 minute to rebuild indexes on large lists

[Barry A. Warsaw]

>>>>> "BG" == Ben Gertzfield <che@debian.org> writes:

    BG> On a massive list (Mailman 2.0.6) I run that regularly gets a
    BG> few hundred or more emails every day, things begin to slow
    BG> down to molasses after a week or two each month, with the
    BG> qrunner process taking literally HUNDREDS of megabytes of RAM,
    BG> and 100% CPU, all the time.

FWIW, the combined {zope,python}.org gets many hundreds of messages a
day on the python-list and zope mailing lists alone.  I'm not online
at the moment, so I can't check, but I know I receive several digests
every day from both lists -- they're very high traffic.  Needless to
say, I haven't noticed any such performance problems...

Here are a few things you can do to improve matters.  First and
foremost, make sure GZIP_ARCHIVE_TXT_FILES is 0 -- this should be the
default though.  Don't even try to gzip your .txt files in real-time
since this will absolutely clobber your system.  Use cron/nightly_gzip
instead (it doesn't have to be nightly, that's up to your crontab).

If your system still can't handle things, then the next step is to set
ARCHIVE_TO_MBOX to 1.  This way, Mailman will simply append the
message to the .mbox file, which ought to be extremely quick, but it
won't attempt to run the Pipermail archiver in real time.  Then you
can use whatever archiving scheme you want (e.g. bin/arch nightly, or
an external archiver).

In MM2.1, you'll be able to lower the priority of your archive
qrunner, so that it processes messages, say once per hour.  It also
won't be in-line with the normal thru-path of messages, so even if the
archiver is slow, it won't gum up delivery of list messages.

And yes, the Pipermail stuff is a mess.  I don't have time to rewrite
it, so it'll have to wait for volunteers.

-Barry