[Mailman-Developers] (2.0.6) pipermail takes >1 minute to rebuild indexes on large lists

[Ben Gertzfield]

On a massive list (Mailman 2.0.6) I run that regularly gets a few
hundred or more emails every day, things begin to slow down to
molasses after a week or two each month, with the qrunner process
taking literally HUNDREDS of megabytes of RAM, and 100% CPU, all the
time.

It's gotten so bad that the pipermail databases have gotten massive:

-rw-rw-r--    1 list     list     40866061 Sep 13 19:24 2001-September-article
-rw-rw-r--    1 list     list       686273 Sep 13 19:24 2001-September-author
-rw-rw-r--    1 list     list       561479 Sep 13 19:24 2001-September-date
-rw-rw-r--    1 list     list       796534 Sep 13 19:24 2001-September-subject
-rw-rw-r--    1 list     list       569238 Sep 13 19:24 2001-September-thread

And, of course, pipermail takes over a *MINUTE* to rebuild the indexes
now.  This is with a bunch of syslog() debugging in pipermail.py.
Notice the timestamps -- this is how long it takes a *single* message
to make it through the ToArchive step of the pipeline..

Sep 13 19:24:02 2001 (29357) in pipermail._update_simple_index() now
Sep 13 19:24:02 2001 (29357) opening index as stdout
Sep 13 19:24:02 2001 (29357) done opening index, writing index header
Sep 13 19:24:02 2001 (29357) getting article 200109010000.JAA26551@admin.interq.or.jp from db
Sep 13 19:24:03 2001 (29357) writing index entry 0
Sep 13 19:24:03 2001 (29357) going to next entry

(snip LOTS of index entry writing -- note the timestamp and the count below)

Sep 13 19:24:52 2001 (29357) getting article 3B95AB3EA.4F08HONDA@192.168.1.190 from db
Sep 13 19:24:52 2001 (29357) writing index entry 4794
Sep 13 19:24:52 2001 (29357) going to next db
Sep 13 19:24:52 2001 (29357) calling self.write_index_footer()
Sep 13 19:24:52 2001 (29357) in pipermail._update_thread_index
Sep 13 19:24:53 2001 (29357) in pipermail.write_TOC()
Sep 13 19:24:53 2001 (29357) done writing TOC
Sep 13 19:24:53 2001 (29357) picking archive state into /var/lib/mailman/archives/private/sysadmin/pipermail.pck

While it does this, the size of the python process that's doing the
qrunner grows by *megabytes* each second.  I've seen it get to 200MB
before it finishes, and it's always taking up 99% CPU.

As you can guess, things rapidly get out of hand, and messages start
arriving faster than pipermail can deal with them.  And since it's
first in the pipeline, nothing gets delivered until pipermail archives
the mail.. and mail queues up and queues up, until it's taking 7-8
hours to send out each mail -- no joke.

What can I do to help solve this?  I know I can use an external
archiver, but I don't know anything about that.  I've commented out
the ToArchive step from the pipeline for now, but I don't know what
else I can do.  There *MUST* be a way to not have to loop over every
single article in the simple index every time a message arrives,
right?

Other people must have run into this, and I'd like to know what I can
do to only have to write the changed articles in the simple index.

Thanks,

Ben

-- 
Brought to you by the letters F and O and the number 12.
"Johnny! Don't go! It's too dangerous!" "I don't care!"
Debian GNU/Linux maintainer of Gimp and GTK+ -- http://www.debian.org/