[Mailman-Developers] [ mailman-Bugs-558988 ] bad performance for big queue dirs

[Norbert Bollow]

> > Many filesystems (e.g. the popular ext2) have horrible
> > performance when there are many files in the same
> > directory.  The queue system should be modified to
> > avoid this situation.  As a test case, try adding
> > 20,000 test address in such a way that Mailman will try
> > to send a welcome message to each of them.
> 
> Isn't this optomising for a rather uncommon case.

It may be an uncommon case, but if/when it happens, processing the
queue becomes so slow that it's a real problem.

I've just taken some timings:

Adding 20,000 test addresses took about 40 minutes.  (They're all
addresses at a remote machine where the the whole domain is forwarded
to /dev/null ). I consider this to be acceptable.

Now the welcome messages are going out, at a rate of one message every
6.6 seconds.  This is on inexpensive hardware (a Cobalt RaQ3) but the
server is otherwise idle.  At this rate, it would take more than 36
hours to send them all out.  I consider this to be unacceptably slow
(but it's not catastrophic unless it's normal for your server to put
ten or more messages into the virgin queue per minute on the avarage).

What I'm really concerned about is the possiblity of a similar thing
happening on the bounces queue.  On a busy system, you can easily have
bounces coming in with a rate greater than 10 bounces per minute.
Temporary network problems can easily result in you getting a large
number of bounces at the same time.  This could put so many bounces
into the queue that processing bounces becomes slower than the rate at
which new bounces arrive.  Then the bounce-handler is permanently
screwed.

Greetings, Norbert.

-- 
A founder of the http://DotGNU.org project and Steering Committee member
Norbert Bollow, Weidlistr.18, CH-8624 Gruet   (near Zurich, Switzerland)
Tel +41 1 972 20 59         Fax +41 1 972 20 69        http://norbert.ch
List hosting with GNU Mailman on your own domain name   http://cisto.com