[Mailman-Developers] [ mailman-Bugs-558988 ] bad performance for big queue dirs

[Barry A. Warsaw]

>>>>> "NB" == Norbert Bollow <nb@thinkcoach.com> writes:

    NB> Now the welcome messages are going out, at a rate of one
    NB> message every 6.6 seconds.  This is on inexpensive hardware (a
    NB> Cobalt RaQ3) but the server is otherwise idle.  At this rate,
    NB> it would take more than 36 hours to send them all out.  I
    NB> consider this to be unacceptably slow (but it's not
    NB> catastrophic unless it's normal for your server to put ten or
    NB> more messages into the virgin queue per minute on the
    NB> avarage).

There are a lot of unknowns here though.  For example, we don't know
if your rate limits are caused by MTA or network throttles.  If your
server is idle then i/o is suspect but we don't know if creating more
outgoing qrunner processes would help you (by splitting up the queue
hash space among parallel processes).  Is your MTA spooling to the
same filesystem that Mailman is spooling off of?  If your MTA is
throttling and it's running synchronously with the outgoing qrunner
then Mailman may just be sitting around blocked on output to your
MTA.  Or maybe your disk subsystem can't take the pressure of Mailman
reading off of it while your MTA is writing to it.  What happens if
you put them on different disks and/or controllers?  And what effect
does using some other filesystem (e.g. reiserfs) have on throughput?

    NB> What I'm really concerned about is the possiblity of a similar
    NB> thing happening on the bounces queue.  On a busy system, you
    NB> can easily have bounces coming in with a rate greater than 10
    NB> bounces per minute.  Temporary network problems can easily
    NB> result in you getting a large number of bounces at the same
    NB> time.  This could put so many bounces into the queue that
    NB> processing bounces becomes slower than the rate at which new
    NB> bounces arrive.  Then the bounce-handler is permanently
    NB> screwed.

I don't think we can correlate your outgoing queue draining behavior
with any supposed behavior of the internal bounce queue draining.

-Barry