[Mailman-Users] Why does Mailman have a queue?

[Barry A. Warsaw]

My mailman-users subscription got disabled, probably due to the
digicool.com -> zope.com switch.  I probably missed a bunch of
messages on this list but didn't notice because I've been literally
deluged with email from all the other sundry forums I'm on.  I'm
convinced email is like Washington DC traffic: there's more of it
every year and about the only time you don't wallow in it up to your
eyeballs is 2am (usually after coming home from a gig).

Greg Ward asks in:

http://mail.python.org/pipermail/mailman-users/2001-July/013126.html

why Mailman has a queue.  One simple reason for MM2.0.x: most MTAs
impose a time limit on how long they let a mail program run.  Postfix
specifically uncatchably SIGKILLs a mail program after a configurable
amount of time (I think the default is 1000 seconds).  Before you say
that that should be enough for anything Mailman wants to do with a
message, consider a really busy server with lots of list locks.
Before the incoming queue was implemented, we used to see lots of
messages just get hard killed by Postfix because the list lock
couldn't be acquired in time.  You could argue that the list lock
implementation could be improved so as to guarantee that a mail
program could finish in the allotted time, but 1) I don't think you
could guarantee that; 2) my mantra is "no email shall be lost"; 3) the
queue idea actually has other benefits, mostly to be realized in
MM2.1.

E.g. there are now 7 queues (possibly to be 8 if there's time) for
moving mail around the system.  There's are two incoming queues, 3
outgoing queues, a "virgin" queue (for messages generated by Mailman),
and a "shunt" queue for errors.  We can individually control how many
processes run through each queue, and how often they run.  So for
example, we could put a high priority on the incoming and outgoing
queues, possibly throwing multiple processors or multiple servers at
these queues.  We could put a lower priority on archiving, delivering
to news, or processing email commands by running fewer processes, or
running them less often.  We can also apply different processing steps
to each queue.  And we can better separate read/write operations from
read-only operations, e.g. so that the "modify-and-munge" pipeline
that all incoming-queue messages go through need to acquire the list
lock, but the outgoing-queue messages, which only get handoffs to the
MTA don't need to acquire the list lock.  Thus if the MTA is slow to
respond for some reason (e.g. it's swamped and is backing off
accepts), the rest of Mailman won't get bogged down behind it.  We can
also (in theory) do more finely grained rate limiting types of things.

-Barry