[Mailman-Developers] Exim tweak

[J C Lawrence]

As an experiment in analysing my system loads I added the following to
my exim.conf:

  # When this option is set, a delivery process is started whenever a
  # message is received, directing and routing is performed, and local
  # deliveries take place. However, if any SMTP deliveries are
  # required for domains that match queue_smtp_domains, they are not
  # immediately delivered, but instead the message waits on the queue
  # for the next queue run. Since routing of the message has taken
  # place, Exim knows to which remote hosts it must be delivered, and
  # so when the queue run happens, multiple messages for the same host
  # are delivered over a single SMTP connection. This option is
  # checked against the domains supplied in the incoming addresses,
  # before any widening is done (because that is part of routing). The
  # -odqs command line option causes all SMTP deliveries to be queued
  # in this way, and is equivalent to setting queue_smtp_domains to
  # *'. See also queue_remote_domains, which is subtly different.

  queue_smtp_domains = "*"

Note that all of my lists are hand moderated resulting in outbound mail
occurring in batches, typically in the range of 5K - 10K MTA spool
entries per batch.

The primary effect of adding this config item is that (in general) a
compleat moderation batch is enqueued by the MTA before delivery
attempts start on any of the component messages.  This turns out to have
a significant effect on delivery time performance.

Loosely what seems to be happening is that without the above config Exim
and Mailman's qrunner end up competing for CPU resources, with system
loads climbing into the 30's and 40's with parallel deliveries to the
same MX being infrequent to the point of nonexistence. [1] However with
the above config Exim has the opportunity to batch all mail targeted to
a given MX and parallel deliveries are the default. [2] This shows
clearly in the logs with blocks of deliveries to the same MX all
together.  The apparent result of this batching is that the total
delivery time [3] for a moderation batch is now reduced by a factor of
almost 5. [3]

Not a bad side effect for a little idle poking about.

-- 
J C Lawrence               
---------(*)                Satan, oscillate my metallic sonatas. 
claw@kanga.nu               He lived as a devil, eh?		  
http://www.kanga.nu/~claw/  Evil is a name of a foeman, as I live.

1) I use 20 queue runners with up to 10 parallel deliveries per MX.

2) I have low rates of duplicate MX use in my subscriber base.  AOL +
MSN + Hotmail together form less than 3% of my subscriber base.  Outside
of resource contention this low collision rate is the prime factor in
parallel deliveries not happening.

3) Loosely defining total delivery time as the taken to handle all
deliveries to all fast MX'es (no errors, warnings, or responses taking
more than a second).

4) This depends heavily on whether Mailman's qrunner can get the
compleat moderation queue dumped to the MTA before it starts another
queue run.  Typically for my setup I can get ~4K spool entries queued
between MTA queue runs.  I *should* be able to get at least an order of
magnitude more, and almost two (as determined by trivial test code doing
a canned loop).  I haven't yet analysed why the qrunner path is so slow
and am unlikely to before I deploy v2.1.  However as a general
observation Mailman v2.0's qrunner seems absurdly slow in iterating
across the membership roster and dumping RCPT TO bundles to the MTA.