[Mailman-Developers] Regarding Handlers/SMTPDirect.py and "chunkify"

[Stefan Förster]

* Mark Sapiro <mark at msapiro.net> wrote:
> Stefan Förster wrote:
>> I don't think I'm qualified to discuss my concern (which involves VERP
>> delivery using a specific MTAs VERP implementation and a highly I/O
>> saturated mail server) on this list.
> 
> See
> <http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq04.012.htp>
> for some background.

I apologize, I should have found that entry myself. This sure makes up
for an interesting reading. Was bandwith really that expensive back in
2001? :)

> 2. is only partially correct. The purpose of grouping is to try to
> minimize the number of transactions between the outgoing MTA and the
> remote MTA. E.g., if all the aol.com addresses are passed from mailman
> to the outgoing MTA in one SMTP transaction, then the MTA is able (if
> so configured, etc., ...) to deliver the message to aol.com in one
> transaction with the same recipients. It has nothing to do with trying
> to prevent blocking, as the chunks are delivered serially anyway.

I can fully understand the thoughts behind this. OTOH, as the FAQ
entry you mentioned, if you do parallel deliveries to a single
location, you get a greater throughput: "Cut network bandwidth but
slow delivery to the larger domains." I remember a posting on this
list about cuddling MTAs or trying to outsmart them ;-) [1]

> In any case, if you have 'tuning' suggestions, either in general or
> for specific cases, we would be glad to receive them and either
> incorporate them in the code or add them to the FAQ as appropriate.

I don't know whether this is really a "tuning" suggestion. Let me
please describe my specific setup in hopes that the reason for my
request becomes clearer:

I recently had to take care of a server which was MX for a lagre
company and their outgoing mail relay (common setup). This machine was
doing fine until another server, dedicated to handle the companys many
mailig lists, broke down and it had to take care of mailing list
deliveries, too. The added load of permanent VERPed deliveries was
totally killing the machine in terms of I/O capacity, a situation
which could be resolved as of yesterday by letting the MTA do the
VERPing.

Now, the reason this change was so effective was that the number of
(disk) I/O operations was greatly reduced: Where a posting to a list
with 25000 subscribers would inevitably lead to the creation of 25000
queue files (even more if message content and control information are
kept in different files) before the change, only 25000/500 were
created afterwards (or "should have been", see next paragraph).
Additionally (but this has not really that much of an influence), a
larger number of destinations is available much faster to the MTA,
allowing it's queue manager to make decisions in a "more educated"
way.

Since the server is still quite close to it's I/O limits, I would very
much like to further reduce the number of I/O transactions. Therefore,
I would like to see a configurable parameter, something like
FLAT_RECIPIENT_CHUNKIFY: Right now, if mm_cfg.SMTP_MAX_RCPTS is not
set <= 0, "chunkify" will always (i.e. for any real world recipient
distribution) return more than one chunk of recipients, even if the
total number of recipients is smaller than  mm_cfg.SMTP_MAX_RCPTS.
With the addition of the aforementioned "tunable", we could have
chunkify simple split recipients into batches of mm_cfg.SMTP_MAX_RCPTS
and don't do any groupings. This way, a delivery to 25000 recipients
would really only involve (25000/mm_cfg.SMTP_MAX_RCPTS)*(number of
queue files per message) file creation operations.

Let me please emphasize that I'm fully aware that I will need better
disks or a new server, or get rid off VERP in the long run. I was just
thinking that the addition of a tunable like that _might_ be a
reasonable thing to propose if one absolutely wants to reduce the
number of I/O operations during a "bulkdeliver" without having to set
mm_cfg.SMTP_MAX_RCPTS to <= 0. But I think having one more option (and
I don't think this is hard to code, even I could do that and I have
read my first lines of Python on Friday evening) to adapt Mailman (or
it's delivery strategy) to a local site's needs would not really hurt.
If this option defaults to false, nobody would even notice it.

OTOH, it is one configuration parameter more, and one should not
easily include these things to avoid unnecessary configuration
complexity.

If you feel this is an unreasonable request, please do say so. If not,
I'd be glad to get some feedback from the people on this list
regarding "optimal" delivery strategies to the local MTA in this (or
other) cases.


Cheers
Stefan

[1] This was meant to be a joke. Just to be safe.
-- 
Stefan Förster     http://www.incertum.net/     Public Key: 0xBBE2A9E9
"Wenn es sich um Geld handelt, gehört jeder der gleichen Religion an."
    -- Voltaire