Efficient final message disposition (was Re: [Mailman-Developers] Requirements for a new archiver)

[Barry Warsaw]

Ok, I'm beat up enough, so let me open things up to a hopefully more
productive thread.  How can Mailman more efficiently hand off messages
to a local mail server for final delivery?

Some problems with the current approach include:

- The desire/requirement that Mailman chunk and sort recipients

- The ability for Mailman to swamp the mail server or cause the mail
server to consume all available cpu

- The fact that failures in upstream mail server are reported to Mailman
as bounces instead of as error codes

- Inefficiencies in VERP/personalization/mail-merge because of the lack
of cooperation

- The need for Mailman to queue outgoing messages that aren't completely
delivered

I'm sure you guys can identify more issues <wink>.  Look at the
complexity in SMTPDirect.py, and even there, we still have problems.

So how do we design a system where we can push the complexity and
efficiency concerns out past our boundary?  Here's a rough sketch of
what I'd like:

Mailman has a list of recipients, or at least knows how to calculate
that list.  It has a message template as encoded 7-bit ascii.  It has a
dictionary (association table, hash table) of substitution placeholders
to values for each recipient, or knows how to calculate that.

Mailman wants to simply hand that data off to some agent and forget
about it.  It wants to know that the agent will make best effort to mail
merge and deliver.  It wants to be informed of any final delivery
failures.  And that's it.  Mailman doesn't want to chunkify recipients,
and it doesn't want to sort them.  It doesn't want to worry about a mail
server effectively managing system resources.  I'd rather not have to
hand it a couple of meg of recipient or substitution data, but there
seems to be no other way.

So what can we do here to improve matters?

-Barry