Efficient final message disposition (was Re: [Mailman-Developers] Requirements for a new archiver)

[Brad Knowles]

At 8:20 PM -0500 2003/10/30, J C Lawrence wrote:

>  While I don't disagree, this is really an MTA's job, not Mailman's.
>  This is why I've been doing log analysis of MXes and routing mail to
>  customised outbound MTAs on the basis of responsiveness, since early
>  2000.  Adaptive MX routing is great stuff.

	There is a need for this function, and no MTA available today 
does it.  MLMs throughout the history of the Internet have 
incorporated a variety of features for SMTP performance enhancement 
that are unique to mailing lists or are usually found primarily in 
mailing lists, and this is no different.

	If you want to externalize all these functions outside of 
mailman, that's fine.  But then someone has to pick up the ball and 
start hacking on bulk_mailer or some other program to provide these 
features.

>  Yup.  I did it at the first level with an initial SMTP proxy which
>  routed based on MX response records pulled from a DB.

	Again, this is a feature which is not found on any MTA available 
today, and which is known to have a huge impact on mailing list 
performance.  This feature needs to be provided somewhere, by someone.

>  I'm generally of the view that Mailman should do opportunistic domain
>  sorting and per-MTA customised VERP handoffs (because nobody has
>  standardised VERP across MTAs), and beyond that to back off.  Mailman's
>  job is to get the outbound mail into the MTA's spool as quickly as
>  possible, wrapped in transactions (ie RCPT TO bundles) that are friendly
>  to efficient processing, and that's it.

	If you go back to Barry's message, he was talking about getting 
even further involved, by doing a mail-merge process.  Since there is 
no MMTP (something that Bryan Costales, Eric Allman, and I had worked 
on for a while, before we realized that it would just make the spam 
problem worse and then dropped all further efforts), there is a need 
for an intermediate program that is called by mailman and then hands 
the messages off to the MTA.

	Either that intermediate program can be provided by mailman 
itself, or it can come from a third party.  But it needs to come from 
somewhere.

>  We're not in the game of second guessing the MTAs.  That way lies wasted
>  time and madness.

	If there were MLTAs which were optimized for this function, I 
would agree with you.  Since we're trying to take standard MTAs which 
may have only some optimizations that might be generally applicable 
to most situations (including mailing lists), I must disagree.


	For the mailing list specific optimizations that we know are not 
provided by many common MTAs or MTA versions, we need to perform 
those optimizations before the message gets to the MTA.

	We also need to be able to selectively turn them off, in the case 
that there are MTAs that can do that specific job themselves and 
don't need our interference.

>  Where Mailman's performance hurts is in the handling of the list
>  configs, especially for lists with very large memberships rosters and in
>  queue runner performance and overhead (try watching queue runner's
>  system resource profile in v2.1 for lists with > 50,000 members).  For
>  me those are the obvious low hanging fruit,

	You should definitely go after the low-hanging fruit when you 
can.  However, you also have to consider how much work would go into 
fixing those problems.

	A high priority item that would require re-engineering the entire 
system is something that should be planned for the long term, perhaps 
in conjunction with other things that would likewise require 
significant re-engineering efforts as well.

	Meanwhile, if there are other performance issues that can be 
addressed which do not require such significant re-engineering, those 
should be given serious consideration in the shorter term.

-- 
Brad Knowles, <brad.knowles at skynet.be>

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)