[Mailman-Developers] Adding headers to mailman generated mails

[Somuchfun]

Brad,
I did some testing with a 50,000 members list and the time it takes to send
when using personalization is twice as long (8hrs instead of 4).
Of course that is bad news. I also tested some other mailing list software
like Gordano's Communicator and Lyris Listmanager and they both seem to be
able to send in batches but still add or include personalized fields to
trace the messages.
So if they can do it my simple question is why cannot mailman do it but
still send in batches?

> -----Original Message-----
> From: Brad Knowles [mailto:brad.knowles at skynet.be] 
> Sent: Thursday, January 22, 2004 3:21 PM
> To: Barry Warsaw
> Cc: Brad Knowles; Somuchfun; mailman-developers at python.org
> Subject: RE: [Mailman-Developers] Adding headers to mailman 
> generated mails
> 
> At 5:50 PM -0500 2004/01/22, Barry Warsaw wrote:
> 
> >  Nigel can provide details, but I think the same embedding 
> feature could
> >  be used to have the MTA do the final stitching of content 
> template and
> >  personalized data.  It would be A Project to hack 
> together, but I think
> >  it could be a neat idea to play with, although I'm not 
> sure how much it
> >  would help.
> 
> 	This is similar to what Eric Allman (at that time, before 
> Sendmail Inc. existed), Bryan Costales (at the time, working for 
> InfoBeat/Mercury Mail), and I (working at AOL) were discussing back 
> in 1996, in the creation of a Mail-Merge Transport Protocol (MMTP) 
> server, based on a modified version of sendmail along with a standard 
> language for transmitting that content.  With MMTP servers on both 
> ends, it would not matter how many thousands or millions of 
> recipients you might have, only one copy of the message body would be 
> transmitted, and all the rest would be filled in on the remote end.
> 
> 	We ultimately gave up on this idea because we realized that it 
> would make the spam problem much, much worse.  The same things that 
> help regular MTAs transmit millions of customized messages per hour 
> to their paying customers would probably allow spammers to transmit 
> billions of messages per hour to everyone in the universe.
> 
> 
> 	Certainly, before any serious discussion of creating something 
> like an MMTP server, and trying to make that a standard which you 
> would expect programs like sendmail, postfix, and Exim to implement, 
> I believe that the spam issue needs to be addressed.  You need to be 
> able to prove how this cannot be abused to generate spam instead.
> 
> >  If the MTA could do what Mailman does here -- not creating 
> a disk image
> >  for each instance of the message, but stitching it 
> together in member as
> >  it's going out on the wire -- I think you'd greatly improve disk
> >  contention.
> 
> 	I'm not sure that the MTA could safely do that in memory.  At 
> least, it would be difficult to ensure that the MTA gets this done 
> right.  This would be akin to handling the entire message queue in 
> memory for all messages, something which can't really be done safely 
> except under very strict circumstances.
> 
> 	The only MTA I know of that is capable of doing things 
> like this 
> is the latest release of version 8 sendmail, and even then it 
> defaults to handling messages in memory that are no larger than 4KB.
> 
> 
> 	Yes, filesystem I/O is the number one killer, specifically 
> synchronous meta-data updates.
> 
> 	But then people on this list have heard me harping on this 
> subject for a long time, and should know by now that I will refer 
> them to Nick Christenson's book _Sendmail Performance Tuning_ (see 
> <http://www.jetcafe.org/~npc/book/sendmail/>), or my own slides from 
> an invited talk entitled "Sendmail Performance Tuning for Large 
> Systems" at 
> <http://www.shub-internet.org/brad/papers/sendmail-tuning/>.
> 
> >  I don't think Postfix has the same embedding capabilities, 
> although I
> >  haven't looked at what Postfix 2.1 may provide.
> 
> 	I'm not aware of anything like this, but I'd have to check.
> 
> >  In a sense, that's what we've talked about before.  If there were a
> >  standard language that the mail server and list manager 
> could agree on
> >  for both defining the template, and defining the per-recipient data
> >  source, we could have a more efficient mechanism, with 
> perhaps a hope of
> >  mta agnosticism.
> 
> 	That would be nice.  However, I fear that we have much 
> more basic 
> problems that are much more serious, and which need to be resolved 
> before we can expect to start worrying about such subjects as 
> increasing efficiency in the interfaces between MLMs and MTAs.
> 
> -- 
> Brad Knowles, <brad.knowles at skynet.be>
> 
> "They that can give up essential liberty to obtain a little temporary
> safety deserve neither liberty nor safety."
>      -Benjamin Franklin, Historical Review of Pennsylvania.
> 
> GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ 
> !E-(---) W+++(--) N+
> !w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) 
> X++(+++) R+(+++)
> tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- 
> r---(+++)* z(+++)
>