[Mailman-Developers] Re: To VERP or not to VERP?

[Chuq Von Rospach]

On Sunday, June 17, 2001, at 03:20 AM, Fil wrote:

>  My
> systems delivers 10K messages at the mean rate of 18000 per hour. When I
> send a message down a 130 000 subscribers' list, well, I have to wait 
> quite
> long befoore it's gone ; maybe I should buy some more bandwith.

It depends on how time sensitive the stuff is and when you send it. If 
you send out at 10PM and expect people to read it when they wake up, 
does it matter if they actually get it at 3AM or 6AM? But if it's time 
sensitive -- you have to worry about that issue. If the delay adds a few 
minutes, no big deal (but if it adds a few minutes * 100 messages a day; 
big deal, especially for the messages at the end of the day where you 
can get the "bad weather over O'hare" problem...)

> However there's another performance issue: how does postfix react when 
> it's
> got sent 130 000 emails to store and forward?

Disk I/O. That's the other issue here. You're adding to yoru network 
usage, you're also adding to your Disk I/O problem.

>  Currently I send it about
> 10000 messages that it breaks up one by one using. I don't know about 
> memory
> or disk issues there, but 130 000 * 10k = 1.3 Gbytes on disk:

It's a problem. network and disk are (IMHO) the two big performance 
issues in delivering e-mail (at least the two under your control. The 
third is the speed at which receiving machines will accept messages, but 
you can't buy everyone in the universe faster e-mail servers...)

the amount of disk it takes isn't an issue (within reason) -- remember, 
it's going to start sending right away, so message will be gone out of 
the queue. You don't queue it all up and then start delivery. But this 
generates disk IO, and that can start bogging you down if you don't tune 
things properly. That means taking advantage of sub-folders in your mail 
queue for any MTA that allows them (the #1 performance death for a 
typical sendmail system: write-locks on /var/spool/mqueue, since every 
sendmail process has to create files in that directory. If you haven't 
set up subfolders, you are (I say this in a nice way) an idiot. if you 
aren't using a version of sendmail that allows for them, I'll call you 
an idiot in a not-nice way. Anyone not running at least 8.10 is hosed, 
so forget them...

Most MTAs are sensitive to this and work to minimize the impact. Even 
sendmail finally figured it out. But you still need, in large e-mail 
environments, to look at splitting this across heads and spindles. My 
experiments have indicated you're better off having mail on separate 
spindles than you are building a RAID using those same spindles, for 
whatever that's worth. And if you have lots of RAM, you can start using 
ram disks, and then you have lots of fun... (yes, I've done that. It's 
amazing how much faster sendmail is when you remove the disk I/O on 
those directory inodes...)


--
Chuq Von Rospach, Internet Gnome <http://www.chuqui.com>
[<chuqui@plaidworks.com> = <me@chuqui.com> = <chuq@apple.com>]
Yes, yes, I've finally finished my home page. Lucky you.

95% of being a net.god is sounding persuasive
and convincing people you know what you're talking about, even when 
you're
making it up as you go along. (chuq von rospach, 1992)