[Mailman-Users] Mailman performance / sends per hour

[Brad Knowles]

At 5:01 PM -0400 2003/07/25, Jon Carnes wrote:

>  Right! Given decent equipment the MTA is the primary worry.  For best
>  performance you really want to use an optimized MTA like Postfix.
>  Postfix shuffles slow responding mail sites to the end of the queue so
>  that they don't hold up the outflow of mail.  The increase in efficiency
>  this gives you is tremendous and far outweighs any other gains.

	Actually, what postfix does is handle multiple copies of the 
message being transmitted to separate domains in parallel.  This 
helps ensure that fast sites further down the list don't get hung up 
by slower sites that come earlier.  However, there is a limit to this 
parallelism.  Mailman could help this process by tracking the average 
delivery time per recipient, and then sorting the recipient list when 
handing the messages to postfix -- fastest first, slowest last.

	You could further tune this process by making the initial 
delivery attempt time out very quickly (like, five seconds instead of 
the typical two minutes), and then have "reaper" processes set to 
retry delivery with a more normal timeout.  But this is an MTA tuning 
process, and would differ depending on which MTA you use.

	Overall, postfix is pretty well optimized for use with mailing 
lists, but the architecture does guarantee a certain minimum amount 
of overhead that cannot be escaped.  With some work, you can create a 
sendmail configuration that will avoid even this level of overhead, 
and can be considerably faster.  But it does take more work to get 
there, whereas postfix will do a pretty good job out-of-the-box.

>  The next best thing you can do is use a nice fast LVD disk subsystem
>  (maybe even a RAID) for your /var volume. MTA's follow a specification
>  that ensures the delivery of mail, even if the server goes down.  This
>  means that each and every transaction is written somewhere on disk.  So
>  moving mail around takes a very large number of read and writes! We say
>  that mail servers are disk I/O bound.

	Increasing filesystem performance is a big issue.  The best 
choice here is with a *BSD operating system and "softupdates", 
because that will avoid writing files to disk if they are created and 
then deleted very quickly (they're safely queued in memory before 
being written to disk, and the creat() is removed if the file is 
unlink()'ed before the data has been flushed).

	Using Linux-style asynchronous writes will cause the data to 
still be written to disk, even if in a more efficient manner.  This 
is not as good as avoiding the disk operations entirely.

	But a Journaling extent-based filesystem with a hashed directory 
structure (such as SGI XFS) would be a big step forward over what 
most people are likely to have.

>  So, you really didn't give us enough information to guess at how well
>  your server will do. Still you can probably count on at least 2k
>  messages/minute if you have a SCSI disk subsystem and use something like
>  Postfix.

	Certainly, a high-speed multi-user oriented disk subsystem would 
also help, and here SCSI is preferred over ATA/EIDE (it's the 
multi-user part that SCSI does so much better).

>  BTW: if you choose to use Sendmail, I have some tweaks that let it run
>  in an quasi-optimized mode (similar to Postfix).  You will find them
>  described here:
>    http://www.trilug.org/~jonc/mailserver/PartIII.html

	Anybody who is serious about this topic should read the book 
_Sendmail Performance Tuning_ by Nick Christensen (see 
<http://www.jetcafe.org/~npc/book/sendmail/>).  Of course, I'm a bit 
biased, since I did the first public talk on this topic at SANE'98 
(see <http://www.shub-internet.org/brad/papers/sendmail-tuning/>) 
which Nick quickly followed (see 
<http://www.jetcafe.org/~npc/doc/performance_tuning.pdf>), and I was 
a technical reviewer of his book.

	With regards to the tuning of MTAs for performance specifically 
with mailing lists, make sure you read Rob Kolstad's paper "Tuning 
Sendmail for Large Mailing Lists" at 
<http://www.usenix.org/publications/library/proceedings/lisa97/21.kolstad.html> 
and Strata Chalup's paper "Drinking from the Fire(walls) Hose: 
Another Approach to Very Large Mailing Lists" at 
<http://www.usenix.org/publications/library/proceedings/lisa98/chalup.html>. 
Both of these papers were written before mailman existed (and before 
VERP existed), but most of the issues raised and suggestions made are 
still relevant today.

-- 
Brad Knowles, <brad.knowles at skynet.be>

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)