[Mailman-Users] Mailman performance / sends per hour

[Jon Carnes]

On Fri, 2003-07-25 at 17:36, Brad Knowles wrote:
> At 5:01 PM -0400 2003/07/25, Jon Carnes wrote:
> 
> >  Right! Given decent equipment the MTA is the primary worry.  For best
> >  performance you really want to use an optimized MTA like Postfix.
> >  Postfix shuffles slow responding mail sites to the end of the queue so
> >  that they don't hold up the outflow of mail.  The increase in efficiency
> >  this gives you is tremendous and far outweighs any other gains.
> 
> 	Actually, what postfix does is handle multiple copies of the 
> message being transmitted to separate domains in parallel.  This 
> helps ensure that fast sites further down the list don't get hung up 
> by slower sites that come earlier.  However, there is a limit to this 
> parallelism.  Mailman could help this process by tracking the average 
> delivery time per recipient, and then sorting the recipient list when 
> handing the messages to postfix -- fastest first, slowest last.
> 
Actually Brad, it looks like your knowledge of Sendmail is rather dated.
Sendmail has been doing this since 2001.

  http://www.sendmail.org/~ca/email/doc8.12/RELEASE_NOTES

Postfix has some very interesting features that make it much better to
use than Sendmail, but the one that sets it most apart in added
efficiency is its default queueing structure.

I do agree with you though, that if the MTA (or Mailman) could
periodically sweep the MTA delivery logs and sort the domains from
fastest to slowest, there would be an increase in efficiency.

I regularly run MTA reports that give me these stats and there are easy
to see regularities in the list (the slow staying slow, the fast always
finishing near the top). 

> 	You could further tune this process by making the initial 
> delivery attempt time out very quickly (like, five seconds instead of 
> the typical two minutes), and then have "reaper" processes set to 
> retry delivery with a more normal timeout.  But this is an MTA tuning 
> process, and would differ depending on which MTA you use.
> 
This is the Timeout.iconnect feature (if you use M4 to setup Sendmail).

> 	Overall, postfix is pretty well optimized for use with mailing 
> lists, but the architecture does guarantee a certain minimum amount 
> of overhead that cannot be escaped.  With some work, you can create a 
> sendmail configuration that will avoid even this level of overhead, 
> and can be considerably faster.  But it does take more work to get 
> there, whereas postfix will do a pretty good job out-of-the-box.
> 
> >  The next best thing you can do is use a nice fast LVD disk subsystem
> >  (maybe even a RAID) for your /var volume. MTA's follow a specification
> >  that ensures the delivery of mail, even if the server goes down.  This
> >  means that each and every transaction is written somewhere on disk.  So
> >  moving mail around takes a very large number of read and writes! We say
> >  that mail servers are disk I/O bound.
> 
> 	Increasing filesystem performance is a big issue.  The best 
> choice here is with a *BSD operating system and "softupdates", 
> because that will avoid writing files to disk if they are created and 
> then deleted very quickly (they're safely queued in memory before 
> being written to disk, and the creat() is removed if the file is 
> unlink()'ed before the data has been flushed).
> 
> 	Using Linux-style asynchronous writes will cause the data to 
> still be written to disk, even if in a more efficient manner.  This 
> is not as good as avoiding the disk operations entirely.
> 
> 	But a Journaling extent-based filesystem with a hashed directory 
> structure (such as SGI XFS) would be a big step forward over what 
> most people are likely to have.
> 
> >  So, you really didn't give us enough information to guess at how well
> >  your server will do. Still you can probably count on at least 2k
> >  messages/minute if you have a SCSI disk subsystem and use something like
> >  Postfix.
> 
> 	Certainly, a high-speed multi-user oriented disk subsystem would 
> also help, and here SCSI is preferred over ATA/EIDE (it's the 
> multi-user part that SCSI does so much better).
> 
> >  BTW: if you choose to use Sendmail, I have some tweaks that let it run
> >  in an quasi-optimized mode (similar to Postfix).  You will find them
> >  described here:
> >    http://www.trilug.org/~jonc/mailserver/PartIII.html
> 

For larger lists and Mailman, I have found that nothing beats using a
RAM disk and accessing the list database files via the mounted RAM disk.
The speed increase can be 100x faster.

Take care and again, good luck!

Jon Carnes