[Mailman-Users] Call for suggestions

[J C Lawrence]

On Tue, 08 May 2001 17:55:21 -0600 
Ashley M Kirchner <ashley at pcraft.com> wrote:

> Chuq Von Rospach wrote:
>> And services like yahoo have programmers on staff to write this
>> stuff, and admins on staff to manage it, and budgets for the
>> hardware, and...

> Something I don't have. :) That's why I'm trying to go as light as
> I can, with currently (working) software.

If I were to go for a first order attempt at reliability/scalability
I'd be tempted to do something like (which is actually what we ended
up doing at a previous client):

  Lists run on system X.  

  Outbound list mail is not delivered to the local MTA but to a set
  of remote MTAs hidden behind a DNS round robin (later was a single
  IP address which load shared across multiple systems behind the
  switch)..

  Spool on the MTA systems is on a battery backed RUPP Silicon disk
  (10Gig units IIRC, not cheap).  Big RAM heavy machines capable of
  running mid thousands of simultaneous queue runners (under QMail
  as happens).

  Domain routing based on historical MX profiling partitioned the
  spool base and the resultant spool entries are smarthosted out to
  a pool of delivery servers (more RUPP disks), with the really slow
  MXes being pulled out and dumped on the black hole box (constantly
  overwhalmed and struggling to deliver mail to boxes that were
  mostly not there).

>> How big are these lists? How many messages a day? How many total
>> e-mails are you thinking to send out a day? What's the hardware? 
>> What's the network and the pipe?

> Right now, the single machine running lists is a 200Mhz RISC
> machine, hosting roughly 30 lists, each one with subscriber base
> between 1,500 to 5,000 (only about 5 of those lists actually hit
> the 5,000 mark, the rest are all under 2,500).  Each list is
> generating an average of 20 msgs per hour, most of them being
> between the hours of 6AM and 9PM.  

~500K messages a day.  Not too bad really.

> The machine is sitting on a 100BT hub, connected to a T1.  I don't
> have a problem with the bandwidth generated (considering these are
> small text msgs), however I do have a problem with the system load
> and the time it takes for messages to be spewed back out because
> of load.

I suspect that your real problems are in choice of MTA, MTA
configuration, and lack of a local cacheing name server.

Suggest:

  Toss sendmail for Postfix, Exim, or QMail.  Recommend Postfix.

  Configure the MTA to not do DNS verification for messages received
  from localhost (127.0.0.1).

  Install a local cacheing name server.  Suggest DJBDNS.  

  Check your RAM usage patterns.  Watch swap.  Add RAM if indicated.
  You are *NOT* CPU bound at this point.  You may be disk bound if
  your disks are really slow, syslog is on the same spindle as
  spool, and/or you have syslog configured to sync mail log writes.

> Yes I know, I should probably look into buying a faster machine,
> however I don't have that luxury (at the moment).  I do however
> have other machines at my disposal that I can use, providing I can
> tie them in this (main) one.

The first tep to distributing load would be to use a second system
as the outbound MTA.  Fairly simple to do really.  Either have list
mail spool to the localhost and then smarthost out thru the second
box, or have list mail delivered straight to the second box (both
work).

RAM and disk bandwidth are your enemies at this point.

-- 
J C Lawrence                                       claw at kanga.nu
---------(*)                          http://www.kanga.nu/~claw/
The pressure to survive and rhetoric may make strange bedfellows