[Mailman-Users] Large Lists manipulations

[Jon Carnes]

Let's not solve the wrong problem here.  The problem as stated is that it
takes too long to edit the config.db file.  So long in fact, that it can no
longer be done via the web interface.
Contributing factors to this are the size of the list, hence the size of the
database, the disk subsystem the database is stored on, and to a minor
degree the CPU and memory of the box.

One possible solution is to split the database into smaller parts.  You
could create 26 databases and move the users into the database that
corresponded to the first letter of their email.  Then you would change your
old list into an umbrella list that simply relayed mail to these 26 other
lists.
This would be a royal pain to setup, but fairly easy to maintain.
 - Allow folks to subscribe to the main umbrella list. No "welcome" or
password reminders from the main list.
 - Run a script hourly to output then new users of the main list and that
subscribes them to the proper sub-list, and then removes them from the main
list.
 - Each sub-list sends out the welcome and the password reminders.

Of course the easiest solution is to add an LVD SCSI chain to your server
and move the volumes over to that disk subsystem.  This should give a 8x to
20x improvement in speed.

Jon Carnes

----- Original Message -----
From: "J C Lawrence" <claw at kanga.nu>
To: "Tass" <tass at kenderhome.com>
Cc: <mailman-users at python.org>
Sent: Wednesday, December 05, 2001 12:59 PM
Subject: Re: [Mailman-Users] Large Lists manipulations


> On Wed, 5 Dec 2001 08:28:53 -0500 (EST)
> tass  <tass at kenderhome.com> wrote:
>
> > SCSI alas is not an option (which is where I think the bottleneck
> > is).  It is an 80GB IBM EIDE hardrive with UDMA66 Celeron 500
> > 686class 512M Ram
>
> Its a little difficult to say much without knowing how much list
> traffic you'd be dealing with or what your outbound bandwidth is.
> Assuming reasonable outbound bandwidth (T1+), a posting rate of
> ~10 messages a day, and ~10% slow MX'es, I'd:
>
>   -- set SMTP_MAX_RCPTS to be reasonably large (in the 50 - 100
>   range)
>
>   -- make sure I had a local cacheing nameserver installed
>   (recommend: pdnsd)
>
>   -- tune my MTA to be sensitive to system load.  Using Exim as the
>   MTA would make this particularly easy.
>
>   -- tune the MTA for rapid fall-offs for slow MX'es and hard
>   bounces no later than 4 days (as per RFC recommendation).
>
> Note that running a 10K member list on such hardware is not
> inherently a problem.  It can and should work quite nice without
> much stress or strain.  It all really depends on two  things:
>
>   1) How busy the list is
>
>   2) What your percentage of slow MX'es is (which controls what your
>   average queue size is).
>
> The less saturated the system is, the more attractive Postfix is as
> an MTA.  Nicely enough, Postfix is fast enough and clever enough in
> its spool handling that it will delay the saturation point
> significantly.  However, if you are at or near saturation point the
> Exim's queue handling will be a lot nicer to you and your system,
> with its graceful fall-offs for system load.
>
> > I have about 80% disk cap still open
>
> If you can, throw another disk in there, then put /var/spool/<MTA>
> on it and put it on a __different___ IO chain from your main drive.
> This will reduce IO and head contention for your mail spool.
>
> --
> J C Lawrence
> ---------(*)                Satan, oscillate my metallic sonatas.
> claw at kanga.nu               He lived as a devil, eh?
> http://www.kanga.nu/~claw/  Evil is a name of a foeman, as I live.
>
> ------------------------------------------------------
> Mailman-Users maillist  -  Mailman-Users at python.org
> http://mail.python.org/mailman/listinfo/mailman-users