[Mailman-Users] Umbrella List Spam Protection

[Links at Momsview]

JC,
Thanks for the suggestion and the link.
This is what I'm seeing with my single list of 140,000 subs

(1) MTA performance is not a bottleneck
OutgoingRunner is sending to Postfix at between 15-30 emails /sec.  Since
the mail queue never grows beyond 1000 (from failed deliveries) I'm assuming
that Mailman (or the I/O to /var/log/maillog) is the bottleneck limiting
performance.  In any case I'm more than satisfied with  900-1800
emails/minute.

(2) Excessive Memory usage
BounceRunner, OutgoingRunner, CommandRunner, ArchRunner, IncomingRunner and
VirginRunner are EACH using 120-135 Megabytes of Memory.  That's over
700Mbytes RAM just to load these processes (and periodically trigger swap on
my 1.5GB 2GHZ P4).  I've been able to track the memory usage to  the use of
a Python dictionary  to hold the mailing list  (the extra memory required
for the hash space probably results in 500% memory overhead compared to a
Python list structure).

(3) Excessive Processing Time
BounceRunner was processing over 80 bounces per minute peak with a list size
of 48,000 subs.  That went to 40 bounces per minute peak at 89,000 subs and
25 per minute at 138,000. I believe this is due to the CPU time and disk I/O
required to write out 17 megabytes of data for every bounce (at 142,000
subs).  CommandRunner is showing a similar drop in performance probably due
to the same reasons

If there is another approach I should take PLEASE PLEASE let me know.  But
from what I'm seeing, short of re-writing a lot of code, splitting the list
is my only hope of reducing both the memory usage and the processing time.
John

----- Original Message -----
From: "JC Dill" <lists at vo.cnchost.com>
To: "Links at Momsview" <links at momsview.com>
Cc: <mailman-users at python.org>
Sent: Wednesday, February 12, 2003 3:36 PM
Subject: Re: [Mailman-Users] Umbrella List Spam Protection


> Links at Momsview wrote:
> > I currently have a VERY large ( >100k)  announce only list (always
> > moderated) running mailman  2.1 under Postfix.
> >
> > For performance reasons I'm considering splitting this list into 26
sublists
> > based on the first letter of each subscriber email.
> > I would then use an umbrella list to send to all of the sublists at
once.
> >
> > As I understand it, unless I want to separately approve the message for
the
> > 26 sublists I need to allow at least one email address to post
unmoderated
> > on ALL of the sublists.  This is very scary since spammers currently try
to
> > post on my current large list almost daily.
>
> Before you go down this road, be sure you have done everything possible
> in terms of selecting and configuring your outgoing mail server to
> handle the volume.  I suspect that handling the outgoing volume will be
> easier and more effective when you direct your attention towards your
> mail server, than setting up sublists on the list server.
>
> FAQ entry 1.15 has more on this topic, and directs you to related topics:
>
> <http://www.python.org/cgi-bin/faqw-mm.py?req=all#1.15>
>
> HTH
>
> jc
>
>
>