[Mailman-Users] Options for increasing throughput

[Fletcher Cocquyt]

Mike, many thanks for your (as always) very helpful response - I added the 1
liner to mm_cfg.py to increase the threads to 16.
Now I am observing (via memory trend graphs) an acceleration of what looks
like a memory leak - maybe from python - currently at 2.4

I am compiling the latest 2.5.2 to see if that helps - for now the
workaround is to restart mailman occasionally.

(and yes the spamassassin checks are the source of the 4-10 second delay -
now those happen in parallel x16 - so no spikes in the backlog...)

Thanks again


On 6/20/08 9:01 AM, "Mark Sapiro" <mark at msapiro.net> wrote:

> Fletcher Cocquyt wrote:
> 
>> Hi, I am observing periods of qfiles/in backlogs in the 400-600 message
>> count range that take 1-2hours to clear with the standard Mailman 2.1.9 +
>> Spamassassin (the vette log shows these messages process in an avg of ~10
>> seconds each)
> 
> 
> Is Spamassassin invoked from Mailman or from the MTA before Mailman? If
> this plain Mailman, 10 seconds is a hugely long time to process a
> single post through IncomingRunner.
> 
> If you have some Spamassassin interface like
> <http://sourceforge.net/tracker/index.php?func=detail&aid=640518&group_id=103&
> atid=300103>
> that calls spamd from a Mailman handler, you might consider moving
> Spamassassin ahead of Mailman and using something like
> <http://sourceforge.net/tracker/index.php?func=detail&aid=840426&group_id=103&
> atid=300103>
> or just header_filter_rules instead.
> 
> 
>> Is there an easy way to parallelize what looks like a single serialized
>> Mailman queue?
>> I see some posts re: multi-slice ­ but nothing definitive
> 
> 
> See the section of Defaults.py headed with
> 
> #####
> # Qrunner defaults
> #####
> 
> In order to run multiple, parallel IncomingRunner processes, you can
> either copy the entire QRUNNERS definition from Defaults.py to
> mm_cfg.py
> and change
> 
>     ('IncomingRunner', 1), # posts from the outside world
> 
> to
> 
>     ('IncomingRunner', 4), # posts from the outside world
> 
> 
> which says run 4 IncomingRunner processes, or you can just add
> something like
> 
> QRUNNERS[QRUNNERS.index(('IncomingRunner',1))] = ('IncomingRunner',4)
> 
> to mm_cfg.py. You can use any power of two for the number.
> 
> 
>> I would also like the option of working this into an overall loadbalancing
>> scheme where I have multiple smtp nodes behind an F5 loadbalancer and the
>> nodes share an NFS backend...
> 
> 
> The following search will return some information.
> 
> <http://www.google.com/search?q=site%3Amail.python.org++inurl%3Amailman++%22lo
> ad+balancing%22>

-- 
Fletcher Cocquyt
Senior Systems Administrator
Information Resources and Technology (IRT)
Stanford University School of Medicine

Email: fcocquyt at stanford.edu
Phone: (650) 724-7485