[Mailman-Developers] Really slow bounce processing

[Robert Anderson]

I have a rather large mailman mailing list (around 180,000 opt-in subs
over 7-8 lists) have had system problems earlier in the week which
shunted all bounce and subscriber requests because of corrupt pending
lock file.

Well I sorted out the problems but ended up with a HUGE amount of bounce
requests to process (over 25,000).

Problem is that the bounce processing is so slow.  I have tried the
following optimisations but can't get any more than 10 bounce requests
processed per minute:
- mounted locks directory under ramfs
- mounted lists directory under ramfs (tried to see if it improved
performance but it didn't so I changed it back)
- removed all unessential disk writes (commented out a bunch of
syslog's)
- placed debugging code to test and readjust the order of the bounce
processing code in BounceAPI.py to try the most used bounce processes
first.
- tried breaking down the large bounce directory into smaller batches to
make sure the huge directory size wasn't killing the system.
- redirected incoming bounce to "bounce_in" directory so I don't
re-process the same addresses again and again.
- Reduced lock timeout settings.
- Sending out all new system messages using VERP as this seems to reduce
the about of code to process future bounces.
- tested and tried 1,2,4 and 8 bounce runner processes (locks and 100%
CPU usage killed any parallelisation gains and I went back to 1 process)
- Pulled my hair out!!!

My system is a P4 2.0 GHZ with 1.5GB ram and 2x 7200 rpm HDD (lists
folder on second disk to reduce i/o bottlenecks).

Does anyone have any idea's on how to tune bounce processing to get
through this backlog.  I can imagine 5-10 bounce file processes per
minute is right.

I'm hoping I have missed some magical silver bullet that will fix all my
problems.

Any help, assistance, suggestion are very welcome.  while I'm going to
get all the bounces processed over the weekend, I'm not looking forward
to the disabled reminder processing in a weeks time!!!

Cheers		
-- 
Robert Anderson

Webmaster
http://www.Aarons-Jokes.com
+64 21 808 525