[Mailman-Users] Running large announce list & forums

[Brad Knowles]

At 8:11 PM -0700 2006-09-06, Jack Stone wrote:

>  I'll have to check on the things you suggested, but just a quick
>  response without more specifics, I've noticed it takes a long, long
>  time for the announcement list to reach the sendmail queue -- and
>  the "qfiles/out" is usually choked with queued messages for quite a
>  while. It appears the discussion list has to wait in line..

In that case, it seems likely that your delay is in the Mailman queue 
management process, and not within sendmail.  In that case, I 
recently did a slight cleanup on some good stuff that Nigel 
Metheringham put into the FAQ Wizard for FAQ 6.6, and I'd recommend 
you take a look at it.

You may also want to look into running multiple queue runners. 
Within sendmail, this is not too difficult, but this is a pretty 
advanced concept for Mailman.  IIRC, Barry has had some things to say 
about this in the archives of the mailman-users or mailman-developers 
mailing lists, and I'd encourage you to search the archives.

>  By past experience with majordomo, it would "cook" the big one-way
>  announcement list for a couple hours, but still send out the
>  discussion postings (plus regular user mail) immediately without delay
>  -- and although the dam would burst into sendmail queue from the
>  eventual fully-cooked (batched) announcements (10,000+), it would
>  still pass all other requests to sendmail's queue.

The issue here is that Majordomo didn't have a separate queue 
management process for handling mail as it was being processed 
through the system.  Instead, they would fork a separate Majordomo 
process for each message that was posted, and then that process would 
be responsible for handling everything to do with processing that 
message, up to the point where everything was handed off to sendmail.

The Majordomo approach has good points and bad ones.  One bad point 
is that it requires a hell of a lot more memory because you can have 
a large number of simultaneous processes in memory at any given time, 
and each one of those processes could grow to gargantuan sizes for 
even a moderately sized list.  In addition, classic Majordomo didn't 
do any chunking of the messages being sent, which causes all the 
outgoing copies of a given incoming message to be single-threaded, 
and a slow site in the middle of the list of recipients could cause 
mail for all the following recipients to be greatly delayed.

Rob Kolstad and Strata Chalup both talked about tuning Majordomo and 
sendmail for large mailing lists in their respective papers, which I 
linked from FAQ 6.3.  If you really want to know the deep internals 
of how these things work (or sometimes don't work), I'd encourage you 
to read those papers.


In the long run, the Majordomo approach is not very scalable.  IMO, 
Mailman is more scalable, but because it has it's own internal queue 
processing system, it places different demands on the system and 
requires different types of tuning for maximum performance.

I've tried to collect as much wisdom as I can in the various FAQ 
entries which have the keyword "performance" in them, but the 
internal tuning of Mailman for maximum performance is still a pretty 
black art for most of us -- I can tune sendmail or postfix with the 
best of them, but I'm way out of my depth when it comes to tuning 
Mailman itself -- that's the kind of thing for which I'd have to turn 
to Barry.

>  I know this is not majordomo, just looking for anything I may have
>  mis-configured that may cause this delay. Sendmail has always handled
>  my loads with ease -- all mail lists, plus individual users.

I'm not convinced that there is anything Majordomo-specific that you 
may have done to your server(s) that causes it/them to be less 
suitable for use with Mailman, but then I don't know everything 
you've done to your server(s).

I can tell you, with some confidence, that I know of some areas 
within Mailman that I believe could be significant bottlenecks, and 
may take a fair bit of work to resolve.

-- 
Brad Knowles, <brad at stop.mail-abuse.org>

"Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety."

     -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
     Assembly to the Governor, November 11, 1755

  Founding Individual Sponsor of LOPSA.  See <http://www.lopsa.org/>.