[Mailman-Developers] very large lists

[Mark Sapiro]

Ian Eiloart wrote:
>
>We got huge improvements delivering mail when we introduced parallel q 
>runners. Our problem was that delivery to small but time sensitive lists 
>was held up when someone sent mail to a large list. So, it still takes a 
>while to deliver to a large list, but other jobs aren't held up.


It is only partly true that other jobs are not held up. When you slice
a qrunner, you don't get "one queue/many servers". You get "many
queues/many servers" any specific message still has a 1/n chance of
being behind one with a long service time.

I note that you have 32 outgoing runner slices, so it's probably rare
that a message is waiting in the same queue slice as the one to 10,000
recipients, but it can happen.


Bob,

As far as the millions of recipients is concerned, with a default
Mailman 2.x. the size of the lists/LISTNAME/config.pck is going to be
huge. I think you'd need to break it into several subset lists and an
umbrella. I also think you'd want to investigate alternate
MemberAdaptors, although the MySQL ones (see thread at
<http://mail.python.org/pipermail/mailman-developers/2008-September/020363.html>
are the only ones I am aware of that support bounce processing.

With some changes like this, it might be feasable in Mailman 2.1/2.2.
This is somewhat uncharted territory, all though 100K lists are not
unheard of.  See the FAQ at <http://wiki.list.org/x/NoA9>.

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan