[Mailman-Users] Mailman performance / sends per hour

[Brad Knowles]

At 9:43 AM -0400 2003/07/26, Jon Carnes wrote:

>  Here is the section from the Release Notes that is pertinent to our
>  "pissing contest":
>
>     Add parallel queue runner code.  Allows multiple queue runners per work
>  		group (one or more queues in a multi-queue environment
>  		collected together) to process the same work list at the
>  		same time.

	I said:

>>  	Actually, what postfix does is handle multiple copies of the
>>  message being transmitted to separate domains in parallel.  This
>>  helps ensure that fast sites further down the list don't get hung up
>>  by slower sites that come earlier.  However, there is a limit to this
>>  parallelism.  Mailman could help this process by tracking the average
>>  delivery time per recipient, and then sorting the recipient list when
>>  handing the messages to postfix -- fastest first, slowest last.

	You responded:

>  Actually Brad, it looks like your knowledge of Sendmail is rather
>  dated. Sendmail has been doing this since 2001.
>
>   http://www.sendmail.org/~ca/email/doc8.12/RELEASE_NOTES

	You just said "this".  You didn't say what "this" you were 
talking about.  I (naturally) assumed you meant to reference the most 
recent part of the paragraph you were responding to, which was the 
issue of the recipient sorting feature based on previous average 
delivery times -- a feature that neither sendmail nor postfix has, 
and which would be a notable improvement for mailman.


	If you had wanted to refer to the issue of having multiple queue 
runners going in parallel, you should have trimmed the paragraph at 
the appropriate point, or you should have been more specific.

	Either way, I would then have mentioned that this feature did not 
work correctly when originally added to the system in version 8.10 
(dated 2000/03/01) with "multiple queue directories":

         Support multiple queue directories.  To use multiple queues, supply
                 a QueueDirectory option value ending with an asterisk.  For
                 example, /var/spool/mqueue/q* will use all of the
                 directories or symbolic links to directories beginning with
                 'q' in /var/spool/mqueue as queue directories.  Keep in
                 mind, the queue directory structure should not be changed
                 while sendmail is running.  Queue runs create a separate
                 process for running each queue unless the verbose flag is
                 given on a non-daemon queue run.  New items are randomly
                 assigned to a queue.  Contributed by Exactis.com, Inc.

	The problem is that there was no way you could keep sendmail from 
firing off a queue runner per multiple queue, and if you wanted to 
have a large directory hierarchy of multiple levels of queues with 
lots of queues at each level (a la postfix, but better), this could 
cause thousands or millions of queue runners to be started -- 
obviously, a highly undesirable result.  This caused no end of 
problems for us at a previous employer, end I ended up turning off 
all of sendmail's control over multiple queues and managing them 
myself.

	I was also an early adopter of running multiple queue runners in 
the same directory, some with "QueueSortOrder=host", some with 
"QueueSortOrder=time", some with "QueueSortOrder=random", etc... so 
as to try to clear the queue as best as possible but with as little 
lock contention as possible.  I was doing this from ~1995.


	Besides, with limiting the number of recipients per envelope 
(either within the MLM or within the MTA) and then allowing multiple 
processes to handle each chunk separately, you get pretty much the 
same effect.

>  Re-read my statement above and maybe look into the old archives of this
>  list.  You will discover that due to Mailman's current design (well
>  really a limitation of Python) large lists can be very slow to maintain
>  and process.  the solution is to move the MAILMAN (not sendmail you
>  oaf!) list databases into a RAM drive.  And (duh!) a battery backed up
>  RAM drive sure would be best.

	List databases or the MTA mqueues, it doesn't matter.  The same 
requirement for reliability is there.  If you choose to be so casual 
with the management of your mailing lists, it sure makes me wonder 
about your competence in other areas.

-- 
Brad Knowles, <brad.knowles at skynet.be>

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)