[Mailman-Users] Mailman throughput

[Mark Sapiro]

Ivan Fetch wrote:
>
>What is a reasonable / realistic way to benchmark a Mailman installation? Are there details of other similarly sized instlalations and throughput numbers which I can compare?


I don't have any data for a comparable size installation. The benchmark
test you describe seems reasonable to me.

See below for some comments.


>I am sending some test messages through the VM, but don't know whether this is useful or pointless, in telling me how capable the VM is.


It seems this should be useful.


>The VM processed 10000 messages in an hour and 10 minutes. The messages went to two lists, one with 25 recipients and the other with 500 recipients. The VM (Linux) peaked at a load average of 4 (2 VCPUs), and was using 800Mb of it's 2G ERAM for IO caching. I could add more resources, but it doesn't look like they would get used.
>
>Initially (maybe for the first 15-20 minutes) the Postfix queue had to catch up submitting messages to Mailman. After that, qfiles/in was empty, and the work which remained was to process qfiles/out and SMTP the outbound messages to the relay. I raised SMTP_MAX_RCPTS from 500 to 1000, but this did not seem to make a difference.


There are several things going on.

1) Processing mail through Postfix to Mailman. This is almost entirely
Postfix. Delivery is piping the message to the mail wrapper which
involves very little Mailman processing - only making a queue entry
and storing it in qfiles/in. Any tuning would have to be in Postfix,
but I don't know what would be applicable beyond ensuring Postfix has
enough resources do do the job.

2) Mailman's IncommingRunner picking up the message from qfiles/in,
processing in through the pipeline and queuing the result in
qfiles/out and qfiles/archive. Also if the list is digestable, the
message will be added to the list's digest.mbox and possibly a digest
will be triggered on size.

3) Mailman's ArchRunner picking up the message from qfiles/archive and
adding it to the list's archive.

4) Mailman's OutgoingRunner picking up the message from qfiles/out and
delivering it to the outgoing MTA.


>SO I may be able to tune Postfix handing off to Mailman, as well as Mailman handing off to the SMTP relay. Any suggestoins here?


It seems the major hurdle is in processing the 'out' queue. It is
possible to slice OutgoingRunner to provide some parallelism in this
process and that may speed things up, but I suspect that a lot of the
time is in network communications between OutgoingRunner and the
remote Postfix and that slicing OutgoingRunner may not help much, but
it would be worth rerunning your benchmark with 2 or 4 outgoing runner
slices to see if it helps.

Your raising of SMTP_MAX_RCPTS from 500 to 1000 would not have any
effect because your larger list had only 500 members so no outgoing
message had more than 500 recipients. Even if this were not the case,
I don't think raising SMTP_MAX_RCPTS would make much difference.
Messages are sent via SMTP transactions which look like

MAIL FROM ...
  reply
RCPT TO ...
  reply
RCPT TO ...
  reply
... repeated for each recipient
DATA
  reply
<message text>
  reply

If SMTP_MAX_RCPTS = 500, and there are a total of 500 recipients, the
above is done once with 500 recipients. If SMTP_MAX_RCPTS = 50, the
above would be done 10 times with 50 recipients per transaction which
would result in 9 additional MAIL FROM, DATA and <message text>
interactions, but I don't think this would add significantly to the
processing time.

There are some MTA tuning tips in the FAQ
<http://wiki.list.org/x/AgA3>, but some are only applicable to Mailman
2.0 so be careful.

The main outgoing MTA performance killer is doing DNS verification on
recipient domains during SMTP from Mailman. This should be avoided.

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan