[Mailman-Users] Performance issues with 12K subscribers and 14K subscribers and personalized footers.

[Bryan Wright]

Mark,

Thanks for the help.  Followed the instructions in the FAQ  to get sendmail listening on an alternate port in deferred mode.  That made a huge difference, although not as speedy as your environment, 8122 messages made it from mailman to sendmail in about 30 minutes.

Sep 30 10:07:18 2013 (22267) <AA2751A60453C346982E6748CC55DFB54C7E93D2 at RAC-BCS-EXCHG1.rac.local> smtp to catherine.klein for 8122 recips, completed in 1782.344 seconds

Now I just need to fine-tune sendmail so it speeds up as well, but that's a question for elsewhere.

Thanks again for the help.

Bryan Wright


-----Original Message-----
From: Mark Sapiro [mailto:mark at msapiro.net] 
Sent: Saturday, September 28, 2013 3:33 PM
To: Bryan Wright; mailman-users at python.org
Subject: Re: [Mailman-Users] Performance issues with 12K subscribers and 14K subscribers and personalized footers.

On 09/27/2013 10:40 PM, Bryan Wright wrote:
> 
> So back to the speed.   If multiple qrunners won't fix the issue, is
> there any way to speed this up? To get it to process multiple outgoing 
> messages at once?


Before trying to send from Mailman in parallel, you need to look into speeding up the SMTP interaction between Mailman and sendmail, bevcause that's where the real delay exists.


> Could I create multiple lists and then just have the lists subscribed 
> to the one list?  Then each of these lists could use a qrunner?


Yes and no. You could create an umbrella list or use regular_include_lists (see the FAQ at <http://wiki.list.org/x/TIA9>) to create multiple outgoing messages with fewer recipients, but there is no guarantee that these message's out/ queue entries would all wind up in different OutgoingRunner slices. Actually ensuring that each message is processed by a different OutgoingRunner would, I think, require some kludgey code mods. It would be easier and more productive to modify SMTPDirect to use multiple threads. (There is an 'experimental'
MAX_DELIVERY_THREADS setting documented in Defaults.py, but it is not currently implemented.)


> Could disk access time be an issue?  Could I place the mailman queue 
> directory in a ram disk?


Putting the queue on a ram disk won't help. The queue entry is only picked up at the start of processing and whether it takes microseconds or milliseconds, it is not the source of any significant delay.

Putting the server's swap space on a ram disk could help, but there are other things to try first.

As I said in my original reply, see the FAQs at <http://wiki.list.org/x/doA9> and <http://wiki.list.org/x/q4A9>. A local caching DNS may help and if sendmail is doing DNS verification at incoming SMTP time, stopping that will really help. Also ensure you haven't changed the default

SMTP_MAX_SESSIONS_PER_CONNECTION = 0

setting.

Note: as a point of reference, my production server is not very high powered, runs CentOS 5, and has no specific performance tuning. Messages from Mailman use Mailman's VERP feature so each recipient gets an individual message whether or not personalization is on. Mailman delivers over 50 messages per second to Postfix or 12,000 messages in 4 minutes. This is based on up to 600+ messages (my maximum list size is currently 629), and may slow down some with larger lists as Postfix gets busier sending, but I'm sure delivery to Postfix of 12,000 messages would take minutes as opposed to hours.

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan