[Mailman-Developers] First big Mailing

[J C Lawrence]

On Thu, 10 Jan 2002 22:08:19 -0800 
Marc Perkel <marc@perkel.com> wrote:

> I'm doing my first big mailing with Mailman/Exim to deliver
> effector for the Electronic Frontier Foundation. The list is about
> 20,000 names.

I'm going to do some guessing here, so make allowances.

> Anyhow - started out moving right along, maybe too well -
> saturated the T1 pretty quick and the system slowed down. But the
> email logs were really going fast. This went on for about 45
> minutes.

Translation: A message came in, was approved (automatically or by
moderation) and moved to ~mailman/qfiles along with a copy of the
the distribution list.  qunner ran a little while later (cron job),
and started delivering the broadcast to the MTA (thus all the
activity).

> During this time I saw a number of errors. Messages indicating the
> I had too many files open (running tail on the exim logs). A few
> messages that looked like something couldn't open something.db or
> something like that.

Not good.  Very not good.  Loose guess:  You have configured Exim
for more queue runners, parallel deliveries, and/or simultaneous
incoming deliveries that Exim exceeded either your kernel's maximum
number of file handles per process, or the total number of file
handles in the system.

You need to track this down and fix it.  Now.  Before you do
anything else.

It may be enough (for the short term) to just reconfigure Exim to
use a smaller number of processes etc, but that's a stopgap, not a
fix.

> The delivery slowed down as it it were done. System load dropped
> back to low normal levels. 

This sounds like qrunner processes being forked by cron, each trying
to deliver more of your 20K messages to Exim.  Qrunner has an
internal timeout (15mins IIRC) after which it will be reaped and a
new process forked by the next cron pass.

> This lasted a while - then things started back up again really
> delivering messages. These deliveries come in spurts.

Which would explain the above bit.

> Anyhow - even though Exim is delivering other email. Messages sent
> to mailman are getting "stuck". 

Odds are good that a qrunner process was ungracefully reaped
resulting in a stale lock file in ~mailman/locks.  As a result
subsequent qrunner processes are doing nothing, waiting for the lock
to timeout.  

Fix:

  Check that there are no qrunner processes running.  If so, delete
  ~mailman/locks/*

Notes:

  You *REALLY* need to fix your file handle problem.  Its not
  unlikely that that's is a fundamental cause of your problems.

  qrunner is responsible for all motion of mail thru the Mailman
  system, in receipt, moderation, and broadcasting.  If qrunner is
  locked, nothing will happen until it is unlocked.

> It's as if nothing in mailman is working. I see messages being
> sent to mailman. But mailman isn't responding. 

Nope.  They're being stashed in ~mailman/data, awaiting qrunner to
Do The Right Thing (deliver to list, process as bounce, broadcast,
etc.

> I don't know if something is holding these messages and this is
> waiting in some queue - or if Mailman has crashed and is eating
> messages - or someing is corrupt or locked or overloaded or what.

See above.

Again:  FIX YOUR FILE HANDLE PROBLEM FIRST!  

Until you do you can lose mail and have inexplicable impossible to
debug problems.

Notes:

  If you can spare the systems, the first thing you'll need to do is
  separate the MTA that is handling final delivery from your Mailman
  machine.  Bounce processing can and will *really* screw with your
  efficiencies and load with lists of that size.  
  
  Recommended architecture for scaling Mailman in your sort of
  situation is to have your Mailman system deliver all outbound
  mail to a smarthost (either via a smarthost rule on your MTA, or
  directly via SMTP config in Mailman).  Ensure that the MXing for
  bounces will *NOT* go back to your smarthost, but will go directly
  to your Mailman system,

  This allows your smarthost to be tuned for what you need it to do:
  handle outbound deliveries efficiently, and allows your Mailman
  system to remain responsive (and under less load as there's no
  local overburdened MTA queue) for processing inbound bounces etc.

  Set SMTP_MAX_RCPTS in ~mailman/Mailman/mm_cfg.py to something
  reasonably large.  Suggest something in the 50 - 100 range.  Do
  not go any larger.  This may help with temporarily resolving your
  file handle problem.  It will also decrease system load in general
  and help smooth things along a bit.  Later, when everything is
  known working, you can start tuning for performance and look at
  dropping SMTP_MAX_RCPTS down to around 5 (usually the sweet spot).

> Anyhow - I'd like some general feedback on what might be
> happening. The newsletter contains an important story about Norway
> inditing Jon Johansen criminally. He's the guy who wrote the DVD
> code.

Yeah, I read it.  

ObOffer: 

  If you would like some help offloading your mail traffic, I'm
  willing to smarthost a chunk of it for you (will need to verify
  with my upstreams).  Basic idea would be to smarthost route a
  couple TLDs to me for final delivery (I've got a couple T3's so I
  should be able to take a fair percentage of your load).

-- 
J C Lawrence                
---------(*)                Satan, oscillate my metallic sonatas. 
claw@kanga.nu               He lived as a devil, eh?		  
http://www.kanga.nu/~claw/  Evil is a name of a foeman, as I live.