[Mailman-Users] Re: [Mailman-Developers] SMTPDirect.py using sleep command

[Richard Barrett]

Without comment on whether this whole approach is correct, I would suggest 
that you do not want to modify the deliver function.

I'm assuming you have not enabled threaded delivery, i.e. set the 
mm_cfg.MAX_DELIVERY_THREADS to a non-zero value. If you have, you might 
like to reconsider that decision as it will mean the threaded_deliver 
function may be being used, assuming your Python is installed to support 
threads. If it is, that may be part of the problem. I'll also assume you 
haven't changed the SMTP_MAX_RCPTS to an unreasonable value in mm_cfg.py

Looking at the problem you pose; Mailman divides the total recipient list 
for a mail in to a sequence of chunks in the process function of 
SMTPDirect.py. By default I think SMTP_MAX_RCPTS=500. It then iterates over 
that chunk sequence in the process function of SMTPDirect.py, calling the 
deliver function, which opens a new SMTP connection to pass the message to 
the SMTP server for each chunk's-worth of recipients.

You would be better off putting your delay into the loop over the chunks in 
the process function, to give some breathing space between connections 
being opened, rather than delaying the send to the SMTP server once the 
connection has been opened, which is what your proposal does.

Doing the delay in the process function should also have the advantage of 
less impact on operation when the number of recipients for a message is 
more modest, if you make the delay conditional on there being a large 
number of recipients. You could also play some games so that the amount of 
throttling by Mailman is conditional on the total number of recipients 
using some suitable algorithm to calculate the delay.

Just for fun I've included the content of a samll patch file for 
SMTPDirect.py below which:

1. adds a function for computing the inter-chunk deliver delay. Change the 
algorithm to suit your prejudices.

2. adds a conditional sleep between calls to deliver for each chunk of 
recipients.

Somebody will probably say this is all of load of codswallop and they may 
well be right. But I hope you solve your problem and would be interested to 
hear how you do so.

cut here------------------------------------------------------
--- SMTPDirect.py.orig  Fri Sep  6 10:54:06 2002
+++ SMTPDirect.py       Fri Sep  6 11:17:53 2002
@@ -44,8 +44,15 @@



+def throttle(number_of_recipients):
+    # or some algorithm to compute inter-connection delay
+    delay = int(number_of_recipients / 2000)
+    return delay
+
+
  def process(mlist, msg, msgdata):
      recips = msgdata.get('recips')
+    throttle_delay = throttle(len(recips))
      if not recips:
          # Nobody to deliver to!
          return
@@ -72,6 +79,8 @@
          else:
              for chunk in chunks:
                  deliver(admin, msgtext, chunk, refused)
+                if throttle_delay:
+                    time.sleep(throttle_delay)
      finally:
          t1 = time.time()
          mlist.Lock()
cut here------------------------------------------------------


At 16:12 05/09/2002 -0600, Kory Wheatley wrote:
>Using Mailman 2.0.13 on Red Hat 7.2
>
>I've got a Mailman mailing list that contains 14,000 subscribers, on a
>Linux box that's
>using sendmail has the delivery agent that sends all of the messages to
>one mail host, because all 14,000 subscribers have an email
>account on our mail server. This Linux box is only used to deliver our
>Mailman mailing to our Mail server which is a HPUX 11 server, using
>sendmail as the routing agent.
>
>We sent a message to the 14,000 recipient  Mailman mailing list the
>other day and
>it put the load on our Unix Box that contains our mail server to the
>roof
>
>What I have done to maybe slow down the delivery  to sendmail  that
>creates the queue files then delivers them to our mail server is  in the
>Mailman SMTPDirect.py I have used the sleep command to sleep after
>delivering  so that our mail server can catch up before the next batch
>is delivered, this way its not being bombarded.
>
>Now here is where I would appreciate help, I'm not a experienced Python
>programmer and maybe what I did below is  stupid and will not work
>(using the time.sleep(25) command), but please look at the code and if
>it will not work  please let me know what I can code that will work that
>will slow down the delivery to our mail server.
>
>------------CODE--------------------
>
>def deliver(envsender, msgtext, recips, failures):
>     refused = {}
>     # Gather statistics on how long each SMTP dialog takes.
>##    t0 = time.time()
>     try:
>         conn = smtplib.SMTP(mm_cfg.SMTPHOST, mm_cfg.SMTPPORT)
>         try:
>             # make sure the connect happens, which won't be done by the
>             # constructor if SMTPHOST is false
>             time.sleep(25)
>             refused = conn.sendmail(envsender, recips, msgtext)
>         finally:
>##            t1 = time.time()
>##            syslog('smtp', 'smtp for %d recips, completed in %.3f
>seconds' %
>##                   (len(recips), (t1-t0)))
>             conn.quit()
>     except smtplib.SMTPRecipientsRefused, e:
>         refused = e.recipients
>     # MTA not responding, or other socket problems, or any other kind of
>
>     # SMTPException.  In that case, nothing got delivered
>     except (socket.error, smtplib.SMTPException), e:
>
>------------END-----------------
>
>
>_______________________________________________
>Mailman-Developers mailing list
>Mailman-Developers at python.org
>http://mail.python.org/mailman-21/listinfo/mailman-developers