[Mailman-Developers] duplicate messages

[Ken Manheimer]

  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.
  Send mail to mime@docserver.cac.washington.edu for more info.

---559023410-1712289374-903035598=:1139
Content-Type: TEXT/PLAIN; charset=US-ASCII

Scott and developers,

I've got a couple of things to try re the duplicate delivery problem.  I
suspect it's related to the new queuing mechanism, and also to the linux
permissions problem that i unravelled for corbett last week.

I suspect it's related because of the sensitivity to forking - and the
fact that you're on a linux system.  When forking on recent rh linux,
you lose the effective UID needed to access the files already written in
the queue directory, so the queued files remain around despite being
successfully sent.  The residual queue files then get processed next
time around, yielding the duplicate messages.  I don't have the time to
verify this, but it sounds like a likely prospect to me.  If it is in
fact the cause, you might profit from the same workaround as for
corbett's posting failure problems - set the ownership of the
~mailman/data directory to 'mail', or whatever ID your MTA runs with,
and see if that helps.

Alternately, before i went away last week i refined the queuing
mechanism a bit to take care of some particular exceptions that can foul
the delivery.  You might try substituting the following replacements for
TrySMTPDelivery() and DeliverToUser() in ~mailman/Mailman/Utils.py, and
using the attached version of ~mailman/scripts/contact_transport instead
of the installed one, to get some error detection for uncooperative
queue files.

NOTE that i don't have a vanilla 1.0b5 installation to test these
against, so try them a bit at a time - i really can't guarantee that
they're compatable, though the interface changes ought to be small
enough to be ok...

If any of you try these things, please let me know what you find.

Ken


def DeliverToUser(msg, recipient, add_headers=[]):
    """Use smtplib to deliver message.

    Optional argument add_headers should be a list of headers to be added
    to the message, e.g. for Errors-To and X-No-Archive."""
    # We fork to ensure no deadlock.  Otherwise, even if sendmail is
    # invoked in forking mode, if it eg detects a bad address before
    # forking, then it will try deliver to the errorsto addr *in the
    # foreground*.  If the errorsto happens to be the list owner for a list
    # that is doing the send - and holding a lock - then the delivery will
    # hang pending release of the lock - deadlock.

    if os.fork():
        return
    sender = msg.GetSender()

    try:
        try:
            msg.headers.remove('\n')
        except ValueError:
            pass
        if not msg.getheader('to'):
            msg.headers.append('To: %s\n' % recipient)
        for i in add_headers:
            if i and i[-1] != '\n':
              i = i + '\n'
            msg.headers.append(i)

        text = string.join(msg.headers, '')+ '\n'+ QuotePeriods(msg.body)
        import OutgoingQueue
        queue_id = OutgoingQueue.enqueueMessage(sender, recipient, text)
        TrySMTPDelivery(recipient,sender,text,queue_id)
        # Just in case there's still something waiting to be sent...
        OutgoingQueue.processQueue()
    finally:
        os._exit(0)

def TrySMTPDelivery(recipient, sender, text, queue_entry):
    import sys, socket
    import smtplib
    import OutgoingQueue

    try:
        con = smtplib.SmtpConnection(mm_cfg.SMTPHOST)
        con.helo(mm_cfg.DEFAULT_HOST_NAME)
        con.send(to=recipient,frm=sender,text=text)
        con.quit()
        dequeue = 1
        failure = None

    # Any exceptions that warrant leaving the message on the queue should
    # be identified by their exception, below, with setting 'dequeue' to 1
    # and 'failure' to something suitable.  Without a particular exception
    # we fall through to the blanket 'except:', which dequeues the message.

    except socket.error:
        # MTA not responding, or other socket prob - leave on queue.
        dequeue = 0
        failure = sys.exc_info()

    except:
        # Unanticipated cause of delivery failure - *don't* leave message 
        # queued, or it may stay, with reattempted delivery, forever...
        dequeue = 1
        failure = sys.exc_info()

    if dequeue:
        OutgoingQueue.dequeueMessage(queue_entry)
    if failure:
        # XXX Here may be the place to get the failure info back to the
        #     list object, so it can disable the recipient, etc.  But how?
        from Logging.StampedLogger import StampedLogger
        l = StampedLogger("smtp-failures", "TrySMTPDelivery", immediate=1)
        l.write("To %s:\n" % recipient)
        l.write("\t %s / %s\n" % (failure[0], failure[1]))
        l.flush()







---559023410-1712289374-903035598=:1139
Content-Type: TEXT/PLAIN; charset=US-ASCII; name=contact_transport
Content-Transfer-Encoding: BASE64
Content-ID: <Pine.GSO.3.96.980813151318.1139C@glyph.cnri.reston.va.us>
Content-Description: ~mailman/scripts/contact_transport

IyEgL3Vzci9iaW4vZW52IHB5dGhvbg0KIw0KIyBDb3B5cmlnaHQgKEMpIDE5
OTggYnkgdGhlIEZyZWUgU29mdHdhcmUgRm91bmRhdGlvbiwgSW5jLg0KIw0K
IyBUaGlzIHByb2dyYW0gaXMgZnJlZSBzb2Z0d2FyZTsgeW91IGNhbiByZWRp
c3RyaWJ1dGUgaXQgYW5kL29yDQojIG1vZGlmeSBpdCB1bmRlciB0aGUgdGVy
bXMgb2YgdGhlIEdOVSBHZW5lcmFsIFB1YmxpYyBMaWNlbnNlDQojIGFzIHB1
Ymxpc2hlZCBieSB0aGUgRnJlZSBTb2Z0d2FyZSBGb3VuZGF0aW9uOyBlaXRo
ZXIgdmVyc2lvbiAyDQojIG9mIHRoZSBMaWNlbnNlLCBvciAoYXQgeW91ciBv
cHRpb24pIGFueSBsYXRlciB2ZXJzaW9uLg0KIyANCiMgVGhpcyBwcm9ncmFt
IGlzIGRpc3RyaWJ1dGVkIGluIHRoZSBob3BlIHRoYXQgaXQgd2lsbCBiZSB1
c2VmdWwsDQojIGJ1dCBXSVRIT1VUIEFOWSBXQVJSQU5UWTsgd2l0aG91dCBl
dmVuIHRoZSBpbXBsaWVkIHdhcnJhbnR5IG9mDQojIE1FUkNIQU5UQUJJTElU
WSBvciBGSVRORVNTIEZPUiBBIFBBUlRJQ1VMQVIgUFVSUE9TRS4gIFNlZSB0
aGUNCiMgR05VIEdlbmVyYWwgUHVibGljIExpY2Vuc2UgZm9yIG1vcmUgZGV0
YWlscy4NCiMgDQojIFlvdSBzaG91bGQgaGF2ZSByZWNlaXZlZCBhIGNvcHkg
b2YgdGhlIEdOVSBHZW5lcmFsIFB1YmxpYyBMaWNlbnNlDQojIGFsb25nIHdp
dGggdGhpcyBwcm9ncmFtOyBpZiBub3QsIHdyaXRlIHRvIHRoZSBGcmVlIFNv
ZnR3YXJlIA0KIyBGb3VuZGF0aW9uLCBJbmMuLCA1OSBUZW1wbGUgUGxhY2Ug
LSBTdWl0ZSAzMzAsIEJvc3RvbiwgTUEgMDIxMTEtMTMwNywgVVNBLg0KDQoi
IiJTZW5kIGEgbWVzc2FnZSB2aWEgbG9jYWwgU01UUCwgb3IgcXVldWUgaXQg
aWYgU01UUCBwb3J0IGlzIG5vdCByZXNwb25kaW5nLg0KDQpUaGUgc2NyaXB0
IHRha2VzIHRoZSBmb2xsb3dpbmcgcHJvdG9jb2wgb24gc3RkaW46DQoNCiBs
aW5lIFsxXTogc2VuZGVyDQogbGluZSBbMjpuKzFdOiBuIHJlY2lwaWVudHMN
CiBsaW5lIFtuKzJdOiA8ZW1wdHk+IC0gZGVsaW1pdGluZyBlbmQgb2YgcmVj
aXBpZW50cw0KIGxpbmUgW24rMzpdOiBtZXNzYWdlIGNvbnRlbnQNCiIiIg0K
DQppbXBvcnQgc3lzLCBvcw0KaW1wb3J0IHBhdGhzDQoNCiMgWFhYOiB0aGlz
IHJlYWxseSBzaG91bGQgYmUgbWVyZ2VkIHdpdGggUHl0aG9uJ3Mgc3RhbmRh
cmQgc210cGxpYiBsaWJyYXJ5DQpmcm9tIE1haWxtYW4gaW1wb3J0IG1tX2Nm
Zw0KZnJvbSBNYWlsbWFuIGltcG9ydCBzbXRwbGliDQpmcm9tIE1haWxtYW4g
aW1wb3J0IFV0aWxzDQpmcm9tIE1haWxtYW4gaW1wb3J0IE91dGdvaW5nUXVl
dWUNCg0KZnJvbSBNYWlsbWFuLkxvZ2dpbmcuVXRpbHMgaW1wb3J0IExvZ1N0
ZEVycg0KTG9nU3RkRXJyKCJlcnJvciIsICJjb250YWN0X3RyYW5zcG9ydCIp
DQoNCmZyb21fYWRkciA9IHN5cy5zdGRpbi5yZWFkbGluZSgpWzotMV0NCnRv
X2FkZHJzICA9IFtdDQp3aGlsZSAxOg0KICAgIGwgPSBzeXMuc3RkaW4ucmVh
ZGxpbmUoKVs6LTFdDQogICAgaWYgbm90IGw6DQogICAgICAgIGJyZWFrDQog
ICAgdG9fYWRkcnMuYXBwZW5kKGwpDQp0ZXh0ID0gc3lzLnN0ZGluLnJlYWQo
KQ0KDQpxdWV1ZV9pZCA9IE91dGdvaW5nUXVldWUuZW5xdWV1ZU1lc3NhZ2Uo
ZnJvbV9hZGRyLCB0b19hZGRycywgdGV4dCkNClV0aWxzLlRyeVNNVFBEZWxp
dmVyeSh0b19hZGRycywgZnJvbV9hZGRyLCB0ZXh0LCBxdWV1ZV9pZCkNCk91
dGdvaW5nUXVldWUucHJvY2Vzc1F1ZXVlKCkNCg0K
---559023410-1712289374-903035598=:1139--