[Mailman-Developers] Patching qrunner to never send duplicates w/ same Message-ID

[Barry A. Warsaw]

>>>>> "BG" == Ben Gertzfield <che@debian.org> writes:

    BG> I would like to work on a different solution; I think it might
    BG> be possible to modify the Mailman queue runner to never send
    BG> out multiple messages with the same Message-ID to the same
    BG> recipient.  Looking through the source in Mailman/Queue, I see
    BG> that each mail's destination is put into its own marshalled
    BG> .db file in /var/lib/mailman/qfiles, so we will have to keep
    BG> state to avoid sending duplicates.

    BG> The way Gnus (the best mail reader on the planet, natch)
    BG> handles this is by keeping a cache of the past X Message-IDs
    BG> (1000 by default, obviously tuneable); we could pretty easily
    BG> keep a list of Message-IDs and recipient pairs, and do one of
    BG> the following if a message is about to be sent out to a
    BG> recipient that has already received an email with that
    BG> Message-ID:

    | 1) Do nothing. (to keep current behavior)
    | 2) Add a warning header to let the users filter on their own
    | 3) Don't send out the message.

    BG> This could be configurable per-user, and I think it'd be a
    BG> great improvement over the current state of things.

    BG> Would a patch for this be accepted?  At which point of the
    BG> queue runner do we want to deal with this?
    BG> Mailman.Queue.OutgoingRunner might be the right place, but I'm
    BG> not sure.

I would try to implement this is a handler module (see GLOBAL_PIPELINE
in Defaults.py) to be run just after CalcRecips.py.  To avoid locking
issues, keep the table as an in memory dictionary; you'll lose it when
you restart the qrunner, but that's probably better than having to
lock a list-external resource (creating a potential bottleneck).

As a bonus, you might also want to cull recipients if you notice their
address is explicitly listed in the To: or Cc: headers, on the
assumption that they will probably see this message twice.  Again,
that should be user configurable too.

The actual implementation of this feature shouldn't be too hard --
it's the GUI aspects that'll take some time.  You'll have to modify
options.html and options.py to expose the user controls, and you'll
have to add a user option flag in Defaults.py.in.

I might accept such a patch for MM2.1 if it gets submitted fairly soon
and cleanly applies to cvs.  I'm hoping to get alpha3 out today or
tomorrow, and this will likely be the last alpha (meaning feature
freeze with the following beta1 release).

-Barry