[Mailman-Developers] Initial Mailman v2.0 with TMDA and Mime filtering

[J C Lawrence]

On Wed, 31 Jul 2002 10:39:52 -0400 
Barry A Warsaw <barry@python.org> wrote:

>>>>>> "JCL" == J C Lawrence <claw@kanga.nu> writes:

JCL> Its now section 6.7 in the mailman user FAQ:

> Thanks JC.

Welcome.

> Would you mind writing up a couple of paragraphs of motivation for
> inclusion in a README.TMDA file?  E.g. Why did you go through all this
> effort?  What problem were you trying to solve?  Maybe: why did you
> choose this particular toolset?

> It doesn't need to go into any of the detail of your FAQ entry.  We'll
> just include a reference to it and be done.

Sure (I've not added the below to the FAQ -- please feel free to):

Among of the basic problems if operating mailing lists are control and
handling of valid posts from non-subscribers to a list, and control of
SPAM.  The two problems are related but not identical.

Proper handling of posts from non-subscribed addresses can be a
significant problem.  SPAM comes from non-subscribed addresses (and is
easily detected when it doesn't), but there are also more interesting
cases; such as for (technical) support lists where the people reporting
a bug have no real wish or need to subscribe to the list, for university
students whose schools seemingly randomly rewrite their email addresses,
or even just the people who have multiple email addresses and never
quite remember which one they subscribed with.  There are many useful
cases beyond that where desirable list mail is sent from non-subscribed
addresses.

  The standard answer to the fellow with multiple email addresses is
  that he should subscribe them all and set all but one to NoMail --
  neither an attractive or particularly approach, especially if he moves
  jobs or IPSs with frequency, or he does not have direct control over
  what address is pasted onto the email he sends (more common than you
  may think).  It also tends to leave your subscriber roster cluttered
  with increasing numbers of NoMail accounts over time as people leave
  the list and don't bother to unsubscribe addresses which don't get
  list mail.

Properly or effectively handling SPAM is a near universal problem.  The
current cold war between SPAMmers and SPAM detection systems like
SpamAssassin leave far too much SPAM getting through.  While SPAM can be
effectively controlled at a list's moderation interface, its expensive
on the moderator's time and attention.  The -owner and -admin addresses
pose a larger problem: Commonly the -admin and -owner addresses for well
known mailing lists receive hundreds if not thousands of SPAM for every
valid message to them.

TMDA (http://www.tmda.net/) can effectively address and help both
problems.

Loosely, TMDA is a "whitelist system".  A whitelist system is an
automated system of building lists of known-good or known-bad email
addresses, and then filtering mail on the basis of those lists.  TMDA
does quite a bit more than this, but we're primarily interested in its
whitelist management at this time.

  Note: The list of subscribed addresses that Mailman maintains for each
  list is effectively a whitelist.  As each address is confirmed at
  subscription time it is both known to be a valid address, and the
  address of someone who is interested in the list.

Putting TMDA in front of Mailman can gain the following behaviors:

  All mail send to any of the list related addresses (list itself,
  -owner and -admin) passes through the TMDA filter system and:

    a) As TMDA can read and process Mailman config.db files (where the
    subscriber data is) mail from subscribers (who have effectively
    whitelisted themselves by confirming their address at subscription
    time) goes straight through to the list or -owner or -admin
    addresses.

    b) Mail from blacklisted addresses is silently discarded

    c) Mail from whitelisted addresses is passed through to the list or
    -owner or -admin addresses.

    d) All other mail is held by TMDA.

  TMDA then provides a confirmation system for held mail by sending a
  message back to the address that sent the held message, requesting
  confirmation that the poster not only exists, but really meant to send
  that message.  The confirmation message contains both instructions on
  how to confirm, and a copy of the original message.

    If the original sender follows the instructions in the confirmation
    request the held message will be sent straight through to the list
    or or -owner or -admin addresses as if TMDA had never intercepted
    it.  Further, TMDA will add that address to its whitelist so that
    future email from that address will pass straight through TMDA and
    not be held.  In this manner non-subscribers can post to the list
    without having to subscribe a NoMail account or putting more load on
    the list moderator.

    If the original sender doesn't confirm his message TMDA will hold it
    for a while awaiting confirmation before silently deleting it.  As
    SPAMers almost universally don't run proper mail systems they never
    receive the confirmation requests from TMDA and so never confirm and
    thus their messages never get to the list, -owner, or -admin.  While
    it is possible that the very few SPAMmers who do run proper mail
    systems could build systems that can confirm TMDA requests (I
    haven't heard of even one yet that does), their population is so
    low, and they are so easily detected, that the effort of manually
    blacklisting them is not so large.

      Should SPAMmers get the hang of TMDA, TMDA can trivially change
      its confirmation process to be less and less automateble --
      essentially making the confirmation process more and more of a
      Turing test.  Happily, that's not necessary yet.  The current TMDA
      method of, "Just reply to this message to confirm," works well for
      now.

The end result is that subscription and posting rights for TMDA-fronted
lists have effectively been separated.  You now subscribe to a list to
receive mail from the list (and establish an initial address from which
you can post to the list).  You simply send mail to the list (and
confirm each email address) to be able to send messages to a list.

Inserting TMDA in this manner in front of the -owner and -admin
addresses has the following results:

  -- Mail from subscribers and other whitelisted addresses gets pass
  straight through to the list owner.

  -- Mail from other addresses is held for confirmation.  

In this way the primary audience (your list subscribers and whitelist
members) can message the -owner or -admin without any extra effort.
Other people merely need to confirm their address to gain the same
benefit.

Note: 

  TMDA is a very flexible system.  The above details one possible
  application and configuration of TMDA with mailman.  Other simple
  variations could have every inbound message needing to be confirmed,
  every message from a non-subscriber needing confirmation, having TMDA
  manage all posting rights and never checking the Mailman list
  subscriber list, etc.  TMDA also supports a significant feature set
  over and above just whitelist handling.  There are very interesting
  things that can be done with TMDA's custom addresses for instance.
  Read the TMDA documentation and adapt the following HOW-TO to suit
  your needs.

-- 
J C Lawrence
---------(*)                Satan, oscillate my metallic sonatas. 
claw@kanga.nu               He lived as a devil, eh?		  
http://www.kanga.nu/~claw/  Evil is a name of a foeman, as I live.