Fighting Spam with Python

[David MacQuigg]

On Thu, 25 Aug 2005 10:18:37 -0400, Peter Hansen <peter at engcorp.com>
wrote:

>David MacQuigg wrote:
>> Are you as mad about spam as I am?  Are you frustrated with the
>> pessimism and lack of progress these last two years?  Do you have
>> faith that an open-source project can do better than the big companies
>> competing for a lock-in solution?  If so, you might be interested in
>> the Open-Mail project.
>> 
>> I'm writing some scripts to check incoming mail against a registry of
>> reputable senders, using the new authentication methods.  Python is
>> ideal for this because it will give mail-system admins the ability to
>> experiment with the different methods, and provide some real-world
>> feedback sorely needed by the advocates of each method.  So far, we
>> have SPF and CSV.  See http://purl.net/macquigg/email/python for the
>> latest project status.
>
>You might find www.spambayes.org of interest, in several ways.

Integration of a good spam filter is one of our top priorities.
Spambayes looks like a good candidate.  The key new features needed in
a spam filter are the ability to extract the sender's identity (not
that of the latest forwarder), and to factor into the spam score the
reputation of that identity.  We could use some help on this
integration.

I guess I should have said a little more about the Open-Mail project.
We are not focused on developing new authentication or filtering
methods, but rather, providing a platform that will bring these pieces
together and allow the mail admin to chose which methods are used and
in what order.  Interoperability has been the main barrier to
widescale use of authentication.  Python is superb at gluing these
pieces together.

In the flow we envision, the spam filter is the final process, used
only on the 5% that is hard to classify.  80% will get an immediate
reject.  15% will get an immediate accept without filtering, because
the sender is authenticated and has a good reputation.  Eventually,
all reputable senders will join the 15%, and the 5% will shrink to
where we can ignore it.

--
Dave