[Spambayes] Proposing to make chi-combining the default

[Jim Bublitz]

On 26-Oct-02 Tim Peters wrote:
> This reminds me that Jim Bublitz reported that using a system
> "for real" day to day gave even better results than contrived
> tests, and while I'm not running controlled experiments on my
> own email, that's my (subjective) impression too.  For my
> personal use in real life, it's been pure delight.

Just like Beetlejuice, if you say my name, I appear :)

Previously I just did testing in chronological order and got much
better results than random testing. As of Sunday I turned on my new
mail system which includes the spam filter, but also replaces
fetchmail, procmail, cron (for mail anyway), and some of qmail
(qmail still acts as my local smtp/pop3 server). I also have a
whitelist in front of the spam filter (my fps can be expensive).
It's completely in Python, of course.

Over 6 full days of use, 1 or 2 spams per day get through and I've
had a couple of fps total. The fns/fps are less than 1%.  I haven't
written anything to parse the logs yet so I don't have actual
stats, and for the first few days I had to restart the mail system
a number of times (bugs), so there hasn't been any way to accumulate
actual results except the logs.

As several other people have mentioned, I'm also juggling msgs
between ham and spam folders based on the results of "manual"
review. The only thing I think deserves mention is the review
process. Every 20 spams received, the mail system puts together a
msg with a list of Subj and From lines from 20 spams in score order.
Next to each msg is a checkbox []. This email msg gets sent to a
user (all 2 of us) on an alternating basis. The user replies to the
msg to confirm the scoring was correct (leaves the checkboxes
empty), or if a score looks wrong, puts an [x] in the box. When the
mail system receives the reply, it forwards any checked msgs back
to the user. If the checked msg was really spam, the user places it
in a local spam folder, along with any fns; if it *was* ham they do
nothing (the mail system has already moved the msg to the ham folder
temporarily). At the end of the day the mail system empties all of
the local spam folders and shifts msgs around again if req'd, and
then retrains on the new mail.

Reviewing 20 msgs at a time takes less than a minute. Doing it via
email makes it more likely the msgs will actually get reviewed. The
review email is much easier than having to scan a folder and delete
each unwanted msg, but it still gives users a sense of control over
the process, demonstrates how much spam is actually being blocked,
and offers a sense of "victory over spammers". My wife likes it
anyway, and she usually hates my UIs.

We still end up looking at spam subject lines because we can't
afford any fps, but real mail gets through more quickly and sorting
is much more accurate and done more quickly with about the absolute
minimum of user activity. In a few months I might have enough
confidence to trust the .99 (spam) scores without review, or else
have constructed a sizable blacklist that doesn't require scoring
or review. The couple of fps so far have scored around .502 (0.5
cutoff) - one was a legitimate mail from a guy who works for a
company that's well represented in my spam corpus.

Jim