OT: spam filtering idea

[Tim Peters]

[Paul Wright]
> I thought the point of Bayesian filtering was that it learned about your
> spam and your legitimate email, so that learning what other people
> considered spam wouldn't be as effective. I'm no expert on this, though.
> I expect Tim Peters will be along in a minute :-)

Cursed to do so, yes <wink>.  Trying to train one of these classifiers to
serve a diverse group of users at once is demonstrably and quantifiably much
less effective.  Training one for a focused mailing list (like
comp.lang.python) appears extremely effective, though (the spambayes
classifier had error rates too low to measure reliably across a 20K c.l.py
ham + 14K spam test).

> <http://www.jerf.org/irights/2002/11/18.html> argues that human malice
> can and will defeat Bayesian filters, and that widespread adoption of
> them will end up making spam harder to recognize by hand. I'm a little
> concerned that the author of the article overestimates the intelligence
> of spammers, but I suppose there's a selection pressure on them to get
> more cunning as time goes on.

He's probably right that the way to beat this generation of filters is to
create spam statistically indistinguishable from ham.  The unknown not
addressed there is that all forms of advertising are a percentage game, and
current spam uses (e.g.) ALL CAPS and huge fonts and bright colors because
those tricks increase response rate.  Spam so bland that it looks like it
came from your grandmother may not draw a response rate large enough to
repay the costs of spamming (which, while tiny on a per-msg basis, aren't
zero).

> ...
> [1] How many boneheaded keyword filters will now bounce this post when
> it goes out as mail on the python list, I wonder? There's an awful lot
> of snake oil out there being sold as spam filters.

I received it via the c.l.py mailing list gateway.  My personal spambayes
filter gave your msg a score of 0.9998 for haminess (0.0 = spam, 1.0 = ham),
and 1.422e-07 for spaminess (0.0 = ham, 1.0 = spam).  So, overall, it was
certain the msg was ham, and despite that it contained some mildly strong
spam words:

'intelligence'  0.958257
'legitimate'    0.961219
'businesses'    0.963789

An individual word can neither damn nor redeem a msg in this kind of
approach; it's more a "preponderance of evidence" approach.  Effective spam
has to get around to selling you something, and the language of advertising
as a whole is very damning (for example, in c.l.py tests, the putative ham
we had the most trouble with was conference announcements:  they're trying
to sell you on attending a conference, and share much language with hardcore
spam because of it).