From tim.one@comcast.net  Thu Sep  5 18:57:17 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 05 Sep 2002 13:57:17 -0400
Subject: [Spambayes] RE: [Python-Dev] Getting started with GBayes testing
In-Reply-To: <3D772EC2.30217.184B6C78@localhost>
Message-ID: <BIEJKCLHCIOIHAGOKOLHAEENDKAA.tim.one@comcast.net>

[Followups directed to spambayes@python.org
 http://mail.python.org/mailman-21/listinfo/spambayes
]

[Brad Clements]
> ...
> My feeling is that the presentation of "the message" is independent of the
> message itself, so if I get a message in Text, HTML, RTF only the actual
> content is important, not the markup method.

Everything's A Clue.  Everything that gets ignored partly blinds the
classifier, so the question isn't whether there's a difference, it's how
much of a difference it makes.

> Though I suppose using lots of red and large fonts might be an
> indicator of spam, the text of the message should still suffice.

Indeed, Graham reported that the hex color code for bright red was one of
the strongest spam indicators in his database.

> Tim's comments in timtest.py hint that stripping tags isn't a
> catastrophe for f-n's, but he's not planning on doing that for use on
> technical lists.

When HTML-only email is a 99.99% spam indicator on a tech list, it would be
crazy to ignore that clue.  But note that the comments *also* say I'd be
delighted to remove HTML tags even there if some other way of slashing the
f-n rate is proven to work (and most people who have tried it say that
mining more header lines does do it -- but then I haven't seen anything from
them about how they do when they ignore the header lines.  I was happy to
ignore header lines in order to get *some* kind of handle on how well could
be done on "pure content", and turned out that works remarkably well).

>> # So if a message is multipart/alternative with both text/plain
>> # and text/html branches, we ignore the latter, else newbies would never
>> # get a message through.  If a message is just HTML, it has virtually no
>> # chance of getting through

> Tells me (spammer hat on) that I can send message with a
> non-spammish text only part, and a spam html part since most
> "non-techie" email client users automatically display the html
> version when available, however Tim's implementation will ignore it.

Sure.  It *certainly* isn't a problem on my test data (as witnessed by the
measured error rates).  If the nature of the world changes, the code has to
adapt along with it.  But 90% of the spam I receive (and I get a lot) is
still trivial to recognize from a mere glance at the subject line, and I
don't buy that spammers are a class of ubergeek with formidable skill.
Response rates are a percentage game, and more so than anti-spammers I
expect spammers are keen to go for high-percentage wins at the expense of
esoterica.

> Most "average users" never even see the text-only part of
> multipart messages. In Tim's application, that's okay since he's going
> to use the text-only part anyway. But for my  purposes, I need to consider
> both portions. So it's simpler for me to strip html and combine that text
> with the text-only part and then "test" the combined parts.

Not unreasonable <wink>, but testing remains the only way to decide.  It's
rare you can out-think a fraction of a percent!


From tim.one@comcast.net  Thu Sep  5 19:30:03 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 05 Sep 2002 14:30:03 -0400
Subject: [Spambayes] RE: [Python-Dev] GBayes design
In-Reply-To: <002b01c254f2$f6c7c020$71b53bd0@othello>
Message-ID: <BIEJKCLHCIOIHAGOKOLHGEEPDKAA.tim.one@comcast.net>

[Followups directed to spambayes@python.org
 http://mail.python.org/mailman-21/listinfo/spambayes
]

[Raymond Hettinger]
> Is it too late to challenge a core design decision?

Never too late, but somebody has to do real work to prove that a change is
justified.  Plausible ideas are cheaper than dirt, alas.

> Instead of multiplying probablities, use fuzzy logic methods.
> Classify the indicators into damning, strong, weak, neautral, ...

Think about how that differs from 0.99, 0.80, 0.20 and 0.50.  Does it?

> After counting the number of indicators in each class, make
> a spam/ham decision that can be easily tweaked.  This would
> make it easy to implement variations of Tim's recent clear
> win, where additional indicators are gathered until the
> balance shifts sharply to one side.
>
> Some other advantages are:
> -- easily interpreted score vectors (6 damning, 7 strong, 4 weak, ... )

I've seen people see the current prob("TV") = 0.99 style cold and pick it up
at once.  With character n-grams I think it's frustrating, but word-like
tokenization gives easily recognized clues.

> -- avoids mathematical issues with indicators not being independent

How do you know this?

> -- allows the addition of non-token based indicators.  for instance,
>     a preponderance of caps would be a weak indicator.  the presence
>     of caps separated by spaces would be a strong indicator.

As far as the current classifier is concerned, "a token" is any Python
object usable as a dict key.  There are already several ways in which the
current tokenization scheme in timtest.py uses strings to *represent*
non-textual indicators.  For example, if the headers lack an Organization
line, a 'bool:noorg' "token" is generated.  For large blobs of text that get
skipped, a token is generated that records both the first character in that
blob and the number of bytes skipped (chopped to the nearest multiple of
10).  And so on -- you can inject anything you like into the scheme,
including stuff like

    "number of caps separated by spaces: more than 10"

(BTW, I happen to know that this particular "clue" acts to block relevant
conference announcements, not just spam)

I got some interesting results by injecting a crude characters/word
statistic:

    yield "cpw:%.1g" % (float(len(text)) / len(text.split()))

There are certain values of that statistic that turned out to be
killer-strong spam indicators, but there's a potential problem I've
mentioned before:  if you have an unbounded number of free parameters you
can fiddle, you can train a system to fit any given dataset exactly.  That's
in part why replication of results by others is necessary to make schemes
like this superb (I can only make one merely excellent on my own <wink>).

> -- the decision logic would be more intuitive
> -- avoids the issue of having equal amounts of spam and ham in
>     the sample

It's not clear that this matters; some results of preliminary experiments
are written up in the code comments.  The way Graham computes P(Spam | Word)
is via ratios, *as if* there were an equal number of each; and that's
consistent with the other bogus <wink> equality assumption in the scorer.  I
haven't yet changed all these guys at the same time to take P(Spam) and
P(Ham) into account.

BTW, note that all the results I've reported had a ham/spam training ratio
of 4000/2750.  I left that non-unity on purpose.

> The core concept would stay the same -- it's really just a shift from
> continuous to discrete.

Let us know how it turns out <wink>.


From nas@python.ca  Thu Sep  5 20:04:21 2002
From: nas@python.ca (Neil Schemenauer)
Date: Thu, 5 Sep 2002 12:04:21 -0700
Subject: [Spambayes] all but one testing
Message-ID: <20020905190420.GB19726@glacier.arctrix.com>

I've written a driver script the does "all but one testing".  The basic
algorithm is:

    gb = GrahamBayes()
    for msg in spam:
        gb.learn(msg, is_spam=True)
    for msg in ham:
        gb.learn(msg, is_spam=False)
    for msg in spam:
        gb.unlearn(msg, is_spam=True)
        gb.spamprob(msg)
        gb.lear(msg, is_spam=True)
    for msg in ham:
        gb.unlearn(msg, is_spam=False)
        gb.spamprob(msg)
        gb.lear(msg, is_spam=False)
    print summary

Is this type of testing useful?  As understand it, it's most useful when
you have a small amount of testing and training data.  That doesn't seem
to be a problem for us.  Also, it's really slow.

  Neil

From neale@woozle.org  Thu Sep  5 20:47:26 2002
From: neale@woozle.org (Neale Pickett)
Date: 05 Sep 2002 12:47:26 -0700
Subject: [Spambayes] spamcan release, finally
Message-ID: <w53k7m0fcc1.fsf@woozle.org>

Although at this point its value is limited (if existant at all),
yesterday I finally got the green light to release my spamcan package.

  http://woozle.org/~neale/src/spamcan/spamcan.html

Who knows, maybe there's something useful in there.  For starters, it
uses an anydbm file back-end.  I don't know that a dbm file is the best
solution, but the way the Debian distribution is set up currently, you
can either have ZODB (Python2.1) or you can have generators
(Python2.2).  I suspect that the dbm method will result is smaller
files, but that just a hunch.

The more I look at classifier.py, the more it looks like something
well-suited for a cdb file (heresy, I know).  But I haven't seen any
data showing that cdb is better in any way than Berkeley DB hashes--all
I've managed to find so far is dogma.

Anyway, if it wasn't obvious before, I'm currently trying to make stuff
run quickly, with as little disk usage as possible.

Aloha :)

Neale

From bkc@murkworks.com  Thu Sep  5 20:56:19 2002
From: bkc@murkworks.com (Brad Clements)
Date: Thu, 05 Sep 2002 15:56:19 -0400
Subject: [Spambayes] RE: [Python-Dev] Getting started with GBayes testing
In-Reply-To: <BIEJKCLHCIOIHAGOKOLHAEENDKAA.tim.one@comcast.net>
References: <3D772EC2.30217.184B6C78@localhost>
Message-ID: <3D777F05.5725.1984F990@localhost>

On 5 Sep 2002 at 13:57, Tim Peters wrote:


> Not unreasonable <wink>, but testing remains the only way to decide.  It's
> rare you can out-think a fraction of a percent!

Hence my interest in getting a test rig setup so I can generate my own percentages to 
crow about ;-)

..  or not :-(



Brad Clements,                bkc@murkworks.com   (315)268-1000
http://www.murkworks.com                          (315)268-9812 Fax
AOL-IM: BKClements


From neale@woozle.org  Thu Sep  5 20:49:44 2002
From: neale@woozle.org (Neale Pickett)
Date: 05 Sep 2002 12:49:44 -0700
Subject: [Spambayes] spamcan release, finally
In-Reply-To: <w53k7m0fcc1.fsf@woozle.org>
References: <w53k7m0fcc1.fsf@woozle.org>
Message-ID: <w53fzwofc87.fsf@woozle.org>

So then, Neale Pickett <neale@woozle.org> is all like:

> Although at this point its value is limited (if existant at all),
> yesterday I finally got the green light to release my spamcan package.

Oh, I should have mentioned, I have zero interest in maintaining spamcan
now that spambayes exists.  I present it mainly for code-lifting
purposes :)

Neale

From skip@pobox.com  Thu Sep  5 21:53:23 2002
From: skip@pobox.com (Skip Montanaro)
Date: Thu, 5 Sep 2002 15:53:23 -0500
Subject: [Spambayes] test sets?
Message-ID: <15735.50243.135743.32180@12-248-11-90.client.attbi.com>

Tim,

Any thought to wrapping up your spam and ham test sets for inclusion w/ the
spambayes project?

Skip

From gward@python.net  Thu Sep  5 22:03:12 2002
From: gward@python.net (Greg Ward)
Date: Thu, 5 Sep 2002 17:03:12 -0400
Subject: [Spambayes] test sets?
In-Reply-To: <15735.50243.135743.32180@12-248-11-90.client.attbi.com>
References: <15735.50243.135743.32180@12-248-11-90.client.attbi.com>
Message-ID: <20020905210312.GA16171@cthulhu.gerg.ca>

On 05 September 2002, Skip Montanaro said:
> Any thought to wrapping up your spam and ham test sets for inclusion w/ the
> spambayes project?

Might be more useful to have other people working other test sets.
Variety is the spice of life, and you don't want an algorithm completely
biased towards one particular dataset.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Dyslexics of the world, untie!

From skip@pobox.com  Thu Sep  5 22:06:55 2002
From: skip@pobox.com (Skip Montanaro)
Date: Thu, 5 Sep 2002 16:06:55 -0500
Subject: [Spambayes] test sets?
In-Reply-To: <20020905210312.GA16171@cthulhu.gerg.ca>
References: <15735.50243.135743.32180@12-248-11-90.client.attbi.com>
        <20020905210312.GA16171@cthulhu.gerg.ca>
Message-ID: <15735.51055.618575.247684@12-248-11-90.client.attbi.com>


    >> Any thought to wrapping up your spam and ham test sets for inclusion
    >> w/ the spambayes project?

    Greg> Might be more useful to have other people working other test sets.
    Greg> Variety is the spice of life, and you don't want an algorithm
    Greg> completely biased towards one particular dataset.

Agreed, but for the purposes of comparing new stuff with the baseline it
seems to make sense to have a standard test set.  Sort of a spam-stone. ;-)

Skip


From paul-bayes@svensson.org  Thu Sep  5 22:18:13 2002
From: paul-bayes@svensson.org (Paul Svensson)
Date: Thu, 5 Sep 2002 17:18:13 -0400 (EDT)
Subject: [Spambayes] GBayes spam filtering
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKECKBCAB.tim.one@comcast.net>
Message-ID: <Pine.LNX.4.44.0209051346240.6840-100000@familjen.svensson.org>

I've been following the discussion on spam filtering on the python-dev
ist with great interest.  It looks very promising so far, but there's one
issue I would like to explore further: we don't all have a pre-filtered
corpus and a Tim or Brad to hand it to, to turn into a well tuned filter,
and even if we did, how often would we need to bring them back to re-tune
the filter as the flavor of spam changes over time ?

Thus my interest in the operational side of corpus-collecting.

This is more an issue for person-to-person email than for large mailing
lists, as the later are more likely to actually have a Tom or Brad available.

I don't think it's realistic to expect users to mark everything they
read as ham or spam.  For a single-user setup, I would consider a
mail reader command "delete as spam"; everything that's read and not
thusly marked would go in the ham list.  However, for a multi-user system,
I think something a little more sofisticated would be neccesary.

Here's my idea:

The message corpus database needs to contain, for each message,
	the message-id
	a timestamp (for removal of old stuff)
	the word count histogram
	a spam/ham flag

On SMTP receipt of a message, it's scanned, and if it smells like spam,
it's bounced.  It's NOT automatically added to the corpus.
If the message does not smell like spam, it's delivered, and
added to the corpus as ham.

When a user reads a message and find that it's spam that got thru the filter,
they need a way to send the message-id to the corpus, to flag it as spam.
At this point, it would be a good idea to compare the histogram of the new
spam to each histogram in the ham corpus, and remove any that are similar
(any good ideas how to do the comparison?), or maybe if they are VERY
similar simply flag them as spam.  After recomputing the filter from the
modified corpus, we could also re-filter the ham corpus, and remove more
newfound spam that way.

Characteristically of this system, the spam corpus will be reasonably clean
(assuming the users don't abuse it too much), but the ham corpus will be
quite dirty, containing spam that's not yet read, and spam that the recipient
didn't bother to mark.  I'm curious how GBayes would handle this situation;
I assume the false negative rate would go up, but how much ?

	/Paul


From tim.one@comcast.net  Thu Sep  5 23:20:33 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 05 Sep 2002 18:20:33 -0400
Subject: [Spambayes] all but one testing
In-Reply-To: <20020905190420.GB19726@glacier.arctrix.com>
Message-ID: <BIEJKCLHCIOIHAGOKOLHEEGCDKAA.tim.one@comcast.net>

[Neil Schemenauer]
> I've written a driver script the does "all but one testing".  The basic
> algorithm is:
>
>     gb = GrahamBayes()
>     for msg in spam:
>         gb.learn(msg, is_spam=True)
>     for msg in ham:
>         gb.learn(msg, is_spam=False)
>     for msg in spam:
>         gb.unlearn(msg, is_spam=True)
>         gb.spamprob(msg)
>         gb.lear(msg, is_spam=True)
>     for msg in ham:
>         gb.unlearn(msg, is_spam=False)
>         gb.spamprob(msg)
>         gb.lear(msg, is_spam=False)
>     print summary
>
> Is this type of testing useful?

It's sure better than nothing <wink>.  Also better than nothing, but not as
good, is doing the same thing but skipping the learn/unlearn calls after
initial training.

> As understand it, it's most useful when you have a small amount of testing
> and training data.

I've run no experiments on training set size yet, and won't hazard a guess
as to how much is enough.  I'm nearly certain that the 4000h+2750s I've been
using is way more than enough, though.  It's a question of practical
importance open for fresh triumphs <wink>.

> That doesn't seem> to be a problem for us.  Also, it's really slow.

Each call to learn() and to unlearn() computes a new probability for every
word in the database.  There's an official way to avoid that in the first
two loops, e.g.

    for msg in spam:
        gb.learn(msg, True, False)
    gb.update_probabilities()

In each of the last two loops, the total # of ham and total # of spam in the
"learned" set is invariant across loop trips, and you *could* break into the
abstraction to exploit that:  the only probabilities that actually change
across those loop trips are those associated with the words in msg.  Then
the runtime for each trip would be proportional to the # of words in the msg
rather than the number of words in the database.

Another area for potentially fruitful study:  it's clear that the
highest-value indicators usually appear "early" in msgs, and for spam
there's an actual reason for that:  advertising has to strive to get your
attention early.  So, for example, if we only bothered to tokenize the first
90% of a msg, would results get worse?  I doubt it.  And if not, what about
the first 50%?  The first 10%?  The first 1000 bytes?  max(1000 bytes, first
10%)?  That could also yield a major speed boost, and *may* even improve
results -- e.g., sometimes an on-topic message starts well but then rambles.


From whisper@oz.net  Thu Sep  5 23:42:38 2002
From: whisper@oz.net (David LeBlanc)
Date: Thu, 5 Sep 2002 15:42:38 -0700
Subject: [Spambayes] All but one testing
Message-ID: <GCEDKONBLEFPPADDJCOEMECOENAA.whisper@oz.net>

Errr... not to be pedantic or anything, but this is called "omit one
testing" or OOT in the literature IIRC. Helpful in case you're searching for
additional information, say at http://citeseer.nj.nec.com/ for instance.

David LeBlanc
Seattle, WA USA


From nas@python.ca  Thu Sep  5 23:49:23 2002
From: nas@python.ca (Neil Schemenauer)
Date: Thu, 5 Sep 2002 15:49:23 -0700
Subject: [Spambayes] all but one testing
In-Reply-To: <BIEJKCLHCIOIHAGOKOLHEEGCDKAA.tim.one@comcast.net>
References: <20020905190420.GB19726@glacier.arctrix.com>
	<BIEJKCLHCIOIHAGOKOLHEEGCDKAA.tim.one@comcast.net>
Message-ID: <20020905224923.GA20480@glacier.arctrix.com>

Tim Peters wrote:
> I've run no experiments on training set size yet, and won't hazard a guess
> as to how much is enough.  I'm nearly certain that the 4000h+2750s I've been
> using is way more than enough, though.

Okay, I believe you.

> Each call to learn() and to unlearn() computes a new probability for every
> word in the database.  There's an official way to avoid that in the first
> two loops, e.g.
> 
>     for msg in spam:
>         gb.learn(msg, True, False)
>     gb.update_probabilities()

I did that.  It's still really slow when you have thousands of messages.

> In each of the last two loops, the total # of ham and total # of spam in the
> "learned" set is invariant across loop trips, and you *could* break into the
> abstraction to exploit that:  the only probabilities that actually change
> across those loop trips are those associated with the words in msg.  Then
> the runtime for each trip would be proportional to the # of words in the msg
> rather than the number of words in the database.

I hadn't tried that.  I figured it was better to find out if "all but
one" testing had any appreciable value.  It looks like it doesn't so
I'll forget about it.

> Another area for potentially fruitful study:  it's clear that the
> highest-value indicators usually appear "early" in msgs, and for spam
> there's an actual reason for that:  advertising has to strive to get your
> attention early.  So, for example, if we only bothered to tokenize the first
> 90% of a msg, would results get worse?

Spammers could exploit this including a large MIME part at the beginning
of the message.  In pratice that would probably work fine.  

> sometimes an on-topic message starts well but then rambles.

Never.  I remember the time when I was ten years old and went down to
the fishing hole with my buddies.  This guy named Gordon had a really
huge head.  Wait, maybe that was Joe.  Well, no matter.  As I recall, it
was a hot day and everyone was tired...Human Growth Hormone...girl with
huge breasts...blah blah blah......


From nas@python.ca  Thu Sep  5 23:56:01 2002
From: nas@python.ca (Neil Schemenauer)
Date: Thu, 5 Sep 2002 15:56:01 -0700
Subject: [Spambayes] All but one testing
In-Reply-To: <GCEDKONBLEFPPADDJCOEMECOENAA.whisper@oz.net>
References: <GCEDKONBLEFPPADDJCOEMECOENAA.whisper@oz.net>
Message-ID: <20020905225601.GA20578@glacier.arctrix.com>

David LeBlanc wrote:
> Errr... not to be pedantic or anything, but this is called "omit one
> testing" or OOT in the literature IIRC.

I have no idea.  I made up the name.  Thanks for the correction.

  Neil

From tim.one@comcast.net  Fri Sep  6 03:14:07 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 05 Sep 2002 22:14:07 -0400
Subject: [Spambayes] test sets?
In-Reply-To: <15735.50243.135743.32180@12-248-11-90.client.attbi.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEFEBCAB.tim.one@comcast.net>

[Skip Montanaro]
> Any thought to wrapping up your spam and ham test sets for
> inclusion w/ the spambayes project?

I gave it all the thought it deserved <wink>.  It would be wonderful to get
several people cranking on the same test data, and I'm all in favor of that.
OTOH, my Data/ subtree currently has more than 35,000 files slobbering over
134 million bytes -- even if I had a place to put that much stuff, I'm not
sure my ISP would let me email it in one msg <wink>.

Apart from that, there was a mistake very early on whose outcome was that
this isn't the data I hoped I was using.  I *hoped* I was using a snapshot
of only recent msgs (to match the snapshot this way of only spam from 2002),
but turns out they actually go back to the last millennium.  Greg Ward is
currently capturing a stream coming into python.org, and I hope we can get a
more modern, and cleaner, test set out of that.  But if that stream contains
any private email, it may not be ethically possible to make that available.
Can you think of anyplace to get a large, shareable ham sample apart from a
public mailing list?  Everyone's eager to share their spam, but spam is so
much alike in so many ways that's the easy half of the data collection
problem.


From skip@pobox.com  Fri Sep  6 03:41:13 2002
From: skip@pobox.com (Skip Montanaro)
Date: Thu, 5 Sep 2002 21:41:13 -0500
Subject: [Spambayes] test sets?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCIEFEBCAB.tim.one@comcast.net>
References: <15735.50243.135743.32180@12-248-11-90.client.attbi.com>
        <LNBBLJKPBEHFEDALKOLCIEFEBCAB.tim.one@comcast.net>
Message-ID: <15736.5577.157228.229200@12-248-11-90.client.attbi.com>


    Tim> I gave it all the thought it deserved <wink>.  It would be
    Tim> wonderful to get several people cranking on the same test data, and
    Tim> I'm all in favor of that.  OTOH, my Data/ subtree currently has
    Tim> more than 35,000 files slobbering over 134 million bytes -- even if
    Tim> I had a place to put that much stuff, I'm not sure my ISP would let
    Tim> me email it in one msg <wink>.

Do you have a dialup or something more modern <wink>?  134MB of messages
zipped would probably compress pretty well - under 50MB I'd guess with all
the similarity in the headers and such.  You could zip each of the 10 sets
individually and upload them somewhere.

    Tim> Can you think of anyplace to get a large, shareable ham sample
    Tim> apart from a public mailing list?  Everyone's eager to share their
    Tim> spam, but spam is so much alike in so many ways that's the easy
    Tim> half of the data collection problem.

How about random sampling lots of public mailing lists via gmane or
something similar, manually cleaning it (distributing that load over a
number of people) and then relying on your clever code and your rebalancing
script to help further cleanse it?  The "problem" with the ham is it tends
to be much more tied to one person (not just intimate, but unique) than the
spam.

I save all incoming email for ten days (gzipped mbox format) before it rolls
over and disappears.  At any one time I think I have about 8,000-10,000
messages.  Most of it isn't terribly personal (which I would cull before
passing along anyway) and much of it is machine-generated, so would be of
marginal use.  Finally, it's all ham-n-spam mixed together.  Do we call that
an omelette or a Denny's Grand Slam?

Skip

From tim.one@comcast.net  Fri Sep  6 07:09:11 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 06 Sep 2002 02:09:11 -0400
Subject: [Spambayes] all but one testing
In-Reply-To: <20020905224923.GA20480@glacier.arctrix.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEFOBCAB.tim.one@comcast.net>

[Tim]
> Another area for potentially fruitful study:  it's clear that the
> highest-value indicators usually appear "early" in msgs, and for spam
> there's an actual reason for that:  advertising has to strive
> to get your attention early.  So, for example, if we only bothered to
> tokenize the first 90% of a msg, would results get worse?

[Neil Schemenauer]
> Spammers could exploit this including a large MIME part at the beginning
> of the message.  In pratice that would probably work fine.

Note that timtest.py's current tokenizer only looks at decoded text/* MIME
sections (or raw message text if no MIME exists); spammers could put
megabytes of other crap before that and it wouldn't even be looked at
(except that the email package has to parse non-text/* parts well enough to
skip over them, and tokens for the most interesting parts of Content-{Type,
Disposition, Transfer-Encoding} decorations are generated for all MIME
sections).

Schemes that remain ignorant of MIME are vulnerable to spammers putting
arbitrary amounts of "nice text" in the preamble area (after the headers and
before the first MIME section), which most mail readers don't display, but
which appear first in the file so are latched on to by Graham's scoring
scheme.

But I don't worry about clever spammers -- I've seen no evidence that they
exist <0.5 wink>.  Even if they do, the Open Source zoo is such that no
particular scheme will gain dominance, and there's no percentage for
spammers in trying to fool just one scheme.  Even if they did, for the kind
of scheme we're using here they can't *know* what "nice text" is, not unless
they pay a lot of attention to the spam targets and highly tailor their
messages to each different one.  At that point they'd be doing targeted
marketing, and the cost of the game to them would increase enormously.

if-you're-out-to-make-a-quick-buck-you-don't-waste-a-second-on-hard-
    targets-ly y'rs  - tim


From anthony@interlink.com.au  Fri Sep  6 08:59:38 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Fri, 06 Sep 2002 17:59:38 +1000
Subject: [Spambayes] test sets?
Message-ID: <200209060759.g867xcV03853@localhost.localdomain>


I've got a test set here that's the last 3 and a bit years email to
info@ekit.com and info@ekno.com - it's a really ugly set of 20,000+
messages, currently broken into 7,000 spam, 9,000 ham, 9,000 currently
unclassified. These addresses are all over the 70-some different 
ekit/ekno/ISIConnect websites, so they get a LOT of spam.

As well as the usual spam, it also has customers complaining about 
credit card charges, it has people interested in the service and 
asking questions about long distance rates, &c &c &c. Lots and lots 
of "commercial" speech, in other words. Stuff that SA gets pretty 
badly wrong.

I'm currently mangling it by feeding all parts (text, html, whatever 
else :) into the filters, as well as both a selected number of headers 
(to, from, content-type, x-mailer), and also a list of 
(header,count_of_header). This is showing up some nice stuff - e.g. the 
X-uidl that stoopid spammers blindly copy into their messages.

I did have Received in there, but it's out for the moment, as it causes
rates to drop.

I'm also stripping out HTML tags, except for href="" and src="" - there's
so so much goodness in them (note that I'm only keeping the contents of
the attributes). 


--
Anthony Baxter     <anthony@interlink.com.au>
It's never too late to have a happy childhood.


From anthony@interlink.com.au  Fri Sep  6 09:06:57 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Fri, 06 Sep 2002 18:06:57 +1000
Subject: [Spambayes] Re: [Python-Dev] Getting started with GBayes testing 
In-Reply-To: <BIEJKCLHCIOIHAGOKOLHGEEMDKAA.tim.one@comcast.net> 
Message-ID: <200209060806.g8686ve03964@localhost.localdomain>


>>> Tim Peters wrote
> > I've actually got a bunch of spam like that. The text/plain is something
> > like
> >
> > **This is a HTML message**
> >
> > and nothing else.
> 
> Are you sure that's in a text/plain MIME section?  I've seen that many times
> myself, but it's always been in the prologue (*between* MIME sections -- so
> it's something a non-MIME aware reader will show you).

*nod* I know - on my todo is to feed the prologue into the system as well.

A snippet, hopefully not enough to trigger the spam-filters.


To: into89j@gin.elax.ekorp.com
X-Mailer: Microsoft Outlook Express 4.72.1712.3
X-MimeOLE: Produced By Microsoft MimeOLE V??D.1712.3
Mime-Version: 1.0
Date: Sun, 28 Jan 2001 23:54:39 -0500
Content-Type: multipart/mixed; boundary="----=_NextPart_000_007F_01BDF6C7.FABAC1
B0"
Content-Transfer-Encoding: 7bit

This is a MIME Message

------=_NextPart_000_007F_01BDF6C7.FABAC1B0
Content-Type: multipart/alternative; boundary="----=_NextPart_001_0080_01BDF6C7.
FABAC1B0"

------=_NextPart_001_0080_01BDF6C7.FABAC1B0
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

***** This is an HTML Message ! *****


------=_NextPart_001_0080_01BDF6C7.FABAC1B0
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<!doctype html public "-//w3c//dtd html 4=2E0
 transitional//en">
 <html>
 <head>
    


From anthony@interlink.com.au  Fri Sep  6 09:11:50 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Fri, 06 Sep 2002 18:11:50 +1000
Subject: [Spambayes] test sets? 
In-Reply-To: <200209060759.g867xcV03853@localhost.localdomain> 
Message-ID: <200209060811.g868Bo904031@localhost.localdomain>


>>> Anthony Baxter wrote
> I'm currently mangling it by feeding all parts (text, html, whatever 
> else :) into the filters, as well as both a selected number of headers 
> (to, from, content-type, x-mailer), and also a list of 
> (header,count_of_header). This is showing up some nice stuff - e.g. the 
> X-uidl that stoopid spammers blindly copy into their messages.

The other thing on my todo list (probably tonight's tram ride home) is
to add all headers from non-text parts of multipart messages. If nothing
else, it'll pick up most virus email real quick.



-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.


From gward@python.net  Fri Sep  6 14:44:17 2002
From: gward@python.net (Greg Ward)
Date: Fri, 6 Sep 2002 09:44:17 -0400
Subject: [Spambayes] test sets?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCIEFEBCAB.tim.one@comcast.net>
References: <15735.50243.135743.32180@12-248-11-90.client.attbi.com>
	<LNBBLJKPBEHFEDALKOLCIEFEBCAB.tim.one@comcast.net>
Message-ID: <20020906134417.GA16820@cthulhu.gerg.ca>

On 05 September 2002, Tim Peters said:
> Greg Ward is
> currently capturing a stream coming into python.org, and I hope we can get a
> more modern, and cleaner, test set out of that.

Not yet -- still working on the required config changes.  But I have a
cunning plan...

> But if that stream contains
> any private email, it may not be ethically possible to make that available.

It will!  Part of my cunning plan involves something like this:

  if folder == "accepted":             # ie. not suspected junk mail
      if (len(recipients) == 1 and
          recipients[0] in ("guido@python.org", "barry@python.org", ...)):
          folder = "personal"

If you (and Guido, Barry, et. al.) prefer, I could change that last
statement to "folder = None", so the mail won't be saved at all.  I
*might* also add a "and sender doesn't look like -bounce-*, -request,
-admin, ..." clause to that if statement.

> Can you think of anyplace to get a large, shareable ham sample apart from a
> public mailing list?  Everyone's eager to share their spam, but spam is so
> much alike in so many ways that's the easy half of the data collection
> problem.

I believe the SpamAssassin maintainers have a scheme whereby the corpus
of non-spam is distributed, ie. several people have bodies of non-spam
that they use for collectively evolving the SA score set.  If that
sounds vague, it matches my level of understanding.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Reality is for people who can't handle science fiction.

From guido@python.org  Fri Sep  6 14:54:14 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 06 Sep 2002 09:54:14 -0400
Subject: [Spambayes] test sets?
In-Reply-To: Your message of "Fri, 06 Sep 2002 09:44:17 EDT."
             <20020906134417.GA16820@cthulhu.gerg.ca> 
References: <15735.50243.135743.32180@12-248-11-90.client.attbi.com>
	<LNBBLJKPBEHFEDALKOLCIEFEBCAB.tim.one@comcast.net>  
	<20020906134417.GA16820@cthulhu.gerg.ca> 
Message-ID: <200209061354.g86DsEE14105@pcp02138704pcs.reston01.va.comcast.net>

> I believe the SpamAssassin maintainers have a scheme whereby the corpus
> of non-spam is distributed, ie. several people have bodies of non-spam
> that they use for collectively evolving the SA score set.  If that
> sounds vague, it matches my level of understanding.

See if you can get a hold of that so we can do a level-playing-field
competition. :-)

--Guido van Rossum (home page: http://www.python.org/~guido/)

From gward@python.net  Fri Sep  6 14:57:18 2002
From: gward@python.net (Greg Ward)
Date: Fri, 6 Sep 2002 09:57:18 -0400
Subject: [Spambayes] Re: [Python-Dev] Getting started with GBayes testing
In-Reply-To: <200209060806.g8686ve03964@localhost.localdomain>
References: <BIEJKCLHCIOIHAGOKOLHGEEMDKAA.tim.one@comcast.net>
	<200209060806.g8686ve03964@localhost.localdomain>
Message-ID: <20020906135718.GC16820@cthulhu.gerg.ca>

On 06 September 2002, Anthony Baxter said:
> A snippet, hopefully not enough to trigger the spam-filters.

As an aside: one of the best ways to dodge SpamAssassin is by having an
In-Reply-To header.  Most list traffic should meet this criterion.

Alternately, I can whitelist mail to spambayes@python.org -- that'll
work until spammers get ahold of the list address, which usually seems
to take a few months.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Gee, I feel kind of LIGHT in the head now, knowing I can't make my
satellite dish PAYMENTS!

From barry@python.org  Fri Sep  6 15:23:19 2002
From: barry@python.org (Barry A. Warsaw)
Date: Fri, 6 Sep 2002 10:23:19 -0400
Subject: [Spambayes] test sets?
References: <15735.50243.135743.32180@12-248-11-90.client.attbi.com>
	<LNBBLJKPBEHFEDALKOLCIEFEBCAB.tim.one@comcast.net>
Message-ID: <15736.47703.689156.538539@anthem.wooz.org>


>>>>> "TP" == Tim Peters <tim.one@comcast.net> writes:

    >> Any thought to wrapping up your spam and ham test sets for
    >> inclusion w/ the spambayes project?

    TP> I gave it all the thought it deserved <wink>.  It would be
    TP> wonderful to get several people cranking on the same test
    TP> data, and I'm all in favor of that.  OTOH, my Data/ subtree
    TP> currently has more than 35,000 files slobbering over 134
    TP> million bytes -- even if I had a place to put that much stuff,
    TP> I'm not sure my ISP would let me email it in one msg <wink>.

Check it into the spambayes project.  SF's disks are cheap <wink>.

-Barry

From guido@python.org  Fri Sep  6 15:24:37 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 06 Sep 2002 10:24:37 -0400
Subject: [Spambayes] test sets?
In-Reply-To: Your message of "Fri, 06 Sep 2002 10:23:19 EDT."
             <15736.47703.689156.538539@anthem.wooz.org> 
References: <15735.50243.135743.32180@12-248-11-90.client.attbi.com>
	<LNBBLJKPBEHFEDALKOLCIEFEBCAB.tim.one@comcast.net>  
	<15736.47703.689156.538539@anthem.wooz.org> 
Message-ID: <200209061424.g86EOcd14363@pcp02138704pcs.reston01.va.comcast.net>

> Check it into the spambayes project.  SF's disks are cheap <wink>.

Perhaps more useful would be if Tim could check in the pickle(s?)
generated by one of his training runs, so that others can see how
Tim's training data performs against their own corpora.  This could
also be the starting point for a self-contained distribution (you've
got to start with *something*, and training with python-list data
seems just as good as anything else).

--Guido van Rossum (home page: http://www.python.org/~guido/)

From barry@python.org  Fri Sep  6 15:28:12 2002
From: barry@python.org (Barry A. Warsaw)
Date: Fri, 6 Sep 2002 10:28:12 -0400
Subject: [Spambayes] test sets?
References: <15735.50243.135743.32180@12-248-11-90.client.attbi.com>
	<LNBBLJKPBEHFEDALKOLCIEFEBCAB.tim.one@comcast.net>
	<20020906134417.GA16820@cthulhu.gerg.ca>
Message-ID: <15736.47996.84689.421662@anthem.wooz.org>


>>>>> "GW" == Greg Ward <gward@python.net> writes:

    GW> If you (and Guido, Barry, et. al.) prefer, I could change that
    GW> last statement to "folder = None", so the mail won't be saved
    GW> at all.

I don't care if the mail is foldered on python.org, but personal
messages regardless of who they're for, shouldn't be part of the
public spambayes repository unless specifically approved by both the
recipient and sender.

Note also that we are much more liberal about python.org/zope.org
mailing list traffic than most folks.  Read list-managers for any
length of time and you'll find that there are a lot of people who
assert strict copyright over their collections, are very protective of
their traffic, and got really pissed when gmane just started
gatewaying their messages without asking.

Which might be an appropriate for their lists, but not for ours (don't
think I'm suggesting we do the same -- I /like/ our laissez-faire
approach).

But for personal email, we should be more careful.
-Barry

From guido@python.org  Fri Sep  6 15:31:22 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 06 Sep 2002 10:31:22 -0400
Subject: [Spambayes] Deployment
Message-ID: <200209061431.g86EVM114413@pcp02138704pcs.reston01.va.comcast.net>

Quite independently from testing and tuning the algorithm, I'd like to
think about deployment.

Eventually, individuals and postmasters should be able to download a
spambayes software distribution, answer a few configuration questions
about their mail setup, training and false positives, and install it
as a filter.

A more modest initial goal might be the production of a tool that can
easily be used by individuals (since we're more likely to find
individuals willing to risk this than postmasters).

There are many ways to do this.  Some ideas:

- A program that acts both as a pop client and a pop server.  You
  configure it by telling it about your real pop servers.  You then
  point your mail reader to the pop server at localhost.  When it
  receives a connection, it connects to the remote pop servers, reads
  your mail, and gives you only the non-spam.  To train it, you'd only
  need to send it the false negatives somehow; it can assume that
  anything is ham that you don't say is spam within 48 hours.

- A server with a custom protocol that you send a copy of a message
  and that answers "spam" or "ham".  Then you have a little program
  that is invoked e.g. by procmail that talks to the server.  (The
  server exists so that it doesn't have to load the pickle with the
  scoring database for each message.  I don't know how big that pickle
  would be, maybe loading it each time is fine.  Or maybe
  marshalling.)

- Your idea here.

Takers?  How is ESR's bogofilter packaged?  SpamAssassin?  The Perl
Bayes filter advertised on slashdot?

--Guido van Rossum (home page: http://www.python.org/~guido/)

From barry@python.org  Fri Sep  6 15:38:56 2002
From: barry@python.org (Barry A. Warsaw)
Date: Fri, 6 Sep 2002 10:38:56 -0400
Subject: [Spambayes] test sets?
References: <15735.50243.135743.32180@12-248-11-90.client.attbi.com>
	<LNBBLJKPBEHFEDALKOLCIEFEBCAB.tim.one@comcast.net>
	<15736.47703.689156.538539@anthem.wooz.org>
	<200209061424.g86EOcd14363@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <15736.48640.283430.184348@anthem.wooz.org>


>>>>> "GvR" == Guido van Rossum <guido@python.org> writes:

    GvR> Perhaps more useful would be if Tim could check in the
    GvR> pickle(s?)  generated by one of his training runs, so that
    GvR> others can see how Tim's training data performs against their
    GvR> own corpora.

He could do that too. :)

-Barry

From bkc@murkworks.com  Fri Sep  6 15:39:48 2002
From: bkc@murkworks.com (Brad Clements)
Date: Fri, 06 Sep 2002 10:39:48 -0400
Subject: [Spambayes] Deployment
In-Reply-To: <200209061431.g86EVM114413@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <3D788653.9143.1D8992DA@localhost>

On 6 Sep 2002 at 10:31, Guido van Rossum wrote:

>   your mail, and gives you only the non-spam.  To train it, you'd only need
>   to send it the false negatives somehow; it can assume that anything is
>   ham that you don't say is spam within 48 hours.

I have folks who leave their email programs running 24 hours a day, constantly polling 
for mail. If they go away for a long weekend, lots of "friday night spam" will become 
ham on sunday night.  (Friday night seems to be the most popular time)


> - Your idea here.

Ultimately I'd like to see tight integration into the "most popular email clients".. As a 
stop-gap to the auto-ham ..

How about adding an IMAP server with a spam and deleted-ham folder. Most email 
clients can handle IMAP. Users should be able to quickly move "spam" into the spam 
folder. 

Instead of deleting messages (or, by reprogramming the delete function) they can 
quickly move ham into the ham folder.

In either case, the message would be processed and then destroyed.


Brad Clements,                bkc@murkworks.com   (315)268-1000
http://www.murkworks.com                          (315)268-9812 Fax
AOL-IM: BKClements


From tim.one@comcast.net  Fri Sep  6 15:45:27 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 06 Sep 2002 10:45:27 -0400
Subject: [Spambayes] test sets?
In-Reply-To: <15736.5577.157228.229200@12-248-11-90.client.attbi.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEHCBCAB.tim.one@comcast.net>

[Tim]
> OTOH, my Data/ subtree currently has more than 35,000 files slobbering
> over 134 million bytes -- even if I had a place to put that much stuff,
> I'm not sure my ISP would let me email it in one msg <wink>.

[Skip]
> Do you have a dialup or something more modern <wink>?

Much more modern:  a cable modem with a small upload rate cap.  There's a
reason the less modern uncapped @Home went out of business <wink>.

> 134MB of messages zipped would probably compress pretty well - under 50MB
> I'd guess with all the similarity in the headers and such.  You could zip
> each of the 10 sets individually and upload them somewhere.

I suppose this could finish over the course of an afternoon.  Now where's
"somewhere"?  I expect we'll eventually collect several datasets;
SourceForge isn't a good place for it (they expect projects to distribute
relatively small code files, and complain if even those get big).

> ...
> How about random sampling lots of public mailing lists via gmane or
> something similar, manually cleaning it (distributing that load over a
> number of people) and then relying on your clever code and your
> rebalancing script to help further cleanse it?

What then are we training the classifier to do?  Graham's scoring scheme is
based on an assumption that the ham-vs-spam task is *easy*, and half of that
is due to that the ham has a lot in common.  It was an experiment to apply
his scheme to all the comp.lang.python traffic, which is a lot broader than
he had in mind (c.l.py has long had a generous definition of "on topic"
<wink>).  I don't expect good things to come of making it ever broader,
*unless* your goal is to investigate just how broad it can be made before it
breaks down.

> The "problem" with the ham is it tends to be much more tied to one person
> (not just intimate, but unique) than the spam.

Which is "a feature" from Graham's POV:  the more clues, the better this
"smoking guns only" approach should work.

> I save all incoming email for ten days (gzipped mbox format) before it
rolls
> over and disappears.  At any one time I think I have about 8,000-10,000
> messages.  Most of it isn't terribly personal (which I would cull before
> passing along anyway) and much of it is machine-generated, so would be of
> marginal use.  Finally, it's all ham-n-spam mixed together.  Do we call
> that an omelette or a Denny's Grand Slam?

Unless you're volunteering to clean it, tag it, package it, and distribute
it, I'd call it irrelevant <wink>.


From guido@python.org  Fri Sep  6 15:43:33 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 06 Sep 2002 10:43:33 -0400
Subject: [Spambayes] Deployment
In-Reply-To: Your message of "Fri, 06 Sep 2002 10:39:48 EDT."
             <3D788653.9143.1D8992DA@localhost> 
References: <3D788653.9143.1D8992DA@localhost> 
Message-ID: <200209061443.g86Ehie14557@pcp02138704pcs.reston01.va.comcast.net>

> >   your mail, and gives you only the non-spam.  To train it, you'd only need
> >   to send it the false negatives somehow; it can assume that anything is
> >   ham that you don't say is spam within 48 hours.
> 
> I have folks who leave their email programs running 24 hours a day,
> constantly polling for mail. If they go away for a long weekend,
> lots of "friday night spam" will become ham on sunday night.
> (Friday night seems to be the most popular time)

So we'll make this a config parameter.

> > - Your idea here.
> 
> Ultimately I'd like to see tight integration into the "most popular
> email clients".. As a stop-gap to the auto-ham ..

What's an auto-ham?

> How about adding an IMAP server with a spam and deleted-ham
> folder. Most email clients can handle IMAP. Users should be able to
> quickly move "spam" into the spam folder.

I personally don't think IMAP has a bright future, but for people who
do use it, that's certainly a good approach.

> Instead of deleting messages (or, by reprogramming the delete
> function) they can quickly move ham into the ham folder.

Yes.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.one@comcast.net  Fri Sep  6 15:59:38 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 06 Sep 2002 10:59:38 -0400
Subject: [Spambayes] test sets?
In-Reply-To: <200209060759.g867xcV03853@localhost.localdomain>
Message-ID: <LNBBLJKPBEHFEDALKOLCEEHFBCAB.tim.one@comcast.net>

[Anthony Baxter]
> I've got a test set here that's the last 3 and a bit years email to
> info@ekit.com and info@ekno.com - it's a really ugly set of 20,000+
> messages, currently broken into 7,000 spam, 9,000 ham, 9,000 currently
> unclassified. These addresses are all over the 70-some different
> ekit/ekno/ISIConnect websites, so they get a LOT of spam.
>
> As well as the usual spam, it also has customers complaining about
> credit card charges, it has people interested in the service and
> asking questions about long distance rates, &c &c &c. Lots and lots
> of "commercial" speech, in other words. Stuff that SA gets pretty
> badly wrong.

Can this corpus be shared?  I suppose not.

> I'm currently mangling it by feeding all parts (text, html, whatever
> else :) into the filters, as well as both a selected number of headers
> (to, from, content-type, x-mailer), and also a list of (header,
> count_of_header). This is showing up some nice stuff - e.g. the
> X-uidl that stoopid spammers blindly copy into their messages.

If we ever <wink> have a shared corpus, an easy refactoring of timtest
should allow to plug in different tokenizers.  I've only made three changes
to Graham's algorithm so far (well, I've made dozens -- only three survived
testing as proven winners), all the rest has been refining the tokenization
to provide better clues.

> I did have Received in there, but it's out for the moment, as it causes
> rates to drop.

That's ambiguous.  Accuracy rates or error rates, ham or spam rates?

> I'm also stripping out HTML tags, except for href="" and src="" - there's
> so so much goodness in them (note that I'm only keeping the contents of
> the attributes).

Mining embedded http/https/ftp thingies cut the false negative rate in half
in my tests (not keying off href, just scanning for anything that "looked
like" one); that was the single biggest f-n improvement I've seen.  It
didn't change the false positive rate.  So you know whether src added
additional power, or did you do both at once?


From skip@pobox.com  Fri Sep  6 16:01:51 2002
From: skip@pobox.com (Skip Montanaro)
Date: Fri, 6 Sep 2002 10:01:51 -0500
Subject: [Spambayes] Deployment
In-Reply-To: <200209061431.g86EVM114413@pcp02138704pcs.reston01.va.comcast.net>
References: <200209061431.g86EVM114413@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <15736.50015.881231.510395@12-248-11-90.client.attbi.com>


    Guido> Takers?  How is ESR's bogofilter packaged?  SpamAssassin?  The
    Guido> Perl Bayes filter advertised on slashdot?

Dunno about the other tools, but SpamAssassin is a breeze to incorporate
into a procmail environment.  Lots of people use it in many other ways.  For
performance reasons, many people run a spamd process and then invoke a small
C program called spamc which shoots the message over to spamd and passes the
result back out.  I think spambayes in incremental mode is probably fast
enough to not require such tricks (though I would consider changing the
pickle to an anydbm file).

Basic procmail usage goes something like this:

    :0fw
    | spamassassin -P

    :0
    * ^X-Spam-Status: Yes
    $SPAM

Which just says, "Run spamassassin -P reinjecting its output into the
processing stream.  If the resulting mail has a header which begins
"X-Spam-Status: Yes", toss it into the folder indicated by the variable
$SPAM.

SpamAssassin also adds other headers as well, which give you more detail
about how its tests fared.  I'd like to see spambayes operate in at least
this way: do its thing then return a message to stdout with a modified set
of headers which further processing downstream can key on.

Skip


From bkc@murkworks.com  Fri Sep  6 16:02:11 2002
From: bkc@murkworks.com (Brad Clements)
Date: Fri, 06 Sep 2002 11:02:11 -0400
Subject: [Spambayes] Deployment
In-Reply-To: <200209061443.g86EhXQ14543@pcp02138704pcs.reston01.va.comcast.net>
References: Your message of "Fri, 06 Sep 2002 10:39:48 EDT."
	<3D788653.9143.1D8992DA@localhost> 
Message-ID: <3D788B92.22739.1D9E0FD1@localhost>

Did you want this on the list? I'm replying to the list..

On 6 Sep 2002 at 10:43, Guido van Rossum wrote:

> What's an auto-ham?

Automatically marking something as ham after a given timeout.. regardless of how long 
that timeout is, someone is going to forget to submit the message back as spam. 

How many spams-as-hams can be accepted before the f-n rate gets unacceptable?


> > How about adding an IMAP server with a spam and deleted-ham
> > folder. Most email clients can handle IMAP. Users should be able to
> > quickly move "spam" into the spam folder.
> 
> I personally don't think IMAP has a bright future, but for people who
> do use it, that's certainly a good approach.
> 
> > Instead of deleting messages (or, by reprogramming the delete
> > function) they can quickly move ham into the ham folder.
> 
> Yes.

I view IMAP as a stop-gap measure until tighter integration with various email clients 
can be achieved.

I still feel it's better to require classification feedback from the recipient, rather than 
make any assumptions after some period of time passes. But this is an end-user issue 
and we're still at the algorithm stage.. ;-)




Brad Clements,                bkc@murkworks.com   (315)268-1000
http://www.murkworks.com                          (315)268-9812 Fax
AOL-IM: BKClements


From guido@python.org  Fri Sep  6 16:05:22 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 06 Sep 2002 11:05:22 -0400
Subject: [Spambayes] Deployment
In-Reply-To: Your message of "Fri, 06 Sep 2002 11:02:11 EDT."
             <3D788B92.22739.1D9E0FD1@localhost> 
References: "Your message of Fri, 06 Sep 2002 10:39:48 EDT."
	<3D788653.9143.1D8992DA@localhost>  
	<3D788B92.22739.1D9E0FD1@localhost> 
Message-ID: <200209061505.g86F5MM14762@pcp02138704pcs.reston01.va.comcast.net>

> > What's an auto-ham?
> 
> Automatically marking something as ham after a given
> timeout.. regardless of how long that timeout is, someone is going
> to forget to submit the message back as spam.

OK, here's a refinement.  Assuming very little spam comes through, we
only need to pick a small percentage of ham received as new training
ham to match the new training spam.  The program could randomly select
a sufficient number of saved non-spam msgs and ask the user to
validate this selection.  You could do this once a day or week (config
parameter).

> How many spams-as-hams can be accepted before the f-n rate gets
> unacceptable?

Config parameter.

> I view IMAP as a stop-gap measure until tighter integration with
> various email clients can be achieved.
> 
> I still feel it's better to require classification feedback from the
> recipient, rather than make any assumptions after some period of
> time passes. But this is an end-user issue and we're still at the
> algorithm stage.. ;-)

I'm trying to think about the end-user issues because I have nothing
to contribute to the algorithm at this point.  For deployment we need
both!

--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido@python.org  Fri Sep  6 16:06:26 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 06 Sep 2002 11:06:26 -0400
Subject: [Spambayes] Deployment
In-Reply-To: Your message of "Fri, 06 Sep 2002 10:01:51 CDT."
             <15736.50015.881231.510395@12-248-11-90.client.attbi.com> 
References: <200209061431.g86EVM114413@pcp02138704pcs.reston01.va.comcast.net>
	<15736.50015.881231.510395@12-248-11-90.client.attbi.com> 
Message-ID: <200209061506.g86F6Qo14777@pcp02138704pcs.reston01.va.comcast.net>

> Dunno about the other tools, but SpamAssassin is a breeze to incorporate
> into a procmail environment.  Lots of people use it in many other ways.  For
> performance reasons, many people run a spamd process and then invoke a small
> C program called spamc which shoots the message over to spamd and passes the
> result back out.  I think spambayes in incremental mode is probably fast
> enough to not require such tricks (though I would consider changing the
> pickle to an anydbm file).
> 
> Basic procmail usage goes something like this:
> 
>     :0fw
>     | spamassassin -P
> 
>     :0
>     * ^X-Spam-Status: Yes
>     $SPAM
> 
> Which just says, "Run spamassassin -P reinjecting its output into the
> processing stream.  If the resulting mail has a header which begins
> "X-Spam-Status: Yes", toss it into the folder indicated by the variable
> $SPAM.
> 
> SpamAssassin also adds other headers as well, which give you more detail
> about how its tests fared.  I'd like to see spambayes operate in at least
> this way: do its thing then return a message to stdout with a modified set
> of headers which further processing downstream can key on.

Do you feel capable of writing such a tool?  It doesn't look too hard.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From skip@pobox.com  Fri Sep  6 16:12:58 2002
From: skip@pobox.com (Skip Montanaro)
Date: Fri, 6 Sep 2002 10:12:58 -0500
Subject: [Spambayes] Deployment
In-Reply-To: <200209061443.g86Ehie14557@pcp02138704pcs.reston01.va.comcast.net>
References: <3D788653.9143.1D8992DA@localhost>
        <200209061443.g86Ehie14557@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <15736.50682.911121.462698@12-248-11-90.client.attbi.com>


    >> Ultimately I'd like to see tight integration into the "most popular
    >> email clients"..

The advantage of using a kitchen sink (umm, make that highly programmable)
editor+email package like Emacs+VM is that you can twiddle your key bindings
and write a little ELisp (or Pymacs) glue to toss messages in the right
direction (spam or ham).  For this, spambayes would have to operate in an
incremental fashion when fed a single ham or spam message.

(No, I have no idea what an "auto-ham" is.  A pig run over by a car,
perhaps?)

give-a-dog-a-bone-ly, y'rs,

Skip

From skip@pobox.com  Fri Sep  6 16:19:30 2002
From: skip@pobox.com (Skip Montanaro)
Date: Fri, 6 Sep 2002 10:19:30 -0500
Subject: [Spambayes] Deployment
In-Reply-To: <200209061506.g86F6Qo14777@pcp02138704pcs.reston01.va.comcast.net>
References: <200209061431.g86EVM114413@pcp02138704pcs.reston01.va.comcast.net>
        <15736.50015.881231.510395@12-248-11-90.client.attbi.com>
        <200209061506.g86F6Qo14777@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <15736.51074.369911.905337@12-248-11-90.client.attbi.com>


    >> Dunno about the other tools, but SpamAssassin is a breeze ...
    >> SpamAssassin also adds other headers as well, which give you more
    >> detail ...

    Guido> Do you feel capable of writing such a tool?  It doesn't look too
    Guido> hard.

Sure, but at the moment I have to stop reading email for a few hours and do
some real work. ;-) I'll see if I can modify GBayes.py suitably over the
weekend.

Skip

From tim.one@comcast.net  Fri Sep  6 16:45:43 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 06 Sep 2002 11:45:43 -0400
Subject: [Spambayes] test sets?
In-Reply-To: <200209060759.g867xcV03853@localhost.localdomain>
Message-ID: <LNBBLJKPBEHFEDALKOLCAEHLBCAB.tim.one@comcast.net>

[Anthony Baxter]
> ... and also a list of (header,count_of_header). This is showing up some
> nice stuff - e.g. the X-uidl that stoopid spammers blindly copy into their
> messages.

This is a very cool idea.  I'm currently special-casing the absence of an
Organization line because that proved to give a tiny reduction in the f-n
rate.  But counting the # of each kind of header line clearly gives the same
info at comparable cost and is much more general.

Unfortunately, on my corpora it turns out to be *too* strong, because of
header fields injected by Mailman into the c.l.py msg archive.  Here's a
list of all header fields + counts found in my first spam corpus
(Spam/Set1/*.txt), from most valuable to least ("most valuable" here ==
farthest from spamprob 0.5, and then by total # of hams and spams it
appeared in).

X-UIDL variants are strong spam indicators here in a probability sense, but
are extremely rare even in spam.

Barry, can you please identify for me which of these headers are Mailman
artifacts so I can avoid counting them?

Each line is of the form

    spam_probability  raw_spam_count  raw_ham_count  token

where token is

    "header:"  header_name  ":"  count

Note that the "probabilities" don't make sense <wink>:  as always, the ham
counts are multiplied by 2 (HAMBIAS) artificially before computing a prob.

 prob nspam  nham token
 0.01    19  3559 'header:X-Mailman-Version:1'
 0.01    19  3559 'header:List-Id:1'
 0.01    19  3557 'header:X-BeenThere:1'
 0.01     0  3093 'header:Newsgroups:1'
 0.01     0  3054 'header:Xref:1'
 0.01     0  3053 'header:Path:1'
 0.01     0  2846 'header:References:1'
 0.01    24  2760 'header:Organization:1'
 0.99  2685    14 'header:Content-Length:1'
 0.01    19  2668 'header:List-Unsubscribe:1'
 0.01    19  2668 'header:List-Subscribe:1'
 0.01    19  2668 'header:List-Post:1'
 0.01    19  2668 'header:List-Help:1'
 0.01    19  2668 'header:List-Archive:1'
 0.01     0  2652 'header:NNTP-Posting-Host:1'
 0.01     0  2058 'header:X-Trace:1'
 0.01     0  1756 'header:X-Complaints-To:1'
 0.01     0  1655 'header:NNTP-Posting-Date:1'
 0.01     0  1320 'header:X-Newsreader:1'
 0.01     1   941 'header:User-Agent:1'
 0.99   689     0 'header:Delivered-To:4'
 0.01     2   627 'header:X-Accept-Language:1'
 0.01     0   538 'header:In-Reply-To:1'
 0.99   522     0 'header:Delivered-To:3'
 0.99   519     0 'header:Received:8'
 0.99   466     1 'header:Received:7'
 0.99   364     0 'header:Return-Path:2'
 0.99   273     0 'header:MiME-Version:1'
 0.01     0   149 'header:X-Spam-Status:1'
 0.01     0   103 'header:X-Http-User-Agent:1'
 0.01     0   103 'header:X-Http-Proxy:1'
 0.01     0   103 'header:X-Article-Creation-Date:1'
 0.01     0   103 'header:X-Abuse-Info:2'
 0.01     0    99 'header:X-Spam-Level:1'
 0.01     0    84 'header:X-MyDeja-Info:1'
 0.01     0    77 'header:X-Face:1'
 0.01     0    76 'header:Mail-Followup-To:1'
 0.01     0    68 'header:In-reply-to:1'
 0.99    52     0 'header:Spam-Apparently-To:1'
 0.01     0    49 'header:X-Server-Date:1'
 0.01     0    47 'header:cc:1'
 0.99    44     0 'header:Received:10'
 0.01     0    35 'header:X-Original-NNTP-Posting-Host:1'
 0.01     0    32 'header:Distribution:1'
 0.99    31     0 'header:X-MailID:1'
 0.99    31     0 'header:Complain-To:1'
 0.01     0    30 'header:X-NNTP-Posting-Host:1'
 0.99    29     0 'header:X-Encoding:1'
 0.01     0    28 'header:X-Mimeole:1'
 0.99    27     0 'header:1:1'
 0.01     0    27 'header:X-Originally-To:1'
 0.01     0    26 'header:Cache-Post-Path:1'
 0.99    25     0 'header:X-Mailing-List:1'
 0.01     0    25 'header:Mail-Copies-To:1'
 0.99    24     0 'header:SUBJECT:1'
 0.99    24     0 'header:DATE:1'
 0.01     0    24 'header:X-Received-Date:1'
 0.99    23     0 'header:x-esmtp:1'
 0.99    23     0 'header:X-Precedence-Ref:1'
 0.01     0    23 'header:X-Cache:1'
 0.99    22     0 'header:FROM:1'
 0.01     0    22 'header:X-Report:1'
 0.01     0    22 'header:X-Orig-Message-ID:1'
 0.01     0    22 'header:NNTP-Proxy-Relay:1'
 0.01     0    22 'header:NNTP-Posting-Time:1'
 0.01     0    22 'header:Abuse-Reports-To:1'
 0.01     0    21 'header:X-Comment-To:1'
 0.99    20     0 'header:Received:11'
 0.01     0    20 'header:X-Comments:1'
 0.01     0    20 'header:X-Authenticated-User:1'
 0.01     0    20 'header:X-Abuse-Info:1'
 0.99    19     0 'header:X-List-Name:1'
 0.99    19     0 'header:X-List-Manager:1'
 0.99    19     0 'header:X-Library:1'
 0.01     0    19 'header:X-Comments2:1'
 0.01     0    19 'header:Content-Transfer-Encoding:2'
 0.01     0    18 'header:X-Original-Path:1'
 0.01     0    18 'header:X-Comments3:1'
 0.01     0    16 'header:X-Attribution:1'
 0.99    15     0 'header:X-Stormpost-To:1'
 0.99    15     0 'header:X-List-Unsubscribe:1'
 0.99    15     0 'header:X-EM-Version:1'
 0.99    15     0 'header:X-EM-Registration:1'
 0.01     0    15 'header:X-Posting-Agent:1'
 0.01     0    15 'header:X-Orig-Path:1'
 0.01     0    15 'header:X-In-Reply-To:1'
 0.01     0    14 'header:X-X-Sender:1'
 0.01     0    14 'header:X-UserInfo1:1'
 0.01     0    14 'header:X-Filtered-By:1'
 0.01     0    14 'header:Mime-version:1'
 0.01     0    13 'header:X-scanner:1'
 0.01     0    13 'header:X-Url:1'
 0.01     0    13 'header:X-Filename:1'
 0.99    12     0 'header:TO:1'
 0.01     0    12 'header:X-Gateway:1'
 0.01     0    12 'header:X-FTNADDR:1'
 0.01     0    12 'header:X-Envelope-To:1'
 0.01     0    12 'header:Content-Encoding:1'
 0.01     0    11 'header:X-Nntp-Posting-Host:1'
 0.99    10     0 'header:X-UIDL:1'
 0.01     0    10 'header:X-Reposted-By:1'
 0.01     0    10 'header:X-Repost-Date:1'
 0.01     0    10 'header:X-Original-Message-ID:1'
 0.01     0    10 'header:X-Oblique-Strategy:1'
 0.01     0    10 'header:X-No-Productlinks:1'
 0.01     0    10 'header:X-MS-TNEF-Correlator:1'
 0.01     0    10 'header:X-MS-Has-Attach:1'
 0.01     0    10 'header:X-Comments:5'
 0.99     9     0 'header:Comments:1'
 0.01     0     9 'header:X-eGroups-Return:1'
 0.01     0     9 'header:X-No-Archive:1'
 0.99     8     0 'header:X-PMG-Userid:1'
 0.99     8     0 'header:X-PMG-Recipient:1'
 0.99     8     0 'header:X-PMG-Msgid:1'
 0.99     8     0 'header:X-PMFLAGS:1'
 0.99     8     0 'header:X-Info:2'
 0.01     0     8 'header:X-Path-Notice:1'
 0.01     0     8 'header:Originator:1'
 0.01     0     8 'header:Cancel-Lock:1'
 0.99     7     0 'header:X-x:1'
 0.01     0     7 'header:content-class:1'
 0.01     0     7 'header:X-Organization:1'
 0.01     0     7 'header:Followup-To:1'
 0.99     6     0 'header:X-MDRemoteIP:1'
 0.01     0     6 'header:X-Wren-Trace:1'
 0.01     0     6 'header:X-Squaresville:1'
 0.01     0     6 'header:X-Originating-Host:1'
 0.01     0     6 'header:X-Niggle:1'
 0.01     0     6 'header:X-NETCOM-Date:1'
 0.01     0     6 'header:X-Moon-Phase:1'
 0.01     0     6 'header:X-Licznik:1'
 0.01     0     6 'header:X-Copyright:1'
 0.01     0     6 'header:NNTP-Posting-User:1'
 0.01     0     6 'header:Bytes:1'
 0.99     5     0 'header:X-Info:1'
 0.99     5     0 'header:Received:12'
 0.01     0     5 'header:X-homepage:1'
 0.01     0     5 'header:X-Original-Trace:1'
 0.01     0     5 'header:X-Mail2News-Path:1'
 0.01     0     5 'header:X-Editor:1'
 0.01     0     5 'header:X-Admin:1'
 0.01     0     5 'header:Supersedes:1'
 0.01     0     5 'header:Keywords:1'
 0.01     0     4 'header:X-Trace-PostClient-IP:1'
 0.01     0     4 'header:X-Meow:1'
 0.01     0     4 'header:X-GC-Trace:1'
 0.01     0     4 'header:X-DMCA-Complaints-To:1'
 0.01     0     4 'header:Posted-And-Mailed:1'
 0.01     0     3 'header:X-question:1'
 0.01     0     3 'header:X-Sun-Charset:1'
 0.01     0     3 'header:X-Spam-Status:2'
 0.01     0     3 'header:X-Spam-Level:2'
 0.01     0     3 'header:X-SessionID:1'
 0.01     0     3 'header:X-Path-Stamp:1'
 0.01     0     3 'header:X-PGP-Fingerprint:1'
 0.01     0     3 'header:X-Original-NNTP-Posting-Host:2'
 0.01     0     3 'header:X-Home-Page:1'
 0.01     0     3 'header:X-Hash:1'
 0.01     0     3 'header:X-Hash-Info:1'
 0.01     0     3 'header:X-AntiVirus:1'
 0.01     0     3 'header:Original-Sender:1'
 0.01     0     3 'header:Mail-copies-to:1'
 0.01     0     3 'header:Content-ID:1'
 0.99   133     1 'header:Received:9'
 0.02    55  3559 'header:Errors-To:1'
 0.02    65  3559 'header:Precedence:1'
 0.04     3    49 'header:X-MIMEOLE:1'
 0.95   239     9 'header:X-OriginalArrivalTime:1'
 0.05     2    26 'header:Priority:1'
 0.06     1    11 'header:Thread-Topic:1'
 0.07     1    10 'header:X-MIME-Autoconverted:1'
 0.07   219  3647 'header:Sender:1'
 0.07     1     9 'header:X-URL:1'
 0.11    19   111 'header:Content-Disposition:1'
 0.88   135    13 'header:Received:6'
 0.86  2373   277 'header:Return-Path:1'
 0.14   191   863 'header:X-MimeOLE:1'
 0.86    17     2 'header:X-Return-Path:1'
 0.86    17     2 'header:X-MDaemon-Deliver-To:1'
 0.85    15     2 'header:X-X:1'
 0.16    12    46 'header:Content-transfer-encoding:1'
 0.84    28     4 'header:Content-Class:1'
 0.20     7    21 'header:X-Authentication-Warning:1'
 0.20     4    12 'header:X-MSMail-priority:1'
 0.20     1     3 'header:thread-index:1'
 0.80  1471   273 'header:Delivered-To:1'
 0.76   480   112 'header:Message-Id:1'
 0.74    16     4 'header:Delivered-To:2'
 0.74     4     1 'header:X-SLUIDL:1'
 0.72   117    33 'header:CC:1'
 0.29     5     9 'header:X-Apparently-From:1'
 0.71    17     5 'header:Bcc:1'
 0.71    27     8 'header:Thread-Index:1'
 0.29     8    14 'header:X-Msmail-Priority:1'
 0.69    31    10 'header:X-MIMETrack:1'
 0.31   564   903 'header:Mime-Version:1'
 0.32    13    20 'header:MIME-version:1'
 0.35   349   474 'header:Received:2'
 0.65   756   297 'header:Received:3'
 0.36    94   123 'header:X-Sender:1'
 0.36  1561  3867 'header:Message-ID:1'
 0.37    16    20 'header:Message-id:1'
 0.63   329   141 'header:Importance:1'
 0.62   100    44 'header:Content-type:1'
 0.61    68    31 'header:Received:5'
 0.39   983  1096 'header:X-Mailer:1'
 0.40   754   827 'header:X-MSMail-Priority:1'
 0.41   293   307 'header:Cc:1'
 0.59  1158   587 'header:Reply-To:1'
 0.42  1558  1575 'header:Content-Transfer-Encoding:1'
 0.42    16    16 'header:X-mailer:1'
 0.58   224   119 'header:Received:4'
 0.42   882   875 'header:X-Priority:1'
 0.43  2090  3097 'header:Lines:1'
 0.46  1397  1199 'header:MIME-Version:1'
 0.47  2410  2207 'header:Content-Type:1'
 0.47  2450  4000 'header:Date:1'
 0.48    15    12 'header:X-Originating-IP:1'
 0.52    15    10 'header:Received:1'
 0.48    31    24 'header:Reply-to:1'
 0.49  2627  3989 'header:To:1'
 0.50  2700  3999 'header:Subject:1'
 0.50  2704  4000 'header:From:1'
 0.50     4     0 'header:X-RMD-Text:1'
 0.50     4     0 'header:X-Owner:1'
 0.50     4     0 'header:Cc:90'
 0.50     4     0 'header:2:1'
 0.50     3     0 'header:received:1'
 0.50     3     0 'header:microsoft:1'
 0.50     3     0 'header:X-Unsent:1'
 0.50     3     0 'header:X-UID:1'
 0.50     3     0 'header:X-Set:1'
 0.50     3     0 'header:X-SMTPExp-Version:1'
 0.50     3     0 'header:X-SMTPExp-Registration:1'
 0.50     3     0 'header:X-PLATTER:1'
 0.50     3     0 'header:X-MB-Pid:1'
 0.50     3     0 'header:X-MB-Mid:1'
 0.50     3     0 'header:X-Info-2:1'
 0.50     3     0 'header:X-Info-1:1'
 0.50     3     0 'header:X-IONK:1'
 0.50     3     0 'header:X-Debug:1'
 0.50     3     0 'header:X-CRUNCHERS:1'
 0.50     3     0 'header:X-CORONNA:1'
 0.50     3     0 'header:X-BlackMail:1'
 0.50     3     0 'header:X-Authenticated-Timestamp:1'
 0.50     3     0 'header:Errors-to:1'
 0.50     3     0 'header:Disposition-Notification-To:1'
 0.50     3     0 'header:6:1'
 0.50     3     0 'header:5:1'
 0.50     3     0 'header:4:1'
 0.50     3     0 'header:3:1'
 0.50     2     1 'header:X-MDRcpt-To:1'
 0.50     2     0 'header:X-vsuite-type:1'
 0.50     2     0 'header:X-Tracking:1'
 0.50     2     0 'header:X-Sent-Mail:1'
 0.50     2     0 'header:X-Sender-Ip:1'
 0.50     2     0 'header:X-Originating-Ip:1'
 0.50     2     0 'header:X-Hops:1'
 0.50     2     0 'header:X-Expiredinmiddle:1'
 0.50     2     0 'header:X-AntiAbuse:5'
 0.50     2     0 'header:X-:1'
 0.50     2     0 'header:Date-warning:1'
 0.50     2     0 'header:Content-Location:1'
 0.50     2     0 'header:Content-Language:1'
 0.50     2     0 'header:Cc:95'
 0.50     0     2 'header:X-VirusChecked:1'
 0.50     0     2 'header:X-Uptime:1'
 0.50     0     2 'header:X-RBL-Warning:1'
 0.50     0     2 'header:X-Public-Domain:1'
 0.50     0     2 'header:X-Postfilter:1'
 0.50     0     2 'header:X-Poster-Key:1'
 0.50     0     2 'header:X-Operating-System:1'
 0.50     0     2 'header:X-OS:1'
 0.50     0     2 'header:X-News-Software:1'
 0.50     0     2 'header:X-NFilter:1'
 0.50     0     2 'header:X-MailScanner:1'
 0.50     0     2 'header:X-Lotus-FromDomain:1'
 0.50     0     2 'header:X-Loop-Detect:1'
 0.50     0     2 'header:X-Inktomi-Trace:1'
 0.50     0     2 'header:X-Delivery-Agent:1'
 0.50     0     2 'header:X-DMCA-Notifications:1'
 0.50     0     2 'header:X-Complaints-To:2'
 0.50     0     2 'header:X-Complaints-Info:1'
 0.50     0     2 'header:X-BeenThere:3'
 0.50     0     2 'header:X-Access:1'
 0.50     0     2 'header:X-Abuse-and-DMCA-Info:2'
 0.50     0     2 'header:Status:1'
 0.50     0     2 'header:Mail-From:1'
 0.50     0     2 'header:Content-return:1'
 0.50     0     2 'header:'
 0.50     1     0 'header:http:1'
 0.50     1     0 'header:content-type:1'
 0.50     1     0 'header:X-zippo:1'
 0.50     1     0 'header:X-WSS-ID:2'
 0.50     1     0 'header:X-WSS-ID:1'
 0.50     1     0 'header:X-Sybari-Space:1'
 0.50     1     0 'header:X-Server-Uuid:2'
 0.50     1     0 'header:X-Server-Uuid:1'
 0.50     1     0 'header:X-Sanitizer:1'
 0.50     1     0 'header:X-Reply-To:1'
 0.50     1     0 'header:X-Rcpt-To:1'
 0.50     1     0 'header:X-RAV-Antivirus:1'
 0.50     1     0 'header:X-PROJECT-ID:1'
 0.50     1     0 'header:X-Msmail-priority:1'
 0.50     1     0 'header:X-MessageNo:1'
 0.50     1     0 'header:X-MDSend-Notifications-To:1'
 0.50     1     0 'header:X-MDMailing-List:1'
 0.50     1     0 'header:X-MAILER:1'
 0.50     1     0 'header:X-Lookup-Warning:1'
 0.50     1     0 'header:X-JsMail-Priority:1'
 0.50     1     0 'header:X-Infomail-Spawn:1'
 0.50     1     0 'header:X-Infomail-Id:1'
 0.50     1     0 'header:X-GWIA:1'
 0.50     1     0 'header:X-GCMulti:1'
 0.50     1     0 'header:X-Envelope-From:1'
 0.50     1     0 'header:X-David-Sym:1'
 0.50     1     0 'header:X-David-Flags:1'
 0.50     1     0 'header:X-Auto-Forward:1'
 0.50     1     0 'header:To:45'
 0.50     1     0 'header:Reply-To:2'
 0.50     1     0 'header:Received:13'
 0.50     1     0 'header:MIME-Version:4'
 0.50     1     0 'header:Illegal-Object:1'
 0.50     1     0 'header:Delivered-To:5'
 0.50     1     0 'header:Cc:82'
 0.50     1     0 'header:Cc:80'
 0.50     1     0 'header:Cc:74'
 0.50     1     0 'header:Cc:24'
 0.50     0     1 'header:x-message-flag:1'
 0.50     0     1 'header:subject:1'
 0.50     0     1 'header:X400-MTS-Identifier:1'
 0.50     0     1 'header:X-uri:1'
 0.50     0     1 'header:X-no-archive:1'
 0.50     0     1 'header:X-news-hint:1'
 0.50     0     1 'header:X-news-hint2:1'
 0.50     0     1 'header:X-Virus-Scanned:1'
 0.50     0     1 'header:X-User:1'
 0.50     0     1 'header:X-TMDA-Fingerprint:1'
 0.50     0     1 'header:X-TCP-IDENTITY:1'
 0.50     0     1 'header:X-T.O.S.:1'
 0.50     0     1 'header:X-Silly:1'
 0.50     0     1 'header:X-Report-Abuse-To:1'
 0.50     0     1 'header:X-Real-Address:1'
 0.50     0     1 'header:X-ROUTED:1'
 0.50     0     1 'header:X-Posting-IP:1'
 0.50     0     1 'header:X-Phone:1'
 0.50     0     1 'header:X-PGP:1'
 0.50     0     1 'header:X-PGP-Key:1'
 0.50     0     1 'header:X-PGP-Key-Fingerprint:1'
 0.50     0     1 'header:X-Original-To:1'
 0.50     0     1 'header:X-Original-NNTP-Posting-Host:3'
 0.50     0     1 'header:X-Orig-X-Trace:1'
 0.50     0     1 'header:X-Orig-X-Complaints-To:1'
 0.50     0     1 'header:X-Orig-NNTP-Posting-Host:1'
 0.50     0     1 'header:X-Orig-NNTP-Posting-Date:1'
 0.50     0     1 'header:X-Organisation:1'
 0.50     0     1 'header:X-Notice:2'
 0.50     0     1 'header:X-No-Repost:1'
 0.50     0     1 'header:X-Nntp-Posting-Date:1'
 0.50     0     1 'header:X-Newsposter:1'
 0.50     0     1 'header:X-Newsgroups:1'
 0.50     0     1 'header:X-Mail-Copies-To:1'
 0.50     0     1 'header:X-MTA:1'
 0.50     0     1 'header:X-Location:1'
 0.50     0     1 'header:X-Islamic-Date:1'
 0.50     0     1 'header:X-HomePage:1'
 0.50     0     1 'header:X-Get-A-Real-Newsreader:1'
 0.50     0     1 'header:X-GPG-Key-ID:1'
 0.50     0     1 'header:X-GPG-Fingerprint:1'
 0.50     0     1 'header:X-From:1'
 0.50     0     1 'header:X-Flags:1'
 0.50     0     1 'header:X-Fax:1'
 0.50     0     1 'header:X-Favorite-Dwarf:1'
 0.50     0     1 'header:X-Faculty:1'
 0.50     0     1 'header:X-Eudora-Signature:1'
 0.50     0     1 'header:X-Eric-Conspiracy:1'
 0.50     0     1 'header:X-Enigmail-Version:1'
 0.50     0     1 'header:X-Enigmail-Supports:1'
 0.50     0     1 'header:X-Emacs:1'
 0.50     0     1 'header:X-Emacs-Acronym:1'
 0.50     0     1 'header:X-ELN-Date:1'
 0.50     0     1 'header:X-Draft-From:1'
 0.50     0     1 'header:X-Disclaimer:1'
 0.50     0     1 'header:X-Cyberus:1'
 0.50     0     1 'header:X-Commercial-ReplyTo:1'
 0.50     0     1 'header:X-Bpc-Relay-Sender-Host:1'
 0.50     0     1 'header:X-Bpc-Relay-Info:1'
 0.50     0     1 'header:X-Bpc-Relay-Envelope-From:1'
 0.50     0     1 'header:X-BeOS-Platform:1'
 0.50     0     1 'header:X-Authentication-Info:1'
 0.50     0     1 'header:X-Attachments:1'
 0.50     0     1 'header:X-Added:1'
 0.50     0     1 'header:X-Abuse-Info2:1'
 0.50     0     1 'header:UA-Content-Id:1'
 0.50     0     1 'header:To:2'
 0.50     0     1 'header:Sensitivity:1'
 0.50     0     1 'header:Phone:1'
 0.50     0     1 'header:Original-Encoded-Information-Types:1'
 0.50     0     1 'header:Nntp-Posting-Host:1'
 0.50     0     1 'header:Microsoft:1'
 0.50     0     1 'header:MMDF-Warning:1'
 0.50     0     1 'header:Injector-Info:1'
 0.50     0     1 'header:Favorite-Color:1'
 0.50     0     1 'header:Content-disposition:1'
 0.50     0     1 'header:Content-description:1'
 0.50     0     1 'header:Content-MD5:1'
 0.50     0     1 'header:Content-Identifier:1'
 0.50     0     1 'header:Cc:2'
 0.50     0     1 'header:Bcc:2'
 0.50     0     1 'header:Autoforwarded:1'
 0.50     0     1 'header:Archive-Name:1'


From gward@python.net  Fri Sep  6 16:55:09 2002
From: gward@python.net (Greg Ward)
Date: Fri, 6 Sep 2002 11:55:09 -0400
Subject: [Spambayes] test sets?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCAEHLBCAB.tim.one@comcast.net>
References: <200209060759.g867xcV03853@localhost.localdomain>
	<LNBBLJKPBEHFEDALKOLCAEHLBCAB.tim.one@comcast.net>
Message-ID: <20020906155509.GA17800@cthulhu.gerg.ca>

On 06 September 2002, Tim Peters said:
>  prob nspam  nham token
>  0.99  2685    14 'header:Content-Length:1'

That might be a bias of Bruce Guenter's spam collection.

>  0.99   689     0 'header:Delivered-To:4'

And this *definitely* is, because Bruce is a qmail guy.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Life is too short for ordinary music.

From nas@python.ca  Fri Sep  6 16:57:05 2002
From: nas@python.ca (Neil Schemenauer)
Date: Fri, 6 Sep 2002 08:57:05 -0700
Subject: [Spambayes] Deployment
In-Reply-To: <200209061443.g86Ehie14557@pcp02138704pcs.reston01.va.comcast.net>
References: <3D788653.9143.1D8992DA@localhost>
	<200209061443.g86Ehie14557@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <20020906155705.GA22115@glacier.arctrix.com>

Guido van Rossum wrote:
> I personally don't think IMAP has a bright future, but for people who
> do use it, that's certainly a good approach.

Writing an IMAP server is a non-trivial task.  The specification is huge
and clients do all kinds of weird stuff.  POP is very easy in
comparison.  Perhaps you could forward messages to a special address or
save them in a special folder to mark them as false negatives.

Alternatively, perhaps there could be a separate protocol and client
that could be used to review additions to the training set.  Each day a
few random spam and ham messages could be grabbed as candidates.
Someone would periodically startup the client, review the candidates,
reclassify or remove any messages they don't like and add them to the
training set.

  Neil

From tim.one@comcast.net  Fri Sep  6 17:21:51 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 06 Sep 2002 12:21:51 -0400
Subject: [Spambayes] test sets?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCAEHLBCAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEHPBCAB.tim.one@comcast.net>

[Tim]
> ...
> Unfortunately, on my corpora it turns out to be *too* strong,
> ...

Here's what happens if I leave all the header counts in:

false positive percentages
    0.000  0.000  tied
    0.000  0.000  tied
    0.100  0.025  won    -75.00%
    0.000  0.000  tied
    0.025  0.000  won   -100.00%
    0.025  0.000  won   -100.00%
    0.100  0.025  won    -75.00%
    0.025  0.000  won   -100.00%
    0.025  0.000  won   -100.00%
    0.050  0.000  won   -100.00%
    0.100  0.000  won   -100.00%
    0.025  0.000  won   -100.00%
    0.025  0.025  tied
    0.025  0.000  won   -100.00%
    0.025  0.000  won   -100.00%
    0.025  0.000  won   -100.00%
    0.025  0.025  tied
    0.000  0.000  tied
    0.025  0.000  won   -100.00%
    0.100  0.025  won    -75.00%

won  14 times
tied  6 times
lost  0 times

total unique fp went from 9 to 2

false negative percentages
    0.364  0.145  won    -60.16%
    0.400  0.291  won    -27.25%
    0.400  0.364  won     -9.00%
    0.909  0.618  won    -32.01%
    0.836  0.545  won    -34.81%
    0.618  0.473  won    -23.46%
    0.291  0.291  tied
    1.018  0.654  won    -35.76%
    0.982  0.655  won    -33.30%
    0.727  0.545  won    -25.03%
    0.800  0.618  won    -22.75%
    1.163  0.872  won    -25.02%
    0.764  0.545  won    -28.66%
    0.473  0.291  won    -38.48%
    0.473  0.327  won    -30.87%
    0.727  0.509  won    -29.99%
    0.655  0.400  won    -38.93%
    0.509  0.218  won    -57.17%
    0.545  0.364  won    -33.21%
    0.509  0.436  won    -14.34%

won  19 times
tied  1 times
lost  0 times

total unique fn went from 168 to 124

A false positive *really* has to work hard then, eh?  The long quote of a
Nigerian scam letter is one of the two that made it, and spamprob() looked
at all this stuff before deciding it was spam:

prob = 0.999945196947
prob('domestic') = 0.99
prob('dollars)') = 0.99
prob('solicit') = 0.99
prob('partner.') = 0.99
prob('accounts,') = 0.99
prob('federal') = 0.99
prob('nigeria.') = 0.99
prob('ministry') = 0.99
prob('subject:Business') = 0.99
prob('overseas') = 0.99
prob('housing') = 0.99
prob('nigeria') = 0.99
prob('nigerian') = 0.99
prob('estate') = 0.99
prob('70%') = 0.99
prob('regime') = 0.99
prob('payment') = 0.99
prob('header:X-Complaints-To:1') = 0.01
prob('header:X-BeenThere:1') = 0.01
prob('header:NNTP-Posting-Host:1') = 0.01
prob('ended.') = 0.01
prob('wrote') = 0.01
prob('header:Path:1') = 0.01
prob('header:NNTP-Posting-Date:1') = 0.01
prob('header:X-Mailman-Version:1') = 0.01
prob('header:List-Id:1') = 0.01
prob('header:List-Archive:1') = 0.01
prob('header:X-Trace:1') = 0.01
prob('header:Organization:1') = 0.01
prob('header:Newsgroups:1') = 0.01
prob('header:List-Post:1') = 0.01
prob('header:References:1') = 0.01
prob('header:List-Help:1') = 0.01
prob('header:X-Newsreader:1') = 0.01
prob('states') = 0.959986
prob('united') = 0.96139
prob('money') = 0.964852
prob('country.') = 0.97034
prob('civil') = 0.96754
prob('partner') = 0.969003
prob('complex,') = 0.01
prob('funds') = 0.972142
prob('million') = 0.971369
prob('purchase') = 0.986651
prob('government') = 0.985578
prob('header:Precedence:1') = 0.0306554
prob('header:Xref:1') = 0.01
prob('header:List-Subscribe:1') = 0.01
prob('header:List-Unsubscribe:1') = 0.01
prob('header:Errors-To:1') = 0.0182013

It actually found more 0.01 clues than 0.99 ones then, but the content is
*so* bad nothing can overcome the judgment of guilt.

BTW, the false negative rate in my corpora is also getting near the point
where I won't be able to measure improvement reliably.  Since there are only
2750 spams in a spam set, 1% is 27.5 spams, whereas in the ham corpus 1% is
40 hams.  So, e.g., a f-n rate of 0.364% means a grand total of 10 false
negatives, so even changing that by 1 measly msg makes a 10% difference in
the f-n rate.


From barry@python.org  Fri Sep  6 17:23:33 2002
From: barry@python.org (Barry A. Warsaw)
Date: Fri, 6 Sep 2002 12:23:33 -0400
Subject: [Spambayes] Deployment
References: <3D788653.9143.1D8992DA@localhost>
	<200209061443.g86Ehie14557@pcp02138704pcs.reston01.va.comcast.net>
	<20020906155705.GA22115@glacier.arctrix.com>
Message-ID: <15736.54917.688066.738120@anthem.wooz.org>


>>>>> "NS" == Neil Schemenauer <nas@python.ca> writes:

    NS> Writing an IMAP server is a non-trivial task.

That's what I've been told by everyone I've talked to who's actually
tried to write one.
    
    NS> Alternatively, perhaps there could be a separate protocol and
    NS> client that could be used to review additions to the training
    NS> set.  Each day a few random spam and ham messages could be
    NS> grabbed as candidates.  Someone would periodically startup the
    NS> client, review the candidates, reclassify or remove any
    NS> messages they don't like and add them to the training set.

I think people will be much more motivated to report spam than ham.  I
like the general approach that copies of random messages will be
sequestered for some period of time before they're assumed to be ham.
Matched with a simple spam reporting scheme, this could keep the
training up to date with little effort.  I've sketched out an approach
a listserver like Mailman could do along these lines and if I get some
free time I'll hack something together.

I like the idea of a POP proxy which is classifying messages as
they're pulled from the server.  The easiest way for such a beast to
be notified of spam might be to simply save the spam in a special
folder or file that the POP proxy would periodically consult.

-Barry

From gward@python.net  Fri Sep  6 17:25:05 2002
From: gward@python.net (Greg Ward)
Date: Fri, 6 Sep 2002 12:25:05 -0400
Subject: [Spambayes] Deployment
In-Reply-To: <200209061431.g86EVM114413@pcp02138704pcs.reston01.va.comcast.net>
References: <200209061431.g86EVM114413@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <20020906162505.GB17800@cthulhu.gerg.ca>

On 06 September 2002, Guido van Rossum said:
> Quite independently from testing and tuning the algorithm, I'd like to
> think about deployment.

I was just pondering this this morning.

In case it wasn't obvious, I'm a strong proponent of filtering junk mail
as early as possible, ie. right after the SMTP DATA command has been
completed.  Filtering spam at the MUA just seems stupid to me -- by the
time it gets to me MUA, the spammer has already stolen my bandwidth.  My
public addresses are gward@python.net and gward@mems-exchange.org, so I
want spam stopped by the mail servers for those two domains.  (Hence the
recent MTA switch on starship...)

I guess MUA-level filtering is just a fallback for people who don't have
1) a burning, all-consuming hatred of junk mail, 2) root access to all
mail servers they rely on, and 3) the ability and inclination to install
an MTA with every bell and whistle tweaked to keep out junk mail.

Anyways, here's how I think it should work:

  * as soon as the DATA command is completed, the MTA passes the
    message to some local message-scanning code: a milter with Sendmail,
    local_scan() with Exim.  Dunno if any other MTAs have similar
    provisions.

  * the local scanner feeds the message to spambayes; if it says
    "yep, this is spam", the local scanner generates an SMTP rejection
    message, which the MTA returns to the client, eg.

      DATA
      [...spam...]
      .
      550-rejected -- looks like spam
      550 (see http://mail.python.org/spam/17nLfU-0003IT-00)

    The hypothetical web page (one per rejected message) would give an
    explanation of why the message was considered spam (eg. the top 15
    keywords), and give the sender the option to "request review" --
    what I'm thinking is send email to postmaster, and one of the
    postmasters will pop over to another web page, look at the message,
    and either rescue it or decide that it really is spam.

    Yes, I'm willing to risk giving spammers information in order to
    make life easier for false positive victims.  I very much doubt that
    spammers read SMTP rejection messages.

As for feeding the message to spambayes: for the Exim servers that I
have a hand in, the local_scan() function is written in Python, so there
shouldn't be any need to spawn a sub-process or open a socket to do
this.  Other sites may not be so lucky, in which case a fast,
low-overhead way to evaluate a message is essential.  Python's startup
overhead is not trivial, but I'd bet Python+spambayes is much faster to
startup than Perl+SpamAssassin.  Python has bytecode compilation, and
the spambayes database is much simpler than SpamAssassin's ruleset.
(Especially if the pickle is changed to a DB, DBM, or CDB file.)  So a
spamd-style daemon is worth considering, but not necessarily the answer.

Anyways, I've outlined a way to gather false positives above.  We
already have a protocol for dealing with false negs -- forward them to
spam@python.org.  Just have to figure out what to do with them then.
(Currently they're piling up in /var/mail/nc-spam [nc = not caught] on
mail.python.org.)

> Eventually, individuals and postmasters should be able to download a
> spambayes software distribution, answer a few configuration questions
> about their mail setup, training and false positives, and install it
> as a filter.

Note that SpamAssassin is not as simple as that to install -- I think
the "few configuration questions about their mail setup" is a massive
blackhole that's best avoided.  SA provides tools that tell you whether
something looks like spam, and how spammy it is.  Everything else is up
to the local admin, which makes eminent sense to me.  The mantra is:
SpamAssassin is a tool for *detecting* spam, not for rejecting
it/discarding it/moving it somewhere/whatever.  Do one thing, and do it
well.

The downside of that approach is that every MTA/MDA(/MUA?) community has
to figure out clever ways to integrate SpamAssassin.  This might not be
a bad thing: the ways to integrate SA with Exim just keep getting better
and better, and the SA people don't really have to worry about that.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
A committee is a life form with six or more legs and no brain.

From paul-bayes@svensson.org  Fri Sep  6 17:27:57 2002
From: paul-bayes@svensson.org (Paul Svensson)
Date: Fri, 6 Sep 2002 12:27:57 -0400 (EDT)
Subject: [Spambayes] Corpus Collection (Was: Re: Deployment)
In-Reply-To: <200209061431.g86EVM114413@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <Pine.LNX.4.44.0209061150430.6840-100000@familjen.svensson.org>

On Fri, 6 Sep 2002, Guido van Rossum wrote:

>Quite independently from testing and tuning the algorithm, I'd like to
>think about deployment.
>
>Eventually, individuals and postmasters should be able to download a
>spambayes software distribution, answer a few configuration questions
>about their mail setup, training and false positives, and install it
>as a filter.
>
>A more modest initial goal might be the production of a tool that can
>easily be used by individuals (since we're more likely to find
>individuals willing to risk this than postmasters).

My impression is that a pre-collected corpus would not fit most individuals
very well, but each individual (or group?) should collect their own corpus.

One problem that comes upp immediately: individuals are lazy.

If I currently get 50 spam and 50 ham a day, and I'll have to
press the 'delete' button once for each spam, I'll be happy
to press the 'spam' button instead.  However, if in addition
have to press a 'ham' button for each ham, it starts to look
much less like a win to me.  Add the time to install and setup
the whole machinery, and I'll just keep hitting delete.

The suggestions so far have been to hook something on the delete
action, that adds a message to the ham corpus.  I see two problems
with this: the ham will be a bit skewed; mail that I keep around
without deleting will not be counted.  Secondly, if I by force of
habit happen to press the 'delete' key instead of the 'spam' key,
I'll end up with spam in the ham, anyways.

I would like to look for a way to deal with spam in the ham.

The obvious thing to do is to trigger on the 'spam' button,
and at that time look for messages similar to the deleted one
in the ham corpus, and simply remove them.  To do this we
need a way to compare two word count histograms, to see
how similar they are.  Any ideas ?

Also, I personally would prefer to not see the spam at all.
If they get bounced (preferably already in the SMTP),
false positives become the senders problem, to rewrite
to remove the spam smell.

In a well tuned system then, there spam corpus will be much
smaller than the ham corpus, so it would be possible to be
slightly over-agressive when clearing potential spam from
the ham corpus.  This should make it easier to keep it clean.

Having a good way to remove spam from the ham corpus,
there's less need to worry about it getting there by mistake,
and we might as well simply add all messages to the ham corpus,
that didn't get deleted by the spam filtering.

It might also be useful to have a way to remove messages from
the spam corpus, in case of user ooops.

	/Paul


From barry@wooz.org  Fri Sep  6 17:16:17 2002
From: barry@wooz.org (Barry A. Warsaw)
Date: Fri, 6 Sep 2002 12:16:17 -0400
Subject: [Spambayes] test sets?
References: <200209060759.g867xcV03853@localhost.localdomain>
	<LNBBLJKPBEHFEDALKOLCAEHLBCAB.tim.one@comcast.net>
Message-ID: <15736.54481.733005.644033@anthem.wooz.org>


>>>>> "TP" == Tim Peters <tim.one@comcast.net> writes:

    TP> Barry, can you please identify for me which of these headers
    TP> are Mailman artifacts so I can avoid counting them?

Sure, with a little off-topic commentary added for no charge.

 0.01    19  3559 'header:X-Mailman-Version:1'
 0.01    19  3559 'header:List-Id:1'
 0.01    19  3557 'header:X-BeenThere:1'

These three are definitely MM artifacts, although the second one
/could/ be inserted by other list management software (it's described
in an RFC).

 0.01     0  3093 'header:Newsgroups:1'
 0.01     0  3054 'header:Xref:1'
 0.01     0  3053 'header:Path:1'

These aren't MM artifacts, but are byproducts of gating a message off
of an nntp feed.  Some of the other NNTP-* headers are similar, but I
won't point them out below.

 0.01    19  2668 'header:List-Unsubscribe:1'
 0.01    19  2668 'header:List-Subscribe:1'
 0.01    19  2668 'header:List-Post:1'
 0.01    19  2668 'header:List-Help:1'
 0.01    19  2668 'header:List-Archive:1'

RFC recommended generic listserve headers that MM injects.

 0.99   689     0 'header:Delivered-To:4'

This one's often a byproduct of the mail server.  In particularly,
Postfix and possibly others put the envelope recipient in this header.

 0.99   522     0 'header:Delivered-To:3'

So why do you get two entries for this one?

 0.99   519     0 'header:Received:8'
 0.99   466     1 'header:Received:7'

And this one?

 0.99   273     0 'header:MiME-Version:1'

Note that header names are case insensitive, so this one's no
different than "MIME-Version:".  Similarly other headers in your list.

 0.99    27     0 'header:1:1'

Huh?

 0.01     0    27 'header:X-Originally-To:1'

Mailman copies any To: header found in a message gated off of nntp to
the X-Originally-To: header.  Others possible here in clude
X-Original-To, X-Original-Cc, X-Original-Content-Transfer-Encoding,
and X-Original-Date.

 0.01     0     9 'header:X-No-Archive:1'

Could be MM or not.  This is used to stop the archiving of certain
messages, and MM will inject these into digests and password
reminders, but it's also possible that user agents have added this.

(Aside, in particularly mischievous fashion the value of this header
is "yes", so you see things like "X-No-Archive: yes".
X-Isn't-Not-Nonsense: no).

 0.02    65  3559 'header:Precedence:1'

Could be Mailman, or not.  This header is supposed to tell other
automated software that this message was automated.  E.g. a replybot
should ignore any message with a Precedence: {bulk|junk|list}.

 0.80  1471   273 'header:Delivered-To:1'

Why again?!

 0.50     4     0 'header:2:1'

!?

 0.50     3     0 'header:6:1'
 0.50     3     0 'header:5:1'
 0.50     3     0 'header:4:1'
 0.50     3     0 'header:3:1'
 0.50     2     0 'header:X-:1'

Freaky.

 0.50     0     2 'header:X-BeenThere:3'

X-BeenThere: before :)

 0.50     0     2 'header:'

Heh?

 0.50     0     1 'header:X-Silly:1'

X-Very-Silly: fneh
X-Very-Silly-Indeed: dead parrot

 0.50     0     1 'header:X-Get-A-Real-Newsreader:1'
 0.50     0     1 'header:X-Favorite-Dwarf:1'
 0.50     0     1 'header:X-Eric-Conspiracy:1'
 0.50     0     1 'header:Favorite-Color:1'

Cute. :)

Some headers of course are totally unreliable as to their origin.  I'm
thinking stuff like MIME-Version, Content-Type, To, From, etc, etc.
Everyone sticks those in.

-Barry

From guido@python.org  Fri Sep  6 17:27:01 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 06 Sep 2002 12:27:01 -0400
Subject: [Spambayes] Deployment
In-Reply-To: Your message of "Fri, 06 Sep 2002 12:25:05 EDT."
             <20020906162505.GB17800@cthulhu.gerg.ca> 
References: <200209061431.g86EVM114413@pcp02138704pcs.reston01.va.comcast.net>
	<20020906162505.GB17800@cthulhu.gerg.ca> 
Message-ID: <200209061627.g86GR1p15407@pcp02138704pcs.reston01.va.comcast.net>

> I guess MUA-level filtering is just a fallback for people who don't have
> 1) a burning, all-consuming hatred of junk mail, 2) root access to all
> mail servers they rely on, and 3) the ability and inclination to install
> an MTA with every bell and whistle tweaked to keep out junk mail.

Sure.  But for most people, changing their company's or ISP's server
requires years of lobbying, while they have total and immediate
control over their own MUA.

That said, I agree that we should offer a good solution to
postmasters, and I trust that your ideas are right on the mark!

--Guido van Rossum (home page: http://www.python.org/~guido/)

From jeremy@alum.mit.edu  Fri Sep  6 17:28:09 2002
From: jeremy@alum.mit.edu (Jeremy Hylton)
Date: Fri, 6 Sep 2002 12:28:09 -0400
Subject: [Spambayes] Deployment
In-Reply-To: <200209061431.g86EVM114413@pcp02138704pcs.reston01.va.comcast.net>
References: <200209061431.g86EVM114413@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <15736.55193.38098.486459@slothrop.zope.com>

I think one step towards deployment is creating a re-usable tokenizer
for mail messages.  The current codebase doesn't expose an easy-to-use
or easy-to-customize tokenizer.

The timtest module seems to contain an enormous body of practical
knowledge about how to parse mail messages, but the module wasn't
designed for re-use.  I'd like to see a module that can take a single
message or a collection of messages and tokenize each one.

I'd like to see the tokenize by customizable, too.  Tim had to exclude
some headers from his test data, because there were particular biases
in the test data.  If other people have test data without those
biases, they ought to be able to customize the tokenizer to include
them or exclude others.

Jeremy


From tim.one@comcast.net  Fri Sep  6 17:45:09 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 06 Sep 2002 12:45:09 -0400
Subject: [Spambayes] test sets?
In-Reply-To: <15736.54481.733005.644033@anthem.wooz.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEICBCAB.tim.one@comcast.net>

[Barry A. Warsaw, gives answers and asks questions]

Here's the code that produced the header tokens:

    x2n = {}
    for x in msg.keys():
        x2n[x] = x2n.get(x, 0) + 1
    for x in x2n.items():
        yield "header:%s:%d" % x


Some responses:

>  0.01    19  3559 'header:X-Mailman-Version:1'
>  0.01    19  3559 'header:List-Id:1'
>  0.01    19  3557 'header:X-BeenThere:1'
>
> These three are definitely MM artifacts, although the second one
> /could/ be inserted by other list management software (it's described
> in an RFC).

Since all the ham came from Mailman, and only 19 spam had it, it's quite
safe to assume then that I should ignore these for now.

>  0.01     0  3093 'header:Newsgroups:1'
>  0.01     0  3054 'header:Xref:1'
>  0.01     0  3053 'header:Path:1'
>
> These aren't MM artifacts, but are byproducts of gating a message off
> of an nntp feed.  Some of the other NNTP-* headers are similar, but I
> won't point them out below.

I should ignore these too then.

>  0.01    19  2668 'header:List-Unsubscribe:1'
>  0.01    19  2668 'header:List-Subscribe:1'
>  0.01    19  2668 'header:List-Post:1'
>  0.01    19  2668 'header:List-Help:1'
>  0.01    19  2668 'header:List-Archive:1'
>
> RFC recommended generic listserve headers that MM injects.

Ditto.

> So why do you get two entries for this one?
>
>  0.99   519     0 'header:Received:8'
>  0.99   466     1 'header:Received:7'

Read the code <wink>.  The first line counts msgs that had 8 instances of a
'Received' header, and the second counts msgs that had 7 instances.  I
expect this is a good clue!  The more indirect the mail path, the more of
those thingies we'll see, and if you're posting from a spam trailer park in
Tasmania you may well need to travel thru more machines.

> ...
> Note that header names are case insensitive, so this one's no
> different than "MIME-Version:".  Similarly other headers in your list.

Ignoring case here may or may not help; that's for experiment to decide.
It's plausible that case is significant, if, e.g., a particular spam mailing
package generates unusual case, or a particular clueless spammer
misconfigures his package.

>  0.02    65  3559 'header:Precedence:1'
>
> Could be Mailman, or not.  This header is supposed to tell other
> automated software that this message was automated.  E.g. a replybot
> should ignore any message with a Precedence: {bulk|junk|list}.

Rule of thumb:  if Mailman inserts a thing, I should ignore it.  Or, better,
I should stop trying to out-think the flaws in the test data and get better
test data instead!

>  0.50     4     0 'header:2:1'
>
> !?
> ...
>  0.50     0     2 'header:'
>
> Heh?

I sucked out all the wordinfo keys that began with "header:".  The last line
there  was probably due to unrelated instances of the string "header:" in
message bodies.  Harder to guess about the first line.

> ...
> Some headers of course are totally unreliable as to their origin.  I'm
> thinking stuff like MIME-Version, Content-Type, To, From, etc, etc.
> Everyone sticks those in.

The brilliance of Anthony's "just count them" scheme is that it requires no
thought, so can't be fooled <wink>.  Header lines that are evenly
distributed across spam and ham will turn out to be worthless indicators
(prob near 0.5), so do no harm.


From tim.one@comcast.net  Fri Sep  6 17:55:07 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 06 Sep 2002 12:55:07 -0400
Subject: [Spambayes] test sets?
In-Reply-To: <200209060811.g868Bo904031@localhost.localdomain>
Message-ID: <LNBBLJKPBEHFEDALKOLCEEIEBCAB.tim.one@comcast.net>

[Anthony Baxter]
> The other thing on my todo list (probably tonight's tram ride home) is
> to add all headers from non-text parts of multipart messages. If nothing
> else, it'll pick up most virus email real quick.

See the checkin comments for timtest.py last night.  Adding this code gave a
major reduction in the false negative rate:

def crack_content_xyz(msg):
    x = msg.get_type()
    if x is not None:
        yield 'content-type:' + x.lower()

    x = msg.get_param('type')
    if x is not None:
        yield 'content-type/type:' + x.lower()

    for x in msg.get_charsets(None):
        if x is not None:
            yield 'charset:' + x.lower()

    x = msg.get('content-disposition')
    if x is not None:
        yield 'content-disposition:' + x.lower()

    fname = msg.get_filename()
    if fname is not None:
        for x in fname.lower().split('/'):
            for y in x.split('.'):
                yield 'filename:' + y

    x = msg.get('content-transfer-encoding:')
    if x is not None:
        yield 'content-transfer-encoding:' + x.lower()


...

    t = ''
    for x in msg.walk():
        for w in crack_content_xyz(x):
            yield t + w
        t = '>'

I *suspect* most of that stuff didn't make any difference, but I put it all
in as one blob so don't know which parts did and didn't help.


From barry@python.org  Fri Sep  6 17:59:49 2002
From: barry@python.org (Barry A. Warsaw)
Date: Fri, 6 Sep 2002 12:59:49 -0400
Subject: [Spambayes] test sets?
References: <LNBBLJKPBEHFEDALKOLCAEHLBCAB.tim.one@comcast.net>
	<LNBBLJKPBEHFEDALKOLCKEHPBCAB.tim.one@comcast.net>
Message-ID: <15736.57093.811682.371784@anthem.wooz.org>


    TP> A false positive *really* has to work hard then, eh?  The long
    TP> quote of a Nigerian scam letter is one of the two that made
    TP> it, and spamprob() looked at all this stuff before deciding it
    TP> was spam:

Here's an interesting thing to test: discriminate words differently if
they are on a line that starts with `>' or, to catch styles like
above, that the first occurance on a line of < or > is > (to eliminate
html).

Then again, it may not be worth trying to un-false-positive that
Nigerian scam quote.

-Barry

From neale@woozle.org  Fri Sep  6 18:13:17 2002
From: neale@woozle.org (Neale Pickett)
Date: 06 Sep 2002 10:13:17 -0700
Subject: [Spambayes] Deployment
In-Reply-To: <200209061506.g86F6Qo14777@pcp02138704pcs.reston01.va.comcast.net>
References: <200209061431.g86EVM114413@pcp02138704pcs.reston01.va.comcast.net>
	<15736.50015.881231.510395@12-248-11-90.client.attbi.com>
	<200209061506.g86F6Qo14777@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <w53lm6fca8i.fsf@woozle.org>

So then, Guido van Rossum <guido@python.org> is all like:

> > Basic procmail usage goes something like this:
> > 
> >     :0fw
> >     | spamassassin -P
> > 
> >     :0
> >     * ^X-Spam-Status: Yes
> >     $SPAM
> > 
> 
> Do you feel capable of writing such a tool?  It doesn't look too hard.

Not to beat a dead horse, but that's exactly what my spamcan package
did.  For those just tuning in, spamcan is a thingy I wrote before I
knew about Tim & co's work on this crazy stuff; you can download it from
<http://woozle.org/~neale/src/spamcan/spamcan.html>, but I'm not going
to work on it anymore.

I'm currently writing a new one based on classifier (and timtest's
booty-kicking tokenizer).  I'll probably have something soon, like maybe
half an hour, and no, it's not too hard.  The hard part is storing the
data somewhere.  I don't want to use ZODB, as I'd like something a
person can just drop in with a default Python install.  So anydbm is
looking like my best option.

I already have a setup like this using Xavier Leroy's SpamOracle, which
does the same sort of thing.  You call it from procmail, it adds a new
header, and then you can filter on that header.  Really easy.

Here's how I envision this working.  Everybody gets four new mailboxes:

  train-eggs
  train-spam
  trained-eggs
  trained-spam

You copy all your spam and eggs* into the "train-" boxes as you get it.
How frequently you do this would be up to you, but you'd get better
results if you did it more often, and you'd be wise to always copy over
anything which was misclassified.  Then, every night, the spam fairy
swoops down and reads through your folders, learning about what sorts of
things you think are eggs and what sorts of things are spam.  After she's
done, she moves your mail into the "trained-" folders.

This would work for anybody using IMAP on a Unix box, or folks who read
their mail right off the server.  I've spoken with some fellows at work
about Exchange and they seem to beleive that Exchange exports
appropriate functionality to implement a spam fairy as well.

Advanced users could stay ahead of the game by reprogramming their mail
client to bind the key "S" to "move to train-spam" and "H" to "move to
train-eggs".  Eventually, if enough people used this sort of thing, it'd
start showing up in mail clients.  That's the "delete as spam" button
Paul Graham was talking about.

* The Hormel company might not think well of using the word "ham" as the
  opposite of "spam", and they've been amazingly cool about the use of
  their product name for things thus far.  So I propose we start calling
  non-spam something more innocuous (and more Monty Pythonic) such as
  "eggs".

Neale

From tim.one@comcast.net  Fri Sep  6 18:35:55 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 06 Sep 2002 13:35:55 -0400
Subject: [Spambayes] Deployment
In-Reply-To: <15736.55193.38098.486459@slothrop.zope.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEIHBCAB.tim.one@comcast.net>

[Jeremy Hylton]
> I think one step towards deployment is creating a re-usable tokenizer
> for mail messages.  The current codebase doesn't expose an easy-to-use
> or easy-to-customize tokenizer.

tokenize() couldn't be easier to use:  it takes a string argument, and
produces a stream of tokens (whether via explicit list, or generator, or
tuple, or ... doesn't matter).  All the tokenize() functions in GBayes.py
and timtest.py are freely interchangeable this way.

Note that we have no evidence to support that a customizable tokenizer would
do any good, or, if it would, in which ways customization could be helpful.
That's a research issue on which no work has been done.

> The timtest module seems to contain an enormous body of practical
> knowledge about how to parse mail messages, but the module wasn't
> designed for re-use.

That's partly a failure of imagination <wink>.  Splitting out all knowledge
of tokenization is just a large block cut-and-paste ... there, it's done.
Change the

    from timtoken import tokenize

at the top to use any other tokenizer now.  If you want to make it easier
still, feel free to check in something better.

> I'd like to see a module that can take a single message or a collection of
> messages and tokenize each one.

The Msg and MsgStream classes in timtest.py are a start at that, but it's
hard to do anything truly *useful* here when people use all sorts of
different physical representations for email msgs (mboxes in various
formats, one file per "folder", one file per msg, Skip's gzipped gimmick,
...).  If you're a Python coder <wink>, you *should* find it very easy to
change the guts of Msg and MsgStream to handle your peculiar scheme.
Defining interfaces for these guys should be done.

> I'd like to see the tokenize by customizable, too.  Tim had to exclude
> some headers from his test data, because there were particular biases
> in the test data.  If other people have test data without those
> biases, they ought to be able to customize the tokenizer to include
> them or exclude others.

This sounds like a bottomless pit to me, and there's no easier way to
customize than to edit the code.  As README.txt still says, though, massive
refactoring would help.  Hop to it!


From whisper@oz.net  Fri Sep  6 18:53:24 2002
From: whisper@oz.net (David LeBlanc)
Date: Fri, 6 Sep 2002 10:53:24 -0700
Subject: [Spambayes] Deployment
In-Reply-To: <w53lm6fca8i.fsf@woozle.org>
Message-ID: <GCEDKONBLEFPPADDJCOECEGMENAA.whisper@oz.net>

I think that when considering deployment, a solution that supports all
Python platforms and not just the L|Unix crowd is desirable.

Mac and PC users are more apt to be using a commercial MUA that's unlikely
to offer hooking ability (at least not easily). As mentioned elsewhere, even
L|Unix users may find an MUA solution easier to use then getting it added to
their MTA. (SysOps make programmers look like flaming liberals ;).)

My notion of a solution for Windows/Outlook has been, as Guido described, a
client-server. Client side does pop3/imap/mapi fetching (of which, I'm only
going to implement pop3 initially) potentially on several hosts, spamhams
the incoming mail and puts it into one file per message (qmail style?). The
MUA accesses this "eThunk" as a server to obtain all the ham. Spam is
retained in the eThunk and a simple viewer would be used for manual
oversight on the spam for ultimate rejection (and training of spam filter)
and the ham will go forward (after being used for training) on the next MUA
fetch. eThunk would sit on a timer for 'always online' users, but I am not
clear on how to support dialup users with this scheme.

Outbound mail would use a direct path from the MUA to the MTA. Hopefully all
MUAs can split the host fetch/send URL's

IMO, end users are likely to be more intested in n-way classification. If
this is available, the "simple viewer" could be enhanced to support viewing
via folders and (at least for me) the Outlook nightmare is over - I would
use this as my only MUA. (N.B. according to my recent readings, the best
n-way classifier uses something called a "Support Vector Machine" (SVM)
which is 5-8% more accurate then Naive Bayes (NB) ).

I wonder if the focus of spambayes ought not to be a classifier that leaves
the fetching and feeding of messages to auxillary code? That way, it could
be dropped into whatever harness that suited the user's situation.

David LeBlanc
Seattle, WA USA



From guido@python.org  Fri Sep  6 18:58:14 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 06 Sep 2002 13:58:14 -0400
Subject: [Spambayes] Deployment
In-Reply-To: Your message of "Fri, 06 Sep 2002 10:53:24 PDT."
             <GCEDKONBLEFPPADDJCOECEGMENAA.whisper@oz.net> 
References: <GCEDKONBLEFPPADDJCOECEGMENAA.whisper@oz.net> 
Message-ID: <200209061758.g86HwET15939@pcp02138704pcs.reston01.va.comcast.net>

> I wonder if the focus of spambayes ought not to be a classifier that
> leaves the fetching and feeding of messages to auxillary code? That
> way, it could be dropped into whatever harness that suited the
> user's situation.

I see no reason to restrict the project to developing the classifier
and leave the deployment to others.  Attempts at deployment in the
real world will surely provide additional feedback for the classifier.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From gward@python.net  Fri Sep  6 19:02:23 2002
From: gward@python.net (Greg Ward)
Date: Fri, 6 Sep 2002 14:02:23 -0400
Subject: [Spambayes] test sets?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEICBCAB.tim.one@comcast.net>
References: <15736.54481.733005.644033@anthem.wooz.org>
	<LNBBLJKPBEHFEDALKOLCKEICBCAB.tim.one@comcast.net>
Message-ID: <20020906180223.GA18250@cthulhu.gerg.ca>

On 06 September 2002, Tim Peters said:
> > Note that header names are case insensitive, so this one's no
> > different than "MIME-Version:".  Similarly other headers in your list.
> 
> Ignoring case here may or may not help; that's for experiment to decide.
> It's plausible that case is significant, if, e.g., a particular spam mailing
> package generates unusual case, or a particular clueless spammer
> misconfigures his package.

Case of headers is definitely helpful.  SpamAssassin has a rule for it
-- if you have headers like "DATE" or "SUBJECT", you get a few more
points.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
God is omnipotent, omniscient, and omnibenevolent
---it says so right here on the label.

From skip@pobox.com  Fri Sep  6 19:48:35 2002
From: skip@pobox.com (Skip Montanaro)
Date: Fri, 6 Sep 2002 13:48:35 -0500
Subject: [Spambayes] Deployment
In-Reply-To: <20020906162505.GB17800@cthulhu.gerg.ca>
References: <200209061431.g86EVM114413@pcp02138704pcs.reston01.va.comcast.net>
        <20020906162505.GB17800@cthulhu.gerg.ca>
Message-ID: <15736.63619.488739.691181@12-248-11-90.client.attbi.com>


    Greg> In case it wasn't obvious, I'm a strong proponent of filtering
    Greg> junk mail as early as possible, ie. right after the SMTP DATA
    Greg> command has been completed.  Filtering spam at the MUA just seems
    Greg> stupid to me -- by the time it gets to me MUA, the spammer has
    Greg> already stolen my bandwidth.  

The two problems I see with filtering that early are:

    1. Everyone receiving email via that server will contribute ham to the
       stew, making the Bayesian classification less effective.

    2. Given that there will be some false positives, you absolutely have to
       put the mail somewhere.  You can't simply delete it.  (I also don't
       like the TMDA-ish business of replying with a msg that says, "here's
       what you do to really get your message to me."  That puts an extra
       burden on my correspondents.)  As an individual, I would prefer you
       put spammish messages somewhere where I can review them, not an
       anonymous sysadmin who I might not trust with my personal email
       (nothing against you Greg ;-).

I personally prefer to manage this stuff at the user agent level.  Bandwidth
is a heck of a lot cheaper than my time.

Skip

From harri.pasanen@bigfoot.com  Fri Sep  6 20:07:28 2002
From: harri.pasanen@bigfoot.com (Harri Pasanen)
Date: Fri, 6 Sep 2002 21:07:28 +0200
Subject: [Spambayes] Deployment
In-Reply-To: <15736.63619.488739.691181@12-248-11-90.client.attbi.com>
References: <200209061431.g86EVM114413@pcp02138704pcs.reston01.va.comcast.net>
	<20020906162505.GB17800@cthulhu.gerg.ca>
	<15736.63619.488739.691181@12-248-11-90.client.attbi.com>
Message-ID: <200209062107.28106.harri.pasanen@bigfoot.com>

On Friday 06 September 2002 20:48, Skip Montanaro wrote:
>     Greg> In case it wasn't obvious, I'm a strong proponent of
> filtering Greg> junk mail as early as possible, ie. right after the
> SMTP DATA Greg> command has been completed.  Filtering spam at the
> MUA just seems Greg> stupid to me -- by the time it gets to me MUA,
> the spammer has Greg> already stolen my bandwidth.
>
> The two problems I see with filtering that early are:
>
>     1. Everyone receiving email via that server will contribute ham
> to the stew, making the Bayesian classification less effective.
>
>     2. Given that there will be some false positives, you absolutely
> have to put the mail somewhere.  You can't simply delete it.  (I also
> don't like the TMDA-ish business of replying with a msg that says,
> "here's what you do to really get your message to me."  That puts an
> extra burden on my correspondents.)  As an individual, I would prefer
> you put spammish messages somewhere where I can review them, not an
> anonymous sysadmin who I might not trust with my personal email
> (nothing against you Greg ;-).
>
> I personally prefer to manage this stuff at the user agent level. 
> Bandwidth is a heck of a lot cheaper than my time.
>

I see no reason why both approaches could and should not be used.
MTA level filtering would just need to use a different corpus, one that 
would contain illegal or otherwise commonly unapproved material for the 
group of people using that MTA.   I'm sure that such an approach would 
significantly reduce the mail traffic as a first step, without  giving  
false positives.

MUA corpus would then be personally trained -- although I'd like the 
option of 'down-loadable' corpuses and merge functionality.

Harri

PS.  Just joined the list, so pardon if my thoughts have been hashed 
through before.



From tim.one@comcast.net  Fri Sep  6 20:24:15 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 06 Sep 2002 15:24:15 -0400
Subject: [Spambayes] Deployment
In-Reply-To: <200209061431.g86EVM114413@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEJABCAB.tim.one@comcast.net>

[Guido]
> ...
> - A program that acts both as a pop client and a pop server.  You
>   configure it by telling it about your real pop servers.  You then
>   point your mail reader to the pop server at localhost.  When it
>   receives a connection, it connects to the remote pop servers, reads
>   your mail, and gives you only the non-spam.

FYI, I'll never trust such a scheme:  I have no tolerance for false
positives, and indeed do nothing to try to block spam on any of my email
accounts now for that reason.  Deliver all suspected spam to a Spam folder
instead and I'd love it.


From guido@python.org  Fri Sep  6 20:24:38 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 06 Sep 2002 15:24:38 -0400
Subject: [Spambayes] Deployment
In-Reply-To: Your message of "Fri, 06 Sep 2002 15:24:15 EDT."
             <LNBBLJKPBEHFEDALKOLCIEJABCAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCIEJABCAB.tim.one@comcast.net> 
Message-ID: <200209061924.g86JOc516514@pcp02138704pcs.reston01.va.comcast.net>

> > - A program that acts both as a pop client and a pop server.  You
> >   configure it by telling it about your real pop servers.  You then
> >   point your mail reader to the pop server at localhost.  When it
> >   receives a connection, it connects to the remote pop servers, reads
> >   your mail, and gives you only the non-spam.
> 
> FYI, I'll never trust such a scheme:  I have no tolerance for false
> positives, and indeed do nothing to try to block spam on any of my email
> accounts now for that reason.  Deliver all suspected spam to a Spam folder
> instead and I'd love it.

Another config parameter.

The filter could add a header file.  Or a ~ to the subject if you like
that style. :-)

--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.one@comcast.net  Fri Sep  6 20:21:22 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 06 Sep 2002 15:21:22 -0400
Subject: [Spambayes] Deployment
In-Reply-To: <200209061431.g86EVM114413@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEIPBCAB.tim.one@comcast.net>

[Guido]
>   ...
>   I don't know how big that pickle would be, maybe loading it each time
>   is fine.  Or maybe marshalling.)

My tests train on about 7,000 msgs, and a binary pickle of the database is
approaching 10 million bytes.  I haven't done anything to try to reduce its
size, and know of some specific problem areas (for example, doing character
5-grams of "long words" containing high-bit characters generates a lot of
database entries, and I suspect they're approximately worthless).  OTOH,
adding in more headers will increase the size.  So let's call it 10 meg
<wink>.


From tim.one@comcast.net  Fri Sep  6 20:43:56 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 06 Sep 2002 15:43:56 -0400
Subject: [Spambayes] Deployment
In-Reply-To: <200209061431.g86EVM114413@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCEEJCBCAB.tim.one@comcast.net>

[Guido]
> Takers?  How is ESR's bogofilter packaged?  SpamAssassin?  The Perl
> Bayes filter advertised on slashdot?

WRT the last, it's a small pile of Windows .exe files along with
cygwin1.dll.  The .exes are cmdline programs.  One is a POP3 proxy.  If I
currently have an email server named, say, mail.comcast.net, with user name
timmy, then I change my email reader to say that my server is 127.0.0.1, and
that my user name on that server is mail.comcast.net:timmy.  In that way the
proxy picks up both the real server and user names from what the mail reader
tells it the user name is.

This is an N-way classifier (like ifile that way), and "all it does" is
insert a

    X-Text-Classification:  one_of_the_class_names_you_picked

header into your email before passing it on to your mail reader.  The user
then presumably fiddles their mail reader to look for such headers and "do
something about it" (and even Outlook can handle *that* much <wink>).

The user is responsible for generating text files with appropriate examples
of each class of message, and for running the cmdline tools to train the
classifier.


From whisper@oz.net  Fri Sep  6 20:53:36 2002
From: whisper@oz.net (David LeBlanc)
Date: Fri, 6 Sep 2002 12:53:36 -0700
Subject: [Spambayes] Deployment
In-Reply-To: <LNBBLJKPBEHFEDALKOLCIEJABCAB.tim.one@comcast.net>
Message-ID: <GCEDKONBLEFPPADDJCOECEHJENAA.whisper@oz.net>

You missed the part that said that spam is kept in the "eThunk" and was
viewable by a simple viewer for final disposition?

Of course, with Outbloat, you could fire up PythonWin and stuff the spam
into the Junk Email folder... but then you loose the ability to retrain on
the user classified ham/spam.

David LeBlanc
Seattle, WA USA

> -----Original Message-----
> From: spambayes-bounces+whisper=oz.net@python.org
> [mailto:spambayes-bounces+whisper=oz.net@python.org]On Behalf Of Tim
> Peters
> Sent: Friday, September 06, 2002 12:24
> To: spambayes@python.org
> Subject: RE: [Spambayes] Deployment
>
>
> [Guido]
> > ...
> > - A program that acts both as a pop client and a pop server.  You
> >   configure it by telling it about your real pop servers.  You then
> >   point your mail reader to the pop server at localhost.  When it
> >   receives a connection, it connects to the remote pop servers, reads
> >   your mail, and gives you only the non-spam.
>
> FYI, I'll never trust such a scheme:  I have no tolerance for false
> positives, and indeed do nothing to try to block spam on any of my email
> accounts now for that reason.  Deliver all suspected spam to a Spam folder
> instead and I'd love it.
>
>
> _______________________________________________
> Spambayes mailing list
> Spambayes@python.org
> http://mail.python.org/mailman-21/listinfo/spambayes


From neale@woozle.org  Fri Sep  6 20:58:33 2002
From: neale@woozle.org (Neale Pickett)
Date: 06 Sep 2002 12:58:33 -0700
Subject: [Spambayes] Deployment
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEIPBCAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCGEIPBCAB.tim.one@comcast.net>
Message-ID: <w53znuuc2l2.fsf@woozle.org>

So then, Tim Peters <tim.one@comcast.net> is all like:

> [Guido]
> >   ...
> >   I don't know how big that pickle would be, maybe loading it each time
> >   is fine.  Or maybe marshalling.)
> 
> My tests train on about 7,000 msgs, and a binary pickle of the database is
> approaching 10 million bytes.

My paltry 3000-message training set makes a 6.3MB (where 1MB=1e6 bytes)
pickle.  hammie.py, which I just checked in, will optionally let you
write stuff out to a dbm file.  With that same message base, the dbm
file weighs in at a hefty 21.4MB.  It also takes longer to write:

  Using a database:
   real    8m24.741s
   user    6m19.410s
   sys     1m33.650s

  Using a pickle:
   real    1m39.824s
   user    1m36.400s
   sys     0m2.160s

This is on a PIII at 551.257MHz (I don't know what it's *supposed* to
be, 551.257 is what /proc/cpuinfo says).

For comparison, SpamOracle (currently the gold standard in my mind, at
least for speed) on the same data blazes along:

   real    0m29.592s
   user    0m28.050s
   sys     0m1.180s

Its data file, which appears to be a marshalled hash, is 448KB.
However, it's compiled O'Caml and it uses a much simpler tokenizing
algorithm written with a lexical analyzer (ocamllex), so we'll never be
able to outperform it.  It's something to keep in mind, though.

I don't have statistics yet for scanning unknown messages.  (Actually, I
do, and the database blows the pickle out of the water, but it scores
every word with 0.00, so I'm not sure that's a fair test. ;)  In any
case, 21MB per user is probably too large, and 10MB is questionable.  

On the other hand, my pickle compressed very well with gzip, shrinking
down to 1.8MB.

Neale

From gward@python.net  Fri Sep  6 21:01:11 2002
From: gward@python.net (Greg Ward)
Date: Fri, 6 Sep 2002 16:01:11 -0400
Subject: [Spambayes] Deployment
In-Reply-To: <15736.63619.488739.691181@12-248-11-90.client.attbi.com>
References: <200209061431.g86EVM114413@pcp02138704pcs.reston01.va.comcast.net>
	<20020906162505.GB17800@cthulhu.gerg.ca>
	<15736.63619.488739.691181@12-248-11-90.client.attbi.com>
Message-ID: <20020906200111.GA18381@cthulhu.gerg.ca>

On 06 September 2002, Skip Montanaro said:
> The two problems I see with filtering that early are:
> 
>     1. Everyone receiving email via that server will contribute ham to the
>        stew, making the Bayesian classification less effective.

I agree -- the whole idea of automatically adding mail to the training
corpus makes me really nervous.  Especially when you just assume that
mail not deemed to be spam is really not spam.

>     2. Given that there will be some false positives, you absolutely have to
>        put the mail somewhere.  You can't simply delete it.

Absolutely!  With SpamAssassin, there's a large grey area where false
positives live (scores between 5 and 10, I'd say) -- hence the whole
tedious mechanism of setting that mail aside, mailing nightly summaries
to postmaster, and one of us logging in to rescue the false positives.
Rejecting messages that score above 10 is fairly safe -- if there is the
occasional false positive in there, then someone gets a bounce report
telling their mail looked like spam.  At least it's not discarded.

As Tim has reported elsewhere, there's a much smaller grey area with the
Bayesian classifier, and false positives are clustered up around 0.99
along with all the spam.  Hence the idea I floated, where all rejected
messages are saved, and the rejection message includes a URL where the
sender can request a second chance.  I think that would be much less
burden on everyone -- the sender and the postmasters.

(Also, we could make it more sophisticated then just mailing requests
for a second chance to postmaster -- eg. associate each recipient
address with a "responsible person" who has the authority to rescue FPs
for that address.  Not sure what to do about messages with multiple
recipients.)

>        (I also don't
>        like the TMDA-ish business of replying with a msg that says, "here's
>        what you do to really get your message to me."  That puts an extra
>        burden on my correspondents.)

Err, then I guess you don't like the above -- especially since it's not
a nice friendly TMDA-style message, but an MTA-generated bounce.  Hmmm.

> I personally prefer to manage this stuff at the user agent level.  Bandwidth
> is a heck of a lot cheaper than my time.

Good to hear another perspective.  I'm sure some people will come up
with a great MUA solution, while others concentrate on the server side.
We all have our preferences...

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Those of you who think you know everything really annoy those of us who do.

From tim.one@comcast.net  Fri Sep  6 21:15:18 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 06 Sep 2002 16:15:18 -0400
Subject: [Spambayes] GBayes spam filtering
In-Reply-To: <Pine.LNX.4.44.0209051346240.6840-100000@familjen.svensson.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEJFBCAB.tim.one@comcast.net>

[Paul Svensson]
> ...
> When a user reads a message and find that it's spam that got thru
> the filter, they need a way to send the message-id to the corpus, to
> flag it as spam.  At this point, it would be a good idea to compare the
> histogram of the new spam to each histogram in the ham corpus, and remove
> any that are similar (any good ideas how to do the comparison?),

Read the "memory-based approach" stuff in

    Learning to Filter Spam E-Mail:  A Comparison of a Naive
    Bayesian and a Memory-Based Approach

    http://arxiv.org/ftp/cs/papers/0009/0009009.pdf


> or maybe if they are VERY similar simply flag them as spam.  After
> recomputing the filter from the modified corpus, we could also re-filter
> the ham corpus, and remove more newfound spam that way.
>
> Characteristically of this system, the spam corpus will be
> reasonably clean (assuming the users don't abuse it too much), but
> the ham corpus will be quite dirty, containing spam that's not yet read,
> and spam that the recipient didn't bother to mark.  I'm curious how
> GBayes would handle this situation; I assume the false negative rate
> would go up, but how much ?

You can run an experiment and measure it.  That's almost as easy as, and
much more reliable than, guessing <wink>.


From jeremy@alum.mit.edu  Fri Sep  6 21:03:28 2002
From: jeremy@alum.mit.edu (Jeremy Hylton)
Date: Fri, 6 Sep 2002 16:03:28 -0400
Subject: [Spambayes] understanding high false negative rate
Message-ID: <15737.2576.315460.956295@slothrop.zope.com>

I've tried to do some testing with some personal collections of ham
and spam.  I'm seeing very high false negative rates. 20-30% is
typical.=A0 The false positive rate is 0-3%.  (Finally!  I had to scrub=

a bunch of previously unnoticed spam from my inbox.)  Both collections
have about 1100 messages.

I'd like to figure out why my false negative rate is so high, but I'm
not sure what details I should look at to diagnose.  I'm assuming that
mboxtest.py is basically correct, but it could have bugs.

One possibility is that my ham test set isn't nearly so useful as the
python-list, since it isn't focused on a single topic.  I've got some
python email, personal correspondence, questions about my Shakespeare
web site, and a few email newsletters I get on a regular basis.  I've
got receipts from various online order sites, mail from the company
that manages my student loans, etc.  Maybe the great variety in my
non-spam email makes it harder to find good discriminators for spam?

Here's a sample spam distribution from a test run:

Spam distribution for this pair:
* =3D 3 items
  0.00  73 *************************
  2.50   0=20
  5.00   2 *
  7.50   0=20
 10.00   0=20
 12.50   1 *
 15.00   0=20
 17.50   1 *
 20.00   1 *
 22.50   0=20
 25.00   2 *
 27.50   0=20
 30.00   0=20
 32.50   0=20
 35.00   0=20
 37.50   0=20
 40.00   0=20
 42.50   0=20
 45.00   0=20
 47.50   0=20
 50.00   0=20
 52.50   0=20
 55.00   0=20
 57.50   1 *
 60.00   0=20
 62.50   1 *
 65.00   0=20
 67.50   0=20
 70.00   1 *
 72.50   0=20
 75.00   0=20
 77.50   0=20
 80.00   2 *
 82.50   2 *
 85.00   2 *
 87.50   0=20
 90.00   4 **
 92.50   1 *
 95.00   5 **
 97.50 127 *******************************************

And here's a sample false negative.  (I'll quote the report so it
stands out.)  One thing I don't understand is how the spam probability
for the message is so low, when there are several high indicators and
few low indicators.

> Low prob spam! 1.64654685184e-11
> /home/jeremy/Mail/spam:242 subject: your web site has been mapped
> prob('millions') =3D 0.99
> prob('skip:=3D 40') =3D 0.99
> prob('"remove"') =3D 0.99
> prob('from:email addr:mail') =3D 0.99
> prob('email addr:alum') =3D 0.01
> prob('status') =3D 0.01
> prob('connected') =3D 0.01
> prob('returning') =3D 0.01
> prob('from:email addr:com>') =3D 0.224056
> prob('every') =3D 0.789741
> prob('charges') =3D 0.208406
> prob('free') =3D 0.818103
> prob('survey.') =3D 0.14931
> prob('officer') =3D 0.208406
> prob('its') =3D 0.155044
> prob('added') =3D 0.133131
> prob('current') =3D 0.152639
> prob('email addr:mit') =3D 0.01
> prob('wide') =3D 0.0911528
> prob('mark') =3D 0.136416
> prob('survey') =3D 0.0850202
> prob('http1:asp') =3D 0.88055
> prob("i'd") =3D 0.0470418
> prob('notices') =3D 0.01
>=20
> From VM Mon Jul 24 10:05:39 2000
> Return-Path: <undeliverables@mail.internetseer.com>
> Message-ID: <0112a1010021870MARS1@mars1.internetseer.com>
> Status: RO
> From: "InternetSeer.com" <services@mail.internetseer.com>
> To: jeremy@alum.mit.edu
> Subject: Your web site has been mapped
> Date: 23 Jul 2000 22:10:11 -0400
>=20
> Freewire has added your web site to its map of the World Wide Web.  F=
reewire will continue to monitor millions of links and web sites every =
day during its ongoing web survey.
>=20
> If it is important for you to know that your site is connected to the=
 web at all times, Freewire has arranged with InternetSeer.com to notif=
y you when your site does not respond.  This means that, AT NO CHARGE; =
InternetSeer.com will monitor your Web site every hour and send notific=
ation to you by email whenever your site is not connected to the Web. T=
here are NO current or future charges associated with this service.
>=20
> To begin your FREE monitoring NOW, activate your account at:
> http://www.internetseer.com/signup.asp?email=3Djeremy@alum.mit.edu
>=20
> Mark McLellan
> Chief Technology Officer
> Freewire.com
>=20
> Is your web site status important to you? I'd love your comments. If =
you prefer not to receive any future notices that result from our ongoi=
ng survey please let me know by returning this email with the word "rem=
ove" in the subject line.
>=20
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> ##Remove: jeremy@alum.mit.edu##
>=20
>=20

Jeremy


From guido@python.org  Fri Sep  6 21:26:52 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 06 Sep 2002 16:26:52 -0400
Subject: [Spambayes] understanding high false negative rate
In-Reply-To: Your message of "Fri, 06 Sep 2002 16:03:28 EDT."
             <15737.2576.315460.956295@slothrop.zope.com> 
References: <15737.2576.315460.956295@slothrop.zope.com> 
Message-ID: <200209062026.g86KQqJ03393@pcp02138704pcs.reston01.va.comcast.net>

> > Low prob spam! 1.64654685184e-11
> > /home/jeremy/Mail/spam:242 subject: your web site has been mapped
> > prob('millions') = 0.99
> > prob('skip:= 40') = 0.99
> > prob('"remove"') = 0.99
> > prob('from:email addr:mail') = 0.99
> > prob('email addr:alum') = 0.01
> > prob('status') = 0.01
> > prob('connected') = 0.01
> > prob('returning') = 0.01
> > prob('from:email addr:com>') = 0.224056
> > prob('every') = 0.789741
> > prob('charges') = 0.208406
> > prob('free') = 0.818103
> > prob('survey.') = 0.14931
> > prob('officer') = 0.208406
> > prob('its') = 0.155044
> > prob('added') = 0.133131
> > prob('current') = 0.152639
> > prob('email addr:mit') = 0.01
> > prob('wide') = 0.0911528
> > prob('mark') = 0.136416
> > prob('survey') = 0.0850202
> > prob('http1:asp') = 0.88055
> > prob("i'd") = 0.0470418
> > prob('notices') = 0.01

Looks like your ham corpus by and large has To: jeremy@alum.mit.edu in
a header while your spam corpus by and large doesn't.  But this one
does.

Where did you gather your spam corpus?  Could it be a collection of
edge cases that SA didn't kill, like Barry's collection of SA false
negatives?

--Guido van Rossum (home page: http://www.python.org/~guido/)

From jeremy@alum.mit.edu  Fri Sep  6 21:45:37 2002
From: jeremy@alum.mit.edu (Jeremy Hylton)
Date: Fri, 6 Sep 2002 16:45:37 -0400
Subject: [Spambayes] understanding high false negative rate
In-Reply-To: <200209062026.g86KQqJ03393@pcp02138704pcs.reston01.va.comcast.net>
References: <15737.2576.315460.956295@slothrop.zope.com>
	<200209062026.g86KQqJ03393@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <15737.5105.690511.94543@slothrop.zope.com>

>>>>> "GvR" == Guido van Rossum <guido@python.org> writes:

  GvR> Looks like your ham corpus by and large has To:
  GvR> jeremy@alum.mit.edu in a header while your spam corpus by and
  GvR> large doesn't.  But this one does.

By and large that's true.  Wouldn't it be true of any mailbox?  Most
of your real mail is addressed to you, but only some of the spam is.

  GvR> Where did you gather your spam corpus?  Could it be a
  GvR> collection of edge cases that SA didn't kill, like Barry's
  GvR> collection of SA false negatives?

A large chunk of my spam collection is from 2000.  The rest is recent,
starting about the same time spambayes did.  None of it was previously
filtered by SA.

Jeremy



From tim.one@comcast.net  Fri Sep  6 22:05:22 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 06 Sep 2002 17:05:22 -0400
Subject: [Spambayes] understanding high false negative rate
In-Reply-To: <15737.2576.315460.956295@slothrop.zope.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEJKBCAB.tim.one@comcast.net>

[Jeremy Hylton]
> I've tried to do some testing with some personal collections of ham
> and spam.  I'm seeing very high false negative rates. 20-30% is
> typical.

That's very high indeed.

> The false positive rate is 0-3%.  (Finally!  I had to scrub
> a bunch of previously unnoticed spam from my inbox.)  Both collections
> have about 1100 messages.

Does this mean you trained on about 1100 of each?

> I'd like to figure out why my false negative rate is so high, but I'm
> not sure what details I should look at to diagnose.  I'm assuming that
> mboxtest.py is basically correct, but it could have bugs.
>
> One possibility is that my ham test set isn't nearly so useful as the
> python-list, since it isn't focused on a single topic.

Heh -- when's the last time you read c.l.py <wink>?  "Python" is a very
strong ham indicator, and that certainly helps.  "wrote:" is an even
stronger ham indicator there, and that helps even more.

> I've got some python email, personal correspondence, questions about my
> Shakespeare web site, and a few email newsletters I get on a regular
basis.
> I've got receipts from various online order sites, mail from the company
> that manages my student loans, etc.  Maybe the great variety in my
> non-spam email makes it harder to find good discriminators for spam?

Can't guess.  You're in a good position to start adding more headers into
the analysis, though.  For example, an easy start would be to uncomment the
header-counting lines in tokenize() (look for "Anthony").  Likely the most
valuable thing it's missing then is some special parsing and tagging of
Received headers.

> Here's a sample spam distribution from a test run:
>
> Spam distribution for this pair:
> * = 3 items
>   0.00  73 *************************
>   2.50   0
>   5.00   2 *
>   7.50   0
>  10.00   0
>  12.50   1 *
>  15.00   0
>  17.50   1 *
>  20.00   1 *
>  22.50   0
>  25.00   2 *
>  27.50   0
>  30.00   0
>  32.50   0
>  35.00   0
>  37.50   0
>  40.00   0
>  42.50   0
>  45.00   0
>  47.50   0
>  50.00   0
>  52.50   0
>  55.00   0
>  57.50   1 *
>  60.00   0
>  62.50   1 *
>  65.00   0
>  67.50   0
>  70.00   1 *
>  72.50   0
>  75.00   0
>  77.50   0
>  80.00   2 *
>  82.50   2 *
>  85.00   2 *
>  87.50   0
>  90.00   4 **
>  92.50   1 *
>  95.00   5 **
>  97.50 127 *******************************************

So the bulk of your f-n woes come from spam scoring near 0.0.  Good to know.

> And here's a sample false negative.  (I'll quote the report so it
> stands out.)  One thing I don't understand is how the spam probability
> for the message is so low, when there are several high indicators and
> few low indicators.

You're hallucinating.  Let's look:

> > Low prob spam! 1.64654685184e-11
> > /home/jeremy/Mail/spam:242 subject: your web site has been mapped
> > prob('millions') = 0.99
> > prob('skip:= 40') = 0.99
> > prob('"remove"') = 0.99
> > prob('from:email addr:mail') = 0.99
> > prob('email addr:alum') = 0.01
> > prob('status') = 0.01
> > prob('connected') = 0.01
> > prob('returning') = 0.01

Those 8 cancel out completely.  They're the strongest indicators it found in
both directions, and it's exactly as if they didn't exist.  I'll sort the
rest from low to high:

> > prob('notices') = 0.01
> > prob('email addr:mit') = 0.01
> > prob("i'd") = 0.0470418
> > prob('survey') = 0.0850202
> > prob('wide') = 0.0911528
> > prob('added') = 0.133131
> > prob('mark') = 0.136416
> > prob('survey.') = 0.14931
> > prob('current') = 0.152639
> > prob('its') = 0.155044
> > prob('officer') = 0.208406
> > prob('charges') = 0.208406
> > prob('from:email addr:com>') = 0.224056

> > prob('every') = 0.789741
> > prob('http1:asp') = 0.88055
> > prob('free') = 0.818103

So you're got 13 indicators below 0.5, versus 3 above 0.5:  it's
overwhelmingly in favor of ham.


> >
> > From VM Mon Jul 24 10:05:39 2000
> > Return-Path: <undeliverables@mail.internetseer.com>
> > Message-ID: <0112a1010021870MARS1@mars1.internetseer.com>
> > Status: RO
> > From: "InternetSeer.com" <services@mail.internetseer.com>
> > To: jeremy@alum.mit.edu
> > Subject: Your web site has been mapped
> > Date: 23 Jul 2000 22:10:11 -0400
> >
> > Freewire has added your web site to its map of the World Wide
> Web.  Freewire will continue to monitor millions of links and web
> sites every day during its ongoing web survey.
> >
> > If it is important for you to know that your site is connected
> to the web at all times, Freewire has arranged with
> InternetSeer.com to notify you when your site does not respond.
> This means that, AT NO CHARGE; InternetSeer.com will monitor your
> Web site every hour and send notification to you by email
> whenever your site is not connected to the Web. There are NO
> current or future charges associated with this service.
> >
> > To begin your FREE monitoring NOW, activate your account at:
> > http://www.internetseer.com/signup.asp?email=jeremy@alum.mit.edu
> >
> > Mark McLellan
> > Chief Technology Officer
> > Freewire.com
> >
> > Is your web site status important to you? I'd love your
> comments. If you prefer not to receive any future notices that
> result from our ongoing survey please let me know by returning
> this email with the word "remove" in the subject line.
> >
> > =============================================
> > ##Remove: jeremy@alum.mit.edu##

Yuck:  it got two 0.01's from embedding your email address at the bottom
here.


From skip@pobox.com  Fri Sep  6 22:20:05 2002
From: skip@pobox.com (Skip Montanaro)
Date: Fri, 6 Sep 2002 16:20:05 -0500
Subject: [Spambayes] Deployment
In-Reply-To: <200209061924.g86JOc516514@pcp02138704pcs.reston01.va.comcast.net>
References: <LNBBLJKPBEHFEDALKOLCIEJABCAB.tim.one@comcast.net>
        <200209061924.g86JOc516514@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <15737.7173.450960.192144@12-248-11-90.client.attbi.com>


    >> FYI, I'll never trust such a scheme: I have no tolerance for false
    >> positives, and indeed do nothing to try to block spam on any of my
    >> email accounts now for that reason.  Deliver all suspected spam to a
    >> Spam folder instead and I'd love it.

    Guido> Another config parameter.

Nix on that idea.  If you make it a config parameter, some sysadmin is bound
to use it and lose important mail for a customer.

Essentially every mail user agent has some sort of filtering capability.
Instead of *ever* deleting mail you should simply tag it with your decision
about its spamminess and send it along.  The user can then configure
Outlook, Outlook Express, procmail, mutt, pine, VM, ... to do what she wants
with the stuff that trips the spam check.

The biggest false accusation people (not limited to the spammers) make about
tools like SpamAssassin is that it deletes email.  It doesn't.  It just tags
email.  Someone (the user or the sysadmin) made a decision to delete spammy
mail instead of squirreling it away for further review.  SA gets wrongly
accused.  Don't provide detractors with more ammunition than necessary.

just-message-tags-ma'am-ly y'rs,

Skip

From tim.one@comcast.net  Fri Sep  6 22:21:12 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 06 Sep 2002 17:21:12 -0400
Subject: [Spambayes] understanding high false negative rate
In-Reply-To: <15737.5105.690511.94543@slothrop.zope.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCEEJNBCAB.tim.one@comcast.net>

>   GvR> Looks like your ham corpus by and large has To:
>   GvR> jeremy@alum.mit.edu in a header while your spam corpus by and
>   GvR> large doesn't.  But this one does.

[Jeremy]
> By and large that's true.  Wouldn't it be true of any mailbox?  Most
> of your real mail is addressed to you, but only some of the spam is.

The "To:" header didn't play any role in this, assuming Jeremy is using
timtoken.tokenize.  I couldn't use To in my test runs because variants of
to:bruceg are extremely common in my spam collection.  The

    prob('email addr:mit') = 0.01
and
    prob('email addr:alum') = 0.01

came from the

    ##Remove: jeremy@alum.mit.edu##

embedded in the body of the msg.

> A large chunk of my spam collection is from 2000.  The rest is recent,
> starting about the same time spambayes did.  None of it was previously
> filtered by SA.

Good!  Even better, if Guido gets into his time machine and starts spamming
you from 2000, you're all set <wink>.


From tim.one@comcast.net  Fri Sep  6 22:32:21 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 06 Sep 2002 17:32:21 -0400
Subject: [Spambayes] Deployment
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEIPBCAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEJOBCAB.tim.one@comcast.net>

[Tim]
> My tests train on about 7,000 msgs, and a binary pickle of the database is
> approaching 10 million bytes.

That shrinks to under 2 million bytes, though, if I delete all the WordInfo
records with spamprob exactly equal to UNKNOWN_SPAMPROB.  Such records
aren't needed when scoring (an unknown word gets a made-up probability of
UNKNOWN_SPAMPROB).  Such records are only needed for training; I've noted
before that a scoring-only database can be leaner.

In part the bloat is due to character 5-gram'ing, part due to that the
database is brand new so has never been cleaned via clearjunk(), and part
due to plain evil gremlins.


From neale@woozle.org  Fri Sep  6 22:44:33 2002
From: neale@woozle.org (Neale Pickett)
Date: 06 Sep 2002 14:44:33 -0700
Subject: [Spambayes] Deployment
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEJOBCAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCGEJOBCAB.tim.one@comcast.net>
Message-ID: <w53vg5ibxoe.fsf@woozle.org>

So then, Tim Peters <tim.one@comcast.net> is all like:

> [Tim]
> > My tests train on about 7,000 msgs, and a binary pickle of the database is
> > approaching 10 million bytes.
> 
> That shrinks to under 2 million bytes, though, if I delete all the WordInfo
> records with spamprob exactly equal to UNKNOWN_SPAMPROB.  Such records
> aren't needed when scoring (an unknown word gets a made-up probability of
> UNKNOWN_SPAMPROB).  Such records are only needed for training; I've noted
> before that a scoring-only database can be leaner.

That's pretty good.  I wonder how much better you could do by using some
custom pickler.  I just checked my little dbm file and found a lot of
what I would call bloat:

  >>> import anydbm, hammie
  >>> d = hammie.PersistentGrahamBayes("ham.db")
  >>> db = anydbm.open("ham.db")
  >>> db["neale"], len(db["neale"])
  ('ccopy_reg\n_reconstructor\nq\x01(cclassifier\nWordInfo\nq\x02c__builtin__\nobject\nq\x03NtRq\x04(GA\xce\xbc{\xfd\x94\xbboK\x00K\x00K\x00G?\xe0\x00\x00\x00\x00\x00\x00tb.', 106)
  >>> d.wordinfo["neale"], len(`d.wordinfo["neale"]`)
  (WordInfo'(1031337979.16197, 0, 0, 0, 0.5)', 42)

Ignoring the fact that there are too many zeros in there, the pickled
version of that WordInfo object is over twice as large as the string
representation.  So we could get a 50% decrease in size just by using
the string representation instead of the pickle, right?

Something about that logic seems wrong to me, but I can't see what it
is.  Maybe pickling is good for heterogeneous data types, but every
value of our big dictionary is going to have the same type, so there's a
ton of redundancy.  I guess that explains why it compressed so well.

Neale

From skip@pobox.com  Fri Sep  6 23:39:48 2002
From: skip@pobox.com (Skip Montanaro)
Date: Fri, 6 Sep 2002 17:39:48 -0500
Subject: [Spambayes] understanding high false negative rate
In-Reply-To: <LNBBLJKPBEHFEDALKOLCIEJKBCAB.tim.one@comcast.net>
References: <15737.2576.315460.956295@slothrop.zope.com>
        <LNBBLJKPBEHFEDALKOLCIEJKBCAB.tim.one@comcast.net>
Message-ID: <15737.11956.18745.619040@12-248-11-90.client.attbi.com>


    >> > ##Remove: jeremy@alum.mit.edu##

    Tim> Yuck: it got two 0.01's from embedding your email address at the
    Tim> bottom here.

Which suggests that tagging email addresses in To/CC headers should be
handled differently than in message bodies?

Skip




From tim.one@comcast.net  Sat Sep  7 00:03:58 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 06 Sep 2002 19:03:58 -0400
Subject: [Spambayes] understanding high false negative rate
In-Reply-To: <15737.11956.18745.619040@12-248-11-90.client.attbi.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEKGBCAB.tim.one@comcast.net>

>     >> > ##Remove: jeremy@alum.mit.edu##
>
>     Tim> Yuck: it got two 0.01's from embedding your email address at the
>     Tim> bottom here.
>
> Which suggests that tagging email addresses in To/CC headers should be
> handled differently than in message bodies?

I don't know whether it suggests that, but they would be tagged differently
in to/cc if I were tagging them at all right now.  If I were tagging To:
addresses, for example, the tokens would look like

    'to:email addr:mit'

instead of

    'email addr:mit'

as they appear when an email-like thingie is found in the body.  Whether
email addresses should be stuck in as one blob or split up as they are now
is something I haven't tested.


From tim.one@comcast.net  Sat Sep  7 00:21:15 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 06 Sep 2002 19:21:15 -0400
Subject: [Spambayes] [ANN] Trained classifier available
In-Reply-To: <20020906162505.GB17800@cthulhu.gerg.ca>
Message-ID: <LNBBLJKPBEHFEDALKOLCMEKHBCAB.tim.one@comcast.net>

    http://sf.net/project/showfiles.php?group_id=61702

    This is the binary pickle of my classifier after training on
    my first spam/ham corpora pair.  All records with
    spamprob == UNKNOWN_SPAMPROB have been purged.

It's in a zip file, and is only half a meg.

Jeremy, it would be interesting if you tried that on your data.  The false
negative rates across my other 4 test sets when run against this are:

      0.364%
      0.400%
      0.400%
      0.909%


From neale@woozle.org  Sat Sep  7 00:50:19 2002
From: neale@woozle.org (Neale Pickett)
Date: 06 Sep 2002 16:50:19 -0700
Subject: [Spambayes] Ditching WordInfo
Message-ID: <w53r8g6brus.fsf@woozle.org>

I hacked up something to turn WordInfo into a tuple before pickling, and
then turn the tuple back into WordInfo right after unpickling.  Without
this hack, my database was 21549056 bytes.  After, it's 9945088 bytes.
That's a 50% savings, not a bad optimization.

So my question is, would it be too painful to ditch WordInfo in favor of
a straight out tuple?  (Or list if you'd rather, although making it a
tuple has the nice side-effect of forcing you to play nice with my
DBDict class).

I hope doing this sort of optimization isn't too far distant from the
goal of this project, even though README.txt says it is :)

Diff attached.  I'm not comfortable checking this in, since I don't
really like how it works (I'd rather just get rid of WordInfo).  But I
guess it proves the point :)

Neale

---8<---
? classifier.pyc
? d
? ham.db
? ham.pickle
? ham.spamoracle
? hammie.pyc
? timtoken.pyc
Index: hammie.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/hammie.py,v
retrieving revision 1.5
diff -u -r1.5 hammie.py
--- hammie.py	6 Sep 2002 20:48:29 -0000	1.5
+++ hammie.py	6 Sep 2002 23:48:34 -0000
@@ -1,7 +1,8 @@
 #! /usr/bin/env python
 
 # A driver for the classifier module.  Currently mostly a wrapper around
-# existing stuff.
+# existing stuff.  Neale Pickett <neale@woozle.org> is the person to
+# blame for this.
 
 """Usage: %(program)s [options]
 
@@ -36,6 +37,7 @@
 import errno
 import anydbm
 import cPickle as pickle
+from types import *
 
 program = sys.argv[0]
 
@@ -69,11 +71,24 @@
 
     def __getitem__(self, key):
         if self.hash.has_key(key):
-            return pickle.loads(self.hash[key])
+            val = pickle.loads(self.hash[key])
+            # XXX: kludge kludge kludge.  There's a more elegant
+            # solution, but this proves the concept for the time being.
+            if type(val) == TupleType \
+                   and len(val) == len(classifier.WordInfo.__slots__):
+                # How does pickle pull this off?
+                w = classifier.WordInfo(0)
+                w.__setstate__(val)
+                val = w
+            return val
         else:
             raise KeyError(key)
 
-    def __setitem__(self, key, val): 
+    def __setitem__(self, key, val):
+        # XXX: This has got to go when the __getitem__ kludge is cleaned
+        # up
+        if isinstance(val, classifier.WordInfo):
+            val = val.__getstate__()
         v = pickle.dumps(val, 1)
         self.hash[key] = v
 
@@ -84,7 +99,7 @@
         k = self.hash.first()
         while k != None:
             key = k[0]
-            val = pickle.loads(k[1])
+            val = self.__getitem__(key)
             if key not in self.iterskip:
                 if fn:
                     yield fn((key, val))

---8<---

From jeremy@alum.mit.edu  Sat Sep  7 01:00:14 2002
From: jeremy@alum.mit.edu (Jeremy Hylton)
Date: Fri, 6 Sep 2002 20:00:14 -0400
Subject: [Spambayes] understanding high false negative rate
In-Reply-To: <LNBBLJKPBEHFEDALKOLCIEJKBCAB.tim.one@comcast.net>
References: <15737.2576.315460.956295@slothrop.zope.com>
	<LNBBLJKPBEHFEDALKOLCIEJKBCAB.tim.one@comcast.net>
Message-ID: <15737.16782.542869.368986@slothrop.zope.com>

>>>>> "TP" == Tim Peters <tim.one@comcast.net> writes:

  >> The false positive rate is 0-3%.  (Finally!  I had to scrub a
  >> bunch of previously unnoticed spam from my inbox.)  Both
  >> collections have about 1100 messages.

  TP> Does this mean you trained on about 1100 of each?

The total collections are 1100 messages.  I trained with 1100/5
messages. 

  TP> Can't guess.  You're in a good position to start adding more
  TP> headers into the analysis, though.  For example, an easy start
  TP> would be to uncomment the header-counting lines in tokenize()
  TP> (look for "Anthony").  Likely the most valuable thing it's
  TP> missing then is some special parsing and tagging of Received
  TP> headers.

I tried the "Anthony" stuff, but it didn't make any appreciable
difference that I could see from staring at the false negative rate.
The numbers are big enough that a quick eyeball suffices.

Then I tried a dirt simple tokenizer for the headers that tokenize the
words in the header and emitted like this "%s: %s" % (hdr, word).
That worked too well :-).  The received and date headers helped the
classifier discover that most of my spam is old and most of my ham is
new.

So I tried a slightly more complex one that skipped received, data,
and x-from_, which all contained timestamps.  I also skipped the X-VM-
headers that my mail reader added:

class MyTokenizer(Tokenizer):

    skip = {'received': 1,
            'date': 1,
            'x-from_': 1,
            }

    def tokenize_headers(self, msg):
        for k, v in msg.items():
            k = k.lower()
            if k in self.skip or k.startswith('x-vm'):
                continue
            for w in subject_word_re.findall(v):
                for t in tokenize_word(w):
                    yield "%s:%s" % (k, t)

This did moderately better.  The false negative rate is 7-21% over the
tests performed so far.  This is versus 11-28% for the previous test
run that used the timtest header tokenizer.

It's interesting to see that the best descriminators are all ham
discriminators.  There's not a single spam-indicator in the list.
Most of the discriminators are header fields.  One thing to note is
that the presence of Mailman-generated headers is a strong non-spam
indicator.  That matches my intuition: I got an awful lot of
Mailman-generated mail, and those lists are pretty good at surpressing
spam.  The other thing is that I get a lot of ham from people who use
XEmacs.  That's probably Barry, Guido, Fred, and me :-).

One final note.  It looks like many of the false positives are from
people I've never met with questions about Shakespeare.  They often
start with stuff like:

> Dear Sir/Madam,
> 
> May I please take some of your precious time to ask you to help me to find a
> solution to a problem that is worrying me greatly. I am old science student

I guess that reads a lot like spam :-(.

Jeremy


238 hams & 221 spams
    false positive: 2.10084033613
    false negative: 9.50226244344
    new false positives: []
    new false negatives: []

    best discriminators:
        'x-mailscanner:clean' 671 0.0483425
        'x-spam-status:IN_REP_TO' 679 0.01
        'delivered-to:skip:s 10' 691 0.0829876
        'x-mailer:Lucid' 699 0.01
        'x-mailer:XEmacs' 699 0.01
        'x-mailer:patch' 699 0.01
        'x-mailer:under' 709 0.01
        'x-mailscanner:Found' 716 0.0479124
        'cc:zope.com' 718 0.01
        "i'll" 750 0.01
        'references:skip:1 20' 767 0.01
        'rossum' 795 0.01
        'x-spam-status:skip:S 10' 825 0.01
        'van' 850 0.01
        'http0:zope' 869 0.01
        'email addr:zope' 883 0.01
        'from:python.org' 895 0.01
        'to:jeremy' 902 0.185401
        'zope' 984 0.01
        'list-archive:skip:m 10' 1058 0.01
        'list-subscribe:skip:m 10' 1058 0.01
        'list-unsubscribe:skip:m 10' 1058 0.01
        'from:zope.com' 1098 0.01
        'return-path:zope.com' 1115 0.01
        'wrote:' 1129 0.01
        'jeremy' 1150 0.01
        'email addr:python' 1257 0.01
        'x-mailman-version:2.0.13' 1311 0.01
        'x-mailman-version:101270' 1395 0.01
        'python' 1401 0.01


From tim.one@comcast.net  Sat Sep  7 01:06:56 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 06 Sep 2002 20:06:56 -0400
Subject: [Spambayes] Ditching WordInfo
In-Reply-To: <w53r8g6brus.fsf@woozle.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCOEKKBCAB.tim.one@comcast.net>

[Neale Pickett]
> I hacked up something to turn WordInfo into a tuple before pickling,

That's what WordInfo.__getstate__ does.

> and then turn the tuple back into WordInfo right after unpickling.

Likewise for WordInfo.__setstate__.

> Without this hack, my database was 21549056 bytes.  After, it's 9945088
bytes.
> That's a 50% savings, not a bad optimization.

I'm not sure what you're doing, but suspect you're storing individual
WordInfo pickles.  If so, most of the administrative pickle bloat is due to
that, and doesn't happen if you pickle an entire classifier instance
directly.

> So my question is, would it be too painful to ditch WordInfo in favor of
> a straight out tuple?  (Or list if you'd rather, although making it a
> tuple has the nice side-effect of forcing you to play nice with my
> DBDict class).
>
> I hope doing this sort of optimization isn't too far distant from the
> goal of this project, even though README.txt says it is :)
>
> Diff attached.  I'm not comfortable checking this in,

I think it's healthy that you're uncomfortable checking things in with

> +            # XXX: kludge kludge kludge.

comments <wink>.

> since I don't really like how it works (I'd rather just get rid of
WordInfo).
> But I guess it proves the point :)

I'm not interested in optimizing anything yet, and get many benefits from
the *ease* of working with utterly vanilla Python instance objects.  Lots of
code all over picks these apart for display and analysis purposes.  Very few
people have tried this code yet, and there are still many questions about it
(see, e.g., Jeremy's writeup of his disappointing first-time experiences
today).  Let's keep it as easy as possible to modify for now.  If you're
desparate to save memory, write a subclass?

Other people are free to vote in other directions, of course <wink>.


From tim.one@comcast.net  Sat Sep  7 01:18:18 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 06 Sep 2002 20:18:18 -0400
Subject: [Spambayes] test sets?
In-Reply-To: <15736.57093.811682.371784@anthem.wooz.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEKLBCAB.tim.one@comcast.net>

[Barry]
> Here's an interesting thing to test: discriminate words differently if
> they are on a line that starts with `>' or, to catch styles like
> above, that the first occurance on a line of < or > is > (to eliminate
> html).

Give me a mod to timtoken.py that does this, and I'll be happy to test it.

> Then again, it may not be worth trying to un-false-positive that
> Nigerian scam quote.

If there's any sanity in the world, even the original poster would be glad
to have his kneejerk response blocked <wink>.  OTOH, you know there are a
great many msgs on c.l.py (all over Usenet) that do nothing except quote a
previous post and add a one-line comment.  Remove the quoted sections from
those, and there may be no content left to judge except for the headers.  So
I can see this nudging the stats in either direction.  The only way to find
out for sure is for you to write some code <wink>.


From tim.one@comcast.net  Sat Sep  7 01:32:26 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 06 Sep 2002 20:32:26 -0400
Subject: [Spambayes] understanding high false negative rate
In-Reply-To: <15737.16782.542869.368986@slothrop.zope.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEKNBCAB.tim.one@comcast.net>

[Jeremy Hylton[
> The total collections are 1100 messages.  I trained with 1100/5
> messages.

I'm reading this now as that you trained on about 220 spam and about 220
ham.  That's less than 10% of the sizes of the training sets I've been
using.  Please try an experiment:  train on 550 of each, and test once
against the other 550 of each.  Do that a few times making a random split
each time (it won't be long until you discover why directories of individual
files are a lot easier to work -- e.g., random.shuffle() makes this kind of
thing trivial for me).


From tim.one@comcast.net  Sat Sep  7 03:51:24 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 06 Sep 2002 22:51:24 -0400
Subject: [Spambayes] understanding high false negative rate
In-Reply-To: <15737.16782.542869.368986@slothrop.zope.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEKOBCAB.tim.one@comcast.net>

[Jeremy]
> The total collections are 1100 messages.  I trained with 1100/5
> messages.

While that's not a lot of training data, I picked random subsets of my
corpora and got much better behavior (this is rates.py output; f-p rate per
run in left column, f-n rate in right):

Training on Data/Ham/Set1 & Data/Spam/Set1 ... 220 hams & 220 spams
      0.000   1.364
      0.000   0.455
      0.000   1.818
      0.000   1.364
Training on Data/Ham/Set2 & Data/Spam/Set2 ... 220 hams & 220 spams
      0.455   2.727
      0.455   0.455
      0.000   0.909
      0.455   2.273
Training on Data/Ham/Set3 & Data/Spam/Set3 ... 220 hams & 220 spams
      0.000   2.727
      0.455   0.909
      0.000   0.909
      0.000   1.818
Training on Data/Ham/Set4 & Data/Spam/Set4 ... 220 hams & 220 spams
      0.000   2.727
      0.000   0.909
      0.000   0.909
      0.000   1.818
Training on Data/Ham/Set5 & Data/Spam/Set5 ... 220 hams & 220 spams
      0.000   1.818
      0.000   1.364
      0.000   0.909
      0.000   2.273
total false pos 4 0.363636363636
total false neg 29 2.63636363636

Another full run with another randomly chosen (but disjoint) 220 of each in
each set was much the same.  The score distribution is also quite sharp:

Ham distribution for all runs:
* = 74 items
  0.00 4381 ************************************************************
  2.50    3 *
  5.00    3 *
  7.50    1 *
 10.00    0
 12.50    0
 15.00    1 *
 17.50    1 *
 20.00    1 *
 22.50    0
 25.00    0
 27.50    0
 30.00    1 *
 32.50    0
 35.00    0
 37.50    0
 40.00    1 *
 42.50    0
 45.00    0
 47.50    0
 50.00    0
 52.50    0
 55.00    0
 57.50    1 *
 60.00    0
 62.50    0
 65.00    0
 67.50    1 *
 70.00    0
 72.50    0
 75.00    0
 77.50    0
 80.00    0
 82.50    0
 85.00    0
 87.50    1 *
 90.00    0
 92.50    2 *
 95.00    0
 97.50    2 *

Spam distribution for all runs:
* = 73 items
  0.00   13 *
  2.50    0
  5.00    4 *
  7.50    5 *
 10.00    0
 12.50    2 *
 15.00    1 *
 17.50    1 *
 20.00    2 *
 22.50    1 *
 25.00    0
 27.50    1 *
 30.00    0
 32.50    3 *
 35.00    0
 37.50    0
 40.00    0
 42.50    0
 45.00    1 *
 47.50    3 *
 50.00   16 *
 52.50    0
 55.00    0
 57.50    0
 60.00    1 *
 62.50    0
 65.00    2 *
 67.50    1 *
 70.00    1 *
 72.50    0
 75.00    1 *
 77.50    0
 80.00    3 *
 82.50    2 *
 85.00    1 *
 87.50    2 *
 90.00    2 *
 92.50    4 *
 95.00    4 *
 97.50 4323 ************************************************************

It's hard to say whether you need better ham or better spam, but I suspect
better spam <wink>.  18 of the 30 most powerful discriminators here were
HTML-related spam indicators; the top 10 overall were:

        '<font' 266 0.99
        'content-type:text/plain' 275 0.172932
        '</body>' 312 0.99
        '</html>' 329 0.99
        'click' 334 0.99
        '<html>' 335 0.99
        'wrote:' 381 0.01
        'skip:< 10' 398 0.99
        'python' 428 0.01
        'content-type:text/html' 454 0.99

The HTML tags come from non-multipart/alternative HTML messages, from which
HTML tags aren't stripped, and there are lots of these in my spam sets.

That doesn't account for it, though.  If I strip HTML tags out of those too,
the rates are only a little worse:

raining on Data/Ham/Set1 & Data/Spam/Set1 ... 220 hams & 220 spams
     0.000   1.364
     0.000   1.818
     0.455   1.818
     0.000   1.818
raining on Data/Ham/Set2 & Data/Spam/Set2 ... 220 hams & 220 spams
     0.000   1.364
     0.455   1.818
     0.455   0.909
     0.000   1.818
raining on Data/Ham/Set3 & Data/Spam/Set3 ... 220 hams & 220 spams
     0.000   2.727
     0.000   0.909
     0.909   0.909
     0.455   1.818
raining on Data/Ham/Set4 & Data/Spam/Set4 ... 220 hams & 220 spams
     0.000   1.818
     0.000   0.909
     0.455   0.909
     0.000   1.364
raining on Data/Ham/Set5 & Data/Spam/Set5 ... 220 hams & 220 spams
     0.000   2.727
     0.000   1.364
     0.455   2.273
     0.455   2.273
otal false pos 4 0.363636363636
otal false neg 34 3.09090909091

The 4th-strongest discriminator *still* finds another HTML clue, though!

        'subject:Python' 164 0.01
        'money' 169 0.99
        'content-type:text/plain' 185 0.2
        'charset:us-ascii' 191 0.127273
        "i'm" 232 0.01
        'content-type:text/html' 248 0.983607
        '&nbsp;' 255 0.99
        'wrote:' 372 0.01
        'python' 431 0.01
        'click' 519 0.99

Heh.  I forgot all about &nbsp;.


From anthony@interlink.com.au  Sat Sep  7 04:38:51 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Sat, 07 Sep 2002 13:38:51 +1000
Subject: [Spambayes] test sets? 
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEICBCAB.tim.one@comcast.net> 
Message-ID: <200209070338.g873cpp20640@localhost.localdomain>


> > Note that header names are case insensitive, so this one's no
> > different than "MIME-Version:".  Similarly other headers in your list.
> 
> Ignoring case here may or may not help; that's for experiment to decide.
> It's plausible that case is significant, if, e.g., a particular spam mailing
> package generates unusual case, or a particular clueless spammer
> misconfigures his package.

I found it made no difference for my testing.

> The brilliance of Anthony's "just count them" scheme is that it requires no
> thought, so can't be fooled <wink>.  Header lines that are evenly
> distributed across spam and ham will turn out to be worthless indicators
> (prob near 0.5), so do no harm.

zactly. I started off doing clever clever things, and, as always with
this stuff, found that stupid with a rock beats smart with scissors,
every time.


-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.


From anthony@interlink.com.au  Sat Sep  7 04:44:50 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Sat, 07 Sep 2002 13:44:50 +1000
Subject: [Spambayes] understanding high false negative rate 
In-Reply-To: 
	<200209062026.g86KQqJ03393@pcp02138704pcs.reston01.va.comcast.net> 
Message-ID: <200209070344.g873io020676@localhost.localdomain>


> Looks like your ham corpus by and large has To: jeremy@alum.mit.edu in
> a header while your spam corpus by and large doesn't.  But this one
> does.

Interestingly, for me, one of the highest value spam indicators was
the name of the mail host that the spam was delivered to, in the To:
line. So mail to info@gin.elax2.ekorp.com was pretty much a dead cert
for the filters.


-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.


From anthony@interlink.com.au  Sat Sep  7 04:50:37 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Sat, 07 Sep 2002 13:50:37 +1000
Subject: [Spambayes] understanding high false negative rate 
In-Reply-To: <15737.16782.542869.368986@slothrop.zope.com> 
Message-ID: <200209070350.g873obE20720@localhost.localdomain>


>>> Jeremy Hylton wrote
> Then I tried a dirt simple tokenizer for the headers that tokenize the
> words in the header and emitted like this "%s: %s" % (hdr, word).
> That worked too well :-).  The received and date headers helped the
> classifier discover that most of my spam is old and most of my ham is
> new.

Heh. I hit the same problem, but the other way round, when I first
started playing with this - I'd collected spam for a week or two, 
then mixed it up with randomly selected messages from my mail boxes.

course, it instantly picked up on 'received:2001' as a non-ham. 

Curse that too-smart-for-me software. Still, it's probably a good
thing to note in the documentation about the software - when collecting
spam/ham, make _sure_ you try and collect from the same source.


Anthony

-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.


From anthony@interlink.com.au  Sat Sep  7 04:52:54 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Sat, 07 Sep 2002 13:52:54 +1000
Subject: [Spambayes] understanding high false negative rate 
In-Reply-To: <200209070350.g873obE20720@localhost.localdomain> 
Message-ID: <200209070352.g873qs820746@localhost.localdomain>



> course, it instantly picked up on 'received:2001' as a non-ham. 
                                                            -spam.

*sigh*


-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.


From guido@python.org  Sat Sep  7 04:51:12 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 06 Sep 2002 23:51:12 -0400
Subject: [Spambayes] hammie.py vs. GBayes.py
Message-ID: <200209070351.g873pC613144@pcp02138704pcs.reston01.va.comcast.net>

There seem to be two "drivers" for the classifier now: Neale Pickett's
hammie.py, and the original GBayes.py.  According to the README.txt,
GBayes.py hasn't been kept up to date.  Is there anything in there
that isn't covered by hammie.py?  About the only useful feature of
GBayes.py that hammie.py doesn't (yet) copy is -u, which calculates
spamness for an entire mailbox.  This feature can easily be copied
into hammie.py.  (GBayes.py also has a large collection of tokenizers;
but timtoken.py rules, so I'm not sure how interesting that is now.)

Therefore I propose to nuke GBayes.py, after adding a -u feature.

Anyone against?  (I imagine that Skip or Barry might have a stake in
GBayes.py; Tim seems to have moved all code he's working to other
modules.)

--Guido van Rossum (home page: http://www.python.org/~guido/)

From anthony@interlink.com.au  Sat Sep  7 05:00:36 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Sat, 07 Sep 2002 14:00:36 +1000
Subject: [Spambayes] test sets? 
In-Reply-To: <LNBBLJKPBEHFEDALKOLCEEHFBCAB.tim.one@comcast.net> 
Message-ID: <200209070400.g8740a520809@localhost.localdomain>


>>> Tim Peters wrote
> > As well as the usual spam, it also has customers complaining about
> > credit card charges, it has people interested in the service and
> > asking questions about long distance rates, &c &c &c. Lots and lots
> > of "commercial" speech, in other words. Stuff that SA gets pretty
> > badly wrong.
> 
> Can this corpus be shared?  I suppose not.

Almost certainly 100% not, at least not without a massive massive
amount of manual cleansing. There's just too much personal data in
there.

> > I did have Received in there, but it's out for the moment, as it causes
> > rates to drop.
> That's ambiguous.  Accuracy rates or error rates, ham or spam rates?

It made both the f-p and f-n rates drop. I need to think a bit more
about why - I'm currently thinking about a special tokeniser just for
received, so that, e.g., hostnames like 
'pcp736393pcs.reston01.va.comcast.net' gets turned into

received:pcp736393pcs.reston01.va.comcast.net
received:reston01.va.comcast.net
received:va.comcast.net
received:comcast.net

Specialising the tokeniser for various headers actually seems to do
some good - in particular, keeping the parameters and their values of
the content-types makes for a good detector of korean spam.

> Mining embedded http/https/ftp thingies cut the false negative rate in half
> in my tests (not keying off href, just scanning for anything that "looked
> like" one); that was the single biggest f-n improvement I've seen.  It
> didn't change the false positive rate.  So you know whether src added
> additional power, or did you do both at once?

Both at once. I added it because <iframe src=cid:foofoofoo> is such a
killer detector of spam/viruses, also because I got a bunch of email
spam that was just

    <img src="bozo.bozo.kr/img34532.jpg">
    <img src="bozo.bozo.kr/img34512.jpg">
    <img src="bozo.bozo.kr/img34237.jpg">
    <img src="bozo.bozo.kr/img34914.jpg">

I'm already stripping out HTML tags - it was producing far far too
many false positives with my corpus with them in. Without the src/hrefs
these spams were pretty much null and void.



From skip@pobox.com  Sat Sep  7 05:05:45 2002
From: skip@pobox.com (Skip Montanaro)
Date: Fri, 6 Sep 2002 23:05:45 -0500
Subject: [Spambayes] Maybe change X-Spam-Disposition to something else...
Message-ID: <15737.31513.555967.915801@12-248-11-90.client.attbi.com>


I actually like Neale's X-Spam-Disposition header, I just wonder if maybe we
should choose something with a different prefix than "X-Spam-" so that
people don't confuse it with SpamAssassin, all of whose headers begin with
that prefix.

Also, some sort of version information would be useful.  Perhaps the CVS
version number for the classifier module could be tacked on.

Skip

From anthony@interlink.com.au  Sat Sep  7 05:10:37 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Sat, 07 Sep 2002 14:10:37 +1000
Subject: [Spambayes] Maybe change X-Spam-Disposition to something else... 
In-Reply-To: <15737.31513.555967.915801@12-248-11-90.client.attbi.com> 
Message-ID: <200209070410.g874Ab620880@localhost.localdomain>


>>> Skip Montanaro wrote
> 
> I actually like Neale's X-Spam-Disposition header, I just wonder if maybe we
> should choose something with a different prefix than "X-Spam-" so that
> people don't confuse it with SpamAssassin, all of whose headers begin with
> that prefix.

I think it's fine, in general, just so long as no-one checks in anything
that puts it into my test corpus.

Or alternately, whatever is chosen should be ignored by the tokenizer. 
I know my mail host (interlink) runs SA, but I also run it, with my own
set of rules and scores. I don't want my spam-filter to be getting messed
up by an upstream spam filter.



From skip@pobox.com  Sat Sep  7 05:30:31 2002
From: skip@pobox.com (Skip Montanaro)
Date: Fri, 6 Sep 2002 23:30:31 -0500
Subject: [Spambayes] hammie.py vs. GBayes.py
In-Reply-To: <200209070351.g873pC613144@pcp02138704pcs.reston01.va.comcast.net>
References: <200209070351.g873pC613144@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <15737.32999.256881.434812@12-248-11-90.client.attbi.com>


    Guido> Therefore I propose to nuke GBayes.py, after adding a -u feature.

    Guido> Anyone against?  (I imagine that Skip or Barry might have a stake
    Guido> in GBayes.py; Tim seems to have moved all code he's working to
    Guido> other modules.)

No argument here.

Skip

From skip@pobox.com  Sat Sep  7 05:46:01 2002
From: skip@pobox.com (Skip Montanaro)
Date: Fri, 6 Sep 2002 23:46:01 -0500
Subject: [Spambayes] speed
Message-ID: <15737.33929.716821.779152@12-248-11-90.client.attbi.com>


If the frequency of my laptop's disk chirps are any indication, I'd say
hammie is about 3-5x faster than SpamAssassin.

Skip


From skip@pobox.com  Sat Sep  7 05:44:07 2002
From: skip@pobox.com (Skip Montanaro)
Date: Fri, 6 Sep 2002 23:44:07 -0500
Subject: [Spambayes] Maybe change X-Spam-Disposition to something else... 
In-Reply-To: <200209070410.g874Ab620880@localhost.localdomain>
References: <15737.31513.555967.915801@12-248-11-90.client.attbi.com>
        <200209070410.g874Ab620880@localhost.localdomain>
Message-ID: <15737.33815.719456.367026@12-248-11-90.client.attbi.com>


---------------------- multipart/mixed attachment

    >> I actually like Neale's X-Spam-Disposition header, I just wonder if
    >> maybe we should choose something with a different prefix than
    >> "X-Spam-" so that people don't confuse it with SpamAssassin, all of
    >> whose headers begin with that prefix.

    Anthony> I think it's fine, in general, just so long as no-one checks in
    Anthony> anything that puts it into my test corpus.

I took a somewhat different approach.  Lots of stuff puts synthetic headers
in my mail.  VM has a whole series of X-VM headers.  SpamAssassin adds
X-Spam headers.  In my .procmailrc file I even add an X-SA-Host header so I
know what machine SpamAssassin was run on.  Any or all of those could
provide spurious clues to hammie.

Accordingly, I wrote unheader.py, which is mostly a ripoff of something
someone else posted to python-dev or c.l.py within the last week or so to
strip out SA-generated headers.  Before checking a mailbox, I run unheader
over it like so:

    unheader python > python.clean

then run hammie on the result to produce python.check.  That then gets
analyzed.

A copy is attached.  It could easily be extended to zap arbitrary header
re's, but I haven't needed that so far.

Skip


---------------------- multipart/mixed attachment
A non-text attachment was scrubbed...
Name: not available
Type: application/octet-stream
Size: 2421 bytes
Desc: not available
Url : http://mail.python.org/pipermail-21/spambayes/attachments/20020906/2454e663/attachment.exe

---------------------- multipart/mixed attachment--

From skip@pobox.com  Sat Sep  7 06:15:54 2002
From: skip@pobox.com (Skip Montanaro)
Date: Sat, 7 Sep 2002 00:15:54 -0500
Subject: [Spambayes] can't write to CVS...
Message-ID: <15737.35722.956784.600958@12-248-11-90.client.attbi.com>

I'm listed as a developer on SF and have the spambayes CVS module checked
out using my SF username, but I'm unable to write to the repository.  CVS
complains:

    % cvs add unheader.py
    cvs [server aborted]: "add" requires write access to the repository

Any thoughts?

Skip

From neale@woozle.org  Sat Sep  7 06:17:28 2002
From: neale@woozle.org (Neale Pickett)
Date: 06 Sep 2002 22:17:28 -0700
Subject: [Spambayes] Ditching WordInfo
In-Reply-To: <LNBBLJKPBEHFEDALKOLCOEKKBCAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCOEKKBCAB.tim.one@comcast.net>
Message-ID: <w53n0qubcpj.fsf@woozle.org>

So then, Tim Peters <tim.one@comcast.net> is all like:

> I'm not sure what you're doing, but suspect you're storing individual
> WordInfo pickles.  If so, most of the administrative pickle bloat is
> due to that, and doesn't happen if you pickle an entire classifier
> instance directly.

Yeah, that's exactly what I was doing--I didn't realize I was incurring
administrative pickle bloat this way.  I'm specifically trying to make
things faster and smaller, so I'm storing individual WordInfo pickles
into an anydbm dict (keyed by token).  The result is that it's almost 50
times faster to score messages one per run our of procmail (.408s vs
18.851s).

However, it *does* say all over the place that the goal of this project
isn't to make the fastest or the smallest implementation, so I guess
I'll hold off doing any further performance tuning until the goal starts
to point more in that direction.  .4 seconds is probably fast enough for
people to use it in their procmailrc, which is what I was after.

> If you're desparate to save memory, write a subclass?

That's probably what I'll do if I get too antsy :)

Trying to think of ways to sneak "administrative pickle boat" into
casual conversation,

Neale

From neale@woozle.org  Sat Sep  7 06:25:45 2002
From: neale@woozle.org (Neale Pickett)
Date: 06 Sep 2002 22:25:45 -0700
Subject: [Spambayes] Maybe change X-Spam-Disposition to something else...
In-Reply-To: <15737.31513.555967.915801@12-248-11-90.client.attbi.com>
References: <15737.31513.555967.915801@12-248-11-90.client.attbi.com>
Message-ID: <w53fzwmbcbq.fsf@woozle.org>

So then, Skip Montanaro <skip@pobox.com> is all like:

> I actually like Neale's X-Spam-Disposition header, I just wonder if maybe we
> should choose something with a different prefix than "X-Spam-" so that
> people don't confuse it with SpamAssassin, all of whose headers begin with
> that prefix.

Good point.  How about X-Spambayes-Disposition (or X-Hammie-Disposition
if there'll be other classifier front-ends)?


From guido@python.org  Sat Sep  7 06:33:23 2002
From: guido@python.org (Guido van Rossum)
Date: Sat, 07 Sep 2002 01:33:23 -0400
Subject: [Spambayes] Ditching WordInfo
In-Reply-To: Your message of "Fri, 06 Sep 2002 22:17:28 PDT."
             <w53n0qubcpj.fsf@woozle.org> 
References: <LNBBLJKPBEHFEDALKOLCOEKKBCAB.tim.one@comcast.net>  
            <w53n0qubcpj.fsf@woozle.org> 
Message-ID: <200209070533.g875XN813509@pcp02138704pcs.reston01.va.comcast.net>

> Yeah, that's exactly what I was doing--I didn't realize I was
> incurring administrative pickle bloat this way.  I'm specifically
> trying to make things faster and smaller, so I'm storing individual
> WordInfo pickles into an anydbm dict (keyed by token).  The result
> is that it's almost 50 times faster to score messages one per run
> our of procmail (.408s vs 18.851s).

This is very worthwhile.

> However, it *does* say all over the place that the goal of this
> project isn't to make the fastest or the smallest implementation, so
> I guess I'll hold off doing any further performance tuning until the
> goal starts to point more in that direction.  .4 seconds is probably
> fast enough for people to use it in their procmailrc, which is what
> I was after.

Maybe.  I batch messages using fetchmail (don't ask why), and adding
.4 seconds per message for a batch of 50 (not untypical) feels like a
real wait to me...

--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.one@comcast.net  Sat Sep  7 06:36:41 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 07 Sep 2002 01:36:41 -0400
Subject: [Spambayes] can't write to CVS...
In-Reply-To: <15737.35722.956784.600958@12-248-11-90.client.attbi.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCCELKBCAB.tim.one@comcast.net>

[Skip Montanaro]
> I'm listed as a developer on SF and have the spambayes CVS module checked
> out using my SF username, but I'm unable to write to the repository.  CVS
> complains:
>
>     % cvs add unheader.py
>     cvs [server aborted]: "add" requires write access to the repository
>
> Any thoughts?

Not really.  Try again?  About half the developers on the spambayes project
were missing some permission or other, so I ran thru all of them and checked
every damned box and clicked on every damn dropdown list I could find.  As
far as SF is concerned, you're all sitting on God's Right Hand now, so if it
still doesn't work I suggest you upgrade to Win98 <wink>.


From skip@pobox.com  Sat Sep  7 06:40:18 2002
From: skip@pobox.com (Skip Montanaro)
Date: Sat, 7 Sep 2002 00:40:18 -0500
Subject: [Spambayes] can't write to CVS...
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCELKBCAB.tim.one@comcast.net>
References: <15737.35722.956784.600958@12-248-11-90.client.attbi.com>
        <LNBBLJKPBEHFEDALKOLCCELKBCAB.tim.one@comcast.net>
Message-ID: <15737.37186.513740.936811@12-248-11-90.client.attbi.com>


    Tim> About half the developers on the spambayes project were missing
    Tim> some permission or other, so I ran thru all of them and checked
    Tim> every damned box and clicked on every damn dropdown list I could
    Tim> find.  As far as SF is concerned, you're all sitting on God's Right
    Tim> Hand now, so if it still doesn't work I suggest you upgrade to
    Tim> Win98 <wink>.

Time to upgrade I guess. :-(

    % cvs add unheader.py 
    cvs [server aborted]: "add" requires write access to the repository

I'll try a checkout into a new directory...

S

From tim.one@comcast.net  Sat Sep  7 06:40:45 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 07 Sep 2002 01:40:45 -0400
Subject: [Spambayes] Maybe change X-Spam-Disposition to something else...
In-Reply-To: <15737.33815.719456.367026@12-248-11-90.client.attbi.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCAELLBCAB.tim.one@comcast.net>

[Skip Montanaro, to Anthony Baxter]
> ...
> Accordingly, I wrote unheader.py, which is mostly a ripoff of something
> someone else posted to python-dev or c.l.py within the last week or so to
> strip out SA-generated headers.

Unless I've grown senile tonight, you got it from Anthony to begin with.
Please check it in to project, and add a short blurb to README.txt!


From skip@pobox.com  Sat Sep  7 06:43:31 2002
From: skip@pobox.com (Skip Montanaro)
Date: Sat, 7 Sep 2002 00:43:31 -0500
Subject: [Spambayes] Maybe change X-Spam-Disposition to something else...
In-Reply-To: <w53fzwmbcbq.fsf@woozle.org>
References: <15737.31513.555967.915801@12-248-11-90.client.attbi.com>
        <w53fzwmbcbq.fsf@woozle.org>
Message-ID: <15737.37379.699338.64767@12-248-11-90.client.attbi.com>


    >> ... choose something with a different prefix than "X-Spam-" so that
    >> people don't confuse it with SpamAssassin ...

    Neale> How about X-Spambayes-Disposition (or X-Hammie-Disposition if
    Neale> there'll be other classifier front-ends)?

I kinda like "hammie".  Your front end was there first, so I suspect it will
rule the front end roost.

S


From skip@pobox.com  Sat Sep  7 06:45:23 2002
From: skip@pobox.com (Skip Montanaro)
Date: Sat, 7 Sep 2002 00:45:23 -0500
Subject: [Spambayes] can't write to CVS...
In-Reply-To: <15737.37186.513740.936811@12-248-11-90.client.attbi.com>
References: <15737.35722.956784.600958@12-248-11-90.client.attbi.com>
        <LNBBLJKPBEHFEDALKOLCCELKBCAB.tim.one@comcast.net>
        <15737.37186.513740.936811@12-248-11-90.client.attbi.com>
Message-ID: <15737.37491.890945.752437@12-248-11-90.client.attbi.com>


    Skip> I'll try a checkout into a new directory...

Which didn't help.  At the very least I think that means it's time for
bed...

Skip


From skip@pobox.com  Sat Sep  7 06:46:41 2002
From: skip@pobox.com (Skip Montanaro)
Date: Sat, 7 Sep 2002 00:46:41 -0500
Subject: [Spambayes] Maybe change X-Spam-Disposition to something else...
In-Reply-To: <LNBBLJKPBEHFEDALKOLCAELLBCAB.tim.one@comcast.net>
References: <15737.33815.719456.367026@12-248-11-90.client.attbi.com>
        <LNBBLJKPBEHFEDALKOLCAELLBCAB.tim.one@comcast.net>
Message-ID: <15737.37569.889973.866955@12-248-11-90.client.attbi.com>


    >> Accordingly, I wrote unheader.py, which is mostly a ripoff of
    >> something someone else posted to python-dev or c.l.py within the last
    >> week or so to strip out SA-generated headers.

    Tim> Unless I've grown senile tonight, you got it from Anthony to begin
    Tim> with.  Please check it in to project, and add a short blurb to
    Tim> README.txt!

Raring to go, once I can write to CVS...

S

From neale@woozle.org  Sat Sep  7 06:48:17 2002
From: neale@woozle.org (Neale Pickett)
Date: 06 Sep 2002 22:48:17 -0700
Subject: [Spambayes] Ditching WordInfo
In-Reply-To: <200209070533.g875XN813509@pcp02138704pcs.reston01.va.comcast.net>
References: <LNBBLJKPBEHFEDALKOLCOEKKBCAB.tim.one@comcast.net>
	<w53n0qubcpj.fsf@woozle.org>
	<200209070533.g875XN813509@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <w537khybba6.fsf@woozle.org>

So then, Guido van Rossum <guido@python.org> is all like:

> Maybe.  I batch messages using fetchmail (don't ask why), and adding
> .4 seconds per message for a batch of 50 (not untypical) feels like a
> real wait to me...

Yeesh.  Sounds like what you need is something to kick up once and score
an entire mailbox.

Wait a second...  So *that's* why you wanted -u.

If you can spare the memory, you might get better performance in this
case using the pickle store, since it only has to go to disk once (but
boy, does it ever go to disk!)  I can't think of anything obvious to
speed things up once it's all loaded into memory, though.  That's
profiler territory, and profiling is exactly the kind of optimization
I just said I wasn't going to do :)

Neale

From sdrees@sdrees2.de  Sat Sep  7 07:07:11 2002
From: sdrees@sdrees2.de (Stefan Drees)
Date: Sat, 7 Sep 2002 08:07:11 +0200
Subject: [Spambayes] can't write to CVS...
In-Reply-To: <15737.35722.956784.600958@12-248-11-90.client.attbi.com>;
	from skip@pobox.com on Sat, Sep 07, 2002 at 12:15:54AM -0500
References: <15737.35722.956784.600958@12-248-11-90.client.attbi.com>
Message-ID: <20020907080711.A22966@sdrees2.de>

On Sat, Sep 07, 2002 at 12:15:54AM -0500 - a wonderful day - 
					Skip Montanaro wrote:
> I'm listed as a developer on SF and have the spambayes CVS 
> module checked out using my SF username, but I'm unable to 
> write to the repository.  CVS complains:
> 
>     % cvs add unheader.py
>     cvs [server aborted]: "add" requires write access to 
>			the repository
> 
> Any thoughts?
cvs -z3 -d:pserver:<user>@cvs.<project>.sourceforge.net:/cvsroot/<project> login

is ok, but for adding and such you have to ensure:
1) export CVS_RSH=ssh
2) cvs -z3 -d:ext:<user>@cvs.<project>.sourceforge.net:/cvsroot/<project> co <project>

change inside project/spam-or-ham

3) inside <project>/ things like cvs -z3 add unheader.py

"""
The cvs abort messages if forgetting 1) or using :pserver: instead of 
:ext: are irritating at best.

HTH.

All the best,
		s t e f a n.
-- 
Stefan Drees, sdrees@acm.org.

From guido@python.org  Sat Sep  7 07:06:31 2002
From: guido@python.org (Guido van Rossum)
Date: Sat, 07 Sep 2002 02:06:31 -0400
Subject: [Spambayes] Ditching WordInfo
In-Reply-To: Your message of "Fri, 06 Sep 2002 22:48:17 PDT."
             <w537khybba6.fsf@woozle.org> 
References: <LNBBLJKPBEHFEDALKOLCOEKKBCAB.tim.one@comcast.net>
	<w53n0qubcpj.fsf@woozle.org>
	<200209070533.g875XN813509@pcp02138704pcs.reston01.va.comcast.net>  
	<w537khybba6.fsf@woozle.org> 
Message-ID: <200209070606.g8766Vf13736@pcp02138704pcs.reston01.va.comcast.net>

> > Maybe.  I batch messages using fetchmail (don't ask why), and adding
> > .4 seconds per message for a batch of 50 (not untypical) feels like a
> > real wait to me...
> 
> Yeesh.  Sounds like what you need is something to kick up once and score
> an entire mailbox.
> 
> Wait a second...  So *that's* why you wanted -u.
> 
> If you can spare the memory, you might get better performance in this
> case using the pickle store, since it only has to go to disk once (but
> boy, does it ever go to disk!)  I can't think of anything obvious to
> speed things up once it's all loaded into memory, though.  That's
> profiler territory, and profiling is exactly the kind of optimization
> I just said I wasn't going to do :)

We could have a server mode (someone described this as an SA option).

--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.one@comcast.net  Sat Sep  7 17:07:27 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 07 Sep 2002 12:07:27 -0400
Subject: [Spambayes] Interesting
In-Reply-To: <200209070606.g8766Vf13736@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCAEMHBCAB.tim.one@comcast.net>

Just to see what would happen, I fiddled spamprob() to merely count the
number of times a word had prob above 0.5, and the number of times below.
Things exactly at 0.5 weren't counted.  The actual value of a prob was
ignored beyond what cmp(prob, 0.5) revealed.  MAX_DISCRIMINATORS is also
ignored here.

spamprob() then returned the high count divided by the sum of the high and
low counts (so if a quarter of all non-0.5 probs were greater than 0.5, 0.25
is the return value).  The virtue of all this is that the computation is
easy to picture.

Here's the aggregate distribution across 20 runs on 500-ham + 500-spam
pairs:

Ham distribution for all runs:
* = 27 items
  0.00    4 *
  2.50   30 **
  5.00  109 *****
  7.50  293 ***********
 10.00  603 ***********************
 12.50 1079 ****************************************
 15.00 1430 *****************************************************
 17.50 1490 ********************************************************
 20.00 1576 ***********************************************************
 22.50 1106 *****************************************
 25.00  872 *********************************
 27.50  533 ********************
 30.00  371 **************
 32.50  185 *******
 35.00  106 ****
 37.50   72 ***
 40.00   51 **
 42.50   32 **
 45.00   28 **
 47.50   11 *
 50.00    8 *
 52.50    4 *
 55.00    1 *
 57.50    1 *
 60.00    1 *
 62.50    0
 65.00    0
 67.50    0
 70.00    0
 72.50    0
 75.00    0
 77.50    0
 80.00    2 *
 82.50    0
 85.00    1 *
 87.50    0
 90.00    1 *
 92.50    0
 95.00    0
 97.50    0

Spam distribution for all runs:
* = 18 items
  0.00    0
  2.50    0
  5.00    0
  7.50    0
 10.00    0
 12.50    0
 15.00    0
 17.50    0
 20.00    0
 22.50    0
 25.00    0
 27.50    0
 30.00    3 *
 32.50    1 *
 35.00    0
 37.50    4 *
 40.00    4 *
 42.50    9 *
 45.00   28 **
 47.50   31 **
 50.00   94 ******
 52.50   77 *****
 55.00  112 *******
 57.50  184 ***********
 60.00  282 ****************
 62.50  336 *******************
 65.00  461 **************************
 67.50  568 ********************************
 70.00  643 ************************************
 72.50  619 ***********************************
 75.00  729 *****************************************
 77.50  755 ******************************************
 80.00  939 *****************************************************
 82.50 1035 **********************************************************
 85.00  909 ***************************************************
 87.50  803 *********************************************
 90.00  585 *********************************
 92.50  342 *******************
 95.00  277 ****************
 97.50  170 **********

Whatever this statistic measures <wink>, it sure looks like a normal
distribution for both ham and spam, that the ham distribution is tighter
(smaller variance), and that their means are 60 points apart.

Now those who have been following the checkins know that I've been
cautiously removing the biases built in to Graham's scoring scheme.  Last
night I officially got rid of MINCOUNT, and it could be that HAMBIAS=2.0 is
the only deliberate bias remaining.  Here's what happens if I repeat the
above after setting HAMBIAS to 1.0:

Ham distribution for all runs:
* = 22 items
  0.00    0
  2.50    0
  5.00    4 *
  7.50    3 *
 10.00    7 *
 12.50   27 **
 15.00   70 ****
 17.50  122 ******
 20.00  271 *************
 22.50  442 *********************
 25.00  722 *********************************
 27.50  893 *****************************************
 30.00 1229 ********************************************************
 32.50 1297 ***********************************************************
 35.00 1226 ********************************************************
 37.50 1135 ****************************************************
 40.00  916 ******************************************
 42.50  609 ****************************
 45.00  403 *******************
 47.50  235 ***********
 50.00  165 ********
 52.50   86 ****
 55.00   52 ***
 57.50   31 **
 60.00   27 **
 62.50   14 *
 65.00    4 *
 67.50    4 *
 70.00    2 *
 72.50    0
 75.00    0
 77.50    0
 80.00    1 *
 82.50    1 *
 85.00    1 *
 87.50    0
 90.00    1 *
 92.50    0
 95.00    0
 97.50    0

Spam distribution for all runs:
* = 30 items
  0.00    0
  2.50    0
  5.00    0
  7.50    0
 10.00    0
 12.50    0
 15.00    0
 17.50    0
 20.00    0
 22.50    0
 25.00    0
 27.50    0
 30.00    0
 32.50    0
 35.00    0
 37.50    0
 40.00    1 *
 42.50    0
 45.00    1 *
 47.50    0
 50.00   29 *
 52.50    4 *
 55.00   11 *
 57.50   13 *
 60.00   22 *
 62.50   43 **
 65.00   63 ***
 67.50   76 ***
 70.00  133 *****
 72.50  178 ******
 75.00  405 **************
 77.50  446 ***************
 80.00  674 ***********************
 82.50  894 ******************************
 85.00 1082 *************************************
 87.50 1359 **********************************************
 90.00 1775 ************************************************************
 92.50 1619 ******************************************************
 95.00  864 *****************************
 97.50  308 ***********

So without forced biases, it's the spam distribution that's tighter,
suggesting that spam really is more alike than ham is in this data.  Note
that one bizarre effect of HAMBIAS > 1.0 is that a word can get a spamprob <
0.5 even if it appears more often in spam than in ham.  Something about that
isn't right <wink>.


From jeremy@alum.mit.edu  Sat Sep  7 18:18:17 2002
From: jeremy@alum.mit.edu (Jeremy Hylton)
Date: Sat, 7 Sep 2002 13:18:17 -0400
Subject: [Spambayes] understanding high false negative rate
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCEKNBCAB.tim.one@comcast.net>
References: <15737.16782.542869.368986@slothrop.zope.com>
	<LNBBLJKPBEHFEDALKOLCCEKNBCAB.tim.one@comcast.net>
Message-ID: <15738.13529.407748.635725@slothrop.zope.com>

>>>>> "TP" == Tim Peters <tim.one@comcast.net> writes:

  TP> [Jeremy Hylton[
  >> The total collections are 1100 messages.  I trained with 1100/5
  >> messages.

  TP> I'm reading this now as that you trained on about 220 spam and
  TP> about 220 ham.  That's less than 10% of the sizes of the
  TP> training sets I've been using.  Please try an experiment: train
  TP> on 550 of each, and test once against the other 550 of each.

This helps a lot!  Here are results with the stock tokenizer:

Training on <mbox: /home/jeremy/Mail/INBOX 0> & <mbox: /home/jeremy/Mail/spam 8>
 ... 644 hams & 557 spams
      0.000  10.413
      1.398   6.104
      1.398   5.027
Training on <mbox: /home/jeremy/Mail/INBOX 0> & <mbox: /home/jeremy/Mail/spam 0>
 ... 644 hams & 557 spams
      0.000   8.259
      1.242   2.873
      1.242   5.745
Training on <mbox: /home/jeremy/Mail/INBOX 2> & <mbox: /home/jeremy/Mail/spam 3>
 ... 644 hams & 557 spams
      1.398   5.206
      1.398   4.488
      0.000   9.336
Training on <mbox: /home/jeremy/Mail/INBOX 2> & <mbox: /home/jeremy/Mail/spam 0>
 ... 644 hams & 557 spams
      1.553   5.206
      1.553   5.027
      0.000   9.874
total false pos 139 5.39596273292
total false neg 970 43.5368043088

And results from the tokenizer that looks at all headers except Date,
Received, and X-From_:

Training on <mbox: /home/jeremy/Mail/INBOX 0> & <mbox: /home/jeremy/Mail/spam 8>
 ... 644 hams & 557 spams
      0.000   7.540
      0.932   4.847
      0.932   3.232
Training on <mbox: /home/jeremy/Mail/INBOX 0> & <mbox: /home/jeremy/Mail/spam 0>
 ... 644 hams & 557 spams
      0.000   7.181
      0.621   2.873
      0.621   4.847
Training on <mbox: /home/jeremy/Mail/INBOX 2> & <mbox: /home/jeremy/Mail/spam 3>
 ... 644 hams & 557 spams
      1.087   4.129
      1.087   3.052
      0.000   6.822
Training on <mbox: /home/jeremy/Mail/INBOX 2> & <mbox: /home/jeremy/Mail/spam 0>
 ... 644 hams & 557 spams
      0.776   3.411
      0.776   3.411
      0.000   6.463
total false pos 97 3.76552795031
total false neg 738 33.1238779174

Is it safe to conclude that avoiding any cleverness with headers is a
good thing?

  TP> Do that a few times making a random split each time (it won't be
  TP> long until you discover why directories of individual files are
  TP> a lot easier to work -- e.g., random.shuffle() makes this kind
  TP> of thing trivial for me).

You haven't looked at mboxtest.py, have you?  I'm using
random.shuffle(), too.  You don't need to have many files to make an
arbitrary selection of messages from an mbox.

I'll report some more results when they're done.

Jeremy



From tim.one@comcast.net  Sat Sep  7 20:45:18 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 07 Sep 2002 15:45:18 -0400
Subject: [Spambayes] understanding high false negative rate
In-Reply-To: <15738.13529.407748.635725@slothrop.zope.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCOENBBCAB.tim.one@comcast.net>

>   TP> I'm reading this now as that you trained on about 220 spam and
>   TP> about 220 ham.  That's less than 10% of the sizes of the
>   TP> training sets I've been using.  Please try an experiment: train
>   TP> on 550 of each, and test once against the other 550 of each.

[Jeremy]
> This helps a lot!

Possibly.  I checked in a change to classifier.py overnight (getting rid of
MINCOUNT) that gave a major improvement in the f-n rate too, independent of
tokenization.

> Here are results with the stock tokenizer:

Unsure what "stock tokenizer" means to you.  For example, it might mean
tokenizer.tokenize, or mboxtest.MyTokenizer.tokenize.

> Training on <mbox: /home/jeremy/Mail/INBOX 0> & <mbox:
> /home/jeremy/Mail/spam 8>
>  ... 644 hams & 557 spams
>       0.000  10.413
>       1.398   6.104
>       1.398   5.027
> Training on <mbox: /home/jeremy/Mail/INBOX 0> & <mbox:
> /home/jeremy/Mail/spam 0>
>  ... 644 hams & 557 spams
>       0.000   8.259
>       1.242   2.873
>       1.242   5.745
> Training on <mbox: /home/jeremy/Mail/INBOX 2> & <mbox:
> /home/jeremy/Mail/spam 3>
>  ... 644 hams & 557 spams
>       1.398   5.206
>       1.398   4.488
>       0.000   9.336
> Training on <mbox: /home/jeremy/Mail/INBOX 2> & <mbox:
> /home/jeremy/Mail/spam 0>
>  ... 644 hams & 557 spams
>       1.553   5.206
>       1.553   5.027
>       0.000   9.874
> total false pos 139 5.39596273292
> total false neg 970 43.5368043088

Note that those rates remain much higher than I got using just 220 of ham
and 220 of spam.  That remains A Mystery.

> And results from the tokenizer that looks at all headers except Date,
> Received, and X-From_:

Unsure what that means too.  For example, "looks at" might mean you enabled
Anthony's count-them gimmick, and/or that you're tokenizing them yourself,
and/or ...?

> Training on <mbox: /home/jeremy/Mail/INBOX 0> & <mbox:
> /home/jeremy/Mail/spam 8>
>  ... 644 hams & 557 spams
>       0.000   7.540
>       0.932   4.847
>       0.932   3.232
> Training on <mbox: /home/jeremy/Mail/INBOX 0> & <mbox:
> /home/jeremy/Mail/spam 0>
>  ... 644 hams & 557 spams
>       0.000   7.181
>       0.621   2.873
>       0.621   4.847
> Training on <mbox: /home/jeremy/Mail/INBOX 2> & <mbox:
> /home/jeremy/Mail/spam 3>
>  ... 644 hams & 557 spams
>       1.087   4.129
>       1.087   3.052
>       0.000   6.822
> Training on <mbox: /home/jeremy/Mail/INBOX 2> & <mbox:
> /home/jeremy/Mail/spam 0>
>  ... 644 hams & 557 spams
>       0.776   3.411
>       0.776   3.411
>       0.000   6.463
> total false pos 97 3.76552795031
> total false neg 738 33.1238779174
>
> Is it safe to conclude that avoiding any cleverness with headers is a
> good thing?

Since I don't know what you did, exactly, I can't guess.  What you seemed to
show is that you did *something* clever with headers and that doing so
helped (the "after" numbers are better than the "before" numbers, right?).
Assuming that what you did was override what's now
tokenizer.Tokenizer.tokenize_headers with some other routine, and didn't
call the base Tokenizer.tokenize_headers at all, then you're missing
carefully tested treatment of just a few header fields, but adding many
dozens of other header fields.  There's no question that adding more header
fields should help; tokenizer.Tokenizer.tokenize_headers doesn't do so only
because my testing corpora are such that I can't add more headers without
getting major benefits for bogus reasons.

Apart from all that, you said you're skipping Received.  By several
accounts, that may be the most valuable of all the header fields.  I'm
(meaning tokenizer.Tokenizer.tokenize_headers) skipping them too for the
reason explained above.  Offline a week or two ago, Neil Schemenauer
reported good results from this scheme:

    ip_re = re.compile(r'(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})')

    for header in msg.get_all("received", ()):
        for ip in ip_re.findall(header):
            parts = ip.split(".")
            for n in range(1, 5):
                yield 'received:' + '.'.join(parts[:n])

This makes a lot of sense to me; I just checked it in, but left it disabled
for now.


From tim.one@comcast.net  Sat Sep  7 21:11:36 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 07 Sep 2002 16:11:36 -0400
Subject: [Spambayes] test sets?
In-Reply-To: <200209061424.g86EOcd14363@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCOENCBCAB.tim.one@comcast.net>

[Guido]
> Perhaps more useful would be if Tim could check in the pickle(s?)
> generated by one of his training runs, so that others can see how
> Tim's training data performs against their own corpora.

I did that yesterday, but seems like nobody bit.  Just in case <wink>, I
uploaded a new version just now.  Since MINCOUNT went away, UNKNOWN_SPAMPROB
is much less likely, and there's almost nothing that can be pruned away (so
the file is about 5x larger now).

    http://sf.net/project/showfiles.php?group_id=61702


> This could also be the starting point for a self-contained distribution
> (you've got to start with *something*, and training with python-list data
> seems just as good as anything else).

The only way to know anything here is for someone to try it.


From jeremy@alum.mit.edu  Sat Sep  7 21:15:03 2002
From: jeremy@alum.mit.edu (Jeremy Hylton)
Date: Sat, 7 Sep 2002 16:15:03 -0400
Subject: [Spambayes] understanding high false negative rate
In-Reply-To: <LNBBLJKPBEHFEDALKOLCOENBBCAB.tim.one@comcast.net>
References: <15738.13529.407748.635725@slothrop.zope.com>
	<LNBBLJKPBEHFEDALKOLCOENBBCAB.tim.one@comcast.net>
Message-ID: <15738.24135.294137.640570@slothrop.zope.com>

Here's clarification of why I did:

First test results using tokenizer.Tokenizer.tokenize_headers()
unmodified. 

Training on 644 hams & 557 spams
      0.000  10.413
      1.398   6.104
      1.398   5.027
Training on 644 hams & 557 spams
      0.000   8.259
      1.242   2.873
      1.242   5.745
Training on 644 hams & 557 spams
      1.398   5.206
      1.398   4.488
      0.000   9.336
Training on 644 hams & 557 spams
      1.553   5.206
      1.553   5.027
      0.000   9.874
total false pos 139 5.39596273292
total false neg 970 43.5368043088

Second test results using mboxtest.MyTokenizer.tokenize_headers().
This uses all headers except Received, Data, and X-From_.

Training on 644 hams & 557 spams
      0.000   7.540
      0.932   4.847
      0.932   3.232
Training on 644 hams & 557 spams
      0.000   7.181
      0.621   2.873
      0.621   4.847
Training on 644 hams & 557 spams
      1.087   4.129
      1.087   3.052
      0.000   6.822
Training on 644 hams & 557 spams
      0.776   3.411
      0.776   3.411
      0.000   6.463
total false pos 97 3.76552795031
total false neg 738 33.1238779174

Jeremy


From guido@python.org  Sat Sep  7 21:19:16 2002
From: guido@python.org (Guido van Rossum)
Date: Sat, 07 Sep 2002 16:19:16 -0400
Subject: [Spambayes] test sets?
In-Reply-To: Your message of "Sat, 07 Sep 2002 16:11:36 EDT."
             <LNBBLJKPBEHFEDALKOLCOENCBCAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCOENCBCAB.tim.one@comcast.net> 
Message-ID: <200209072019.g87KJGe15250@pcp02138704pcs.reston01.va.comcast.net>

> [Guido]
> > Perhaps more useful would be if Tim could check in the pickle(s?)
> > generated by one of his training runs, so that others can see how
> > Tim's training data performs against their own corpora.

[Tim]
> I did that yesterday, but seems like nobody bit.

I downloaded and played with it a bit, but had no time to do anything
systematic.  It correctly recognized a spam that slipped through SA.
But it also identified as spam everything in my inbox that had any
MIME structure or HTML parts, and several messages in my saved 'zope
geeks' list that happened to be using MIME and/or HTML.

So I guess I'll have to retrain it (yes, you told me so :-).

> Just in case <wink>, I
> uploaded a new version just now.  Since MINCOUNT went away, UNKNOWN_SPAMPROB
> is much less likely, and there's almost nothing that can be pruned away (so
> the file is about 5x larger now).
> 
>     http://sf.net/project/showfiles.php?group_id=61702

I'll try this when I have time.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From skip@pobox.com  Sun Sep  8 00:11:19 2002
From: skip@pobox.com (Skip Montanaro)
Date: Sat, 7 Sep 2002 18:11:19 -0500
Subject: [Spambayes] spambayes package?
Message-ID: <15738.34711.467756.145336@12-248-11-90.client.attbi.com>

Before we get too far down this road, what do people think of creating a
spambayes package containing classifier and tokenizer?  This is just to
minimize clutter in site-packages.

Skip

From tim.one@comcast.net  Sun Sep  8 00:22:18 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 07 Sep 2002 19:22:18 -0400
Subject: [Spambayes] understanding high false negative rate
In-Reply-To: <15738.24135.294137.640570@slothrop.zope.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCCENKBCAB.tim.one@comcast.net>

[Jeremy Hylton]
> Here's clarification of why I did:

That's pretty much what I had guessed.  Thanks!  One more to try:

> First test results using tokenizer.Tokenizer.tokenize_headers()
> unmodified. 
> ...
> Second test results using mboxtest.MyTokenizer.tokenize_headers().
> This uses all headers except Received, Data, and X-From_.
> ...

Try the latter again, but call the base tokenize_headers() too.

From guido@python.org  Sun Sep  8 03:47:23 2002
From: guido@python.org (Guido van Rossum)
Date: Sat, 07 Sep 2002 22:47:23 -0400
Subject: [Spambayes] spambayes package?
In-Reply-To: Your message of "Sat, 07 Sep 2002 18:11:19 CDT."
             <15738.34711.467756.145336@12-248-11-90.client.attbi.com> 
References: <15738.34711.467756.145336@12-248-11-90.client.attbi.com> 
Message-ID: <200209080247.g882lNt17033@pcp02138704pcs.reston01.va.comcast.net>

> Before we get too far down this road, what do people think of creating a
> spambayes package containing classifier and tokenizer?  This is just to
> minimize clutter in site-packages.

Too early IMO (if you mean to leave the various other tools out of
it).  If and when we package this, perhaps we should use Barry's trick
from the email package for making the package itself the toplevel dir
of the distribution (rather than requiring an extra directory level
just so the package can be a subdir of the distro).

--Guido van Rossum (home page: http://www.python.org/~guido/)


From tim.one@comcast.net  Sun Sep  8 03:55:12 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 07 Sep 2002 22:55:12 -0400
Subject: [Spambayes] test sets?
In-Reply-To: <200209072019.g87KJGe15250@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCOEOBBCAB.tim.one@comcast.net>

[Guido, on the classifier pickle on SF]
> I downloaded and played with it a bit, but had no time to do anything
> systematic.

Cool!

> It correctly recognized a spam that slipped through SA.

Ditto.

> But it also identified as spam everything in my inbox that had any
> MIME structure or HTML parts, and several messages in my saved 'zope
> geeks' list that happened to be using MIME and/or HTML.

Do you know why?  The strangest implied claim there is that it hates MIME
independent of HTML.  For example, the spamprob of 'content-type:text/plain'
in that pickle is under 0.21.  'content-type:multipart/alternative' gets
0.93, but that's not a killer clue, and one bit of good content will more
than cancel it out.

WRT hating HTML, possibilities include:

1. It really had to do with something other than MIME/HTML.

2. These are pure HTML (not multipart/alternative with a text/plain part),
   so that the tags aren't getting stripped.  The pickled classifier
   despises all hints of HTML due to its c.l.py heritage.

3. These are multipart/alternative with a text/plain part, but the
   latter doesn't contain the same text as the text/html part (for
   example, as Anthony reported, perhaps the text/plain part just
   says something like "This is an HMTL message.").

If it's #2, it would be easy to add an optional bool argument to tokenize()
meaning "even if it is pure HTML, strip the tags anyway".  In fact, I'd like
to do that and default it to True.  The extreme hatred of HTML on tech lists
strikes me as, umm, extreme <wink>.

> So I guess I'll have to retrain it (yes, you told me so :-).

That would be a different experiment.  I'm certainly curious to see whether
Jeremy's much-worse-than-mine error rates are typical or aberrant.


From guido@python.org  Sun Sep  8 04:38:47 2002
From: guido@python.org (Guido van Rossum)
Date: Sat, 07 Sep 2002 23:38:47 -0400
Subject: [Spambayes] test sets?
In-Reply-To: Your message of "Sat, 07 Sep 2002 22:55:12 EDT."
             <LNBBLJKPBEHFEDALKOLCOEOBBCAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCOEOBBCAB.tim.one@comcast.net> 
Message-ID: <200209080338.g883clw17223@pcp02138704pcs.reston01.va.comcast.net>

> > But it also identified as spam everything in my inbox that had any
> > MIME structure or HTML parts, and several messages in my saved 'zope
> > geeks' list that happened to be using MIME and/or HTML.
> 
> Do you know why?  The strangest implied claim there is that it hates MIME
> independent of HTML.  For example, the spamprob of 'content-type:text/plain'
> in that pickle is under 0.21.  'content-type:multipart/alternative' gets
> 0.93, but that's not a killer clue, and one bit of good content will more
> than cancel it out.

I reran the experiment (with the new SpamHam1.pik, but it doesn't seem
to make a difference).  Here are the clues for the two spams in my
inbox (in hammie.py's output format, which sorts the clues by
probability; the first two numbers are the message number and overall
probability; then line-folded):

    66 1.00 S 'facility': 0.01; 'speaker': 0.01; 'stretch': 0.01;
    'thursday': 0.01; 'young,': 0.01; 'mistakes': 0.12; 'growth':
    0.85; '>content-type:text/plain': 0.85; 'please': 0.85; 'capital':
    0.92; 'series': 0.92; 'subject:Don': 0.94; 'companies': 0.96;
    '>content-type:text/html': 0.96; 'fee': 0.96; 'money': 0.96;
    '8:00am': 0.99; '9:00am': 0.99; '>content-type:image/gif': 0.99;
    '>content-type:multipart/alternative': 0.99; 'attend': 0.99;
    'companies,': 0.99; 'content-type/type:multipart/alternative':
    0.99; 'content-type:multipart/related': 0.99; 'economy': 0.99;
    'economy"': 0.99

This has 6 content-types as spam clues, only one of which is related
to HTML, despite there being an HTML alternative (and 12 other spam
clues, vs. only 6 ham clues).  This was an announcement of a public
event by our building owners, with a text part that was the same as
the HTML (AFAICT).  Its language may be spammish, but the content-type
clues didn't help.  (BTW, it makes me wonder about the wisdom of
keeping punctuation -- 'economy' and 'economy"' to me don't seem to
deserve two be counted as clues.)

    76 1.00 S '(near': 0.01; 'alexandria': 0.01; 'conn': 0.01;
    'from:adam': 0.01; 'from:email addr:panix': 0.01; 'poked': 0.01;
    'thorugh': 0.01; 'though': 0.03; "i'm": 0.03; 'reflect': 0.05;
    "i've": 0.06; 'wednesday': 0.07; 'content-disposition:inline':
    0.10; 'contacting': 0.93; 'sold': 0.96; 'financially': 0.98;
    'prices': 0.98; 'rates': 0.99; 'discount.': 0.99; 'hotel': 0.99;
    'hotels': 0.99; 'hotels.': 0.99; 'nights,': 0.99; 'plaza': 0.99;
    'rates,': 0.99; 'rates.': 0.99; 'rooms': 0.99; 'season': 0.99;
    'stations': 0.99; 'subject:Hotel': 0.99

Here is the full message (Received: headers stripped), with apologies
to Ziggy and David:

"""
Date: Fri, 06 Sep 2002 17:17:13 -0400
From: Adam Turoff <ziggy@panix.com>
Subject: Hotel information
To: guido@python.org, davida@activestate.com
Message-id: <20020906211713.GK7451@panix.com>
MIME-version: 1.0
Content-type: text/plain; charset=us-ascii
Content-disposition: inline
User-Agent: Mutt/1.4i

I've been looking into hotels.  I poked around expedia for availability
from March 26 to 29 (4 nights, wednesday thorugh saturday).  

I've also started contacting hotels for group rates; some of the group
rates are no better than the regular rates, and they require signing a
contract with a minimum number of rooms sold (with someone financially
responsible for unbooked rooms).  Most hotels are less than responsive...

	Radission - Barcelo Hotel (DuPont Circle)
	$125/night, $99/weekend

	State Plaza hotel (Foggy Bottom; near GWU)
	$119/night, $99/weekend

	Hilton Silver Spring (Near Metro, in suburban MD)
	$99/hight, $74/weekend

	Windsor Park Hotel
	Conn Ave, between DuPont Circle/Woodley Park Metro stations
	$95/night; needs a car

	Econo Lodge Alexandria (Near Metro, in suburban VA)
	$95/night

This is a hand picked list; I ignored anything over $125/night, even
though there are some really well situated hotels nearby at higher rates.
Also, I'm not sure how much these prices reflect an expedia-only
discount.  I can't vouch for any of these hotels, either.

I also found out that the down season for DC Hotels are mid-june through
mid-september, and mid-november through mid-january.

Z.
"""

This one has no MIME structure nor HTML!  It even has a
Content-disposition which is counted as a non-spam clue.  It got
f-p'ed because of the many hospitality-related and money-related
terms.  I'm surprised $125/night and similar aren't clues too.  (And
again, several spam clues are duplicated with different variations:
'hotel', 'hotels', 'hotels.', 'subject:Hotel', 'rates,', 'rates.'.

> WRT hating HTML, possibilities include:
> 
> 1. It really had to do with something other than MIME/HTML.
> 
> 2. These are pure HTML (not multipart/alternative with a text/plain part),
>    so that the tags aren't getting stripped.  The pickled classifier
>    despises all hints of HTML due to its c.l.py heritage.
> 
> 3. These are multipart/alternative with a text/plain part, but the
>    latter doesn't contain the same text as the text/html part (for
>    example, as Anthony reported, perhaps the text/plain part just
>    says something like "This is an HMTL message.").
> 
> If it's #2, it would be easy to add an optional bool argument to tokenize()
> meaning "even if it is pure HTML, strip the tags anyway".  In fact, I'd like
> to do that and default it to True.  The extreme hatred of HTML on tech lists
> strikes me as, umm, extreme <wink>.

I also looked in more detail at some f-p's in my geeks traffic.  The
first one's a doozie (that's the term, right? :-).  It has lots of
HTML clues that are apparently ignored.  It was a multipart/mixed with
two parts: a brief text/plain part containing one or two sentences, a
mondo weird URL:

http://x60.deja.com/[ST_rn=ps]/getdoc.xp?AN=687715863&CONTEXT=973121507.1408827441&hitnum=23

and some employer-generated spammish boilerplate; the second part was
the HTML taken directly from the above URL.  Clues:

    43 1.00 S '"main"': 0.01; '(later': 0.01; '(lots': 0.01; '--paul':
    0.01; '1995-2000': 0.01; 'adopt': 0.01; 'apps': 0.01; 'commands':
    0.01; 'deja.com': 0.01; 'dejanews,': 0.01; 'discipline': 0.01;
    'duct': 0.01; 'email addr:digicool': 0.01; 'email name:paul':
    0.01; 'everitt': 0.01; 'exist,': 0.01; 'forwards': 0.01;
    'framework': 0.01; 'from:email addr:digicool': 0.01; 'from:email
    name:<paul': 0.01; 'from:paul': 0.01; 'height': 0.01;
    'hodge-podge': 0.01; 'http0:deja': 0.01; 'http0:zope': 0.01;
    'http1:[st_rn': 0.01; 'http1:comp': 0.01; 'http1:getdoc': 0.01;
    'http1:ps]': 0.01; 'http>1:22': 0.01; 'http>1:24': 0.01;
    'http>1:57': 0.01; 'http>1:an': 0.01; 'http>1:author': 0.01;
    'http>1:fmt': 0.01; 'http>1:getdoc': 0.01; 'http>1:pr': 0.01;
    'http>1:products': 0.01; 'http>1:query': 0.01; 'http>1:search':
    0.01; 'http>1:viewthread': 0.01; 'http>1:xp': 0.01; 'http>1:zope':
    0.01; 'inventing': 0.01; 'jsp': 0.01; 'jsp.': 0.01; 'logic': 0.01;
    'maps': 0.01; 'neo': 0.01; 'newsgroup,': 0.01; 'object': 0.01;
    'popup': 0.01; 'probable': 0.01; 'query': 0.01; 'query,': 0.01;
    'resizes': 0.01; 'servlet': 0.01; 'skip:? 20': 0.01; 'stems':
    0.01; 'subject:JSP': 0.01; 'sucks!': 0.01; 'templating': 0.01;
    'tempted': 0.01; 'url.': 0.01; 'usenet': 0.01; 'usenet,': 0.01;
    'wrote': 0.01; 'x-mailer:mozilla 4.74 [en] (windows nt 5.0; u)':
    0.01; 'zope': 0.01; '#000000;': 0.99; '#cc0000;': 0.99;
    '#ff3300;': 0.99; '#ff6600;': 0.99; '#ffffff;': 0.99; '&copy;':
    0.99; '&gt;': 0.99; '&nbsp;&nbsp;': 0.99; '&quot;no': 0.99;
    '.med': 0.99; '.small': 0.99; '0pt;': 0.99; '0px;': 0.99; '10px;':
    0.99; '11pt;': 0.99; '12px;': 0.99; '18pt;': 0.99; '18px;': 0.99;
    '1pt;': 0.99; '2px;': 0.99; '640;': 0.99; '8pt;': 0.99; '<!--':
    0.99; '</b>': 0.99; '</body>': 0.99; '</head>': 0.99; '</html>':
    0.99; '</script>': 0.99; '</select>': 0.99; '</span>': 0.99;
    '</style>': 0.99; '</table>': 0.99; '</td>': 0.99; '</td></tr>':
    0.99; '</tr>': 0.99; '</tr><tr': 0.99; '<b><a': 0.99; '<base':
    0.99; '<body': 0.99; '<br>': 0.99; '<br>&nbsp;': 0.99; '<br><a':
    0.99; '<br><span': 0.99; '<font': 0.99; '<form': 0.99; '<head>':
    0.99; '<html>': 0.99; '<img': 0.99; '<input': 0.99; '<meta': 0.99;
    '<option': 0.99; '<p>': 0.99; '<p>a': 0.99; '<script>': 0.99;
    '<select': 0.99; '<span': 0.99; '<style>': 0.99; '<table': 0.99;
    '<td': 0.99; '<td>': 0.99; '<td></td>': 0.99; '<td><img': 0.99;
    '<tr': 0.99; '<tr>': 0.99; '<tr><td': 0.99; '<tr><td><img': 0.99;
    'absolute;': 0.99; 'align="left"': 0.99; 'align=center': 0.99;
    'align=left': 0.99; 'align=middle': 0.99; 'align=right': 0.99;
    'align=right>': 0.99; 'alt=""': 0.99; 'bold;': 0.99; 'border=0':
    0.99; 'border=0>': 0.99; 'color:': 0.99; 'colspan=2': 0.99;
    'colspan=2>': 0.99; 'colspan=4': 0.99; 'face="arial"': 0.99;
    'font-family:': 0.99; 'font-size:': 0.99; 'font-weight:': 0.99;
    'footer': 0.99; 'for<br>': 0.99; 'fucking<br>': 0.99;
    'height="1"': 0.99; 'height="16"': 0.99; 'height=1': 0.99;
    'height=12': 0.99; 'height=125': 0.99; 'height=17': 0.99;
    'height=18': 0.99; 'height=21': 0.99; 'height=4': 0.99;
    'height=57': 0.99; 'height=60': 0.99; 'height=8': 0.99;
    'hspace=0': 0.99; 'http0:g': 0.99; 'http0:web2': 0.99; 'http1:0':
    0.99; 'http1:ads': 0.99; 'http1:d': 0.99; 'http1:page': 0.99;
    'http1:site': 0.99; 'http>1:article': 0.99; 'http>1:back': 0.99;
    'http>1:com': 0.99; 'http>1:d': 0.99; 'http>1:gif': 0.99;
    'http>1:go': 0.99; 'http>1:group': 0.99; 'http>1:http': 0.99;
    'http>1:post': 0.99; 'http>1:ps': 0.99; 'http>1:site': 0.99;
    'http>1:st': 0.99; 'http>1:title': 0.99; 'http>1:yahoo': 0.99;
    'inc.</a>': 0.99; 'jobs!': 0.99; 'normal;': 0.99; 'nowrap': 0.99;
    'nowrap>': 0.99; 'nowrap><font': 0.99; 'padding:': 0.99;
    'rowspan=2': 0.99; 'rowspan=3': 0.99; 'servlets,': 0.99;
    'size=15': 0.99; 'size=35': 0.99; 'skip:< 10': 0.99; 'skip:b 60':
    0.99; 'skip:h 110': 0.99; 'skip:h 170': 0.99; 'skip:h 200': 0.99;
    'skip:h 240': 0.99; 'skip:h 250': 0.99; 'skip:h 290': 0.99;
    'skip:v 40': 0.99; 'solid;': 0.99; 'text=#000000': 0.99; 'to<br>':
    0.99; 'type="image"': 0.99; 'type="text"': 0.99; 'type=hidden':
    0.99; 'type=image': 0.99; 'type=radio': 0.99; 'type=submit': 0.99;
    'type=text': 0.99; 'valign=top': 0.99; 'valign=top>': 0.99;
    'value="">': 0.99; 'visibility:': 0.99; 'width:': 0.99;
    'width="33"': 0.99; 'width=1': 0.99; 'width=100%': 0.99;
    'width=100%>': 0.99; 'width=12': 0.99; 'width=125': 0.99;
    'width=130': 0.99; 'width=137': 0.99; 'width=2': 0.99; 'width=20':
    0.99; 'width=25': 0.99; 'width=4': 0.99; 'width=468': 0.99;
    'width=6': 0.99; 'width=72': 0.99; 'works<br>': 0.99

The second f-p had the same structure (and sender :-); the third f-p
had the same structure and a different sender.  Ditto the fifth, sixth.  (Not posting clues for
brevity.)

The fourth was different: plaintext with one very short sentence and a
URL.  Clues:

   300 1.00 S 'from:email addr:digicool': 0.01; 'http1:news': 0.24;
   'from:email addr:com>': 0.32; 'from:tres': 0.50; 'http>1:1114digi':
   0.50; 'proto:http': 0.50; 'subject:Geeks': 0.50; 'x-mailer:mozilla
   4.75 [en] (x11; u; linux 2.2.14-5.0smp i686)': 0.50; 'take': 0.54;
   'bool:noorg': 0.61; 'http0:com': 0.66; 'skip:h 50': 0.83;
   'http>1:htm': 0.90; 'subject:Software': 0.96; 'http>1:business':
   0.99; 'http>1:local': 0.99; 'subject:firm': 0.99; 'us:': 0.99

The seventh was similar.

I scanned a bunch more until I got bored, and most of them were either
of the first form (brief text with URL followed by quoted HTML from
website) or the second (brief text with one or more URLs).

It's up to you to decide what to call this, but I think these are none
of your #1, #2 or #3 (they're close to #3, but all are multipart/mixed
rather than multipart/alternative).

> > So I guess I'll have to retrain it (yes, you told me so :-).
> 
> That would be a different experiment.  I'm certainly curious to see whether
> Jeremy's much-worse-than-mine error rates are typical or aberrant.

It's possible that the corpus you've trained on is more homogeneous
than you thought.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.one@comcast.net  Sun Sep  8 06:02:32 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 08 Sep 2002 01:02:32 -0400
Subject: [Spambayes] test sets?
In-Reply-To: <200209080338.g883clw17223@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCEEOGBCAB.tim.one@comcast.net>

[Guido]
> I reran the experiment

Thanks!

> (with the new SpamHam1.pik, but it doesn't seem to make a difference).

I can believe that.  One effect of getting rid of MINCOUNT is that it
latches on more strongly to rare clues now, and those can be unique to the
corpus trained on (e.g., one trained ham says "gryndlplyx!", and a followup
new ham quotes it).

> Here are the clues for the two spams in my inbox (in hammie.py's output
> format, which sorts the clues by probability; the first two numbers are
the
> message number and overall probability; then line-folded):
>
>     66 1.00 S 'facility': 0.01; 'speaker': 0.01; 'stretch': 0.01;
>     'thursday': 0.01; 'young,': 0.01; 'mistakes': 0.12; 'growth':
>     0.85; '>content-type:text/plain': 0.85; 'please': 0.85; 'capital':
>     0.92; 'series': 0.92; 'subject:Don': 0.94; 'companies': 0.96;
>     '>content-type:text/html': 0.96; 'fee': 0.96; 'money': 0.96;
>     '8:00am': 0.99; '9:00am': 0.99; '>content-type:image/gif': 0.99;
>     '>content-type:multipart/alternative': 0.99; 'attend': 0.99;
>     'companies,': 0.99; 'content-type/type:multipart/alternative':
>     0.99; 'content-type:multipart/related': 0.99; 'economy': 0.99;
>     'economy"': 0.99
>
> This has 6 content-types as spam clues, only one of which is related
> to HTML, despite there being an HTML alternative (and 12 other spam
> clues, vs. only 6 ham clues).

Gotcha.  I'm afraid they're *correlated* with HTML in my corpora.  You also
got hit because this email attached a gif, and it may be that no legit
c.l.py post in history has done so.

> This was an announcement of a public event by our building owners, with a
> text part that was the same as the HTML (AFAICT).

Naturally, *my* corpora has no clues about our building owners, so you're
likely missing good ham clues (like the name of our building, etc) that
would get into your database once you trained on this.

> Its language may be spammish, but the content-type clues didn't help.

More, they hurt <wink>.

> (BTW, it makes me wonder about the wisdom of keeping punctuation --
'economy'
> and 'economy"' to me don't seem to deserve two be counted as clues.)

Wisdom has nothing to do with it -- it's purely a matter of trying it both
ways and looking at the resulting error rates.  This particular decision is
thoroughly discussed in the "How to tokenize?" comment block in
tokenizer.py, including a full account of both-way error rates and the exact
"ignore punctuation too" tokenization scheme used for comparison.

That doesn't mean I'm using the best possible tokenization scheme, but does
mean the one I'm using decisively beat the specific "search for alphanumeric
runs" scheme spelled out there, and which beat any number of other
tokenization schemes (including word bigrams) I tried that aren't discussed
in the comments.  If you have another specific tokenization scheme in mind,
I'll be happy to run a similar test on it (the only caution is that I can't
test "an idea", I can only test concrete code <wink>).

>     76 1.00 S '(near': 0.01; 'alexandria': 0.01; 'conn': 0.01;
>     'from:adam': 0.01; 'from:email addr:panix': 0.01; 'poked': 0.01;
>     'thorugh': 0.01; 'though': 0.03; "i'm": 0.03; 'reflect': 0.05;
>     "i've": 0.06; 'wednesday': 0.07; 'content-disposition:inline':
>     0.10; 'contacting': 0.93; 'sold': 0.96; 'financially': 0.98;
>     'prices': 0.98; 'rates': 0.99; 'discount.': 0.99; 'hotel': 0.99;
>     'hotels': 0.99; 'hotels.': 0.99; 'nights,': 0.99; 'plaza': 0.99;
>     'rates,': 0.99; 'rates.': 0.99; 'rooms': 0.99; 'season': 0.99;
>     'stations': 0.99; 'subject:Hotel': 0.99
>
> Here is the full message (Received: headers stripped), with apologies
> to Ziggy and David:

Heh -- just from glancing at the clues it sure looks like spam to me <wink>.

> """
> Date: Fri, 06 Sep 2002 17:17:13 -0400
> From: Adam Turoff <ziggy@panix.com>
> Subject: Hotel information
> To: guido@python.org, davida@activestate.com
> Message-id: <20020906211713.GK7451@panix.com>
> MIME-version: 1.0
> Content-type: text/plain; charset=us-ascii
> Content-disposition: inline
> User-Agent: Mutt/1.4i
>
> I've been looking into hotels.  I poked around expedia for availability
> from March 26 to 29 (4 nights, wednesday thorugh saturday).
>
> I've also started contacting hotels for group rates; some of the group
> rates are no better than the regular rates, and they require signing a
> contract with a minimum number of rooms sold (with someone financially
> responsible for unbooked rooms).  Most hotels are less than responsive...
>
> 	Radission - Barcelo Hotel (DuPont Circle)
> 	$125/night, $99/weekend
>
> 	State Plaza hotel (Foggy Bottom; near GWU)
> 	$119/night, $99/weekend
>
> 	Hilton Silver Spring (Near Metro, in suburban MD)
> 	$99/hight, $74/weekend
>
> 	Windsor Park Hotel
> 	Conn Ave, between DuPont Circle/Woodley Park Metro stations
> 	$95/night; needs a car
>
> 	Econo Lodge Alexandria (Near Metro, in suburban VA)
> 	$95/night
>
> This is a hand picked list; I ignored anything over $125/night, even
> though there are some really well situated hotels nearby at higher rates.
> Also, I'm not sure how much these prices reflect an expedia-only
> discount.  I can't vouch for any of these hotels, either.
>
> I also found out that the down season for DC Hotels are mid-june through
> mid-september, and mid-november through mid-january.
>
> Z.
> """
>
> This one has no MIME structure nor HTML!  It even has a
> Content-disposition which is counted as a non-spam clue.  It got
> f-p'ed because of the many hospitality-related and money-related
> terms.

Sure -- such things are rarely discussed on c.l.py, but often appear in
spam.  The system is very good at learning what it's taught!  BTW, one of
the remaining fps in my test data is a c.l.py msg from someone looking to
save money by sharing a conference hotel room.  It would have saved itself
had it mentioned it was a Python conference, but it didn't.

> I'm surprised $125/night and similar aren't clues too.

They don't appear anywhere in legit c.l.py traffic.  Phrases like
'$1,500+/wk' have 0.99 spamprob, though.

> (And again, several spam clues are duplicated with different variations:
> 'hotel', 'hotels', 'hotels.', 'subject:Hotel', 'rates,', 'rates.'.

It definitely helps to keep the Subject line words separate from the rest,
so I don't count 'subject:Hotel' as a variation:  it's a very distinct form
of clue from a variant of 'hotel' appearing in the body.  Within the body,
research reports say that lemmatization helps classic Bayesian classifiers
("lemmatization" is what stemming is called in the literature).  But then
you need a lemmatizer too, and that requires a hell of a lot of
language-specific knowledge.  It's also highly open to question whether it
would help or hurt this "smoking guns only" scoring gimmick.

BTW, if you were looking at correctly classified msgs too, you'd likely find
that "pointless variations" are also helping your marginal ham a lot.  For
example, if you've got "Python?" and "Python" in a msg now, you get two
strong brownie points; but you'd only get one if punctuation were stripped.

> I also looked in more detail at some f-p's in my geeks traffic.  The
> first one's a doozie (that's the term, right? :-).  It has lots of
> HTML clues that are apparently ignored.

?? The clues below are *loaded* with snippets unique to HTML (like '<br>').


> It was a multipart/mixed with two parts: a brief text/plain part
containing
> one or two sentences, a mondo weird URL:
>
>
http://x60.deja.com/[ST_rn=ps]/getdoc.xp?AN=687715863&CONTEXT=973121507.1408
827441&hitnum=23
>
> and some employer-generated spammish boilerplate; the second part was
> the HTML taken directly from the above URL.

What type was this part?  text/html?  If it was text/plain, the tokenizer
should have stripped all the '<br>' thingies.  Not enough info here to tell
whether there's a tokenizer bug in stripping such things.  If it was
text/html, then the previously mentioned tokenizer optional "strip tags from
HTML too" arg would deal with this.

>  Clues:
>
>     43 1.00 S '"main"': 0.01; '(later': 0.01; '(lots': 0.01; '--paul':
>     0.01; '1995-2000': 0.01; 'adopt': 0.01; 'apps': 0.01; 'commands':
>     0.01; 'deja.com': 0.01; 'dejanews,': 0.01; 'discipline': 0.01;
>     'duct': 0.01; 'email addr:digicool': 0.01; 'email name:paul':
>     0.01; 'everitt': 0.01; 'exist,': 0.01; 'forwards': 0.01;
>     'framework': 0.01; 'from:email addr:digicool': 0.01; 'from:email
>     name:<paul': 0.01; 'from:paul': 0.01; 'height': 0.01;
>     'hodge-podge': 0.01; 'http0:deja': 0.01; 'http0:zope': 0.01;
>     'http1:[st_rn': 0.01; 'http1:comp': 0.01; 'http1:getdoc': 0.01;
>     'http1:ps]': 0.01; 'http>1:22': 0.01; 'http>1:24': 0.01;
>     'http>1:57': 0.01; 'http>1:an': 0.01; 'http>1:author': 0.01;
>     'http>1:fmt': 0.01; 'http>1:getdoc': 0.01; 'http>1:pr': 0.01;
>     'http>1:products': 0.01; 'http>1:query': 0.01; 'http>1:search':
>     0.01; 'http>1:viewthread': 0.01; 'http>1:xp': 0.01; 'http>1:zope':
>     0.01; 'inventing': 0.01; 'jsp': 0.01; 'jsp.': 0.01; 'logic': 0.01;
>     'maps': 0.01; 'neo': 0.01; 'newsgroup,': 0.01; 'object': 0.01;
>     'popup': 0.01; 'probable': 0.01; 'query': 0.01; 'query,': 0.01;
>     'resizes': 0.01; 'servlet': 0.01; 'skip:? 20': 0.01; 'stems':
>     0.01; 'subject:JSP': 0.01; 'sucks!': 0.01; 'templating': 0.01;
>     'tempted': 0.01; 'url.': 0.01; 'usenet': 0.01; 'usenet,': 0.01;
>     'wrote': 0.01; 'x-mailer:mozilla 4.74 [en] (windows nt 5.0; u)':
>     0.01; 'zope': 0.01; '#000000;': 0.99; '#cc0000;': 0.99;
>     '#ff3300;': 0.99; '#ff6600;': 0.99; '#ffffff;': 0.99; '&copy;':
>     0.99; '&gt;': 0.99; '&nbsp;&nbsp;': 0.99; '&quot;no': 0.99;
>     '.med': 0.99; '.small': 0.99; '0pt;': 0.99; '0px;': 0.99; '10px;':
>     0.99; '11pt;': 0.99; '12px;': 0.99; '18pt;': 0.99; '18px;': 0.99;
>     '1pt;': 0.99; '2px;': 0.99; '640;': 0.99; '8pt;': 0.99; '<!--':
>     0.99; '</b>': 0.99; '</body>': 0.99; '</head>': 0.99; '</html>':
>     0.99; '</script>': 0.99; '</select>': 0.99; '</span>': 0.99;
>     '</style>': 0.99; '</table>': 0.99; '</td>': 0.99; '</td></tr>':
>     0.99; '</tr>': 0.99; '</tr><tr': 0.99; '<b><a': 0.99; '<base':
>     0.99; '<body': 0.99; '<br>': 0.99; '<br>&nbsp;': 0.99; '<br><a':
>     0.99; '<br><span': 0.99; '<font': 0.99; '<form': 0.99; '<head>':
>     0.99; '<html>': 0.99; '<img': 0.99; '<input': 0.99; '<meta': 0.99;
>     '<option': 0.99; '<p>': 0.99; '<p>a': 0.99; '<script>': 0.99;
>     '<select': 0.99; '<span': 0.99; '<style>': 0.99; '<table': 0.99;
>     '<td': 0.99; '<td>': 0.99; '<td></td>': 0.99; '<td><img': 0.99;
>     '<tr': 0.99; '<tr>': 0.99; '<tr><td': 0.99; '<tr><td><img': 0.99;
>     'absolute;': 0.99; 'align="left"': 0.99; 'align=center': 0.99;
>     'align=left': 0.99; 'align=middle': 0.99; 'align=right': 0.99;
>     'align=right>': 0.99; 'alt=""': 0.99; 'bold;': 0.99; 'border=0':
>     0.99; 'border=0>': 0.99; 'color:': 0.99; 'colspan=2': 0.99;
>     'colspan=2>': 0.99; 'colspan=4': 0.99; 'face="arial"': 0.99;
>     'font-family:': 0.99; 'font-size:': 0.99; 'font-weight:': 0.99;
>     'footer': 0.99; 'for<br>': 0.99; 'fucking<br>': 0.99;
>     'height="1"': 0.99; 'height="16"': 0.99; 'height=1': 0.99;
>     'height=12': 0.99; 'height=125': 0.99; 'height=17': 0.99;
>     'height=18': 0.99; 'height=21': 0.99; 'height=4': 0.99;
>     'height=57': 0.99; 'height=60': 0.99; 'height=8': 0.99;
>     'hspace=0': 0.99; 'http0:g': 0.99; 'http0:web2': 0.99; 'http1:0':
>     0.99; 'http1:ads': 0.99; 'http1:d': 0.99; 'http1:page': 0.99;
>     'http1:site': 0.99; 'http>1:article': 0.99; 'http>1:back': 0.99;
>     'http>1:com': 0.99; 'http>1:d': 0.99; 'http>1:gif': 0.99;
>     'http>1:go': 0.99; 'http>1:group': 0.99; 'http>1:http': 0.99;
>     'http>1:post': 0.99; 'http>1:ps': 0.99; 'http>1:site': 0.99;
>     'http>1:st': 0.99; 'http>1:title': 0.99; 'http>1:yahoo': 0.99;
>     'inc.</a>': 0.99; 'jobs!': 0.99; 'normal;': 0.99; 'nowrap': 0.99;
>     'nowrap>': 0.99; 'nowrap><font': 0.99; 'padding:': 0.99;
>     'rowspan=2': 0.99; 'rowspan=3': 0.99; 'servlets,': 0.99;
>     'size=15': 0.99; 'size=35': 0.99; 'skip:< 10': 0.99; 'skip:b 60':
>     0.99; 'skip:h 110': 0.99; 'skip:h 170': 0.99; 'skip:h 200': 0.99;
>     'skip:h 240': 0.99; 'skip:h 250': 0.99; 'skip:h 290': 0.99;
>     'skip:v 40': 0.99; 'solid;': 0.99; 'text=#000000': 0.99; 'to<br>':
>     0.99; 'type="image"': 0.99; 'type="text"': 0.99; 'type=hidden':
>     0.99; 'type=image': 0.99; 'type=radio': 0.99; 'type=submit': 0.99;
>     'type=text': 0.99; 'valign=top': 0.99; 'valign=top>': 0.99;
>     'value="">': 0.99; 'visibility:': 0.99; 'width:': 0.99;
>     'width="33"': 0.99; 'width=1': 0.99; 'width=100%': 0.99;
>     'width=100%>': 0.99; 'width=12': 0.99; 'width=125': 0.99;
>     'width=130': 0.99; 'width=137': 0.99; 'width=2': 0.99; 'width=20':
>     0.99; 'width=25': 0.99; 'width=4': 0.99; 'width=468': 0.99;
>     'width=6': 0.99; 'width=72': 0.99; 'works<br>': 0.99

That sure sets the record for longest list of cancelling extreme clues!

> The second f-p had the same structure (and sender :-); the third f-p
> had the same structure and a different sender.  Ditto the fifth,
> sixth.  (Not posting clues for brevity.)

We thank you <wink>.

> The fourth was different: plaintext with one very short sentence and a
> URL.  Clues:
>
>    300 1.00 S 'from:email addr:digicool': 0.01; 'http1:news': 0.24;
>    'from:email addr:com>': 0.32; 'from:tres': 0.50; 'http>1:1114digi':
>    0.50; 'proto:http': 0.50; 'subject:Geeks': 0.50; 'x-mailer:mozilla
>    4.75 [en] (x11; u; linux 2.2.14-5.0smp i686)': 0.50; 'take': 0.54;
>    'bool:noorg': 0.61; 'http0:com': 0.66; 'skip:h 50': 0.83;
>    'http>1:htm': 0.90; 'subject:Software': 0.96; 'http>1:business':
>    0.99; 'http>1:local': 0.99; 'subject:firm': 0.99; 'us:': 0.99

That there are *any* 0.50 clues in here means the scheme ran out of anything
interesting to look at.  Adding in more header lines should cure that.

> The seventh was similar.
>
> I scanned a bunch more until I got bored, and most of them were either
> of the first form (brief text with URL followed by quoted HTML from
> website)

If those were text/plain, the HTML tags should have been stripped.  I'm
still confused about this part.

> or the second (brief text with one or more URLs).
>
> It's up to you to decide what to call this, but I think these are none
> of your #1, #2 or #3 (they're close to #3, but all are multipart/mixed
> rather than multipart/alternative).

That HTML tags aren't getting stripped remains the biggest mystery to me.

>>> So I guess I'll have to retrain it (yes, you told me so :-).

>> That would be a different experiment.  I'm certainly curious to
>> see whether Jeremy's much-worse-than-mine error rates are typical or
>> aberrant.

> It's possible that the corpus you've trained on is more homogeneous
> than you thought.

This seems confused:  Jeremy didn't use my trained classifier pickle, he
trained his own classifier from scratch on his own corpora.  That's an
entirely different kind of experiment from the one you're trying (indeed,
you're the only one so far to report results from trying my pickle on their
own email, and I never expected *that* to work well; it's a much bigger
mystery to me why Jeremy got such relatively worse results from training his
own -- and he's the only one so far to report results from *that*
experiment).


From guido@python.org  Sun Sep  8 06:38:45 2002
From: guido@python.org (Guido van Rossum)
Date: Sun, 08 Sep 2002 01:38:45 -0400
Subject: [Spambayes] test sets?
In-Reply-To: Your message of "Sun, 08 Sep 2002 01:02:32 EDT."
             <LNBBLJKPBEHFEDALKOLCEEOGBCAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCEEOGBCAB.tim.one@comcast.net> 
Message-ID: <200209080538.g885cjk17553@pcp02138704pcs.reston01.va.comcast.net>

> > I also looked in more detail at some f-p's in my geeks traffic.  The
> > first one's a doozie (that's the term, right? :-).  It has lots of
> > HTML clues that are apparently ignored.
> 
> ?? The clues below are *loaded* with snippets unique to HTML (like '<br>').

I *meant* to say that they were 0.99 clues cancelled out by 0.01
clues.  But that's wrong too!  It looks I haven't grokked this part of
your code yet; this one has way more than 16 clues, and it seems the
classifier basically ended up counting way more 0.99 than 0.01 clues,
and no others made it into the list.  I thought it was looking for
clues with values in between; apparently it found none that weren't
exactly 0.5?

> That sure sets the record for longest list of cancelling extreme clues!

This happened to be the longest one, but there were quite a few
similar ones.  I wonder if there's anything we can learn from looking
at the clues and the HTML.  It was heavily marked-up HTML, with ads in
the sidebar, but the body text was a serious discussion of "OO and
soft coding" with lots of highly technical words as clues (including
Zope and ZEO).

> That there are *any* 0.50 clues in here means the scheme ran out of
> anything interesting to look at.  Adding in more header lines should
> cure that.

Are there any minable-but-unmined header lines in your corpus left?
Or do we have to start with a different corpus before we can make
progress there?

> > The seventh was similar.
> >
> > I scanned a bunch more until I got bored, and most of them were either
> > of the first form (brief text with URL followed by quoted HTML from
> > website)
> 
> If those were text/plain, the HTML tags should have been stripped.  I'm
> still confused about this part.

No, sorry.  These were all of the following structure:

  multipart/mixed
      text/plain        (brief text plus URL(s))
      text/html         (long HTML copied from website)

I guess you get this when you click on "mail this page" in some
browsers.

> That HTML tags aren't getting stripped remains the biggest mystery to me.

Still?

> This seems confused: Jeremy didn't use my trained classifier pickle,
> he trained his own classifier from scratch on his own corpora.
> That's an entirely different kind of experiment from the one you're
> trying (indeed, you're the only one so far to report results from
> trying my pickle on their own email, and I never expected *that* to
> work well; it's a much bigger mystery to me why Jeremy got such
> relatively worse results from training his own -- and he's the only
> one so far to report results from *that* experiment).

I think it's still corpus size.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.one@comcast.net  Sun Sep  8 08:18:49 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 08 Sep 2002 03:18:49 -0400
Subject: [Spambayes] test sets?
In-Reply-To: <200209080538.g885cjk17553@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEOIBCAB.tim.one@comcast.net>

[Guido]
> I *meant* to say that they were 0.99 clues cancelled out by 0.01
> clues.  But that's wrong too!  It looks I haven't grokked this part of
> your code yet; this one has way more than 16 clues, and it seems the
> classifier basically ended up counting way more 0.99 than 0.01 clues,
> and no others made it into the list.  I thought it was looking for
> clues with values in between; apparently it found none that weren't
> exactly 0.5?

There's a brief discussion of this before the definition of
MAX_DISCRIMINATORS.  All clues with prob MIN_SPAMPROB and MAX_SPAMPROB are
saved in min and max lists, and all other clues are fed into the nbest heap.
Then the shorter of the min and max lists cancels out the same number of
clues in the longer list.  Whatever remains of the longer list (if anything)
is then fed into the nbest heap too, but no more than MAX_DISCRIMINATORS of
them.  In no case do more than MAX_DISCRIMINATORS clues enter into the final
probability calculation, but all of the min and max lists go into the list
of clues (else you'd have no clue that massive cancellation was occurring;
and massive cancellation may yet turn out to be a hook to signal that manual
review is needed).  In your specific case, the excess of clues in the longer
MAX_SPAMPROB list pushed everything else out of the nbest heap, and that's
why you didn't see anything other than 0.01 and 0.99.

Before adding these special lists, the outcome when faced with many 0.01 and
0.99 clues was too often a coin toss:  whichever flavor just happened to
appear MAX_DISCRIMINATORS//2 + 1 times first determined the final outcome.

>> That sure sets the record for longest list of cancelling extreme clues!

> This happened to be the longest one, but there were quite a few
> similar ones.

I just beat it <wink>:  a tokenization scheme that folds case, and ignores
punctuation, and strips a trailing 's' from words, and saves both word
bigrams and word unigrams, turned up a low-probability very long spam with a
list of 410 0.01 clues and 125 0.99 clues!  Yikes.

> I wonder if there's anything we can learn from looking at the clues and
the
> HTML.  It was heavily marked-up HTML, with ads in the sidebar, but the
body
> text was a serious discussion of "OO and soft coding" with lots of highly
> technical words as clues (including Zope and ZEO).

No matter how often it says Zope, it gets only one 0.01 clue from doing so.
Ditto for ZEO.  In contrast, HTML markup has many unique "words" that get
0.99.  BTW, this is a clear case where the assumption of
conditionally-independent word probabilities is utterly bogus -- e.g., the
probability that "<body>" appears in a message is highly correlated with the
prob of "<br>" appearing.  By treating them as independent, naive Bayes
grossly misjudges the probability that both appear, and the only thing you
get in return is something that can actually be computed <wink>.

Read the "What about HTML?" section in tokenizer.py.  From the very start,
I've been investigating what would work best for the mailing lists hosted at
python.org, and HTML decorations have so far been too strong a clue to
justify ignoring it in that specific context.  I haven't done anything
geared toward personal email, including the case of non-mailing-list email
that happens to go through python.org.

I'd prefer to strip HTML tags from everything, but last time I tried that it
still had bad effects on the error rates in my corpora (the full test
results with and without HTML tag stripping is included in the "What about
HTML?" comment block).  But as the comment block also says,

# XXX So, if another way is found to slash the f-n rate, the decision here
# XXX not to strip HTML from HTML-only msgs should be revisited.

and we've since done several things that gave significant f-n rate
reductions.  I should test that again now.

> Are there any minable-but-unmined header lines in your corpus left?

Almost all of them -- apart from MIME decorations that appear in both
headers and bodies (like Content-Type), the *only* header lines the base
tokenizer looks at now are Subject, From, X-Mailer, and Organization.

> Or do we have to start with a different corpus before we can make
> progress there?

I would need different data, yes.  My ham is too polluted with Mailman
header decorations (which I may or may not be able to clean out, but fudging
the data is a Mortal Sin and I haven't changed a byte so far), and my spam
too polluted with header clues about the fellow who collected it.  In
particular I have to skip To and Received headers now, and I suspect they're
going to be very valuable in real life (for example, I don't even catch
"undisclosed recipients" in the To header now!).

> ...
> No, sorry.  These were all of the following structure:
>
>   multipart/mixed
>       text/plain        (brief text plus URL(s))
>       text/html         (long HTML copied from website)

Ah!  That explains why the HTML tags didn't get stripped.  I'd again offer
to add an optional argument to tokenize() so that they'd get stripped here
too, but if it gets glossed over a third time that would feel too much like
a loss <wink>.

>> This seems confused: Jeremy didn't use my trained classifier pickle,
>> he trained his own classifier from scratch on his own corpora.
>> ...

> I think it's still corpus size.

I reported on tests I ran with random samples of 220 spams and 220 hams from
my corpus (that means training on sets of those sizes as well as predicting
on sets of those sizes), and while that did harm the error rates, the error
rates I saw were still much better than Jeremy reported when using 500 of
each.


Ah, a full test run just finished, on the

   tokenization scheme that folds case, and ignores punctuation, and strips
a
   trailing 's' from words, and saves both word bigrams and word unigrams

This is the code:

            # Tokenize everything in the body.
            lastw = ''
            for w in word_re.findall(text):
                n = len(w)
                # Make sure this range matches in tokenize_word().
                if 3 <= n <= 12:
                    if w[-1] == 's':
                        w = w[:-1]
                    yield w
                    if lastw:
                        yield lastw + w
                    lastw = w + ' '

                elif n >= 3:
                    lastw = ''
                    for t in tokenize_word(w):
                        yield t

where

    word_re = re.compile(r"[\w$\-\x80-\xff]+")

This at least doubled the process size over what's done now.  It helped the
f-n rate significantly, but probably hurt the f-p rate (the f-p rate is too
low with only 4000 hams per run to be confident about changes of such small
*absolute* magnitude -- 0.025% is a single message in the f-p table):

false positive percentages
    0.000  0.000  tied
    0.000  0.075  lost  +(was 0)
    0.050  0.125  lost  +150.00%
    0.025  0.000  won   -100.00%
    0.075  0.025  won    -66.67%
    0.000  0.050  lost  +(was 0)
    0.100  0.175  lost   +75.00%
    0.050  0.050  tied
    0.025  0.050  lost  +100.00%
    0.025  0.000  won   -100.00%
    0.050  0.125  lost  +150.00%
    0.050  0.025  won    -50.00%
    0.050  0.050  tied
    0.000  0.025  lost  +(was 0)
    0.000  0.025  lost  +(was 0)
    0.075  0.050  won    -33.33%
    0.025  0.050  lost  +100.00%
    0.000  0.000  tied
    0.025  0.100  lost  +300.00%
    0.050  0.150  lost  +200.00%

won   5 times
tied  4 times
lost 11 times

total unique fp went from 13 to 21

false negative percentages
    0.327  0.218  won    -33.33%
    0.400  0.218  won    -45.50%
    0.327  0.218  won    -33.33%
    0.691  0.691  tied
    0.545  0.327  won    -40.00%
    0.291  0.218  won    -25.09%
    0.218  0.291  lost   +33.49%
    0.654  0.473  won    -27.68%
    0.364  0.327  won    -10.16%
    0.291  0.182  won    -37.46%
    0.327  0.254  won    -22.32%
    0.691  0.509  won    -26.34%
    0.582  0.473  won    -18.73%
    0.291  0.255  won    -12.37%
    0.364  0.218  won    -40.11%
    0.436  0.327  won    -25.00%
    0.436  0.473  lost    +8.49%
    0.218  0.218  tied
    0.291  0.255  won    -12.37%
    0.254  0.364  lost   +43.31%

won  15 times
tied  2 times
lost  3 times

total unique fn went from 106 to 94


From tim.one@comcast.net  Sun Sep  8 08:48:28 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 08 Sep 2002 03:48:28 -0400
Subject: [Spambayes] test sets?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEOIBCAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCEEOJBCAB.tim.one@comcast.net>

[Tim]
> ...
> I'd prefer to strip HTML tags from everything, but last time I
> tried that it still had bad effects on the error rates in my
> corpora (the full test results with and without HTML tag stripping
> is included in the "What about HTML?" comment block).  But as the
> comment block also says,
>
> # XXX So, if another way is found to slash the f-n rate, the decision here
> # XXX not to strip HTML from HTML-only msgs should be revisited.
>
> and we've since done several things that gave significant f-n rate
> reductions.  I should test that again now.

I did so.  Alas, stripping HTML tags from all text still hurts the f-n rate
in my test data:

false positive percentages
    0.000  0.000  tied
    0.000  0.000  tied
    0.050  0.075  lost   +50.00%
    0.025  0.025  tied
    0.075  0.025  won    -66.67%
    0.000  0.000  tied
    0.100  0.100  tied
    0.050  0.075  lost   +50.00%
    0.025  0.025  tied
    0.025  0.000  won   -100.00%
    0.050  0.075  lost   +50.00%
    0.050  0.050  tied
    0.050  0.025  won    -50.00%
    0.000  0.000  tied
    0.000  0.000  tied
    0.075  0.075  tied
    0.025  0.025  tied
    0.000  0.000  tied
    0.025  0.025  tied
    0.050  0.050  tied

won   3 times
tied 14 times
lost  3 times

total unique fp went from 13 to 11

false negative percentages
    0.327  0.400  lost   +22.32%
    0.400  0.400  tied
    0.327  0.473  lost   +44.65%
    0.691  0.654  won     -5.35%
    0.545  0.473  won    -13.21%
    0.291  0.364  lost   +25.09%
    0.218  0.291  lost   +33.49%
    0.654  0.654  tied
    0.364  0.473  lost   +29.95%
    0.291  0.327  lost   +12.37%
    0.327  0.291  won    -11.01%
    0.691  0.654  won     -5.35%
    0.582  0.655  lost   +12.54%
    0.291  0.400  lost   +37.46%
    0.364  0.436  lost   +19.78%
    0.436  0.582  lost   +33.49%
    0.436  0.364  won    -16.51%
    0.218  0.291  lost   +33.49%
    0.291  0.400  lost   +37.46%
    0.254  0.327  lost   +28.74%

won   5 times
tied  2 times
lost 13 times

total unique fn went from 106 to 122

Last time I tried this (see tokenizer.py comments), the f-n rate after
stripping tags ranged from 0.982% to 1.781%, with a median of about 1.34%,
so we've made tons of progress on the f-n rate since then.  But the mere
presence of HTML tags still remains a significant clue for c.l.py traffic,
so I'm left with the same comment:

> # XXX So, if another way is found to slash the f-n rate, the decision here
> # XXX not to strip HTML from HTML-only msgs should be revisited.

If we want to take the focus of this away from c.l.py traffic, I can't say
what effect HTML stripping would have (I don't have suitable test data to
measure that on).


From skip@pobox.com  Sun Sep  8 12:03:04 2002
From: skip@pobox.com (Skip Montanaro)
Date: Sun, 8 Sep 2002 06:03:04 -0500
Subject: [Spambayes] spambayes package?
In-Reply-To: <200209080247.g882lNt17033@pcp02138704pcs.reston01.va.comcast.net>
References: <15738.34711.467756.145336@12-248-11-90.client.attbi.com>
        <200209080247.g882lNt17033@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <15739.11880.95033.293165@12-248-11-90.client.attbi.com>


    >> Before we get too far down this road, what do people think of
    >> creating a spambayes package containing classifier and tokenizer?
    >> This is just to minimize clutter in site-packages.

    Guido> Too early IMO (if you mean to leave the various other tools out
    Guido> of it).

Well, I mentioned classifier and tokenize only because I thought they were
the only importable modules.  The rest represent script-level code, right?

    Guido> If and when we package this, perhaps we should use Barry's trick
    Guido> from the email package for making the package itself the toplevel
    Guido> dir of the distribution (rather than requiring an extra directory
    Guido> level just so the package can be a subdir of the distro).

That would be perfect.  I tried in the naive way last night, but wound up
with all .py files in the package, which wasn't my intent.

Skip



From nas@python.ca  Sun Sep  8 18:21:13 2002
From: nas@python.ca (Neil Schemenauer)
Date: Sun, 8 Sep 2002 10:21:13 -0700
Subject: [Spambayes] testing results
Message-ID: <20020908172113.GA26741@glacier.arctrix.com>

These results are from timtest.py.  I've got three sets of spam and ham
with about 500 messages in each set.  Here's what happens when I enable
my latest "received" header code:

    false positive percentages
        0.187  0.187  tied
        0.749  0.562  won    -24.97%
        0.780  0.585  won    -25.00%

    won   2 times
    tied  1 times
    lost  0 times

    total unique fp went from 19 to 17

    false negative percentages
        2.072  1.318  won    -36.39%
        2.448  1.318  won    -46.16%
        0.574  0.765  lost   +33.28%

    won   2 times
    tied  0 times
    lost  1 times

    total unique fn went from 43 to 28

Anthony's header counting code does not seem to help.

  Neil

From tim.one@comcast.net  Sun Sep  8 19:28:02 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 08 Sep 2002 14:28:02 -0400
Subject: [Spambayes] testing results
In-Reply-To: <20020908172113.GA26741@glacier.arctrix.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCOEPHBCAB.tim.one@comcast.net>

[Neil Schemenauer]
> These results are from timtest.py.  I've got three sets of spam and ham
> with about 500 messages in each set.  Here's what happens when I enable
> my latest "received" header code:

If you've still got the summary files, please cvs up and try running cmp.py
again -- in the process of generalizing cmp.py, you managed to make it skip
half the lines <wink>.  That is, if you've got N sets, you *should* get
N**2-N pairs for each error rate.  You have 3 sets, so you should get 6
pairs of f-n rates and 6 pairs of f-p rates.

>     false positive percentages
>         0.187  0.187  tied
>         0.749  0.562  won    -24.97%
>         0.780  0.585  won    -25.00%
>
>     won   2 times
>     tied  1 times
>     lost  0 times
>
>     total unique fp went from 19 to 17
>
>     false negative percentages
>         2.072  1.318  won    -36.39%
>         2.448  1.318  won    -46.16%
>         0.574  0.765  lost   +33.28%
>
>     won   2 times
>     tied  0 times
>     lost  1 times
>
>     total unique fn went from 43 to 28

Looks promising!  Getting 6 lines of output for each block would give a
clearer picture, of course.

> Anthony's header counting code does not seem to help.

It helps my test data too much <wink/sigh>.


From tim.one@comcast.net  Sun Sep  8 20:48:13 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 08 Sep 2002 15:48:13 -0400
Subject: [Spambayes] testing results
In-Reply-To: <20020908172113.GA26741@glacier.arctrix.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEPKBCAB.tim.one@comcast.net>

Neil trained a classifier using 3 sets with about 500 ham and spam in each.
We're missing half his test run results due to a cmp.py bug (since fixed);
the "before custom fiddling" figures on the 3 reported runs were:

    false positive percentages
        0.187
        0.749
        0.780
    total unique fp 19

    false negative percentages
        2.072
        2.448
        0.574
    total unique fn 43

The "total unique" figures counts all 6 runs; it's just the individual-run
fp and fn percentages we're missing for 3 runs.

Jeremy reported these "before custom fiddling" figures on 4 sets with about
600 ham and spam in each:

    false positive percentages
        0.000
        1.398
        1.398
        0.000
        1.242
        1.242
        1.398
        1.398
        0.000
        1.553
        1.553
        0.000
    total unique fp 139

    false negative percentages
       10.413
        6.104
        5.027
        8.259
        2.873
        5.745
        5.206
        4.488
        9.336
        5.206
        5.027
        9.874
    total unique fn 970

So things are clearly working much better for Neil.  Both reported
significant improvements in both f-n and f-p rates by folding in more header
lines.  Neal added Received analysis to the base tokenizer's header
analysis, and Jeremy skipped the base tokenizer's header analysis completely
but added base-subject-line-like but case-folded tokenization for almost all
header lines (excepting only Received, Data, X-From_, and, I *suspect*, all
those starting with 'x-vm').

When I try 5 random pairs of 500-ham + 500-spam subsets in my test data, I
see:

    false positive percentages
        0.000
        0.000
        0.200
        0.000
        0.200
        0.000
        0.200
        0.000
        0.000
        0.200
        0.400
        0.000
        0.200
        0.000
        0.200
        0.400
        0.000
        0.400
        0.200
        0.600
    total unique fp 10

    false negative percentages
        0.800
        0.400
        0.200
        0.600
        1.000
        0.000
        0.600
        1.200
        1.200
        0.800
        0.400
        0.800
        1.800
        0.800
        0.400
        1.000
        1.000
        0.400
        0.000
        0.600
    total unique fn 36

This is much closer to what Neil saw, but still looks better.  Another run
on a disjoint 5 random pairs looked much the same; total unique fp rose to
12 and fn fell to 27; on a third run with another set of disjoint 5 random
pairs, likewise, with fp 12 and fn 40.  So I'm pretty confident that it's
not going to matter which random subsets of 500 I take from my data.

It's hard to conclude anything given Jeremy's much worse results.  If they
were in line with Neil's results, I'd suspect that I've over-tuned the
algorithm to statistical quirks in my corpora.


From tim.one@comcast.net  Sun Sep  8 21:01:00 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 08 Sep 2002 16:01:00 -0400
Subject: [Spambayes] test sets?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCEEOGBCAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEPLBCAB.tim.one@comcast.net>

[Tim]
> One effect of getting rid of MINCOUNT is that it latches on more
> strongly to rare clues now, and those can be unique to the corpus
> trained on (e.g., one trained ham says "gryndlplyx!", and a followup
> new ham quotes it).

This may be a systematic bias in the testing procedure:  in real life, msgs
come ordered in time.  Say there's a thread that spans N messages on c.l.py.
In our testing setup, we'll train on a random sampling throughout its whole
lifetime, and test likewise.  New ham "in the middle" of this thread gets
benefit from that we trained on msgs that appeared both before and *after*
it in real life.  It's quite plausible that the f-p rate would rise without
this effect; in real life, at any given time some number of ham threads will
just be starting their lives, and if they're at all unusual the trained data
will know little to nothing about them.


From tim.one@comcast.net  Sun Sep  8 21:13:40 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 08 Sep 2002 16:13:40 -0400
Subject: [Spambayes] test sets?
In-Reply-To: <20020906180223.GA18250@cthulhu.gerg.ca>
Message-ID: <LNBBLJKPBEHFEDALKOLCOEPLBCAB.tim.one@comcast.net>

[Greg Ward]
> Case of headers is definitely helpful.  SpamAssassin has a rule for it
> -- if you have headers like "DATE" or "SUBJECT", you get a few more
> points.

Across my data, all-caps DATE, SUBJECT, TO, etc indeed appear only in the
spam collections.  OTOH, they don't appear often -- less than 1% of spam
messages have at least one of these all-cap header lines.  But when I'm
fighting what are now sub-1% f-n rates, even rare clues can help!


From tim.one@comcast.net  Sun Sep  8 21:36:15 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 08 Sep 2002 16:36:15 -0400
Subject: [Spambayes] Ditching WordInfo
In-Reply-To: <w537khybba6.fsf@woozle.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEPNBCAB.tim.one@comcast.net>

[Neale Pickett]
> ...
> If you can spare the memory, you might get better performance in this
> case using the pickle store, since it only has to go to disk once (but
> boy, does it ever go to disk!)  I can't think of anything obvious to
> speed things up once it's all loaded into memory, though.

On my box the current system scores about 50 msgs per second (starting in
memory, of course).  While that can be a drag while waiting for one of my
full test runs to complete (one of those scores a message more than 120,000
times, and trains more than 30,000 times), I've got no urge to do any speed
optimizations -- if I were using this for my own email, I'd never notice the
drag.  Guido will bitch like hell about waiting an extra second for his
50-msg batches to score, but he's the boss so he bitches about everything
<wink>.

> That's profiler territory, and profiling is exactly the kind of
> optimization I just said I wasn't going to do :)

I haven't profiled yet, but *suspect* there aren't any egregious hot spots.
5-gram'ing of long words with high-bit characters is likely overly expensive
*when* it happens, but it doesn't happen that often, and as an approach to
non-English languages it sucks anyway (i.e., there's no point speeding
something that ought to be replaced entirely).


From tim.one@comcast.net  Sun Sep  8 21:46:47 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 08 Sep 2002 16:46:47 -0400
Subject: [Spambayes] hammie.py vs. GBayes.py
In-Reply-To: <200209070351.g873pC613144@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEPOBCAB.tim.one@comcast.net>

[Guido]
> There seem to be two "drivers" for the classifier now: Neale Pickett's
> hammie.py, and the original GBayes.py.  According to the README.txt,
> GBayes.py hasn't been kept up to date.

It seemed that way to me when I ripped the classifier out of it -- I don't
think anyone has touched it after.

> Is there anything in there that isn't covered by hammie.py?

Someone else will have to answer that (I don't use GBayes or hammie, at
least not yet).

> About the only useful feature of GBayes.py that hammie.py doesn't (yet)
> copy is -u, which calculates spamness for an entire mailbox.  This
> feature can easily be copied into hammie.py.

That's been done now, right?

> (GBayes.py also has a large collection of tokenizers; but timtoken.py
> rules, so I'm not sure how interesting that is now.)

Those tokenizers are truly trivial to rewrite from scratch if they're
interesting.  The tiny spam/ham collections in GBayes are also worthless
now.  The "self test" feature didn't do anything except print its results;
Tester.py since became doctest'ed and verifies that some basic machinery
actually delivers what it's supposed to deliver.

> Therefore I propose to nuke GBayes.py, after adding a -u feature.

+1 here.

> Anyone against?


From bkc@murkworks.com  Sun Sep  8 21:45:49 2002
From: bkc@murkworks.com (Brad Clements)
Date: Sun, 08 Sep 2002 16:45:49 -0400
Subject: [Spambayes] All Cap or Cap Word Subjects
Message-ID: <3D7B7F11.22376.29256B69@localhost>

Just curious if subject line capitalization can be used as an indicator.

Either the percentage of characters that are caps..

Or, percentage starting with a capital letter (if number of words > xx)



Brad Clements,                bkc@murkworks.com   (315)268-1000
http://www.murkworks.com                          (315)268-9812 Fax
AOL-IM: BKClements


From tim.one@comcast.net  Sun Sep  8 21:56:59 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 08 Sep 2002 16:56:59 -0400
Subject: [Spambayes] All Cap or Cap Word Subjects
In-Reply-To: <3D7B7F11.22376.29256B69@localhost>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEPPBCAB.tim.one@comcast.net>

[Brad Clements]
> Just curious if subject line capitalization can be used as an indicator.
>
> Either the percentage of characters that are caps..
>
> Or, percentage starting with a capital letter (if number of words > xx)

Supply a mod to tokenizer.py and I'll test it (eventually <wink>).  Note
that the tokenizer already *preserves* case in subject-line words, because
experiment showed that this was better than folding case away in this
specific context (but experiment also showed-- against my
expectations --that preserving case everywhere didn't make a significant
difference to either error rate -- the subject line is a special case for
this).


From tim.one@comcast.net  Mon Sep  9 01:00:55 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 08 Sep 2002 20:00:55 -0400
Subject: [Spambayes] test sets?
In-Reply-To: <200209070400.g8740a520809@localhost.localdomain>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEAEBDAB.tim.one@comcast.net>

[Tim, to Anthony]
>> So you know whether src added additional power, or did you do both at
>> once?

[Anthony Baxter]
> Both at once. I added it because <iframe src=cid:foofoofoo> is such a
> killer detector of spam/viruses, also because I got a bunch of email
> spam that was just
>
>     <img src="bozo.bozo.kr/img34532.jpg">
>     <img src="bozo.bozo.kr/img34512.jpg">
>     <img src="bozo.bozo.kr/img34237.jpg">
>     <img src="bozo.bozo.kr/img34914.jpg">

I added src= tokenization, and the result was identical f-n and f-p rates
across all 20 of my "standard runs":  0 differences.  I suspect this is
because img tags *really* look more like

     <img src="http://bozo.bozo.kr/img34914.jpg">

and our tokenizer was already picking up http thingies regardless of their
context.

> I'm already stripping out HTML tags - it was producing far far too
> many false positives with my corpus with them in. Without the src/hrefs
> these spams were pretty much null and void.

I'd like to strip them too <wink>; but see earlier recent msgs about that.


From nas@python.ca  Mon Sep  9 02:20:51 2002
From: nas@python.ca (Neil Schemenauer)
Date: Sun, 8 Sep 2002 18:20:51 -0700
Subject: [Spambayes] testing results
In-Reply-To: <LNBBLJKPBEHFEDALKOLCOEPHBCAB.tim.one@comcast.net>
References: <20020908172113.GA26741@glacier.arctrix.com>
	<LNBBLJKPBEHFEDALKOLCOEPHBCAB.tim.one@comcast.net>
Message-ID: <20020909012051.GD27510@glacier.arctrix.com>

Tim Peters wrote:
> If you've still got the summary files, please cvs up and try running cmp.py
> again -- in the process of generalizing cmp.py, you managed to make it skip
> half the lines <wink>.

Woops.  I didn't have the summary files so I regenerated them using a
slightly different set of data.  Here are the results of enabling the
"received" header processing:

    false positive percentages
        0.707  0.530  won    -25.04%
        0.873  0.524  won    -39.98%
        0.301  0.301  tied
        1.047  1.047  tied
        0.602  0.452  won    -24.92%
        0.353  0.177  won    -49.86%

    won   4 times
    tied  2 times
    lost  0 times

    total unique fp went from 17 to 14 won    -17.65%

    false negative percentages
        2.167  1.238  won    -42.87%
        0.969  0.969  tied
        1.887  1.372  won    -27.29%
        1.616  1.292  won    -20.05%
        1.029  0.858  won    -16.62%
        1.548  1.548  tied

    won   4 times
    tied  2 times
    lost  0 times

    total unique fn went from 50 to 38 won    -24.00%

My test set is different than Tim's in that all the email was received
by the same account.  Also, my set contains email sent to me, not to
mailing lists (I use a different addresses for mailing lists).  If
people cook up more ideas I will be happy to test them.

  Neil

From tim.one@comcast.net  Mon Sep  9 03:16:18 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 08 Sep 2002 22:16:18 -0400
Subject: [Spambayes] Ditching WordInfo
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCEPNBCAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEAHBDAB.tim.one@comcast.net>

[Tim]
> ...
> On my box the current system scores about 50 msgs per second (starting
> in memory, of course).

That was a guess.  Bothering to get a clock out, it was more like 80 per
second.  See?  A 60% speedup without changing a thing <wink>.


From tim.one@comcast.net  Mon Sep  9 04:36:00 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 08 Sep 2002 23:36:00 -0400
Subject: [Spambayes] testing results
In-Reply-To: <20020909012051.GD27510@glacier.arctrix.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEAMBDAB.tim.one@comcast.net>

[Neil Schemenauer]
> Woops.  I didn't have the summary files so I regenerated them using a
> slightly different set of data.  Here are the results of enabling the
> "received" header processing:
>
>     false positive percentages
>         0.707  0.530  won    -25.04%
>         0.873  0.524  won    -39.98%
>         0.301  0.301  tied
>         1.047  1.047  tied
>         0.602  0.452  won    -24.92%
>         0.353  0.177  won    -49.86%
>
>     won   4 times
>     tied  2 times
>     lost  0 times
>
>     total unique fp went from 17 to 14 won    -17.65%
>
>     false negative percentages
>         2.167  1.238  won    -42.87%
>         0.969  0.969  tied
>         1.887  1.372  won    -27.29%
>         1.616  1.292  won    -20.05%
>         1.029  0.858  won    -16.62%
>         1.548  1.548  tied
>
>     won   4 times
>     tied  2 times
>     lost  0 times
>
>     total unique fn went from 50 to 38 won    -24.00%
>
> My test set is different than Tim's in that all the email was received
> by the same account.  Also, my set contains email sent to me, not to
> mailing lists (I use a different addresses for mailing lists).

Enabling the Received headers works even better for me <wink>; here's the
f-n section from a quick run on 500-element subsets:

    0.600  0.200  won    -66.67%
    0.200  0.200  tied
    0.200  0.000  won   -100.00%
    0.800  0.400  won    -50.00%
    0.400  0.200  won    -50.00%
    0.400  0.000  won   -100.00%
    0.200  0.000  won   -100.00%
    1.000  0.400  won    -60.00%
    0.800  0.200  won    -75.00%
    1.200  0.600  won    -50.00%
    0.400  0.200  won    -50.00%
    2.000  0.800  won    -60.00%
    0.400  0.400  tied
    1.200  0.600  won    -50.00%
    0.400  0.000  won   -100.00%
    2.000  1.000  won    -50.00%
    0.400  0.000  won   -100.00%
    0.800  0.000  won   -100.00%
    0.000  0.200  lost  +(was 0)
    0.400  0.000  won   -100.00%

won  17 times
tied  2 times
lost  1 times

total unique fn went from 38 to 15 won    -60.53%

A huge improvement, but for wrong reasons ... except not entirely!  The most
powerful discriminator in the whole database on one training set became:

        'received:unknown' 881 0.99

That's got nothing to do with BruceG, right?

        'received:bfsmedia.com'

was also a strong spam indicator across all training sets.  I'm jealous.

> If people cook up more ideas I will be happy to test them.

Neil, are using your own tokenizer now, or the tokenizer.Tokenizer.tokenize
generator?  Whichever, someone who's not afraid of their headers should try
adding mboxtest.MyTokenizer.tokenize_headers into the mix, once in lieu of
tokenizer.Tokenizer.tokenize_headers(), and again in addition to it.  Jeremy
reported on just the former.


From gward@python.net  Mon Sep  9 14:16:06 2002
From: gward@python.net (Greg Ward)
Date: Mon, 9 Sep 2002 09:16:06 -0400
Subject: [Spambayes] spambayes package?
In-Reply-To: <200209080247.g882lNt17033@pcp02138704pcs.reston01.va.comcast.net>
References: <15738.34711.467756.145336@12-248-11-90.client.attbi.com>
	<200209080247.g882lNt17033@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <20020909131606.GA1136@cthulhu.gerg.ca>

On 07 September 2002, Guido van Rossum said:
> If and when we package this, perhaps we should use Barry's trick
> from the email package for making the package itself the toplevel dir
> of the distribution (rather than requiring an extra directory level
> just so the package can be a subdir of the distro).

It's not a *trick*!  It just requires this

  package_dir = {'spambayes': '.'}

in the setup script.

harrumph!  "trick" indeed...

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
A committee is a life form with six or more legs and no brain.

From guido@python.org  Mon Sep  9 15:10:00 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 09 Sep 2002 10:10:00 -0400
Subject: [Spambayes] test sets?
In-Reply-To: Your message of "Sun, 08 Sep 2002 03:18:49 EDT."
             <LNBBLJKPBEHFEDALKOLCGEOIBCAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCGEOIBCAB.tim.one@comcast.net> 
Message-ID: <200209091410.g89EA0428928@pcp02138704pcs.reston01.va.comcast.net>

> I'd prefer to strip HTML tags from everything, but last time I tried
> that it still had bad effects on the error rates in my corpora

Your corpora are biased in this respect though -- newsgroups have a
strong social taboo on posting HTML, but in many people's personal
inboxes it is quite abundant.

Getting a good ham corpus may prove to be a bigger hurdle than I
though!  My own saved mail doesn't reflect what I receive, since I
save and throw away selectively (much more so than in the past :-).

> >   multipart/mixed
> >       text/plain        (brief text plus URL(s))
> >       text/html         (long HTML copied from website)
> 
> Ah!  That explains why the HTML tags didn't get stripped.  I'd again
> offer to add an optional argument to tokenize() so that they'd get
> stripped here too, but if it gets glossed over a third time that
> would feel too much like a loss <wink>.

I'll bite.  Sounds like a good idea to strip the HTML in this case;
I'd like to see how this improves the f-p rate on this corpus.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.one@comcast.net  Mon Sep  9 15:45:52 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 09 Sep 2002 10:45:52 -0400
Subject: [Spambayes] test sets?
In-Reply-To: <200209091410.g89EA0428928@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCIECBBDAB.tim.one@comcast.net>

[Tim]
>> I'd prefer to strip HTML tags from everything, but last time I tried
>> that it still had bad effects on the error rates in my corpora

[Guido]
> Your corpora are biased in this respect though -- newsgroups have a
> strong social taboo on posting HTML, but in many people's personal
> inboxes it is quite abundant.

We're in violent agreement there:  the comments in tokenizer.py say that as
strongly as possible, and I've repeated it endlessly here too.  But so long
as I was the only one doing serious testing, it was a dubious idea to make
the code maximally clumsy for me to use on the c.l.py task <wink>.

> Getting a good ham corpus may prove to be a bigger hurdle than I
> though!  My own saved mail doesn't reflect what I receive, since I
> save and throw away selectively (much more so than in the past :-).

Yup, the system picks up on *everything* in the tokens.  Graham's proposed
"delete as ham" and "delete as spam" keys would probably work very well for
motivated geeks.  But Paul Svensson has pointed out here that they probably
wouldn't work nearly so well for real people.

>> Ah!  That explains why the HTML tags didn't get stripped.  I'd again
>> offer to add an optional argument to tokenize() so that they'd get
>> stripped here too, but if it gets glossed over a third time that
>> would feel too much like a loss <wink>.

> I'll bite.  Sounds like a good idea to strip the HTML in this case;
> I'd like to see how this improves the f-p rate on this corpus.

I'll soon check in this change:

    def tokenize_body(self, msg, retain_pure_html_tags=False):
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
        """Generate a stream of tokens from an email Message.

        If a multipart/alternative section has both text/plain and text/html
        sections, the text/html section is ignored.  This may not be a good
        idea (e.g., the sections may have different content).

        HTML tags are always stripped from text/plain sections.

        By default, HTML tags are also stripped from text/html sections.
        However, doing so hurts the false negative rate on Tim's
        comp.lang.python tests (where HTML-only messages are almost never
        legitimate traffic).  If optional argument retain_pure_html_tags
        is specified and True, HTML tags are retained in text/html sections.
        """

You should do a cvs up and establish a new baseline first, as I checked in a
pure-win change in the wee hours that cut the fp and fn rates in my tests.


From tim.one@comcast.net  Mon Sep  9 15:53:39 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 09 Sep 2002 10:53:39 -0400
Subject: [Spambayes] test sets?
Message-ID: <LNBBLJKPBEHFEDALKOLCGECCBDAB.tim.one@comcast.net>

I see that most people here aren't on the spambayes-checkins list.  You
should be aware of this change.  If anyone thinks one of the headers in the
safe_headers set is prone to systematic bias in their test data, let me know
and I'll take it out.

-----Original Message-----
From: spambayes-checkins-bounces@python.org
[mailto:spambayes-checkins-bounces@python.org]On Behalf Of Tim Peters
Sent: Monday, September 09, 2002 12:56 AM
To: spambayes-checkins@python.org
Subject: [Spambayes-checkins] spambayes tokenizer.py,1.9,1.10

Update of /cvsroot/spambayes/spambayes
In directory usw-pr-cvs1:/tmp/cvs-serv29522

Modified Files:
	tokenizer.py
Log Message:
Pure win, from enabling Anthony's "count the mere # of various header
lines, case-sensitively" on a small subset of header lines.  This
avoids all the header lines the union of Greg and Barry told me *might*
be artifacts of Mailman and/or BruceG's (the spam collector's)
email setup.  It's an open question how much this may merely be
discriminating newsgroup traffic from non-newsgroup mail, but I also
left out what I thought were obvious newsgroupy headers (like References:).
The presence of X-Complaints-To happens to be a very strong discriminator
in my data, and accounts for redeeming 6 of the 14 previous false
positives.

false positive percentages
    0.000  0.000  tied
    0.000  0.000  tied
    0.100  0.050  won    -50.00%
    0.025  0.000  won   -100.00%
    0.050  0.025  won    -50.00%
    0.000  0.000  tied
    0.075  0.075  tied
    0.050  0.025  won    -50.00%
    0.025  0.025  tied
    0.025  0.000  won   -100.00%
    0.050  0.050  tied
    0.050  0.000  won   -100.00%
    0.050  0.025  won    -50.00%
    0.000  0.000  tied
    0.000  0.000  tied
    0.075  0.050  won    -33.33%
    0.025  0.025  tied
    0.000  0.000  tied
    0.025  0.025  tied
    0.100  0.050  won    -50.00%

won   9 times
tied 11 times
lost  0 times

total unique fp went from 14 to 8 won    -42.86%

false negative percentages
    0.291  0.255  won    -12.37%
    0.364  0.364  tied
    0.254  0.254  tied
    0.582  0.509  won    -12.54%
    0.545  0.436  won    -20.00%
    0.218  0.218  tied
    0.218  0.182  won    -16.51%
    0.654  0.582  won    -11.01%
    0.364  0.327  won    -10.16%
    0.255  0.255  tied
    0.400  0.254  won    -36.50%
    0.654  0.582  won    -11.01%
    0.618  0.545  won    -11.81%
    0.291  0.255  won    -12.37%
    0.291  0.291  tied
    0.436  0.400  won     -8.26%
    0.436  0.291  won    -33.26%
    0.218  0.218  tied
    0.255  0.218  won    -14.51%
    0.182  0.145  won    -20.33%

won  14 times
tied  6 times
lost  0 times

total unique fn went from 101 to 89 won    -11.88%


Index: tokenizer.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/tokenizer.py,v
retrieving revision 1.9
retrieving revision 1.10
diff -C2 -d -r1.9 -r1.10
*** tokenizer.py	8 Sep 2002 23:48:50 -0000	1.9
--- tokenizer.py	9 Sep 2002 04:56:12 -0000	1.10
***************
*** 745,748 ****
--- 745,770 ----
          yield '.'.join(parts[:i])

+ # We're merely going to count the number of these, and case-sensitively.
+ safe_headers = Set("""
+     abuse-reports-to
+     date
+     errors-to
+     from
+     importance
+     in-reply-to
+     message-id
+     mime-version
+     organization
+     received
+     reply-to
+     return-path
+     subject
+     to
+     user-agent
+     x-abuse-info
+     x-complaints-to
+     x-face
+ """.split())
+

...


From skip@pobox.com  Mon Sep  9 15:11:16 2002
From: skip@pobox.com (Skip Montanaro)
Date: Mon, 9 Sep 2002 09:11:16 -0500
Subject: [Spambayes] spambayes package?
In-Reply-To: <20020909131606.GA1136@cthulhu.gerg.ca>
References: <15738.34711.467756.145336@12-248-11-90.client.attbi.com>
        <200209080247.g882lNt17033@pcp02138704pcs.reston01.va.comcast.net>
        <20020909131606.GA1136@cthulhu.gerg.ca>
Message-ID: <15740.44036.651321.285627@12-248-11-90.client.attbi.com>


    >> If and when we package this, perhaps we should use Barry's trick ...

    Greg> It's not a *trick*!  It just requires this

    Greg>   package_dir = {'spambayes': '.'}

    Greg> in the setup script.

That has the nasty side effect of placing all .py files in the package.
What about obvious executable scripts (like timtest or hammie)?  How can I
keep them out of the package?

Skip


From guido@python.org  Mon Sep  9 16:22:52 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 09 Sep 2002 11:22:52 -0400
Subject: [Spambayes] spambayes package?
In-Reply-To: Your message of "Mon, 09 Sep 2002 09:11:16 CDT."
             <15740.44036.651321.285627@12-248-11-90.client.attbi.com> 
References: <15738.34711.467756.145336@12-248-11-90.client.attbi.com>
	<200209080247.g882lNt17033@pcp02138704pcs.reston01.va.comcast.net>
	<20020909131606.GA1136@cthulhu.gerg.ca>  
	<15740.44036.651321.285627@12-248-11-90.client.attbi.com> 
Message-ID: <200209091522.g89FMqL29316@pcp02138704pcs.reston01.va.comcast.net>

> That has the nasty side effect of placing all .py files in the
> package.  What about obvious executable scripts (like timtest or
> hammie)?  How can I keep them out of the package?

Why would we care about installing a few extra files, as long as
they're inside a package?

--Guido van Rossum (home page: http://www.python.org/~guido/)

From marklists@mceahern.com  Mon Sep  9 17:26:51 2002
From: marklists@mceahern.com (Mark McEahern)
Date: Mon, 9 Sep 2002 11:26:51 -0500
Subject: [Spambayes] spambayes package?
In-Reply-To: <15740.44036.651321.285627@12-248-11-90.client.attbi.com>
Message-ID: <JHEOKEOOLIGLDHCMAHMOGEGJDNAA.marklists@mceahern.com>

[Skip Montanaro]
> That has the nasty side effect of placing all .py files in the package.
> What about obvious executable scripts (like timtest or hammie)?  How can I
> keep them out of the package?

Put them in a scripts folder?

// m

-


From skip@pobox.com  Mon Sep  9 17:31:12 2002
From: skip@pobox.com (Skip Montanaro)
Date: Mon, 9 Sep 2002 11:31:12 -0500
Subject: [Spambayes] deleting "duplicate" spam before training?  good idea or
	bad?
Message-ID: <15740.52432.861148.597750@12-248-11-90.client.attbi.com>


Because I get mail through several different email addresses, I frequently
get duplicates (or triplicates or more-plicates) of various spam messages.
In saving spam for later analysis I haven't always been careful to avoid
saving such duplicates.

I wrote a script some time ago to try an minimize the duplicates I see by
calculating a loose checksum, but I still have some duplicates.  Should I
delete the duplicates before training or not?  Would people be interested in
the script?  I'd be happy to extricate it from my local modules and check it
into CVS.

Skip


From jeremy@alum.mit.edu  Mon Sep  9 17:37:05 2002
From: jeremy@alum.mit.edu (Jeremy Hylton)
Date: Mon, 9 Sep 2002 12:37:05 -0400
Subject: [Spambayes] understanding high false negative rate
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCENKBCAB.tim.one@comcast.net>
References: <15738.24135.294137.640570@slothrop.zope.com>
	<LNBBLJKPBEHFEDALKOLCCENKBCAB.tim.one@comcast.net>
Message-ID: <15740.52785.769385.831666@slothrop.zope.com>

>>>>> "TP" == Tim Peters <tim.one@comcast.net> writes:

  >> First test results using tokenizer.Tokenizer.tokenize_headers()
  >> unmodified.  ...  Second test results using
  >> mboxtest.MyTokenizer.tokenize_headers().  This uses all headers
  >> except Received, Data, and X-From_.  ...

  TP> Try the latter again, but call the base tokenize_headers() too.

Sorry.  I haven't found the time to try any more test runs.  Perhaps
later today.

Jeremy


From tim.one@comcast.net  Mon Sep  9 17:46:55 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 09 Sep 2002 12:46:55 -0400
Subject: [Spambayes] deleting "duplicate" spam before training?  good idea
 orbad?
In-Reply-To: <15740.52432.861148.597750@12-248-11-90.client.attbi.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCIECKBDAB.tim.one@comcast.net>

[Skip Montanaro]
> Because I get mail through several different email addresses, I
> frequently get duplicates (or triplicates or more-plicates) of
> various spam messages.  In saving spam for later analysis I haven't
> always been careful to avoid saving such duplicates.
>
> I wrote a script some time ago to try an minimize the duplicates I see
> by calculating a loose checksum, but I still have some duplicates.
> Should I delete the duplicates before training or not?

People just can't stop thinking <wink>.  The classifier should work best
when trained on a wholly random spattering of real life.  If real life
contains duplicates, then that's what the classifier should see.

> Would people be interested in the script?  I'd be happy to extricate
> it from my local modules and check it into CVS.

Sure!  I think it's relevant, but maybe for another purpose.  Paul Svensson
is thinking harder about real people <wink> than the rest of us, and he may
be able to get use out of approaches that identify closely related spam.
For example, some amount of spam is going to end up in the ham training data
in real life use, and any sort of similarity score to a piece of known spam
may be an aid in finding and purging it.


From skip@pobox.com  Mon Sep  9 19:15:37 2002
From: skip@pobox.com (Skip Montanaro)
Date: Mon, 9 Sep 2002 13:15:37 -0500
Subject: [Spambayes] spambayes package?
In-Reply-To: <200209091522.g89FMqL29316@pcp02138704pcs.reston01.va.comcast.net>
References: <15738.34711.467756.145336@12-248-11-90.client.attbi.com>
        <200209080247.g882lNt17033@pcp02138704pcs.reston01.va.comcast.net>
        <20020909131606.GA1136@cthulhu.gerg.ca>
        <15740.44036.651321.285627@12-248-11-90.client.attbi.com>
        <200209091522.g89FMqL29316@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <15740.58697.697962.77038@12-248-11-90.client.attbi.com>


    Guido> Why would we care about installing a few extra files, as long as
    Guido> they're inside a package?

I guess you needn't worry about that.  It just doesn't seem "clean" to me.

S


From skip@pobox.com  Mon Sep  9 20:19:04 2002
From: skip@pobox.com (Skip Montanaro)
Date: Mon, 9 Sep 2002 14:19:04 -0500
Subject: [Spambayes] deleting "duplicate" spam before training?  good idea
 orbad?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCIECKBDAB.tim.one@comcast.net>
References: <15740.52432.861148.597750@12-248-11-90.client.attbi.com>
        <LNBBLJKPBEHFEDALKOLCIECKBDAB.tim.one@comcast.net>
Message-ID: <15740.62504.533596.995365@12-248-11-90.client.attbi.com>

    >> I wrote a script some time ago to try an minimize the duplicates I
    >> see by calculating a loose checksum, but I still have some
    >> duplicates.  Should I delete the duplicates before training or not?

    Tim> People just can't stop thinking <wink>.  The classifier should work
    Tim> best when trained on a wholly random spattering of real life.  If
    Tim> real life contains duplicates, then that's what the classifier
    Tim> should see.

A bit more detail.  I get destined for many addresses: skip@pobox.com,
skip@calendar.com, concerts@musi-cal.com, webmaster@mojam.com, etc.  I
originally wrote (a slightly different version of) the loosecksum.py script
I'm about to check in to avoid manually scanning all those presumed spams
which are really identical.  Once a message was identified as spam, what I
refer to as a loose checksum was computed to try and avoid saving the same
spam multiple times for later review.

    >> Would people be interested in the script?  I'd be happy to extricate
    >> it from my local modules and check it into CVS.

    Tim> Sure!  I think it's relevant, but maybe for another purpose.  Paul
    Tim> Svensson is thinking harder about real people <wink> than the rest
    Tim> of us, and he may be able to get use out of approaches that
    Tim> identify closely related spam.  For example, some amount of spam is
    Tim> going to end up in the ham training data in real life use, and any
    Tim> sort of similarity score to a piece of known spam may be an aid in
    Tim> finding and purging it.

I'll check it in.  Let me know if you find it useful.

Skip


From gward@python.net  Mon Sep  9 20:25:42 2002
From: gward@python.net (Greg Ward)
Date: Mon, 9 Sep 2002 15:25:42 -0400
Subject: [Spambayes] deleting "duplicate" spam before training?  good idea
	orbad?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCIECKBDAB.tim.one@comcast.net>
References: <15740.52432.861148.597750@12-248-11-90.client.attbi.com>
	<LNBBLJKPBEHFEDALKOLCIECKBDAB.tim.one@comcast.net>
Message-ID: <20020909192542.GB2002@cthulhu.gerg.ca>

On 09 September 2002, Tim Peters said:
> > Would people be interested in the script?  I'd be happy to extricate
> > it from my local modules and check it into CVS.
> 
> Sure!  I think it's relevant, but maybe for another purpose.  Paul Svensson
> is thinking harder about real people <wink> than the rest of us, and he may
> be able to get use out of approaches that identify closely related spam.
> For example, some amount of spam is going to end up in the ham training data
> in real life use, and any sort of similarity score to a piece of known spam
> may be an aid in finding and purging it.

OTOH, look into DCC (Distributed Checksum Clearinghouse,
http://www.rhyolite.com/anti-spam/dcc/), which uses fuzzy checksums.
It's quite likely that DCC's checksumming scheme is better than
something any of us would throw together for personal use (no offense,
Skip!).  But I have no personal experience of it.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
If it can't be expressed in figures, it is not science--it is opinion.

From skip@pobox.com  Mon Sep  9 20:35:04 2002
From: skip@pobox.com (Skip Montanaro)
Date: Mon, 9 Sep 2002 14:35:04 -0500
Subject: [Spambayes] deleting "duplicate" spam before training?  good idea
        orbad?
In-Reply-To: <20020909192542.GB2002@cthulhu.gerg.ca>
References: <15740.52432.861148.597750@12-248-11-90.client.attbi.com>
        <LNBBLJKPBEHFEDALKOLCIECKBDAB.tim.one@comcast.net>
        <20020909192542.GB2002@cthulhu.gerg.ca>
Message-ID: <15740.63464.611324.2220@12-248-11-90.client.attbi.com>


    Greg> OTOH, look into DCC (Distributed Checksum Clearinghouse,
    Greg> http://www.rhyolite.com/anti-spam/dcc/), which uses fuzzy
    Greg> checksums.  It's quite likely that DCC's checksumming scheme is
    Greg> better than something any of us would throw together for personal
    Greg> use (no offense, Skip!).

None taken.  I wrote my little script before I was aware DCC existed.  Even
now, it seems like overkill for my use.

Skip

From gward@python.net  Mon Sep  9 22:01:10 2002
From: gward@python.net (Greg Ward)
Date: Mon, 9 Sep 2002 17:01:10 -0400
Subject: [Spambayes] python.org email harvesting ready to roll
Message-ID: <20020909210110.GA2224@cthulhu.gerg.ca>

[followups to spambayes@python.org please, unless you're specifically
 concerned about some particular bit of email policy for python.org]

OK, after much fiddling with and tweaking of /etc/exim/exim4.conf and
/etc/exim/local_scan.py on mail.python.org, I am fairly confident that
I can start harvesting all incoming email at a moment's notice.  For the
record, here's how it all works:

  * exim4.conf works almost exactly the same as before if the file
    /etc/exim/harvest does not exist.  That is, any "junk mail
    condition" that can be detected by Exim ACLs (access control lists)
    is handled entirely in exim4.conf: the message is rejected before it
    ever gets to local_scan.py.  This covers such diverse cases as
    "message from known spammer" (reject after every RCPT TO command),
    "no message-id header", and "8-bit chars in subject" (both rejected
    after the message headers/body are read).

    The main things I have changed in the absence of /etc/exim/harvest
    are:
      - don't check for 8-bit chars in "From" header -- the vast
        majority of hits for this test were bounces from some
        Asian ISP; the remaining hits should be handled by SpamAssassin
      - do header sender verification (ie. ensure that there's a
        verifiable email address in at least one of "From", "Reply-to",
        and "Sender") as late as possible, because it requires DNS
        lookups which can be slow (and can also make messages that
        should have been rejected merely be deferred, if those DNS
        lookups timeout)

  * if /etc/exim/harvest exists, then the behaviour of all of those
    ACLs in exim4.conf suddenly changes: instead of rejecting recipients
    or messages, they add an X-reject header to the message.  This
    header is purely for internal use; it records the name of the folder
    to which the rejected message should be saved, and also gives the
    SMTP error message which should ultimately be used to reject
    the message.

    Thus, those messages will now be seen by local_scan.py, which now
    looks for the X-reject header.  If found, it uses the folder name
    specified there to save the message, and then rejects it with the
    SMTP error message also given in X-reject.  (Currently X-reject is
    retained in saved messages.)

    If a message was not tagged with X-reject, then local_scan.py
    runs the usual virus and spam checks.  (Namely, my homebrew
    scan for attachments with filenames that look like Windows
    executables, and a run through SpamAssassin.)  The logic is
    basically this:
      if virus:
          folder = "virus"
      else:
          run through SpamAssassin
          if score >= 10.0:
              folder = "rejected-spam"
          elif score >= 5.0:
              folder = "caught-spam"

    Finally, local_scan.py writes the message to the designated folder.
    By far the biggest folder will be "accepted" -- the server handles
    2000-5000 incoming messages per day, of which maybe 100-500 are junk
    mail.  (Oops, just realized I haven't written the code that actually
    saves the message -- d'ohh!  Also haven't written anything to
    discriminate personal email, which I must do.  Sigh.)

  * finally, the big catch: waiting until after you've read the message
    headers and body to actually reject the message is problematic,
    because certain broken MTAs (including those used by some spammers)
    don't consider a 5xx after DATA as a permanent error, but keep
    retrying.  D'ohh.  This is a minor annoyance currently, where a fair
    amount of stuff is rejected at RCPT TO time.  But in harvest mode,
    *everything* (with the exception of people probing for open relays)
    will be rejected at DATA time.  So I have cooked up something called
    the ASBL, or automated sender blacklist.  This is just a Berkeley DB
    file that maps (sender_ip, sender_address) to an expiry time.  When
    local_scan() rejects a message from (sender_ip, sender_address) --
    for whatever reason, including finding an X-reject header added by
    an ACL in exim4.conf -- it adds a record to the ASBL, with an expiry
    time 3 hours in the future.  Meanwhile, there's an ACL in exim4.conf
    that checks for records in the ASBL; if there's a record for the
    current (sender_ip, sender_address) that hasn't expired yet, we
    reject all recipients without ever looking at the message headers or
    body.

    The downside of this from the point-of-view of corpus collection is
    that if some jerk is busily spamming *@python.org, one SMTP
    connection per address, we will most likely only get one copy.  This
    is a win if you're just thinking about reducing server load and
    bandwidth, but I'm not sure if it's helpful for training spam
    detectors.  Tim?

Happy harvesting --

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Budget's in the red?  Let's tax religion!
    -- Dead Kennedys

From tim.one@comcast.net  Tue Sep 10 04:18:25 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 09 Sep 2002 23:18:25 -0400
Subject: [Spambayes] Current histograms
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCEPNBCAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEFBBDAB.tim.one@comcast.net>

We've not only reduced the f-p and f-n rates in my test runs, we've also
made the score distributions substantially sharper.  This is bad news for
Greg, because the non-existent "middle ground" is becoming even less
existent <wink>:

Ham distribution for all runs:
* = 1333 items
  0.00 79975 ************************************************************
  2.50     1 *
  5.00     0
  7.50     0
 10.00     2 *
 12.50     1 *
 15.00     0
 17.50     0
 20.00     0
 22.50     1 *
 25.00     0
 27.50     0
 30.00     0
 32.50     0
 35.00     0
 37.50     1 *
 40.00     0
 42.50     0
 45.00     0
 47.50     0
 50.00     0
 52.50     0
 55.00     0
 57.50     0
 60.00     1 *
 62.50     0
 65.00     1 *
 67.50     0
 70.00     0
 72.50     0
 75.00     0
 77.50     0
 80.00     0
 82.50     0
 85.00     0
 87.50     0
 90.00     0
 92.50     0
 95.00     0
 97.50    17 *

Spam distribution for all runs:
* = 914 items
  0.00   118 *
  2.50     7 *
  5.00     0
  7.50     2 *
 10.00     1 *
 12.50     1 *
 15.00     3 *
 17.50     1 *
 20.00     1 *
 22.50     1 *
 25.00     0
 27.50     0
 30.00     4 *
 32.50     3 *
 35.00     4 *
 37.50     2 *
 40.00     0
 42.50     1 *
 45.00     1 *
 47.50     0
 50.00     2 *
 52.50     3 *
 55.00     1 *
 57.50     2 *
 60.00     0
 62.50     1 *
 65.00     1 *
 67.50    10 *
 70.00     2 *
 72.50     1 *
 75.00     2 *
 77.50     1 *
 80.00     0
 82.50     0
 85.00     1 *
 87.50     4 *
 90.00     2 *
 92.50     5 *
 95.00    14 *
 97.50 54806 ************************************************************

As usual for me, this is an aggregate of 20 runs, each both training and
predicting on 4000 c.l.py ham + ~2750 BruceG spam.

Only 25 ham scores out of 80,000 are above 0.025 now (and, yes, the
"Nigerian scam"-quoting msg is still counted as ham -- I haven't taken
anything out of the ham corpus since remving the "If AOL were a car" spam),
the f-p rate wouldn't have changed at all if the spamprob cutoff were
dropped from 0.90 to 0.675, dropping the cutoff to 0.40 would have added
only 2 false positives, and dropping it to 0.15 would have added only
another 2 more!

It's spooky.


From anthony@interlink.com.au  Tue Sep 10 10:29:19 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Tue, 10 Sep 2002 19:29:19 +1000
Subject: [Spambayes] Current histograms 
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEFBBDAB.tim.one@comcast.net> 
Message-ID: <200209100929.g8A9TJV27347@localhost.localdomain>


>>> Tim Peters wrote
> We've not only reduced the f-p and f-n rates in my test runs, we've also
> made the score distributions substantially sharper.  This is bad news for
> Greg, because the non-existent "middle ground" is becoming even less
> existent <wink>:

Well, I've finally got around to pulling down the SF code. Starting
with it, and absolutely zero local modifications, I see the following:

Ham distribution for all runs:
* = 589 items
  0.00 35292 ************************************************************
  2.50    36 *
  5.00    21 *
  7.50    12 *
 10.00     6 *
 12.50     9 *
 15.00     6 *
 17.50     3 *
 20.00     8 *
 22.50     5 *
 25.00     3 *
 27.50    18 *
 30.00     9 *
 32.50     1 *
 35.00     4 *
 37.50     3 *
 40.00     0 
 42.50     3 *
 45.00     3 *
 47.50     4 *
 50.00     9 *
 52.50     5 *
 55.00     5 *
 57.50     3 *
 60.00     4 *
 62.50     2 *
 65.00     2 *
 67.50     6 *
 70.00     1 *
 72.50     3 *
 75.00     2 *
 77.50     4 *
 80.00     3 *
 82.50     3 *
 85.00     6 *
 87.50     8 *
 90.00     4 *
 92.50     8 *
 95.00    15 *
 97.50   441 *

Spam distribution for all runs:
* = 504 items
  0.00   393 *
  2.50    17 *
  5.00    18 *
  7.50    12 *
 10.00     4 *
 12.50     6 *
 15.00    11 *
 17.50    10 *
 20.00    10 *
 22.50     5 *
 25.00     3 *
 27.50    19 *
 30.00     8 *
 32.50     2 *
 35.00     0 
 37.50     1 *
 40.00     5 *
 42.50     5 *
 45.00     7 *
 47.50     2 *
 50.00     5 *
 52.50     1 *
 55.00     9 *
 57.50    11 *
 60.00     6 *
 62.50     4 *
 65.00     3 *
 67.50     5 *
 70.00     7 *
 72.50     9 *
 75.00     2 *
 77.50    13 *
 80.00     3 *
 82.50     7 *
 85.00    15 *
 87.50    16 *
 90.00    11 *
 92.50    16 *
 95.00    45 *
 97.50 30226 ************************************************************


My next (current) task is to complete the corpus I've got - it's currently
got ~ 9000 ham, 7800 spam, and about 9200 currently unsorted. I'm tossing 
up using either hammie or spamassassin to do the initial sort  - previously
I've used various forms of 'grep' for keywords and a little gui thing to 
pop a message up and let me say 'spam/ham', but that's just getting too, too
tedious.

I can't make it available en masse, but I will look at finding some of
the more 'interesting' uglies. One thing I've seen (consider this 
'anecdotal' for now) is that the 'skip' tokens end up in a _lot_ of the 
f-ps.

Anthony

From skip@pobox.com  Tue Sep 10 17:36:57 2002
From: skip@pobox.com (Skip Montanaro)
Date: Tue, 10 Sep 2002 11:36:57 -0500
Subject: [Spambayes] timtest broke?
Message-ID: <15742.8105.960711.715561@12-248-11-90.client.attbi.com>


After my latest cvs up, timtest fails with

    Traceback (most recent call last):
      File "/home/skip/src/spambayes/timtest.py", line 294, in ?
        drive(nsets)
      File "/home/skip/src/spambayes/timtest.py", line 264, in drive
        d = Driver()
      File "/home/skip/src/spambayes/timtest.py", line 152, in __init__
        self.global_ham_hist = Hist(options.nbuckets)
    AttributeError: 'OptionsClass' object has no attribute 'nbuckets'

I'm running it as

    timtest -n5 > Data/timtest.out

from my ~/Mail directory (not from my ~/src/spambayes directory).  If I
create a symlink to ~/src/spambayes/bayes.ini it works once again, but
shouldn't there be an nbuckets attribute with a default value already?

Skip


From tim.one@comcast.net  Tue Sep 10 18:12:51 2002
From: tim.one@comcast.net (Tim Peters)
Date: Tue, 10 Sep 2002 13:12:51 -0400
Subject: [Spambayes] timtest broke?
In-Reply-To: <15742.8105.960711.715561@12-248-11-90.client.attbi.com>
Message-ID: <BIEJKCLHCIOIHAGOKOLHIEHJDKAA.tim.one@comcast.net>

[Skip Montanaro]
> After my latest cvs up, timtest fails with
>
>     Traceback (most recent call last):
>       File "/home/skip/src/spambayes/timtest.py", line 294, in ?
>         drive(nsets)
>       File "/home/skip/src/spambayes/timtest.py", line 264, in drive
>         d = Driver()
>       File "/home/skip/src/spambayes/timtest.py", line 152, in __init__
>         self.global_ham_hist = Hist(options.nbuckets)
>     AttributeError: 'OptionsClass' object has no attribute 'nbuckets'
>
> I'm running it as
>
>     timtest -n5 > Data/timtest.out
>
> from my ~/Mail directory (not from my ~/src/spambayes directory).  If I
> create a symlink to ~/src/spambayes/bayes.ini it works once again, but
> shouldn't there be an nbuckets attribute with a default value already?

I never used ConfigParser before, but I read that its read() method silently
ignores files that don't exist.  If 'bayes.ini' isn't found, *none* of the
options will be defined.  Since you want to run this from a directory other
than my spambayes directory, it's up to you to check in changes to make that
possible <wink>.


From tim.one@comcast.net  Tue Sep 10 19:26:05 2002
From: tim.one@comcast.net (Tim Peters)
Date: Tue, 10 Sep 2002 14:26:05 -0400
Subject: [Spambayes] timtest broke?
Message-ID: <ac49dabc2f.abc2fac49d@icomcast.net>

[Tim]
> I never used ConfigParser before, but I read that its read() 
> method silently ignores files that don't exist.  If 'bayes.ini'
> isn't found, *none* of the options will be defined. ...

Note that I since got rid of bayes.ini (it's embedded in Options.py 
now), so search-path issues won't burn you here anymore.  The intended 
way to customize the tokenizer and testers is via creating your own 
bayescustomize.ini.  You'll get burned by search-path issues wrt that 
instead now <0.7 wink>.


From tim.one@comcast.net  Tue Sep 10 20:01:36 2002
From: tim.one@comcast.net (Tim Peters)
Date: Tue, 10 Sep 2002 15:01:36 -0400
Subject: [Spambayes] Current histograms
In-Reply-To: <200209100929.g8A9TJV27347@localhost.localdomain>
Message-ID: <BIEJKCLHCIOIHAGOKOLHEEHMDKAA.tim.one@comcast.net>

[Anthony Baxter]
> Well, I've finally got around to pulling down the SF code. Starting
> with it, and absolutely zero local modifications, I see the following:

How many runs is this summarizing?  For each, how many ham&spam were in the
training set?  How many in the prediction sets?  What were the error rates
(run rates.py over your output file)?

The effect of set sizes on accuracy rates isn't known.  I've informally
reported some results from just a few controlled experiments on that.
Jeremy reported improved accuracy by doubling the training set size, but
that wasn't a controlled experiment (things besides just training set size
changed between "before" and "after").

> Ham distribution for all runs:
> * = 589 items
>   0.00 35292 ************************************************************
>   2.50    36 *
>   5.00    21 *
>   7.50    12 *
>  10.00     6 *
> ...
>  90.00     4 *
>  92.50     8 *
>  95.00    15 *
>  97.50   441 *
>
> Spam distribution for all runs:
> * = 504 items
>   0.00   393 *
>   2.50    17 *
>   5.00    18 *
>   7.50    12 *
>  10.00     4 *
> ...
>  90.00    11 *
>  92.50    16 *
>  95.00    45 *
>  97.50 30226 ************************************************************
>
>
> My next (current) task is to complete the corpus I've got - it's currently
> got ~ 9000 ham, 7800 spam, and about 9200 currently unsorted. I'm tossing
> up using either hammie or spamassassin to do the initial sort  -
previously
> I've used various forms of 'grep' for keywords and a little gui thing to
> pop a message up and let me say 'spam/ham', but that's just getting too,
too
> tedious.

Yup, tagging data is mondo tedious, and mistakes hurt.

I expect hammie will do a much better job on this already than hand
grepping.  Be sure to stare at the false positives and get the spam out of
there.

> I can't make it available en masse, but I will look at finding some of
> the more 'interesting' uglies. One thing I've seen (consider this
> 'anecdotal' for now) is that the 'skip' tokens end up in a _lot_ of the
> f-ps.

With probabilities favoring ham or spam?  A skip token is produced in lieu
of "word" more than 12 chars long and without any high-bit characters.  It's
possible that they helped me because raw HTML produces lots of these.
However, if you're running current CVS, Tokenizer/retain_pure_html_tags
defaults to False now, so HTML decorations should vanish before body
tokenization.


From tim.one@comcast.net  Wed Sep 11 04:11:25 2002
From: tim.one@comcast.net (Tim Peters)
Date: Tue, 10 Sep 2002 23:11:25 -0400
Subject: [Spambayes] XTreme Training
In-Reply-To: <200209091410.g89EA0428928@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEGPBDAB.tim.one@comcast.net>

Let's get silly:

    remove a ham at random from the set of all ham
    remove a spam at random from the set of all spam
    train a new classifier on just those two
    pick and remove another spam + ham at random
    score just those two
    repeat the above a total of 1000 times

Note that the only "probabilities" that a word can get then are 0.01, 0.99,
and 0.5 (trust me <wink>).  Is it better than flipping a coin?  Yup!  In
fact, it's curious how closely it matches rolling a die.  Tnis is on my
20,000 c.l.py ham + 13,752 BruceG spam collection, with options

[TestDriver]
show_histograms = True
show_ham_lo = 1.0
show_best_discriminators = False
save_trained_pickles = False
show_spam_lo = 1.0
show_false_positives = False
show_ham_hi = 0.0
pickle_basename = class
show_false_negatives = False
nbuckets = 10
show_spam_hi = 0.0

[Tokenizer]
safe_headers = abuse-reports-to
        date
        errors-to
        from
        importance
        in-reply-to
        message-id
        mime-version
        organization
        received
        reply-to
        return-path
        subject
        to
        user-agent
        x-abuse-info
        x-complaints-to
        x-face
mine_received_headers = False
retain_pure_html_tags = True
count_all_header_lines = False

Here's what happens:

Ham distribution for all runs:
* = 11 items
  0.00 630 **********************************************************
 10.00   0
 20.00   0
 30.00   0
 40.00   0
 50.00  57 ******
 60.00   0
 70.00   0
 80.00   0
 90.00 313 *****************************

Spam distribution for all runs:
* = 14 items
  0.00 176 *************
 10.00   0
 20.00   0
 30.00   0
 40.00   0
 50.00  38 ***
 60.00   0
 70.00   0
 80.00   0
 90.00 786 *********************************************************

It's better at nailing spam than ham, suggesting something <wink>.  The odds
that ham will score near 0 are about 2/3, and near 1 about 1/3.  But for
spam the odds of scoring near 1 are about 5/6, and near 0 about 1/6.  The
f-p rate is over 31%, but the f-n rate is under 22% (a score of 90 is the
cutoff point for spam).

Training on 2 ham + 2 spam (but still scoring only 1 of each) helps a lot:

Ham distribution for all runs:
* = 12 items
  0.00 710 ************************************************************
 10.00  22 **
 20.00   0
 30.00   8 *
 40.00   1 *
 50.00   1 *
 60.00   3 *
 70.00   1 *
 80.00   9 *
 90.00 245 *********************

Spam distribution for all runs:
* = 15 items
  0.00 112 ********
 10.00   7 *
 20.00   0
 30.00   3 *
 40.00   0
 50.00   7 *
 60.00   3 *
 70.00   5 *
 80.00   2 *
 90.00 861 **********************************************************

f-p down to 24.5%, f-n down to 13.9%.  Boosting the training set to 3 each
makes those 22.3% and 8.8% (not much help for f-p!):

Ham distribution for all runs:
* = 13 items
  0.00 743 **********************************************************
 10.00   8 *
 20.00   8 *
 30.00   3 *
 40.00   4 *
 50.00   0
 60.00   2 *
 70.00   2 *
 80.00   7 *
 90.00 223 ******************

Spam distribution for all runs:
* = 16 items
  0.00  69 *****
 10.00   1 *
 20.00   3 *
 30.00   1 *
 40.00   4 *
 50.00   4 *
 60.00   0
 70.00   2 *
 80.00   4 *
 90.00 912 *********************************************************

Boosting to 4 gives a larger benefit to f-p, suggesting that maybe the one
run I did at 3 may have been atypical.  But the f-n rate doesn't move much
now, and neither are helped much more by boosting to 5:

Ham distribution for all runs:
* = 14 items
  0.00 787 *********************************************************
 10.00   7 *
 20.00   2 *
 30.00   9 *
 40.00   1 *
 50.00   3 *
 60.00   2 *
 70.00   3 *
 80.00   4 *
 90.00 182 *************

Spam distribution for all runs:
* = 16 items
  0.00  62 ****
 10.00   3 *
 20.00   1 *
 30.00   0
 40.00   1 *
 50.00   7 *
 60.00   2 *
 70.00   0
 80.00   6 *
 90.00 918 **********************************************************

Training on 5 of each:

Ham distribution for all runs:
* = 14 items
  0.00 796 *********************************************************
 10.00   5 *
 20.00   3 *
 30.00   5 *
 40.00   1 *
 50.00   3 *
 60.00   2 *
 70.00   1 *
 80.00   6 *
 90.00 178 *************

Spam distribution for all runs:
* = 16 items
  0.00  54 ****
 10.00   3 *
 20.00   3 *
 30.00   1 *
 40.00   2 *
 50.00   4 *
 60.00   0
 70.00   3 *
 80.00   2 *
 90.00 928 **********************************************************

I guess the moral of the story is that unless you have a lot of ham and
spam, you may as well stop when you've trained on 4 of each <0.99 wink>.

On this corpus, we know that at *some* point before boosting training to
4000 ham + ~2750 spam, the f-p rate eventually gets much better than the f-n
rate, and that the ham distribution eventually gets tighter than the spam
distribution.  I don't know where the crossover point is, and this
methodology can't find it (by doing random selection without replacement,
just doing the last train-on-5 run used 1000*6 = 6000 = close to a half of
my total spam collection).

At the other extreme, training on half my ham&spam, and scoring aginst the
other half ("all runs" here is a single run):

Ham distribution for all runs:
* = 167 items
  0.00 9998 ************************************************************
 10.00    0
 20.00    0
 30.00    0
 40.00    0
 50.00    0
 60.00    0
 70.00    0
 80.00    0
 90.00    1 *

Spam distribution for all runs:
* = 115 items
  0.00   20 *
 10.00    1 *
 20.00    0
 30.00    0
 40.00    0
 50.00    1 *
 60.00    0
 70.00    0
 80.00    3 *
 90.00 6850 ************************************************************

[Note:  a bug in the driver caused one ham and one spam to get dropped on
the floor; that's why the top histogram only shows 9999 results; the bottom
is missing a spam too]

    false positive rate: 0.0100%
    false negative rate: 0.3636%

so that the f-n rate is 30x worse than the f-p rate, and both are
substantially lower than I get with training on "my usual" 4000 ham + 2750
spam.  It's a good sign that the system keeps getting more the more I train
on it with.

Something else to note is that 3 of Anthony's "just count the # of header
lines" gimmicks are in the top 10 discriminators (note that due to the
multi-source nature of my corpora, the only ones I *tried* to count are
listed as the value of option safe_headers above):

        'subject:Python' 2168 0.01
        '</body>' 2245 0.99
        'header:User-Agent:1' 2287 0.01
        '</html>' 2436 0.99
        '<html>' 2616 0.99
        'skip:< 10' 2956 0.99
        'wrote:' 4342 0.01
        'header:X-Complaints-To:1' 4401 0.01
        'python' 4977 0.01
        'header:Organization:1' 6916 0.01

Alas, all 4 of the 0.99 clues there are HTML-related.  Indeed, the only 0.99
clue in the top 30 discriminators that *wasn't* HTML-related in this run was
8 occurrences of a Received header:

        'header:Received:8' 1331 0.99

Double alas, I'm not entirely sure that one isn't an artifact of BruceG's
collection procedure.


From tim.one@comcast.net  Wed Sep 11 04:46:15 2002
From: tim.one@comcast.net (Tim Peters)
Date: Tue, 10 Sep 2002 23:46:15 -0400
Subject: [Spambayes] XTreme Training
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEGPBDAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCOEHDBDAB.tim.one@comcast.net>

[Tim]
> ...
> At the other extreme, training on half my ham&spam, and scoring aginst
> the other half
> ...
>     false positive rate: 0.0100%
>     false negative rate: 0.3636%
> ...
> Alas, all 4 of the 0.99 clues there are HTML-related.

That begged to try it again but with Tokenize/retain_pure_html_tags false.
The random halves getting trained on and scored against are different here,
and I repaired the bug that dropped 1 ham and 1 spam on the floor, so this
isn't exactly a 1-change difference between runs.

Ham distribution for all runs:
* = 167 items
  0.00 9999 ************************************************************
 10.00    0
 20.00    0
 30.00    0
 40.00    0
 50.00    0
 60.00    0
 70.00    0
 80.00    0
 90.00    1 *

Spam distribution for all runs:
* = 115 items
  0.00   21 *
 10.00    0
 20.00    0
 30.00    1 *
 40.00    0
 50.00    0
 60.00    1 *
 70.00    0
 80.00    1 *
 90.00 6852 ************************************************************

    false positive rate: 0.0100%
    false negative rate: 0.3490%

Yay!  That may mean that HTML tags aren't really needed in my test data
provided it's trained on enough stuff.  Curiously, the sole false positive
here is the same as the sole false positive on the half&half run reported in
the preceding msg (I assume the Nigerian scam "false positive" just happened
to end up in the training data both times):

************************************************************************
Data/Ham/Set4/107687.txt
prob = 0.999632042904
prob('python.') = 0.01
prob('alteration') = 0.01
prob('edinburgh') = 0.01
prob('subject:Python') = 0.01
prob('header:Errors-To:1') = 0.0216278
prob('thanks,') = 0.0319955
prob('help?') = 0.041806
prob('road,') = 0.0462364
prob('there,') = 0.0722794
prob('us.') = 0.906609
prob('our') = 0.919118
prob('company,') = 0.921852
prob('visit') = 0.930785
prob('sent.') = 0.939882
prob('e-mail') = 0.949765
prob('courses') = 0.954726
prob('received') = 0.955209
prob('analyst') = 0.960756
prob('investment') = 0.975139
prob('regulated') = 0.99
prob('e-mails') = 0.99
prob('mills') = 0.99

Received: from [195.171.5.71] (helo=node401.dmz.standardlife.com)
        by mail.python.org with esmtp (Exim 3.21 #1)
        id 15rDsu-00085k-00
        for python-list@python.org; Wed, 10 Oct 2001 03:34:32 -0400
Received: from slukdcn4.internal.standardlife.com (slukdcn4.standardlife.com
        [10.3.2.72])
        by node401.dmz.standardlife.com (Pro-8.9.3/Pro-8.9.3) with SMTP id
        IAA53660for <python-list@python.org>; Wed, 10 Oct 2001 08:34:00
+0100
Received: from sl079320 ([172.31.88.231]) by
        slukdcn4.internal.standardlife.com (Lotus SMTP MTA v4.6.6  (890.1
7-16-1999))
        with SMTP id 80256AE1.00294B60; Wed, 10 Oct 2001 08:31:02 +0100
Message-ID:
<007e01c1515d$bb255940$e7581fac@sl079320.internal.standardlife.com>
From: "Vickie Mills" <vickie_mills@standardlife.com>
To: <python-list@python.org>
Subject: Training Courses in Python in UK
Date: Wed, 10 Oct 2001 08:32:30 +0100
MIME-Version: 1.0
Content-Type: text/plain;
        charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 4.72.3155.0
X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3155.0
Sender: python-list-admin@python.org
Errors-To: python-list-admin@python.org
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.0.6 (101270)
Precedence: bulk
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Post: <mailto:python-list@python.org>
List-Subscribe: <http://mail.python.org/mailman/listinfo/python-list>,
        <mailto:python-list-request@python.org?subject=subscribe>
List-Id: General discussion list for the Python programming language
        <python-list.python.org>
List-Unsubscribe: <http://mail.python.org/mailman/listinfo/python-list>,
        <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>

Hi there,

I am looking for you recommendations on training courses available in the UK
on Python.  Can you help?

Thanks,

Vickie Mills
IS Training Analyst

Tel:    0131 245 1127
Fax:    0131 245 1550
E-mail:    vickie_mills@standardlife.com

For more information on Standard Life, visit our website
http://www.standardlife.com/   The Standard Life Assurance Company, Standard
Life House, 30 Lothian Road, Edinburgh EH1 2DH, is registered in Scotland
(No SZ4) and regulated by the Personal Investment Authority.  Tel: 0131 225
2552 - calls may be recorded or monitored.  This confidential e-mail is for
the addressee only.  If received in error, do not retain/copy/disclose it
without our consent and please return it to us.  We virus scan all e-mails
but are not responsible for any damage caused by a virus or alteration by a
third party after it is sent.
************************************************************************

The top 30 discriminators are more interesting now:

        'income' 629 0.99
        'http0:python' 643 0.01
        'header:MiME-Version:1' 672 0.99
        'http1:remove' 693 0.99
        'content-type:text/html' 711 0.982345
        'string' 714 0.01
        'http>1:jpg' 776 0.99
        'object' 813 0.01
        'python,' 852 0.01
        'python.' 882 0.01
        'language' 883 0.01
        '>>>' 907 0.01
        'header:Return-Path:2' 907 0.99
        'unsubscribe' 975 0.99
        'header:Received:7' 1113 0.99
        'def' 1142 0.01
        'http>1:gif' 1168 0.99
        'module' 1169 0.01
        'import' 1332 0.01
        'header:Received:8' 1342 0.99
        'header:Errors-To:1' 1377 0.0216278
        'header:In-Reply-To:1' 1402 0.01
        'wrote' 1753 0.01
        '&nbsp;' 2067 0.99
        'subject:Python' 2140 0.01
        'header:User-Agent:1' 2322 0.01
        'header:X-Complaints-To:1' 4351 0.01
        'wrote:' 4370 0.01
        'python' 4972 0.01
        'header:Organization:1' 6921 0.01

There are still two HTML clues remaining there ("&nbsp;" and
"content-type:text/html").  Anthony's trick accounts for almost a third of
these.  "Python" appears in 5 of them ('http0:python' means that 'python'
was found in the 1st field of an embedded http:// URL).  Sticking a .gif or
a .jpg in  a URL both score as 0.99 spam clues.  Note the damning pattern of
capitalization in 'header:MiME-Version:1'!  This counting is case-sensitive,
and nobody ever would have guessed that MiME is more damning than SUBJECT or
DATE.  Why would spam be likely to end up with two instances of Return-Path
in the headers?


From anthony@interlink.com.au  Wed Sep 11 05:16:32 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Wed, 11 Sep 2002 14:16:32 +1000
Subject: [Spambayes] XTreme Training 
In-Reply-To: <LNBBLJKPBEHFEDALKOLCOEHDBDAB.tim.one@comcast.net> 
Message-ID: <200209110416.g8B4GWp07653@localhost.localdomain>


>>> Tim Peters wrote
> Why would spam be likely to end up with two instances of Return-Path
> in the headers?

Because idiot spammers copy message headers blindly, and put this 
header in - then the destination MTA puts in another. 

This is one of the strong points of the header-counting trick - it 
picks up a bunch of these nasties. E.g. X-UIDL is a POP header -
so if your email doesn't get picked up with POP, then you can be
fairly sure it's a badspamnodonut case if you see it in the message
headers.

In a world where the spammers evolved as fast as the Klez virus,
these wouldn't last very long - but in the spamiverse they're
much much stupider than that.

Anthony
-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.


From anthony@interlink.com.au  Wed Sep 11 05:23:38 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Wed, 11 Sep 2002 14:23:38 +1000
Subject: [Spambayes] Current histograms 
In-Reply-To: <BIEJKCLHCIOIHAGOKOLHEEHMDKAA.tim.one@comcast.net> 
Message-ID: <200209110423.g8B4Ndh07749@localhost.localdomain>


> How many runs is this summarizing?  For each, how many ham&spam were in the
> training set?  How many in the prediction sets?  What were the error rates
> (run rates.py over your output file)?

5 sets, each of 1800ham/1550spam, just ran the once (it matched all 5 to
each other...)

rates.py sez:

Training on Data/Ham/Set1 & Data/Spam/Set1 ... 1798 hams & 1548 spams
      0.445   0.388
      0.445   0.323
      2.108   4.072
      0.556   1.097
Training on Data/Ham/Set2 & Data/Spam/Set2 ... 1798 hams & 1546 spams
      2.113   0.517
      1.335   0.194
      3.106   5.365
      2.113   2.903
Training on Data/Ham/Set3 & Data/Spam/Set3 ... 1798 hams & 1547 spams
      2.447   0.646
      0.945   0.388
      2.884   3.426
      2.058   1.097
Training on Data/Ham/Set4 & Data/Spam/Set4 ... 1803 hams & 1547 spams
      1.057   2.584
      0.723   1.682
      0.890   1.164
      0.445   0.452
Training on Data/Ham/Set5 & Data/Spam/Set5 ... 1798 hams & 1550 spams
      0.779   4.328
      0.501   3.299
      0.667   3.361
      0.388   4.977
total false pos 273 3.03501945525
total false neg 367 4.74282760403

> The effect of set sizes on accuracy rates isn't known.  I've informally
> reported some results from just a few controlled experiments on that.
> Jeremy reported improved accuracy by doubling the training set size, but
> that wasn't a controlled experiment (things besides just training set size
> changed between "before" and "after").

I'll try with 2 sets with half the messages each.

> Yup, tagging data is mondo tedious, and mistakes hurt.
> 
> I expect hammie will do a much better job on this already than hand
> grepping.  Be sure to stare at the false positives and get the spam out of
> there.

Yah, but there's a chicken-and-egg problem there - I want stuff that's
_known_ to be right to test this stuff, so using the spambayes code to
tell me whether it's spam is not going to help.


> With probabilities favoring ham or spam?  A skip token is produced in lieu
> of "word" more than 12 chars long and without any high-bit characters.  It's
> possible that they helped me because raw HTML produces lots of these.
> However, if you're running current CVS, Tokenizer/retain_pure_html_tags
> defaults to False now, so HTML decorations should vanish before body
> tokenization.

Yep, it shows up in a lot of spam, but also in different forms in hams. 
But the hams each manage to pick a different variant of 
~~~~~~~~~~~~~~~~~~~~~~
or whatever - so they don't end up counteracting the various bits in the
spam.

Looking further, a _lot_ of the bad skip rubbish is coming from uuencoded
viruses &c in the spam-set.

Anthony


From neale@woozle.org  Wed Sep 11 06:14:36 2002
From: neale@woozle.org (Neale Pickett)
Date: 10 Sep 2002 22:14:36 -0700
Subject: [Spambayes] stack.pop() ate my multipart message
Message-ID: <w53lm6985vn.fsf@woozle.org>

I've been running hammie on all my incoming messages, and I noticed that
multipart/alternative messages are totally hosed: they have no content,
just the MIME boundaries.  For instance, the following message:

------------------------------8<------------------------------
From: somebody <someone@somewhere.org>
To: neale@woozle.org
Subject: Booga
Content-type: multipart/alternative; boundary="snot"

This is a multi-part message in MIME format.

--snot
Content-type: text/plain; charset=iso-8859-1
Content-transfer-encoding: 7BIT

Hi there.
--snot
Content-type: text/html; charset=iso-8859-1
Content-transfer-encoding: 7BIT

<pre>Hi there.</pre>
--snot--
------------------------------8<------------------------------

Comes out like this:

------------------------------8<------------------------------
From: somebody <someone@somewhere.org>
To: neale@woozle.org
Subject: Booga
Content-type: multipart/alternative; boundary="snot"
X-Hammie-Disposition: No; 0.74; [unrelated gar removed]

This is a multi-part message in MIME format.

--snot

--snot--
------------------------------8<------------------------------

I'm using "Python 2.3a0 (#1, Sep  9 2002, 22:56:24)".

I've fixed it with the following patch to Tim's tokenizer, but I have to
admit that I'm baffled as to why it works.  Maybe there's some subtle
interaction between generators and lists that I can't understand.  Or
something.  Being as I'm baffled, I don't imagine any theory I come up
with will be anywhere close to reality.

In any case, be advised that (at least for me) hammie will eat
multipart/alternative messages until this patch is applied.  The patch
seems rather bogus though, so I'm not checking it in, in the hope that
there's a better fix I just wasn't capable of discovering :)

------------------------------8<------------------------------
Index: tokenizer.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/tokenizer.py,v
retrieving revision 1.15
diff -u -r1.15 tokenizer.py
--- tokenizer.py	10 Sep 2002 18:15:49 -0000	1.15
+++ tokenizer.py	11 Sep 2002 05:01:16 -0000
@@ -1,3 +1,4 @@
+#! /usr/bin/env python
 """Module to tokenize email messages for spam filtering."""
 
 import email
@@ -507,7 +508,8 @@
             htmlpart = textpart = None
             stack = part.get_payload()
             while stack:
-                subpart = stack.pop()
+                subpart = stack[0]
+                stack = stack[1:]
                 ctype = subpart.get_content_type()
                 if ctype == 'text/plain':
                     textpart = subpart
------------------------------8<------------------------------



From neale@woozle.org  Wed Sep 11 08:10:29 2002
From: neale@woozle.org (Neale Pickett)
Date: 11 Sep 2002 00:10:29 -0700
Subject: [Spambayes] stack.pop() ate my multipart message
In-Reply-To: <w53lm6985vn.fsf@woozle.org>
References: <w53lm6985vn.fsf@woozle.org>
Message-ID: <w53znup6ly2.fsf@woozle.org>

So then, Neale Pickett <neale@woozle.org> is all like:

> Maybe there's some subtle interaction between generators and lists
> that I can't understand.  Or something.  Being as I'm baffled, I don't
> imagine any theory I come up with will be anywhere close to reality.

And then, just as I was about to fall asleep, I figured it out.  The
tokenizer now has an extra [:], and all is well.  I feel like a real
chawbacon for not realizing this earlier.  :*)

Blaming it on staying up past bedtime,

Neale

From gward@python.net  Wed Sep 11 13:23:08 2002
From: gward@python.net (Greg Ward)
Date: Wed, 11 Sep 2002 08:23:08 -0400
Subject: [Spambayes] XTreme Training
In-Reply-To: <LNBBLJKPBEHFEDALKOLCOEHDBDAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCKEGPBDAB.tim.one@comcast.net>
	<LNBBLJKPBEHFEDALKOLCOEHDBDAB.tim.one@comcast.net>
Message-ID: <20020911122308.GB5866@cthulhu.gerg.ca>

On 10 September 2002, Tim Peters said:
> Why would spam be likely to end up with two instances of Return-Path
> in the headers?

Possibly another qmail-ism from Bruce Guenter's spam collection.  Or
maybe Anthony's right about spammers being stupid and blindly copying
headers.  (Well, of course he's right about spammers being stupid; it's
just this particular aspect of stupidity that's open to question.)

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Think honk if you're a telepath.

From tim.one@comcast.net  Thu Sep 12 01:13:06 2002
From: tim.one@comcast.net (Tim Peters)
Date: Wed, 11 Sep 2002 20:13:06 -0400
Subject: [Spambayes] Current histograms
In-Reply-To: <200209110423.g8B4Ndh07749@localhost.localdomain>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEJBBDAB.tim.one@comcast.net>

[Anthony Baxter]
> 5 sets, each of 1800ham/1550spam, just ran the once (it matched all 5 to
> each other...)
>
> rates.py sez:
>
> Training on Data/Ham/Set1 & Data/Spam/Set1 ... 1798 hams & 1548 spams
>       0.445   0.388
>       0.445   0.323
>       2.108   4.072
>       0.556   1.097
> Training on Data/Ham/Set2 & Data/Spam/Set2 ... 1798 hams & 1546 spams
>       2.113   0.517
>       1.335   0.194
>       3.106   5.365
>       2.113   2.903
> Training on Data/Ham/Set3 & Data/Spam/Set3 ... 1798 hams & 1547 spams
>       2.447   0.646
>       0.945   0.388
>       2.884   3.426
>       2.058   1.097
> Training on Data/Ham/Set4 & Data/Spam/Set4 ... 1803 hams & 1547 spams
>       1.057   2.584
>       0.723   1.682
>       0.890   1.164
>       0.445   0.452
> Training on Data/Ham/Set5 & Data/Spam/Set5 ... 1798 hams & 1550 spams
>       0.779   4.328
>       0.501   3.299
>       0.667   3.361
>       0.388   4.977
> total false pos 273 3.03501945525
> total false neg 367 4.74282760403

How were these msgs broken up into the 5 sets?  Set4 in particular is giving
the other sets severe problems, and Set5 blows the f-n rate on everything
it's predicting -- when the rates across runs within a training set vary by
as much as a factor of 25, it suggests there was systematic bias in the way
the sets were chosen.  For example, perhaps they were broken into sets by
arrival time.  If that's what you did, you should go back and break them
into sets randomly instead.  If you did partition them randomly, the wild
variance across runs is mondo mysterious.


>> I expect hammie will do a much better job on this already than hand
>> grepping.  Be sure to stare at the false positives and get the
>> spam out of there.

> Yah, but there's a chicken-and-egg problem there - I want stuff that's
> _known_ to be right to test this stuff,

Then you have to look at every message by eyeball -- any scheme has non-zero
error rates of both kinds.

> so using the spambayes code to tell me whether it's spam is not
> going to help.

Trust me <wink> -- it helps a *lot*.  I expect everyone who has done any
testing here has discovered spam in their ham, and vice versa.  Results
improve as you improve the categorization.  Once the gross mistakes are
straightened out, it's much less tedious to scan the rest by eyeball.

[on skip tokens]
> Yep, it shows up in a lot of spam, but also in different forms in hams.
> But the hams each manage to pick a different variant of
> ~~~~~~~~~~~~~~~~~~~~~~
> or whatever - so they don't end up counteracting the various bits in the
> spam.
>
> Looking further, a _lot_ of the bad skip rubbish is coming from
> uuencoded viruses &c in the spam-set.

For whatever reason, there appear to be few of those in BruceG's spam
collection.  I added code to strip uuencoded sections, and pump out uuencode
summary tokens instead.  I'll check it in.  It didn't make a significant
difference on my usual test run (a single spam in my Set4 is now judged as
ham by the other 4 sets; nothing else changed).  It does shrink the database
size here by a few percent.  Let us know whether it helps you!

Before and after stripping uuencoded sections:

false positive percentages
    0.000  0.000  tied
    0.000  0.000  tied
    0.050  0.050  tied
    0.000  0.000  tied
    0.025  0.025  tied
    0.000  0.000  tied
    0.075  0.075  tied
    0.025  0.025  tied
    0.025  0.025  tied
    0.000  0.000  tied
    0.050  0.050  tied
    0.000  0.000  tied
    0.025  0.025  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.050  0.050  tied
    0.025  0.025  tied
    0.000  0.000  tied
    0.025  0.025  tied
    0.050  0.050  tied

won   0 times
tied 20 times
lost  0 times

total unique fp went from 8 to 8 tied

false negative percentages
    0.255  0.255  tied
    0.364  0.364  tied
    0.254  0.291  lost   +14.57%
    0.509  0.509  tied
    0.436  0.436  tied
    0.218  0.218  tied
    0.182  0.218  lost   +19.78%
    0.582  0.582  tied
    0.327  0.327  tied
    0.255  0.255  tied
    0.254  0.291  lost   +14.57%
    0.582  0.582  tied
    0.545  0.545  tied
    0.255  0.255  tied
    0.291  0.291  tied
    0.400  0.400  tied
    0.291  0.291  tied
    0.218  0.218  tied
    0.218  0.218  tied
    0.145  0.182  lost   +25.52%

won   0 times
tied 16 times
lost  4 times

total unique fn went from 89 to 90 lost    +1.12%


From anthony@interlink.com.au  Thu Sep 12 01:23:51 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Thu, 12 Sep 2002 10:23:51 +1000
Subject: [Spambayes] Current histograms 
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEJBBDAB.tim.one@comcast.net> 
Message-ID: <200209120023.g8C0NpQ18478@localhost.localdomain>



> How were these msgs broken up into the 5 sets?  Set4 in particular is giving
> the other sets severe problems, and Set5 blows the f-n rate on everything
> it's predicting -- when the rates across runs within a training set vary by
> as much as a factor of 25, it suggests there was systematic bias in the way
> the sets were chosen.  For example, perhaps they were broken into sets by
> arrival time.  If that's what you did, you should go back and break them
> into sets randomly instead.  If you did partition them randomly, the wild
> variance across runs is mondo mysterious.

They weren't partitioned in any particular scheme - I think I'll write a
reshuffler and move them all around, just in case (fwiw, I'm using MH 
style folders with numbered files - means you can just use MH tools to 
manipulate the sets.)


> For whatever reason, there appear to be few of those in BruceG's spam
> collection.  I added code to strip uuencoded sections, and pump out uuencode
> summary tokens instead.  I'll check it in.  It didn't make a significant
> difference on my usual test run (a single spam in my Set4 is now judged as
> ham by the other 4 sets; nothing else changed).  It does shrink the database
> size here by a few percent.  Let us know whether it helps you!

I'll give it a go.


-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.


From tim.one@comcast.net  Thu Sep 12 01:37:13 2002
From: tim.one@comcast.net (Tim Peters)
Date: Wed, 11 Sep 2002 20:37:13 -0400
Subject: [Spambayes] stack.pop() ate my multipart message
In-Reply-To: <w53znup6ly2.fsf@woozle.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEJDBDAB.tim.one@comcast.net>

[Neale Pickett]
> And then, just as I was about to fall asleep, I figured it out.  The
> tokenizer now has an extra [:], and all is well.  I feel like a real
> chawbacon for not realizing this earlier.  :*)

Good eye, Neale!  Thanks for the fix.

> Blaming it on staying up past bedtime,

Blame it on Barry.  I do <wink>.

From tim.one@comcast.net  Thu Sep 12 01:44:56 2002
From: tim.one@comcast.net (Tim Peters)
Date: Wed, 11 Sep 2002 20:44:56 -0400
Subject: [Spambayes] XTreme Training
In-Reply-To: <20020911122308.GB5866@cthulhu.gerg.ca>
Message-ID: <LNBBLJKPBEHFEDALKOLCAEJEBDAB.tim.one@comcast.net>

[Tim]
>> Why would spam be likely to end up with two instances of Return-Path
>> in the headers?

[Greg Ward]
> Possibly another qmail-ism from Bruce Guenter's spam collection.

Doesn't seem *likely*, as it appeared in about 900 of about 14,000 spams.
It could be specific to one of his bait addresses, though -- don't know.  A
nice thing about a statistical inferencer is that you really don't have to
know why a thing works, just whether it works <wink>.

> Or maybe Anthony's right about spammers being stupid and blindly copying
> headers. (Well, of course he's right about spammers being stupid; it's
> just this particular aspect of stupidity that's open to question.)

I'm going to blow it off -- it's just another instance of being pointlessly
baffled by a mixed corpus half of which I don't know enough about.


From skip@pobox.com  Thu Sep 12 02:10:50 2002
From: skip@pobox.com (Skip Montanaro)
Date: Wed, 11 Sep 2002 20:10:50 -0500
Subject: [Spambayes] Current histograms 
In-Reply-To: <200209120023.g8C0NpQ18478@localhost.localdomain>
References: <LNBBLJKPBEHFEDALKOLCKEJBBDAB.tim.one@comcast.net>
        <200209120023.g8C0NpQ18478@localhost.localdomain>
Message-ID: <15743.59802.802210.914537@12-248-11-90.client.attbi.com>


    Anthony> They weren't partitioned in any particular scheme - I think
    Anthony> I'll write a reshuffler and move them all around, ...

Hmmm.   How about you create empty Data/Ham/Set[12345], stuff all your
files into a Data/Ham/reservoir folder, then run the rebal.py script to
randomly parcel messages out to the various real directories?

I suspect you can pull the same stunt for your Data/Spam stuff.

Skip

From tim.one@comcast.net  Thu Sep 12 04:06:24 2002
From: tim.one@comcast.net (Tim Peters)
Date: Wed, 11 Sep 2002 23:06:24 -0400
Subject: [Spambayes] Current histograms
In-Reply-To: <15743.59802.802210.914537@12-248-11-90.client.attbi.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEJJBDAB.tim.one@comcast.net>

[Skip]
> Hmmm.   How about you create empty Data/Ham/Set[12345], stuff all your
> files into a Data/Ham/reservoir folder, then run the rebal.py script to
> randomly parcel messages out to the various real directories?

I'm afraid rebal is quadratic-time in the # of msgs it shuffles around --
since it was only intended to move a few files around, it's dead simple.

An easy thing is to start the same way:  move all the files into a single
directory.  Then do random.shuffle() on an os.listdir() of that directory.
Then it's trivial to split the result into N slices, and move the files into
N other directories accordingly.

> I suspect you can pull the same stunt for your Data/Spam stuff.

Yup!


From anthony@interlink.com.au  Thu Sep 12 05:26:41 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Thu, 12 Sep 2002 14:26:41 +1000
Subject: [Spambayes] Current histograms 
Message-ID: <200209120426.g8C4QfI23085@localhost.localdomain>


> They weren't partitioned in any particular scheme - I think I'll write a
> reshuffler and move them all around, just in case (fwiw, I'm using MH 
> style folders with numbered files - means you can just use MH tools to 
> manipulate the sets.)

Freak show. Obviously there _was_ some sort of patterns to the data:

Training on Data/Ham/Set1 & Data/Spam/Set1 ... 1798 hams & 1546 spams
      0.779   0.582
      0.834   0.840
      0.945   0.452
      0.667   1.164
Training on Data/Ham/Set2 & Data/Spam/Set2 ... 1798 hams & 1547 spams
      1.112   0.776
      0.834   0.969
      0.779   0.646
      0.667   1.100
Training on Data/Ham/Set3 & Data/Spam/Set3 ... 1798 hams & 1548 spams
      1.168   0.582
      1.001   0.646
      0.834   0.582
      0.667   0.453
Training on Data/Ham/Set4 & Data/Spam/Set4 ... 1798 hams & 1547 spams
      0.779   0.712
      0.779   0.582
      0.556   0.840
      0.779   0.970
Training on Data/Ham/Set5 & Data/Spam/Set5 ... 1798 hams & 1546 spams
      0.612   0.517
      0.779   0.517
      0.723   0.711
      0.667   0.582
total false pos 144 1.60177975528
total false neg 101 1.30592190328

(before the shuffle, I was seeing:
total false pos 273 3.03501945525
total false neg 367 4.74282760403
)

For sake of comparision, here's what I see for partitioned into 2 sets:

Training on Data/Ham/Set1 & Data/Spam/Set1 ... 4492 hams & 3872 spams
      0.490   0.776
Training on Data/Ham/Set2 & Data/Spam/Set2 ... 4493 hams & 3868 spams
      0.401   0.491
total false pos 40 0.445186421814
total false neg 49 0.633074935401

more later...

Anthony

From neale@woozle.org  Thu Sep 12 08:26:21 2002
From: neale@woozle.org (Neale Pickett)
Date: 12 Sep 2002 00:26:21 -0700
Subject: [Spambayes] Results of playing with CDB
Message-ID: <w537khr7joi.fsf@woozle.org>

In keeping with my tradition of doing precisely what the goal of this
project isn't, I've been tinkering with CDB as a storage back-end.  The
hype about CDB is that it's incredibly fast to look stuff up, but not so
fast at making updates.  In fact, you have to rewrite the entire
database if you update anything.

So what I did was write a front-end that caches writes, and then outputs
a new database on destruction.  I wrapped it around Neil's cdb.py with
cdbwrap.py (now in CVS).  Then, I wrote a cdbhammie.py to use it.

So far, it outputs a *giant* text file which can be converted to cdb
with djb's cdbmake (much faster than doing it in Python).  My text file
weighs in at a ridiculous 106 million bytes.  Maybe something changed
with the tokenizer to make it so much larger, or maybe it's that I'm
using a derivative of shelve.Shelf in cdbhammie.  

I suspect that the 106MB file is indicative of something really wrong
with what I'm doing, since the same dataset only made me a 20MB file
with a Berkeley hash a few days ago, and CDB appears to be a very
lightweight format.  It scores a message in under 1s, which I suppose is
pretty impressive considering the immensity of the database it's working
with (111MB).  So it may well outperform Berkeley hashes with the same
data.

Anyhow, I'm out of hours in the day to mess with it.  If anyone else
wants to poke around at it, be my guest.

Neale

From robertoschler@hotmail.com  Thu Sep 12 10:22:04 2002
From: robertoschler@hotmail.com (Robert Oschler)
Date: Thu, 12 Sep 2002 05:22:04 -0400
Subject: [Spambayes] Link Grammar?
Message-ID: <OE61wSoRKbpG1Kiaexd00001674@hotmail.com>

This is a multi-part message in MIME format.

---------------------- multipart/alternative attachment
Hi I'm new to this list.  Has anyone mentioned Link Grammar?  It's an =
open source natural language parser that is very impressive.  I'm =
starting to work with it now.  I would think that knowing the usage of a =
word (noun, subject, preposition), _and_ its relation other words in the =
sentence (e.g., object of a verb, complement to the verb "to be/is", =
would be a boon to really refining the probability trees.

Just a thought.  Here's the "link" to Link Grammar.  Try playing with =
the Parser via their web site:

http://www.link.cs.cmu.edu/link/

thx


---------------------- multipart/alternative attachment
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail-21/spambayes/attachments/20020912/32062e48/attachment.html

---------------------- multipart/alternative attachment--

From robertoschler@hotmail.com  Thu Sep 12 10:28:52 2002
From: robertoschler@hotmail.com (Robert Oschler)
Date: Thu, 12 Sep 2002 05:28:52 -0400
Subject: [Spambayes] "Difficult cases" archive?
Message-ID: <OE18O3hRDT6Z3LstPGt00001659@hotmail.com>

This is a multi-part message in MIME format.

---------------------- multipart/alternative attachment
Is there an archive of spam cases that have "stumped the filters" or "if =
filtered create too many false positives"?  I'd like to try my hand and =
at nuking those.  Again, newbie here.  If it's a dumb question then mea =
culpa.

thx


---------------------- multipart/alternative attachment
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail-21/spambayes/attachments/20020912/1ed3a32f/attachment.html

---------------------- multipart/alternative attachment--

From gward@python.net  Thu Sep 12 13:43:16 2002
From: gward@python.net (Greg Ward)
Date: Thu, 12 Sep 2002 08:43:16 -0400
Subject: [Spambayes] Link Grammar?
In-Reply-To: <OE61wSoRKbpG1Kiaexd00001674@hotmail.com>
References: <OE61wSoRKbpG1Kiaexd00001674@hotmail.com>
Message-ID: <20020912124316.GA11728@cthulhu.gerg.ca>

On 12 September 2002, Robert Oschler said:
> Hi I'm new to this list.  Has anyone mentioned Link Grammar?  It's an
> open source natural language parser that is very impressive.  I'm
> starting to work with it now.  I would think that knowing the usage of
> a word (noun, subject, preposition), _and_ its relation other words in
> the sentence (e.g., object of a verb, complement to the verb "to
> be/is", would be a boon to really refining the probability trees.

To channel the Tim-bot: it's not necessary to be so smart.  Simple and
stupid seems to work best here, despite everyone's instinctive reaction.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Money is truthful.  If a man speaks of his honor, make him pay cash.

From gward@python.net  Thu Sep 12 13:44:41 2002
From: gward@python.net (Greg Ward)
Date: Thu, 12 Sep 2002 08:44:41 -0400
Subject: [Spambayes] "Difficult cases" archive?
In-Reply-To: <OE18O3hRDT6Z3LstPGt00001659@hotmail.com>
References: <OE18O3hRDT6Z3LstPGt00001659@hotmail.com>
Message-ID: <20020912124441.GB11728@cthulhu.gerg.ca>

On 12 September 2002, Robert Oschler said:
> Is there an archive of spam cases that have "stumped the filters" or
> "if filtered create too many false positives"?  I'd like to try my
> hand and at nuking those.  Again, newbie here.  If it's a dumb
> question then mea culpa.

I have a collection of spam that 1) made it past SpamAssassin on
mail.python.org, and 2) was reported by someone (by forwarding it to
spam@python.org).

That's not necessarily what you asked for, but it would be interesting
to see how well spambayes does on the spam that SA didn't detect.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
An elephant: a mouse built to government specifications.

From anthony@interlink.com.au  Thu Sep 12 14:05:55 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Thu, 12 Sep 2002 23:05:55 +1000
Subject: [Spambayes] "Difficult cases" archive? 
In-Reply-To: <OE18O3hRDT6Z3LstPGt00001659@hotmail.com> 
Message-ID: <200209121305.g8CD5un27070@localhost.localdomain>



> Is there an archive of spam cases that have "stumped the filters" or "if 
> filtered create too many false positives"?  I'd like to try my hand and 
> at nuking those.  Again, newbie here.  If it's a dumb question then mea 
> culpa.

Part of the problem is that individual difficult cases vary based on the
context - without the rest of the collection of ham/spam, it's not hugely
useful to try and work on individual cases...



-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.


From guido@python.org  Thu Sep 12 14:15:28 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 12 Sep 2002 09:15:28 -0400
Subject: [Spambayes] "Difficult cases" archive?
In-Reply-To: Your message of "Thu, 12 Sep 2002 08:44:41 EDT."
             <20020912124441.GB11728@cthulhu.gerg.ca> 
References: <OE18O3hRDT6Z3LstPGt00001659@hotmail.com>  
            <20020912124441.GB11728@cthulhu.gerg.ca> 
Message-ID: <200209121315.g8CDFS111538@pcp02138704pcs.reston01.va.comcast.net>

> That's not necessarily what you asked for, but it would be interesting
> to see how well spambayes does on the spam that SA didn't detect.

The few times I had spam in my inbox that reached me via SA, SB scored
it perfectly.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From anthony@interlink.com.au  Thu Sep 12 14:34:51 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Thu, 12 Sep 2002 23:34:51 +1000
Subject: [Spambayes] "Difficult cases" archive? 
In-Reply-To: 
	<200209121315.g8CDFS111538@pcp02138704pcs.reston01.va.comcast.net> 
Message-ID: <200209121334.g8CDYpu27391@localhost.localdomain>


>>> Guido van Rossum wrote
> > That's not necessarily what you asked for, but it would be interesting
> > to see how well spambayes does on the spam that SA didn't detect.
> 
> The few times I had spam in my inbox that reached me via SA, SB scored
> it perfectly.

In my personal mailbox, I find SA is too trigger happy, rather than 
not trigger happy enough. This is more annoying, as it means I have to
keep a close-ish eye on the +spam folder...




From guido@python.org  Thu Sep 12 15:26:11 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 12 Sep 2002 10:26:11 -0400
Subject: [Spambayes] "Difficult cases" archive?
In-Reply-To: Your message of "Thu, 12 Sep 2002 23:34:51 +1000."
             <200209121334.g8CDYpu27391@localhost.localdomain> 
References: <200209121334.g8CDYpu27391@localhost.localdomain> 
Message-ID: <200209121426.g8CEQBl28353@odiug.zope.com>

> In my personal mailbox, I find SA is too trigger happy, rather than 
> not trigger happy enough. This is more annoying, as it means I have to
> keep a close-ish eye on the +spam folder...

Oh yes, SA is too trigger-happy for me too.  The kind postmasters at
python.org save at least one mail per week for me (though mostly of
dubious value, like monthly "newsletters" that I've signed up for).

--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.one@comcast.net  Thu Sep 12 16:14:45 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 12 Sep 2002 11:14:45 -0400
Subject: [Spambayes] Results of playing with CDB
In-Reply-To: <w537khr7joi.fsf@woozle.org>
Message-ID: <BIEJKCLHCIOIHAGOKOLHAEJIDKAA.tim.one@comcast.net>

[Neale Pickett]
> ...
> So far, it outputs a *giant* text file which can be converted to cdb
> with djb's cdbmake (much faster than doing it in Python).  My text file
> weighs in at a ridiculous 106 million bytes.  Maybe something changed
> with the tokenizer to make it so much larger, or maybe it's that I'm
> using a derivative of shelve.Shelf in cdbhammie.

As of last night, tokenizer changes were such that a binary pickle of the
dict database trained on 4000 ham + ~2750 spam shrunk to about 7MB (down
from about 9MB before last night's changes -- I dropped character
5-gram'ming of long words with high-bit chars, replaced with a much simpler
gimmick, and a full test run said it made no difference).

> I suspect that the 106MB file is indicative of something really wrong
> with what I'm doing, since the same dataset only made me a 20MB file
> with a Berkeley hash a few days ago,

Note that database size has no meaning without at least knowing how many
messages the databse was trained on, or, better, how many keys it contains.
Binary pickles are really quite space-efficient.

> ...
> It scores a message in under 1s, which I suppose is pretty impressive
> considering the immensity of the database it's working with (111MB).  So
> it may well outperform Berkeley hashes with the same data.

I'm curious about why you resist a server-like solution?  The plain dict
version of this code scores about 80 msgs/sec on my 866MHz home box, and
that includes the time to open and read each msg from a separate file.


From tim.one@comcast.net  Thu Sep 12 16:44:21 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 12 Sep 2002 11:44:21 -0400
Subject: [Spambayes] "Difficult cases" archive?
In-Reply-To: <OE18O3hRDT6Z3LstPGt00001659@hotmail.com>
Message-ID: <BIEJKCLHCIOIHAGOKOLHOEJIDKAA.tim.one@comcast.net>

[Robert Oschler]
> Is there an archive of spam cases that have "stumped the filters"

If you check out the software from CVS and use it on some email, you'll
quickly get examples of false positives and false negatives.

> or "if filtered create too many false positives"?

This is a statistical approach, so no single msg can (well, should <wink>)
have a dramatic effect.

> I'd like to try my hand and at nuking those.  Again, newbie here.  If it's
> a dumb question then mea culpa.

Not a dumb question at all.  What I expect you'll find is that the false
negatives under this scheme are breathtakingly false -- screamingly obvious
spam, often long and chatty.  Graham's scoring scheme ignores almost all the
words in a msg, picking on just the about-a-dozen that have "probabilities"
farthest from 0.5.  This seems amazingly effective given enough training
data, but when it fails it's often a spectacular failure.  When you're got
100 clues with prob 0.01 and a hundred with 0.99, the scheme is just lost.
I made an improvement over Graham's original scoring scheme that reduced the
frequency of these gross errors, but they still happen.  A scheme to do
better would be most welcome!


From tim.one@comcast.net  Thu Sep 12 16:51:29 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 12 Sep 2002 11:51:29 -0400
Subject: [Spambayes] "Difficult cases" archive?
In-Reply-To: <20020912124441.GB11728@cthulhu.gerg.ca>
Message-ID: <BIEJKCLHCIOIHAGOKOLHOEJJDKAA.tim.one@comcast.net>

[Greg Ward]
> I have a collection of spam that 1) made it past SpamAssassin on
> mail.python.org, and 2) was reported by someone (by forwarding it to
> spam@python.org).
>
> That's not necessarily what you asked for, but it would be interesting
> to see how well spambayes does on the spam that SA didn't detect.

I'm waiting for Barry to write a little script to rip out the forwarding
artifacts; he says it's easy to do that, but has presumably been working on
it for more than a week <wink>.

You can't let that archive out of the building because it's loaded with
personal email.  I had a quick glance at it with one eye blinded, and from
the little I *saw* it didn't look promising:  everyone subscribes to *some*
bulk email lists, and from what I saw there was no way to distinguish bulk
email that someone wants from bulk email the same person doesn't want,
unless the classifier was trained on a specific-to-that-person corpus.
Else, e.g., Barry would get Guido's weekly list of "Dutch Celebrities in the
News", while Guido would have to endure Barry's weekly list of "Bass Players
Who Choked to Death on the Drummer's Vomit This Week" <wink>.


From robertoschler@hotmail.com  Thu Sep 12 17:44:57 2002
From: robertoschler@hotmail.com (Robert Oschler)
Date: Thu, 12 Sep 2002 12:44:57 -0400
Subject: [Spambayes] Big spam messages file you care to share?
Message-ID: <OE75xWnefxGZ2Lig41U00001a90@hotmail.com>

This is a multi-part message in MIME format.

---------------------- multipart/alternative attachment
If anyone has a big file of spam and non-spam (for false positive =
detection) messages that they care to share, which means they don't =
contain any confidential data, I'd like to know about it.  I get about 3 =
spam messages a week and I haven't been archiving them.

thx


---------------------- multipart/alternative attachment
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail-21/spambayes/attachments/20020912/24da23bb/attachment.html

---------------------- multipart/alternative attachment--

From gward@python.net  Thu Sep 12 19:24:34 2002
From: gward@python.net (Greg Ward)
Date: Thu, 12 Sep 2002 14:24:34 -0400
Subject: [Spambayes] "Difficult cases" archive?
In-Reply-To: <BIEJKCLHCIOIHAGOKOLHOEJJDKAA.tim.one@comcast.net>
References: <20020912124441.GB11728@cthulhu.gerg.ca>
	<BIEJKCLHCIOIHAGOKOLHOEJJDKAA.tim.one@comcast.net>
Message-ID: <20020912182434.GA13136@cthulhu.gerg.ca>

[me]
> I have a collection of spam that 1) made it past SpamAssassin on
> mail.python.org, and 2) was reported by someone (by forwarding it to
> spam@python.org).
>
> That's not necessarily what you asked for, but it would be interesting
> to see how well spambayes does on the spam that SA didn't detect.

[Tim]
> I'm waiting for Barry to write a little script to rip out the forwarding
> artifacts; he says it's easy to do that, but has presumably been working on
> it for more than a week <wink>.

Oh, I already wrote it months ago.  OK, I've just put it in
~greg/process_ncspam on mail.python.org.

> You can't let that archive out of the building because it's loaded with
> personal email.

Hang on a sec -- I think you're talking about /var/mail/fp-spam, which
are the false positives from SpamAssassin -- and it does indeed have
some personal email in it.  Not sure if I would say "loaded with", but
whatever.

I was talking about /var/mail/nc-spam, which is spam that made it past
the filters and was forwarded by some conscientious soul to
spam@python.org.  It is chock full of spam and little else (occasionally
virus-related junk mail that should have been sent to virus@python.org).
About the only personal information you could glean from it is who's
getting spam, and who's reporting it to spam@python.org.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Gee, I feel kind of LIGHT in the head now, knowing I can't make my
satellite dish PAYMENTS!

From gward@python.net  Thu Sep 12 19:32:25 2002
From: gward@python.net (Greg Ward)
Date: Thu, 12 Sep 2002 14:32:25 -0400
Subject: [Spambayes] Big spam messages file you care to share?
In-Reply-To: <OE75xWnefxGZ2Lig41U00001a90@hotmail.com>
References: <OE75xWnefxGZ2Lig41U00001a90@hotmail.com>
Message-ID: <20020912183225.GA13199@cthulhu.gerg.ca>

On 12 September 2002, Robert Oschler said:
> If anyone has a big file of spam and non-spam (for false positive
> detection) messages that they care to share, which means they don't
> contain any confidential data, I'd like to know about it.  I get about
> 3 spam messages a week and I haven't been archiving them.

Currently gathering one on mail.python.org.  Should be a doozy: it
currently has (after about 30 hours of gathering)

  3944 bounce messages  (20529 kB)
  1269 regular messages ( 3373 kB)
       (includes spam that snuck past SpamAssassin -- not much so far)
       (also includes bounces with a non-empty envelope sender)
   457 junk messages    (24707 kB)
       (includes spam, viruses, and SpamAssassin false positives)

The biggest artifact is the overwhelming preponderance of bounce
messages, but that's reality for mail.python.org.  (I bet "MAILER-DAEMON"
will be a popular ham word.  "qmail", "Exim", and "postfix" might also
be, since they pop up in bounce messages lot too.)

I'll have to do something about distinguishing spam, viruses, and other
junk mail in the final corpus.  And of course, there will be the usual
exercise of removing ham from the spam folder and vice-versa.

But I think this will be a pretty good corpus for training spam
detectors for python.org traffic, once I've gathered about 10 days'
worth of traffic.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
All the world's a stage and most of us are desperately unrehearsed.

From tim.one@comcast.net  Thu Sep 12 23:03:17 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 12 Sep 2002 18:03:17 -0400
Subject: [Spambayes] Big spam messages file you care to share?
In-Reply-To: <OE75xWnefxGZ2Lig41U00001a90@hotmail.com>
Message-ID: <BIEJKCLHCIOIHAGOKOLHIEKDDKAA.tim.one@comcast.net>

[Robert Oschler]
> If anyone has a big file of spam and non-spam (for false positive
detection)
> messages that they care to share, which means they don't contain any
> confidential data, I'd like to know about it.  I get about 3 spam messages
a
> week and I haven't been archiving them.

Getting a Hotmail account will cure that <wink>.

I've been using Bruce Guenter's 2002 archive of about 14,000 spam.  Links to
that and two other collections can be found here:

    http://www.paulgraham.com/spamarchives.html

For ham I've been using 20,000 articles selected at random from the
comp.lang.python newsgroup == python-list mailing list.

Something you'll soon discover is that using ham and spam from different
sources creates a lot of headaches, as the classifier picks up on *every*
clue it sees.  For example, Jeremy had an older collection of spam sitting
around, and by tokenizing Received headers the classifier latched on to that
all msgs from around that time and that box were almost certainly spam.  For
that reason, several of us are using crippled versions of the classifier
now.


From neale@woozle.org  Fri Sep 13 00:14:10 2002
From: neale@woozle.org (Neale Pickett)
Date: 12 Sep 2002 16:14:10 -0700
Subject: [Spambayes] Results of playing with CDB
In-Reply-To: <BIEJKCLHCIOIHAGOKOLHIEJHDKAA.tim@zope.com>
References: <BIEJKCLHCIOIHAGOKOLHIEJHDKAA.tim@zope.com>
Message-ID: <w53lm666bst.fsf@woozle.org>

So then, "Tim Peters" <tim@zope.com> is all like:

> Note that database size has no meaning without at least knowing how
> many messages the databse was trained on, or, better, how many keys it
> contains.

Okay, here are some Truly Accurate (tm) results.

Training on a fixed input set of 1000 spam and 1000 ham, I get the
following numbers for the three methods:

              pickle         hash          cdb

Training:     59.61s      438.42s      642.09s
Testing:       9.18s        0.60s        7.57s
Store size:  2948KiB      8976KiB     89404KiB

I'm still not sure why cdb is that much bigger and slower, since it's
using the exact same inputs.  But lining them up side by side has shown
me that the more interesting task is making the hash smaller.  I suspect
that when I do this, I'll see a speed increase as well.

> Binary pickles are really quite space-efficient.

Verily.


> I'm curious about why you resist a server-like solution?  The plain
> dict version of this code scores about 80 msgs/sec on my 866MHz home
> box, and that includes the time to open and read each msg from a
> separate file.

I'm not sure a server would work for multiple users.  I have 80 users;
at 20MB per server, that's 1600MB of RAM, which would require a new hard
drive just for the swap :).  I'm sure it wouldn't be quite that bad if
it was just one server for everyone, but in that case I suspect the dict
would quickly become so large that it would no longer be practical to
have it all in RAM, and then I'm back to dbm files.

Of course, I could be missing some obvious implementation detail.  The
spam database could be shared, for instance, if it was sufficiently
attack-resistant.  In any case, speeding up the current implementation
seems like a sufficiently low-hanging fruit for the time being, and
hopefully will be useful work for a larger project.

Jonesin' for fructose,

Neale

From anthony@interlink.com.au  Fri Sep 13 00:38:18 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Fri, 13 Sep 2002 09:38:18 +1000
Subject: [Spambayes] Big spam messages file you care to share? 
In-Reply-To: <20020912183225.GA13199@cthulhu.gerg.ca> 
Message-ID: <200209122338.g8CNcJx00454@localhost.localdomain>


>>> Greg Ward wrote
> The biggest artifact is the overwhelming preponderance of bounce
> messages, but that's reality for mail.python.org.  (I bet "MAILER-DAEMON"
> will be a popular ham word.  "qmail", "Exim", and "postfix" might also
> be, since they pop up in bounce messages lot too.)

This reminds me of something that came up yesterday - it's more for the
mimelib list, but it will probably bite here, too. Postfix (at least, the
version we're using) when sending a bounce message, will truncate the bounced
message at some limit (I think it's about 50K). It does this rather abruptly,
to the point where if the message is MIME, it's structure is stuffed (no end
tags, that sorta thing). So when the email parser descends into the message/rfc822,
it barfs out.

Anthony
-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.


From JasonR.Mastaler  Fri Sep 13 00:29:22 2002
From: JasonR.Mastaler (JasonR.Mastaler)
Date: Thu, 12 Sep 2002 17:29:22 -0600
Subject: [Spambayes] Re: Results of playing with CDB
References: <w537khr7joi.fsf@woozle.org>
Message-ID: <hhk7lqyegd.fsf@hrothgar.la.mastaler.com>

Neale Pickett <neale@woozle.org> writes:

> So what I did was write a front-end that caches writes, and then
> outputs a new database on destruction.  I wrapped it around Neil's
> cdb.py with cdbwrap.py (now in CVS).  Then, I wrote a cdbhammie.py
> to use it.
>
> So far, it outputs a *giant* text file which can be converted to cdb
> with djb's cdbmake (much faster than doing it in Python).

Have you considered using the python-cdb module
(http://pilcrow.madison.wi.us/#pycdb)?

It's a C module which I've had very good results with over the years.

-- 
(http://tmda.net/)



From neale@woozle.org  Fri Sep 13 01:01:41 2002
From: neale@woozle.org (Neale Pickett)
Date: 12 Sep 2002 17:01:41 -0700
Subject: [Spambayes] Re: Results of playing with CDB
In-Reply-To: <hhk7lqyegd.fsf@hrothgar.la.mastaler.com>
References: <w537khr7joi.fsf@woozle.org>
	<hhk7lqyegd.fsf@hrothgar.la.mastaler.com>
Message-ID: <w534rcu69lm.fsf@woozle.org>

So then, Jason R.Mastaler <jason@mastaler.com> is all like:

> Neale Pickett <neale@woozle.org> writes:
> 
> > So what I did was write a front-end that caches writes, and then
> > outputs a new database on destruction.  I wrapped it around Neil's
> > cdb.py with cdbwrap.py (now in CVS).  Then, I wrote a cdbhammie.py
> > to use it.
> >
> > So far, it outputs a *giant* text file which can be converted to cdb
> > with djb's cdbmake (much faster than doing it in Python).
> 
> Have you considered using the python-cdb module
> (http://pilcrow.madison.wi.us/#pycdb)?

Yeah, I was playing with that for a while.  Two problems:

1.  It looked like it was going to be a total pain to get that code to
    look like a dict.  Well, not a total pain.  But Neil's code was
    going to be easier.

2.  Although it claims to be licensed under the GPL, a cursory glance at
    his .c files shows they were lifted right out of DJB's cdb package.
    I don't know what his licensing terms are, and apparently neither
    does anybody else :)

So I just stuck with the one in cvs, which was good enough to do some
testing and convince me that I didn't want to use CDB :)

From tim.one@comcast.net  Fri Sep 13 01:41:01 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 12 Sep 2002 20:41:01 -0400
Subject: [Spambayes] FW: [Spambayes-checkins] spambayes
 Options.py,1.11,1.12timtest.py,1.22,1.23
Message-ID: <LNBBLJKPBEHFEDALKOLCOELJBDAB.tim.one@comcast.net>

Note this incompatible change.  So sue me <wink>.

-----Original Message-----
From: spambayes-checkins-bounces@python.org
[mailto:spambayes-checkins-bounces@python.org]On Behalf Of Tim Peters
Sent: Thursday, September 12, 2002 8:28 PM
To: spambayes-checkins@python.org
Subject: [Spambayes-checkins] spambayes
Options.py,1.11,1.12timtest.py,1.22,1.23


Update of /cvsroot/spambayes/spambayes
In directory usw-pr-cvs1:/tmp/cvs-serv25548

Modified Files:
	Options.py timtest.py 
Log Message:
Incompatible change:  show_best_discriminators has changed from a bool
option to an int option, now giving the number of best discriminators
to show.  Set to 0 if you don't want to see any.

From gward@python.net  Fri Sep 13 02:41:17 2002
From: gward@python.net (Greg Ward)
Date: Thu, 12 Sep 2002 21:41:17 -0400
Subject: [Spambayes] Big spam messages file you care to share?
In-Reply-To: <200209122338.g8CNcJx00454@localhost.localdomain>
References: <20020912183225.GA13199@cthulhu.gerg.ca>
	<200209122338.g8CNcJx00454@localhost.localdomain>
Message-ID: <20020913014117.GA14192@cthulhu.gerg.ca>

On 13 September 2002, Anthony Baxter said:
> This reminds me of something that came up yesterday - it's more for the
> mimelib list, but it will probably bite here, too. Postfix (at least, the
> version we're using) when sending a bounce message, will truncate the bounced
> message at some limit (I think it's about 50K).

Exim will do the same thing, if so configured.  Dunno what the default
limit is, but I'm pretty sure it's a lot higher than 50 K.  I think I
set it to 30 K on mail.python.org.

IOW, malformed MIME messages do exist, and they're not just viruses!

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Laziness, Impatience, Hubris.

From nas@python.ca  Fri Sep 13 03:39:22 2002
From: nas@python.ca (Neil Schemenauer)
Date: Thu, 12 Sep 2002 19:39:22 -0700
Subject: [Spambayes] best discriminator
Message-ID: <20020913023922.GA5589@glacier.arctrix.com>

>From a recent training run (not sorted by me):

    best discriminators:
        'from:guido' 62 0.01
        '(home' 63 0.01
        '--guido' 63 0.01
        'from:rossum' 63 0.01
        'from:van' 63 0.01
        'page:' 63 0.01
        'url:~guido' 63 0.01

Hee hee,

  Neil

From tim.one@comcast.net  Fri Sep 13 07:05:44 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 13 Sep 2002 02:05:44 -0400
Subject: [Spambayes] "Difficult cases" archive?
In-Reply-To: <20020912182434.GA13136@cthulhu.gerg.ca>
Message-ID: <LNBBLJKPBEHFEDALKOLCEEMDBDAB.tim.one@comcast.net>

[Tim]
> I'm waiting for Barry to write a little script to rip out the forwarding
> artifacts; ...

[Greg Ward]
> Oh, I already wrote it months ago.  OK, I've just put it in
> ~greg/process_ncspam on mail.python.org.

I have no access to that machine; that's why I'm waiting for Barry <wink>.

> ..
> Hang on a sec -- I think you're talking about /var/mail/fp-spam,

As above, I have no idea what mail.python.org looks like.

> which are the false positives from SpamAssassin -- and it does indeed
> have some personal email in it.  Not sure if I would say "loaded with",
> but whatever.

I only looked at a few msgs in whatever it is got passed to me under the
table.  The hit rate for "personal email" was 100%; I extrapolated <wink>.

> I was talking about /var/mail/nc-spam, which is spam that made it past
> the filters and was forwarded by some conscientious soul to
> spam@python.org.  It is chock full of spam and little else (occasionally
> virus-related junk mail that should have been sent to virus@python.org).
> About the only personal information you could glean from it is who's
> getting spam, and who's reporting it to spam@python.org.

If someone can sneak me a copy of that, cleaned of forwarding artifacts, and
artifical transformations (e.g., making up a To: header if one didn't
originally exist:  the lack of a To: header happens to be a strong spam
indicator in my database), I'll be happy to run predictions on it against
"my usual" 5 c.l.py ham + bruceg spam random training pairs.  That will give
us a good idea of both expected error rate and variance, albeit via training
against a spam corpus that came from elsewhere.


From tim.one@comcast.net  Fri Sep 13 09:07:51 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 13 Sep 2002 04:07:51 -0400
Subject: [Spambayes] Testing methodology
In-Reply-To: <BIEJKCLHCIOIHAGOKOLHOEJIDKAA.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEMGBDAB.tim.one@comcast.net>

With contempt for compatibility <wink>, I'd like to switch to using k-fold
cross validation for evaluating test results.  10-fold cv seems to be a de
facto standard for this kind of thing now.

How that works:  You divide your data into K equal-sized pieces, at random.
Then the rest is like Neil's spontaneous leave-one-out design, but applied
to the K pieces:

1. Train on pieces 2 thru K; predict against piece 1.
.
i. Train on all pieces except piece i; predict against piece i.
.
K. Train on pieces 1 thru K-1; predict against piece K.

Then averaging the K observed rates is known to give a robust estimation of
"the true" rate under a wide variety of reaasonable assumptions.  You also
get to predict using (K-1)/K of your data for training, which is especially
helpful if you don't have a lot of good training data (seems a common
problem!).

The KxK test grid (less the diagonal) I've been using was good at the start
because it makes biases stand out clearly (c.f. the recent exchange with
Anthony).  But because it trains on only 1/K of your data at a time, people
are generally running with K too small.  Under k-fold cv with k=10 you get
to train on 90% of your data each time, so I expect people will be much more
willing to run a 10-fold cv (predicting based on 90% of your data each time)
than a 5x5 test grid (predicting based on 20% of your data each time).

If you have N messages, note that K-fold cv doesn't really require training
on N*(K-1)/K of them K times -- because our model is such that we can both
learn and unlearn, I would first train on all N, and for each of the K "big
iterations" take that as a base and unlearn the appropriate N/K messages
from it.  So the total training time would be similar to doing full training
twice, and this is independent of K.  The prediction time is also
independent of K, being K predictions on N/K messages each time, for a total
of N messages predicted.  OTOH, the database size will grow (and
substantially, but substantially less than a factor of K), and two copies
will be sitting around in memory (the base plus the current "partly
unlearned" version).

If there are no killer objections, I'll just do that someday <wink>.


From marklists@mceahern.com  Fri Sep 13 14:39:28 2002
From: marklists@mceahern.com (Mark McEahern)
Date: Fri, 13 Sep 2002 08:39:28 -0500
Subject: [Spambayes] FW: [Spambayes-checkins] spambayes
	Options.py,1.11,1.12timtest.py,1.22,1.23
In-Reply-To: <LNBBLJKPBEHFEDALKOLCOELJBDAB.tim.one@comcast.net>
Message-ID: <JHEOKEOOLIGLDHCMAHMOOEHODOAA.marklists@mceahern.com>

[Tim Peters]
> Log Message:
> Incompatible change:  show_best_discriminators has changed from a bool
> option to an int option, now giving the number of best discriminators
> to show.  Set to 0 if you don't want to see any.

Laura Creighton would love this one.  <wink>

// m

From tim@zope.com  Sat Sep 14 02:53:27 2002
From: tim@zope.com (Tim Peters)
Date: Fri, 13 Sep 2002 21:53:27 -0400
Subject: [Spambayes] Testing methodology
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEMGBDAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCOEPIBDAB.tim@zope.com>

This is a multi-part message in MIME format.

---------------------- multipart/mixed attachment
[double heh!  now python.org blacklisted mu email address for "sending
 spam", so the second attempt also bounced -- third time's the charm?]

[heh -- python.org rejected this mail because of a spam example
 buried near the bottom; you'll have to dig it out of the attached
 zip file if you want to see it]


[Tim]
> With contempt for compatibility <wink>, I'd like to switch to using k-
> fold cross validation for evaluating test results. ...

I've checked in changes for that now, and I hope I *didn't* lose
compatibility in mboxtest.py.  If I did, it's shallow, provided you know how
every stick of this works <wink>.  If I screwed up there, scream and I'll
help fix it.

Note that the format of statistics lines printed by Driver() have changed,
and rates.py and cmp.py changed accordingly.  See the checkin comments for
more.

I haven't chnaged my directories around yet to exploit the new c-v code.
Instead I'm running a 5-fold c-v (timcv.py -n5) on my current setup as an
experiment now.  That means it's training on 16000 ham and about 11000 spam,
then predicting against 4000 disjoint ham and 2750 disjoint spam.  Repeat 5
times.  Note that this should actually run substantially faster than my old
5x5 grid, for reasons explained last time around (and, indeed, 60% of the
full run has completed since I started typing this).

The first thing I note is good news:  despite that it's training on 4x as
many msgs, a pickle of the classifier grows only by a factor of 2.4.  That
means we're getting a lot of words in common.  But it also means we're
getting a lot of unique words, which was thoroughly predictable and is why
GrahamBayes has had a clearjunk() method since its first day (albeit that
I've *still* never tried it <wink>).

16K ham + 11K spam is a huge amount of training data.  The point of c-v is
that we should be able to get good results with much less.  However, as I'm
watching this run, it occurs to me that if the number of msgs predicted is
N, no scheme can possibly measure an error rate smaller than 1/N.  Grumble.
This wasn't a problem in speech recognition because there was no chance we'd
get error rates as low as this guy is getting ...

And the run is done!  Here are the pickle sizes, each representing about
27000 msgs, or about 600 bytes/msg:

16469934 class1.pik
16521522 class2.pik
16362901 class3.pik
16289795 class4.pik
16465748 class5.pik

The variance is very low.

Test results:

-> <stat> tested 4000 hams & 2750 spams against 16000 hams & 11002 spams
-> <stat> false positive %: 0.025
-> <stat> false negative %: 0.327272727273
-> <stat> 1 new false positives
-> <stat> 9 new false negatives

-> <stat> tested 4000 hams & 2750 spams against 16000 hams & 11002 spams
-> <stat> false positive %: 0.0
-> <stat> false negative %: 0.254545454545
-> <stat> 0 new false positives
-> <stat> 7 new false negatives

-> <stat> tested 4000 hams & 2750 spams against 16000 hams & 11002 spams
-> <stat> false positive %: 0.0
-> <stat> false negative %: 0.181818181818
-> <stat> 0 new false positives
-> <stat> 5 new false negatives

-> <stat> tested 4000 hams & 2751 spams against 16000 hams & 11001 spams
-> <stat> false positive %: 0.025
-> <stat> false negative %: 0.181752090149
-> <stat> 1 new false positives
-> <stat> 5 new false negatives

-> <stat> tested 4000 hams & 2751 spams against 16000 hams & 11001 spams
-> <stat> false positive %: 0.0
-> <stat> false negative %: 0.436205016358
-> <stat> 0 new false positives
-> <stat> 12 new false negatives

total unique false pos 2
total unique false neg 38

average fp % 0.01
average fn % 0.276318694029

So that's 2 fp out of 20000 ham, and 38 fn out of 13752 spam.  The reason
these are better than "my usual" results is almost certainly just because
there's 4x as much training data on each run.

One of the fps is the fellow who quoted the entire Nigerian scam msg.  The
other is poor Vickie Mills, still trying to find a Python training course in
the UK <wink>, and still getting killed by her employer's obnoxious sig.
That's all of 'em!

At least one of the f-n's is a ham by anyone's definition:  it's a technical
discussion of difficulties with Unicode, sent to the Saskatoon Linux Group
Mailing List.  BruceG may not have wanted to read it, and I know I sure
didn't <wink>, but it's straight nerd talk and not trying to sell anything.
Another f-n is output from a cron job, with subject line

    Subject: Cron <bruce@lorien> run-parts cron/hourly

and indeed looks like email he arranged to send to himself, summarizing
email statistics on a spam collection address!  I'm going to take both of
these out of the spam set.

As always, some of the f-n's remain incredibly lame -- you look at these and
marvel it catches *any* spam (let alone more then 99.5% of it).  Just one
example to make your day <wink>:  [dig it out of the attached zip file]

Finally, aggregate distribution histograms for this test:

Ham distribution for all runs:
* = 334 items
  0.00 19995 ************************************************************
  2.50     0
  5.00     1 *
  7.50     0
 10.00     0
 12.50     0
 15.00     0
 17.50     0
 20.00     0
 22.50     0
 25.00     0
 27.50     0
 30.00     0
 32.50     0
 35.00     1 *
 37.50     0
 40.00     0
 42.50     0
 45.00     0
 47.50     0
 50.00     0
 52.50     0
 55.00     0
 57.50     0
 60.00     0
 62.50     0
 65.00     0
 67.50     0
 70.00     0
 72.50     0
 75.00     1 *
 77.50     0
 80.00     0
 82.50     0
 85.00     0
 87.50     0
 90.00     0
 92.50     0
 95.00     0
 97.50     2 *

Spam distribution for all runs:
* = 229 items
  0.00    31 *
  2.50     0
  5.00     0
  7.50     1 *
 10.00     0
 12.50     0
 15.00     1 *
 17.50     0
 20.00     0
 22.50     0
 25.00     0
 27.50     0
 30.00     0
 32.50     0
 35.00     0
 37.50     0
 40.00     0
 42.50     0
 45.00     1 *
 47.50     1 *
 50.00     1 *
 52.50     1 *
 55.00     0
 57.50     0
 60.00     0
 62.50     0
 65.00     0
 67.50     0
 70.00     0
 72.50     0
 75.00     0
 77.50     0
 80.00     0
 82.50     0
 85.00     0
 87.50     1 *
 90.00     1 *
 92.50    10 *
 95.00     1 *
 97.50 13702 ************************************************************

---------------------- multipart/mixed attachment
A non-text attachment was scrubbed...
Name: spam.zip
Type: application/x-zip-compressed
Size: 1974 bytes
Desc: not available
Url : http://mail.python.org/pipermail-21/spambayes/attachments/20020913/f7f1fee9/spam.bin

---------------------- multipart/mixed attachment--


From skip@pobox.com  Sat Sep 14 05:07:24 2002
From: skip@pobox.com (Skip Montanaro)
Date: Fri, 13 Sep 2002 23:07:24 -0500
Subject: [Spambayes] Large false negative ...
Message-ID: <15746.46588.788996.794513@12-248-11-90.client.attbi.com>


I see extremely large false negative percentages.  I didn't say anything
earlier because I had relatively small training set sizes before the advent
of timcv.py (400/333 ham/spam per set).  I tried a run this evening with
timcv.  The results were similar (fn down to about 15% from around 22%).
Here's the final summary chunk from the rates.py output:

    total unique false pos 0
    total unique false neg 262
    average fp % 0.0
    average fn % 15.7357357357

Scanning through the reported false negatives, nothing much jumped out as
unusual except the viruses.  Figuring viruses were not spam and might be
throwing things off, I went through my spam collection and deleted all the
obvious viruses then rebalanced the spam sets (leaving 328 per set) and
tried again:

    total unique false pos 0
    total unique false neg 249
    average fp % 0.0
    average fn % 15.1829268293

I'm headed in the right direction, but am nowhere close to the sorts of
results Tim and others have been getting.  I'd be happy with 3-4% fn.

On a somewhat brighter note, I'm quite happy with the fp percentage...

Can someone with a larger collection of ham 'n spam try running rebal.py to
get 400 ham per set and 328 spam per set, then try "timcv -n5" and let me
know what the overall fn percentage is?  Assuming Data is a subdirectory of
the current directory and Data/{Ham,Spam}/reservoir are your two reservoirs,
you'd execute:

    rebal.py -n 400 -r Data/Ham/reservoir -s Data/Ham/Set -Q
    rebal.py -n 328 -r Data/Spam/reservoir -s Data/Spam/Set -Q

Files should be migrated to your reservoirs.  The -Q flag just shuts up
rebal.py. After your timcv run you can run rebal.py again with different -n
values to restore the numbers of ham and spam you had before.

Skip

From tim.one@comcast.net  Sat Sep 14 07:21:15 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 14 Sep 2002 02:21:15 -0400
Subject: [Spambayes] HAMBIAS
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEMGBDAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEPOBDAB.tim.one@comcast.net>

HAMBIAS=2.0 is the last deliberate bias in my rework of Graham's scoring
algorithm.  Since HAMBIAS > 1.0 artificially lowers spam probabilities, we
can expect it to cut the f-p rate at the expense of boosting the f-n rate.

Now that I can run 10-fold validation on my relatively large datasets, I
thought it might be interesting to try a pair of runs at HAMBIAS=2.0 and at
HAMBIAS=1.0.  It was <wink>.

Here are the options for the "before" run:

"""
[TestDriver]
save_trained_pickles = False
show_histograms = True
show_ham_lo = 1.0
show_best_discriminators = 50
show_spam_lo = 1.0
show_ham_hi = 0.0
show_false_positives = True
pickle_basename = class
show_false_negatives = True
nbuckets = 40
show_charlimit = 100000
show_spam_hi = 0.0

[Classifier]
spambias = 1.0
min_spamprob = 0.01
unknown_spamprob = 0.5
hambias = 2.0
max_discriminators = 16
max_spamprob = 0.99

[Tokenizer]
safe_headers = abuse-reports-to
        date
        errors-to
        from
        importance
        in-reply-to
        message-id
        mime-version
        organization
        received
        reply-to
        return-path
        subject
        to
        user-agent
        x-abuse-info
        x-complaints-to
        x-face
mine_received_headers = False
retain_pure_html_tags = False
count_all_header_lines = False
"""

Note that retain_pure_html_tags is False now:  with enough training data,
the c.l.py corpus no longer gets significant benefit from retaining HTML
tags in pure HTML msgs.

The "after" run was identical except for HAMBIAS=1.0.  The outcome, with the
10 runs each having these counts:

-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams

(that's a huge amount of training data!)

false positive percentages
    0.000  0.100  lost  +(was 0)
    0.000  0.050  lost  +(was 0)
    0.000  0.050  lost  +(was 0)
    0.000  0.050  lost  +(was 0)
    0.050  0.050  tied
    0.000  0.100  lost  +(was 0)
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.100  0.150  lost   +50.00%

won   0 times
tied  4 times
lost  6 times

total unique fp went from 3 to 11 lost  +266.67%
mean fp % went from 0.015 to 0.055 lost  +266.67%

false negative percentages
    0.218  0.145  won    -33.49%
    0.436  0.145  won    -66.74%
    0.073  0.000  won   -100.00%
    0.218  0.073  won    -66.51%
    0.218  0.073  won    -66.51%
    0.291  0.000  won   -100.00%
    0.291  0.218  won    -25.09%
    0.218  0.145  won    -33.49%
    0.291  0.000  won   -100.00%
    0.073  0.000  won   -100.00%

won  10 times
tied  0 times
lost  0 times

total unique fn went from 32 to 11 won    -65.63%
mean fn % went from 0.232727272727 to 0.0800000000002 won    -65.62%

The rates are too low to measure reliably, but check it out anyway:  the f-n
and f-p rates are very close after (vs a factor of 17 apart before), and
there are 11 mistakes of each kind after (vs 3 fn and 32 fp before).

The obvious conjecture is that, given enough training data, removing the
last deliberate bias in fact yields results favoring neither ham nor spam at
the expense of the other.

That's comforting, at least from a useless theoretical POV <wink>.


From tim.one@comcast.net  Sat Sep 14 08:04:51 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 14 Sep 2002 03:04:51 -0400
Subject: [Spambayes] Large false negative ...
In-Reply-To: <15746.46588.788996.794513@12-248-11-90.client.attbi.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEPPBDAB.tim.one@comcast.net>

[Skip Montanaro]
> I see extremely large false negative percentages.  I didn't say anything
> earlier because I had relatively small training set sizes before
> the advent of timcv.py (400/333 ham/spam per set).  I tried a run this
> evening with timcv.  The results were similar (fn down to about 15% from
> around 22%).

That's a huge decrease, and the primary effect of timcv is to increase the
amount of data trained on per prediction run.

> Here's the final summary chunk from the rates.py output:
>
>     total unique false pos 0
>     total unique false neg 262
>     average fp % 0.0
>     average fn % 15.7357357357

There's not enough here, Skip.  rates.py prints more stuff than that, and
does so because it's important information.  timcv also prints all the
options in effect right at the start, because that's important info too.
All that showing just these 4 numbers can tell me is that, ya, your fn rate
is supernaturally high and your fp rate supernaturally low.  The list of
best discriminators (provided you have that enabled -- but I can't guess
that either <wink>) often has the best clues.

> Scanning through the reported false negatives, nothing much jumped out as
> unusual except the viruses.  Figuring viruses were not spam and might be
> throwing things off, I went through my spam collection and deleted all
> the obvious viruses then rebalanced the spam sets (leaving 328 per set)
> and tried again:
>
>     total unique false pos 0
>     total unique false neg 249
>     average fp % 0.0
>     average fn % 15.1829268293
>
> I'm headed in the right direction,

It would be easier to have confidence in that had you run cmp.py against the
two summary files and posted the comparison.  You're effectively revealing
two numbers per test run, and that's all ("total unique" is close to 100%
correlated with "average" in timcv output, so half the numbers I'm seeing
here are redundant).

> but am nowhere close to the sorts of results Tim and others have been
> getting.

Note that I'm using a factor of about 45x more training data.

> I'd be happy with 3-4% fn.

Try using 50x more data <wink>.

> On a somewhat brighter note, I'm quite happy with the fp percentage...

You shouldn't be, though -- the extreme imbalance in rates is as suspicious
as the absolute magnitude of the fn rate.  It's as if all your ham have
something trivial in common that the classifier is latching onto (wouldn't
be the first time someone got tripped up by this!), and that a fair amount
of your spam also has that.  Looking at the best discriminators may reveal
something of this nature.

> Can someone with a larger collection of ham 'n spam try running
> rebal.py to get 400 ham per set and 328 spam per set, then try
> "timcv -n5" and let me know what the overall fn percentage is?

Is it the case that you're only running 5-fold c-v?

> Assuming Data is a subdirectory of the current directory and
> Data/{Ham,Spam}/reservoir are your two reservoirs, you'd execute:
>
>     rebal.py -n 400 -r Data/Ham/reservoir -s Data/Ham/Set -Q
>     rebal.py -n 328 -r Data/Spam/reservoir -s Data/Spam/Set -Q
>
> Files should be migrated to your reservoirs.  The -Q flag just shuts up
> rebal.py. After your timcv run you can run rebal.py again with
> different -n values to restore the numbers of ham and spam you had
> before.

rebal was very helpful in rearranging my directories, and thanks for
generalizing it!  For those with a larger corpus, I suggest it's easier to
fiddle MsgStream.produce to pick smaller subsets at random; e.g.,

    def produce(self):
        import random
        keep = 'Spam' in self.directories[0] and 328 or 400
        for directory in self.directories:
            all = os.listdir(directory)
            random.seed(hash(max(all))) # reproducible across calls
            random.shuffle(all)
            for fname in all[:keep]:
                yield Msg(directory, fname)

Here are the options I used:

"""
[TestDriver]
save_trained_pickles = False
show_histograms = True
show_ham_lo = 1.0
show_best_discriminators = 50
show_spam_lo = 1.0
show_ham_hi = 0.0
show_false_positives = True
pickle_basename = class
show_false_negatives = True
nbuckets = 40
show_charlimit = 100000
show_spam_hi = 0.0

[Classifier]
spambias = 1.0
min_spamprob = 0.01
unknown_spamprob = 0.5
hambias = 2.0
max_discriminators = 16
max_spamprob = 0.99

[Tokenizer]
safe_headers = abuse-reports-to
        date
        errors-to
        from
        importance
        in-reply-to
        message-id
        mime-version
        organization
        received
        reply-to
        return-path
        subject
        to
        user-agent
        x-abuse-info
        x-complaints-to
        x-face
mine_received_headers = False
retain_pure_html_tags = False
count_all_header_lines = False
"""

Here's the summary file:

"""
-> Training on Data/Ham/Set2-5 & Data/Spam/Set2-5 ... 1600 hams & 1312 spams
-> Predicting Data/Ham/Set1 & Data/Spam/Set1 ...
-> <stat> tested 400 hams & 328 spams against 1600 hams & 1312 spams
-> <stat> false positive %: 0.0
-> <stat> false negative %: 0.30487804878
      0.000   0.305
-> <stat> 0 new false positives
-> <stat> 1 new false negatives
-> Training on Data/Ham/Set1 & Data/Spam/Set1 ... 400 hams & 328 spams
-> Forgetting Data/Ham/Set2 & Data/Spam/Set2 ... 400 hams & 328 spams
-> Predicting Data/Ham/Set2 & Data/Spam/Set2 ...
-> <stat> tested 400 hams & 328 spams against 1600 hams & 1312 spams
-> <stat> false positive %: 0.0
-> <stat> false negative %: 0.0
      0.000   0.000
-> <stat> 0 new false positives
-> <stat> 0 new false negatives
-> Training on Data/Ham/Set2 & Data/Spam/Set2 ... 400 hams & 328 spams
-> Forgetting Data/Ham/Set3 & Data/Spam/Set3 ... 400 hams & 328 spams
-> Predicting Data/Ham/Set3 & Data/Spam/Set3 ...
-> <stat> tested 400 hams & 328 spams against 1600 hams & 1312 spams
-> <stat> false positive %: 0.5
-> <stat> false negative %: 0.609756097561
      0.500   0.610
-> <stat> 2 new false positives
-> <stat> 2 new false negatives
-> Training on Data/Ham/Set3 & Data/Spam/Set3 ... 400 hams & 328 spams
-> Forgetting Data/Ham/Set4 & Data/Spam/Set4 ... 400 hams & 328 spams
-> Predicting Data/Ham/Set4 & Data/Spam/Set4 ...
-> <stat> tested 400 hams & 328 spams against 1600 hams & 1312 spams
-> <stat> false positive %: 0.0
-> <stat> false negative %: 0.0
      0.000   0.000
-> <stat> 0 new false positives
-> <stat> 0 new false negatives
-> Training on Data/Ham/Set4 & Data/Spam/Set4 ... 400 hams & 328 spams
-> Forgetting Data/Ham/Set5 & Data/Spam/Set5 ... 400 hams & 328 spams
-> Predicting Data/Ham/Set5 & Data/Spam/Set5 ...
-> <stat> tested 400 hams & 328 spams against 1600 hams & 1312 spams
-> <stat> false positive %: 0.25
-> <stat> false negative %: 0.0
      0.250   0.000
-> <stat> 1 new false positives
-> <stat> 0 new false negatives
total unique false pos 3
total unique false neg 3
average fp % 0.15
average fn % 0.182926829268
"""

It's clearly very much better than you're seeing.

Here are the score distributions:

Ham distribution for all runs:
* = 34 items
  0.00 1994 ***********************************************************
  2.50    0
  5.00    0
  7.50    0
 10.00    0
 12.50    0
 15.00    0
 17.50    0
 20.00    1 *
 22.50    0
 25.00    0
 27.50    0
 30.00    0
 32.50    0
 35.00    0
 37.50    0
 40.00    1 *
 42.50    0
 45.00    0
 47.50    0
 50.00    0
 52.50    0
 55.00    1 *
 57.50    0
 60.00    0
 62.50    0
 65.00    0
 67.50    0
 70.00    0
 72.50    0
 75.00    0
 77.50    0
 80.00    0
 82.50    0
 85.00    0
 87.50    0
 90.00    0
 92.50    0
 95.00    0
 97.50    3 *

Spam distribution for all runs:
* = 28 items
  0.00    1 *
  2.50    0
  5.00    0
  7.50    0
 10.00    0
 12.50    0
 15.00    0
 17.50    0
 20.00    0
 22.50    0
 25.00    0
 27.50    0
 30.00    0
 32.50    0
 35.00    1 *
 37.50    0
 40.00    0
 42.50    0
 45.00    0
 47.50    0
 50.00    0
 52.50    0
 55.00    0
 57.50    0
 60.00    0
 62.50    0
 65.00    0
 67.50    0
 70.00    0
 72.50    0
 75.00    0
 77.50    1 *
 80.00    0
 82.50    0
 85.00    0
 87.50    0
 90.00    0
 92.50    1 *
 95.00    0
 97.50 1636 ***********************************************************

Here are the best 20 discriminators from the last run:

        'url:python' 188 0.01
        'content-type:text/html' 193 0.97645
        'module' 194 0.01
        'header:Return-Path:2' 211 0.99
        'unsubscribe' 222 0.99
        'def' 235 0.01
        'wrote' 238 0.0120086
        'import' 255 0.01
        'header:Received:7' 267 0.99
        'header:In-Reply-To:1' 292 0.01
        'url:gif' 304 0.99
        'url:remove' 315 0.99
        'header:Received:8' 333 0.99
        'subject:Python' 437 0.01
        'header:Errors-To:1' 454 0.0201643
        'header:User-Agent:1' 471 0.01
        'header:Organization:1' 634 0.0120482
        'wrote:' 864 0.01
        'header:X-Complaints-To:1' 885 0.01
        'python' 1000 0.01


From skip@pobox.com  Sat Sep 14 13:56:46 2002
From: skip@pobox.com (Skip Montanaro)
Date: Sat, 14 Sep 2002 07:56:46 -0500
Subject: [Spambayes] Large false negative ...
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEPPBDAB.tim.one@comcast.net>
References: <15746.46588.788996.794513@12-248-11-90.client.attbi.com>
        <LNBBLJKPBEHFEDALKOLCGEPPBDAB.tim.one@comcast.net>
Message-ID: <15747.12814.571767.593491@12-248-11-90.client.attbi.com>


    >> Here's the final summary chunk from the rates.py output:

    Tim> There's not enough here, Skip.  rates.py prints more stuff than
    Tim> that, and does so because it's important information.  

Okay, I've tried to mimic your settings and report more information below.
Here are my latest settings:

    [TestDriver]
    save_trained_pickles = False
    show_histograms = True
    show_ham_lo = 1.0
    show_best_discriminators = 50
    show_spam_lo = 1.0
    show_ham_hi = 0.0
    show_false_positives = True
    pickle_basename = class
    show_false_negatives = True
    nbuckets = 40
    show_charlimit = 3000
    show_spam_hi = 0.0

    [Classifier]
    spambias = 1.0
    min_spamprob = 0.01
    unknown_spamprob = 0.5
    hambias = 2.0
    max_discriminators = 16
    max_spamprob = 0.99

    [Tokenizer]
    safe_headers = abuse-reports-to
            date
            errors-to
            from
            importance
            in-reply-to
            message-id
            mime-version
            organization
            received
            reply-to
            return-path
            subject
            to
            user-agent
            x-abuse-info
            x-complaints-to
            x-face
    mine_received_headers = False
    retain_pure_html_tags = False
    count_all_header_lines = False


    Tim> Note that I'm using a factor of about 45x more training data.

    >> I'd be happy with 3-4% fn.

    Tim> Try using 50x more data <wink>.

I would have thought that I'd be within spitting distance of the sorts of
numbers you reported with the 5x5 scheme.  I'm training on 1600 hams and
1300 spams.  You were training on 4000 hams and 2750 spams I believe.

    >> On a somewhat brighter note, I'm quite happy with the fp percentage...

    Tim> You shouldn't be, though -- the extreme imbalance in rates is as
    Tim> suspicious as the absolute magnitude of the fn rate.  It's as if
    Tim> all your ham have something trivial in common that the classifier
    Tim> is latching onto (wouldn't be the first time someone got tripped up
    Tim> by this!), and that a fair amount of your spam also has that.
    Tim> Looking at the best discriminators may reveal something of this
    Tim> nature.

Here's the set of best discriminators from my latest run:

        'x-mailer:none' 96 0.481519
        'header:MIME-Version:1' 97 0.282956
        'subject:GB2312' 97 0.99
        'idaho' 100 0.01
        '8bit%:92' 101 0.99
        'url:dolphin' 102 0.01
        'url:listinfo' 102 0.0329182
        'felt' 104 0.01
        'header:Subject:1' 104 0.497099
        'kept)' 104 0.01
        '(text' 105 0.01
        'cedu.' 109 0.01
        'charset:gb2312' 111 0.99
        'email addr:dolphin.mojam.com' 112 0.01
        'stripmime' 112 0.01
        'subject:CEDU' 112 0.01
        'header:Message-ID:1' 116 0.245064
        '[cedu]' 117 0.01
        'header:Errors-To:1' 117 0.0915332
        'jan' 117 0.01
        'content-type:text/plain' 118 0.405034
        'header:User-Agent:1' 118 0.01
        'workshop' 120 0.01
        'daughter' 123 0.01
        'venue' 123 0.01
        'rma' 124 0.01
        'folk' 125 0.01
        'header:To:1' 126 0.495146
        'from:email addr:aol.com' 135 0.0148768
        'para' 137 0.99
        'skip' 141 0.0122271
        'concert' 142 0.01
        'header:Message-Id:1' 146 0.60187
        'header:Received:1' 156 0.964159
        'parents' 158 0.0109178
        'ascent' 161 0.01
        'parent' 180 0.01
        '8bit%:100' 181 0.99
        'bca' 192 0.01
        'wrote:' 202 0.01
        'sent:' 213 0.01
        'montanaro' 215 0.01
        'header:In-Reply-To:1' 245 0.01
        'kids' 250 0.01
        'url:manatee' 485 0.01
        'email addr:manatee.mojam.com' 526 0.01
        'url:cedu' 570 0.01
        'email name:cedu' 600 0.01
        'subject:cedu' 685 0.01
        'cedu' 694 0.01

A few things jump out at me looking at this list:

 * There are very few 0.99's.  I would have thought some of the obvious spam
   tripwords would have made it into the set.

 * It's a bit suspicious that "dolphin" and "dolphin.mojam.com" are such a
   good ham indicators.  I haven't used it as my email machine for quite
   awhile (a couple years at least).  In fact, it's been turned off for the
   past year or so.

 * There are lots of 0.01's associated with "cedu" ("workshop", "daughter",
   "[cedu]", "idaho", etc).  I participate in three pretty disjoint
   communities on the net: Python, CEDU and old British cars.  Note that
   there are no Python terms in the discriminator set (not even "Guido" or
   "Tim").  I have a ton of unread CEDU mail I dumped into the training set,
   but have very little Python data saved.  Can the training data be
   overwhelmed by too much input from one population?

    Tim> Is it the case that you're only running 5-fold c-v?

Yes.  400 hams, now down to 325 spams per Set[1-5] directory.

    Tim> For those with a larger corpus, I suggest it's easier to fiddle
    Tim> MsgStream.produce to pick smaller subsets at random; e.g.,

    Tim>     def produce(self):
    Tim>         import random
    Tim>         keep = 'Spam' in self.directories[0] and 328 or 400
    Tim>         for directory in self.directories:
    Tim>             all = os.listdir(directory)
    Tim>             random.seed(hash(max(all))) # reproducible across calls
    Tim>             random.shuffle(all)
    Tim>             for fname in all[:keep]:
    Tim>                 yield Msg(directory, fname)

Perhaps the keep values should be settable options?

    Tim> Here's the summary file:

Here's my latest summary file:

-> Training on Data/Ham/Set2-5 & Data/Spam/Set2-5 ... 1600 hams & 1300 spams
-> Predicting Data/Ham/Set1 & Data/Spam/Set1 ...
-> <stat> tested 400 hams & 325 spams against 1600 hams & 1300 spams
-> <stat> false positive %: 0.0
-> <stat> false negative %: 16.3076923077
      0.000  16.308
-> <stat> 0 new false positives
-> <stat> 53 new false negatives
-> Training on Data/Ham/Set1 & Data/Spam/Set1 ... 400 hams & 325 spams
-> Forgetting Data/Ham/Set2 & Data/Spam/Set2 ... 400 hams & 325 spams
-> Predicting Data/Ham/Set2 & Data/Spam/Set2 ...
-> <stat> tested 400 hams & 325 spams against 1600 hams & 1300 spams
-> <stat> false positive %: 0.0
-> <stat> false negative %: 14.4615384615
      0.000  14.462
-> <stat> 0 new false positives
-> <stat> 47 new false negatives
-> Training on Data/Ham/Set2 & Data/Spam/Set2 ... 400 hams & 325 spams
-> Forgetting Data/Ham/Set3 & Data/Spam/Set3 ... 400 hams & 325 spams
-> Predicting Data/Ham/Set3 & Data/Spam/Set3 ...
-> <stat> tested 400 hams & 325 spams against 1600 hams & 1300 spams
-> <stat> false positive %: 0.0
-> <stat> false negative %: 14.7692307692
      0.000  14.769
-> <stat> 0 new false positives
-> <stat> 48 new false negatives
-> Training on Data/Ham/Set3 & Data/Spam/Set3 ... 400 hams & 325 spams
-> Forgetting Data/Ham/Set4 & Data/Spam/Set4 ... 400 hams & 325 spams
-> Predicting Data/Ham/Set4 & Data/Spam/Set4 ...
-> <stat> tested 400 hams & 325 spams against 1600 hams & 1300 spams
-> <stat> false positive %: 0.0
-> <stat> false negative %: 15.3846153846
      0.000  15.385
-> <stat> 0 new false positives
-> <stat> 50 new false negatives
-> Full commercial single (1-3 songs) released to major stores and DJ's
-> Major recording, engineering, and production from a Sonic Wave producer
-> Professional Graphic Design (industry recognized and popular graphic
-> Unlimited Manufacturing and complete distribution to all major music 
-> 15% royalty on sales
-> sales analysis and recommendations
-> professional reports and graphs.
-> Training on Data/Ham/Set4 & Data/Spam/Set4 ... 400 hams & 325 spams
-> Forgetting Data/Ham/Set5 & Data/Spam/Set5 ... 400 hams & 325 spams
-> Predicting Data/Ham/Set5 & Data/Spam/Set5 ...
-> <stat> tested 400 hams & 325 spams against 1600 hams & 1300 spams
-> <stat> false positive %: 0.0
-> <stat> false negative %: 15.3846153846
      0.000  15.385
-> <stat> 0 new false positives
-> <stat> 50 new false negatives
total unique false pos 0
total unique false neg 248
average fp % 0.0
average fn % 15.2615384615

    Tim> Here are the score distributions:

And mine:

Ham distribution for all runs:
* = 34 items
  0.00 2000 ***********************************************************
  2.50    0 
  5.00    0 
  7.50    0 
 10.00    0 
 12.50    0 
 15.00    0 
 17.50    0 
 20.00    0 
 22.50    0 
 25.00    0 
 27.50    0 
 30.00    0 
 32.50    0 
 35.00    0 
 37.50    0 
 40.00    0 
 42.50    0 
 45.00    0 
 47.50    0 
 50.00    0 
 52.50    0 
 55.00    0 
 57.50    0 
 60.00    0 
 62.50    0 
 65.00    0 
 67.50    0 
 70.00    0 
 72.50    0 
 75.00    0 
 77.50    0 
 80.00    0 
 82.50    0 
 85.00    0 
 87.50    0 
 90.00    0 
 92.50    0 
 95.00    0 
 97.50    0 

Spam distribution for all runs:
* = 23 items
  0.00  213 **********
  2.50    7 *
  5.00    3 *
  7.50    3 *
 10.00    3 *
 12.50    0 
 15.00    2 *
 17.50    0 
 20.00    1 *
 22.50    2 *
 25.00    0 
 27.50    1 *
 30.00    0 
 32.50    0 
 35.00    0 
 37.50    0 
 40.00    0 
 42.50    0 
 45.00    1 *
 47.50    0 
 50.00    0 
 52.50    2 *
 55.00    0 
 57.50    0 
 60.00    1 *
 62.50    0 
 65.00    2 *
 67.50    0 
 70.00    1 *
 72.50    1 *
 75.00    0 
 77.50    1 *
 80.00    2 *
 82.50    0 
 85.00    1 *
 87.50    1 *
 90.00    2 *
 92.50    4 *
 95.00    1 *
 97.50 1370 ************************************************************

My best guess at this point is that lots of spam goes to ancient addresses
and I've mixed in a fair amount of old (but ham) mail which was sent to such
old addresses.  Perhaps deleting old ham from the corpus will improve
things.  I'll try moving old mail out of the way temporarily and see how
that changes the results.

Skip

From nas@python.ca  Sat Sep 14 16:18:07 2002
From: nas@python.ca (Neil Schemenauer)
Date: Sat, 14 Sep 2002 08:18:07 -0700
Subject: [Spambayes] Testing methodology
In-Reply-To: <LNBBLJKPBEHFEDALKOLCOEPIBDAB.tim@zope.com>
References: <LNBBLJKPBEHFEDALKOLCKEMGBDAB.tim.one@comcast.net>
	<LNBBLJKPBEHFEDALKOLCOEPIBDAB.tim@zope.com>
Message-ID: <20020914151807.GA9581@glacier.arctrix.com>

Here are my results using the default options:

-> Training on Data/Ham/Set2-6 & Data/Spam/Set2-6 ... 1500 hams & 1500 spams
-> Predicting Data/Ham/Set1 & Data/Spam/Set1 ...
-> <stat> tested 300 hams & 300 spams against 1500 hams & 1500 spams
-> <stat> false positive %: 0.333333333333
-> <stat> false negative %: 0.666666666667
      0.333   0.667
-> <stat> 1 new false positives
-> <stat> 2 new false negatives
-> Training on Data/Ham/Set1 & Data/Spam/Set1 ... 300 hams & 300 spams
-> Forgetting Data/Ham/Set2 & Data/Spam/Set2 ... 300 hams & 300 spams
-> Predicting Data/Ham/Set2 & Data/Spam/Set2 ...
-> <stat> tested 300 hams & 300 spams against 1500 hams & 1500 spams
-> <stat> false positive %: 0.0
-> <stat> false negative %: 1.33333333333
      0.000   1.333
-> <stat> 0 new false positives
-> <stat> 4 new false negatives
-> Training on Data/Ham/Set2 & Data/Spam/Set2 ... 300 hams & 300 spams
-> Forgetting Data/Ham/Set3 & Data/Spam/Set3 ... 300 hams & 300 spams
-> Predicting Data/Ham/Set3 & Data/Spam/Set3 ...
-> <stat> tested 300 hams & 300 spams against 1500 hams & 1500 spams
-> <stat> false positive %: 1.0
-> <stat> false negative %: 1.66666666667
      1.000   1.667
-> <stat> 3 new false positives
-> <stat> 5 new false negatives
-> Training on Data/Ham/Set3 & Data/Spam/Set3 ... 300 hams & 300 spams
-> Forgetting Data/Ham/Set4 & Data/Spam/Set4 ... 300 hams & 300 spams
-> Predicting Data/Ham/Set4 & Data/Spam/Set4 ...
-> <stat> tested 300 hams & 300 spams against 1500 hams & 1500 spams
-> <stat> false positive %: 0.333333333333
-> <stat> false negative %: 0.333333333333
      0.333   0.333
-> <stat> 1 new false positives
-> <stat> 1 new false negatives
-> Training on Data/Ham/Set4 & Data/Spam/Set4 ... 300 hams & 300 spams
-> Forgetting Data/Ham/Set5 & Data/Spam/Set5 ... 300 hams & 300 spams
-> Predicting Data/Ham/Set5 & Data/Spam/Set5 ...
-> <stat> tested 300 hams & 300 spams against 1500 hams & 1500 spams
-> <stat> false positive %: 0.0
-> <stat> false negative %: 2.0
      0.000   2.000
-> <stat> 0 new false positives
-> <stat> 6 new false negatives
-> Training on Data/Ham/Set5 & Data/Spam/Set5 ... 300 hams & 300 spams
-> Forgetting Data/Ham/Set6 & Data/Spam/Set6 ... 300 hams & 300 spams
-> Predicting Data/Ham/Set6 & Data/Spam/Set6 ...
-> <stat> tested 300 hams & 300 spams against 1500 hams & 1500 spams
-> <stat> false positive %: 0.0
-> <stat> false negative %: 2.0
      0.000   2.000
-> <stat> 0 new false positives
-> <stat> 6 new false negatives
total unique false pos 5
total unique false neg 24
average fp % 0.277777777778
average fn % 1.33333333333

The fp are mostly pretty questionable things (e.g. a joke in the form of
a sales pitch forwarded to me).  Turning on mine_received_headers gives
a small improvement:

false positive percentages
    0.333  0.333  tied
    0.000  0.000  tied
    1.000  1.000  tied
    0.333  0.000  won   -100.00%
    0.000  0.000  tied
    0.000  0.000  tied

won   1 times
tied  5 times
lost  0 times

total unique fp went from 5 to 4 won    -20.00%
mean fp % went from 0.277777777778 to 0.222222222222 won    -20.00%

false negative percentages
    0.667  0.667  tied
    1.333  0.667  won    -49.96%
    1.667  1.000  won    -40.01%
    0.333  0.333  tied
    2.000  1.667  won    -16.65%
    2.000  1.000  won    -50.00%

won   4 times
tied  2 times
lost  0 times

total unique fn went from 24 to 16 won    -33.33%
mean fn % went from 1.33333333333 to 0.88888888889 won    -33.33%

  Neil

From gward@python.net  Sat Sep 14 16:34:08 2002
From: gward@python.net (Greg Ward)
Date: Sat, 14 Sep 2002 11:34:08 -0400
Subject: [Spambayes] Testing methodology
In-Reply-To: <LNBBLJKPBEHFEDALKOLCOEPIBDAB.tim@zope.com>
References: <LNBBLJKPBEHFEDALKOLCKEMGBDAB.tim.one@comcast.net>
	<LNBBLJKPBEHFEDALKOLCOEPIBDAB.tim@zope.com>
Message-ID: <20020914153408.GA3254@cthulhu.gerg.ca>

On 13 September 2002, Tim Peters said:
> [double heh!  now python.org blacklisted mu email address for "sending
>  spam", so the second attempt also bounced -- third time's the charm?]

Hmmm -- you managed to tickle SpamAssassin so hard (by quoting spam)
that my local_scan() ended up rejecting your message rather than just
quarantining it.  That landed you (or rather, the pair
("tim.one@comcast.net", 24.153.64.2)) on the automated sender blacklist.
Luckily ASBL entries only last for three hours, so the third time was
indeed the charm.

I'm going to add tim.one@comcast.net to the sender whitelist, and
spambayes@python.org to the recipient whitelist.  That'll knock 20
points off anything you send to this list, which should pretty much keep
SpamAssassin from bothering you again.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Life is too short for ordinary music.

From tim.one@comcast.net  Sat Sep 14 17:22:37 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 14 Sep 2002 12:22:37 -0400
Subject: [Spambayes] Testing methodology
In-Reply-To: <20020914153408.GA3254@cthulhu.gerg.ca>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEAIBEAB.tim.one@comcast.net>

[Tim]
> [double heh!  now python.org blacklisted mu email address for "sending
>  spam", so the second attempt also bounced -- third time's the charm?]

[Greg Ward]
> Hmmm -- you managed to tickle SpamAssassin so hard (by quoting spam)
> that my local_scan() ended up rejecting your message rather than just
> quarantining it.  That landed you (or rather, the pair
> ("tim.one@comcast.net", 24.153.64.2)) on the automated sender blacklist.
> Luckily ASBL entries only last for three hours, so the third time was
> indeed the charm.

The third time I sent it using a different email account, so it didn't
tickle the blacklist.  Then the message was blocked by Mailman, for
"non-member posting to a moderated list".  Luckily, I'm a list admin for
this list, so I was able to approve it myself.  All this joy just so
python.org clients can remain ignorant of how small their penises really are
<wink>?

> I'm going to add tim.one@comcast.net to the sender whitelist, and
> spambayes@python.org to the recipient whitelist.  That'll knock 20
> points off anything you send to this list, which should pretty much keep
> SpamAssassin from bothering you again.

This is encouraging, but I don't think it's quite yet time to reveal that
it's all part of my plan to become an unstoppable spammer.


From tim.one@comcast.net  Sat Sep 14 18:31:25 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 14 Sep 2002 13:31:25 -0400
Subject: [Spambayes] Large false negative ...
In-Reply-To: <15747.12814.571767.593491@12-248-11-90.client.attbi.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEAKBEAB.tim.one@comcast.net>

[Skip Montanaro, reveals more info]
> Here's the set of best discriminators from my latest run:

Very important:  such a list gets printed 5 times during an -n5 run.  Are
you showing the first such list, the last such list, or ...?  I showed you
the last such list I got (that's what I meant by "the last run").  Without
knowing which list it is, it's impossible to interpret the displayed
killcounts (they generally get larger from first run to last, and I can't
tell whether you're seeing high killcounts on the first run or low
killcounts on the last run).

>         '8bit%:92' 101 0.99
>         '8bit%:100' 181 0.99

Do you have a lot of Asian spam?  It appears that my (bruceg's) collection
doesn't, as I never see an '8bit%:nn' in the top 50 discriminators.  The
tokenizer really doesn't have a clue about what to do with high-bit
languages, but without a corpus that contains a bunch of that stuff I can't
test strategies for tokenizing it effectively.  Someone else should jump on
this!

>         'x-mailer:none' 96 0.481519
>         'header:Subject:1' 104 0.497099
>         'content-type:text/plain' 118 0.405034
>         'header:To:1' 126 0.495146
>         'header:Message-Id:1' 146 0.60187

This is bizarre:  this means, for example, that in 126 messages, a clue with
spamprob 0.495146 was one of the 16 *strongest* clues.  Strongest ==
farthest from 0.5, but 0.495146 essentially *is* 0.5 -- it's no clue at all.
I've never seen even one word with a probability near 0.5 in the list of top
50 discriminators, and you've got 5 like that.  More, they're all derived
from the headers.  It's as if it's not finding a message *body* in these
cases (else it would find *something* better than the mere presence of a
"To:" line to pick on!).  It's also curious that the presence of a
Message-Id line is a (weak) spam indicator in your data.

>         'from:email addr:aol.com' 135 0.0148768

You may have the world's only corpus where email from AOL is a Good Thing
<wink>.

>         'charset:gb2312' 111 0.99

Fascinating!

>         'subject:GB2312' 97 0.99

Weird.

>         'idaho' 100 0.01

Disturbing <wink>.

The lack of 'url:gif' and 'url:remove' as spam indicators is surprising to
me.

The top 4:

>         'url:cedu' 570 0.01
>         'email name:cedu' 600 0.01
>         'subject:cedu' 685 0.01
>         'cedu' 694 0.01

>  * There are very few 0.99's.  I would have thought some of the
>    obvious spam tripwords would have made it into the set.

Me too.  It's possible that you have a lot of spam of the form Anthony
warned about:

    multipart/alternative
        text/html
            The real spam is here.
        text/plain
            Something innocuous is here.

In that case, the tokenizer ignores the text/html part, and only looks at
the text/plain part.  As I've said to the point of obnoxiousness, though,
the changes I make are driven by test results, so if my corpus doesn't have
stuff like that I'm not going to do anything about it (indeed, given the
spectacularly low rates I'm getting via this approach, I couldn't even
measure an improvement on my data if it *did* have such stuff).

You could do a gross test to see whether this is important in your data by
changing the last line of tokenizer.textparts() from

    return text - redundant_html

to

    return text + redundant_html

> ...
> I have a ton of unread CEDU mail I dumped into the training set,

If you haven't read it, how do you know whether it's ham or spam?

>    but have very little Python data saved.  Can the training data be
>    overwhelmed by too much input from one population?

Since I got rid of MINCOUNT, a clue in a single ham message is as strong as
a clue that appears in every ham message.  This makes me squirm, but testing
said it was a win.  (BTW, I've tried, but not reported on, a dozen schemes
to give less weight to high-prob clues that appear in few messages; every
such attempt has been a loser.)

Can you set

    show_false_positives = False
    show_false_negatives = True
    show_charlimit = 500000

and stick full output from a run on the web somewhere?  Suppressing false
positives means we won't see any of your ham.  Setting show_charlimit means
the output will contain the full content of the false negatives (provided
they're under half a meg).


From tim.one@comcast.net  Sat Sep 14 19:10:59 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 14 Sep 2002 14:10:59 -0400
Subject: [Spambayes] Testing methodology
In-Reply-To: <20020914151807.GA9581@glacier.arctrix.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEALBEAB.tim.one@comcast.net>

[Neil Schemenauer, running a 6-fold cross validation]

> Here are my results using the default options:
>
> [assorted lines elided by tim]
>
> -> <stat> tested 300 hams & 300 spams against 1500 hams & 1500 spams
>       0.333   0.667
> -> <stat> 1 new false positives
> -> <stat> 2 new false negatives

> -> <stat> tested 300 hams & 300 spams against 1500 hams & 1500 spams
>       0.000   1.333
> -> <stat> 0 new false positives
> -> <stat> 4 new false negatives

> -> <stat> tested 300 hams & 300 spams against 1500 hams & 1500 spams
>       1.000   1.667
> -> <stat> 3 new false positives
> -> <stat> 5 new false negatives

> -> <stat> tested 300 hams & 300 spams against 1500 hams & 1500 spams
>       0.333   0.333
> -> <stat> 1 new false positives
> -> <stat> 1 new false negatives

> -> <stat> tested 300 hams & 300 spams against 1500 hams & 1500 spams
>       0.000   2.000
> -> <stat> 0 new false positives
> -> <stat> 6 new false negatives

> -> <stat> tested 300 hams & 300 spams against 1500 hams & 1500 spams
>       0.000   2.000
> -> <stat> 0 new false positives
> -> <stat> 6 new false negatives

> total unique false pos 5
> total unique false neg 24

> average fp % 0.277777777778
> average fn % 1.33333333333

On a similar 6-fold all-defaults run, also with

-> <stat> tested 300 hams & 300 spams against 1500 hams & 1500 spams

my bottom line was

total unique false pos 3
total unique false neg 6
average fp % 0.166666666667
average fn % 0.333333333333

If I take recieved, return-path, and in-reply-to out of the default
safe-headers option, it gets worse:

total unique false pos 3
total unique false neg 9
average fp % 0.166666666667
average fn % 0.5

I still have a nagging suspicion that just counting received and return-path
instances is discriminating between c.l.py and bruceg's spam for bogus
reasons.

> The fp are mostly pretty questionable things (e.g. a joke in the form of
> a sales pitch forwarded to me).  Turning on mine_received_headers gives
> a small improvement:

I disagree!  It's a major improvement.  Indeed, cutting the f-n rate by a
third, and without harming the f-p rate, is a huge win.  From my time in
speech recog, I've been very careful all along to quote error rates instead
of accuracy rates, because it seems psychologically impossible for people to
grok that improving accuracy from, say, 98% to 99% is a spectacular
improvement.  Phrasing that as cutting the error rate from 2% to 1% says
exactly the same thing, but for whatever reason makes it much clearer that
performance has gotten twice as good.  Alas, when the error rates get under
1% (which they never did in speech recog), it appears it's also
psychologically impossible not to think of them as "well, two small numbers
are pretty alike -- no big deal either way".  But cutting an error rate in
half is still a spectacular improvement, and is actually harder to do the
smaller the absolute rate to begin with.  Cutting one by a third is merely a
huge win <wink>.

> false positive percentages
>     0.333  0.333  tied
>     0.000  0.000  tied
>     1.000  1.000  tied
>     0.333  0.000  won   -100.00%
>     0.000  0.000  tied
>     0.000  0.000  tied
>
> won   1 times
> tied  5 times
> lost  0 times
>
> total unique fp went from 5 to 4 won    -20.00%
> mean fp % went from 0.277777777778 to 0.222222222222 won    -20.00%
>
> false negative percentages
>     0.667  0.667  tied
>     1.333  0.667  won    -49.96%
>     1.667  1.000  won    -40.01%
>     0.333  0.333  tied
>     2.000  1.667  won    -16.65%
>     2.000  1.000  won    -50.00%
>
> won   4 times
> tied  2 times
> lost  0 times
>
> total unique fn went from 24 to 16 won    -33.33%
> mean fn % went from 1.33333333333 to 0.88888888889 won    -33.33%

That's a pure win, and a big one.


From tim.one@comcast.net  Sat Sep 14 19:42:04 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 14 Sep 2002 14:42:04 -0400
Subject: [Spambayes] Large false negative ...
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEAKBEAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEAMBEAB.tim.one@comcast.net>

[Tim]
> ...
> You could do a gross test to see whether this is important in your data
> by changing the last line of tokenizer.textparts() from
>
>     return text - redundant_html
> to
>     return text + redundant_html

Oops!  Make that

    return text | redundant_html

"+" gives a TypeError.

From tim.one@comcast.net  Sat Sep 14 21:13:31 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 14 Sep 2002 16:13:31 -0400
Subject: [Spambayes] Large false negative ...
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEAKBEAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEBABEAB.tim.one@comcast.net>

[Tim]
> ...
> It's possible that you have a lot of spam of the form Anthony
> warned about:
>
>     multipart/alternative
>         text/html
>             The real spam is here.
>         text/plain
>             Something innocuous is here.
>
> In that case, the tokenizer ignores the text/html part, and only looks at
> the text/plain part.

Not anymore:  I just introduced new option

    [Tokenizer]ignore_redundant_html

and it defaults to false.  This may change results!  Try it both ways and
report what happens.  Setting it to True (in your bayescustomize.ini)
restores pre-checkin behavior.


From tim.one@comcast.net  Sat Sep 14 22:05:03 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 14 Sep 2002 17:05:03 -0400
Subject: [Spambayes] Large false negative ...
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCEBABEAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEBCBEAB.tim.one@comcast.net>

[Tim]
> Not anymore:  I just introduced new option
>
>     [Tokenizer]ignore_redundant_html
>
> and it defaults to false.  This may change results!  Try it both ways
> and report what happens.

I'll start.  "Before" is the former behavior, and "after" with
ignore_redundant_html false (the new default).  This is on a 10-fold cross
validation run, each run testing 2000 hams & 1375 spams against training on
18000 hams & 12375 spams:

false positive percentages
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.050  0.050  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.050  lost  +(was 0)
    0.000  0.000  tied
    0.100  0.100  tied

won   0 times
tied  9 times
lost  1 times

total unique fp went from 3 to 4 lost   +33.33%
mean fp % went from 0.015 to 0.02 lost   +33.33%

false negative percentages
    0.218  0.218  tied
    0.436  0.364  won    -16.51%
    0.073  0.000  won   -100.00%
    0.218  0.218  tied
    0.218  0.218  tied
    0.291  0.291  tied
    0.291  0.218  won    -25.09%
    0.218  0.145  won    -33.49%
    0.291  0.291  tied
    0.073  0.073  tied

won   4 times
tied  6 times
lost  0 times

total unique fn went from 32 to 28 won    -12.50%
mean fn % went from 0.232727272727 to 0.203636363636 won    -12.50%


The new false positive is due to a multipart/alternative where the text/html
portion has a very long <!-- ... --> comment, containing stuff like

    @page Section1
        {size:8.5in 11.0in;
        margin:1.0in 1.25in 1.0in 1.25in;
        mso-header-margin:.5in;
        mso-footer-margin:.5in;
        mso-paper-source:0;}
    div.Section1
        {page:Section1;}

The regexp trickery the tokenizer uses to strip HTML tags puts a bound on
how much text it's willing to believe a tag consumes, lest an innocent
non-HTML '<' thingie consume a huge portion of the body.  As a result, this
comment isn't getting stripped, and the msg picks up lots of 0.99 clues from
the comment contents:

prob('1.25in') = 0.99
prob('div.section1') = 0.99
prob('@page') = 0.99
prob('margin:1.0in') = 0.99
prob('<!--') = 0.99
prob('roman";') = 0.99
prob('margin:0in;') = 0.99
prob('roman";}') = 0.99
prob('section1') = 0.99
prob('p.msonormal,') = 0.99
prob('11.0in;') = 0.99
prob('1.25in;') = 0.99
prob('1.0in') = 0.99
prob('{margin:0in;') = 0.99
prob('{size:8.5in') = 0.99

If anyone wants to improve HTML tag-stripping, please do!  The current
scheme is efficient, can't blow the re stack, and works well "almost all" of
the time.  OTOH, it makes mistakes like the above, and leaves character
entities (like &nbsp;) intact (although I can't say whether that's good or
bad!).


From tim.one@comcast.net  Sat Sep 14 23:05:16 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 14 Sep 2002 18:05:16 -0400
Subject: [Spambayes] Large false negative ...
In-Reply-To: <15747.12814.571767.593491@12-248-11-90.client.attbi.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCMEBDBEAB.tim.one@comcast.net>

[Tim]
> ...
> For those with a larger corpus, I suggest it's easier to fiddle
> MsgStream.produce to pick smaller subsets at random; e.g., ...

[Skip Montanaro]
> Perhaps the keep values should be settable options?

timcv.py now has cmdline options to do this.

From skip@pobox.com  Sun Sep 15 00:51:35 2002
From: skip@pobox.com (Skip Montanaro)
Date: Sat, 14 Sep 2002 18:51:35 -0500
Subject: [Spambayes] Large false negative ...
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEAKBEAB.tim.one@comcast.net>
References: <15747.12814.571767.593491@12-248-11-90.client.attbi.com>
        <LNBBLJKPBEHFEDALKOLCGEAKBEAB.tim.one@comcast.net>
Message-ID: <15747.52103.424920.545869@12-248-11-90.client.attbi.com>


    >> Here's the set of best discriminators from my latest run:

    Tim> Very important: such a list gets printed 5 times during an -n5 run.
    Tim> Are you showing the first such list, the last such list, or ...? 

Sorry, the last.

    >> '8bit%:92' 101 0.99
    >> '8bit%:100' 181 0.99

    Tim> Do you have a lot of Asian spam?

Oodles and oodles of the crap.  In my training set, I have 189 alone which
mention "gb2312" in either the subject or as the charset.  Of those that
mention a charset, 638 mention something other than ascii or one of the
iso-8859 charsets.

    >> 'x-mailer:none' 96 0.481519
    >> 'header:Subject:1' 104 0.497099
    >> 'content-type:text/plain' 118 0.405034
    >> 'header:To:1' 126 0.495146
    >> 'header:Message-Id:1' 146 0.60187

    Tim> It's as if it's not finding a message *body* in these cases (else
    Tim> it would find *something* better than the mere presence of a "To:"
    Tim> line to pick on!).  It's also curious that the presence of a
    Tim> Message-Id line is a (weak) spam indicator in your data.

Don't know what to make of that.  There are some spams with message bodies
consisting of just one or two words.

    >> 'from:email addr:aol.com' 135 0.0148768

    Tim> You may have the world's only corpus where email from AOL is a Good
    Tim> Thing <wink>.

Yes, many of the CEDU and Musi-Cal concert site correspondents are AOLers.

    >> 'charset:gb2312' 111 0.99

    Tim> Fascinating!

    >> 'subject:GB2312' 97 0.99

    Tim> Weird.

    >> 'idaho' 100 0.01

    Tim> Disturbing <wink>.

    Tim> The lack of 'url:gif' and 'url:remove' as spam indicators is surprising to
    Tim> me.

    Tim> The top 4:

    >> 'url:cedu' 570 0.01
    >> 'email name:cedu' 600 0.01
    >> 'subject:cedu' 685 0.01
    >> 'cedu' 694 0.01

Yup, there are a lot of CEDU messages in my ham corpus.

    >> * There are very few 0.99's.  I would have thought some of the
    >> obvious spam tripwords would have made it into the set.

    Tim> Me too.  It's possible that you have a lot of spam of the form
    Tim> Anthony warned about:

    Tim>     multipart/alternative
    Tim>         text/html
    Tim>             The real spam is here.
    Tim>         text/plain
    Tim>             Something innocuous is here.

    Tim> In that case, the tokenizer ignores the text/html part, and only
    Tim> looks at the text/plain part. 

Maybe the relative size of the two parts could be used to decide if the
text/html section should be retained?  Or only emit words from the text/html
section if they don't occur in the text/plain section?  (Just thinking out
loud here.)

    Tim> You could do a gross test to see whether this is important in your
    Tim> data by changing the last line of tokenizer.textparts() from

    Tim>     return text - redundant_html

    Tim> to

    Tim>     return text + redundant_html

I will 

    >> I have a ton of unread CEDU mail I dumped into the training set,

    Tim> If you haven't read it, how do you know whether it's ham or spam?

It's in my CEDU mailbox, put there by procmail based upon seeing a 

    Sender: cedu-admin@manatee.mojam.com

header.  The CEDU list is a closed (only subscribers can post) list.  Also,
I'm the list moderator, so even though I haven't read every message, I have
at least scanned the headers.

    Tim> Can you set

    Tim>     show_false_positives = False
    Tim>     show_false_negatives = True
    Tim>     show_charlimit = 500000

    Tim> and stick full output from a run on the web somewhere?  

I'll give it a go.

Skip

From skip@pobox.com  Sun Sep 15 00:55:10 2002
From: skip@pobox.com (Skip Montanaro)
Date: Sat, 14 Sep 2002 18:55:10 -0500
Subject: [Spambayes] Large false negative ...
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEAMBEAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCGEAKBEAB.tim.one@comcast.net>
        <LNBBLJKPBEHFEDALKOLCGEAMBEAB.tim.one@comcast.net>
Message-ID: <15747.52318.398983.873152@12-248-11-90.client.attbi.com>


    Tim> Oops!  Make that

    Tim>     return text | redundant_html

Can two timcv jobs be run at the same time?

Skip


From tim.one@comcast.net  Sun Sep 15 00:58:40 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 14 Sep 2002 19:58:40 -0400
Subject: [Spambayes] Need testers!
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEAKBEAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEBGBEAB.tim.one@comcast.net>

[Tim]
> ...
> Since I got rid of MINCOUNT, a clue in a single ham message is as
> strong as a clue that appears in every ham message.  This makes me
> squirm, but testing said it was a win.  (BTW, I've tried, but not
> reported on, a dozen schemes to give less weight to high-prob clues
> that appear in few messages; every such attempt has been a loser.)

The one I just checked in may be a winner, though.  It needs testing on a
wide variety of corpora and corpora sizes.

1. Run a baseline and save a summary file (rates.py).

2. Make exactly one change, adding

[Classifier]
adjust_probs_by_evidence_mass: True

   to your bayescustomize.ini file.

3. Run the same test scenario again, and create another summary file.

4. Run cmp.py over the summary files and post the cmp.py output
   (all of it, please).  Or mail it to me, but I think there's value
   in public humiliation <wink> if it helps get better results.

Here's my before-and-after cmp.py:

Before: adjust_probs_by_evidence_mass: False
After:  adjust_probs_by_evidence_mass: True

"""
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams

false positive percentages
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.050  lost  +(was 0)
    0.000  0.000  tied
    0.050  0.050  tied
    0.000  0.050  lost  +(was 0)
    0.000  0.000  tied
    0.050  0.050  tied
    0.000  0.000  tied
    0.100  0.050  won    -50.00%

won   1 times
tied  7 times
lost  2 times

total unique fp went from 4 to 5 lost   +25.00%
mean fp % went from 0.02 to 0.025 lost   +25.00%

false negative percentages
    0.218  0.073  won    -66.51%
    0.364  0.218  won    -40.11%
    0.000  0.000  tied
    0.218  0.145  won    -33.49%
    0.218  0.218  tied
    0.291  0.218  won    -25.09%
    0.218  0.291  lost   +33.49%
    0.145  0.218  lost   +50.34%
    0.291  0.291  tied
    0.073  0.000  won   -100.00%

won   5 times
tied  3 times
lost  2 times

total unique fn went from 28 to 23 won    -17.86%
mean fn % went from 0.203636363636 to 0.167272727273 won    -17.86%
"""

If you can make time, also try two other variations, changing the line

                    dist *= sum / (sum + 1.0)

in classifier.GrahamBayes.update_probabilities().  Try it once replacing 1.0
with 2.0, and another time replacing 1.0 with 0.5.  (The closer this
constant is to 0, the more this will act as if the new code weren't there.)

If you want to have some fun too <wink>, read the comments and try a
different way to get what it's aiming at.  A little thought separated by
hours of testing is how real progress is made on this kind of thing, though.


From tim.one@comcast.net  Sun Sep 15 01:17:23 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 14 Sep 2002 20:17:23 -0400
Subject: [Spambayes] Large false negative ...
In-Reply-To: <15747.52318.398983.873152@12-248-11-90.client.attbi.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCMEBHBEAB.tim.one@comcast.net>

[Skip]
> Can two timcv jobs be run at the same time?

Oh yes!  Even three <wink>.  The database is constructed internally in a
Python dict, and nothing is changed on disk.  bayescustomize.ini is read up
early at startup; you can change it before starting a second run and not
affect a run already in progress (the option values timcv prints at the
start are the ones that run will use for its life).


From tim.one@comcast.net  Sun Sep 15 06:16:28 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 15 Sep 2002 01:16:28 -0400
Subject: [Spambayes] An unusual spam
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCEBABEAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCAEBPBEAB.tim.one@comcast.net>

I just noticed the scores on this false negative from my last run.  Check it
out:  it doesn't have *any* clues (among the most extreme 16) on the spam
side of the ledger!

*************************************************************************
Data/Spam/Set4/3434.txt
prob = 1.2373429363e-029
prob('hacker,') = 0.01
prob('hackers.') = 0.01
prob('newbie,') = 0.01
prob('tutorial,') = 0.01
prob('hack') = 0.01
prob('btw') = 0.01
prob('way),') = 0.01
prob('forums.') = 0.0121951
prob('hackers,') = 0.0121951
prob('url:forums') = 0.0121951
prob('way?') = 0.0156571
prob('point.') = 0.0237802
prob('distinct') = 0.0244564
prob('anyway,') = 0.0324544
prob('although,') = 0.040078
prob("i'm") = 0.0431125

Return-Path: <MassMail@ecforums.com>
Delivered-To: em-ca-bait@em.ca
Received: (qmail 3757 invoked from network); 10 May 2002 22:09:01 -0000
Received: from pcp01365376pcs.jamisn01.pa.comcast.net (HELO ecforums.com)
(68.80.210.138)
  by churchill.factcomp.com with SMTP; 10 May 2002 22:09:01 -0000
From: "BlackIntrepid" <MassMail@ecforums.com>
To: <bait@em.ca>
Subject: ECForums.com [Please Read Me]
Sender: "BlackIntrepid" <MassMail@ecforums.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="ISO-8859-1"
Date: Fri, 10 May 2002 18:08:54 -0400
Reply-To: "BlackIntrepid" <MassMail@ecforums.com>
Content-Transfer-Encoding: 8bit
Content-Length: 1287
Lines: 24

Hello, my Name is BlackIntrepid

I'm a hacker. Yes, a Hacker. I'll make this message short and to the point.
I just opened a new website, called ECForums.com it's at www.ecforums.com

We're a forum about, yes, hackers. Don't be afraid of the subject, their
are very distinct categories of hackers, White hat, or hackers that only
do good, and help people out, and Black hat, hackers that purposely try to
destroy servers and hurt the internet. Although, some people think that
Hackers Cannot do good, wouldn't you rather have someone from our White
hat community hack into the FBI's server, and then alert them about it, so
a black hat hacker couldn't get in the same way? This is what we do.
Anyway, we need a couple more people at our site to visit our forums. If
you want to learn to be a hacker, or just want to learn more about the
computer, visit out "N00bi3 forums" n00bi3 stands for Newbie, which is
someone who is new, or doesn't know much about the computer. Soon we hope
to have a complete tutorial, on what to learn and study, if your new to
the computer, or want to be a hacker. Please feel free to come visit, and
Register on our Forums!

-ECForums.com Staff

BTW (BY the way), to get straight to our forums, just type this in your
browser: Http://www.ecforums.com/forums/

*************************************************************************


From tim.one@comcast.net  Sun Sep 15 06:30:25 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 15 Sep 2002 01:30:25 -0400
Subject: [Spambayes] Large false negative ...
In-Reply-To: <15747.52103.424920.545869@12-248-11-90.client.attbi.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCMEBPBEAB.tim.one@comcast.net>

[Skip Montanaro]
> ...
> Maybe the relative size of the two parts could be used to decide if the
> text/html section should be retained?  Or only emit words from
> the text/html section if they don't occur in the text/plain section?
> (Just thinking out loud here.)

Thought doesn't help <0.9 wink>.  For any idea, try it and see what the data
tells you.  If you find something that might be a winner, share it so that
other people can test it too.


From tim@zope.com  Sun Sep 15 07:04:15 2002
From: tim@zope.com (Tim Peters)
Date: Sun, 15 Sep 2002 02:04:15 -0400
Subject: [Spambayes] Results of playing with CDB
In-Reply-To: <w53lm666bst.fsf@woozle.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCAECCBEAB.tim@zope.com>

[Neale Pickett]
> ...
> I'm not sure a server would work for multiple users.  I have 80 users;
> at 20MB per server, that's 1600MB of RAM, which would require a new hard
> drive just for the swap :).  I'm sure it wouldn't be quite that bad if
> it was just one server for everyone, but in that case I suspect the dict
> would quickly become so large that it would no longer be practical to
> have it all in RAM, and then I'm back to dbm files.
>
> Of course, I could be missing some obvious implementation detail.  The
> spam database could be shared, for instance, if it was sufficiently
> attack-resistant.  In any case, speeding up the current implementation
> seems like a sufficiently low-hanging fruit for the time being, and
> hopefully will be useful work for a larger project.

Hmm.  When you say "a server", you seem to have in mind a shared machine
sitting in another room.  When I say "a server", I just have in mind a
process that usually sits idle, waiting for a client to ask it to do
something.  There's no rule against a server and client process running on
the same box, neither against each user running their own server process for
their own email.


From guido@python.org  Sun Sep 15 07:16:56 2002
From: guido@python.org (Guido van Rossum)
Date: Sun, 15 Sep 2002 02:16:56 -0400
Subject: [Spambayes] Results of playing with CDB
In-Reply-To: Your message of "Sun, 15 Sep 2002 02:04:15 EDT."
             <LNBBLJKPBEHFEDALKOLCAECCBEAB.tim@zope.com> 
References: <LNBBLJKPBEHFEDALKOLCAECCBEAB.tim@zope.com> 
Message-ID: <200209150617.g8F6GuQ32732@pcp02138704pcs.reston01.va.comcast.net>

> [Neale Pickett]
> > ...
> > I'm not sure a server would work for multiple users.  I have 80 users;
> > at 20MB per server, that's 1600MB of RAM, which would require a new hard
> > drive just for the swap :).  I'm sure it wouldn't be quite that bad if
> > it was just one server for everyone, but in that case I suspect the dict
> > would quickly become so large that it would no longer be practical to
> > have it all in RAM, and then I'm back to dbm files.
> >
> > Of course, I could be missing some obvious implementation detail.  The
> > spam database could be shared, for instance, if it was sufficiently
> > attack-resistant.  In any case, speeding up the current implementation
> > seems like a sufficiently low-hanging fruit for the time being, and
> > hopefully will be useful work for a larger project.

[Tim]
> Hmm.  When you say "a server", you seem to have in mind a shared machine
> sitting in another room.  When I say "a server", I just have in mind a
> process that usually sits idle, waiting for a client to ask it to do
> something.  There's no rule against a server and client process running on
> the same box, neither against each user running their own server process for
> their own email.

That's not how I read Neale's post.  He's considering one server
*process* per user though, and thinks that 80 server processes with a
virtual memory address space (what he calls "RAM") of 20 MB would take
up too much swap space.  Is 20 MB a reasonable estimate for the
in-memory dict?

But I don't buy his assumption that if you share the server between 80
users the dict becomes too large.  Why would it?  Because their ham
collections would be disjoint?  Not completely, I expect.  But that is
indeed a question to ponder: can 80 users effectively share the
classifier database?

--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.one@comcast.net  Sun Sep 15 07:17:35 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 15 Sep 2002 02:17:35 -0400
Subject: [Spambayes] FW: [Spambayes-checkins]
 spambayesOptions.py,1.11,1.12timtest.py,1.22,1.23
In-Reply-To: <JHEOKEOOLIGLDHCMAHMOOEHODOAA.marklists@mceahern.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCAECDBEAB.tim.one@comcast.net>

[Tim]
>> Log Message:
>> Incompatible change:  show_best_discriminators has changed from a bool
>> option to an int option, now giving the number of best discriminators
>> to show.  Set to 0 if you don't want to see any.

[Mark McEahern]
> Laura Creighton would love this one.  <wink>

LOL!  Good thing I'm a list admin here, so I can keep her out.

From tim@zope.com  Sun Sep 15 08:05:29 2002
From: tim@zope.com (Tim Peters)
Date: Sun, 15 Sep 2002 03:05:29 -0400
Subject: [Spambayes] Results of playing with CDB
In-Reply-To: <200209150617.g8F6GuQ32732@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCGECFBEAB.tim@zope.com>

[Guido]
> That's not how I read Neale's post.  He's considering one server
> *process* per user though, and thinks that 80 server processes with a
> virtual memory address space (what he calls "RAM") of 20 MB would take
> up too much swap space.

Sure.  My post suggested a different approach:  one client box, one client
server process running on the client box.  I understood that Neale had in
mind one server machine and N server processes running on that single
machine.  That's not what I had in mind.

> Is 20 MB a reasonable estimate for the in-memory dict?

It depends on how much training data you feed it, how often it's cleaned,
how aggressively it's cleaned, and what the user considers to be acceptable
error rates (of both kinds).  The only solid data we have on any of that is
the qualitative observation that both error rates decrease as the training
data increases.

BTW, I've noticed that my process data size on test runs is about 3-4x
larger than the classifier pickle on disk.

Note that saving an 8-byte timestamp and an 8-byte probability (both
doubles) per word is merely convenient for now.

I expect that, in the end, 20MB RAM will prove enough for a 0.2% fp rate and
a 1% fn rate, and assuming English.

> But I don't buy his assumption that if you share the server between 80
> users the dict becomes too large.  Why would it?  Because their ham
> collections would be disjoint?  Not completely, I expect.

The only figure I have on that was posted yesterday:  when I increased the
training data by a factor of 4, the pickle size grew by a factor of 2.4 (to
about 600 bytes per message, and about 27,000 messages trained on).

> But that is indeed a question to ponder: can 80 users effectively
> share the classifier database?

I expect that depends most on their tolerance for errors.  Just about
everyone subscribes to some bulk email, and it seems likely that a
classifier trained to accept the union of 80 peoples' private delusions
<wink> would suffer a lot.  At the start, there would be a lot of false
positives as the system struggled to learn that stuff looking exactly like
spam is actually desired by someone.  OTOH, only testing can answer this,
and we've got a lot more question-askers than testers here.


From tim.one@comcast.net  Sun Sep 15 08:45:47 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 15 Sep 2002 03:45:47 -0400
Subject: [Spambayes] Need testers!
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEBGBEAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCKECGBEAB.tim.one@comcast.net>

[Tim]
> 1. Run a baseline and save a summary file (rates.py).
>
> 2. Make exactly one change, adding
>
> [Classifier]
> adjust_probs_by_evidence_mass: True
>
>    to your bayescustomize.ini file.
>
> 3. Run the same test scenario again, and create another summary file.
>
> 4. Run cmp.py over the summary files and post the cmp.py output
>    (all of it, please).  Or mail it to me, but I think there's value
>    in public humiliation <wink> if it helps get better results.

This worked well for my very large training sets, but was a clear loss when
I ran on smaller subsets.  I've checked in another version that repaired
this for me.  To try it, do cvs up and change step #2 to this:

[Classifier]
adjust_probs_by_evidence_mass: True
min_spamprob: 0.001
max_spamprob: 0.999
hambias: 1.5


On my giant

-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams

10-fold c-v run, this is the difference:

false positive percentages
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.050  0.050  tied
    0.000  0.050  lost  +(was 0)
    0.000  0.000  tied
    0.050  0.050  tied
    0.000  0.000  tied
    0.100  0.050  won    -50.00%

won   1 times
tied  8 times
lost  1 times

total unique fp went from 4 to 4 tied
mean fp % went from 0.02 to 0.02 tied

false negative percentages
    0.218  0.145  won    -33.49%
    0.364  0.364  tied
    0.000  0.073  lost  +(was 0)
    0.218  0.218  tied
    0.218  0.218  tied
    0.291  0.145  won    -50.17%
    0.218  0.073  won    -66.51%
    0.145  0.145  tied
    0.291  0.218  won    -25.09%
    0.073  0.000  won   -100.00%

won   5 times
tied  4 times
lost  1 times

total unique fn went from 28 to 22 won    -21.43%
mean fn % went from 0.203636363636 to 0.16 won    -21.43%


On much smaller random-subset 10-fold c-v runs with

-> <stat> tested 300 hams & 300 spams against 2700 hams & 2700 spams

the effect is usually a significant decrease in the f-n rate, and a similar
(to the huge test) random bump up or down in the f-p stats.  Dropping the
value of hambias probably accounts for the f-n goodness; the rest is largely
to prevent f-p badness at the same time; expanding the min/max-prob range,
coupled with taking into account the number of msgs that go into each
probablity estimate, all but eliminates "massive cancellation" of
MIN_SPAMPROB and MAX_SPAMPROB clues.


From gward@python.net  Sun Sep 15 14:28:27 2002
From: gward@python.net (Greg Ward)
Date: Sun, 15 Sep 2002 09:28:27 -0400
Subject: [Spambayes] An unusual spam
In-Reply-To: <LNBBLJKPBEHFEDALKOLCAEBPBEAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCCEBABEAB.tim.one@comcast.net>
	<LNBBLJKPBEHFEDALKOLCAEBPBEAB.tim.one@comcast.net>
Message-ID: <20020915132827.GA3951@cthulhu.gerg.ca>

On 15 September 2002, Tim Peters said:
> I just noticed the scores on this false negative from my last run.  Check it
> out:  it doesn't have *any* clues (among the most extreme 16) on the spam
> side of the ledger!

Maybe we should run a grammar checker to detect possible spam?  ;-)

> Hello, my Name is BlackIntrepid
            ^^^^ not a proper noun -- English != German

> I'm a hacker. Yes, a Hacker. I'll make this message short and to the point.
                       ^^^^^^ ditto

> I just opened a new website, called ECForums.com it's at www.ecforums.com

> We're a forum about, yes, hackers. Don't be afraid of the subject, their
                                    ^ should be two spaces

> are very distinct categories of hackers, White hat, or hackers that only
> do good, and help people out, and Black hat, hackers that purposely try to
> destroy servers and hurt the internet.
                               ^^^^^^^^ proper noun - should be capitalized

>                                         Although, some people think that
                                                  ^ incorrect comma

Can I stop now?

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
"What do you mean -- a European or an African swallow?"

From nas@python.ca  Sun Sep 15 16:22:12 2002
From: nas@python.ca (Neil Schemenauer)
Date: Sun, 15 Sep 2002 08:22:12 -0700
Subject: [Spambayes] Need testers!
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKECGBEAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCKEBGBEAB.tim.one@comcast.net>
	<LNBBLJKPBEHFEDALKOLCKECGBEAB.tim.one@comcast.net>
Message-ID: <20020915152212.GA15538@glacier.arctrix.com>

The results for 6 sets with 300 in each:

false positive percentages
    0.667  1.333  lost   +99.85%
    0.000  0.000  tied
    1.000  1.000  tied
    0.333  0.333  tied
    0.000  0.667  lost  +(was 0)
    0.000  0.000  tied

won   0 times
tied  4 times
lost  2 times

total unique fp went from 6 to 10 lost   +66.67%
mean fp % went from 0.333333333333 to 0.555555555555 lost   +66.67%

false negative percentages
    0.333  0.333  tied
    1.333  1.000  won    -24.98%
    1.667  1.333  won    -20.04%
    0.333  0.333  tied
    1.333  1.333  tied
    1.667  0.667  won    -59.99%

won   3 times
tied  3 times
lost  0 times

total unique fn went from 20 to 15 won    -25.00%
mean fn % went from 1.11111111111 to 0.833333333332 won    -25.00%


From neale@woozle.org  Sun Sep 15 20:06:33 2002
From: neale@woozle.org (Neale Pickett)
Date: 15 Sep 2002 12:06:33 -0700
Subject: [Spambayes] Results of playing with CDB
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGECFBEAB.tim@zope.com>
References: <LNBBLJKPBEHFEDALKOLCGECFBEAB.tim@zope.com>
Message-ID: <w53hegr3wee.fsf@woozle.org>

So then, "Tim Peters" <tim@zope.com> is all like:

> Sure.  My post suggested a different approach: one client box, one
> client server process running on the client box.  I understood that
> Neale had in mind one server machine and N server processes running on
> that single machine.  That's not what I had in mind.

Unfortunately, that's the only situation I can afford :) I have 80
users, one box, and if what SpamAssassin says is correct, 1/3 of all my
incoming email is spam.  I suspect it's closer to 1/2.  In any case, I
have a big problem and limited resources.  The "quick-scoring" Paul
Graham approach looks really appealing, but I have to make sure it's
fast and runs quickly.  About the only thing I really can spare is
storage space.

> I expect that, in the end, 20MB RAM will prove enough for a 0.2% fp
> rate and a 1% fn rate, and assuming English.

So that basically eliminates one server process per user for me, and
probably for a lot of other people, too.  204 users would fill up all
addressable 4GiB on a 32-bit architecture.

> > But that is indeed a question to ponder: can 80 users effectively
> > share the classifier database?
> 
> I expect that depends most on their tolerance for errors.  Just about
> everyone subscribes to some bulk email, and it seems likely that a
> classifier trained to accept the union of 80 peoples' private
> delusions <wink> would suffer a lot.  At the start, there would be a
> lot of false positives as the system struggled to learn that stuff
> looking exactly like spam is actually desired by someone.  OTOH, only
> testing can answer this, and we've got a lot more question-askers than
> testers here.

I volunteer my hapless 80 as testers!  (Actually, I doubt more than four
or five would sign up for such a thing, but maybe that's enough.)  Do
you think they should all contribute to one big vat-o-meat database, or
should individual words be tagged per-user?


From guido@python.org  Sun Sep 15 20:35:12 2002
From: guido@python.org (Guido van Rossum)
Date: Sun, 15 Sep 2002 15:35:12 -0400
Subject: [Spambayes] Results of playing with CDB
In-Reply-To: Your message of "Sun, 15 Sep 2002 12:06:33 PDT."
             <w53hegr3wee.fsf@woozle.org> 
References: <LNBBLJKPBEHFEDALKOLCGECFBEAB.tim@zope.com>  
            <w53hegr3wee.fsf@woozle.org> 
Message-ID: <200209151935.g8FJZCX09918@pcp02138704pcs.reston01.va.comcast.net>

> So that basically eliminates one server process per user for me, and
> probably for a lot of other people, too.  204 users would fill up all
> addressable 4GiB on a 32-bit architecture.

Neale, I'm not sure this calculation makes sense.  Each process has
its own address space, which is limited to 4 GB.  I don't think it's
written anywhere that the combined address space must also fit in 4
GB.  It must fit in swap space, sure, but you can configure that as
large as you want to.  Or not?

--Guido van Rossum (home page: http://www.python.org/~guido/)

From crunch@shopip.com  Sat Sep 14 10:24:33 2002
From: crunch@shopip.com (John Draper)
Date: Sat, 14 Sep 2002 02:24:33 -0700
Subject: [Spambayes] Hi - just got on the list....
Message-ID: <v03110751b9a8adb94c7f@[192.168.0.2]>

Allow me to introduce myself...

  Being such a "public" person,  but reluctant to change my well known EMAIL=
 	address,  it's no wonder I'm  getting 2500 spam messages per week.   So=
 about a year ago,   I solicited the help of spamcop in an attempt to reduce=
 it to a dull roar,  and now realize I really needed something to fight off=
 the constant ads reminding me how small a certain piece of anatomy is....

  I just found out about this list,  and have been working on a 100% way of=
 identifying spam.   It's purely theoritical at this point,  and my=
 programming skills are lacking,  but want to share this with others.

  My partner (who is very good programmer) wrote a MTA in Python (in just a=
 half hour),  then I modified it to handle multiple threads,  and wish to=
 use this as a base of a development of a "proof of concept" to see how well=
 this might work.

  Right now,  I'm just trying to determine what's out there,  so want to=
 participate in this list to see what the current anti-spam technology level=
 is at,  and what you guys are doing.

  Being a python programmer,  I was surprised to learn so many people=
 working on this project.

  As a result of my efforts to eliminate spam,  I have 33 megs of spam I've=
 collected (About 6000 spam messages) over the past year.   I didn't save=
 ALL of it,   as I've eliminated redundancy,  and can make them available if=
 anyone wants it.

  I have them in a Mac Eudora Mailbox,  with I think is in "mbox" format, =
 but not sure.   If anyonw can use this,  let me know. =20

   Before joining the list,  I read all the archives to get up to date on=
 the discussions,  and learned people were collecting spam messages.

   Anyway,  I'll be reading these messages with interest,  and might come in=
 ocassionally to ask some questions...   but does anyone have any FAQ=
 written up on some of these tools you talk about?    I want to read up on=
 whats being done in the group first,  before getting on and asking lame que=
stions.

Thanx Guys...

John



From crunch@shopip.com  Sat Sep 14 20:24:26 2002
From: crunch@shopip.com (John Draper)
Date: Sat, 14 Sep 2002 12:24:26 -0700
Subject: [Spambayes] Large false negative ...
In-Reply-To: <15747.12814.571767.593491@12-248-11-90.client.attbi.com>
References: <LNBBLJKPBEHFEDALKOLCGEPPBDAB.tim.one@comcast.net>
 <15746.46588.788996.794513@12-248-11-90.client.attbi.com>       
 <LNBBLJKPBEHFEDALKOLCGEPPBDAB.tim.one@comcast.net>
Message-ID: <v03110755b9a93c4a65da@[192.168.0.2]>

>I would have thought that I'd be within spitting distance of the sorts of
>numbers you reported with the 5x5 scheme.  I'm training on 1600 hams and
>1300 spams.  You were training on 4000 hams and 2750 spams I believe.

Sorry to seem like a lamer,  but what's the difference between a ham and a spam?
Being unable to find this info on-line,   leads me to ask here.

John



From neale@woozle.org  Sun Sep 15 21:05:32 2002
From: neale@woozle.org (Neale Pickett)
Date: 15 Sep 2002 13:05:32 -0700
Subject: [Spambayes] Results of playing with CDB
In-Reply-To: <200209151935.g8FJZCX09918@pcp02138704pcs.reston01.va.comcast.net>
References: <LNBBLJKPBEHFEDALKOLCGECFBEAB.tim@zope.com>
	<w53hegr3wee.fsf@woozle.org>
	<200209151935.g8FJZCX09918@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <w53d6rf3to3.fsf@woozle.org>

So then, Guido van Rossum <guido@python.org> is all like:

> > So that basically eliminates one server process per user for me, and
> > probably for a lot of other people, too.  204 users would fill up all
> > addressable 4GiB on a 32-bit architecture.
> 
> Neale, I'm not sure this calculation makes sense.  Each process has
> its own address space, which is limited to 4 GB.  I don't think it's
> written anywhere that the combined address space must also fit in 4
> GB.  It must fit in swap space, sure, but you can configure that as
> large as you want to.  Or not?

Oops, I was confused.  You're correct, the 4GiB limit ony applies
per-process.  So my argument isn't as strong as I thought :)

But I think the point is still valid--20MB per mail user is too much.
Anyway, I'm probably preaching to the choir at this point.  hammiesrv
will be coming right up.  Any objections to using XML-RPC as the
"protocol"?

From tim.one@comcast.net  Sun Sep 15 23:20:44 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 15 Sep 2002 18:20:44 -0400
Subject: [Spambayes] An unusual spam
In-Reply-To: <20020915132827.GA3951@cthulhu.gerg.ca>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEDLBEAB.tim.one@comcast.net>

[Greg Ward]
> Maybe we should run a grammar checker to detect possible spam?  ;-)

Word bigrams picked up (just) a few spams that nothing else was able to
detect, due to fractured English (the very brief "Rudolph The Rednose
Hooters Here" spam mentioned earlier was its greatest triumph).  OTOH,
deduct for bad grammar, spelling, or layout, and lots of mailing list
traffic would look suspicious.


From tim.one@comcast.net  Sun Sep 15 23:30:54 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 15 Sep 2002 18:30:54 -0400
Subject: [Spambayes] Need testers!
In-Reply-To: <20020915152212.GA15538@glacier.arctrix.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEDMBEAB.tim.one@comcast.net>

[Neil Schemenauer]
> The results for 6 sets with 300 in each:

Thanks!  Was this the earlier-suggested

    [Classifier]
    adjust_probs_by_evidence_mass: True

experiment, or the later-suggested

    [Classifier]
    adjust_probs_by_evidence_mass: True
    min_spamprob: 0.001
    max_spamprob: 0.999
    hambias: 1.5

experiment?

> false positive percentages
>     0.667  1.333  lost   +99.85%
>     0.000  0.000  tied
>     1.000  1.000  tied
>     0.333  0.333  tied
>     0.000  0.667  lost  +(was 0)
>     0.000  0.000  tied
>
> won   0 times
> tied  4 times
> lost  2 times
>
> total unique fp went from 6 to 10 lost   +66.67%
> mean fp % went from 0.333333333333 to 0.555555555555 lost   +66.67%

Do you care?  That is, are the new ones false positives in a disturbing
sense?  There are about 10 msgs in my 20,000 ham that I don't care about --
if they flip from f-p to f-n or back again, I don't give that any weight.
Of course this includes the quote of the Nigerian scam msg, but also
includes two-word "unsubscribe me" messages from first-time posters wrapped
in 5KB of HTML decorations.  I keep these things in the ham set because
they're simply not spam, but nobody would care if they got squashed.

> false negative percentages
>     0.333  0.333  tied
>     1.333  1.000  won    -24.98%
>     1.667  1.333  won    -20.04%
>     0.333  0.333  tied
>     1.333  1.333  tied
>     1.667  0.667  won    -59.99%
>
> won   3 times
> tied  3 times
> lost  0 times
>
> total unique fn went from 20 to 15 won    -25.00%
> mean fn % went from 1.11111111111 to 0.833333333332 won    -25.00%

Cool!


From tim.one@comcast.net  Sun Sep 15 23:51:43 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 15 Sep 2002 18:51:43 -0400
Subject: [Spambayes] Results of playing with CDB
In-Reply-To: <w53hegr3wee.fsf@woozle.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCMEDOBEAB.tim.one@comcast.net>

[Neale Pickett]
> Unfortunately, that's the only situation I can afford :) I have 80
> users, one box,

I don't understand.  You have 80 users attached to dumb terminals with no
CPU of their own ... or what?

> and if what SpamAssassin says is correct, 1/3 of all my incoming email
> is spam.  I suspect it's closer to 1/2.  In any case, I have a big
> problem and limited resources.  The "quick-scoring" Paul Graham approach
> looks really appealing, but I have to make sure it's fast and runs
> quickly.  About the only thing I really can spare is storage space.

>> I expect that, in the end, 20MB RAM will prove enough for a 0.2% fp
>> rate and a 1% fn rate, and assuming English.

> So that basically eliminates one server process per user for me, and
> probably for a lot of other people, too.

These are tradeoffs.  You can limit the size to anything you like, provided
you're willing to take the hit in error rates.  Guido mentioned 20MB
specifically, so I took a guess at what error rates we could get *given*
that size.  This was possible because I've run many tests that happened to
take about that much RAM, so have a feel for what can be done in that size.

> ...
> I volunteer my hapless 80 as testers!  (Actually, I doubt more than four
> or five would sign up for such a thing, but maybe that's enough.)
>
> Do you think they should all contribute to one big vat-o-meat
> database, or should individual words be tagged per-user?

Nobody has reported experiments on this, and I don't know anything about
your users.  For example, this is work email and non-work email is strictly
prohibited, or they're 80 random people around the world sitting at home, or
whatever.  I *expect* to get good results from pooling python.org mailing
lists because of high topic and poster commonality.  This has yet to be
demonstrated, though.  I *don't* expect it would also work well to feed in
personal email that happens to be handled by python.org; I put a
c.l.py-trained classifier up on SourceForge twice so that people could
experiment with that against their personal email, but AFAIK nobody tried it
(and so I haven't bothered to update it either).


From tim.one@comcast.net  Mon Sep 16 00:06:03 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 15 Sep 2002 19:06:03 -0400
Subject: [Spambayes] Large false negative ...
In-Reply-To: <v03110755b9a93c4a65da@[192.168.0.2]>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEDPBEAB.tim.one@comcast.net>

[John Draper]
> Sorry to seem like a lamer,  but what's the difference between a
> ham and a spam?

ham == not spam; spam == not ham; any given email is exactly one of ham or
spam.

> Being unable to find this info on-line,   leads me to ask here.

Download the project files from

    http://sf.net/cvs/?group_id=61702

browse them online via the "Browse CVS Repository" link on that page.


From nas@python.ca  Mon Sep 16 00:13:03 2002
From: nas@python.ca (Neil Schemenauer)
Date: Sun, 15 Sep 2002 16:13:03 -0700
Subject: [Spambayes] Getting rid of max_spamprob and min_spamprob
Message-ID: <20020915231302.GA16483@glacier.arctrix.com>

I don't like the max_spamprob and min_spamprob limits.  I've written a
version of spamprob() that uses long integers, does not clamp the
probabilities and uses all evidence.

    def spamprob(self, wordstream, evidence=False):
        wordinfoget = self.wordinfo.get
        numerator = denominator = 1L
        nham = self.nham
        nspam = self.nspam
        for word in Set(wordstream):
            record = wordinfoget(word)
            if record is None:
                continue
            hamcount = record.hamcount
            spamcount = record.spamcount
            if record.hamcount == 0:
                numerator *= nspam
                denominator *= (nham + 1) * spamcount
            elif record.spamcount == 0:
                numerator *= (nspam + 1) * hamcount
                denominator *= nham
            else:
                numerator *= nspam * hamcount
                denominator *= nham * spamcount
        real, frac = divmod(numerator, denominator)
        huge = 1L<<30
        if real > 0:
            if real > huge:
                prob = 0.0
            else:
                prob = 1.0 / (real + 1.0)
        else:
            if frac > huge:
                prob = 1.0
            else:
                prob = frac / (1.0 + frac)
        if evidence:
            return (prob, [])
        else:
            return prob

The results are interesting, IMHO.  First the rate summary:

    total unique false pos 113
    total unique false neg 0
    average fp % 6.27777777778
    average fn % 0.0

The fp rate sucks but the fn rate is great.  Here is the histograms for
all runs:

Ham distribution for all runs:
* = 28 items
  0.00 1668 ************************************************************
  2.50    7 *
  5.00    0 
  7.50    3 *
 10.00    0 
 12.50    0 
 15.00    0 
 17.50    0 
 20.00    1 *
 22.50    0 
 25.00    3 *
 27.50    0 
 30.00    0 
 32.50    2 *
 35.00    0 
 37.50    0 
 40.00    0 
 42.50    0 
 45.00    0 
 47.50    0 
 50.00    3 *
 52.50    0 
 55.00    0 
 57.50    0 
 60.00    0 
 62.50    0 
 65.00    0 
 67.50    0 
 70.00    0 
 72.50    0 
 75.00    0 
 77.50    0 
 80.00    0 
 82.50    0 
 85.00    0 
 87.50    0 
 90.00    0 
 92.50    0 
 95.00    0 
 97.50  113 *****

Spam distribution for all runs:
* = 30 items
  0.00    0 
  2.50    0 
  5.00    0 
  7.50    0 
 10.00    0 
 12.50    0 
 15.00    0 
 17.50    0 
 20.00    0 
 22.50    0 
 25.00    0 
 27.50    0 
 30.00    0 
 32.50    0 
 35.00    0 
 37.50    0 
 40.00    0 
 42.50    0 
 45.00    0 
 47.50    0 
 50.00    0 
 52.50    0 
 55.00    0 
 57.50    0 
 60.00    0 
 62.50    0 
 65.00    0 
 67.50    0 
 70.00    0 
 72.50    0 
 75.00    0 
 77.50    0 
 80.00    0 
 82.50    0 
 85.00    0 
 87.50    0 
 90.00    0 
 92.50    0 
 95.00    0 
 97.50 1800 ************************************************************

Perhaps there is some way we can swap the two rates by introducing some
bias.

  Neil

From guido@python.org  Mon Sep 16 01:06:01 2002
From: guido@python.org (Guido van Rossum)
Date: Sun, 15 Sep 2002 20:06:01 -0400
Subject: [Spambayes] Results of playing with CDB
In-Reply-To: Your message of "Sun, 15 Sep 2002 13:05:32 PDT."
             <w53d6rf3to3.fsf@woozle.org> 
References: <LNBBLJKPBEHFEDALKOLCGECFBEAB.tim@zope.com>
	<w53hegr3wee.fsf@woozle.org>
	<200209151935.g8FJZCX09918@pcp02138704pcs.reston01.va.comcast.net>  
	<w53d6rf3to3.fsf@woozle.org> 
Message-ID: <200209160006.g8G061N10408@pcp02138704pcs.reston01.va.comcast.net>

> But I think the point is still valid--20MB per mail user is too much.
> Anyway, I'm probably preaching to the choir at this point.  hammiesrv
> will be coming right up.  Any objections to using XML-RPC as the
> "protocol"?

Sounds about right; that way others can easily connect to the same
server.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.one@comcast.net  Mon Sep 16 01:06:56 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 15 Sep 2002 20:06:56 -0400
Subject: [Spambayes] Getting rid of max_spamprob and min_spamprob
In-Reply-To: <20020915231302.GA16483@glacier.arctrix.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCAEEDBEAB.tim.one@comcast.net>

[Neil Schemenauer]
> I don't like the max_spamprob and min_spamprob limits.

They seem mostly to prevent .spamprobs of 0.0 or 1.0 from getting created
when a word appears in only one corpus.  Any msg with a spamprob 0.0 word
(but not also a spamprob 1.0 word) will get rated as 0; any with a spamprob
1.0 word (but not also a spamprob 0.0 word) will get rated as 1; and, if a
msg has both kinds, ZeroDivisionError will occur.  All of that is what
*should* happen with probabilities of 0 and 1, but no finite amount of
training data justifies certainty.

> I've written a version of spamprob() that uses long integers,

Very long, in fact <wink>.

> does not clamp the probabilities and uses all evidence.

Not quite:  the number of times a word appears in a single msg feeds into
spamcount and hamcount, but doesn't in spamprob() (due to the Set()).  Note
that this also means hamcount/nham can be larger than 1 (similarly for
spam); I'm not sure whether you think you've taken that into account.

It would be very helpful if you could try explaining in English what you
think you're computing here, as parts of the code don't make sense to me.
For example,

>         real, frac = divmod(numerator, denominator)
>         huge = 1L<<30
>         if real > 0:
>             if real > huge:
>                 prob = 0.0
>             else:
>                 prob = 1.0 / (real + 1.0)
>         else:
>             if frac > huge:
>                 prob = 1.0
>             else:
>                 prob = frac / (1.0 + frac)
>         if evidence:

Suppose numerator = denominator-1:  they're as close as possible without
being equal.  Then real=0 and frac=numerator.  If numerator > 2**30, this
comes out with prob 1.0.  I'm unclear on what you intended to compute, but
it doesn't seem right that nearly equal numerator and denominator could lead
to certainty that a msg is spam.  Perhaps this is part of why your f-n rate
is 0.

> The results are interesting, IMHO.  First the rate summary:
> ...
> Perhaps there is some way we can swap the two rates by introducing some
> bias.

Here we go again <wink>.  It also suggests a way to get "a middle ground"
for Greg:  use two scoring schemes, one that has no false negatives and one
that has no false positives.  If a given msg gets conflicting predictions
from these two extremes, Greg reviews it by eyeball.


From nas@python.ca  Mon Sep 16 02:28:11 2002
From: nas@python.ca (Neil Schemenauer)
Date: Sun, 15 Sep 2002 18:28:11 -0700
Subject: [Spambayes] Getting rid of max_spamprob and min_spamprob
In-Reply-To: <LNBBLJKPBEHFEDALKOLCAEEDBEAB.tim.one@comcast.net>
References: <20020915231302.GA16483@glacier.arctrix.com>
	<LNBBLJKPBEHFEDALKOLCAEEDBEAB.tim.one@comcast.net>
Message-ID: <20020916012811.GA16631@glacier.arctrix.com>

Tim Peters wrote:
> [Neil Schemenauer]
> > I've written a version of spamprob() that uses long integers,
> 
> Very long, in fact <wink>.

It's seems pretty quick.  Score one for Python's long integers.

> Note that this also means hamcount/nham can be larger than 1
> (similarly for spam); I'm not sure whether you think you've taken that
> into account.

Nope.  I'll have to think about that.

> It would be very helpful if you could try explaining in English what you
> think you're computing here, as parts of the code don't make sense to me.

I'm probably doing things in silly ways or even incorrectly.

> For example,
> 
> >         real, frac = divmod(numerator, denominator)
> >         huge = 1L<<30
> >         if real > 0:
> >             if real > huge:
> >                 prob = 0.0
> >             else:
> >                 prob = 1.0 / (real + 1.0)
> >         else:
> >             if frac > huge:
> >                 prob = 1.0
> >             else:
> >                 prob = frac / (1.0 + frac)
> >         if evidence:
> 
> Suppose numerator = denominator-1:  they're as close as possible without
> being equal.  Then real=0 and frac=numerator.  If numerator > 2**30, this
> comes out with prob 1.0.

The basic equation I'm computing is:

    P = 1 / (1 + (S^n * h_1 * h_2 ... h_n)/(H^n * s_1 * s_2 ... s_n))

    where:

        P is the probability of the message being spam

        S is the number of spam messages

        H is the number of ham messages

        s_n is the number of spam messages containing word n
    
        h_n is the number of ham messages containing word n

In the code, numerator = (S^n * h_1 * ...) and denominator = (H^n * s_1
...).  The last bit of code is (now <wink>) obviously incorrect.  What I
was tring to do was compute numerator/denominator as a float without
causing an overflow error.

> Here we go again <wink>.  It also suggests a way to get "a middle ground"
> for Greg:  use two scoring schemes, one that has no false negatives and one
> that has no false positives.

Yup, after introducing a bias:

    total unique false pos 1
    total unique false neg 88
    average fp % 0.0555555555555
    average fn % 4.88888888889

4.8% is bad but I should be able to improve that.

  Neil

From tim.one@comcast.net  Mon Sep 16 05:03:02 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 16 Sep 2002 00:03:02 -0400
Subject: [Spambayes] Getting rid of max_spamprob and min_spamprob
In-Reply-To: <20020916012811.GA16631@glacier.arctrix.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEEKBEAB.tim.one@comcast.net>

[Neil Schemenauer]
> The basic equation I'm computing is:
>
>     P = 1 / (1 + (S^n * h_1 * h_2 ... h_n)/(H^n * s_1 * s_2 ... s_n))
> ...
>
> What I was tring to do was compute numerator/denominator as a float
> without causing an overflow error.

Quick idea:  this kind of thing is why a classic Bayesian classifier works
in log space.  Forgetting the 1/ and 1+ things, we're left with nothing but
multiplies and divides.  In log space, * and / become + and -, and there's
no worry about overflow.  Refactor as

    product(for i = 1 to n, h_i/s_i) * (S/H)**n

Then the logarithm of that (pick any base you like) is

    sum(for i = 1 to n, log(h_i/s_i)) + n * log(S/H)

log(h_i/s_i) can be precomputed for each i by update_probabilities() and
stored in the WordInfo record for word i.  Likewise log(S/H) can be computed
then too and attached to an attribute of the classifier.

Then your scoring loop would consist of one floating multiply and n float
additions.  Coming back from log space to real space is left as an exercise
for the reader.  To avoid massive cancellation at the end, it's probably
numerically better to do

    sum(for i = 1 to n, log(h_i/s_i) + log(S/H))

where log(S/H) is a loop invariant.  That doubles the # of adds; BFD.

Now whether this computation makes *sense* is something I don't know.  I
posted a while ago about an experiment wherein I just counted the number of
indicators with prob >.5 and <.5, ignoring their values beyond that, and
showed that the resulting statistic appeared to follow a normal distribution
on hams and spams, with means separated by about 60 points.  I didn't report
on followup experiments that found simple ways to make those distributions
tighter and farther apart.  In fact, there appear to be any number of ways
to do that.  I didn't find a way that worked better than the way that's
checked in, although I found ways at the time that appeared just as good.

It's certainly open to experiment!  There's no meaningful sense in which
"probabilities" are being computed now anyway (although the scheme starts
with a reasonable probabilistic argument, by the time the bogus conditional
independence assumption is made, and most of the info is thrown away by the
scoring loop anyway, the numbers that come out in the end are just numbers).

> It's seems pretty quick.  Score one for Python's long integers.

I expect it's more that I/O and tokenization are expensive; from all the
clues I've seen, the scoring step has consumed an insignificant part of the
total time.  Not Python's long ints aren't pretty quick <wink>.


From neale@woozle.org  Mon Sep 16 07:31:52 2002
From: neale@woozle.org (Neale Pickett)
Date: 15 Sep 2002 23:31:52 -0700
Subject: [Spambayes] Results of playing with CDB
In-Reply-To: <LNBBLJKPBEHFEDALKOLCMEDOBEAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCMEDOBEAB.tim.one@comcast.net>
Message-ID: <w538z224f8n.fsf@woozle.org>

So then, Tim Peters <tim.one@comcast.net> is all like:

> [Neale Pickett]
> > Unfortunately, that's the only situation I can afford :) I have 80
> > users, one box,
> 
> I don't understand.  You have 80 users attached to dumb terminals with
> no CPU of their own ... or what?

Sort of.  About half of them use pine on my server, and the other half
are just figuring out how to browse the web, and asking them to install
stuff is too tall an order.  On the plus side, their inboxes are likely
to be very jargon-free.

My end goal is a centralized classifier for an entire organization on an
embedded device.  At $FIRM, our current-generation devices have between
4MB and 16MB of flash and no hard drive.  The next ones we make will
have more, but I don't know what that'll be yet.  This is why I'm so
concerned about storage space, though :)

> > Do you think they should all contribute to one big vat-o-meat
> > database, or should individual words be tagged per-user?

> I *expect* to get good results from pooling python.org mailing lists
> because of high topic and poster commonality.  This has yet to be
> demonstrated, though.  I *don't* expect it would also work well to
> feed in personal email that happens to be handled by python.org;
[snip]

Plagued by conscience, I've just run my 1000 test hams against your
SpamHam1.pik classifier.  It came back saying that 51 were spam, and 949
were ham.  Not bad, eh?  I've put the output of a "hammie.py -u" run at

  http://woozle.org/~neale/tmp/results.txt

All default values in my .ini file.  That is to say, I have no .ini file
:)  I'll go over unread messages and see if you need any other tests run.
Sorry I've been slackin'.

Neale

From neale@woozle.org  Mon Sep 16 07:51:13 2002
From: neale@woozle.org (Neale Pickett)
Date: 15 Sep 2002 23:51:13 -0700
Subject: [Spambayes] Need testers!
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKECGBEAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCKECGBEAB.tim.one@comcast.net>
Message-ID: <w534rcq4ece.fsf@woozle.org>

Here's the result of my inbox vs. spambox.  The inbox is my 1000
personal messages, and some back posts to the thttpd list and the
virgule-dev list.  I've checked them all over for spam and I'm pretty
sure (but not certain) that they're clean.  The spambox is my spam
folder, which I'm pretty sure (but again not certain) is totally dirty.
For the second run I added the lines:

  [Classifier]
  adjust_probs_by_evidence_mass: True
  min_spamprob: 0.001
  max_spamprob: 0.999
  hambias: 1.5

to bayescustomize.ini.


results1s -> results2s
-> <stat> tested 382 hams & 417 spams against 1002 hams & 1000 spams
-> <stat> tested 327 hams & 690 spams against 1002 hams & 1000 spams
-> <stat> tested 1002 hams & 1000 spams against 382 hams & 417 spams
-> <stat> tested 327 hams & 690 spams against 382 hams & 417 spams
-> <stat> tested 1002 hams & 1000 spams against 327 hams & 690 spams
-> <stat> tested 382 hams & 417 spams against 327 hams & 690 spams
-> <stat> tested 382 hams & 417 spams against 1002 hams & 1000 spams
-> <stat> tested 327 hams & 690 spams against 1002 hams & 1000 spams
-> <stat> tested 1002 hams & 1000 spams against 382 hams & 417 spams
-> <stat> tested 327 hams & 690 spams against 382 hams & 417 spams
-> <stat> tested 1002 hams & 1000 spams against 327 hams & 690 spams
-> <stat> tested 382 hams & 417 spams against 327 hams & 690 spams

false positive percentages
    2.094  3.665  lost   +75.02%
    0.612  2.141  lost  +249.84%
    34.331  40.419  lost   +17.73%
    13.456  11.927  won    -11.36%
    52.695  59.481  lost   +12.88%
    44.503  57.330  lost   +28.82%

won   1 times
tied  0 times
lost  5 times

total unique fp went from 798 to 898 lost   +12.53%
mean fp % went from 24.6150141717 to 29.1603737144 lost   +18.47%

false negative percentages
    0.959  0.480  won    -49.95%
    1.014  1.014  tied
    0.800  0.700  won    -12.50%
    1.449  1.304  won    -10.01%
    0.100  0.100  tied
    0.240  0.240  tied

won   3 times
tied  3 times
lost  0 times

total unique fn went from 29 to 25 won    -13.79%
mean fn % went from 0.760468147221 to 0.639710840023 won    -15.88%

From richie@entrian.com  Mon Sep 16 08:57:25 2002
From: richie@entrian.com (Richie Hindle)
Date: Mon, 16 Sep 2002 08:57:25 +0100
Subject: [Spambayes] pop3proxy.py
Message-ID: <o83boucjsrighptscos5h60sh6m0gc4jkj@4ax.com>

Hi,

I've just committed a spam-classifying POP3 proxy (pop3proxy.py) to the
spambayes project.  It adds a score header to each mail as it's retrieved,
so you can use your email client's filters to deal with them without
needing to fiddle with your email delivery system.

It also includes a test POP3 server which serves up a fixed set of two
emails, one spam and one ham, so you can test it without letting it loose
on your real email account (and it can run a self-test using that test
server).

A couple of notes and questions:

 o The score header it adds looks like 'X-Bayes-Score: 0.99' but it should
   probably be 'X-Hammie-Disposition:...' - is that peculiar to hammie, or
   is it up for grabs?  There's no single 'formatheader' or similar in
   hammie.py, but I'll be happy to refactor one in and make pop3proxy use
   it.

 o It doesn't distinguish between different users, mostly because I'm
   running it on my own Windows box at home.  It could be deployed on a
   server machine and use multiple databases, one per POP3 account (but
   if you're installing things on a server, why aren't you using hammie?)

 o It doesn't address the problem of updating the database with new
   messages.  One idea might be to make it an SMTP proxy as well, and
   forward hams and spams to a special address.  Not sure I like that,
   but I don't have any better ideas.

 o There's a known problem with large emails / slow connections / email
   clients that like to time out - see POP3ProxyBase.found_terminator for
   the gory details.  I'll commit a fix RSN.

 o Since this is my first commit to the project, I've made sure to only
   commit my file and to leave everything else alone.  That means I
   haven't added an entry for pop3proxy.py in README.txt - should I go
   ahead and do that, or send patches to the list for review until people
   are convinced I'm not going to break everything?  8-)

PS. I should have introduced myself: I'm Richie Hindle, software engineer,
enthusiastic Python programmer, spam-hater, lurker on comp.lang.python,
not very good at introductions.

-- 
Richie Hindle
richie@entrian.com
http://entrian.com


From sjoerd@acm.org  Mon Sep 16 16:00:59 2002
From: sjoerd@acm.org (Sjoerd Mullender)
Date: Mon, 16 Sep 2002 17:00:59 +0200
Subject: [Spambayes] Need testers!
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEBGBEAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCKEBGBEAB.tim.one@comcast.net> 
Message-ID: <200209161500.g8GF0xH04783@indus.ins.cwi.nl>

I've been saving all my incoming mail for just over 2 weeks now, and
tried this test on my data.  I have collected 3117 hams and 633 spams
which I divided into 4 sets of 150 messages each (with some left in
the reservoirs).

I didn't include messages to python-list-admin@python.org or
postmaster@oratrix.com in my corpus.

Here is the result of the test run with your first suggested change
(just adjust_probs_by_evidence_mass: True)

Before: adjust_probs_by_evidence_mass: False
After:  adjust_probs_by_evidence_mass: True

"""
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams

false positive percentages
    0.000  0.000  tied
    0.667  0.000  won   -100.00%
    0.000  0.000  tied
    0.667  0.667  tied

won   1 times
tied  3 times
lost  0 times

total unique fp went from 2 to 1 won    -50.00%
mean fp % went from 0.333333333334 to 0.166666666667 won    -50.00%

false negative percentages
    2.000  1.333  won    -33.35%
    0.667  1.333  lost   +99.85%
    0.667  1.333  lost   +99.85%
    2.000  2.000  tied

won   1 times
tied  1 times
lost  2 times

total unique fn went from 8 to 9 lost   +12.50%
mean fn % went from 1.33333333333 to 1.5 lost   +12.50%
"""

And here with the later suggested change
[Classifier]
adjust_probs_by_evidence_mass: True
min_spamprob: 0.001
max_spamprob: 0.999
hambias: 1.5

"""
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams

false positive percentages
    0.000  0.000  tied
    0.667  0.667  tied
    0.000  0.000  tied
    0.667  0.667  tied

won   0 times
tied  4 times
lost  0 times

total unique fp went from 2 to 2 tied
mean fp % went from 0.333333333334 to 0.333333333334 tied

false negative percentages
    2.000  1.333  won    -33.35%
    0.667  0.667  tied
    0.667  0.667  tied
    2.000  1.333  won    -33.35%

won   2 times
tied  2 times
lost  0 times

total unique fn went from 8 to 6 won    -25.00%
mean fn % went from 1.33333333333 to 0.999999999998 won    -25.00%
"""

-- Sjoerd Mullender <sjoerd@acm.org>

From tim.one@comcast.net  Mon Sep 16 16:25:28 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 16 Sep 2002 11:25:28 -0400
Subject: [Spambayes] Results of playing with CDB
In-Reply-To: <w538z224f8n.fsf@woozle.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEGNBEAB.tim.one@comcast.net>

[Neale Pickett]
> Sort of.  About half of them use pine on my server, and the other half
> are just figuring out how to browse the web, and asking them to install
> stuff is too tall an order.

So they do have their own CPUs and disk drives?  Whether it's too hard to
install a thing depends on how the installer is written; if a newbie can
click a button, they can install.

> On the plus side, their inboxes are likely to be very jargon-free.

The lack of jargon is likely to hurt more than help -- the classifier gets
as much good out of finding "good words" as "bad" ones, but the set of good
words likely varies across users.

> My end goal is a centralized classifier for an entire organization on an
> embedded device.  At $FIRM, our current-generation devices have between
> 4MB and 16MB of flash and no hard drive.  The next ones we make will
> have more, but I don't know what that'll be yet.  This is why I'm so
> concerned about storage space, though :)

I'll confess that I'm not opposed to slashing memory requirements <wink>.

> ...
> Plagued by conscience, I've just run my 1000 test hams against your
> SpamHam1.pik classifier.

Using the current code base?  The current code base doesn't match that
classifier, alas -- e.g., whenever the tokenization changes, the "words"
produced can be very different.  There are, for example, lots of spam words
in that database that the current tokenizer *can't* produce anymore, due to
things like changing the way URLs are tokenized.

> It came back saying that 51 were spam, and 949 were ham.  Not bad, eh?

Most people would find a 5% f-p rate unacceptable, and I've demonstrated
rates over 100x smaller with the current code base and sufficient training.

>  I've put the output of a "hammie.py -u" run at
>
>   http://woozle.org/~neale/tmp/results.txt

Case in point <wink>:  there's one f-p there containing all this stuff:

'clergy': 0.99;
'cart': 0.99;
'literature,': 0.99;
'assortment': 0.99;
'headquarters': 0.99;
'postal': 0.99;
'subject:confirmation': 0.99;
'ministry,': 0.99;
'conducting':
'ceremonies': 0.99;
'orders': 0.99;
'ordination.': 0.99;
'shopping': 0.99;
'email name:sales':
'ordination': 0.99;
'ministry.': 0.99;
'andrews': 0.99;
'baptisms': 0.99;
'booklet': 0.99;
'from:"mr.': 0.99;
'california.': 0.99;
'comply': 0.99;
'price,': 0.99;
'minister': 0.99;
'materials': 0.99;
'ordained': 0.99;
'deliveries': 0.99;
'officiating': 0.99;
'funerals,': 0.99;
'subject:Welcome': 0.99;
'shipping*': 0.99;
'rites': 0.99

Now if you've got one user who sucked for a minister-by-mail scam, training
a classifier to view this as ham is going to let similar scams through to
all your users.

> All default values in my .ini file.  That is to say, I have no .ini file
> :)  I'll go over unread messages and see if you need any other tests run.
> Sorry I've been slackin'.

Don't use SpamHam1.pik with the current codebase; I'll delete that file now
so nobody else falls into this trap.


From gward@python.net  Mon Sep 16 16:40:36 2002
From: gward@python.net (Greg Ward)
Date: Mon, 16 Sep 2002 11:40:36 -0400
Subject: [Spambayes] Hi - just got on the list....
In-Reply-To: <v03110751b9a8adb94c7f@[192.168.0.2]>
References: <v03110751b9a8adb94c7f@[192.168.0.2]>
Message-ID: <20020916154036.GA6292@cthulhu.gerg.ca>

On 14 September 2002, John Draper said:
> Allow me to introduce myself...

You forgot to introduce yourself!  Luckily Google helped out.  I'm not
sure if it's depressing or heartening that even the old-guard phone
phreaks are overwhelmed by spam.

> My partner (who is very good programmer) wrote a MTA in Python (in
> just a half hour), then I modified it to handle multiple threads,
> and wish to use this as a base of a development of a "proof of
> concept" to see how well this might work.

I have been having a load of fun with Python embedded in Exim, my
MTA-of-choice.  I imagine I would have even more fun if the whole MTA
were written in Python.  But I have to admit I'm skeptical that a
half-hour job can approach the stability, robustness, performance, and
flexibility of something like postfix, Exim, or qmail.

> Right now, I'm just trying to determine what's out there, so want to
> participate in this list to see what the current anti-spam technology
> level is at, and what you guys are doing.

If I were a spammer, I'd be worried.  For the first couple years, the
main force behind anti-spam technology was anger.  That hasn't worked --
look at how many open relays there still are.  Look at how many ISPs
wink and nod at spammers, as long as they pay the bills.  Now the tide
is shifting towards intelligence.  Spammers are mean, they're spiteful,
and above all they're persistent -- but intelligent they are not.

> As a result of my efforts to eliminate spam, I have 33 megs of spam
> I've collected (About 6000 spam messages) over the past year.  I
> didn't save ALL of it, as I've eliminated redundancy, and can make
> them available if anyone wants it.

That *could* be a useful contribution.  Probably the best thing you can
do with it is put it on a web site somewhere and let us know the URL.
However, the fact that all of this spam is from your inbox -- as opposed
to a front-line MTA somewhere -- means that it will be indelibly tagged
with the path from various MTAs to your inbox.  Lots of it will have
"To" headers pointing at you (OK, some spammers have figured out that
one), and all of it will have "Received" headers outlining the path from
your ISP/employer/web host/email host/whatever to your inbox.  All of
those can provide clues for a content-scanner to latch onto, which is
precisely what we don't want: eg. a large body of known spam, 50% of
which has "To: crunch@shopip.com", means that "To: crunch@shopip.com" is
an excellent spam clue *in the context of that corpus*.  Not exactly
what you want.

Anyways, you could do much worse than put your spam collection on the
web somewhere.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
God made machine language; all the rest is the work of man.

From gward@python.net  Mon Sep 16 16:43:43 2002
From: gward@python.net (Greg Ward)
Date: Mon, 16 Sep 2002 11:43:43 -0400
Subject: [Spambayes] Results of playing with CDB
In-Reply-To: <w53d6rf3to3.fsf@woozle.org>
References: <LNBBLJKPBEHFEDALKOLCGECFBEAB.tim@zope.com>
	<w53hegr3wee.fsf@woozle.org>
	<200209151935.g8FJZCX09918@pcp02138704pcs.reston01.va.comcast.net>
	<w53d6rf3to3.fsf@woozle.org>
Message-ID: <20020916154343.GB6292@cthulhu.gerg.ca>

On 15 September 2002, Neale Pickett said:
> Any objections to using XML-RPC as the "protocol"?

Only one: performance.  If you can make it as fast as a simple custom
protocol, go for it.

Oh yeah, and the fact that XML-RCP can't represent None.  Or has that
been fixed yet?

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Those of you who think you know everything really annoy those of us who do.

From gward@python.net  Mon Sep 16 16:49:56 2002
From: gward@python.net (Greg Ward)
Date: Mon, 16 Sep 2002 11:49:56 -0400
Subject: [Spambayes] Getting rid of max_spamprob and min_spamprob
In-Reply-To: <LNBBLJKPBEHFEDALKOLCAEEDBEAB.tim.one@comcast.net>
References: <20020915231302.GA16483@glacier.arctrix.com>
	<LNBBLJKPBEHFEDALKOLCAEEDBEAB.tim.one@comcast.net>
Message-ID: <20020916154956.GC6292@cthulhu.gerg.ca>

On 15 September 2002, Tim Peters said:
> Here we go again <wink>.  It also suggests a way to get "a middle ground"
> for Greg:  use two scoring schemes, one that has no false negatives and one
> that has no false positives.  If a given msg gets conflicting predictions
> from these two extremes, Greg reviews it by eyeball.

You keep talking as though I *enjoy* reviewing 5-20 spams every morning
for false positives.  Anything but!  I want a 100% accurate spam
detector, where every message is either definitely spam or definitely
not-spam.

But then, who doesn't?

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
If it can't be expressed in figures, it is not science--it is opinion.

From jeremy@alum.mit.edu  Mon Sep 16 16:46:48 2002
From: jeremy@alum.mit.edu (Jeremy Hylton)
Date: Mon, 16 Sep 2002 11:46:48 -0400
Subject: [Spambayes] Results of playing with CDB
In-Reply-To: <20020916154343.GB6292@cthulhu.gerg.ca>
References: <LNBBLJKPBEHFEDALKOLCGECFBEAB.tim@zope.com>
	<w53hegr3wee.fsf@woozle.org>
	<200209151935.g8FJZCX09918@pcp02138704pcs.reston01.va.comcast.net>
	<w53d6rf3to3.fsf@woozle.org>
	<20020916154343.GB6292@cthulhu.gerg.ca>
Message-ID: <15749.64744.278642.266456@slothrop.zope.com>

It seems like ZODB+ZEO would provide a straightforward solution here.
Have one ZEO server with a separate classifier object for each user.
The one-off script establishes a ZEO connection to the server and does
its work.  

Jeremy


From guido@python.org  Mon Sep 16 16:51:57 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 16 Sep 2002 11:51:57 -0400
Subject: [Spambayes] Results of playing with CDB
In-Reply-To: Your message of "Mon, 16 Sep 2002 11:46:48 EDT."
             <15749.64744.278642.266456@slothrop.zope.com> 
References: <LNBBLJKPBEHFEDALKOLCGECFBEAB.tim@zope.com>
	<w53hegr3wee.fsf@woozle.org>
	<200209151935.g8FJZCX09918@pcp02138704pcs.reston01.va.comcast.net>
	<w53d6rf3to3.fsf@woozle.org> <20020916154343.GB6292@cthulhu.gerg.ca>  
	<15749.64744.278642.266456@slothrop.zope.com> 
Message-ID: <200209161551.g8GFpvq26494@pcp02138704pcs.reston01.va.comcast.net>

> It seems like ZODB+ZEO would provide a straightforward solution here.
> Have one ZEO server with a separate classifier object for each user.
> The one-off script establishes a ZEO connection to the server and does
> its work.  

Have you timed how long it takes to start up a script that import
enough stuff to be able to connect to a ZEO server?  I'd expect that
the time to load the layers and layers of ZODB code would kill the
performance.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From jeremy@alum.mit.edu  Mon Sep 16 17:55:21 2002
From: jeremy@alum.mit.edu (Jeremy Hylton)
Date: Mon, 16 Sep 2002 12:55:21 -0400
Subject: [Spambayes] Results of playing with CDB
In-Reply-To: <200209161551.g8GFpvq26494@pcp02138704pcs.reston01.va.comcast.net>
References: <LNBBLJKPBEHFEDALKOLCGECFBEAB.tim@zope.com>
	<w53hegr3wee.fsf@woozle.org>
	<200209151935.g8FJZCX09918@pcp02138704pcs.reston01.va.comcast.net>
	<w53d6rf3to3.fsf@woozle.org>
	<20020916154343.GB6292@cthulhu.gerg.ca>
	<15749.64744.278642.266456@slothrop.zope.com>
	<200209161551.g8GFpvq26494@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <15750.3321.564064.870545@slothrop.zope.com>

>>>>> "GvR" == Guido van Rossum <guido@python.org> writes:

  >> It seems like ZODB+ZEO would provide a straightforward solution
  >> here.  Have one ZEO server with a separate classifier object for
  >> each user.  The one-off script establishes a ZEO connection to
  >> the server and does its work.

  GvR> Have you timed how long it takes to start up a script that
  GvR> import enough stuff to be able to connect to a ZEO server?  I'd
  GvR> expect that the time to load the layers and layers of ZODB code
  GvR> would kill the performance.

It's a bit slow, but I don't know what is acceptable.

from ZEO.ClientStoratge import ClientStorage

takes 0.23 sec w/ python2.1
takes 0.09 sec w/ python2.1 -O

I guess there's a non-trivial amount of Python code getting executed
during the imports.  Not a surprise given all the import tricks that
go on.

Jeremy


From tim.one@comcast.net  Mon Sep 16 18:33:30 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 16 Sep 2002 13:33:30 -0400
Subject: [Spambayes] Need testers!
In-Reply-To: <200209161500.g8GF0xH04783@indus.ins.cwi.nl>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEHMBEAB.tim.one@comcast.net>

[Sjoerd Mullender]
> I've been saving all my incoming mail for just over 2 weeks now, and
> tried this test on my data.  I have collected 3117 hams and 633 spams
> which I divided into 4 sets of 150 messages each (with some left in
> the reservoirs).

If these came to the same box, you should be able to improve results via

    [Tokenizer]
    count_all_header_lines: True
    mine_received_headers: True

They're off by default because they improve results *too* much (i.e., for
bogus reasons) when the ham and spam come from different sources.  Maybe I
should enable them by default now?

OTOH, your error rates are already too low to measure reliably with the
amount of data you have (0.667% of 150 messages is a single message, so is
the smallest non-zero error rate you can possibly see).

> I didn't include messages to python-list-admin@python.org or
> postmaster@oratrix.com in my corpus.

Any particular reason?

> Here is the result of the test run with your first suggested change
> (just adjust_probs_by_evidence_mass: True)
>
> Before: adjust_probs_by_evidence_mass: False
> After:  adjust_probs_by_evidence_mass: True

That suggestion was made for an earlier version of the code.  It's
interesting to try it anyway, though <wink>.

> """
> -> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
> -> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
> -> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
> -> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
> -> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
> -> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
> -> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
> -> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
>
> false positive percentages
>     0.000  0.000  tied
>     0.667  0.000  won   -100.00%
>     0.000  0.000  tied
>     0.667  0.667  tied
>
> won   1 times
> tied  3 times
> lost  0 times
>
> total unique fp went from 2 to 1 won    -50.00%
> mean fp % went from 0.333333333334 to 0.166666666667 won    -50.00%
>
> false negative percentages
>     2.000  1.333  won    -33.35%
>     0.667  1.333  lost   +99.85%
>     0.667  1.333  lost   +99.85%
>     2.000  2.000  tied
>
> won   1 times
> tied  1 times
> lost  2 times
>
> total unique fn went from 8 to 9 lost   +12.50%
> mean fn % went from 1.33333333333 to 1.5 lost   +12.50%
> """

An increase in the f-n rate is also what I saw when I tried small training
sets (compared to mine, yours is small <wink>), and that's why I changed the
code and suggested a different experiment to go along with that.  Namely:

> And here with the later suggested change
> [Classifier]
> adjust_probs_by_evidence_mass: True
> min_spamprob: 0.001
> max_spamprob: 0.999
> hambias: 1.5

Bingo!  This is the correct experiment to run with the current codebase.

> """
> -> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
> -> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
> -> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
> -> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
> -> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
> -> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
> -> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
> -> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
>
> false positive percentages
>     0.000  0.000  tied
>     0.667  0.667  tied
>     0.000  0.000  tied
>     0.667  0.667  tied
>
> won   0 times
> tied  4 times
> lost  0 times
>
> total unique fp went from 2 to 2 tied
> mean fp % went from 0.333333333334 to 0.333333333334 tied
>
> false negative percentages
>     2.000  1.333  won    -33.35%
>     0.667  0.667  tied
>     0.667  0.667  tied
>     2.000  1.333  won    -33.35%
>
> won   2 times
> tied  2 times
> lost  0 times
>
> total unique fn went from 8 to 6 won    -25.00%
> mean fn % went from 1.33333333333 to 0.999999999998 won    -25.00%
> """

Cool!  It's working for you much as it worked for me, although I can be more
confident because I've got enough data to run 10-fold c-v experiments, and
across many distinct small random subsets.  It's encouraging that someone
other than me <wink> is able to get sub-1% error rates with this little
data.

Thanks for the report, Sjoerd!  You may want to look at the lists of best
discrimators to try to guess whether your results are so good for bogus
reasons.  Staring at the details of the false positives and negatives may
also suggest weaknesses in the tokenization algorithms (anyone can play!
trying out changes is fun, although testing them is tedious, time-consuming,
and unfortunately necessary).


From tim.one@comcast.net  Mon Sep 16 18:45:06 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 16 Sep 2002 13:45:06 -0400
Subject: [Spambayes] Getting rid of max_spamprob and min_spamprob
In-Reply-To: <20020916154956.GC6292@cthulhu.gerg.ca>
Message-ID: <LNBBLJKPBEHFEDALKOLCEEHNBEAB.tim.one@comcast.net>

[Greg Ward]
> You keep talking as though I *enjoy* reviewing 5-20 spams every morning
> for false positives.  Anything but!

Heh.  Greg, I have a hard time believing that reviewing 20 msgs by eyeball
for spam-vs-ham can consume more than two minutes.  I do this on hundreds of
msgs every day, and it's barely reached annoyance level.

> I want a 100% accurate spam detector, where every message is either
> definitely spam or definitely not-spam.
>
> But then, who doesn't?

Easy:  hire me to look at your spam.  I'll be happy to review this stuff for
you, at $1 per message, plus a 25-cent surcharge for each farm porn reviewed
(I have *some* standards <wink>).

Every scheme makes mistakes, so Neil and I were speculating about a way that
*might* produce a useful "I'm not sure about this one" indicator from a
Graham-like scheme.  As is, the scheme claims to be nearly 100% certain of
its decisions nearly 100% of the time, and is wrong so seldom in my tests
that I can't reliably measure the error rates any more despite training on
more than 10,000 of each per run.


From neale@woozle.org  Mon Sep 16 19:12:06 2002
From: neale@woozle.org (Neale Pickett)
Date: 16 Sep 2002 11:12:06 -0700
Subject: [Spambayes] pop3proxy.py
In-Reply-To: <o83boucjsrighptscos5h60sh6m0gc4jkj@4ax.com>
References: <o83boucjsrighptscos5h60sh6m0gc4jkj@4ax.com>
Message-ID: <w53znuh3itl.fsf@woozle.org>

So then, Richie Hindle <richie@entrian.com> is all like:

> Hi,
> 
> I've just committed a spam-classifying POP3 proxy (pop3proxy.py) to the
> spambayes project.  It adds a score header to each mail as it's retrieved,
> so you can use your email client's filters to deal with them without
> needing to fiddle with your email delivery system.

Coolness!

>  o The score header it adds looks like 'X-Bayes-Score: 0.99' but it should
>    probably be 'X-Hammie-Disposition:...' - is that peculiar to hammie, or
>    is it up for grabs?  There's no single 'formatheader' or similar in
>    hammie.py, but I'll be happy to refactor one in and make pop3proxy use
>    it.

I hold no claim to the header.  I think as long as the header you add is
formatted the same, you should use the same name.  If you format it
differently, probably best to give it a different name.  Better yet, if
you come up with a better format, check the change in to hammie.py
too :)

>  o Since this is my first commit to the project, I've made sure to only
>    commit my file and to leave everything else alone.  That means I
>    haven't added an entry for pop3proxy.py in README.txt - should I go
>    ahead and do that, or send patches to the list for review until people
>    are convinced I'm not going to break everything?  8-)

I was told to modify README when submitting new files, and I consider
myself pretty much in the same boat as you (enthusiastic outsider).  So
I think you should probably modify the README, but I'm not authoritative
:)

Neale


From gward@python.net  Mon Sep 16 19:35:22 2002
From: gward@python.net (Greg Ward)
Date: Mon, 16 Sep 2002 14:35:22 -0400
Subject: [Spambayes] Getting rid of max_spamprob and min_spamprob
In-Reply-To: <LNBBLJKPBEHFEDALKOLCEEHNBEAB.tim.one@comcast.net>
References: <20020916154956.GC6292@cthulhu.gerg.ca>
	<LNBBLJKPBEHFEDALKOLCEEHNBEAB.tim.one@comcast.net>
Message-ID: <20020916183522.GA7076@cthulhu.gerg.ca>

On 16 September 2002, Tim Peters said:
> Every scheme makes mistakes, so Neil and I were speculating about a way that
> *might* produce a useful "I'm not sure about this one" indicator from a
> Graham-like scheme.

Ahh, OK.  I missed that detail.  The ability to admit uncertainty would
definitely be a nice feature -- the highly bipolar distribution of spam
scores is alluring at first, until you realize that most FPs score the
same as real spam, and most FNs score the same as regular mail.

        Greg

From neale@woozle.org  Mon Sep 16 19:36:56 2002
From: neale@woozle.org (Neale Pickett)
Date: 16 Sep 2002 11:36:56 -0700
Subject: [Spambayes] Results of playing with CDB
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCEGNBEAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCCEGNBEAB.tim.one@comcast.net>
Message-ID: <w53vg553ho7.fsf@woozle.org>

So then, Tim Peters <tim.one@comcast.net> is all like:

> > On the plus side, their inboxes are likely to be very jargon-free.
> 
> The lack of jargon is likely to hurt more than help -- the classifier
> gets as much good out of finding "good words" as "bad" ones, but the
> set of good words likely varies across users.

A jargon-free mailbox or two seems a good proving ground for whether or
not an all-users classifier is feasable.

> > Plagued by conscience, I've just run my 1000 test hams against your
> > SpamHam1.pik classifier.
> 
> Using the current code base?  

Yeah :(  I hesitated.  I lost.

> Case in point <wink>:  there's one f-p there containing all this stuff:
> 
> [ ordianed-by-mail keywords removed ]
> 
> Now if you've got one user who sucked for a minister-by-mail scam, training
> a classifier to view this as ham is going to let similar scams through to
> all your users.

That's exactly what it was, someone forwarded to me a message certifying
them as a minister.  My classifier database scores it as ham (p=0.00) on
the following words:

        '$33.90': 0.01;
        'california,': 0.01;
        'california!': 0.01; 
        "minister's": 0.01; 
        'divinity': 0.01;
        'church,': 0.01;
        'funerals,': 0.01;
        'ministers),': 0.01;
        'ministers).': 0.01;
        're-ordained.': 0.01;
        'wedding': 0.01;
        'hardbound': 0.01;
        'deliveries': 0.01;
        'ordained': 0.01; 
        'pass,': 0.01;
        'processed,': 0.01

Of course, that's because I *trained* it on this message (among others).
But what's interesting is that two of the big tip-offs from SpamHam1.pik
("ordained": 0.99, "funerals,": 0.99) show up as strong ham indicators
in my database.  Doesn't 0.01 mean it's never been seen as spam?  I know
it's not a lot of words, but I wonder if this is evidence that the
character of people's spam is just as individual as the character of
their ham.  That would point toward spammers doing more targetting than
I thought they were.

From tim.one@comcast.net  Mon Sep 16 19:45:01 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 16 Sep 2002 14:45:01 -0400
Subject: [Spambayes] Need testers!
In-Reply-To: <w534rcq4ece.fsf@woozle.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCOEIBBEAB.tim.one@comcast.net>

[Neale Pickett]
> Here's the result of my inbox vs. spambox.  The inbox is my 1000
> personal messages, and some back posts to the thttpd list and the
> virgule-dev list.  I've checked them all over for spam and I'm pretty
> sure (but not certain) that they're clean.  The spambox is my spam
> folder, which I'm pretty sure (but again not certain) is totally dirty.
> For the second run I added the lines:
>
>   [Classifier]
>   adjust_probs_by_evidence_mass: True
>   min_spamprob: 0.001
>   max_spamprob: 0.999
>   hambias: 1.5
>
> to bayescustomize.ini.
>
>
> results1s -> results2s
> -> <stat> tested 382 hams & 417 spams against 1002 hams & 1000 spams
> -> <stat> tested 327 hams & 690 spams against 1002 hams & 1000 spams
> -> <stat> tested 1002 hams & 1000 spams against 382 hams & 417 spams
> -> <stat> tested 327 hams & 690 spams against 382 hams & 417 spams
> -> <stat> tested 1002 hams & 1000 spams against 327 hams & 690 spams
> -> <stat> tested 382 hams & 417 spams against 327 hams & 690 spams
> -> <stat> tested 382 hams & 417 spams against 1002 hams & 1000 spams
> -> <stat> tested 327 hams & 690 spams against 1002 hams & 1000 spams
> -> <stat> tested 1002 hams & 1000 spams against 382 hams & 417 spams
> -> <stat> tested 327 hams & 690 spams against 382 hams & 417 spams
> -> <stat> tested 1002 hams & 1000 spams against 327 hams & 690 spams
> -> <stat> tested 382 hams & 417 spams against 327 hams & 690 spams

I *assume* you're not using timcv.py to run the tests, and aren't using "the
standard" one-msg-per-file setup, and so also can't use rebal.py to balance
the # of hams and spams across sets, and also aren't doing cross validation
runs here (this looks like you were running a 3x3 test grid).

> false positive percentages
>     2.094  3.665  lost   +75.02%
>     0.612  2.141  lost  +249.84%
>     34.331  40.419  lost   +17.73%
>     13.456  11.927  won    -11.36%
>     52.695  59.481  lost   +12.88%
>     44.503  57.330  lost   +28.82%

Those numbers are spectacularly bad, both the "before" and "after" numbers,
much worse than anyone else has reported.  It's hard not to suspect a bug in
whichever test driver script you used (I'm not set up to run, e.g.,
mboxtest.py, so don't have any feel for how that's working these days -- but
if that's what you were using, it looks like it's not working these days
<wink>).

> won   1 times
> tied  0 times
> lost  5 times
>
> total unique fp went from 798 to 898 lost   +12.53%
> mean fp % went from 24.6150141717 to 29.1603737144 lost   +18.47%
>
> false negative percentages
>     0.959  0.480  won    -49.95%
>     1.014  1.014  tied
>     0.800  0.700  won    -12.50%
>     1.449  1.304  won    -10.01%
>     0.100  0.100  tied
>     0.240  0.240  tied
>
> won   3 times
> tied  3 times
> lost  0 times
>
> total unique fn went from 29 to 25 won    -13.79%
> mean fn % went from 0.760468147221 to 0.639710840023 won    -15.88%

Too weird.  Who else runs mboxtest.py (assuming that's what Neale runs)?
Has it gone to hell recently for you?


From neale@woozle.org  Mon Sep 16 20:37:57 2002
From: neale@woozle.org (Neale Pickett)
Date: 16 Sep 2002 12:37:57 -0700
Subject: [Spambayes] Need testers!
In-Reply-To: <LNBBLJKPBEHFEDALKOLCOEIBBEAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCOEIBBEAB.tim.one@comcast.net>
Message-ID: <w53r8ft3eui.fsf@woozle.org>

So then, Tim Peters <tim.one@comcast.net> is all like:

> I *assume* you're not using timcv.py to run the tests, and aren't using "the
> standard" one-msg-per-file setup, and so also can't use rebal.py to balance
> the # of hams and spams across sets, and also aren't doing cross validation
> runs here (this looks like you were running a 3x3 test grid).

I ran timtest.py, and do have the standard setup.  I'll re-create my
messages and rebal them, then I'll run it again and let you know how it
went, along with the exact commands I used to set everything up :)

From sjoerd@acm.org  Mon Sep 16 21:00:48 2002
From: sjoerd@acm.org (Sjoerd Mullender)
Date: Mon, 16 Sep 2002 22:00:48 +0200
Subject: [Spambayes] Need testers!
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCEHMBEAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCCEHMBEAB.tim.one@comcast.net> 
Message-ID: <200209162000.g8GK0mw15197@indus.ins.cwi.nl>

On Mon, Sep 16 2002 Tim Peters wrote:

> [Sjoerd Mullender]
> > I've been saving all my incoming mail for just over 2 weeks now, and
> > tried this test on my data.  I have collected 3117 hams and 633 spams
> > which I divided into 4 sets of 150 messages each (with some left in
> > the reservoirs).
> 
> If these came to the same box, you should be able to improve results via
> 
>     [Tokenizer]
>     count_all_header_lines: True
>     mine_received_headers: True
> 
> They're off by default because they improve results *too* much (i.e., for
> bogus reasons) when the ham and spam come from different sources.  Maybe I
> should enable them by default now?
> 
> OTOH, your error rates are already too low to measure reliably with the
> amount of data you have (0.667% of 150 messages is a single message, so is
> the smallest non-zero error rate you can possibly see).

I guess there are four interesting comparisons now:
no bayescustomize.ini - Tokenizer options
no bayescustomize.ini - Tokenizer + Classifier options
Tokenizer options - Tokenizer + Classifier options
Classifier options - Tokenizer + Classifier options

Here they are in that order:

"""
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams

false positive percentages
    0.000  0.000  tied
    0.667  0.000  won   -100.00%
    0.000  0.000  tied
    0.667  0.667  tied

won   1 times
tied  3 times
lost  0 times

total unique fp went from 2 to 1 won    -50.00%
mean fp % went from 0.333333333334 to 0.166666666667 won    -50.00%

false negative percentages
    2.000  2.000  tied
    0.667  2.000  lost  +199.85%
    0.667  0.000  won   -100.00%
    2.000  2.000  tied

won   1 times
tied  2 times
lost  1 times

total unique fn went from 8 to 9 lost   +12.50%
mean fn % went from 1.33333333333 to 1.5 lost   +12.50%
"""

"""
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams

false positive percentages
    0.000  0.000  tied
    0.667  0.667  tied
    0.000  0.000  tied
    0.667  0.667  tied

won   0 times
tied  4 times
lost  0 times

total unique fp went from 2 to 2 tied
mean fp % went from 0.333333333334 to 0.333333333334 tied

false negative percentages
    2.000  1.333  won    -33.35%
    0.667  2.000  lost  +199.85%
    0.667  0.667  tied
    2.000  2.667  lost   +33.35%

won   1 times
tied  1 times
lost  2 times

total unique fn went from 8 to 10 lost   +25.00%
mean fn % went from 1.33333333333 to 1.66666666667 lost   +25.00%
"""

"""
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams

false positive percentages
    0.000  0.000  tied
    0.000  0.667  lost  +(was 0)
    0.000  0.000  tied
    0.667  0.667  tied

won   0 times
tied  3 times
lost  1 times

total unique fp went from 1 to 2 lost  +100.00%
mean fp % went from 0.166666666667 to 0.333333333334 lost  +100.00%

false negative percentages
    2.000  1.333  won    -33.35%
    2.000  2.000  tied
    0.000  0.667  lost  +(was 0)
    2.000  2.667  lost   +33.35%

won   1 times
tied  1 times
lost  2 times

total unique fn went from 9 to 10 lost   +11.11%
mean fn % went from 1.5 to 1.66666666667 lost   +11.11%
"""

"""
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams
-> <stat> tested 150 hams & 150 spams against 450 hams & 450 spams

false positive percentages
    0.000  0.000  tied
    0.667  0.667  tied
    0.000  0.000  tied
    0.667  0.667  tied

won   0 times
tied  4 times
lost  0 times

total unique fp went from 2 to 2 tied
mean fp % went from 0.333333333334 to 0.333333333334 tied

false negative percentages
    1.333  1.333  tied
    0.667  2.000  lost  +199.85%
    0.667  0.667  tied
    1.333  2.667  lost  +100.08%

won   0 times
tied  2 times
lost  2 times

total unique fn went from 6 to 10 lost   +66.67%
mean fn % went from 0.999999999998 to 1.66666666667 lost   +66.67%
"""

> > I didn't include messages to python-list-admin@python.org or
> > postmaster@oratrix.com in my corpus.
> 
> Any particular reason?

The postmaster address gets a lot of bounced spam, and the
python-list-admin address is currently being overwhelmed with warnings
about delayed mail from some French site (328 of 511 messages).

> > And here with the later suggested change
> > [Classifier]
> > adjust_probs_by_evidence_mass: True
> > min_spamprob: 0.001
> > max_spamprob: 0.999
> > hambias: 1.5
> 
> Bingo!  This is the correct experiment to run with the current codebase.

[...]

> Cool!  It's working for you much as it worked for me, although I can be more
> confident because I've got enough data to run 10-fold c-v experiments, and
> across many distinct small random subsets.  It's encouraging that someone
> other than me <wink> is able to get sub-1% error rates with this little
> data.
> 
> Thanks for the report, Sjoerd!  You may want to look at the lists of best
> discrimators to try to guess whether your results are so good for bogus
> reasons.  Staring at the details of the false positives and negatives may
> also suggest weaknesses in the tokenization algorithms (anyone can play!
> trying out changes is fun, although testing them is tedious, time-consuming,
> and unfortunately necessary).

Two messages keep coming back as false positives.  One is an
announcement from Palm (PDA's) and the other from the new CEO of
United Airlines, os both messages are a much like UCE, except
technically they weren't U.

The false negatives are much more varied, and one of them wasn't even
a negative but was misplaced in the spam folder.

I don't really see anything weird in the best discriminators (the ones
I looked at, that is).

-- Sjoerd Mullender <sjoerd@acm.org>

From neale@woozle.org  Mon Sep 16 21:07:23 2002
From: neale@woozle.org (Neale Pickett)
Date: 16 Sep 2002 13:07:23 -0700
Subject: [Spambayes] Need testers!
In-Reply-To: <w53r8ft3eui.fsf@woozle.org>
References: <LNBBLJKPBEHFEDALKOLCOEIBBEAB.tim.one@comcast.net>
	<w53r8ft3eui.fsf@woozle.org>
Message-ID: <w53n0qh3dhg.fsf@woozle.org>

So then, Neale Pickett <neale@woozle.org> is all like:

> So then, Tim Peters <tim.one@comcast.net> is all like:
> 
> > I *assume* you're not using timcv.py to run the tests, and aren't using "the
> > standard" one-msg-per-file setup, and so also can't use rebal.py to balance
> > the # of hams and spams across sets, and also aren't doing cross validation
> > runs here (this looks like you were running a 3x3 test grid).
> 
> I ran timtest.py, and do have the standard setup.  I'll re-create my
> messages and rebal them, then I'll run it again and let you know how it
> went, along with the exact commands I used to set everything up :)

Okay, timcv on a well-balanced set of spam n' ham produces numbers which
to me look more reasonable.  Since my set consists exclusively of mail
I've received, my dataset is a bit small.

If I did this all right, I think I'd like to check in a script to chain
all these actions together, so knuckleheads like me don't go running
useless tests again.


gwydion:~/src/import/spambayes$ python rebal.py -r Data/Ham/reservoir -s Data/Ham/Set -n 200 -Q
gwydion:~/src/import/spambayes$ python rebal.py -r Data/Spam/reservoir -s Data/Spam/Set -n 200 -Q
gwydion:~/src/import/spambayes$ rm -f bayescustomize.ini
gwydion:~/src/import/spambayes$ python timcv.py -n 5 > run1.txt
gwydion:~/src/import/spambayes$ cp bayescustomize.mod.ini bayescustomize.ini     
gwydion:~/src/import/spambayes$ cat bayescustomize.ini
[Classifier]
adjust_probs_by_evidence_mass: True
min_spamprob: 0.001
max_spamprob: 0.999
hambias: 1.5
gwydion:~/src/import/spambayes$ python timcv.py -n 5 > run2.txt
gwydion:~/src/import/spambayes$ python rates.py run1 run2 > runrates.txt
gwydion:~/src/import/spambayes$ python cmp.py run1s run2s
run1s -> run2s
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams

false positive percentages
    2.000  1.000  won    -50.00%
    1.000  1.000  tied
    2.500  1.000  won    -60.00%
    1.000  0.500  won    -50.00%
    1.000  1.000  tied

won   3 times
tied  2 times
lost  0 times

total unique fp went from 15 to 9 won    -40.00%
mean fp % went from 1.5 to 0.9 won    -40.00%

false negative percentages
    3.000  2.000  won    -33.33%
    3.500  2.000  won    -42.86%
    3.000  2.500  won    -16.67%
    2.500  2.000  won    -20.00%
    3.000  3.000  tied

won   4 times
tied  1 times
lost  0 times

total unique fn went from 30 to 23 won    -23.33%
mean fn % went from 3.0 to 2.3 won    -23.33%


From neale@woozle.org  Mon Sep 16 22:44:26 2002
From: neale@woozle.org (Neale Pickett)
Date: 16 Sep 2002 14:44:26 -0700
Subject: [Spambayes] What is spam?
Message-ID: <w53it1538zp.fsf@woozle.org>

My datasets aren't as pure as I thought :( While sorting through my FNs
and FPs, I've found some trends:

1.  When people forward spam to me, it gets tagged as spam.  I have a
    lot of forwarded spam in my inbox; I've asked people to send me
    stuff is the past so I can get a feel for what sort of spam is being
    sent to my users.  I use this feel to blacklist domains in the MAIL
    FROM: SMTP command.  It works pretty well when I stay on top of it.

2.  Non-spam I'd erroneously entered into my spam corpus gets most of
    the false negatives.  Neato!

3.  Stupid forwards (mostly urban legends, exultations to
    pray/boycott/vote a certain way, jokes, or inspirational stories)
    are not tagged as spam.  I get a lot of these too, from my
    grandmother and certain friends who seem to do nothing but relay
    chain letters.  But I don't get enough to train the filter against
    them, apparently.

With only one or two exceptions, that is the extent of my false
positives and false negatives.

I have to wonder, though, if the forwards (#3) are really false
negatives.  Should I have those in the ham folder, and be using another
method to weed out garbage of that type?  I'm not sure the current
classifier is up to sorting out urban legends :)

In any case, it's becoming clear to me that in the future when we're all
trying to help our grandmothers install spambayes, there will have to be
some way of reviewing FPs and FNs the way we're all doing it now.  In my
case at least, a lot of FPs and FNs aren't really F.

From tim.one@comcast.net  Tue Sep 17 00:18:25 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 16 Sep 2002 19:18:25 -0400
Subject: [Spambayes] Getting rid of max_spamprob and min_spamprob
In-Reply-To: <20020916183522.GA7076@cthulhu.gerg.ca>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEJCBEAB.tim.one@comcast.net>

[Greg Ward]
> Ahh, OK.  I missed that detail.  The ability to admit uncertainty would
> definitely be a nice feature -- the highly bipolar distribution of spam
> scores is alluring at first, until you realize that most FPs score the
> same as real spam, and most FNs score the same as regular mail.

I'm not sure it would be a nice feature.  It would introduce two new
measures, both of which carry their own headaches:

    the percentage of ham tagged "unsure"
    the percentage of spam tagged "unsure"

In addition, there will still be non-zero false negative and false positive
rates, due to ham that scores high for spamness under both measures, and
similarly spam scoring high for hamness under both measures.

You've done an admirable job of dodging this issue <wink>, but it won't go
away:  you have to decide on concrete non-zero f-p and f-n rates that are
acceptable.

after-that-it's-just-a-small-matter-of-engineering-ly y'rs  - tim


From tim.one@comcast.net  Tue Sep 17 00:30:05 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 16 Sep 2002 19:30:05 -0400
Subject: [Spambayes] Results of playing with CDB
In-Reply-To: <w53vg553ho7.fsf@woozle.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCEEJDBEAB.tim.one@comcast.net>

[Neale Pickett]
> A jargon-free mailbox or two seems a good proving ground for whether or
> not an all-users classifier is feasable.

I'm afraid the only way to test whether an all-users classifier is usable is
to run tests on an all-users classifier.  When we have enough testers *here*
set up with reasonably trained classifiers on their own mail, we could
combine them into one classifier (classifier merge code hasn't been written
yet but is an easy task), then everyone can run their tests again with the
combined classifier.  That would give us good clues.

>> Now if you've got one user who sucked for a minister-by-mail
>> scam, training a classifier to view this as ham is going to let
>> similar scams through to all your users.

> That's exactly what it was, someone forwarded to me a message certifying
> them as a minister.  My classifier database scores it as ham (p=0.00) on
> the following words:
>
>         '$33.90': 0.01;

LOL!

>         'california,': 0.01;
>         'california!': 0.01;
>         "minister's": 0.01;
>         'divinity': 0.01;
>         'church,': 0.01;
>         'funerals,': 0.01;
>         'ministers),': 0.01;
>         'ministers).': 0.01;
>         're-ordained.': 0.01;
>         'wedding': 0.01;
>         'hardbound': 0.01;
>         'deliveries': 0.01;
>         'ordained': 0.01;
>         'pass,': 0.01;
>         'processed,': 0.01
>
> Of course, that's because I *trained* it on this message (among others).
> But what's interesting is that two of the big tip-offs from SpamHam1.pik
> ("ordained": 0.99, "funerals,": 0.99) show up as strong ham indicators
> in my database.

They likely appeared only in the message you trained it on, telling the
system it was ham.  In the very much larger spam collection I trained the
.pik file on, they appear in several spams -- but remember that you didn't
train using my spam collection.

> Doesn't 0.01 mean it's never been seen as spam?

Either that, or that it's been seen substantially more often in ham than in
spam.  Intuitively, at least 100x more often, but that's not the real truth
because of the distorting effects of HAMBIAS.

> I know it's not a lot of words, but I wonder if this is evidence that
> the character of people's spam is just as individual as the character of
> their ham.  That would point toward spammers doing more targetting than
> I thought they were.

I expect it's evidence that minister-by-email spams are relatively rare,
that I picked some up because my spam corpus is relatively large, and that
you didn't pick some up because yours is relatively small.  A classifier
knows nothing about the world apart from what it's been trained on, of
course.


From gward@python.net  Tue Sep 17 02:09:26 2002
From: gward@python.net (Greg Ward)
Date: Mon, 16 Sep 2002 21:09:26 -0400
Subject: [Spambayes] What is spam?
In-Reply-To: <w53it1538zp.fsf@woozle.org>
References: <w53it1538zp.fsf@woozle.org>
Message-ID: <20020917010926.GA8305@cthulhu.gerg.ca>

On 16 September 2002, Neale Pickett said:
> 1.  When people forward spam to me, it gets tagged as spam.  I have a
>     lot of forwarded spam in my inbox; I've asked people to send me
>     stuff is the past so I can get a feel for what sort of spam is being
>     sent to my users.  I use this feel to blacklist domains in the MAIL
>     FROM: SMTP command.  It works pretty well when I stay on top of it.

I do something similar with python.org -- mail sent to spam@python.org
is dumped straight into /var/mail/nc-spam ("not caught") without going
through SpamAssassin.  In theory, I occasionally look at that folder and
tweak SpamAssassin to do a better job on those messages.

Rather than try to fool SpamAssassin into accepting anything for
spam@python.org, I simply bypass SA entirely; since this is all done in
Python code (I embed a Python interpreter because I didn't want to write
the local_scan() function in C):

  def local_scan (fd, headers, info):
      # Immediately accept any messages that originated locally.
      sender_ip = info.sender_host_address
      if (sender_ip is None or
          sender_ip == "127.0.0.1"):
          return

      # Immediately accept any messages destined for the "uncaught spam"
      # or "uncaught virus" folders.
      if (len(info.recipients_list) == 1 and
          info.recipients_list[0] in ("spam@python.org",
                                      "virus@python.org",
                                      "spam@zope.org",
                                      "virus@zope.org")):
          return

      [... now we do virus- and spam-checking ...]

For other MTAs, your mileage may vary.  But if you ask people to send
you spam, you probably want to keep that mail from going close to your
spam detector in the first place.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Money is truthful.  If a man speaks of his honor, make him pay cash.

From tim.one@comcast.net  Tue Sep 17 02:34:14 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 16 Sep 2002 21:34:14 -0400
Subject: [Spambayes] What is spam?
In-Reply-To: <w53it1538zp.fsf@woozle.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCAEJLBEAB.tim.one@comcast.net>

[Neale Pickett]
> My datasets aren't as pure as I thought :(

Nobody's have been -- it seems to come as a shock <wink>.

> While sorting through my FNs and FPs, I've found some trends:
>
> 1.  When people forward spam to me, it gets tagged as spam.  I have a
>     lot of forwarded spam in my inbox; I've asked people to send me
>     stuff is the past so I can get a feel for what sort of spam is being
>     sent to my users.  I use this feel to blacklist domains in the MAIL
>     FROM: SMTP command.  It works pretty well when I stay on top of it.

When mine-received-headers is true (it's false by default), Neil's code
picks out IP addresses (and their prefixes) and machine names from the
Received headers, and delivers them as tagged tokens to the classifier.  The
classifier will then learn which of these guys are and aren't good spam
indicators.  However, it will also pick up clues about the people who
forwarded this stuff, unless you're careful to strip away forwarding
artifacts.

> 2.  Non-spam I'd erroneously entered into my spam corpus gets most of
>     the false negatives.  Neato!

I've even found hams in bruceg's spam collections (e.g., one was the output
from a cron job he apparently ran on one of his collection machines).

> 3.  Stupid forwards (mostly urban legends, exultations to
>     pray/boycott/vote a certain way, jokes, or inspirational stories)
>     are not tagged as spam.  I get a lot of these too, from my
>     grandmother and certain friends who seem to do nothing but relay
>     chain letters.  But I don't get enough to train the filter against
>     them, apparently.

Can't guess whether *you* classified these as ham or spam.  In any case, you
don't have much training data yet (your last report had a thousand of each).

> With only one or two exceptions, that is the extent of my false
> positives and false negatives.
>
> I have to wonder, though, if the forwards (#3) are really false
> negatives.

False negative means that you put them in a spam folder but that the system
said they were ham.  Is that what you meant to say?

> Should I have those in the ham folder, and be using another
> method to weed out garbage of that type?  I'm not sure the current
> classifier is up to sorting out urban legends :)

In an ideal world you'd never see a message classified as spam.  Do you want
to see these messages or not?  Train accordingly; the system will learn over
time.

> In any case, it's becoming clear to me that in the future when we're all
> trying to help our grandmothers install spambayes, there will have to be
> some way of reviewing FPs and FNs the way we're all doing it now.  In my
> case at least, a lot of FPs and FNs aren't really F.

Put the things you think are ham in your ham collection, and the things you
think are spam in your spam collection.  By definition, a false positive is
a ham identified as spam -- if an FP is "not really F", it was *correctly*
identified as spam, and so didn't belong in your ham collection to begin
with; likewise for an FN that's not really F, it should have been in your
ham collection.  It learns what you teach it.


From tim.one@comcast.net  Tue Sep 17 03:33:10 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 16 Sep 2002 22:33:10 -0400
Subject: [Spambayes] Need testers!
In-Reply-To: <w53n0qh3dhg.fsf@woozle.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCMEJPBEAB.tim.one@comcast.net>

[Neale Pickett]
> Okay, timcv on a well-balanced set of spam n' ham produces numbers which
> to me look more reasonable.  Since my set consists exclusively of mail
> I've received, my dataset is a bit small.

Excellent report, Neale!  A later msg said you found a number of
misclassified msgs.  For purposes of deciding whether to make

> [Classifier]
> adjust_probs_by_evidence_mass: True
> min_spamprob: 0.001
> max_spamprob: 0.999
> hambias: 1.5

the default, it would help if you did this all over again.  It was a pure
and significant win for both error rates in this test report, but that may
change after msgs are shuffled between the spam and ham sets.

> If I did this all right, I think I'd like to check in a script to chain
> all these actions together, so knuckleheads like me don't go running
> useless tests again.

I agree that would be helpful (and, yes, it was all impeccably right).  I
have a collection of Windows .bat files to chain things together, but didn't
want to insult you Unixoids by checking them in <wink>.


From tim.one@comcast.net  Tue Sep 17 03:54:56 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 16 Sep 2002 22:54:56 -0400
Subject: [Spambayes] Need testers!
In-Reply-To: <200209162000.g8GK0mw15197@indus.ins.cwi.nl>
Message-ID: <LNBBLJKPBEHFEDALKOLCOEKABEAB.tim.one@comcast.net>

[Sjoerd Mullender]
> ...
> I guess there are four interesting comparisons now:
> no bayescustomize.ini - Tokenizer options
> no bayescustomize.ini - Tokenizer + Classifier options
> Tokenizer options - Tokenizer + Classifier options
> Classifier options - Tokenizer + Classifier options

There are millions:  every parameter than can be fiddled should be fiddled,
and exhaustively tested by everyone.  That's not going to happen, so for now
I'm just trying to determine whether a specific set of four connected
classifier changes is a widespread win.

> Here they are in that order:

I'm going to skip this, because:

>> OTOH, your error rates are already too low to measure reliably with the
>> amount of data you have (0.667% of 150 messages is a single
>> message, so is the smallest non-zero error rate you can possibly see).

When one measly msg more or less shows up as a 100% change, any conclusion
that can be drawn is as random as one measly message more or less.  Because
the error rates are so low for you already, you need more data before finer
distinctions can be drawn.

> ...
> The postmaster address gets a lot of bounced spam, and the
> python-list-admin address is currently being overwhelmed with warnings
> about delayed mail from some French site (328 of 511 messages).

Do you consider that to be ham or spam <wink>?  I agree it was a good idea
to leave this crud out of the mix.

> ...
> Two messages keep coming back as false positives.  One is an
> announcement from Palm (PDA's) and the other from the new CEO of
> United Airlines, os both messages are a much like UCE, except
> technically they weren't U.

Graham doesn't buy the UCE definition of spam (see "Appendix: Defining Spam"
in <http://www.paulgraham.com/spam.html>).  I don't care how people define
it, but it's clear that the classifier isn't going to learn that you *want*
Palm and United Airlines bulk marketing messages until you've used the
system long enough to train it on a significant number of instances of that
stuff.

> The false negatives are much more varied, and one of them wasn't even
> a negative but was misplaced in the spam folder.

All non-trivial systems have non-zero error rates of both kinds, and that
includes people.  I believe that on my huge training sets, the error rates
I'm seeing are better than I could do by hand.  For one thing, unlike me
running classifier tests, the classifier itself doesn't get bored to death
after doing this crap 20 hours straight <wink>.

> I don't really see anything weird in the best discriminators (the ones
> I looked at, that is).

Looking at the last 20 should nail this:  if there's any "funny stuff" going
on, it usually shows up in the top 10.  For example, if I tokenize Sender
headers,

    sender:email name:python-list-admin

becomes *the* most powerful indicator in the entire database (all my ham
comes from c.l.py, and none of my spam does; the Sender field is a 100%
accurate indicator in my data).


From tim.one@comcast.net  Tue Sep 17 04:04:41 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 16 Sep 2002 23:04:41 -0400
Subject: [Spambayes] pop3proxy.py
In-Reply-To: <w53znuh3itl.fsf@woozle.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEKCBEAB.tim.one@comcast.net>

If you were able to bribe Barry to give you checkin privileges, you should
feel free to use what you paid for <wink>.  It's CVS, and we can back out
changes if anyone gets mortally offended.  For that matter, Barry can revoke
your privileges if you don't keep paying him.

goes-double-for-me-ly y'rs  - tim


PS:  Read the Python style guide PEP:

    http://www.python.org/peps/pep-0008.html


You're in pretty good shape already, except that Guido will burn this code
alive if he sees the many repetitions of

    def handle_error( self ):

"""
Guido hates whitespace in the following places:

    - Immediately inside parentheses, brackets or braces, as in:
      "spam( ham[ 1 ], { eggs: 2 } )".  Always write this as
      "spam(ham[1], {eggs: 2})".
"""

PPS:  We haven't talking about licensing for this codebase.
      The SourceForge project page says it's under the PSF (Python
      Software Foundation) license, and that was certainly my
      intent.  Would you object to putting pop3proxy.py under the
      PSF license?  Assigning copyright to, or sharing copyright
      with, the PSF?


From guido@python.org  Tue Sep 17 04:16:53 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 16 Sep 2002 23:16:53 -0400
Subject: [Spambayes] pop3proxy.py
In-Reply-To: Your message of "Mon, 16 Sep 2002 23:04:41 EDT."
             <LNBBLJKPBEHFEDALKOLCCEKCBEAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCCEKCBEAB.tim.one@comcast.net> 
Message-ID: <200209170316.g8H3GrE01036@pcp02138704pcs.reston01.va.comcast.net>

> PPS:  We haven't talking about licensing for this codebase.
>       The SourceForge project page says it's under the PSF (Python
>       Software Foundation) license, and that was certainly my
>       intent.  Would you object to putting pop3proxy.py under the
>       PSF license?  Assigning copyright to, or sharing copyright
>       with, the PSF?

Yes please!  Separate copyright notices and licenses for individual
files are a major pain for any project.

I'm not the project admin, but if I were, I would set a strict rule
that all code committed to the project must use the PSF license unless
special dispensation is requested and obtained from the project
admins.  (This doesn't mean you can't be listed as the author of a
particular file.  But no separate licenses or copyright notices.)  The
PSF is supposed to have rules and forms for contributing code, but
they aren't official yet, pending legal review.  But here are the
drafts:

  http://www.python.org/psf/psf-contributor-agreement.html

--Guido van Rossum (home page: http://www.python.org/~guido/)

From neale@woozle.org  Tue Sep 17 05:49:25 2002
From: neale@woozle.org (Neale Pickett)
Date: 16 Sep 2002 21:49:25 -0700
Subject: [Spambayes] Need testers!
In-Reply-To: <LNBBLJKPBEHFEDALKOLCMEJPBEAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCMEJPBEAB.tim.one@comcast.net>
Message-ID: <w53admh2pbe.fsf@woozle.org>

So then, Tim Peters <tim.one@comcast.net> is all like:

> it would help if you did this all over again.  It was a pure and
> significant win for both error rates in this test report, but that may
> change after msgs are shuffled between the spam and ham sets.

On, man, I've been running tests repeatedly all day long.  I don't know
why it's so addictive.  This may supplant my frozen-bubble habit :)

The results of my most recent run are below.

> > If I did this all right, I think I'd like to check in a script to
> > chain all these actions together, so knuckleheads like me don't go
> > running useless tests again.
> 
> I agree that would be helpful (and, yes, it was all impeccably right).
> I have a collection of Windows .bat files to chain things together,
> but didn't want to insult you Unixoids by checking them in <wink>.

Mine's a bourne shell script.  I checked it in so you don't have to feel
bad about checking your batch files in now :)

Here's that report.  I've done seven just while composing this letter.
This one finally looks like the input data was clean.  Something I
noticed while looking over all the FPs and FNs was that there were some
message that continually got misclassified.  My feel is that about half
of the misclassifications are just corner cases that the classifier
didn't know enough about because my training sets are too small.
Anyway, here ya go.

run1s -> run2s
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams

false positive percentages
    2.000  2.500  lost   +25.00%
    0.500  1.500  lost  +200.00%
    3.000  3.500  lost   +16.67%
    1.500  0.500  won    -66.67%
    0.500  0.000  won   -100.00%

won   2 times
tied  0 times
lost  3 times

total unique fp went from 15 to 16 lost    +6.67%
mean fp % went from 1.5 to 1.6 lost    +6.67%

false negative percentages
    3.500  4.000  lost   +14.29%
    1.500  1.500  tied
    3.500  3.000  won    -14.29%
    1.500  1.000  won    -33.33%
    1.000  1.000  tied

won   2 times
tied  2 times
lost  1 times

total unique fn went from 22 to 21 won     -4.55%
mean fn % went from 2.2 to 2.1 won     -4.55%

From neale@woozle.org  Tue Sep 17 05:56:31 2002
From: neale@woozle.org (Neale Pickett)
Date: 16 Sep 2002 21:56:31 -0700
Subject: [Spambayes] What is spam?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCAEJLBEAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCAEJLBEAB.tim.one@comcast.net>
Message-ID: <w5365x52ozk.fsf@woozle.org>

So then, Tim Peters <tim.one@comcast.net> is all like:

> > With only one or two exceptions, that is the extent of my false
> > positives and false negatives.
> >
> > I have to wonder, though, if the forwards (#3) are really false
> > negatives.
> 
> False negative means that you put them in a spam folder but that the system
> said they were ham.  Is that what you meant to say?

Yeah.  In other words, grandma sends me a lot of stuff which looks, even
to me, to be a hand-written essay which was well thought-out by
*somebody*, even if it does happen to be totally false :)

The classifier is saying that most of these are ham, but I've got them
in the spam corpus.  So what I'm thinking is, maybe I'm expecting too
much of it (especially given my small datasets) to identify these things
as spam.

Given that, do you think it would be more useful to the tests if I
classified them as ham?  I realize that with enough training it'll start
to recognize "good times" and other urban legends, but I don't want to
screw up my tests just because I'm capable of discerning content and the
classifier isn't.  I mean, these messages are pretty obviously not in
the same category as real UCE.

Neale

From guido@python.org  Tue Sep 17 05:58:46 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 17 Sep 2002 00:58:46 -0400
Subject: [Spambayes] What is spam?
In-Reply-To: Your message of "Mon, 16 Sep 2002 21:56:31 PDT."
             <w5365x52ozk.fsf@woozle.org> 
References: <LNBBLJKPBEHFEDALKOLCAEJLBEAB.tim.one@comcast.net>  
            <w5365x52ozk.fsf@woozle.org> 
Message-ID: <200209170458.g8H4wkV04043@pcp02138704pcs.reston01.va.comcast.net>

> Given that, do you think it would be more useful to the tests if I
> classified them as ham?  I realize that with enough training it'll start
> to recognize "good times" and other urban legends, but I don't want to
> screw up my tests just because I'm capable of discerning content and the
> classifier isn't.  I mean, these messages are pretty obviously not in
> the same category as real UCE.

Anything for which somebody hit the Send button should be classified
as ham, no matter how objectionable the contents.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From neale@woozle.org  Tue Sep 17 06:06:35 2002
From: neale@woozle.org (Neale Pickett)
Date: 16 Sep 2002 22:06:35 -0700
Subject: [Spambayes] What is spam?
In-Reply-To: <20020917010926.GA8305@cthulhu.gerg.ca>
References: <w53it1538zp.fsf@woozle.org>
	<20020917010926.GA8305@cthulhu.gerg.ca>
Message-ID: <w531y7t2ois.fsf@woozle.org>

So then, Greg Ward <gward@python.net> is all like:

> For other MTAs, your mileage may vary.  But if you ask people to send
> you spam, you probably want to keep that mail from going close to your
> spam detector in the first place.

Hmm, that's a good point.  Since SA is mutable (the Auto whitelist), if
people send me their spam, it's going to screw things up a bit.  

Fortunately, that's only the case if that person *only* sends spam to
me.  Since I have SA set up to only add headers (not modify headers or
body), it doesn't do any real harm to the message, and it's actually
somewhat useful to see what SA thought of the message.

IIRC, you have a quarantine system set up, so spamalyzing messages has
more serious repercussions on your server; namely, way too much time
spent by you going over messages which everybody knows are spam :)

As an aside, you can look at my pretty spam graph here:

  http://woozle.org/stats/spam.html

You'll notice how the "unique rejected senders" and "messages tagged as
spam" lines start to become increasingly divergent about the time the
spambayes project began, when I stopped going over the honeypot every
day :)

(Hmm, I wonder if I could use spambayes to look over my honeypot's
mailbox and come up with my list of senders to reject!)



From neale@woozle.org  Tue Sep 17 06:10:16 2002
From: neale@woozle.org (Neale Pickett)
Date: 16 Sep 2002 22:10:16 -0700
Subject: [Spambayes] What is spam?
In-Reply-To: <200209170458.g8H4wkV04043@pcp02138704pcs.reston01.va.comcast.net>
References: <LNBBLJKPBEHFEDALKOLCAEJLBEAB.tim.one@comcast.net>
	<w5365x52ozk.fsf@woozle.org>
	<200209170458.g8H4wkV04043@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <w53wupl19s7.fsf@woozle.org>

So then, Guido van Rossum <guido@python.org> is all like:

> Anything for which somebody hit the Send button should be classified
> as ham, no matter how objectionable the contents.

Wow, is this the criteria other people are using?  That would radically
alter my corpa, but I suspect I'd get much better FP and FN rates.

From guido@python.org  Tue Sep 17 06:15:32 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 17 Sep 2002 01:15:32 -0400
Subject: [Spambayes] What is spam?
In-Reply-To: Your message of "Mon, 16 Sep 2002 22:10:16 PDT."
             <w53wupl19s7.fsf@woozle.org> 
References: <LNBBLJKPBEHFEDALKOLCAEJLBEAB.tim.one@comcast.net>
	<w5365x52ozk.fsf@woozle.org>
	<200209170458.g8H4wkV04043@pcp02138704pcs.reston01.va.comcast.net>  
	<w53wupl19s7.fsf@woozle.org> 
Message-ID: <200209170515.g8H5FWG04327@pcp02138704pcs.reston01.va.comcast.net>

> > Anything for which somebody hit the Send button should be classified
> > as ham, no matter how objectionable the contents.
> 
> Wow, is this the criteria other people are using?  That would radically
> alter my corpa, but I suspect I'd get much better FP and FN rates.

Paul Graham defines spam as *automated* unwanted email.  A forwarded
joke may be unwanted but it's definitely not automated.  A monthly
Amazon.com newsletter is automated, but technically wanted (since you
can check a box on the website to turn it off).  Neither should be
called spam.  If you have to err, calling the Amazon.com newsletter
spam is forgivable.  Calling your grandma's wit or wisdom spam is not.

--Guido van Rossum (home page: http://www.python.org/~guido/)


From neale@woozle.org  Tue Sep 17 06:42:32 2002
From: neale@woozle.org (Neale Pickett)
Date: 16 Sep 2002 22:42:32 -0700
Subject: [Spambayes] What is spam?
In-Reply-To: <200209170515.g8H5FWG04327@pcp02138704pcs.reston01.va.comcast.net>
References: <LNBBLJKPBEHFEDALKOLCAEJLBEAB.tim.one@comcast.net>
	<w5365x52ozk.fsf@woozle.org>
	<200209170458.g8H4wkV04043@pcp02138704pcs.reston01.va.comcast.net>
	<w53wupl19s7.fsf@woozle.org>
	<200209170515.g8H5FWG04327@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <w53ofax18af.fsf@woozle.org>

So then, Guido van Rossum <guido@python.org> is all like:

> Paul Graham defines spam as *automated* unwanted email.

No doubt.  Here's a run against a properly-sorted set of ham and spam,
then.  I don't imagine these variances are even worth looking at, since
0.5% is a single message.

run1s -> run2s
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams

false positive percentages
    1.500  2.000  lost   +33.33%
    2.500  1.000  won    -60.00%
    1.000  0.500  won    -50.00%
    0.500  0.500  tied
    0.000  0.000  tied

won   2 times
tied  2 times
lost  1 times

total unique fp went from 11 to 8 won    -27.27%
mean fp % went from 1.1 to 0.8 won    -27.27%

false negative percentages
    2.000  2.000  tied
    1.500  1.500  tied
    0.000  0.500  lost  +(was 0)
    1.000  1.000  tied
    1.500  1.500  tied

won   0 times
tied  4 times
lost  1 times

total unique fn went from 12 to 13 lost    +8.33%
mean fn % went from 1.2 to 1.3 lost    +8.33%

From guido@python.org  Tue Sep 17 06:53:12 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 17 Sep 2002 01:53:12 -0400
Subject: [Spambayes] What is spam?
In-Reply-To: Your message of "Mon, 16 Sep 2002 22:42:32 PDT."
             <w53ofax18af.fsf@woozle.org> 
References: <LNBBLJKPBEHFEDALKOLCAEJLBEAB.tim.one@comcast.net>
	<w5365x52ozk.fsf@woozle.org>
	<200209170458.g8H4wkV04043@pcp02138704pcs.reston01.va.comcast.net>
	<w53wupl19s7.fsf@woozle.org>
	<200209170515.g8H5FWG04327@pcp02138704pcs.reston01.va.comcast.net>  
	<w53ofax18af.fsf@woozle.org> 
Message-ID: <200209170553.g8H5rCv04872@pcp02138704pcs.reston01.va.comcast.net>

> X-Thought: The only "intuitive" interface is the nipple. After that,
>     it's all learned.
> X-Thought-Attribution: Bruce Ediger <bediger@teal.csn.org>,
>     in comp.os.linux.misc, on X interfaces

I object.  For the past 10 months I've observed first-hand that the
nipple response is not intuitive, it is a reflex.

Bruce has got his concepts mixed-up IMO: intuition is based on
learning!  However the learning is so subtle that it defies "why"
questions.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From richie@entrian.com  Tue Sep 17 12:51:41 2002
From: richie@entrian.com (Richie Hindle)
Date: Tue, 17 Sep 2002 12:51:41 +0100
Subject: [Spambayes] pop3proxy.py
In-Reply-To: <w53znuh3itl.fsf@woozle.org>
References: <o83boucjsrighptscos5h60sh6m0gc4jkj@4ax.com>
	<w53znuh3itl.fsf@woozle.org>
Message-ID: <464eougbmbqjg8qgj262jpr47v2g8r2voq@4ax.com>


[Neale]
> I hold no claim to the header.  I think as long as the header you add is
> formatted the same, you should use the same name.  If you format it
> differently, probably best to give it a different name.  Better yet, if
> you come up with a better format, check the change in to hammie.py
> too :)

I'd like to use the same header, but there's one problem: POP3 lets you
ask for the size of a message, so pop3proxy needs to add on the length of
the header before it's seen the message.  But X-Hammie-Disposition is
variable-length...  We could make the 'clues' section of the header
optional, so that pop3proxy adds "X-Hammie-Disposition: Yes; 0.99" or
"X-Hammie-Disposition:  No; 0.99" [note the extra space, yuck].  Is that
close enough to "formatted the same"?  I'd vote yes - "practicality beats
purity" 8-)

[Tim]
> Guido hates whitespace in the following places:

I'll make it obey the style guide, no problem.

[Tim]
> Would you object to putting pop3proxy.py under the
> PSF license?  Assigning copyright to, or sharing copyright with, the PSF?

Not at all - I only gave it a license to introduce a (theoretically)
enforceable disclaimer.  In the absence of a formal mechanism, you can
take this message as proof that I "transfer rights in and to the
Contribution [...] to the Python Software Foundation" as the draft
agreement puts it.  I'll remove my copyright and the MIT license the next
time I commit.

-- 
Richie Hindle
richie@entrian.com


From neale@woozle.org  Tue Sep 17 17:13:23 2002
From: neale@woozle.org (Neale Pickett)
Date: 17 Sep 2002 09:13:23 -0700
Subject: [Spambayes] pop3proxy.py
In-Reply-To: <464eougbmbqjg8qgj262jpr47v2g8r2voq@4ax.com>
References: <o83boucjsrighptscos5h60sh6m0gc4jkj@4ax.com>
	<w53znuh3itl.fsf@woozle.org>
	<464eougbmbqjg8qgj262jpr47v2g8r2voq@4ax.com>
Message-ID: <w53fzw81tng.fsf@woozle.org>

So then, Richie Hindle <richie@entrian.com> is all like:

> I'd like to use the same header, but there's one problem: POP3 lets you
> ask for the size of a message, so pop3proxy needs to add on the length of
> the header before it's seen the message.  But X-Hammie-Disposition is
> variable-length...  We could make the 'clues' section of the header
> optional, so that pop3proxy adds "X-Hammie-Disposition: Yes; 0.99" or
> "X-Hammie-Disposition:  No; 0.99" [note the extra space, yuck].  Is that
> close enough to "formatted the same"?  I'd vote yes - "practicality beats
> purity" 8-)

I reckon that's close enough.  I doubt any sorting rules will look past
the "Yes" or "No"; if someone needs that, they can use hammie :)
However, I would suggest using "No; 0.01 " instead of " No; 0.01", so
you can regex-match "^No" instead of "^ No".

^ Neale

From tim.one@comcast.net  Tue Sep 17 17:38:13 2002
From: tim.one@comcast.net (Tim Peters)
Date: Tue, 17 Sep 2002 12:38:13 -0400
Subject: [Spambayes] pop3proxy.py
In-Reply-To: <464eougbmbqjg8qgj262jpr47v2g8r2voq@4ax.com>
Message-ID: <BIEJKCLHCIOIHAGOKOLHIELFDKAA.tim.one@comcast.net>

[Richie Hindle]
> I'd like to use the same header, but there's one problem: POP3 lets you
> ask for the size of a message, so pop3proxy needs to add on the length of
> the header before it's seen the message.  But X-Hammie-Disposition is
> variable-length...  We could make the 'clues' section of the header
> optional, so that pop3proxy adds "X-Hammie-Disposition: Yes; 0.99" or
> "X-Hammie-Disposition:  No; 0.99" [note the extra space, yuck].  Is that
> close enough to "formatted the same"?  I'd vote yes - "practicality beats
> purity" 8-)

Note that the "probabilities" computed by this scheme are almost always very
near 1.0 or very near 0.0.  For that reason, I believe adding 0.99 (etc)
does no good and may mislead.  "spam" and "nyet" have the same # of letters
<wink>.


From tim.one@comcast.net  Tue Sep 17 21:11:28 2002
From: tim.one@comcast.net (Tim Peters)
Date: Tue, 17 Sep 2002 16:11:28 -0400
Subject: [Spambayes] What is spam?
In-Reply-To: <w53wupl19s7.fsf@woozle.org>
Message-ID: <BIEJKCLHCIOIHAGOKOLHGELNDKAA.tim.one@comcast.net>

[Guido]
> Anything for which somebody hit the Send button should be classified
> as ham, no matter how objectionable the contents.

[Neale Pickett]
> Wow, is this the criteria other people are using?  That would radically
> alter my corpa, but I suspect I'd get much better FP and FN rates.

It's the criterion I'm using because this started as a project to filter the
tech mailing lists hosted at python.org.  So, e.g., in my ham set I keep the
long msg that did nothing but add a useless one-line comment to a full quote
of a previous Nigerian scam post.  It looks like that will always be a
"false positive" in my tests, although it's surprising that it's the *only*
message of its kind that still routinely shows up as a false positive.  The
others of that kind were sent by more frequent posters, quote only part of
the spam in question, and try to add some real content (if only a sarcastic
comment).  They get points for all that.

OTOH, when I run on smaller training subsets, sometimes that message gets
into the ham training, and as a consequence direct Nigerian scam messages
become false negatives.  Because HAMBIAS is 2 by default, it takes about 2
Nigerian scam messages in the spam training just to "cancel out" the quoted
Nigerian scam in the ham training set -- and it takes a lot more than 2 to
overcome the 0.90 "it's spam" cutoff at the end.

There's another problem here, though, which is that real people aren't going
to let Paul Graham (or even Guido <wink>) tell them what spam is.  I always
thought it was hilarious that SpamCop felt it needed to lecture would-be
reporters at length about their definition:

    http://spamcop.net/fom-serve/cache/125.html

For my personal use, all I want is something that shuffles probable spam
into a different folder.  I don't care much about the f-p rate, because I'm
going to review every "probable spam" msg by hand anyway.  The value of any
spam gimmick to me is just in keeping the most likley spam out of my main
inbox folder, so that the spam-deletion ritual doesn't interfere so much
with my normal workflow.  So for my own case I care more about reducing the
f-n rate, and consequently I would indeed file my sisters' forwarding of
junk email in my spam collection, and your grandmother's too.

BTW, this is another reason I suspect it's not going to work well to pool
users:  Greg has almost no tolerance for spam review, and I appear to be on
the opposite end of that scale.  No fixed policy can leave both of us happy,
and the more training data you have the harder it is to get any numbers
other than 0.0 and 1.0 out of this scheme.


From robertoschler@hotmail.com  Tue Sep 17 23:49:52 2002
From: robertoschler@hotmail.com (Robert Oschler)
Date: Tue, 17 Sep 2002 18:49:52 -0400
Subject: [Spambayes] Slashdot article
Message-ID: <OE35PrvCWO6hm8Ua8d900000fab@hotmail.com>

This is a multi-part message in MIME format.

---------------------- multipart/related attachment

------=_NextPart_001_0008_01C25E7B.023B8A40

More on Bayesian Spam FilteringPosted by michael on Tuesday September =
17, @03:26PM
from the snake-eyes dept.
michaeld writes "The "Bayesian" techniques for spam filtering recently =
publicized in Paul Graham's essay A Plan for Spam doesn't actually seem =
to have anything Bayesian about it, according to Gary Robinson (an =
expert on collaborative filtering). It is based on a non-Bayesian =
probabilistic approach. It works well enough, because it is frequently =
the case that technology doesn't have to be 100% perfect in order to do =
something that really needs to be done. The problem interested Robinson, =
and he posted his thoughts about trying to fix the problems in the =
Graham approach, including adding an actual Bayesian element to the =
calculations."=20

www.slashdot.org

------=_NextPart_001_0008_01C25E7B.023B8A40
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail-21/spambayes/attachments/20020917/4abc17a1/attachment.html

------=_NextPart_001_0008_01C25E7B.023B8A40--

---------------------- multipart/related attachment
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 2319 bytes
Desc: not available
Url : http://mail.python.org/pipermail-21/spambayes/attachments/20020917/4abc17a1/attachment.gif

---------------------- multipart/related attachment--

From crunch@shopip.com  Tue Sep 17 23:30:24 2002
From: crunch@shopip.com (John Draper)
Date: Tue, 17 Sep 2002 15:30:24 -0700
Subject: [Spambayes] An interesting thing I saw on Slashdot...
Message-ID: <v031107b1b9ad5d398d7d@[192.168.0.2]>

http://radio.weblogs.com/0101454/stories/2002/09/16/spamDetection.html
Check out Gary Robinson's Rants you guys...  he claims to have some good ideas.  Perhaps someone might invite him on the list.

John



From tim.one@comcast.net  Tue Sep 17 23:58:06 2002
From: tim.one@comcast.net (Tim Peters)
Date: Tue, 17 Sep 2002 18:58:06 -0400
Subject: [Spambayes] Critique of Graham's math
In-Reply-To: <LNBBLJKPBEHFEDALKOLCOEKABEAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEMKBEAB.tim.one@comcast.net>

<http://radio.weblogs.com/0101454/stories/2002/09/16/spamDetection.html>

appears to be a reasonable critique of Paul Graham's math.  I griped about
some of these things on Python-Dev before the results got so good that I
couldn't care much anymore <0.5 wink>.  Note that we've already removed most
of the biases in Graham's original formulation, but HAMBIAS still does too
much good to get rid of, and I'm still wrestling with trying to give more
weight to indicators with more evidence to back them up.

Gary Robinson has other ideas worth testing, although he's mistaken about
"It is also worth noting that this reduces to Paul's original approach if
you set a=0":  it's another peculiarity of Graham's scheme that
update_probabilities() counts the number of times a word appears in a msg,
but that spamprob() only looks at "no times, or at least once".  Robinson's
claim would be true if update_probabilities() (like spamprob()) also limited
itself to a pure "it's there or it's not there" distinction.

Heh:  I see that Graham's own writeup has quietly changed what we call
UNKNOWN_SPAMPROB from 0.2 to 0.4.  I'm leaving it at our 0.5 <wink>.


From grobinson@transpose.com  Wed Sep 18 00:14:10 2002
From: grobinson@transpose.com (Gary Robinson)
Date: Tue, 17 Sep 2002 19:14:10 -0400
Subject: [Spambayes] Some ideas
Message-ID: <B9AD2F82.16038%grobinson@transpose.com>

Hi,

John Draper pointed this list out to me today after seeing that I was
slashdotted... in case anyone here finds it of interest, I have some
comments on Bayesian spam filtering here. In particular I've discussed some
potential improvements to Paul Graham's approach:

http://radio.weblogs.com/0101454/stories/2002/09/16/spamDetection.html

Best,
Gary


-- 
Gary Robinson
CEO
Transpose, LLC
grobinson@transpose.com
207-942-3463
http://www.emergentmusic.com
http://radio.weblogs.com/0101454



From tim.one@comcast.net  Wed Sep 18 00:34:41 2002
From: tim.one@comcast.net (Tim Peters)
Date: Tue, 17 Sep 2002 19:34:41 -0400
Subject: [Spambayes] Paul Graham's math
Message-ID: <LNBBLJKPBEHFEDALKOLCKEMMBEAB.tim.one@comcast.net>

Hi!  I enjoyed your article, and am running a part-time project that can
test some of this:

    http://mail.python.org/mailman-21/listinfo/spambayes

is the mailing list, and a code project of the same name is on SourceForge.

The results of previous testing caused us to remove the biases in Graham's
scheme one by one.  We're still left with artificially boosting the count of
words in non-spam by a factor of 2, as all attempts to get rid of that have
increased the false negative rate significantly, and especially on smaller
test data sets.

One correction:

    It is also worth noting that this reduces to Paul's original
    approach if you set a=0.

The difficulty is that Graham's scheme isn't symmetric here either:  his
original estimates of P(spam | word) count word duplications.  For example,
if "Nigeria" appears in a single spam 10 times, 10 is added to the spam
count for Nigeria.  This is why his Lisp snippets have expressions like

    (min 1 (/ b nbad))

For words like "the", the counts routinely exceed the total number of
messages, so this gimmick is needed to keep each ratio no larger than 1.0.

On the other hand, when Graham is scoring a message, he ignores word
duplications (each word in the message being scored is counted at most
once).  This isn't entirely clear from his article, although it's implied by
his examples.  He's confirmed this in email.

I used to worry <wink> a lot more about the math until I got enough data to
do serious testing.  Running a 10-fold cross validation on a collection of
20,000 non-spam and 13,750 spam gives a mean false positive rate of 0.02%,
and a mean false negative rate of 0.20%.  There were a grand total of 4
false positives and 28 false negatives across the 10 runs.  I think you'll
agree these are spectacularly low error rates!  One of the false positives
consisted of a one-line comment attached to a quote of an entire Nigerian
scam spam.

If I remove the factor-of-2 bias for non-spam words, it makes 11 mistakes in
each direction.  But false positives are hated more than false negatives in
this application, so *some* bias toward "not spam" is desirable.

A practical concern remaining is that this is a huge testing corpus compared
to what most people can reasonably be expected to have, and the results
aren't as good, and show higher variance, on smaller corpora.  We haven't
tried to quantify that yet, it's simply obvious <wink>.

I look forward to testing your ideas!  Feel free to participate here.  None
of us has ever posted on slashdot <wink>.


From tim.one@comcast.net  Wed Sep 18 00:51:50 2002
From: tim.one@comcast.net (Tim Peters)
Date: Tue, 17 Sep 2002 19:51:50 -0400
Subject: [Spambayes] Paul Graham's math
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEMMBEAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCMEMNBEAB.tim.one@comcast.net>

[Tim]
> ...
> Running a 10-fold cross validation on a collection of 20,000 non-spam
> and 13,750 spam gives a mean false positive rate of 0.02%, and a mean
> false negative rate of 0.20%.  There were a grand total of 4 false
> positives and 28 false negatives across the 10 runs.

Sorry to follow up to my own post, but I think you'd find a histogram of the
score distributions revealing.  There are the final "probabilities"
multiplied by 100, and broken into bins 2.5 wide:

Ham distribution for all runs:
* = 334 items
  0.00 19992 ************************************************************
  2.50     2 *
  5.00     1 *
  7.50     0
 10.00     0
 12.50     0
 15.00     0
 17.50     0
 20.00     0
 22.50     0
 25.00     0
 27.50     0
 30.00     0
 32.50     0
 35.00     0
 37.50     0
 40.00     0
 42.50     0
 45.00     0
 47.50     0
 50.00     0
 52.50     0
 55.00     0
 57.50     0
 60.00     0
 62.50     0
 65.00     0
 67.50     0
 70.00     0
 72.50     0
 75.00     0
 77.50     0
 80.00     0
 82.50     0
 85.00     0
 87.50     1 *
 90.00     0
 92.50     0
 95.00     0
 97.50     4 *

Spam distribution for all runs:
* = 229 items
  0.00    25 *
  2.50     1 *
  5.00     1 *
  7.50     0
 10.00     0
 12.50     0
 15.00     0
 17.50     1 *
 20.00     0
 22.50     0
 25.00     0
 27.50     0
 30.00     0
 32.50     0
 35.00     0
 37.50     0
 40.00     0
 42.50     0
 45.00     0
 47.50     0
 50.00     0
 52.50     0
 55.00     0
 57.50     0
 60.00     0
 62.50     0
 65.00     0
 67.50     0
 70.00     0
 72.50     0
 75.00     0
 77.50     0
 80.00     0
 82.50     0
 85.00     0
 87.50     0
 90.00     2 *
 92.50     1 *
 95.00    11 *
 97.50 13708 ************************************************************

So, e.g., we could have reduced the "spam cutoff" from 0.90 to 0.075 and
gotten only 1 more false positive.  The separation is extremely sharp.  I
should note that we've already made several changes to Graham's scheme, of
course (mostly eliminating biases).


From tim.one@comcast.net  Wed Sep 18 02:08:28 2002
From: tim.one@comcast.net (Tim Peters)
Date: Tue, 17 Sep 2002 21:08:28 -0400
Subject: [Spambayes] Need testers!
In-Reply-To: <LNBBLJKPBEHFEDALKOLCMEJPBEAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCCENDBEAB.tim.one@comcast.net>

Following my own advice (read TESTING.txt if you haven't yet <wink>), I'm
giving up on this one and will rip the supporting code out of the project
(adjust_probs_by_evidence_mass will go away):

> [Classifier]
> adjust_probs_by_evidence_mass: True
> min_spamprob: 0.001
> max_spamprob: 0.999
> hambias: 1.5

The reports from the people who have tested this (for which I thank you!)
simply don't show a consistent and significant improvement.  Jeremy put the
last nail in its coffin today, by running a reasonable but ham-heavy 10-fold
c-v test where it was a pure loss for his f-n rates.  This is a conflated
result because he's also tokenizing more header lines (which I can't do,
because doing so is "too good" on my mixed-source corpora), which is
something we all want to do eventually.  But even without that, the case for
making this change wasn't strong enough.

Thanks for the help!


From crunch@shopip.com  Wed Sep 18 00:59:37 2002
From: crunch@shopip.com (John Draper)
Date: Tue, 17 Sep 2002 16:59:37 -0700
Subject: [Spambayes] This list just got slashdotted today.
Message-ID: <v031107b7b9ad71704d4e@[192.168.0.2]>

I have a feeling there will be s BIG influx if people joining this list,  according to an article on Slashdot.  http://slashdot.org/articles/02/09/17/1746248.shtml?tid=111

Some fairly knowledgeable people have responded....   some may come here on this list.

Several have independently implemented Graham's system.  Getting good results.
Actually,  Gary Robinson was mentioned in the slashdot forum.

John



From nas@python.ca  Wed Sep 18 05:19:40 2002
From: nas@python.ca (Neil Schemenauer)
Date: Tue, 17 Sep 2002 21:19:40 -0700
Subject: [Spambayes] Getting rid of max_spamprob and min_spamprob
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEEKBEAB.tim.one@comcast.net>
References: <20020916012811.GA16631@glacier.arctrix.com>
	<LNBBLJKPBEHFEDALKOLCKEEKBEAB.tim.one@comcast.net>
Message-ID: <20020918041939.GA23161@glacier.arctrix.com>

Tim Peters wrote:
> Quick idea:  this kind of thing is why a classic Bayesian classifier works
> in log space.

The idea of working with logs had crossed my mind but I didn't pursue
it.  I've now changed my code.  Much nicer.  Thanks for the nudge.

> To avoid massive cancellation at the end, it's probably numerically
> better to do
> 
>     sum(for i = 1 to n, log(h_i/s_i) + log(S/H))


How about calculating and storing log(h_i/s_i * S/H) with the rest of the
word info?

> It's certainly open to experiment!

I've played with it for about an hour tonight.  I can't achieve the
accuracy that the CVS code is getting.  Here's the latest result:

    total unique false pos 49
    total unique false neg 1
    average fp % 2.72222222222
    average fn % 0.0555555555555

That one false negative is pretty odd.¹  The false positives look like
reasonable mistakes.

  Neil

¹ http://arctrix.com/nas/bizarre_fn.txt

From tim.one@comcast.net  Wed Sep 18 05:28:47 2002
From: tim.one@comcast.net (Tim Peters)
Date: Wed, 18 Sep 2002 00:28:47 -0400
Subject: [Spambayes] First result from Gary Robinson's ideas
In-Reply-To: <B9AD2F82.16038%grobinson@transpose.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKENMBEAB.tim.one@comcast.net>

Sorry that I have to be telegraphic here -- no time to write up full
details, or even check in the code in a sane way for others to try it=
, and
I'll likely not get back to this at all until Thursday night.

This result comes from implementing just the first suggestion in

<http://radio.weblogs.com/0101454/stories/2002/09/16/spamDetection.ht=
ml>

If you want to try it (and I sure hope someone does on a smaller test
corpus!), you need to change two places:

1. Tester.Test.predict:  change 0.90 to 0.50.  Somebody make that an
   .ini-file option?

2. classifier.GrahamBayes.spamprob:  Replace the final probability
   calculation with this:

        P =3D Q =3D 1.0
        num_clues =3D 0
        for distance, prob, word, record in nbest:
            if prob is None:  # it's one of the dummies nbest started=
 with
                continue
            if record is not None:  # else wordinfo doesn't know abou=
t it
                record.killcount +=3D 1
            if evidence:
                clues.append((word, prob))
            num_clues +=3D 1
            P *=3D 1.0 - prob
            Q *=3D prob

        if num_clues:
            P =3D 1.0 - P**(1./num_clues)
            Q =3D 1.0 - Q**(1./num_clues)
            prob =3D (P-Q)/(P+Q)  # in -1 .. 1
            prob =3D 0.5 + prob/2 # shift to 0 .. 1
        else:
            prob =3D 0.5

A 10-fold cross-validation run against "my usual" monster corpus show=
s
almost no difference in results, but this isn't the interesting part =
of the
story <wink>:

"""
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 sp=
ams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 sp=
ams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 sp=
ams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 sp=
ams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 sp=
ams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 sp=
ams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 sp=
ams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 sp=
ams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 sp=
ams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 sp=
ams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 sp=
ams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 sp=
ams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 sp=
ams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 sp=
ams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 sp=
ams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 sp=
ams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 sp=
ams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 sp=
ams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 sp=
ams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 sp=
ams

false positive percentages
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.050  0.050  tied
    0.000  0.050  lost  +(was 0)
    0.000  0.000  tied
    0.050  0.050  tied
    0.000  0.000  tied
    0.100  0.100  tied

won   0 times
tied  9 times
lost  1 times

total unique fp went from 4 to 5 lost   +25.00%
mean fp % went from 0.02 to 0.025 lost   +25.00%

false negative percentages
    0.218  0.218  tied
    0.364  0.364  tied
    0.000  0.000  tied
    0.218  0.218  tied
    0.218  0.218  tied
    0.291  0.291  tied
    0.218  0.218  tied
    0.145  0.145  tied
    0.291  0.291  tied
    0.073  0.073  tied

won   0 times
tied 10 times
lost  0 times

total unique fn went from 28 to 28 tied
mean fn % went from 0.203636363636 to 0.203636363636 tied
"""

The *interesting* part of this story is the score histograms:

Ham distribution for all runs:
* =3D 311 items
  0.00 18608 ********************************************************=
****
  2.50   301 *
  5.00   110 *
  7.50    54 *
 10.00    78 *
 12.50   177 *
 15.00   163 *
 17.50   140 *
 20.00    93 *
 22.50    53 *
 25.00    66 *
 27.50    44 *
 30.00    35 *
 32.50    22 *
 35.00    18 *
 37.50    12 *
 40.00    10 *
 42.50     6 *
 45.00     3 *
 47.50     2 *
 50.00     1 *
 52.50     0
 55.00     0
 57.50     0
 60.00     2 *
 62.50     0
 65.00     1 *
 67.50     0
 70.00     0
 72.50     0
 75.00     0
 77.50     0
 80.00     0
 82.50     0
 85.00     0
 87.50     0
 90.00     0
 92.50     0
 95.00     0
 97.50     1 *

Spam distribution for all runs:
* =3D 211 items
  0.00     2 *
  2.50     0
  5.00     0
  7.50     0
 10.00     0
 12.50     0
 15.00     0
 17.50     0
 20.00     0
 22.50     0
 25.00     1 *
 27.50     2 *
 30.00     1 *
 32.50     0
 35.00     1 *
 37.50     4 *
 40.00     3 *
 42.50     4 *
 45.00     6 *
 47.50     4 *
 50.00     4 *
 52.50     9 *
 55.00    11 *
 57.50    18 *
 60.00    42 *
 62.50    26 *
 65.00    41 *
 67.50    46 *
 70.00    55 *
 72.50    54 *
 75.00    59 *
 77.50    65 *
 80.00   160 *
 82.50   158 *
 85.00    61 *
 87.50    20 *
 90.00    11 *
 92.50    45 *
 95.00   233 **
 97.50 12604 ********************************************************=
****

This is much more spread out than when using Graham's combining formu=
la, and
has a useful "middle ground":  manual review of msgs with spamprob in=
 [0.4,
0.6] would stop 1 of the false positives and 17 (of the 28 total) fal=
se
negatives.  There are also no cases where false positives or negative=
s have
insane "probabilities" like 1e-30 or 1.0000000000.  The sole very-hig=
h
scoring false positive has prob 0.989806241076, and is the fellow who=
 added
one comment to a quote of an entire Nigerian scam.

There are two very low-scoring false negatives.  One is the "Hello, m=
y Name
is BlackIntrepid" spam mentioned a few days ago, which simply has no =
words
that even score above 0.05 for spamness (its prob is 0.0173583933026 =
now).
The other is an extremely long base64-encoded spam that suffers
"cancellation disease" (Gary, because Graham clamps the word probs to=
 lie
within [0.01, 0.99], sometimes we get messages with hundreds of each;=
 we
changed his algorithm to do better than flipping a coin <0.5 wink> wh=
en this
happens, but it remains a challenge; your suggestion for normalizing
probabilities would stop this, although I won't know whether that wor=
ks
better or worse than what we've got now until I can make time to code=
 that
and test it).

Anyway, on my large corpus this looks very much worth pursuing, as th=
e
results are as good but the numbers it produces "feel" much more like=
 actual
probabilities.

The one new false positive that snuck into this is a revival of an ol=
d
friend, and it just barely scored over 0.50:

*********************************************************************=
***
Data/Ham/Set6/24252.txt
prob =3D 0.517172469699
prob('url:rpm') =3D 0.01
prob('header:Organization:1') =3D 0.01
prob('url:fi') =3D 0.01
prob('url:linux') =3D 0.0107383
prob('header:Errors-To:1') =3D 0.0200348
prob('header:Message-ID:1') =3D 0.364341
prob('x-mailer:none') =3D 0.388829
prob('header:Date:1') =3D 0.471899
prob('header:To:1') =3D 0.489291
prob('header:Subject:1') =3D 0.495598
prob('header:From:1') =3D 0.496521
prob('url:phtml') =3D 0.523918
prob('url:www') =3D 0.55991
prob('url:com') =3D 0.654736
prob('url:net') =3D 0.752554
prob('url:html') =3D 0.785737
prob('url:links') =3D 0.897196
prob('url:es') =3D 0.951256
prob('url:index') =3D 0.95655
prob('url:pid') =3D 0.99
prob('url:1110') =3D 0.99
prob('url:id') =3D 0.99

=46rom: "agc" <agc@redestb.es>
Newsgroups: comp.lang.python
Subject: enlaces
Date: Thu, 17 Feb 2000 17:36:18 +0100
Organization: Iddeo - Retevisi=F3n
Lines: 5
Message-ID: <88h8gg$1vnl84@SGI3651ef0.iddeo.es>
NNTP-Posting-Host: 62.82.233.22
X-Priority: 3
X-MSMail-Priority: Normal
X-Newsreader: Microsoft Outlook Express 5.00.2014.211
X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2014.211
Path:
news!ffx.uu.net!uunet!ams.uu.net!do.de.uu.net!newsfeed01.sul.t-online=
.de!new
sfeed00.sul.t-online.de!t-online.de!bignews.mediaways.net!newsfeed.ne=
ttuno.i
t!server-b.cs.interbusiness.it!news-1.retevision.es!news.iddeo.es!not=
-for-ma
il
Xref: news comp.lang.python:84746
To: python-list@python.org
Sender: python-list-admin@python.org
Errors-To: python-list-admin@python.org
X-BeenThere: python-list@python.org
X-Mailman-Version: 1.2 (experimental)
Precedence: bulk
List-Id: General discussion list for the Python programming language
        <python-list.python.org>

http://www.noguska.net/linux/rpm/rpm-HOWTO.html
http://www.ceu.fi.udc.es/GPUL/links/
http://www.demasiado.com/index.phtml?id=3D1110&pid=3Dinform=E1tica
*********************************************************************=
***

Note that there's so little to go on here that it's reduced to taking=
 the
mere presence of a Subject header as "a clue".



From tim.one@comcast.net  Wed Sep 18 05:47:42 2002
From: tim.one@comcast.net (Tim Peters)
Date: Wed, 18 Sep 2002 00:47:42 -0400
Subject: [Spambayes] This list just got slashdotted today.
In-Reply-To: <v031107b7b9ad71704d4e@[192.168.0.2]>
Message-ID: <LNBBLJKPBEHFEDALKOLCGENOBEAB.tim.one@comcast.net>

[John Draper]
> I have a feeling there will be s BIG influx if people joining
> this list,  according to an article on Slashdot.
> http://slashdot.org/articles/02/09/17/1746248.shtml?tid=111
>
> Some fairly knowledgeable people have responded....   some may
> come here on this list.

I'm happy to see knowledgeable ones; I'd be even happier to see anal
detail-oriented testers with no life to distract them <0.9 wink>.

> Several have independently implemented Graham's system.  Getting
> good results.

Many people have implemented Graham's scheme.  Our README.txt points out
what's been different about this project:

    The focus of this project so far has not been to produce the fastest
    or smallest filters, but to set up a flexible pure-Python
    implementation for doing algorithm research.  Lots of people are
    making fast/small implementations, and it takes an entirely different
    kind of effort to make genuine algorithm improvements.  I think we've
    done quite well at that so far.

And we did better still after I wrote that.  I have cautious hopes that
Gary's ideas will make it even better, but that's another way in which this
project has been different:  I'm hardline about not accepting ideas that
don't prove themselves by testing.  Indeed, I've probably insulted half the
people already on this list by dismissing ideas with a "then do the work to
test it or go away" snipe.  As a result, we've been getting some spectacular
results (albeit in both directions <ahem>).


From popiel@wolfskeep.com  Wed Sep 18 18:00:02 2002
From: popiel@wolfskeep.com (T. Alexander Popiel)
Date: Wed, 18 Sep 2002 10:00:02 -0700
Subject: [Spambayes] Another optimization
Message-ID: <20020918170002.6059EF49B@cashew.wolfskeep.com>

Hello, folks.

Some weeks ago I independently implemented Graham's filter
algorithm, based on his essay on the topic.  Since my
implementation is in perl (please, no flames), I doubt
my code will be all that helpful... and anyway, I just
sort of hacked it together.  If you still want to see it
after that disclaimer, it's available at
http://wolfskeep.com/mailfilter/ .

About a week and a half ago, I got a reference to your
collective work here; I scanned through your code gleaning
ideas.  I figured I should return the favor. ;-)

There are a few things I was wondering about, though:

* Graham was very specific in describing his tokenizer...
  and you folks seem to have ignored that description.
  Instead, you're using split-on-whitespace augmented by
  a few handcrafted hacks for URLs, addresses, and the like.
  This puzzles me, since I seem to get better results using
  the tokenization that Graham suggested.

* Why do you throw out super-long words?  Yes, they
  potentially increase the database size, but I find that
  lines of dashes of various lengths to be particularly
  good spam and non-spam indicators.  Also, some of the
  worms seem to be caught by recognizing their uuencoded
  forms.  Since I'm not storing _all_ words encountered
  (I'm only keeping the probability computations after
  ignoring the few-occurences words) in the database,
  keeping the long words doesn't seem to use much space.
  And disk space is cheap, right?

* One key feature of Graham's algorithm is that it only
  considers the N most significant words in a message;
  however, in your cancellation of .01 and .99 words, you
  seem to throw away the the most significant words in
  favor of less significant words.  (Yes, I understand
  that the math for cancellation works out assuming the
  same set of less significant words is considered...
  but by throwing out the matching words, I think your
  code then considers _more_ of the less significant words.)

  In my code, instead of cancelling out the .01 and .99
  words, I inflate the considered words count to include
  all of these 'maximally significant' words that appear
  in the message, even if that number is >N.  However, I
  do _not_ consider additional less significant words if
  the number of maximally significant words is >N.  I have
  found this to be _much_ more effective; it lowered the
  FN rate while leaving the FP rate constant, whereas
  cancelling and considering more of the less significant
  words increased both FN and FP for me.

  If I managed to misread your code, I apologize for the
  mischaracterization.

My testing is not as rigorous as yours is; in particular,
I don't retrain using several different sets of training
data in the course of evaluating my changes.  As a result,
my experience may be biased. :-/

- Alex

From skip@pobox.com  Wed Sep 18 18:00:55 2002
From: skip@pobox.com (Skip Montanaro)
Date: Wed, 18 Sep 2002 12:00:55 -0500
Subject: [Spambayes] we'll win them over yet...
Message-ID: <15752.45383.84504.165493@12-248-11-90.client.attbi.com>

On the SA-Talk, Justin Mason wrote:


    Skip Montanaro said:

    > I've seen several recent references in SA-talk to "S/O" and "S/O
    > Ratio"....  All I can think of is "Spam to Ollie ratio".  What is it
    > really?

    Spam-to-Overall Ratio: a Dan Q innovation! ;) it's the ratio of spam
    hits to all hits in a given corpus, ie.

                         nspam
        s/o ratio =  --------------
                     (nspam + nham)

    (Excuse my use of the "ham" terminology, but I like it ;)

Maybe we can't get them to use a different language, but we'll get them to
use our terminology. <wink>

Skip

(Taking a several day break from spambayes to try and get some paying work
done...)


From gward@python.net  Wed Sep 18 18:25:07 2002
From: gward@python.net (Greg Ward)
Date: Wed, 18 Sep 2002 13:25:07 -0400
Subject: [Spambayes] Another optimization
In-Reply-To: <20020918170002.6059EF49B@cashew.wolfskeep.com>
References: <20020918170002.6059EF49B@cashew.wolfskeep.com>
Message-ID: <20020918172507.GA16930@cthulhu.gerg.ca>

On 18 September 2002, T. Alexander Popiel said:
> * Graham was very specific in describing his tokenizer...
>   and you folks seem to have ignored that description.
>   Instead, you're using split-on-whitespace augmented by
>   a few handcrafted hacks for URLs, addresses, and the like.
>   This puzzles me, since I seem to get better results using
>   the tokenization that Graham suggested.

If you haven't read the archive for this list yet, do so!  If you're
really keen, check the python-dev archive for the week or so before this
list was created -- there was a fair amount of discussion there.

Anyways, in the early days of this project, Tim Peters experimented a
lot with various tokenization schemes.  The current scheme is the one
that did the best on his corpus.  I suspect that suggested tweaks to the
tokenization algorithm will only be entertained if you back them up with
solid experimental evidence that they improve things.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
I'm a lumberjack and I'm OK / I sleep all night and I work all day

From randy.diffenderfer@eds.com  Wed Sep 18 18:34:07 2002
From: randy.diffenderfer@eds.com (Diffenderfer, Randy)
Date: Wed, 18 Sep 2002 13:34:07 -0400
Subject: [Spambayes] Slashdot mention of MS patent
Message-ID: <8AA870658244D4119AF600508BDF0A360C6BC174@usahm014.exmi01.exch.eds.com>

>From a slightly more than casual read of the patent referenced in the
slashdot series mentioning the "spambayes" technique, it would appear MS is
stomping around in this space.  Anybody have a more learned/legal opinion?

US Patent 6,161,130 -- 12/12/2000

From neale@woozle.org  Wed Sep 18 18:43:32 2002
From: neale@woozle.org (Neale Pickett)
Date: 18 Sep 2002 10:43:32 -0700
Subject: [Spambayes] First result from Gary Robinson's ideas
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKENMBEAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCKENMBEAB.tim.one@comcast.net>
Message-ID: <w537khj19dn.fsf@woozle.org>

So then, Tim Peters <tim.one@comcast.net> is all like:

> If you want to try it (and I sure hope someone does on a smaller test
> corpus!), you need to change two places:

Hey, that's me for sure!

Here ya go:

"""
run1s -> run2s
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams

false positive percentages
    2.500  0.000  won   -100.00%
    2.500  0.000  won   -100.00%
    2.000  0.000  won   -100.00%
    2.000  0.000  won   -100.00%
    1.000  0.000  won   -100.00%

won   5 times
tied  0 times
lost  0 times

total unique fp went from 20 to 0 won   -100.00%
mean fp % went from 2.0 to 0.0 won   -100.00%

false negative percentages
    1.500  100.000  lost  +6566.67%
    1.000  100.000  lost  +9900.00%
    1.500  100.000  lost  +6566.67%
    1.000  100.000  lost  +9900.00%
    1.000  100.000  lost  +9900.00%

won   0 times
tied  0 times
lost  5 times

total unique fn went from 12 to 1000 lost  +8233.33%
mean fn % went from 1.2 to 100.0 lost  +8233.33%
"""

Those are spectacular false negative percentages!  But that's what I
should expect with this change, right?

Here's the breakdown:

"""
Ham distribution for all runs:
* = 17 items
  0.00 977 **********************************************************
  2.50   1 *
  5.00   0 
  7.50   0 
 10.00   0 
 12.50   0 
 15.00   0 
 17.50   0 
 20.00   0 
 22.50   0 
 25.00   0 
 27.50   0 
 30.00   0 
 32.50   0 
 35.00   0 
 37.50   0 
 40.00   0 
 42.50   0 
 45.00   0 
 47.50   0 
 50.00   0 
 52.50   0 
 55.00   0 
 57.50   0 
 60.00   0 
 62.50   0 
 65.00   0 
 67.50   0 
 70.00   0 
 72.50   0 
 75.00   0 
 77.50   1 *
 80.00   0 
 82.50   0 
 85.00   1 *
 87.50   0 
 90.00   1 *
 92.50   0 
 95.00   1 *
 97.50  18 **

Spam distribution for all runs:
* = 17 items
  0.00   8 *
  2.50   0 
  5.00   0 
  7.50   1 *
 10.00   1 *
 12.50   0 
 15.00   0 
 17.50   0 
 20.00   0 
 22.50   0 
 25.00   0 
 27.50   0 
 30.00   0 
 32.50   0 
 35.00   0 
 37.50   0 
 40.00   0 
 42.50   0 
 45.00   1 *
 47.50   0 
 50.00   1 *
 52.50   0 
 55.00   0 
 57.50   0 
 60.00   0 
 62.50   0 
 65.00   0 
 67.50   0 
 70.00   0 
 72.50   0 
 75.00   0 
 77.50   0 
 80.00   0 
 82.50   0 
 85.00   0 
 87.50   0 
 90.00   0 
 92.50   0 
 95.00   0 
 97.50 988 ***********************************************************
"""

Still pretty black and white, but the gray area does appear to have a
few more shades.

HTH

Neale

From grobinson@transpose.com  Wed Sep 18 18:49:02 2002
From: grobinson@transpose.com (Gary Robinson)
Date: Wed, 18 Sep 2002 13:49:02 -0400
Subject: [Spambayes] Re: First result from Gary Robinson's ideas
In-Reply-To: <LNBBLJKPBEHFEDALKOLCIEPKBEAB.tim.one@comcast.net>
Message-ID: <B9AE34CE.160FF%grobinson@transpose.com>

> The clearest difference:
> suppose Nigeria appears in 50 ham and 50 spam.  You'll give it a spamprob
> approximately equal to 1/2 (fiddled a little by a and a*b).  But I haven't
> given enough information there to say what Graham will give it!  In his
> scheme, it also depends on the total number of hams and of spams.  Say there
> are 100 spams and 200 hams.  Then Graham effectively says "OK, it appears in
> half of the spams, but only a quarter of the hams, so if I see Nigeria it's
> twice as likely to be a spam, so the spamprob is really 2/3"
...
> I haven't tested this difference, but I *expect* this one can't be glossed
> over.  Most people doing testing appear to have a lot more ham than spam to
> work with, and I expect that Graham's scheme of computing probs "as if" the
> numbers were equal (via comparing population ratios instead of raw counts)
> is really helping them.

Hmm. That's a good point. I'm pretty sure the key to it is to use the
Bayesian calcs, but TWICE.

That is:

First, look only at the spams. Let h represent the number of "hits", that
is, the number of spams containing the word "Nigeria", and let n be the
total number of spams.

Then the Bayesian probability of the next spam to be looked at containing
Nigeria is 

a + h
-------
ab + n

where, again, a/b is the default assumed probability in the absence of data
and a is the weight of the assumption. (Again, in strict terms this is a
binomial probability based on a beta distribution prior.)

If we let a=0, it's the same as Paul (ignoring the little tricks like
multiplying by 2); h is 50 and n is 100 and the result is .5.

Now do the same thing on the ham side, and you get .25.

Then do the same combining trick to get 2/3.

Now, the advantage of using the Bayesian trick of having a and b is for the
cases where a word hasn't been seen before or has only been seen rarely
before. In such cases, you don't really know enough about it to make
reliable extrapolations about the probabilities. So, a and b serve to handle
that uncertain situation. As I said in the original essay, Paul handles it
manually by inserting .4 in the case of no earlier data, but that does
nothing for the cases where there is little data. If I understand you
correctly, you use .5. That is equivalent to a=1 and b=2, but the Bayesian
calc is much more general.

This is all a little bit weird because it's giving equal weight to the data
that comes from the spam and non-spam cases, which there is no real
justification for.

But I understand Tim's point in the following:

> Also, when it comes to deploying this, I doubt it
> will be possible to get people to keep training the system with a number of
> ham and spam in correct proportion to the real-life ratio; they'll tend to
> feed all their spam into their training data and skip adding any more ham;
> so it I expect it will really help if the system *doesn't* assume that the
> ratio of spam to ham it's trained on is any sort of estimate of the
> real-life ratio.

So if we assume that everything has been classified, which is an assumption
behind my suggestion, we end up with the distortion Tim describes above.

In that way, Paul's trick seems to be more realistic.

HOWEVER I think that rather than give the two cases equal weight, I think we
ought to find out what weighting leads to the best performance and use that
weight. Using equal weight seems like a good compromise during this
development period, and then we can tune it.

That's my $.02 anyway! :)


--Gary


-- 
Gary Robinson
CEO
Transpose, LLC
grobinson@transpose.com
207-942-3463
http://www.emergentmusic.com
http://radio.weblogs.com/0101454


> From: Tim Peters <tim.one@comcast.net>
> Date: Wed, 18 Sep 2002 11:51:17 -0400
> To: Gary Robinson <grobinson@transpose.com>
> Subject: RE: First result from Gary Robinson's ideas
> 
>> Thanks for doing this testing, I really appreciate it!!!!
> 
> Not as much as I appreciate your ideas <wink>.
> 
>> Very interesting about how the spread could be used to look for things to
>> check manually.
> 
> Not something I would have thought of; the people currently using
> SpamAssassin keep bringing this up.  People running large mail hubs are
> willing to put *some* time into manual review, so it's great if a scheme has
> a useful and relatively small "middle ground".
> 
>> Should I assume that, in general, I can post your emails to my
>> site and link to them from my essay when they help to illuminae the
> issues?
> 
> No problem.  It would be better for both of us, though, if you replied to
> the spambayes mailing list.  That's a small group of bright people, and
> they'll have helpful things to say too, and then neither of us (especially
> me <wink>) will have to spend more time repackaging and regurgitating what
> we say here.
> 
>> One thing... I also received test results this morning from Raymond Chen,
>> who found that my technique works much better when S is .71 than
>> when it is .5. If I am reading his results correctly, on Raymond's corpus
>> mine had about 60% fewer classification errors than Graham's.
> 
> We already differ from Graham's original scheme in many details, and each of
> a few dozen changes made significant improvement in at least one of the
> error rates.  For example, special tagging of embedded URLs cut the false
> negative rate in half, and boosting the "unknown word" probability from 0.2
> to 0.5 cut it by another third (BTW, Graham's writeup *currently* says he
> uses 0.4 for this; when we started, it said 0.2).  And we've removed all but
> one of Graham's deliberate "let's favor ham" biases (we set the unknown word
> prob to 0.5, and got rid of the gimmick that assigned the unknown-word prob
> to words for which 2*hamcount + spamcount isn't at least 5).  If Chen hasn't
> yet been driven to similar changes, his baseline performance is almost
> certainly much worse than ours, and I bet his baseline score histograms are
> much fuzzier too.
> 
> I expect it's also likely that he's using smaller data sets, and perhaps not
> running a testing procedure as careful as N-fold cross validation
> (especially for smaller datasets, I think that last is crucial:  the
> variance across runs has sometimes been high in smaller-dataset tests;  I've
> seen other projects adopt or reject changes based on a single run on a
> single corpus, and that's just a recipe for random thrashing).
> 
>> Raymond doesn't explicitly say so, but it looks like he only implemented
> the
>> first part of my ideas.
> 
> The second part turns out to have several differences from what Graham does,
> beyond just whether to count word duplications.  The clearest difference:
> suppose Nigeria appears in 50 ham and 50 spam.  You'll give it a spamprob
> approximately equal to 1/2 (fiddled a little by a and a*b).  But I haven't
> given enough information there to say what Graham will give it!  In his
> scheme, it also depends on the total number of hams and of spams.  Say there
> are 100 spams and 200 hams.  Then Graham effectively says "OK, it appears in
> half of the spams, but only a quarter of the hams, so if I see Nigeria it's
> twice as likely to be a spam, so the spamprob is really 2/3" (note that I'm
> ignoring the complication introduced here by his multiplying the hamcounts
> by 2 -- in this specific example, he'll actually give Nigeria a spamprob of
> 1/2 too, but only because he pretends Nigeria actually appeared in 2*50 =
> 100 spam, so that the fiddled ratios 50/100 and 2*50/200 are equal).
> 
> I haven't tested this difference, but I *expect* this one can't be glossed
> over.  Most people doing testing appear to have a lot more ham than spam to
> work with, and I expect that Graham's scheme of computing probs "as if" the
> numbers were equal (via comparing population ratios instead of raw counts)
> is really helping them.  Also, when it comes to deploying this, I doubt it
> will be possible to get people to keep training the system with a number of
> ham and spam in correct proportion to the real-life ratio; they'll tend to
> feed all their spam into their training data and skip adding any more ham;
> so it I expect it will really help if the system *doesn't* assume that the
> ratio of spam to ham it's trained on is any sort of estimate of the
> real-life ratio.
> 


From grobinson@transpose.com  Wed Sep 18 19:02:15 2002
From: grobinson@transpose.com (Gary Robinson)
Date: Wed, 18 Sep 2002 14:02:15 -0400
Subject: [Spambayes] Re: First result from Gary Robinson's ideas
In-Reply-To: <B9AE34CE.160FF%grobinson@transpose.com>
Message-ID: <B9AE37E7.1610B%grobinson@transpose.com>

Responding to my own post, sorry. I think I used the "unknown word" examples
of .4 and .5 in a screwed-up way below so please ignore those comments.

Also, I think that a and b would be overwhelmed by the real-world data if
the is doing a lot of classifying, so there may really be no need to bother
with them at all; they'd only be useful in the situation where there were a
very small number of emails being classified.

Hmmmm. I want to think about this a bit more.



--Gary


-- 
Gary Robinson
CEO
Transpose, LLC
grobinson@transpose.com
207-942-3463
http://www.emergentmusic.com
http://radio.weblogs.com/0101454


> From: Gary Robinson <grobinson@transpose.com>
> Date: Wed, 18 Sep 2002 13:49:02 -0400
> To: Tim Peters <tim.one@comcast.net>, SpamBayes <spambayes@python.org>
> Subject: Re: First result from Gary Robinson's ideas
> 
>> The clearest difference:
>> suppose Nigeria appears in 50 ham and 50 spam.  You'll give it a spamprob
>> approximately equal to 1/2 (fiddled a little by a and a*b).  But I haven't
>> given enough information there to say what Graham will give it!  In his
>> scheme, it also depends on the total number of hams and of spams.  Say there
>> are 100 spams and 200 hams.  Then Graham effectively says "OK, it appears in
>> half of the spams, but only a quarter of the hams, so if I see Nigeria it's
>> twice as likely to be a spam, so the spamprob is really 2/3"
> ...
>> I haven't tested this difference, but I *expect* this one can't be glossed
>> over.  Most people doing testing appear to have a lot more ham than spam to
>> work with, and I expect that Graham's scheme of computing probs "as if" the
>> numbers were equal (via comparing population ratios instead of raw counts)
>> is really helping them.
> 
> Hmm. That's a good point. I'm pretty sure the key to it is to use the Bayesian
> calcs, but TWICE.
> 
> That is:
> 
> First, look only at the spams. Let h represent the number of "hits", that is,
> the number of spams containing the word "Nigeria", and let n be the total
> number of spams.
> 
> Then the Bayesian probability of the next spam to be looked at containing
> Nigeria is 
> 
> a + h
> -------
> ab + n
> 
> where, again, a/b is the default assumed probability in the absence of data
> and a is the weight of the assumption. (Again, in strict terms this is a
> binomial probability based on a beta distribution prior.)
> 
> If we let a=0, it's the same as Paul (ignoring the little tricks like
> multiplying by 2); h is 50 and n is 100 and the result is .5.
> 
> Now do the same thing on the ham side, and you get .25.
> 
> Then do the same combining trick to get 2/3.
> 
> Now, the advantage of using the Bayesian trick of having a and b is for the
> cases where a word hasn't been seen before or has only been seen rarely
> before. In such cases, you don't really know enough about it to make reliable
> extrapolations about the probabilities. So, a and b serve to handle that
> uncertain situation. As I said in the original essay, Paul handles it manually
> by inserting .4 in the case of no earlier data, but that does nothing for the
> cases where there is little data. If I understand you correctly, you use .5.
> That is equivalent to a=1 and b=2, but the Bayesian calc is much more general.
> 
> This is all a little bit weird because it's giving equal weight to the data
> that comes from the spam and non-spam cases, which there is no real
> justification for.
> 
> But I understand Tim's point in the following:
> 
>> Also, when it comes to deploying this, I doubt it
>> will be possible to get people to keep training the system with a number of
>> ham and spam in correct proportion to the real-life ratio; they'll tend to
>> feed all their spam into their training data and skip adding any more ham;
>> so it I expect it will really help if the system *doesn't* assume that the
>> ratio of spam to ham it's trained on is any sort of estimate of the
>> real-life ratio.
> 
> So if we assume that everything has been classified, which is an assumption
> behind my suggestion, we end up with the distortion Tim describes above.
> 
> In that way, Paul's trick seems to be more realistic.
> 
> HOWEVER I think that rather than give the two cases equal weight, I think we
> ought to find out what weighting leads to the best performance and use that
> weight. Using equal weight seems like a good compromise during this
> development period, and then we can tune it.
> 
> That's my $.02 anyway! :)
> 
> 
> --Gary
> 
> 
> -- 
> Gary Robinson
> CEO
> Transpose, LLC
> grobinson@transpose.com
> 207-942-3463
> http://www.emergentmusic.com
> http://radio.weblogs.com/0101454
> 
> 
>> From: Tim Peters <tim.one@comcast.net>
>> Date: Wed, 18 Sep 2002 11:51:17 -0400
>> To: Gary Robinson <grobinson@transpose.com>
>> Subject: RE: First result from Gary Robinson's ideas
>> 
>>> Thanks for doing this testing, I really appreciate it!!!!
>> 
>> Not as much as I appreciate your ideas <wink>.
>> 
>>> Very interesting about how the spread could be used to look for things to
>>> check manually.
>> 
>> Not something I would have thought of; the people currently using
>> SpamAssassin keep bringing this up.  People running large mail hubs are
>> willing to put *some* time into manual review, so it's great if a scheme has
>> a useful and relatively small "middle ground".
>> 
>>> Should I assume that, in general, I can post your emails to my
>>> site and link to them from my essay when they help to illuminae the
>> issues?
>> 
>> No problem.  It would be better for both of us, though, if you replied to
>> the spambayes mailing list.  That's a small group of bright people, and
>> they'll have helpful things to say too, and then neither of us (especially
>> me <wink>) will have to spend more time repackaging and regurgitating what
>> we say here.
>> 
>>> One thing... I also received test results this morning from Raymond Chen,
>>> who found that my technique works much better when S is .71 than
>>> when it is .5. If I am reading his results correctly, on Raymond's corpus
>>> mine had about 60% fewer classification errors than Graham's.
>> 
>> We already differ from Graham's original scheme in many details, and each of
>> a few dozen changes made significant improvement in at least one of the
>> error rates.  For example, special tagging of embedded URLs cut the false
>> negative rate in half, and boosting the "unknown word" probability from 0.2
>> to 0.5 cut it by another third (BTW, Graham's writeup *currently* says he
>> uses 0.4 for this; when we started, it said 0.2).  And we've removed all but
>> one of Graham's deliberate "let's favor ham" biases (we set the unknown word
>> prob to 0.5, and got rid of the gimmick that assigned the unknown-word prob
>> to words for which 2*hamcount + spamcount isn't at least 5).  If Chen hasn't
>> yet been driven to similar changes, his baseline performance is almost
>> certainly much worse than ours, and I bet his baseline score histograms are
>> much fuzzier too.
>> 
>> I expect it's also likely that he's using smaller data sets, and perhaps not
>> running a testing procedure as careful as N-fold cross validation
>> (especially for smaller datasets, I think that last is crucial:  the
>> variance across runs has sometimes been high in smaller-dataset tests;  I've
>> seen other projects adopt or reject changes based on a single run on a
>> single corpus, and that's just a recipe for random thrashing).
>> 
>>> Raymond doesn't explicitly say so, but it looks like he only implemented
>> the
>>> first part of my ideas.
>> 
>> The second part turns out to have several differences from what Graham does,
>> beyond just whether to count word duplications.  The clearest difference:
>> suppose Nigeria appears in 50 ham and 50 spam.  You'll give it a spamprob
>> approximately equal to 1/2 (fiddled a little by a and a*b).  But I haven't
>> given enough information there to say what Graham will give it!  In his
>> scheme, it also depends on the total number of hams and of spams.  Say there
>> are 100 spams and 200 hams.  Then Graham effectively says "OK, it appears in
>> half of the spams, but only a quarter of the hams, so if I see Nigeria it's
>> twice as likely to be a spam, so the spamprob is really 2/3" (note that I'm
>> ignoring the complication introduced here by his multiplying the hamcounts
>> by 2 -- in this specific example, he'll actually give Nigeria a spamprob of
>> 1/2 too, but only because he pretends Nigeria actually appeared in 2*50 =
>> 100 spam, so that the fiddled ratios 50/100 and 2*50/200 are equal).
>> 
>> I haven't tested this difference, but I *expect* this one can't be glossed
>> over.  Most people doing testing appear to have a lot more ham than spam to
>> work with, and I expect that Graham's scheme of computing probs "as if" the
>> numbers were equal (via comparing population ratios instead of raw counts)
>> is really helping them.  Also, when it comes to deploying this, I doubt it
>> will be possible to get people to keep training the system with a number of
>> ham and spam in correct proportion to the real-life ratio; they'll tend to
>> feed all their spam into their training data and skip adding any more ham;
>> so it I expect it will really help if the system *doesn't* assume that the
>> ratio of spam to ham it's trained on is any sort of estimate of the
>> real-life ratio.
>> 


From tim.one@comcast.net  Wed Sep 18 19:08:14 2002
From: tim.one@comcast.net (Tim Peters)
Date: Wed, 18 Sep 2002 14:08:14 -0400
Subject: [Spambayes] Another optimization
In-Reply-To: <20020918170002.6059EF49B@cashew.wolfskeep.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCMEAFBFAB.tim.one@comcast.net>

[T. Alexander Popiel]
> Some weeks ago I independently implemented Graham's filter
> algorithm, based on his essay on the topic.  Since my
> implementation is in perl (please, no flames), I doubt
> my code will be all that helpful... and anyway, I just
> sort of hacked it together.  If you still want to see it
> after that disclaimer, it's available at
> http://wolfskeep.com/mailfilter/ .

Thanks!  I don't hate Perl, although sometimes Perl has hated me <wink>.

> About a week and a half ago, I got a reference to your
> collective work here; I scanned through your code gleaning
> ideas.  I figured I should return the favor. ;-)
>
> There are a few things I was wondering about, though:
>
> * Graham was very specific in describing his tokenizer...
>   and you folks seem to have ignored that description.
>   Instead, you're using split-on-whitespace augmented by
>   a few handcrafted hacks for URLs, addresses, and the like.
>   This puzzles me, since I seem to get better results using
>   the tokenization that Graham suggested.

The answer to every question is "because careful experiments said it worked
better".  This one is discussed in a long tokenizer.py comment block, with
title "How to tokenize?", and some full both-way test results are given
there.

> * Why do you throw out super-long words?

Likewise, but under the section "How big should 'a word' be?".

>   Yes, they potentially increase the database size, but I find that
>   lines of dashes of various lengths to be particularly
>   good spam and non-spam indicators.

We get some of that via the compressed 'skip' tokens we generate for
"super-long" words.

>   Also, some of the worms seem to be caught by recognizing their
>   uuencoded forms.

We throw out uuencoded sections entirely, except to specially tag the
permission mask and pieces of the filename.

>   Since I'm not storing _all_ words encountered (I'm only keeping the
>   probability computations after ignoring the few-occurences words)

(Perhaps not so) Strangely enough, my experiments showed that never ignoring
a word worked better.

>   in the database, keeping the long words doesn't seem to use much space.
>   And disk space is cheap, right?

Heh -- ask Neale here about that <wink>.

Do read our project's TESTING.txt:  I never argue with what the data tells
me, and "good arguments" don't count for beans compared to that -- an idea
genuinely reduces an error rate, or it doesn't, and only the ideas that do
survive.  But as TESTING.txt also warns, ideas aren't independent, and the
evaluation of a specific idea has to be done in the context all other ideas
implemented (for example, you very likely get some extra benefit out of
"super long" words *because* you don't toss uuencoded sections).  In theory,
every previous decision should be revisited from scratch whenever anything
changes.  I'm not married to anything here, but I simply will not make a
change unless testing says it's a winner.

> * One key feature of Graham's algorithm is that it only
>   considers the N most significant words in a message;
>   however, in your cancellation of .01 and .99 words, you
>   seem to throw away the the most significant words in
>   favor of less significant words.  (Yes, I understand
>   that the math for cancellation works out assuming the
>   same set of less significant words is considered...
>   but by throwing out the matching words, I think your
>   code then considers _more_ of the less significant words.)

We cancel out *equal numbers* of 0.01 and 0.99 words, and that's all.
Including them has no effect whatsoever on the computed result, *except*
that including them prevented other words from being considered in Graham's
original scheme.  It was a measurable improvement to do what we do here, but
IMO the scheme remains a crock when dozens of .01 and .99 clues both appear,

>   In my code, instead of cancelling out the .01 and .99
>   words, I inflate the considered words count to include
>   all of these 'maximally significant' words that appear
>   in the message, even if that number is >N.

If you threw out equal numbers of cancelling clues then, it would make no
difference to the result you compute in the end.  Each .99 & .01 pair
contributes a factor of .99*.01 to Graham's numerator, and also to each term
in Graham's denominator:  they cancel out exactly.

>   However, I do _not_ consider additional less significant words if
>   the number of maximally significant words is >N.

And we do, *if* the number of non-cancelling maxprob words is less than N.
The fact is that the scheme is almost totally lost in these cases
regardless, and Graham's combining rule is such that just a few .01`or .99
clues not cancelled pretty much determines the final outcome.  One of my
most stubborn false negatives has over a hundred .01 clues to a couple dozen
.99 clues -- it doesn't matter what we do to combine those, it's always
going to be a false negative without a more fundamental change elsewhere.
This "only look at smoking guns" scoring scheme seems to be systematically
weak when dealing with long and chatty "just folks" spam (of which there is
blessedly little so far <wink>).

>   I have found this to be _much_ more effective; it lowered the
>   FN rate while leaving the FP rate constant, whereas
>   cancelling and considering more of the less significant
>   words increased both FN and FP for me.

I don't expect I'll be able to figure out *exactly* what you do here without
studying your code.  If someone wants to run an experiment on that, cool,
the data will tell us.  At the moment, though, the error rates on my 30,000+
message tests have such small absolute value that I can no longer measure an
improvement when one is made; I can only detect regressions reliably now.

>   If I managed to misread your code, I apologize for the
>   mischaracterization.
>
> My testing is not as rigorous as yours is; in particular,
> I don't retrain using several different sets of training
> data in the course of evaluating my changes.  As a result,
> my experience may be biased. :-/

Well, it's a statistical approach, and on a single run there are non-zero
chances of a bad idea appearing to win, and of a good idea appearing to
lose.  The odds of making such a mistake decreases as the amount of data
used increases, but the chances of drawing wrong conclusions remain non-zero
regardless.  Running multiple tests over multiple data sets is another way
to sanity-check ideas.  The examples of test runs in tokenizer.py show
several cases where, e.g., a specific idea *appeared* to be a win in 3 runs,
but was actually a loss in 17 others.  More common is that an idea wins in 8
runs, loses in 8, and ties on 2.  You really need multiple test runs to have
confidence you're making progress.


From grobinson@transpose.com  Wed Sep 18 19:23:36 2002
From: grobinson@transpose.com (Gary Robinson)
Date: Wed, 18 Sep 2002 14:23:36 -0400
Subject: [Spambayes] Re: First result from Gary Robinson's ideas
In-Reply-To: <B9AE37E7.1610B%grobinson@transpose.com>
Message-ID: <B9AE3CE8.16113%grobinson@transpose.com>

OK, continuing to reply to myself... here's what I think is a decent
solution.

There are definitely advantages of your way of calculating the word
probabilities compared to the one in my essay, as Tim pointed out.

However there is still reason to use a Bayesian approach LIKE what I pointed
to, in order to handle the "unknown word" case which Paul handles with .4
and you handle with .5 in a way that also deals with the "almost unknown
word" case of just having a very small amount of data for the word.

Let p(w) be the probability for the word as currently calculated (with NO
calculation done for the case of an "unknown word" -- this handles that).
Let n be the number of emails that contain word w that went into the
calculation.

Then let

        a + (n * p(w))
f(w) = ---------------
        (a * b) + n

b = 1/avg(p(w)) across all words (maybe a weighted average that factors in
the relative frequency of the words, not sure about this right now), or
which can simply be tuned by hand, and

a is 1 to start with but is tuned for performance.

Note that unknown words have n=0, so the probability is a/(a*b) which may be
.4, .5 or whatever works best. For example, we could choose a=1 and b=2.5 or
2.0, to arrive at .4 or .5, respectively.

Note that f(w) is not justified by looking it as a binomial distribution
with a beta prior. Actually, I have a way of justifying it based on a
multinomial distribution with a Dirichlet prior that I won't go into here.
Maybe there are other ways of justifying it.

Rigorous justification from a Bayesian POV or not, I am willing to assert
that it makes sense in this context.

Unless I've made another dumb-ass error. ;)



--Gary


-- 
Gary Robinson
CEO
Transpose, LLC
grobinson@transpose.com
207-942-3463
http://www.emergentmusic.com
http://radio.weblogs.com/0101454


> From: Gary Robinson <grobinson@transpose.com>
> Date: Wed, 18 Sep 2002 14:02:15 -0400
> To: Tim Peters <tim.one@comcast.net>, SpamBayes <spambayes@python.org>
> Subject: Re: First result from Gary Robinson's ideas
> 
> Responding to my own post, sorry. I think I used the "unknown word" examples
> of .4 and .5 in a screwed-up way below so please ignore those comments.
> 
> Also, I think that a and b would be overwhelmed by the real-world data if the
> is doing a lot of classifying, so there may really be no need to bother with
> them at all; they'd only be useful in the situation where there were a very
> small number of emails being classified.
> 
> Hmmmm. I want to think about this a bit more.
> 
> 
> 
> --Gary
> 
> 
> -- 
> Gary Robinson
> CEO
> Transpose, LLC
> grobinson@transpose.com
> 207-942-3463
> http://www.emergentmusic.com
> http://radio.weblogs.com/0101454
> 
> 
>> From: Gary Robinson <grobinson@transpose.com>
>> Date: Wed, 18 Sep 2002 13:49:02 -0400
>> To: Tim Peters <tim.one@comcast.net>, SpamBayes <spambayes@python.org>
>> Subject: Re: First result from Gary Robinson's ideas
>> 
>>> The clearest difference:
>>> suppose Nigeria appears in 50 ham and 50 spam.  You'll give it a spamprob
>>> approximately equal to 1/2 (fiddled a little by a and a*b).  But I haven't
>>> given enough information there to say what Graham will give it!  In his
>>> scheme, it also depends on the total number of hams and of spams.  Say there
>>> are 100 spams and 200 hams.  Then Graham effectively says "OK, it appears in
>>> half of the spams, but only a quarter of the hams, so if I see Nigeria it's
>>> twice as likely to be a spam, so the spamprob is really 2/3"
>> ...
>>> I haven't tested this difference, but I *expect* this one can't be glossed
>>> over.  Most people doing testing appear to have a lot more ham than spam to
>>> work with, and I expect that Graham's scheme of computing probs "as if" the
>>> numbers were equal (via comparing population ratios instead of raw counts)
>>> is really helping them.
>> 
>> Hmm. That's a good point. I'm pretty sure the key to it is to use the
>> Bayesian 
>> calcs, but TWICE.
>> 
>> That is:
>> 
>> First, look only at the spams. Let h represent the number of "hits", that is,
>> the number of spams containing the word "Nigeria", and let n be the total
>> number of spams.
>> 
>> Then the Bayesian probability of the next spam to be looked at containing
>> Nigeria is 
>> 
>> a + h
>> -------
>> ab + n
>> 
>> where, again, a/b is the default assumed probability in the absence of data
>> and a is the weight of the assumption. (Again, in strict terms this is a
>> binomial probability based on a beta distribution prior.)
>> 
>> If we let a=0, it's the same as Paul (ignoring the little tricks like
>> multiplying by 2); h is 50 and n is 100 and the result is .5.
>> 
>> Now do the same thing on the ham side, and you get .25.
>> 
>> Then do the same combining trick to get 2/3.
>> 
>> Now, the advantage of using the Bayesian trick of having a and b is for the
>> cases where a word hasn't been seen before or has only been seen rarely
>> before. In such cases, you don't really know enough about it to make reliable
>> extrapolations about the probabilities. So, a and b serve to handle that
>> uncertain situation. As I said in the original essay, Paul handles it
>> manually 
>> by inserting .4 in the case of no earlier data, but that does nothing for the
>> cases where there is little data. If I understand you correctly, you use .5.
>> That is equivalent to a=1 and b=2, but the Bayesian calc is much more
>> general.
>> 
>> This is all a little bit weird because it's giving equal weight to the data
>> that comes from the spam and non-spam cases, which there is no real
>> justification for.
>> 
>> But I understand Tim's point in the following:
>> 
>>> Also, when it comes to deploying this, I doubt it
>>> will be possible to get people to keep training the system with a number of
>>> ham and spam in correct proportion to the real-life ratio; they'll tend to
>>> feed all their spam into their training data and skip adding any more ham;
>>> so it I expect it will really help if the system *doesn't* assume that the
>>> ratio of spam to ham it's trained on is any sort of estimate of the
>>> real-life ratio.
>> 
>> So if we assume that everything has been classified, which is an assumption
>> behind my suggestion, we end up with the distortion Tim describes above.
>> 
>> In that way, Paul's trick seems to be more realistic.
>> 
>> HOWEVER I think that rather than give the two cases equal weight, I think we
>> ought to find out what weighting leads to the best performance and use that
>> weight. Using equal weight seems like a good compromise during this
>> development period, and then we can tune it.
>> 
>> That's my $.02 anyway! :)
>> 
>> 
>> --Gary
>> 
>> 
>> -- 
>> Gary Robinson
>> CEO
>> Transpose, LLC
>> grobinson@transpose.com
>> 207-942-3463
>> http://www.emergentmusic.com
>> http://radio.weblogs.com/0101454
>> 
>> 
>>> From: Tim Peters <tim.one@comcast.net>
>>> Date: Wed, 18 Sep 2002 11:51:17 -0400
>>> To: Gary Robinson <grobinson@transpose.com>
>>> Subject: RE: First result from Gary Robinson's ideas
>>> 
>>>> Thanks for doing this testing, I really appreciate it!!!!
>>> 
>>> Not as much as I appreciate your ideas <wink>.
>>> 
>>>> Very interesting about how the spread could be used to look for things to
>>>> check manually.
>>> 
>>> Not something I would have thought of; the people currently using
>>> SpamAssassin keep bringing this up.  People running large mail hubs are
>>> willing to put *some* time into manual review, so it's great if a scheme has
>>> a useful and relatively small "middle ground".
>>> 
>>>> Should I assume that, in general, I can post your emails to my
>>>> site and link to them from my essay when they help to illuminae the
>>> issues?
>>> 
>>> No problem.  It would be better for both of us, though, if you replied to
>>> the spambayes mailing list.  That's a small group of bright people, and
>>> they'll have helpful things to say too, and then neither of us (especially
>>> me <wink>) will have to spend more time repackaging and regurgitating what
>>> we say here.
>>> 
>>>> One thing... I also received test results this morning from Raymond Chen,
>>>> who found that my technique works much better when S is .71 than
>>>> when it is .5. If I am reading his results correctly, on Raymond's corpus
>>>> mine had about 60% fewer classification errors than Graham's.
>>> 
>>> We already differ from Graham's original scheme in many details, and each of
>>> a few dozen changes made significant improvement in at least one of the
>>> error rates.  For example, special tagging of embedded URLs cut the false
>>> negative rate in half, and boosting the "unknown word" probability from 0.2
>>> to 0.5 cut it by another third (BTW, Graham's writeup *currently* says he
>>> uses 0.4 for this; when we started, it said 0.2).  And we've removed all but
>>> one of Graham's deliberate "let's favor ham" biases (we set the unknown word
>>> prob to 0.5, and got rid of the gimmick that assigned the unknown-word prob
>>> to words for which 2*hamcount + spamcount isn't at least 5).  If Chen hasn't
>>> yet been driven to similar changes, his baseline performance is almost
>>> certainly much worse than ours, and I bet his baseline score histograms are
>>> much fuzzier too.
>>> 
>>> I expect it's also likely that he's using smaller data sets, and perhaps not
>>> running a testing procedure as careful as N-fold cross validation
>>> (especially for smaller datasets, I think that last is crucial:  the
>>> variance across runs has sometimes been high in smaller-dataset tests;  I've
>>> seen other projects adopt or reject changes based on a single run on a
>>> single corpus, and that's just a recipe for random thrashing).
>>> 
>>>> Raymond doesn't explicitly say so, but it looks like he only implemented
>>> the
>>>> first part of my ideas.
>>> 
>>> The second part turns out to have several differences from what Graham does,
>>> beyond just whether to count word duplications.  The clearest difference:
>>> suppose Nigeria appears in 50 ham and 50 spam.  You'll give it a spamprob
>>> approximately equal to 1/2 (fiddled a little by a and a*b).  But I haven't
>>> given enough information there to say what Graham will give it!  In his
>>> scheme, it also depends on the total number of hams and of spams.  Say there
>>> are 100 spams and 200 hams.  Then Graham effectively says "OK, it appears in
>>> half of the spams, but only a quarter of the hams, so if I see Nigeria it's
>>> twice as likely to be a spam, so the spamprob is really 2/3" (note that I'm
>>> ignoring the complication introduced here by his multiplying the hamcounts
>>> by 2 -- in this specific example, he'll actually give Nigeria a spamprob of
>>> 1/2 too, but only because he pretends Nigeria actually appeared in 2*50 =
>>> 100 spam, so that the fiddled ratios 50/100 and 2*50/200 are equal).
>>> 
>>> I haven't tested this difference, but I *expect* this one can't be glossed
>>> over.  Most people doing testing appear to have a lot more ham than spam to
>>> work with, and I expect that Graham's scheme of computing probs "as if" the
>>> numbers were equal (via comparing population ratios instead of raw counts)
>>> is really helping them.  Also, when it comes to deploying this, I doubt it
>>> will be possible to get people to keep training the system with a number of
>>> ham and spam in correct proportion to the real-life ratio; they'll tend to
>>> feed all their spam into their training data and skip adding any more ham;
>>> so it I expect it will really help if the system *doesn't* assume that the
>>> ratio of spam to ham it's trained on is any sort of estimate of the
>>> real-life ratio.
>>> 


From popiel@wolfskeep.com  Wed Sep 18 19:32:02 2002
From: popiel@wolfskeep.com (T. Alexander Popiel)
Date: Wed, 18 Sep 2002 11:32:02 -0700
Subject: [Spambayes] Another optimization 
In-Reply-To: Message from Greg Ward <gward@python.net> 
	of "Wed, 18 Sep 2002 13:25:07 EDT."
	<20020918172507.GA16930@cthulhu.gerg.ca> 
References: <20020918170002.6059EF49B@cashew.wolfskeep.com>
	<20020918172507.GA16930@cthulhu.gerg.ca> 
Message-ID: <20020918183202.D5B82F49B@cashew.wolfskeep.com>

In message:  <20020918172507.GA16930@cthulhu.gerg.ca>
             Greg Ward <gward@python.net> writes:
>
>If you haven't read the archive for this list yet, do so!

Skimmed it before posting, but didn't see reference to the basic
split-on-whitespace choice.  I did not think to check on python-dev
for prior conversation, since I had no way of knowing it would be
there (as opposed to some other unrelated forum). ;-)

>If you're really keen, check the python-dev archive for the week or
>so before this list was created -- there was a fair amount of discussion
>there.

I'll check it out.

>Anyways, in the early days of this project, Tim Peters experimented a
>lot with various tokenization schemes.  The current scheme is the one
>that did the best on his corpus.  I suspect that suggested tweaks to the
>tokenization algorithm will only be entertained if you back them up with
>solid experimental evidence that they improve things.

Yeah... I could (I've got a decent size (1-month) corpus of data
without the bias of trashing much of my non-spam stream, and a 5-year
corpus of stuff with that bias (I was keeping all spam, various mailing
lists, and all mails from various 'important' people, but trashing a
lot of the incidentals))... but since I'm not using the python
implmentation, the effort wouldn't buy me much, and I don't really care
that much.  It's also good to have variant implementation schemes out
in the wild... it gives the spammers more contradictory targets to
fail to tune to properly. ;-)

Unfortunately, my corpora have enough private data in them that
I'm not really willing to give them to the world.  Sorry.

I was just surprised to see such a basic choice made without any
apparent thought or discussion (unlike much of the token cracking
which is backed up with solid data in the source code comments).
Also, on another mailing list I'm on, one of the other people who
also implemented the filter did it with s-o-w instead of Graham's
tokenizer, and was getting results worse than mine by an order
of magnitude... but we never compared our corpora in any rigorous
way, so the difference could have come from variations in the
classifiability of our particular corpora.

- Alex

From grobinson@transpose.com  Wed Sep 18 19:35:38 2002
From: grobinson@transpose.com (Gary Robinson)
Date: Wed, 18 Sep 2002 14:35:38 -0400
Subject: [Spambayes] sorry...
Message-ID: <B9AE3FBA.16116%grobinson@transpose.com>

...for posting entire threads in every one of my messages to this list. I
will refrain from doing that in the future.


--Gary



From popiel@wolfskeep.com  Wed Sep 18 21:37:19 2002
From: popiel@wolfskeep.com (T. Alexander Popiel)
Date: Wed, 18 Sep 2002 13:37:19 -0700
Subject: [Spambayes] Another optimization 
In-Reply-To: Message from Tim Peters <tim.one@comcast.net> 
	of "Wed, 18 Sep 2002 14:08:14 EDT."
	<LNBBLJKPBEHFEDALKOLCMEAFBFAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCMEAFBFAB.tim.one@comcast.net> 
Message-ID: <20020918203719.2742CF49B@cashew.wolfskeep.com>

In message:  <LNBBLJKPBEHFEDALKOLCMEAFBFAB.tim.one@comcast.net>
             Tim Peters <tim.one@comcast.net> writes:
>>
>> * Graham was very specific in describing his tokenizer...
>>   and you folks seem to have ignored that description.
>
>The answer to every question is "because careful experiments said it worked
>better".  This one is discussed in a long tokenizer.py comment block, with
>title "How to tokenize?", and some full both-way test results are given
>there.

Argh.  I just saw that comment block, nestled between n-grams
and HTML discussions.  Somehow I managed to overlook it on my
prior readings.  Sorry.

>> * Why do you throw out super-long words?
>
>Likewise, but under the section "How big should 'a word' be?".

Yeah, but that section only seems to refer to deciding the cutoff
length between 12 and 13 characters, not whether there should be a
cutoff at all.  Hrm.

>>   Since I'm not storing _all_ words encountered (I'm only keeping the
>>   probability computations after ignoring the few-occurences words)
>
>(Perhaps not so) Strangely enough, my experiments showed that never ignoring
>a word worked better.

That's not something I thought to check.  Perhaps I'll try it.

>Do read our project's TESTING.txt:

*nod*  Good stuff.

>>   However, I do _not_ consider additional less significant words if
>>   the number of maximally significant words is >N.
>
>And we do, *if* the number of non-cancelling maxprob words is less than N.

Yeah, that's the distinction I thought I saw.  I tested it both ways
(though not as rigorously as you do) and chose my way.

>The fact is that the scheme is almost totally lost in these cases
>regardless, and Graham's combining rule is such that just a few .01`or .99
>clues not cancelled pretty much determines the final outcome.

The cases that decided me had about 20-30 maximal clues, with about
5 or 6 unpaired .99s.  Allowing the less significant clues (ranging
from .02 to .08 with only a couple .96-.98s) flipped them from 'spam'
to 'ham', increasing the FN rate annoyingly.

Of course, this may be an artifact of my corpora size... those .02-.08
clues may get promoted to .01s with more training data, completely
obviating the difference in our approaches.

>One of my most stubborn false negatives has over a hundred .01 clues
>to a couple dozen .99 clues -- it doesn't matter what we do to combine
>those, it's always going to be a false negative without a more fundamental
>change elsewhere.

Yes.

>This "only look at smoking guns" scoring scheme seems to be systematically
>weak when dealing with long and chatty "just folks" spam (of which there is
>blessedly little so far <wink>).

Indeed.  I'm at a loss for how to combat it, though, unless we
artificially restrict the number of .01s we get through limited
corpora.

>Well, it's a statistical approach, [...]
>You really need multiple test runs to have confidence you're
>making progress.

Yeah, I really ought to be more anal about it. ;-)

- Alex

From grobinson@transpose.com  Wed Sep 18 23:03:15 2002
From: grobinson@transpose.com (Gary Robinson)
Date: Wed, 18 Sep 2002 18:03:15 -0400
Subject: [Spambayes] Mozilla and summary of bayes calcs
Message-ID: <B9AE7063.16172%grobinson@transpose.com>

Did you guys know they are looking at integrating a spam filter into
Mozilla? 

http://bugzilla.mozilla.org/show_bug.cgi?id=163188

They are looking at using the Graham approach.

To summarize my current thinking in one place, since my earlier thoughts
spanned several emails, the Graham approach implicitly does
TWO Bayesian calculations and then combines them. But the priors (expressed
by my a and b) are so overwhelmed by the actual data, that there is no point
in bothering with the priors at all. Nevertheless, they are really the same
calc as the Bayesian based on a beta distribution prior, just recognizing
that there is no point in bothering to have anything but zeroes as the beta
distribution parameters because of the amount of data.

Then those two calculations are combined.

That gives Paul's current "probabilities" for each word.

But after that step, we CAN take the number of occurrences of the word into
account (some words may only have so few occurrences that we can't really
say much about them -- or NO occurrences) with yet another Bayesian
calculation. 

This calc looks much like the one from my original essay, based on the beta
prior but actually needs a different justification, and I have a pretty
abstract one based on a Dirichlet prior although there may be other ways to
justify it.

To repeat, it's

        a + (n * p(w))
f(w) = ---------------
        (a * b) + n

where n is the number of occurrences of word w and p(w) is the current
graham-based calc.

You already account for the n=0 case by manually substituting .5, but this
accounts for low-data situations like n=1 and n=2 as well, when we don't
really have enough info to calculate a realistic probability. Obviously
we're forced to do something when n is 0 because there is no data; but in
reality we should do something when n is any low number -- 0 is just the
most extreme case. This is a smooth way of handling these low-n situations.


--Gary


-- 
Gary Robinson
CEO
Transpose, LLC
grobinson@transpose.com
207-942-3463
http://www.emergentmusic.com
http://radio.weblogs.com/0101454



From richie@entrian.com  Wed Sep 18 23:05:04 2002
From: richie@entrian.com (Richie Hindle)
Date: Wed, 18 Sep 2002 23:05:04 +0100
Subject: [Spambayes] pop3proxy.py
In-Reply-To: <464eougbmbqjg8qgj262jpr47v2g8r2voq@4ax.com>
References: <o83boucjsrighptscos5h60sh6m0gc4jkj@4ax.com>
	<w53znuh3itl.fsf@woozle.org> <464eougbmbqjg8qgj262jpr47v2g8r2voq@4ax.com>
Message-ID: <1qqhou0fc23dotvqhj143bq8usdlvke9mr@4ax.com>


[Neale]
> I doubt any sorting rules will look past
> the "Yes" or "No" [...] I would suggest using "No; 0.01 " instead of
> " No; 0.01", so you can regex-match "^No" instead of "^ No".

[Tim]
> I believe adding 0.99 (etc) does no good and may mislead.

Thanks for the ideas, guys.  pop3proxy.py now adds a simple
X-Hammie-Disposition header that says either "Yes" or "No ".

The code for pop3proxy.py now obeys the style guide.  Unlearning my habit
of putting spaces within parentheses is going to be hard (but the electric
shock device I've wired up to my space bar is helping a lot ) OUCH

-- 
Richie Hindle
richie@entrian.com


From neale@woozle.org  Wed Sep 18 23:43:33 2002
From: neale@woozle.org (Neale Pickett)
Date: 18 Sep 2002 15:43:33 -0700
Subject: [Spambayes] First result from Gary Robinson's ideas
In-Reply-To: <w537khj19dn.fsf@woozle.org>
References: <LNBBLJKPBEHFEDALKOLCKENMBEAB.tim.one@comcast.net>
	<w537khj19dn.fsf@woozle.org>
Message-ID: <w53sn06zzoq.fsf@woozle.org>

So then, Neale Pickett <neale@woozle.org> is all like:

> total unique fn went from 12 to 1000 lost  +8233.33%
> mean fn % went from 1.2 to 100.0 lost  +8233.33%

Please disregard those results.  This says that every single message in
my spam corpus got tagged as ham with this change.  Investigating, I
found that I had neglegted to remove a line calculating prob after
inserting Tim's new code, so everything was getting a probability of
0.5.  On the positive side, my FP rate went to 0!  ;)

So here's another run with the *right* code change.

The first run, run1, is using Tim's original classifier code.  The
second is using the following modification Tim proposed to implement
Gary's first suggestion:

         prob_product = inverse_prob_product = 1.0
+        P = Q = 1.0
+        num_clues = 0
         for distance, prob, word, record in nbest:
             if prob is None:    # it's one of the dummies nbest started
             with
                 continue
             if record is not None:  # else wordinfo doesn't know about
             it
                 record.killcount += 1
             if evidence:
                 clues.append((word, prob))
-            prob_product *= prob
-            inverse_prob_product *= 1.0 - prob
+            num_clues += 1
+            P *= 1.0 - prob
+            Q *= prob
+
+        if num_clues:
+            P = 1.0 - P**(1./num_clues)
+            Q = 1.0 - Q**(1./num_clues)
+            prob = (P-Q)/(P+Q)  # in -1 .. 1
+            prob = 0.5 + prob/2 # shift to 0 .. 1
+        else:
+            prob = 0.5
 
-        prob = prob_product / (prob_product + inverse_prob_product)

Additionally, I changed the "spam cutoff" from 0.9 to 0.5.  Comparing
the results before (run1) and after (run2), I get:

"""
run1s -> run2s
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
-> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams

false positive percentages
    2.000  2.000  tied
    1.500  1.500  tied
    2.000  2.000  tied
    1.000  1.500  lost   +50.00%
    0.500  0.500  tied

won   0 times
tied  4 times
lost  1 times

total unique fp went from 14 to 15 lost    +7.14%
mean fp % went from 1.4 to 1.5 lost    +7.14%

false negative percentages
    1.500  1.000  won    -33.33%
    1.000  1.000  tied
    1.500  1.500  tied
    1.500  1.500  tied
    1.000  1.000  tied

won   1 times
tied  4 times
lost  0 times

total unique fn went from 13 to 12 won     -7.69%
mean fn % went from 1.3 to 1.2 won     -7.69%
"""

So false positives basically stayed the same--the one case where the
false positives got worse, it was only by one message, which I would
imagine is within the margin of error, but I Am Not A Statistician :).

But, as Tim said earlier, what's really interesting is the distribution
of scores across all runs.  The first run, without Gary's modification,
gives me the following distributions:

"""
Ham distribution for all runs:
* = 17 items
  0.00 984 **********************************************************
  2.50   1 *
  5.00   0 
  7.50   0 
 10.00   0 
 12.50   0 
 15.00   0 
 17.50   0 
 20.00   0 
 22.50   0 
 25.00   0 
 27.50   0 
 30.00   0 
 32.50   0 
 35.00   0 
 37.50   0 
 40.00   0 
 42.50   0 
 45.00   0 
 47.50   0 
 50.00   1 *
 52.50   0 
 55.00   0 
 57.50   0 
 60.00   0 
 62.50   0 
 65.00   0 
 67.50   0 
 70.00   0 
 72.50   0 
 75.00   0 
 77.50   0 
 80.00   0 
 82.50   0 
 85.00   0 
 87.50   0 
 90.00   0 
 92.50   0 
 95.00   0 
 97.50  14 *

Spam distribution for all runs:
* = 17 items
  0.00  11 *
  2.50   0 
  5.00   0 
  7.50   0 
 10.00   0 
 12.50   0 
 15.00   0 
 17.50   0 
 20.00   0 
 22.50   0 
 25.00   0 
 27.50   0 
 30.00   0 
 32.50   0 
 35.00   0 
 37.50   0 
 40.00   0 
 42.50   0 
 45.00   0 
 47.50   0 
 50.00   1 *
 52.50   0 
 55.00   0 
 57.50   0 
 60.00   0 
 62.50   0 
 65.00   0 
 67.50   0 
 70.00   0 
 72.50   0 
 75.00   0 
 77.50   0 
 80.00   0 
 82.50   1 *
 85.00   0 
 87.50   0 
 90.00   0 
 92.50   2 *
 95.00   3 *
 97.50 982 **********************************************************
"""

Your typical Grahamian black-or-white picture, with little middle
ground.  With Gary's idea, however, comes many more shades of gray:

"""
Ham distribution for all runs:
* = 12 items
  0.00 681 *********************************************************
  2.50  62 ******
  5.00  18 **
  7.50  10 *
 10.00  14 **
 12.50  33 ***
 15.00  40 ****
 17.50  28 ***
 20.00  14 **
 22.50  22 **
 25.00   5 *
 27.50  11 *
 30.00  13 **
 32.50  10 *
 35.00   6 *
 37.50   5 *
 40.00   3 *
 42.50   5 *
 45.00   5 *
 47.50   0 
 50.00   1 *
 52.50   4 *
 55.00   1 *
 57.50   1 *
 60.00   2 *
 62.50   1 *
 65.00   0 
 67.50   1 *
 70.00   2 *
 72.50   0 
 75.00   0 
 77.50   0 
 80.00   0 
 82.50   0 
 85.00   0 
 87.50   0 
 90.00   0 
 92.50   0 
 95.00   0 
 97.50   2 *

Spam distribution for all runs:
* = 14 items
  0.00   1 *
  2.50   0 
  5.00   0 
  7.50   0 
 10.00   0 
 12.50   0 
 15.00   0 
 17.50   0 
 20.00   0 
 22.50   0 
 25.00   0 
 27.50   0 
 30.00   0 
 32.50   4 *
 35.00   0 
 37.50   1 *
 40.00   1 *
 42.50   1 *
 45.00   3 *
 47.50   0 
 50.00   4 *
 52.50   4 *
 55.00   2 *
 57.50   9 *
 60.00  13 *
 62.50   9 *
 65.00  12 *
 67.50  14 *
 70.00   9 *
 72.50  15 **
 75.00   8 *
 77.50   7 *
 80.00  11 *
 82.50  12 *
 85.00   9 *
 87.50   0 
 90.00   1 *
 92.50   2 *
 95.00  13 *
 97.50 835 ************************************************************
"""

So from my perspective (and again, IANAS) it looks like the algorithm
has gained some humility and is admitting when it's not sure about
stuff.  I can't say this change is a clear win for my miniscule data
set, but it *does* appear to make the probability more meaningful.
Almost like the difference between linear space and log space.

Neale

From neale@woozle.org  Thu Sep 19 00:34:31 2002
From: neale@woozle.org (Neale Pickett)
Date: 18 Sep 2002 16:34:31 -0700
Subject: [Spambayes] idea for tokenizer.crack_filename change
Message-ID: <w53ofauzxbs.fsf@woozle.org>

In going over some of my spam, I was surprised to see that the following
wasn't penalized:

  ------=_NextPart_000_0039_0173A692.99A692D0
  Content-Type: application/octet-stream; name="Video.pif"
  Content-Transfer-Encoding: base64
  Content-Disposition: attachment; filename="Video.pif"

I can guarantee you that I've never been emailed a single .pif file from
an actual human being :)  But tokenizer.crack_filename only splits up
filenames by path elements, so ".pif" never got scored.

I suggest changing fname_sep_re to include ".", like so:

  fname_sep_re = re.compile(r'[./\\:]')

Unfortunately, I can't back up my suspicion that this is a good idea, as
it results in an across-the-board tie on my corpora.  Maybe someone with
larger corpora could try it out.  (Tim?)

Neale

From tim.one@comcast.net  Thu Sep 19 01:23:38 2002
From: tim.one@comcast.net (Tim Peters)
Date: Wed, 18 Sep 2002 20:23:38 -0400
Subject: [Spambayes] Mozilla and summary of bayes calcs
In-Reply-To: <B9AE7063.16172%grobinson@transpose.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCOEBKBFAB.tim.one@comcast.net>

BTW, I'm not a statistician, but I think there's "an obvious" sense in which
Graham's formulation is Bayesian that's been overlooked -- although it's
pretty well hidden <wink>.  I ranted about that earlier:

http://mail.python.org/pipermail/python-dev/2002-August/028216.html

Two footnotes to that:  (1) In email, Paul later said that the proportion of
ham to spam in his inbox is in fact close to 1 to 1, not the 2 to 1 I had
speculated there.  (2) I very briefly experimented with taking P(spam) and
P(not-spam) into account (based on the training set proportion) as explained
there, but doing so was neither a clear win nor a clear loss, and, at the
time, I was getting big clear wins via other means so just dropped this line
of experimentation.


From tim.one@comcast.net  Thu Sep 19 01:38:32 2002
From: tim.one@comcast.net (Tim Peters)
Date: Wed, 18 Sep 2002 20:38:32 -0400
Subject: [Spambayes] idea for tokenizer.crack_filename change
In-Reply-To: <w53ofauzxbs.fsf@woozle.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEBMBFAB.tim.one@comcast.net>

[Neale Pickett]
> In going over some of my spam, I was surprised to see that the following
> wasn't penalized:
>
>   ------=_NextPart_000_0039_0173A692.99A692D0
>   Content-Type: application/octet-stream; name="Video.pif"
>   Content-Transfer-Encoding: base64
>   Content-Disposition: attachment; filename="Video.pif"

The mostly likely reason for this is that you simply don't have many pif
files in your spam training set.  Check it out.

> I can guarantee you that I've never been emailed a single .pif file
> from an actual human being :)  But tokenizer.crack_filename only
> splits up filenames by path elements, so ".pif" never got scored.

Not so:

def crack_filename(fname):
    yield "fname:" + fname
    components = fname_sep_re.split(fname)
    morethan1 = len(components) > 1
    for component in components:
        if morethan1:
            yield "fname comp:" + component
        pieces = urlsep_re.split(component)
        if len(pieces) > 1:
            for piece in pieces:
                yield "fname piece:" + piece

fname_sep_re only splits on *path* components:  forward slash, backward
slash, and colon.  Each component in turn is then split on urlsep_re, which
includes a wide variety of de jure and de facto URL metacharacters.  '.' is
among them, and the pif here should be yielding a

    'fname piece:pif'

token.  It it isn't, there's some sort of bug.  The filename as a whole
should have been extracted via the

    fname = msg.get_filename()
    if fname is not None:
        for x in crack_filename(fname):
            yield 'filename:' + x

portion of crack_content_xyz().  A

    'content-disposition:attachment'

token should have been produced by the code just before that.  Similarly for
Content-Type.  As the comments say, though, Content-Transfer-Encoding is
ignored because test results showed that including it changed results in
minor ways, for both better and worse, across distinct test runs.

>
> I suggest changing fname_sep_re to include ".", like so:
>
>   fname_sep_re = re.compile(r'[./\\:]')

Nope.  That's not what this regexp is for.  If you're not seeing the tokens
mentioned above, there *is* a bug here, and I'd like to know about that.
But the mere fact that a pif token didn't make into the list of best
discriminators for this message doesn't mean anything.

> Unfortunately, I can't back up my suspicion that this is a good idea, as
> it results in an across-the-board tie on my corpora.  Maybe someone with
> larger corpora could try it out.  (Tim?)

I did find value in what crack_filename did else I wouldn't have added the
code <wink>.  I don't know how much value I got specifically from finding
pif tokens, but I noticde once that finding .exe extensions seemed valuable
on a test run.  As always, though, the idea is tokenize everything and not
think too much <wink -- this is ironic given how much sweat has gone into
tokenizing in effective ways!>.


From grobinson@transpose.com  Thu Sep 19 02:01:27 2002
From: grobinson@transpose.com (Gary Robinson)
Date: Wed, 18 Sep 2002 21:01:27 -0400
Subject: [Spambayes] Mozilla and summary of bayes calcs
In-Reply-To: <LNBBLJKPBEHFEDALKOLCOEBKBFAB.tim.one@comcast.net>
Message-ID: <B9AE9A27.161A0%grobinson@transpose.com>

That's really interesting. I had started down that general path of analysis
but stopped before I got to the point of seeing how it made sense. Might be
worthwhile to go farther with your (2) but understand time constraints and
other opportunities for improvement.

I'm going to point to the your
http://mail.python.org/pipermail/python-dev/2002-August/028216.html post
from my essay.

BTW, have you looked at the papers on naive bayesian classification? I'm
curious as to why you are starting with the Graham approach rather than the
nbc approach for spambayes. I'm very interested in the question of which
performs better in real life testing.

--Gary


-- 
Gary Robinson
CEO
Transpose, LLC
grobinson@transpose.com
207-942-3463
http://www.emergentmusic.com
http://radio.weblogs.com/0101454


> From: Tim Peters <tim.one@comcast.net>
> Date: Wed, 18 Sep 2002 20:23:38 -0400
> To: Gary Robinson <grobinson@transpose.com>
> Cc: SpamBayes <spambayes@python.org>
> Subject: RE: [Spambayes] Mozilla and summary of bayes calcs
> 
> BTW, I'm not a statistician, but I think there's "an obvious" sense in which
> Graham's formulation is Bayesian that's been overlooked -- although it's
> pretty well hidden <wink>.  I ranted about that earlier:
> 
> http://mail.python.org/pipermail/python-dev/2002-August/028216.html
> 
> Two footnotes to that:  (1) In email, Paul later said that the proportion of
> ham to spam in his inbox is in fact close to 1 to 1, not the 2 to 1 I had
> speculated there.  (2) I very briefly experimented with taking P(spam) and
> P(not-spam) into account (based on the training set proportion) as explained
> there, but doing so was neither a clear win nor a clear loss, and, at the
> time, I was getting big clear wins via other means so just dropped this line
> of experimentation.
> 


From tim.one@comcast.net  Thu Sep 19 02:11:01 2002
From: tim.one@comcast.net (Tim Peters)
Date: Wed, 18 Sep 2002 21:11:01 -0400
Subject: [Spambayes] Another optimization
In-Reply-To: <20020918203719.2742CF49B@cashew.wolfskeep.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEBOBFAB.tim.one@comcast.net>

[T. Alexander Popiel]

[on tokenization schemes]
> ...
> Argh.  I just saw that comment block, nestled between n-grams
> and HTML discussions.  Somehow I managed to overlook it on my
> prior readings.  Sorry.

No problem, Alex -- it can be mind-numbing reading <wink>.

>> Likewise, but under the section "How big should 'a word' be?".

> Yeah, but that section only seems to refer to deciding the cutoff
> length between 12 and 13 characters, not whether there should be a
> cutoff at all.  Hrm.

I think it's safe to safe that at least a hundred large experiments have
been run that didn't get written up, or got written up only in checkin
comments that got lost when we moved the codebase from a dark corner of the
Python project.

Since I haven't reviewed this particular decision in a long time, here are
comparative results from changing the upper limit on "word" from 12 to
1200000.  This is on a 10-fold cross validation run, across 20,000 randomly
selected legit comp.lang.python messages, and essentially all of Bruce
Guentner's 2002 spam archive:

"""
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams

false positive percentages
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.050  0.050  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.050  0.050  tied
    0.000  0.000  tied
    0.100  0.100  tied

won   0 times
tied 10 times
lost  0 times

total unique fp went from 4 to 4 tied
mean fp % went from 0.02 to 0.02 tied

false negative percentages
    0.218  0.291  lost   +33.49%
    0.364  0.436  lost   +19.78%
    0.000  0.000  tied
    0.218  0.218  tied
    0.218  0.218  tied
    0.291  0.364  lost   +25.09%
    0.218  0.291  lost   +33.49%
    0.145  0.218  lost   +50.34%
    0.291  0.364  lost   +25.09%
    0.073  0.073  tied

won   0 times
tied  4 times
lost  6 times

total unique fn went from 28 to 34 lost   +21.43%
mean fn % went from 0.203636363636 to 0.247272727273 lost   +21.43%
"""

The null hypothesis (that this changes makes no difference) can't be
rejected based on the evidence.  If anything, it *may* hurt the f-n rate a
tiny bit.  But these rates are so low relative to the corpus sizes that,
when looking at the f-n rate, 0.07% is one lousy message out of 1375, so an
increase from (for example) 0.218% to 0.291% *should* be read as "one lousy
message" <wink> instead of sa "whoa!  it got 33.5% worse!".  So, in all, I
see the same thing boosing the upper limit to 1,200,000 as I saw when
boosting it to 13:  no significant difference.  Therefore I'll keep it at
12.

>> (Perhaps not so) Strangely enough, my experiments showed that
>> never ignoring a word worked better.

> That's not something I thought to check.  Perhaps I'll try it.

I suspect Gary Robinson has a better approach to this; giving a spamprob of,
e.g., 0.99, to a word that's appeared exactly one in exactly one spam really
hurt me.  But the test results said it was a real win, so I made that change
and left theory to the theorists <wink>.

> ...
> Yeah, that's the distinction I thought I saw.  I tested it both ways
> (though not as rigorously as you do) and chose my way.

A problem is there are many more things that could and should be tested than
I can possibly make time to test.  So, like you, I pick my battles based on
staring at the mistakes our scheme makes.  I suspect Gary Robinson also has
a better approach to dealing with the minprob/maxprob artificats (Gary has
posted several times in the last two days about cool ideas that beg for
testing -- I suspect I have a better background to make this kind of scheme
really fly than Paul has, and I'm positive Gary has a better background than
I do).

> ...
> The cases that decided me had about 20-30 maximal clues, with about
> 5 or 6 unpaired .99s.  Allowing the less significant clues (ranging
> from .02 to .08 with only a couple .96-.98s) flipped them from 'spam'
> to 'ham', increasing the FN rate annoyingly.

I've only got 28 false negatives left out of 13,750 spam, and the cases
where cancellation still plays a clear role are of the extremely lopsided
nature I mentioned (a hundred .01 vs a few dozen .99).  It's quite possible
that different effects come into play with a smaller corpus, though!  I
haven't had time to test that either.

>> This "only look at smoking guns" scoring scheme seems to be
>> systematically weak when dealing with long and chatty "just folks"
>> spam (of which there is blessedly little so far <wink>).

> Indeed.  I'm at a loss for how to combat it, though, unless we
> artificially restrict the number of .01s we get through limited
> corpora.

I've got no bright ideas here either.  I've become dubious that a
statistical approach *can* catch these, unless it finds good clues in the
headers.

>> You really need multiple test runs to have confidence you're
>> making progress.

> Yeah, I really ought to be more anal about it. ;-)

Do what timcv.py does <wink>.  I don't want to take time to translate it
into Perl, but the idea is simple:  divide your data into 10 sets at random
(for some value of 10 you like ...).  Then do 10 runs, on each run leaving
one of the 10 sets out to predict on, and training on the other 9/10ths.  If
you give your classifier both "learn" and "unlearn" methods, this can be
done more efficiently than it may first appear (on run #i, unlearn set #i,
predict on set #i, and then train on set #i again).  The results across all
10 runs will then give you good evidence about how sensitive ideas are to
statistical quirks, although all runs are still taken from the same corpus.


From tim.one@comcast.net  Thu Sep 19 03:24:19 2002
From: tim.one@comcast.net (Tim Peters)
Date: Wed, 18 Sep 2002 22:24:19 -0400
Subject: [Spambayes] Mozilla and summary of bayes calcs
In-Reply-To: <B9AE9A27.161A0%grobinson@transpose.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCIECBBFAB.tim.one@comcast.net>

[Gary Robinson]
> That's really interesting. I had started down that general path of
> analysis but stopped before I got to the point of seeing how it made
> sense.

Heh -- it took me days of part-time pondering to uncover the connection at
the time.  The anonymous article Paul pointed to neither mentioned nor
invoked Bayes by name, so it was entirely unclear to me too what Bayes had
to do with any of this.  But reading the article very carefully uncovered a
hidden assumption that the (unconditional) probabilities of "the event"
occurring or not occurting were equal, and then it all fell into place:  it
*was* naive Bayesian in this sense, but relying on a highly dubious
assumption that cancelled the P(spam) and P(not-spam) terms I kept looking
for but couldn't see.  It turns out I couldn't see them for the most
surprising of reasons:  they actually weren't there <wink>.

> Might be worthwhile to go farther with your (2) but understand time
> constraints and other opportunities for improvement.

Your ideas are much more interesting to me right now.

> I'm going to point to the your
> http://mail.python.org/pipermail/python-dev/2002-August/028216.html post
> from my essay.

As always, feel free -- we've got no secrets here.

> BTW, have you looked at the papers on naive bayesian classification? I'm
> curious as to why you are starting with the Graham approach rather than
> the nbc approach for spambayes. I'm very interested in the question of
> which performs better in real life testing.

Graham's approach attracted me instantly because of its delightful
presumption that this particular task should be easy:  it took balls of a
kind I don't have <wink> to even *try* a scheme that only scored 15 words in
an entire message.  It also looked very easy to implement (and it was), and
it looked like the scoring loop would be very fast (and it is -- although
the training is not cheap).  The results were so good on my first
implementation that I willingly suspended my doubts about its underpinnings
and just pressed on.

I read several of the papers on NBCs, and knew a little about them before
from a previous life, and my impression was they weren't *this* good at this
specific task, at least not without major implementation effort involving
lots of obscure technical tricks.  There was also the testimony of ifile(*)
users that it didn't do very well when asked just to distinguish spam from
non-spam; indeed, reports were that it did better if told to distinguish
get-rich-quick spam from human-growth-hormone spam from Nigerian-scam-span
etc.

Recent papers on NBCs for the spam versus non-spam task were pulling tricks
like stopword lists, and mutual information calculations, finding that these
improved results by keeping "junk words" out of the calculation.  Graham's
scheme carries that to an extreme in an intuitively appealing way, and for
the specific spam-vs-not-spam task it may well be that the problem *is*
extremely easy if approached in a suitably extreme way.  On a bang for the
buck invested measure, I haven't written code for any other messy real-life
task that performed this well with so little total effort.  (So, yes, the
bottom line is just that this has been fun <wink>.)


(*) ifile is a good implementation of "a classic" N-way NBC:

    http://www.ai.mit.edu/~jrennie/ifile/

    I doubt that Graham's approach would work well for a subtle
    classification problem -- but it's not trying to, and it works
    exceedingly well for what it is trying to do.  That seems a
    stroke of genius to me.


From tim.one@comcast.net  Thu Sep 19 03:44:34 2002
From: tim.one@comcast.net (Tim Peters)
Date: Wed, 18 Sep 2002 22:44:34 -0400
Subject: [Spambayes] First result from Gary Robinson's ideas
In-Reply-To: <w53sn06zzoq.fsf@woozle.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCAECDBFAB.tim.one@comcast.net>

[Neale Pickett, quoting Neale Pickett]
>> total unique fn went from 12 to 1000 lost  +8233.33%
>> mean fn % went from 1.2 to 100.0 lost  +8233.33%

[and then Neale gets on Neale's case]
> Please disregard those results.  This says that every single message in
> my spam corpus got tagged as ham with this change.  Investigating, I
> found that I had neglegted to remove a line calculating prob after
> inserting Tim's new code, so everything was getting a probability of
> 0.5.  On the positive side, my FP rate went to 0!  ;)

You can forgive yourself, if you have to <wink>.  I should have given you a
patch instead of an English description of what to do!  There was another
problem here too, though, as the summary statistics quoted above obviously
didn't come from the same run from which your score histograms came:  the
historgrams showed a low false positive rate.

[and after all that was repaired]
> ...
> Additionally, I changed the "spam cutoff" from 0.9 to 0.5.  Comparing
> the results before (run1) and after (run2), I get:
>
> """
> run1s -> run2s
> -> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
> -> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
> -> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
> -> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
> -> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
> -> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
> -> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
> -> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
> -> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
> -> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
>
> false positive percentages
>     2.000  2.000  tied
>     1.500  1.500  tied
>     2.000  2.000  tied
>     1.000  1.500  lost   +50.00%
>     0.500  0.500  tied
>
> won   0 times
> tied  4 times
> lost  1 times
>
> total unique fp went from 14 to 15 lost    +7.14%
> mean fp % went from 1.4 to 1.5 lost    +7.14%
>
> false negative percentages
>     1.500  1.000  won    -33.33%
>     1.000  1.000  tied
>     1.500  1.500  tied
>     1.500  1.500  tied
>     1.000  1.000  tied
>
> won   1 times
> tied  4 times
> lost  0 times
>
> total unique fn went from 13 to 12 won     -7.69%
> mean fn % went from 1.3 to 1.2 won     -7.69%
> """
>
> So false positives basically stayed the same--the one case where the
> false positives got worse, it was only by one message, which I would
> imagine is within the margin of error, but I Am Not A Statistician :).

Intuition is good enough at extremes:  one lousy message is indeed one lousy
message, no matter what the percentage change.

> But, as Tim said earlier, what's really interesting is the distribution
> of scores across all runs.  The first run, without Gary's modification,
> gives me the following distributions:
>
> """
> Ham distribution for all runs:
> * = 17 items
>   0.00 984 **********************************************************
>   2.50   1 *
>   5.00   0
>   7.50   0
>  10.00   0
>  12.50   0
>  15.00   0
>  17.50   0
>  20.00   0
>  22.50   0
>  25.00   0
>  27.50   0
>  30.00   0
>  32.50   0
>  35.00   0
>  37.50   0
>  40.00   0
>  42.50   0
>  45.00   0
>  47.50   0
>  50.00   1 *
>  52.50   0
>  55.00   0
>  57.50   0
>  60.00   0
>  62.50   0
>  65.00   0
>  67.50   0
>  70.00   0
>  72.50   0
>  75.00   0
>  77.50   0
>  80.00   0
>  82.50   0
>  85.00   0
>  87.50   0
>  90.00   0
>  92.50   0
>  95.00   0
>  97.50  14 *
>
> Spam distribution for all runs:
> * = 17 items
>   0.00  11 *
>   2.50   0
>   5.00   0
>   7.50   0
>  10.00   0
>  12.50   0
>  15.00   0
>  17.50   0
>  20.00   0
>  22.50   0
>  25.00   0
>  27.50   0
>  30.00   0
>  32.50   0
>  35.00   0
>  37.50   0
>  40.00   0
>  42.50   0
>  45.00   0
>  47.50   0
>  50.00   1 *
>  52.50   0
>  55.00   0
>  57.50   0
>  60.00   0
>  62.50   0
>  65.00   0
>  67.50   0
>  70.00   0
>  72.50   0
>  75.00   0
>  77.50   0
>  80.00   0
>  82.50   1 *
>  85.00   0
>  87.50   0
>  90.00   0
>  92.50   2 *
>  95.00   3 *
>  97.50 982 **********************************************************
> """
>
> Your typical Grahamian black-or-white picture, with little middle
> ground.  With Gary's idea, however, comes many more shades of gray:
>
> """
> Ham distribution for all runs:
> * = 12 items
>   0.00 681 *********************************************************
>   2.50  62 ******
>   5.00  18 **
>   7.50  10 *
>  10.00  14 **
>  12.50  33 ***
>  15.00  40 ****
>  17.50  28 ***
>  20.00  14 **
>  22.50  22 **
>  25.00   5 *
>  27.50  11 *
>  30.00  13 **
>  32.50  10 *
>  35.00   6 *
>  37.50   5 *
>  40.00   3 *
>  42.50   5 *
>  45.00   5 *
>  47.50   0

In particular:

>  50.00   1 *
>  52.50   4 *
>  55.00   1 *
>  57.50   1 *

7 (about half) of your false positives scored in the [50, 60) range, and

>  60.00   2 *
>  62.50   1 *
>  65.00   0
>  67.50   1 *

4 scored in [60, 70).  Now I've tweaked more things than I can keep track of
to improve the results using Paul's combining formula, and all of those are
open to re-tweaking to improve results with Gary's.  For the next step,
though, I want to try more of Gary's still-untested ideas.

>  70.00   2 *
>  72.50   0
>  75.00   0
>  77.50   0
>  80.00   0
>  82.50   0
>  85.00   0
>  87.50   0
>  90.00   0
>  92.50   0
>  95.00   0
>  97.50   2 *
>
> Spam distribution for all runs:
> * = 14 items
>   0.00   1 *
>   2.50   0
>   5.00   0
>   7.50   0
>  10.00   0
>  12.50   0
>  15.00   0
>  17.50   0
>  20.00   0
>  22.50   0
>  25.00   0
>  27.50   0

And here 5 of your false negatives lived in [30, 40):

>  30.00   0
>  32.50   4 *
>  35.00   0
>  37.50   1 *

and 5 more in [40, 50) (which leaves only the 1 at 0.00 outside of [30, 50):

>  40.00   1 *
>  42.50   1 *
>  45.00   3 *
>  47.50   0

>  50.00   4 *
>  52.50   4 *
>  55.00   2 *
>  57.50   9 *
>  60.00  13 *
>  62.50   9 *
>  65.00  12 *
>  67.50  14 *
>  70.00   9 *
>  72.50  15 **
>  75.00   8 *
>  77.50   7 *
>  80.00  11 *
>  82.50  12 *
>  85.00   9 *
>  87.50   0
>  90.00   1 *
>  92.50   2 *
>  95.00  13 *
>  97.50 835 ************************************************************
> """
>
> So from my perspective (and again, IANAS) it looks like the algorithm
> has gained some humility and is admitting when it's not sure about
> stuff.  I can't say this change is a clear win for my miniscule data
> set, but it *does* appear to make the probability more meaningful.
> Almost like the difference between linear space and log space.

It's encouraging that this pretty much exactly mirrors my results on my
monster-large-- and very different --test corpus.  I still conjecture that
my error rates are lower almost entirely because my training data is so much
larger, but we still haven't got a solid experimental handle on that, and
probably won't unless more people can be shamed <wink> into playing the
testing game.  I need to check in code to make this particular comparison
easier to do, though.


From anthony@interlink.com.au  Thu Sep 19 04:16:50 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Thu, 19 Sep 2002 13:16:50 +1000
Subject: [Spambayes] Another optimization 
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEBOBFAB.tim.one@comcast.net> 
Message-ID: <200209190316.g8J3GoJ09053@localhost.localdomain>



> I think it's safe to safe that at least a hundred large experiments have
> been run that didn't get written up, or got written up only in checkin
> comments that got lost when we moved the codebase from a dark corner of the
> Python project.

Would it be worth putting up a simple webpage on the spambayes.sf.net 
site that has links to the python-dev threads, links to this mailing list,
and links to a cvs2cl generated Changelog of the old non-dist/sandbox 
area? 

I can probably do this in the next day or so if people really care, 
but it probably won't be the all-singing-all-dancing kinda website that
Barry loves :)

Anthony

-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.


From guido@python.org  Thu Sep 19 04:17:52 2002
From: guido@python.org (Guido van Rossum)
Date: Wed, 18 Sep 2002 23:17:52 -0400
Subject: [Spambayes] Another optimization
In-Reply-To: Your message of "Thu, 19 Sep 2002 13:16:50 +1000."
             <200209190316.g8J3GoJ09053@localhost.localdomain> 
References: <200209190316.g8J3GoJ09053@localhost.localdomain> 
Message-ID: <200209190317.g8J3HqK28582@pcp02138704pcs.reston01.va.comcast.net>

> Would it be worth putting up a simple webpage on the spambayes.sf.net 

Yes please!

> site that has links to the python-dev threads, links to this mailing list,
> and links to a cvs2cl generated Changelog of the old non-dist/sandbox 
> area? 

Just don't expect Tim do do it.

> I can probably do this in the next day or so if people really care, 
> but it probably won't be the all-singing-all-dancing kinda website that
> Barry loves :)

Great!  (That you can do it, not that Barry won't love it.  Though
that's always an added advantage. :-)

--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido@python.org  Thu Sep 19 04:21:55 2002
From: guido@python.org (Guido van Rossum)
Date: Wed, 18 Sep 2002 23:21:55 -0400
Subject: [Spambayes] Need help getting started
Message-ID: <200209190321.g8J3Lti28601@pcp02138704pcs.reston01.va.comcast.net>

I'd like to run experiments for Tim.  My ham corpus is over 80,000
messages, spread over hundreds of MH folders, one message per file
with numeric names.  (Don't laugh.  I've been a packrat.  And that's
only since May 2000.)  As a spam archive I've downloaded Bruce
Guenter's archives from 2000, 2001, 2002, also one message per file,
with a .txt or .lorien extension.  (Does anyone know what the .lorien
means?)

But how to turn this into Tim's standard data setup?  Do I have to
write programs to do this?  Or does something exist that I simply
didn't see in the README.txt file?  How did Tim create his ham
archive?

--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.one@comcast.net  Thu Sep 19 05:40:33 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 19 Sep 2002 00:40:33 -0400
Subject: [Spambayes] Need help getting started
In-Reply-To: <200209190321.g8J3Lti28601@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCKECKBFAB.tim.one@comcast.net>

[Guido]
> I'd like to run experiments for Tim.

I'd prefer that you run them because it's the right thing to do <wink>.

> My ham corpus is over 80,000 messages, spread over hundreds of MH
> folders, one message per file with numeric names.  (Don't laugh.
> I've been a packrat.  And that's only since May 2000.)  As a spam
> archive I've downloaded Bruce Guenter's archives from 2000, 2001,
> 2002, also one message per file, with a .txt or .lorien extension.
> (Does anyone know what the .lorien means?)

I haven't found an official explanation, but when I made my initial mistake
of enabling tokenization of all headers, I quickly learned that

    lorien.untroubled.org

was a killer-strong spam indicator.  It appears to be a specific machine
bruceg used to collect spam from some of his bait addresses.  It may have
more significance to him than just that, but I erased the distinction and
shuffled all the lorien files into my spam directories at random.  They're
definitely spam!

Caution:  bruceg's spams have Windows line endings.  If you ever do
experiments with character n-grams, you too may find that \r\n becomes a
killer-strong spam indicator!  There are lots of traps in using messages
from different sources.  Unfortunately, they're all in the nature of giving
you good results for bogus reasons.

> But how to turn this into Tim's standard data setup?  Do I have to
> write programs to do this?  Or does something exist that I simply
> didn't see in the README.txt file?  How did Tim create his ham
> archive?

I wrote a program, and checked it in as splitndirs.py:

    Split an mbox into N random directories of files.

I used this to split up the c.l.py archive mbox; it's trivial.  I also
removed the initial "From " lines from the mbox msgs (bruceg's corpus
doesn't have those), and normalized line endings in bruceg's messages to \n.
That's it.  Skip has since done a wonderful job of generalizing rebal.py,
which can be used to reshuffle messages among directories (to even out the #
of msgs in each initially, and to replenish them from "the reservoir" as you
find spam in your ham, & vice versa -- there are hams in bruceg's spam
collections!).

Jeremy went down another path, using mboxes directly.  The differences
between mboxtest.py and timcv.py keep shrinking.  It should be easy to write
a similar test driver that works on MH directories directly -- remind me
tomorrow about exactly how those look.  Maybe we can find a better way to
refactor this stuff.


From neale@woozle.org  Thu Sep 19 06:26:58 2002
From: neale@woozle.org (Neale Pickett)
Date: 18 Sep 2002 22:26:58 -0700
Subject: [Spambayes] idea for tokenizer.crack_filename change
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCEBMBFAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCCEBMBFAB.tim.one@comcast.net>
Message-ID: <w534rcmtuql.fsf@woozle.org>

So then, Tim Peters <tim.one@comcast.net> is all like:

> [ neale proposes an idea that was already implemented ]

> The mostly likely reason for this is that you simply don't have many
> pif files in your spam training set.

Man, it's just one dumb thing after another for me.  At least I know
it was a good idea, though :^)

Neale

From neale@woozle.org  Thu Sep 19 06:36:00 2002
From: neale@woozle.org (Neale Pickett)
Date: 18 Sep 2002 22:36:00 -0700
Subject: [Spambayes] Need help getting started
In-Reply-To: <200209190321.g8J3Lti28601@pcp02138704pcs.reston01.va.comcast.net>
References: <200209190321.g8J3Lti28601@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <w53y99ysfr3.fsf@woozle.org>

So then, Guido van Rossum <guido@python.org> is all like:

> I'd like to run experiments for Tim.  My ham corpus is over 80,000
> messages, spread over hundreds of MH folders, one message per file
> with numeric names.

I'm not really up on MH format, but my mail reader (Gnus) stores
messages one per file with numeric names too.  In fact, all the changes
you made to hammie to support MH folders worked great for me.  So I
suspect the format is pretty close, if not identical.

> But how to turn this into Tim's standard data setup?

Super easy.  Create the following directories:

  Data/Ham/Set[1-5]
  Data/Ham/reservoir
  Data/Spam/Set[1-5]
  Data/Spam/reservoir

Then go into the reservoirs, and link (I used hard links to save inodes)
all your ham/spam into the directories:

  $ cd Data/Ham/reservoir
  $ ln ~/Mail/inbox/* .
  $ cd ../../../Data/Spam/reservoir
  $ ln ~/Mail/spam/* .

At that point, you have a standard setup, and can use rebal.py to
populate your Set directories.  If you have a bourne shell, you can
become the second ever user of runtest.sh--call it with the -r option
and it'll rebal for you.

Neale

From anthony@interlink.com.au  Thu Sep 19 06:48:19 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Thu, 19 Sep 2002 15:48:19 +1000
Subject: [Spambayes] Need help getting started 
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKECKBFAB.tim.one@comcast.net> 
Message-ID: <200209190548.g8J5mJK10603@localhost.localdomain>


>>> Tim Peters wrote
> Jeremy went down another path, using mboxes directly.  The differences
> between mboxtest.py and timcv.py keep shrinking.  It should be easy to write
> a similar test driver that works on MH directories directly -- remind me
> tomorrow about exactly how those look.  Maybe we can find a better way to
> refactor this stuff.

Tim's existing tests work fine on MH folders as-is, so long as you've
blown away any , files (deleted messages).

Anthony


From neale@woozle.org  Thu Sep 19 06:52:25 2002
From: neale@woozle.org (Neale Pickett)
Date: 18 Sep 2002 22:52:25 -0700
Subject: [Spambayes] irony?
Message-ID: <w53ptvasezq.fsf@woozle.org>

You know, since about the time I got on this list, the rate at which
I've been getting spammed has more than doubled.

What's bizarre is that I'm /happy/ about this :*)

Cultivating the corpus,

Neale


From tim.one@comcast.net  Thu Sep 19 08:17:25 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 19 Sep 2002 03:17:25 -0400
Subject: Third result ... RE: [Spambayes] First result from Gary Robinson's
 ideas
In-Reply-To: <LNBBLJKPBEHFEDALKOLCAECDBFAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEDABFAB.tim.one@comcast.net>

I've checked in sane changes to the code base now, so that you can try Gary
Robinson's increasingly remarkable probability combining scheme (it was
amazing when I first tried it, and triply so when Neale reported isomorphic
results on a wholly different corpus).  Just set

    [Classifier]
    use_robinson_probability: True

    [TestDriver]
    spam_cutoff: 0.50

as a pair.  The best possible value of spam_cutoff is easy to deduce from
the final score histograms after a run, but for my and Neale's runs 0.50 was
indeed optimal.

I just ran one small-subset test on this before checking it in, on a random
subset of 1000 each of my ham and spam sets.  Results were identical across
all 10 runs (no change in fp or fn rates, no change in total unique fp or
fn).  The score histograms under Graham's combining scheme are "the usual"
bipolar extreme:

Ham distribution for all runs:
* = 17 items
  0.00 998 ***********************************************************
  2.50   0
  5.00   0
  7.50   0
 10.00   0
 12.50   0
 15.00   0
 17.50   0
 20.00   0
 22.50   0
 25.00   0
 27.50   0
 30.00   0
 32.50   0
 35.00   0
 37.50   0
 40.00   0
 42.50   0
 45.00   0
 47.50   0
 50.00   0
 52.50   0
 55.00   0
 57.50   0
 60.00   0
 62.50   0
 65.00   0
 67.50   0
 70.00   0
 72.50   0
 75.00   0
 77.50   0
 80.00   0
 82.50   0
 85.00   0
 87.50   0
 90.00   0
 92.50   0
 95.00   0
 97.50   2 *

Spam distribution for all runs:
* = 17 items
  0.00   2 *
  2.50   0
  5.00   0
  7.50   0
 10.00   0
 12.50   0
 15.00   0
 17.50   0
 20.00   0
 22.50   0
 25.00   0
 27.50   0
 30.00   0
 32.50   0
 35.00   0
 37.50   1 *
 40.00   0
 42.50   0
 45.00   0
 47.50   0
 50.00   1 *
 52.50   0
 55.00   0
 57.50   0
 60.00   0
 62.50   0
 65.00   0
 67.50   0
 70.00   0
 72.50   0
 75.00   0
 77.50   0
 80.00   0
 82.50   0
 85.00   0
 87.50   0
 90.00   0
 92.50   0
 95.00   1 *
 97.50 995 ***********************************************************

Under Gary's scheme, again more interesting:

Ham distribution for all runs:
* = 14 items
  0.00 816 ***********************************************************
  2.50  37 ***
  5.00  13 *
  7.50   2 *
 10.00  12 *
 12.50  34 ***
 15.00  16 **
 17.50  13 *
 20.00   9 *
 22.50  13 *
 25.00  10 *
 27.50   7 *
 30.00   8 *
 32.50   3 *
 35.00   2 *
 37.50   1 *
 40.00   1 *
 42.50   1 *
 45.00   0
 47.50   0

Note that the two false positives live in this range:

 50.00   1 *
 52.50   0
 55.00   0
 57.50   1 *

 60.00   0
 62.50   0
 65.00   0
 67.50   0
 70.00   0
 72.50   0
 75.00   0
 77.50   0
 80.00   0
 82.50   0
 85.00   0
 87.50   0
 90.00   0
 92.50   0
 95.00   0
 97.50   0

Spam distribution for all runs:
* = 15 items
  0.00   0
  2.50   0
  5.00   0
  7.50   0
 10.00   0
 12.50   0
 15.00   0
 17.50   0
 20.00   0
 22.50   0
 25.00   0
 27.50   0
 30.00   0
 32.50   0
 35.00   0

Three of the four false negatives live here:

 37.50   1 *
 40.00   1 *
 42.50   1 *

 45.00   0
 47.50   0

The fourth false negative scored exactly 0.50, and lives here:

 50.00   2 *

 52.50   4 *
 55.00   1 *
 57.50   0
 60.00   4 *
 62.50   1 *
 65.00   8 *
 67.50   6 *
 70.00   3 *
 72.50   5 *
 75.00   8 *
 77.50   9 *
 80.00   9 *
 82.50  16 **
 85.00   9 *
 87.50   0
 90.00   1 *
 92.50   4 *
 95.00  14 *
 97.50 893 ************************************************************

So how did a spam score exactly 0.50?  Like this:

"""
Data/Spam/Set9/1673.txt
prob = 0.5
prob('control: MessageParseError') = 0.5

Return-Path: <qvtlv@tjohoo.se>
Delivered-To: em-ca-bait@em.ca
Received: (qmail 10178 invoked from network); 5 Jun 2002 10:19:14 -0000
Received: from serverw3.easyw3.fr (HELO easyw4.easyw3.fr) (194.3.175.1)
  by churchill.factcomp.com with SMTP; 5 Jun 2002 10:19:14 -0000
Received: from mail.tjohoo.se (216.50.157.50 [216.50.157.50]) by
easyw4.easyw3.fr with SMTP (Microsoft Exchange Internet Mail Service Version
5.5.2655.55)
        id MAQYC3S9; Wed, 5 Jun 2002 07:49:12 +0200
From: "Marietta" <qvtlv@tjohoo.se>
To: "kpe@cs.com
" <kpe@cs.com
>
Subject: look good and have a great summer
Content-Type: text/plain; charset="us-ascii";format=flowed
Content-Transfer-Encoding: 7bit
Content-Length: 1058
Lines: 34

... and the rest is irrelevant ...
"""

That is, the email parser barfed on the malformed "To" header, so
"MessageParseError" was the *only* token generated, and that didn't appear
in the training set at all (so it got our "unknown word prob").  Note that
something is rated spam only if its score is strictly greater than
spam_cutoff, and 0.5 isn't greater than 0.5 <wink>.  I (or someone else --
please <wink>?) should probably change the tokenizer to back off to the raw
message body when the email parser gives up.

This was a long spam, but I guarantee we would have nailed it just from the
first paragraph:

"""
As seen on NBC, CBS, CNN, and even Oprah! The health
discovery that actually reverses aging while burning fat,
without dieting or exercise! This proven discovery has even
been reported on by the New England Journal of Medicine.
Forget  aging and dieting forever! And it's Guaranteed!

Click here:
http://66.231.133.205/hgh/index.html
"""


From anthony@interlink.com.au  Thu Sep 19 09:49:44 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Thu, 19 Sep 2002 18:49:44 +1000
Subject: [Spambayes] initial cut at website is up.
In-Reply-To: <200209190316.g8J3GoJ09053@localhost.localdomain> 
Message-ID: <200209190849.g8J8niv12565@localhost.localdomain>


Ok, there's a first cut at a website at spambayes.sf.net - the
site's built using ht2html (I was already using it for pydns,
so it was easy enough to use it for a quick result). Plus, it
avoids the Barry-rant :)

The website itself is checked in as "website" in CVS. To make changes,
cvs co website, edit the .ht file(s), run 'make' and check it's all ok
in the local directory. 'make install' will push the change to the
web server. And of course, check in the changes. 

If you're one of those steenkin windows users <wink> I'm sure there's
an equivalent technique. 

Anthony

From anthony@interlink.com.au  Thu Sep 19 09:59:12 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Thu, 19 Sep 2002 18:59:12 +1000
Subject: [Spambayes] initial cut at website is up. 
In-Reply-To: <200209190849.g8J8niv12565@localhost.localdomain> 
Message-ID: <200209190859.g8J8xCF12722@localhost.localdomain>


In case it needed to be said: Please, if you don't like what's there,
feel free to fix it. I'm not at all precious about it - working on 
this stuff was a pleasant break from what I was _supposed_ to be doing.


-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.


From anthony@interlink.com.au  Thu Sep 19 10:44:48 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Thu, 19 Sep 2002 19:44:48 +1000
Subject: Third result ... RE: [Spambayes] First result from Gary
	Robinson's ideas 
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEDABFAB.tim.one@comcast.net> 
Message-ID: <200209190944.g8J9inS13207@localhost.localdomain>


>>> Tim Peters wrote
> I've checked in sane changes to the code base now, so that you can try Gary
> Robinson's increasingly remarkable probability combining scheme (it was
> amazing when I first tried it, and triply so when Neale reported isomorphic
> results on a wholly different corpus).  Just set
> 
>     [Classifier]
>     use_robinson_probability: True
> 
>     [TestDriver]
>     spam_cutoff: 0.50

Here's what I'm seeing - run1 was default, run2 with the above settings:


run1s -> run2s
-> <stat> tested 1799 hams & 1547 spams against 7195 hams & 6191 spams
-> <stat> tested 1799 hams & 1548 spams against 7195 hams & 6190 spams
-> <stat> tested 1799 hams & 1549 spams against 7195 hams & 6189 spams
-> <stat> tested 1798 hams & 1548 spams against 7196 hams & 6190 spams
-> <stat> tested 1799 hams & 1546 spams against 7195 hams & 6192 spams
-> <stat> tested 1799 hams & 1547 spams against 7195 hams & 6191 spams
-> <stat> tested 1799 hams & 1548 spams against 7195 hams & 6190 spams
-> <stat> tested 1799 hams & 1549 spams against 7195 hams & 6189 spams
-> <stat> tested 1798 hams & 1548 spams against 7196 hams & 6190 spams
-> <stat> tested 1799 hams & 1546 spams against 7195 hams & 6192 spams

false positive percentages
    0.334  0.445  lost   +33.23%
    0.278  0.334  lost   +20.14%
    0.389  0.445  lost   +14.40%
    0.556  0.556  tied
    0.278  0.334  lost   +20.14%

won   0 times
tied  1 times
lost  4 times

total unique fp went from 33 to 38 lost   +15.15%
mean fp % went from 0.366932315011 to 0.42251875192 lost   +15.15%

false negative percentages
    0.582  0.452  won    -22.34%
    0.775  0.646  won    -16.65%
    0.710  0.710  tied
    0.581  0.581  tied
    0.906  0.647  won    -28.59%

won   3 times
tied  2 times
lost  0 times

total unique fn went from 55 to 47 won    -14.55%
mean fn % went from 0.710811726238 to 0.60736899408 won    -14.55%

Histograms - the "before" case:

Ham distribution for all runs:
* = 150 items
  0.00 8942 ************************************************************
  2.50    1 *
  5.00    1 *
  7.50    1 *
 10.00    0 
 12.50    0 
 15.00    1 *
 17.50    1 *
 20.00    0 
 22.50    0 
 25.00    0 
 27.50    5 *
 30.00    1 *
 32.50    1 *
 35.00    0 
 37.50    0 
 40.00    1 *
 42.50    0 
 45.00    1 *
 47.50    0 
 50.00    0 
 52.50    0 
 55.00    0 
 57.50    0 
 60.00    0 
 62.50    2 *
 65.00    0 
 67.50    1 *
 70.00    0 
 72.50    0 
 75.00    1 *
 77.50    0 
 80.00    0 
 82.50    1 *
 85.00    0 
 87.50    0 
 90.00    0 
 92.50    0 
 95.00    3 *
 97.50   30 *

Spam distribution for all runs:
* = 128 items
  0.00   28 *
  2.50    2 *
  5.00    2 *
  7.50    1 *
 10.00    3 *
 12.50    1 *
 15.00    1 *
 17.50    0 
 20.00    0 
 22.50    0 
 25.00    0 
 27.50    5 *
 30.00    1 *
 32.50    1 *
 35.00    1 *
 37.50    0 
 40.00    1 *
 42.50    0 
 45.00    0 
 47.50    0 
 50.00    0 
 52.50    0 
 55.00    0 
 57.50    2 *
 60.00    2 *
 62.50    0 
 65.00    1 *
 67.50    0 
 70.00    1 *
 72.50    0 
 75.00    0 
 77.50    0 
 80.00    0 
 82.50    1 *
 85.00    0 
 87.50    1 *
 90.00    1 *
 92.50    0 
 95.00    3 *
 97.50 7679 ************************************************************

While the "after" case is more interesting:

Ham distribution for all runs:
* = 69 items
  0.00 4115 ************************************************************
  2.50  808 ************
  5.00  375 ******
  7.50  253 ****
 10.00  166 ***
 12.50  446 *******
 15.00  651 **********
 17.50  440 *******
 20.00  313 *****
 22.50  323 *****
 25.00  317 *****
 27.50  207 ***
 30.00  154 ***
 32.50  140 ***
 35.00   94 **
 37.50   55 *
 40.00   41 *
 42.50   28 *
 45.00   19 *
 47.50   11 *
 50.00    9 *
 52.50    6 *
 55.00    6 *
 57.50    2 *
 60.00    2 *
 62.50    1 *
 65.00    2 *
 67.50    2 *
 70.00    2 *
 72.50    0 
 75.00    0 
 77.50    1 *
 80.00    1 *
 82.50    1 *
 85.00    0 
 87.50    0 
 90.00    0 
 92.50    0 
 95.00    0 
 97.50    3 *

Spam distribution for all runs:
* = 116 items
  0.00    5 *
  2.50    0 
  5.00    0 
  7.50    0 
 10.00    0 
 12.50    0 
 15.00    1 *
 17.50    2 *
 20.00    0 
 22.50    0 
 25.00    0 
 27.50    0 
 30.00    2 *
 32.50    2 *
 35.00    3 *
 37.50    6 *
 40.00    3 *
 42.50    4 *
 45.00    5 *
 47.50   14 *
 50.00   10 *
 52.50   29 *
 55.00   31 *
 57.50   47 *
 60.00   49 *
 62.50   45 *
 65.00   56 *
 67.50   42 *
 70.00   54 *
 72.50   33 *
 75.00   33 *
 77.50   32 *
 80.00   72 *
 82.50   74 *
 85.00   49 *
 87.50   13 *
 90.00    4 *
 92.50   21 *
 95.00   86 *
 97.50 6911 ************************************************************



From sjoerd@acm.org  Thu Sep 19 10:54:08 2002
From: sjoerd@acm.org (Sjoerd Mullender)
Date: Thu, 19 Sep 2002 11:54:08 +0200
Subject: Third result ... RE: [Spambayes] First result from Gary
	Robinson's ideas
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEDABFAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCGEDABFAB.tim.one@comcast.net> 
Message-ID: <200209190954.g8J9s8m07379@indus.ins.cwi.nl>

On Thu, Sep 19 2002 Tim Peters wrote:

> I've checked in sane changes to the code base now, so that you can try Gary
> Robinson's increasingly remarkable probability combining scheme (it was
> amazing when I first tried it, and triply so when Neale reported isomorphic
> results on a wholly different corpus).  Just set
> 
>     [Classifier]
>     use_robinson_probability: True
> 
>     [TestDriver]
>     spam_cutoff: 0.50

Here are my results.  run1 was default, run2 with the above settings.

By the way, I'm using runtest.sh, so I guess I'm the number two and
not Guido.  :-)

run1s -> run2s
-> <stat> tested 159 hams & 159 spams against 636 hams & 636 spams
-> <stat> tested 159 hams & 159 spams against 636 hams & 636 spams
-> <stat> tested 159 hams & 159 spams against 636 hams & 636 spams
-> <stat> tested 159 hams & 159 spams against 636 hams & 636 spams
-> <stat> tested 159 hams & 159 spams against 636 hams & 636 spams
-> <stat> tested 159 hams & 159 spams against 636 hams & 636 spams
-> <stat> tested 159 hams & 159 spams against 636 hams & 636 spams
-> <stat> tested 159 hams & 159 spams against 636 hams & 636 spams
-> <stat> tested 159 hams & 159 spams against 636 hams & 636 spams
-> <stat> tested 159 hams & 159 spams against 636 hams & 636 spams

false positive percentages
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied

won   0 times
tied  5 times
lost  0 times

total unique fp went from 0 to 0 tied
mean fp % went from 0.0 to 0.0 tied

false negative percentages
    2.516  2.516  tied
    0.000  0.000  tied
    1.258  1.258  tied
    2.516  2.516  tied
    1.258  1.258  tied

won   0 times
tied  5 times
lost  0 times

total unique fn went from 12 to 12 tied
mean fn % went from 1.50943396226 to 1.50943396226 tied

The before case:

Ham distribution for all runs:
* = 14 items
  0.00 794 *********************************************************
  2.50   1 *
  5.00   0 
  7.50   0 
 10.00   0 
 12.50   0 
 15.00   0 
 17.50   0 
 20.00   0 
 22.50   0 
 25.00   0 
 27.50   0 
 30.00   0 
 32.50   0 
 35.00   0 
 37.50   0 
 40.00   0 
 42.50   0 
 45.00   0 
 47.50   0 
 50.00   0 
 52.50   0 
 55.00   0 
 57.50   0 
 60.00   0 
 62.50   0 
 65.00   0 
 67.50   0 
 70.00   0 
 72.50   0 
 75.00   0 
 77.50   0 
 80.00   0 
 82.50   0 
 85.00   0 
 87.50   0 
 90.00   0 
 92.50   0 
 95.00   0 
 97.50   0 

Spam distribution for all runs:
* = 14 items
  0.00  12 *
  2.50   0 
  5.00   0 
  7.50   0 
 10.00   0 
 12.50   0 
 15.00   0 
 17.50   0 
 20.00   0 
 22.50   0 
 25.00   0 
 27.50   0 
 30.00   0 
 32.50   0 
 35.00   0 
 37.50   0 
 40.00   0 
 42.50   0 
 45.00   0 
 47.50   0 
 50.00   0 
 52.50   0 
 55.00   0 
 57.50   0 
 60.00   0 
 62.50   0 
 65.00   0 
 67.50   0 
 70.00   0 
 72.50   0 
 75.00   0 
 77.50   0 
 80.00   0 
 82.50   0 
 85.00   0 
 87.50   0 
 90.00   0 
 92.50   0 
 95.00   0 
 97.50 783 ********************************************************

The after case:

Ham distribution for all runs:
* = 14 items
  0.00 787 *********************************************************
  2.50   0 
  5.00   0 
  7.50   0 
 10.00   0 
 12.50   0 
 15.00   1 *
 17.50   2 *
 20.00   0 
 22.50   2 *
 25.00   0 
 27.50   1 *
 30.00   0 
 32.50   0 
 35.00   0 
 37.50   1 *
 40.00   0 
 42.50   0 
 45.00   0 
 47.50   1 *
 50.00   0 
 52.50   0 
 55.00   0 
 57.50   0 
 60.00   0 
 62.50   0 
 65.00   0 
 67.50   0 
 70.00   0 
 72.50   0 
 75.00   0 
 77.50   0 
 80.00   0 
 82.50   0 
 85.00   0 
 87.50   0 
 90.00   0 
 92.50   0 
 95.00   0 
 97.50   0 

Spam distribution for all runs:
* = 12 items
  0.00   1 *
  2.50   0 
  5.00   0 
  7.50   0 
 10.00   0 
 12.50   0 
 15.00   0 
 17.50   0 
 20.00   0 
 22.50   0 
 25.00   2 *
 27.50   1 *
 30.00   1 *
 32.50   0 
 35.00   0 
 37.50   3 *
 40.00   0 
 42.50   2 *
 45.00   2 *
 47.50   0 
 50.00   2 *
 52.50   2 *
 55.00   6 *
 57.50   7 *
 60.00   6 *
 62.50   6 *
 65.00   5 *
 67.50   6 *
 70.00  11 *
 72.50   4 *
 75.00   1 *
 77.50   5 *
 80.00   5 *
 82.50   6 *
 85.00   5 *
 87.50   3 *
 90.00   1 *
 92.50   0 
 95.00   2 *
 97.50 700 ***********************************************************

-- Sjoerd Mullender <sjoerd@acm.org>

From anthony@interlink.com.au  Thu Sep 19 11:33:39 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Thu, 19 Sep 2002 20:33:39 +1000
Subject: [Spambayes] Options ignore_redundant_html is ignored :)
Message-ID: <200209191033.g8JAXdv13809@localhost.localdomain>



Is there a reason for the option 'ignore_redundant_html' to be utterly
ignored and default to 'True'? 

The following patch fixes this, I haven't checked it in, as it's late and
there might be a reason for not doing this.

RCS file: /cvsroot/spambayes/spambayes/tokenizer.py,v
retrieving revision 1.16
diff -u -r1.16 tokenizer.py
--- tokenizer.py        11 Sep 2002 06:58:03 -0000      1.16
+++ tokenizer.py        19 Sep 2002 10:26:37 -0000
@@ -526,7 +526,10 @@
         elif part.get_content_maintype() == 'text':
             text.add(part)
 
-    return text - redundant_html
+    if options.ignore_redundant_html:
+       return text - redundant_html
+    else:
+       return text
 
 url_re = re.compile(r"""
     (https? | ftp)  # capture the protocol

This allowed me to see how parsing the HTML helped:

false positive percentages
    0.278  0.334  lost   +20.14%
    0.278  0.389  lost   +39.93%
    0.278  0.278  tied
    0.390  0.445  lost   +14.10%
    0.334  0.389  lost   +16.47%

won   0 times
tied  1 times
lost  4 times

total unique fp went from 28 to 33 lost   +17.86%
mean fp % went from 0.311500504796 to 0.367124047433 lost   +17.86%

false negative percentages
    0.647  0.517  won    -20.09%
    0.517  0.323  won    -37.52%
    0.969  0.711  won    -26.63%
    0.905  0.840  won     -7.18%
    0.906  0.712  won    -21.41%

won   5 times
tied  0 times
lost  0 times

total unique fn went from 61 to 48 won    -21.31%
mean fn % went from 0.788815790048 to 0.620715037922 won    -21.31%

Anthony
--
Anthony Baxter     <anthony@interlink.com.au>
It's never too late to have a happy childhood.


From anthony@interlink.com.au  Thu Sep 19 12:29:46 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Thu, 19 Sep 2002 21:29:46 +1000
Subject: [Spambayes] well, I'm confused...
Message-ID: <200209191129.g8JBTks14499@localhost.localdomain>


Is there something specifically offensive about canadians? a spam 
that showed up in the middle of the pack in the robinson test opens
with:

  Being Canadian, we sincerely hope  we haven't offended
  you by sending this email



From gward@python.net  Thu Sep 19 14:04:24 2002
From: gward@python.net (Greg Ward)
Date: Thu, 19 Sep 2002 09:04:24 -0400
Subject: [Spambayes] idea for tokenizer.crack_filename change
In-Reply-To: <w53ofauzxbs.fsf@woozle.org>
References: <w53ofauzxbs.fsf@woozle.org>
Message-ID: <20020919130424.GA18260@cthulhu.gerg.ca>

On 18 September 2002, Neale Pickett said:
> In going over some of my spam, I was surprised to see that the following
> wasn't penalized:
> 
>   ------=_NextPart_000_0039_0173A692.99A692D0
>   Content-Type: application/octet-stream; name="Video.pif"
>   Content-Transfer-Encoding: base64
>   Content-Disposition: attachment; filename="Video.pif"

That's almost certainly a virus.  Look at the base64-decoded body --
I betcha it starts with "MZ", which is a DOS/Windows executable.

Spam detectors should not be distracted by trying to detect viruses.
It's OK if spam detector happens to catch some viruses, but -- apart
from some similarities in the headers and the MIME strucuture -- spam
and viruses tend to be very different beasts.  My gut instinct says
viruses should not be included in the "spam" corpus for training a spam
detector -- too many paths to follow, too ambiguous, or something like
that.

However, detecting viruses is pretty damn easy.  See

  http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/elspy/elspy/lib/execontent_simple.py?rev=1.6&content-type=text/vnd.viewcvs-markup

for the code used on mail.python.org and starship.python.net.  In a
nutshell, this code scans the message body for

  /^content-(type|disposition):.*(file)?name=.*\.(\w+)/

and, if the extension that matches that final \w+ is one of
("exe", "com", "vbs", "pif", ...), then you reject the message.
Shelling out big bucks for a virus scanner is kinda silly if
you ask me.

Back to our regularly scheduled programming...

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Rules for Urban Cycling, #1:
Green means go; yellow means go like hell; red means proceed with caution.

From skip@pobox.com  Thu Sep 19 14:16:49 2002
From: skip@pobox.com (Skip Montanaro)
Date: Thu, 19 Sep 2002 08:16:49 -0500
Subject: [Spambayes] Need help getting started
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKECKBFAB.tim.one@comcast.net>
References: <200209190321.g8J3Lti28601@pcp02138704pcs.reston01.va.comcast.net>
        <LNBBLJKPBEHFEDALKOLCKECKBFAB.tim.one@comcast.net>
Message-ID: <15753.52801.854041.964397@12-248-11-90.client.attbi.com>


    Tim> rebal.py, which can be used to reshuffle messages among directories
    Tim> (to even out the # of msgs in each initially, and to replenish them
    Tim> from "the reservoir" as you find spam in your ham, & vice versa --
    Tim> there are hams in bruceg's spam collections!).

Dunno if it's worth mentioning or not, but ...  When you discover ham in
your spam (or vice versa) you should move such messages to the correct
reservoir instead of simply deleting them.  No sense wasting good data.

Skip

From gward@python.net  Thu Sep 19 14:19:35 2002
From: gward@python.net (Greg Ward)
Date: Thu, 19 Sep 2002 09:19:35 -0400
Subject: [Spambayes] well, I'm confused...
In-Reply-To: <200209191129.g8JBTks14499@localhost.localdomain>
References: <200209191129.g8JBTks14499@localhost.localdomain>
Message-ID: <20020919131935.GB18260@cthulhu.gerg.ca>

On 19 September 2002, Anthony Baxter said:
> 
> Is there something specifically offensive about canadians? a spam 
> that showed up in the middle of the pack in the robinson test opens
> with:
> 
>   Being Canadian, we sincerely hope  we haven't offended
>   you by sending this email

Obligatory Canuck jokes... Neil, step in any time...

1) How do you spot a Canadian?  He's the one who apologizes when you step on
   his foot.

2) How do you get 60 rowdy, drunken Canadians out of a swimming pool on a
   hot summer day?  "OK, guys, it's time to get out of the pool now..."

3) A Canadian is someone who boasts about what a self-deprecating
   sense of humour he has.

If you don't get these, you're probably not from around here...

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
A hairstyle's not a lifestyle / Imagine Sid Vicious at 35
    -- Dead Kennedys

From anthony@interlink.com.au  Thu Sep 19 14:21:12 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Thu, 19 Sep 2002 23:21:12 +1000
Subject: [Spambayes] Need help getting started 
In-Reply-To: <15753.52801.854041.964397@12-248-11-90.client.attbi.com> 
Message-ID: <200209191321.g8JDLDR31437@localhost.localdomain>


>>> Skip Montanaro wrote
> Dunno if it's worth mentioning or not, but ...  When you discover ham in
> your spam (or vice versa) you should move such messages to the correct
> reservoir instead of simply deleting them.  No sense wasting good data.

Particularly since usually the reason it's in the wrong bucket is because
it's some ugly edge case that makes it extra useful for testing.

From glouis@dynamicro.on.ca  Thu Sep 19 14:38:46 2002
From: glouis@dynamicro.on.ca (Greg Louis)
Date: Thu, 19 Sep 2002 09:38:46 -0400
Subject: [Spambayes] further test of Gary's ideas
Message-ID: <20020919133845.GA3992@athame.dynamicro.on.ca>

Gary's essay at
http://radio.weblogs.com/0101454/stories/2002/09/16/spamDetection.html
now contains a link to some results I got testing his proposals on a
training database of 1773 spams and 5106 non-spams.  In that report I
promised to follow up with a test of some new messages.  The results
may interest readers of this list who've been following the discussion
of Gary's ideas.

For this test I implemented Gary's S and f(w) calculations as he
describes, but not the g(w) modification.  I used a value of 0.37615
for x (empty precision, I know) that I calculated from the training
database.  The test corpus consisted of 111 spam messages and 144
non-spam ones, none of which had been used for training.

I've been playing with bogofilter (not spambayes, sorry) version 0.7,
modified to ignore all words with individual probabilities between 0.1
and 0.9 (not inclusive).  The modification, as I reported in the
letters Gary posted, improves bogofilter's false-negative count on my
training data from 6.6% to 3.0%.

On today's test data, my "ignore-small-extrema" bogofilter produced 7
false negatives (6.3%) and one false positive (0.7%).  Unmodified,
bogofilter gave 9 false negatives and the same false positive.  Gary's
calculation gave one false negative (0.9%) and the same false positive.
It continues to appear as though Gary's approach does well, on my
emails anyway.

-- 
| G r e g  L o u i s          | gpg public key:      |
|   http://www.bgl.nu/~glouis |   finger greg@bgl.nu |

From robertoschler@hotmail.com  Thu Sep 19 15:14:46 2002
From: robertoschler@hotmail.com (Robert Oschler)
Date: Thu, 19 Sep 2002 10:14:46 -0400
Subject: [Spambayes] [OT] Musings on recent spam content
Message-ID: <OE22nDtKPIrFz3T4nwF00003479@hotmail.com>

This is a multi-part message in MIME format.

---------------------- multipart/alternative attachment
I've noticed that the dominant theme for spam recently is mortage rates. =
 I wonder if this is because of the new spam filters that my ISP has =
implemented.  Possibly words like 'mortage, interest rates' etc. don't =
trigger any of their spam filters?  Just a musing.

thx


---------------------- multipart/alternative attachment
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail-21/spambayes/attachments/20020919/11430956/attachment.html

---------------------- multipart/alternative attachment--

From gward@python.net  Thu Sep 19 15:28:46 2002
From: gward@python.net (Greg Ward)
Date: Thu, 19 Sep 2002 10:28:46 -0400
Subject: [Spambayes] Spam or virus?
Message-ID: <20020919142846.GA18533@cthulhu.gerg.ca>

Hi all --

I've been going over the pile of email collected on mail.python.org over
the last week, and found a message that I can't figure out.  I'm pretty
sure it's either spam or a virus, because all it consists of is a
PowerPoint attachment.  Can someone take a look and give me your
opinion?

The whole message is here:
  http://mail.python.org/corpus/shock-msg

and the attachment itself is here:
  http://mail.python.org/corpus/shock.pps

BE CAREFUL!!  This could very well be a virus-infected PowerPoint
document.  When I tried to load it into AbiWord on my Linux box, it
dumped core; kpresenter said it couldn't handle it either.  It could be
that only genuine Microsoft bugware can handle it.

Thanks --

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Never put off till tomorrow what you can avoid all together.

From grobinson@transpose.com  Thu Sep 19 15:35:04 2002
From: grobinson@transpose.com (Gary Robinson)
Date: Thu, 19 Sep 2002 10:35:04 -0400
Subject: [Spambayes] Advantages of S?
Message-ID: <B9AF58D8.1621E%grobinson@transpose.com>

At present, it appears that S may perform better than relatively untweaked
graham (as I guess bogofilter is using), but about the same as tweaked
graham as spambayes is using, but has the advantage of showing a spread of
results.

As I see it there are two possible advantages to the spread:

1) emails in the middle could be classified as "you should manually check
these", or

2) the user could adjust for the level of sensitivity he wants (similar to
what happens now in Entourage which is otherwise brain-dead as a spam
filter).

(1) is a attractive but one thing worries me: it may give people a false
sense of security -- they may think they only have to look at the middle
ones, but occasionally even extreme cases of seeming to be spam might be
legit. On the other hand, on those rare occasions, the sender can always try
again if he doesn't get a response! ;)


Just a couple of thoughts.


--Gary


-- 
Gary Robinson
CEO
Transpose, LLC
grobinson@transpose.com
207-942-3463
http://www.emergentmusic.com
http://radio.weblogs.com/0101454



From anthony@interlink.com.au  Thu Sep 19 15:43:35 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Fri, 20 Sep 2002 00:43:35 +1000
Subject: Third result ... RE: [Spambayes] First result from Gary
	Robinson's ideas 
Message-ID: <200209191443.g8JEhZV32694@localhost.localdomain>


>>> Anthony Baxter wrote
> total unique fp went from 33 to 38 lost   +15.15%
> mean fp % went from 0.366932315011 to 0.42251875192 lost   +15.15%

> total unique fn went from 55 to 47 won    -14.55%
> mean fn % went from 0.710811726238 to 0.60736899408 won    -14.55%

In private conversation with Gary, he suggested I post a reminder about
my particular test corpus. It's considerably more brutal a test case, as
it's 3 and a bit years worth of email to the contact email addresses for 
the company I work at. So far it's got 9,000 ham and 7,500 spam, there's 
another 12,000 messages that remain to be categorised. 

This means we get email from customers complaining about credit card charges, 
people forwarding spam messages to complain, genuine sales enquiries or 
reseller enquiries, all sorts of things that drive spamassassin mental. 

It turns out that Gary's approach is sensitive to small numbers of very
spammy words (his description) and that this is why I'm getting worse
f-p results...

[gary - sorry for the false alarm - it's actually something with the mail
setup on my laptop that's causing email directly from me to you to bounce.
working on it now...]

Anthony

From glouis@dynamicro.on.ca  Thu Sep 19 15:53:15 2002
From: glouis@dynamicro.on.ca (Greg Louis)
Date: Thu, 19 Sep 2002 10:53:15 -0400
Subject: [Spambayes] Advantages of S?
In-Reply-To: <B9AF58D8.1621E%grobinson@transpose.com>
References: <B9AF58D8.1621E%grobinson@transpose.com>
Message-ID: <20020919145314.GA8615@athame.dynamicro.on.ca>

On 20020919 (Thu) at 1035:04 -0400, Gary Robinson wrote:
> At present, it appears that S may perform better than relatively untweaked
> graham (as I guess bogofilter is using)

Yes, bogofilter 0.7 (haven't looked at the latest release yet) is using
untweaked Graham.

> 1) emails in the middle could be classified as "you should manually check
> these", or
> 
> (1) is a attractive but one thing worries me: it may give people a false
> sense of security -- they may think they only have to look at the middle
> ones, but occasionally even extreme cases of seeming to be spam might be
> legit.

Yup.  In the test I reported earler, the one false positive had an S
value of 0.93 or thereabouts.  It was a message to a bogofilter mailing
list and looked extremely innocuous except for a couple of artefacts in
the training database (words that ought to have been seen in many
messages but actually showed up predominantly in spam).  Ironically,
one of these was "fetchmail-5.9.14" -- I'd been adding a lot of spam
to the corpus since that version came out :)  Shows how easy it is to
bias this kind of classification; you really need to feed it similar
numbers of similarly-dated messages in order to get good training, and
shortcuts can be dangerous...

-- 
| G r e g  L o u i s          | gpg public key:      |
|   http://www.bgl.nu/~glouis |   finger greg@bgl.nu |

From gward@python.net  Thu Sep 19 15:56:18 2002
From: gward@python.net (Greg Ward)
Date: Thu, 19 Sep 2002 10:56:18 -0400
Subject: [Spambayes] Spam or virus?
In-Reply-To: <20020919142846.GA18533@cthulhu.gerg.ca>
References: <20020919142846.GA18533@cthulhu.gerg.ca>
Message-ID: <20020919145618.GA18612@cthulhu.gerg.ca>

> The whole message is here:
>   http://mail.python.org/corpus/shock-msg
> 
> and the attachment itself is here:
>   http://mail.python.org/corpus/shock.pps

Matt Sergeant answered my question offline.  I wish I hadn't asked.
There are some *sick* people out there.  I have removed this file --
don't anyone else waste your time (or risk losing your lunch) on it.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
"... but in the town it was well known that when they got home their fat and
psychopathic wives would thrash them to within inches of their lives ..."

From tim.one@comcast.net  Thu Sep 19 16:11:18 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 19 Sep 2002 11:11:18 -0400
Subject: [Spambayes] Options ignore_redundant_html is ignored :)
Message-ID: <3e6303ba03.3ba033e630@icomcast.net>

[Anthony Baxter]
> Is there a reason for the option 'ignore_redundant_html' to be utterly
> ignored and default to 'True'?

Are you sure you're not hallucinating on both counts?  "defaults" in 
Options.py contains

ignore_redundant_html: False

and tokenizer.py uses it at the top (module) level to define the 
textparts() function in an appropriate way:

if options.ignore_redundant_html:
    def textparts(msg):
...
else:
    # Use all text parts.  If a text/plain and text/html part happen to
    # have redundant content, so it goes.
    def textparts(msg):
...

> The following patch fixes this, I haven't checked it in, as it's 
> late and there might be a reason for not doing this.

Please don't change a thing until you figure out how the acid got in 
your water supply <wink>.

> This allowed me to see how parsing the HTML helped:

Perhaps you had a bayecustomize.ini sitting around that forced it to 
True?  Options are a necessary evil at this point.


From alex@spamcop.net  Thu Sep 19 16:07:18 2002
From: alex@spamcop.net (Alex Bischoff)
Date: Thu, 19 Sep 2002 10:07:18 -0500
Subject: [Spambayes] SpamBayes: announce mailing list?
Message-ID: <3D89E826.6050305@spamcop.net>

Hey there,

I heard about the project through Mozilla bug 163188 
(http://bugzilla.mozilla.org/show_bug.cgi?id=163188). And, I see that 
there's a discussion mailing list and a cvs-checkin mailing list for the 
project.

However, how about an "announce" list for new versions or other news? 
I'm just an ordinary user and, while a full discussion list may be more 
than I need, an announce-list would be just right :).

-- 
   Alex Bischoff                                          3812078 on ICQ
-------------------------------------------------------------------------
export-a-crypto-system-sig RSA-2-lines-PERL
print pack"C*",split/\D+/,`echo "16iII*o\U@{$/=$z;[(pop,pop,unpack"H*",<>
)]}\EsMsKsN0[lN*1lK[d2%Sa2/d0<X+d*lMLa^*lN%0]dsXx++lMlN/dsM0<J]dsJxp"|dc`
-------------------------------------------------------------------------
   Proof That There Is Strength in Numbers --> http://www.distributed.net


From barry@zope.com  Thu Sep 19 16:27:31 2002
From: barry@zope.com (Barry A. Warsaw)
Date: Thu, 19 Sep 2002 11:27:31 -0400
Subject: [Spambayes] Change to this mailing list
Message-ID: <15753.60643.611994.168844@anthem.wooz.org>


I'm making a small change to this mailing list.  You guys /think/
you're working on spam detection software but you're really just
guinea pigs for Mailman 2.1 :)

I've turned on content filtering and will be stripping html parts.
Then, any multipart/alternative that has both a plain text and an html
part will get transformed into just the plain text part.  Since this
is a list about email, Tim should know exactly what I mean.

Let me know if you notice any problems, although I'll be watching too.

-Barry

From tim.one@comcast.net  Thu Sep 19 16:29:53 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 19 Sep 2002 11:29:53 -0400
Subject: [Spambayes] RE: [Spambayes-checkins] spambayes Options.py,1.19,1.20
In-Reply-To: <E17rxi7-0005Kx-00@usw-pr-cvs1.sourceforge.net>
Message-ID: <BIEJKCLHCIOIHAGOKOLHEEMLDKAA.tim.one@comcast.net>

[Anthony Baxter]
> Modified Files:
> 	Options.py
> Log Message:
> if it exists, load options from file(s) specified in env var
> BAYESCUSTOMIZE rather than bayescustomize.ini. Much more convenient.
> ...

That's a good change!  What would the testers find *most* convenient?  The
unwieldy bayescustomize.ini is so named only because there *used* to be a
bayes.ini too.  If someone wants to make option-processing heavenly instead
of merely bearable, please do, without regard for backward compatibility --
in real life, I eat the same thing for lunch every day, even though I don't
like it <wink>.


From nas@python.ca  Thu Sep 19 17:36:57 2002
From: nas@python.ca (Neil Schemenauer)
Date: Thu, 19 Sep 2002 09:36:57 -0700
Subject: [Spambayes] First result from Gary Robinson's ideas
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKENMBEAB.tim.one@comcast.net>
References: <B9AD2F82.16038%grobinson@transpose.com>
	<LNBBLJKPBEHFEDALKOLCKENMBEAB.tim.one@comcast.net>
Message-ID: <20020919163657.GA28786@glacier.arctrix.com>

Here's my result:

bases -> robinsons
-> <stat> tested 300 hams & 300 spams against 1500 hams & 1500 spams
-> <stat> tested 300 hams & 300 spams against 1500 hams & 1500 spams
-> <stat> tested 300 hams & 300 spams against 1500 hams & 1500 spams
-> <stat> tested 300 hams & 300 spams against 1500 hams & 1500 spams
-> <stat> tested 300 hams & 300 spams against 1500 hams & 1500 spams
-> <stat> tested 300 hams & 300 spams against 1500 hams & 1500 spams
-> <stat> tested 300 hams & 300 spams against 1500 hams & 1500 spams
-> <stat> tested 300 hams & 300 spams against 1500 hams & 1500 spams
-> <stat> tested 300 hams & 300 spams against 1500 hams & 1500 spams
-> <stat> tested 300 hams & 300 spams against 1500 hams & 1500 spams
-> <stat> tested 300 hams & 300 spams against 1500 hams & 1500 spams
-> <stat> tested 300 hams & 300 spams against 1500 hams & 1500 spams

false positive percentages
    0.667  0.667  tied
    0.000  0.000  tied
    1.000  1.000  tied
    0.333  0.333  tied
    0.000  0.000  tied
    0.000  0.000  tied

won   0 times
tied  6 times
lost  0 times

total unique fp went from 6 to 6 tied
mean fp % went from 0.333333333333 to 0.333333333333 tied

false negative percentages
    0.333  0.333  tied
    1.333  1.000  won    -24.98%
    1.667  1.667  tied
    0.333  0.000  won   -100.00%
    1.333  1.000  won    -24.98%
    1.667  0.667  won    -59.99%

won   4 times
tied  2 times
lost  0 times

total unique fn went from 20 to 14 won    -30.00%
mean fn % went from 1.11111111111 to 0.777777777778 won    -30.00%

From tim.one@comcast.net  Thu Sep 19 18:02:02 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 19 Sep 2002 13:02:02 -0400
Subject: [Spambayes] initial cut at website is up.
In-Reply-To: <200209190849.g8J8niv12565@localhost.localdomain>
Message-ID: <BIEJKCLHCIOIHAGOKOLHOEMPDKAA.tim.one@comcast.net>

[Anthony Baxter]
> Ok, there's a first cut at a website at spambayes.sf.net -

Bless you, Anthony -- this is great!  The collection of links does a
wonderful job of pointing into the horridly scattered history of this
project.

> ...
> If you're one of those steenkin windows users <wink> I'm sure there's
> an equivalent technique.

Not to worry.  Unlike Linux babies, Windows users solve their own problems
without public crying <wink>.


From tim.one@comcast.net  Thu Sep 19 18:08:35 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 19 Sep 2002 13:08:35 -0400
Subject: [Spambayes] Need help getting started
In-Reply-To: <15753.52801.854041.964397@12-248-11-90.client.attbi.com>
Message-ID: <BIEJKCLHCIOIHAGOKOLHGENADKAA.tim.one@comcast.net>

[Skip Montanaro]
> Dunno if it's worth mentioning or not, but ...  When you discover ham in
> your spam (or vice versa) you should move such messages to the correct
> reservoir instead of simply deleting them.  No sense wasting good data.

Even more, I suggest moving them into the correct category of your "live"
test data.  Randomness means "slice of life" in all possible respects, and
things that some other system misclassified (whether that system be
SpamAssassin or your own eyeballs doesn't matter) should appear in the
training data in the same proportion as they occur in real life.
Systematically moving misclassifications into the normally passive
reservoirs is a distortion of real life, just as feeding the system *only*
misclassifications would be.


From neale@woozle.org  Thu Sep 19 19:43:50 2002
From: neale@woozle.org (Neale Pickett)
Date: 19 Sep 2002 11:43:50 -0700
Subject: [Spambayes] idea for tokenizer.crack_filename change
In-Reply-To: <20020919130424.GA18260@cthulhu.gerg.ca>
References: <w53ofauzxbs.fsf@woozle.org>
	<20020919130424.GA18260@cthulhu.gerg.ca>
Message-ID: <w537khhstuh.fsf@woozle.org>

So then, Greg Ward <gward@python.net> is all like:

> Spam detectors should not be distracted by trying to detect viruses.

But it's such an easy freebie!  At least, I think it should be :)  All
we have to do is tokenize on the extension (which we're doing) and then
".pif" becomes a killer-strong spam indicator.  content-type
"application/octet-stream" is already at like .83 in my list.

Anyway, I'm not seeing a lot of virus in my FPs or FNs--just that one,
actually, which in my mind is evidence that it's already doing a pretty
good job classifying virus.  Nimda, at least, which seems to be about
the only virus in my corpus.

Neale

From neale@woozle.org  Thu Sep 19 19:59:03 2002
From: neale@woozle.org (Neale Pickett)
Date: 19 Sep 2002 11:59:03 -0700
Subject: Third result ... RE: [Spambayes] First result from Gary
	Robinson's ideas
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEDABFAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCGEDABFAB.tim.one@comcast.net>
Message-ID: <w533cs5st54.fsf@woozle.org>

So then, Tim Peters <tim.one@comcast.net> is all like:

> That is, the email parser barfed on the malformed "To" header, so
> "MessageParseError" was the *only* token generated, and that didn't
> appear in the training set at all (so it got our "unknown word prob").
> Note that something is rated spam only if its score is strictly
> greater than spam_cutoff, and 0.5 isn't greater than 0.5 <wink>.  I
> (or someone else -- please <wink>?) should probably change the
> tokenizer to back off to the raw message body when the email parser
> gives up.

I'm still getting my email.Message legs.  How does this look as a first
cut?

--- tokenizer.py        17 Sep 2002 17:57:39 -0000      1.23
+++ tokenizer.py        19 Sep 2002 18:57:42 -0000
@@ -3,6 +3,7 @@
 import email
 import re
 from sets import Set
+from email.MIMEText import MIMEText
 
 from Options import options
 
@@ -839,18 +840,16 @@
         else:
             # Create an email Message object.
             try:
-                if hasattr(obj, "readline"):
-                    return email.message_from_file(obj)
-                else:
-                    return email.message_from_string(obj)
+                if hasattr(obj, "read"):
+                    obj = obj.read()
+                return email.message_from_string(obj)
             except email.Errors.MessageParseError:
-                return None
+                return MIMEText(obj)
 
     def tokenize(self, obj):
         msg = self.get_message(obj)
         if msg is None:
             yield 'control: MessageParseError'
-            # XXX Fall back to the raw body text?
             return
 
         for tok in self.tokenize_headers(msg):

From tim.one@comcast.net  Thu Sep 19 20:22:11 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 19 Sep 2002 15:22:11 -0400
Subject: Third result ... RE: [Spambayes] First result from Gary
	Robinson'sideas
In-Reply-To: <200209190944.g8J9inS13207@localhost.localdomain>
Message-ID: <BIEJKCLHCIOIHAGOKOLHAENFDKAA.tim.one@comcast.net>

[Anthony Baxter]
> >     [Classifier]
> >     use_robinson_probability: True
> >
> >     [TestDriver]
> >     spam_cutoff: 0.50

> false positive percentages
>     0.334  0.445  lost   +33.23%
>     0.278  0.334  lost   +20.14%
>     0.389  0.445  lost   +14.40%
>     0.556  0.556  tied
>     0.278  0.334  lost   +20.14%
>
> won   0 times
> tied  1 times
> lost  4 times
>
> total unique fp went from 33 to 38 lost   +15.15%
> mean fp % went from 0.366932315011 to 0.42251875192 lost   +15.15%
>
> false negative percentages
>     0.582  0.452  won    -22.34%
>     0.775  0.646  won    -16.65%
>     0.710  0.710  tied
>     0.581  0.581  tied
>     0.906  0.647  won    -28.59%
>
> won   3 times
> tied  2 times
> lost  0 times
>
> total unique fn went from 55 to 47 won    -14.55%
> mean fn % went from 0.710811726238 to 0.60736899408 won    -14.55%

Note that for both rates, the "after" run is more eager to call things spam,
and that I had no particular reason to set spam_cutoff to 0.500 exactly
(something "near" 0.5 was clearly best for my first test).

One dead simple way to counteract a tendency to call things spam is simply
to raise the spam_cutoff threshold.  Your second-run score histograms show
that boosting spam_cutoff from 0.500 to 0.525 would have saved 9 false
positives, and added 10 false negatives (using your second-run counts as the
baseline).  Then (using your first-run counts as the baseline) your total
unique fp would have fallen from 33 to 29, and your total unique fn risen
from 55 to 57, and this would be consistent with previous "no real
difference in overall results, but there's an exploitable middle ground"
reports.  OTOH, maybe 0.525016932 would be better still <wink>.

Thanks for the report, Anthony!


From skip@pobox.com  Thu Sep 19 20:27:57 2002
From: skip@pobox.com (Skip Montanaro)
Date: Thu, 19 Sep 2002 14:27:57 -0500
Subject: [Spambayes] Need help getting started
In-Reply-To: <BIEJKCLHCIOIHAGOKOLHGENADKAA.tim.one@comcast.net>
References: <15753.52801.854041.964397@12-248-11-90.client.attbi.com>
        <BIEJKCLHCIOIHAGOKOLHGENADKAA.tim.one@comcast.net>
Message-ID: <15754.9533.56654.176729@12-248-11-90.client.attbi.com>


    Tim> Even more, I suggest moving them into the correct category of your
    Tim> "live" test data.  Randomness means "slice of life" in all possible
    Tim> respects, and things that some other system misclassified (whether
    Tim> that system be SpamAssassin or your own eyeballs doesn't matter)
    Tim> should appear in the training data in the same proportion as they
    Tim> occur in real life.  Systematically moving misclassifications into
    Tim> the normally passive reservoirs is a distortion of real life, just
    Tim> as feeding the system *only* misclassifications would be.

Good point.  I was thinking about the potential loss of data, but moving the
misclassified message to the right Set? directory and running rebal.py would
be better.

Skip


From tim.one@comcast.net  Thu Sep 19 21:00:09 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 19 Sep 2002 16:00:09 -0400
Subject: Third result ... RE: [Spambayes] First result from Gary
	Robinson'sideas
In-Reply-To: <200209190954.g8J9s8m07379@indus.ins.cwi.nl>
Message-ID: <BIEJKCLHCIOIHAGOKOLHIENGDKAA.tim.one@comcast.net>

[Sjoerd Mullender]
> >     [Classifier]
> >     use_robinson_probability: True
> >
> >     [TestDriver]
> >     spam_cutoff: 0.50

> Here are my results.  run1 was default, run2 with the above settings.
>
> By the way, I'm using runtest.sh, so I guess I'm the number two and
> not Guido.  :-)
>
> run1s -> run2s
> -> <stat> tested 159 hams & 159 spams against 636 hams & 636 spams
...
>
> false positive percentages
>     0.000  0.000  tied
>     0.000  0.000  tied
>     0.000  0.000  tied
>     0.000  0.000  tied
>     0.000  0.000  tied
>
> won   0 times
> tied  5 times
> lost  0 times
>
> total unique fp went from 0 to 0 tied
> mean fp % went from 0.0 to 0.0 tied
>
> false negative percentages
>     2.516  2.516  tied
>     0.000  0.000  tied
>     1.258  1.258  tied
>     2.516  2.516  tied
>     1.258  1.258  tied
>
> won   0 times
> tied  5 times
> lost  0 times
>
> total unique fn went from 12 to 12 tied
> mean fn % went from 1.50943396226 to 1.50943396226 tied

Thank you, Sjoerd.  Just noting that (according to the "after" histograms),
boosting spam_cutoff to 0.525 would have left your f-p rate unchanged but
added  2 false negatives to your f-n total.  In the other direction,
lowering it to 0.475 would have saved you 2 false negatives, but added your
first false positive.  The separation near 0.50 is clearly touchy, but it
remains pretty amazing how close 0.50 is to ideal for everyone who has tried
this.


From tim.one@comcast.net  Thu Sep 19 21:59:25 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 19 Sep 2002 16:59:25 -0400
Subject: Third result ... RE: [Spambayes] First result from Gary
	Robinson'sideas
In-Reply-To: <w533cs5st54.fsf@woozle.org>
Message-ID: <BIEJKCLHCIOIHAGOKOLHOENJDKAA.tim.one@comcast.net>

[Tim]
> I (or someone else -- please <wink>?) should probably change the
> tokenizer to back off to the raw message body when the email parser
> gives up.

[Neale Pickett]
> I'm still getting my email.Message legs.  How does this look as a first
> cut?

Thanks!  Some comments:

> --- tokenizer.py        17 Sep 2002 17:57:39 -0000      1.23
> +++ tokenizer.py        19 Sep 2002 18:57:42 -0000
> @@ -3,6 +3,7 @@
>  import email
>  import re
>  from sets import Set
> +from email.MIMEText import MIMEText

We imported email just a few lines above.  MIMEText isn't going to be
referenced enough to justify giving it an abbreviated name.

>  from Options import options
>
> @@ -839,18 +840,16 @@
>          else:
>              # Create an email Message object.
>              try:
> -                if hasattr(obj, "readline"):
> -                    return email.message_from_file(obj)
> -                else:
> -                    return email.message_from_string(obj)
> +                if hasattr(obj, "read"):
> +                    obj = obj.read()
> +                return email.message_from_string(obj)
>              except email.Errors.MessageParseError:
> -                return None
> +                return MIMEText(obj)

Barry suggested doing (and he wrote the email package, so this is a rare
case where we should listen to him <wink>):

    msg = email.Message.Message()
    msg.set_payload(obj)
    return msg

instead.  The difference is that MIMEText() makes up some headers out of
thin air (relative to the original malformed message), but a raw Message
object doesn't.

>      def tokenize(self, obj):
>          msg = self.get_message(obj)
>          if msg is None:
>              yield 'control: MessageParseError'
> -            # XXX Fall back to the raw body text?
>              return

There won't be a way for get_message to return None anymore, so also nuke
the code checking fot that.  Replacing it with

    assert msg is not None

would be OK, but we're hosed in any case then, and even without the assert
the code will raise a None-related exception soon anyway.

Piece o' cake, eh?

piece-o'-python-ly y'rs  - tim


From tim.one@comcast.net  Thu Sep 19 22:36:21 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 19 Sep 2002 17:36:21 -0400
Subject: [Spambayes] idea for tokenizer.crack_filename change
In-Reply-To: <w537khhstuh.fsf@woozle.org>
Message-ID: <BIEJKCLHCIOIHAGOKOLHGENLDKAA.tim.one@comcast.net>

[Greg Ward]
> Spam detectors should not be distracted by trying to detect viruses.

[Neale Pickett]
> But it's such an easy freebie!  At least, I think it should be :)

This is easier to test than to debate.

Everyone keep in mind there are different audiences here too.  Greg has a
cheap yet extremely effective virus trap in place on python.org (if you
prevent users from sharing any useful files, this is easy to do), so for his
purposes it's pure loss if the spambayes filter suffers at all by mixing
virus detection into its mission.  OTOH, Neale wants to filter all the email
for 15 Fortune 500 companies at once on his Palm Pilot in its spare cycles.
I'm sure you'll agree this is a fair characterization all around <wink>.

BTW, one of the first spam this system found in my ham collection was in
fact a virus.  But that's also the last one it found -- I don't know whether
that's because there aren't any more, or because there was only one.  When I
get a free nanosecond, I'll try running Greg's virus detector over the
corpus.


From nas@python.ca  Thu Sep 19 22:52:04 2002
From: nas@python.ca (Neil Schemenauer)
Date: Thu, 19 Sep 2002 14:52:04 -0700
Subject: [Spambayes] idea for tokenizer.crack_filename change
In-Reply-To: <BIEJKCLHCIOIHAGOKOLHGENLDKAA.tim.one@comcast.net>
References: <w537khhstuh.fsf@woozle.org>
	<BIEJKCLHCIOIHAGOKOLHGENLDKAA.tim.one@comcast.net>
Message-ID: <20020919215204.GA29913@glacier.arctrix.com>

Tim Peters wrote:
> When I get a free nanosecond, I'll try running Greg's virus detector
> over the corpus.

Here's a simple script that looks for executable attachments:

    http://arctrix.com/nas/find_executables.py

Run like this:

    python find_executables.py Data/Spam/*/*

If anyone is interested, I have a qmail filter that rejects messages
with executable attachments at SMTP-time.

  Neil

From skip@pobox.com  Thu Sep 19 23:07:30 2002
From: skip@pobox.com (Skip Montanaro)
Date: Thu, 19 Sep 2002 17:07:30 -0500
Subject: [Spambayes] idea for tokenizer.crack_filename change
In-Reply-To: <20020919215204.GA29913@glacier.arctrix.com>
References: <w537khhstuh.fsf@woozle.org>
        <BIEJKCLHCIOIHAGOKOLHGENLDKAA.tim.one@comcast.net>
        <20020919215204.GA29913@glacier.arctrix.com>
Message-ID: <15754.19106.478876.417637@12-248-11-90.client.attbi.com>


    Neil> Here's a simple script that looks for executable attachments:

    Neil>     http://arctrix.com/nas/find_executables.py

    Neil> Run like this:

    Neil>     python find_executables.py Data/Spam/*/*

    Neil> If anyone is interested, I have a qmail filter that rejects
    Neil> messages with executable attachments at SMTP-time.

It seems to me that base64-encoded, all DOS/Windows executables start with
(reciting from memory, since I've deleted all viruses and haven't received
any new ones in the last 15 minutes or so) "TPqAAA" or something similar.
Why rely on finding specific file extensions?  They can just change.

Skip


From neale@woozle.org  Thu Sep 19 23:25:27 2002
From: neale@woozle.org (Neale Pickett)
Date: 19 Sep 2002 15:25:27 -0700
Subject: [Spambayes] idea for tokenizer.crack_filename change
In-Reply-To: <BIEJKCLHCIOIHAGOKOLHGENLDKAA.tim.one@comcast.net>
References: <BIEJKCLHCIOIHAGOKOLHGENLDKAA.tim.one@comcast.net>
Message-ID: <w53k7lhr50o.fsf@woozle.org>

So then, Tim Peters <tim.one@comcast.net> is all like:

> [Greg Ward]
> > Spam detectors should not be distracted by trying to detect viruses.
> 
> [Neale Pickett]
> > But it's such an easy freebie!  At least, I think it should be :)
> 
> This is easier to test than to debate.

My corpus is too small to produce a meaningful test (please, no small
corpus jokes, I'm kind of sensitive about it).  But I'd love to see if
anyone else has good luck with it.

> OTOH, Neale wants to filter all the email for 15 Fortune 500 companies
> at once on his Palm Pilot in its spare cycles.

I wish.  We're using an Atari 2600 retrofitted with an Tulip-based NIC.
Getting the TCP stack to work in 40 bytes of RAM was tough, let me tell
you.

Neale

(The optional "Combat!" module is a really big seller with the more
fascist network admins.)

From anthony@interlink.com.au  Fri Sep 20 00:05:21 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Fri, 20 Sep 2002 09:05:21 +1000
Subject: [Spambayes] Options ignore_redundant_html is ignored :) 
In-Reply-To: <3e6303ba03.3ba033e630@icomcast.net> 
Message-ID: <200209192305.g8JN5Mo03788@localhost.localdomain>


> Are you sure you're not hallucinating on both counts?  "defaults" in 
> Options.py contains

Bah. Ancient version of tokenizer.py hanging around. Don't know why cvs up
didn't make it go away. better now. flashbacks fading.

From gward@python.net  Fri Sep 20 00:14:50 2002
From: gward@python.net (Greg Ward)
Date: Thu, 19 Sep 2002 19:14:50 -0400
Subject: [Spambayes] idea for tokenizer.crack_filename change
In-Reply-To: <BIEJKCLHCIOIHAGOKOLHGENLDKAA.tim.one@comcast.net>
References: <w537khhstuh.fsf@woozle.org>
	<BIEJKCLHCIOIHAGOKOLHGENLDKAA.tim.one@comcast.net>
Message-ID: <20020919231450.GC19802@cthulhu.gerg.ca>

On 19 September 2002, Tim Peters said:
> Greg has a
> cheap yet extremely effective virus trap in place on python.org (if you
> prevent users from sharing any useful files, this is easy to do)

OK, when was the last time someone sent you an .exe or .vbs attachment
that *wasn't* a virus?  And if it happened tomorrow, would you actually
run that attachment?  Or delete it ASAP?

> When I
> get a free nanosecond, I'll try running Greg's virus detector over the
> corpus.

The code (and not much else!) is at elspy.sourceforge.net.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Cheops' Law: Nothing *ever* gets built on schedule or within budget.

From anthony@interlink.com.au  Fri Sep 20 00:15:05 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Fri, 20 Sep 2002 09:15:05 +1000
Subject: [Spambayes] RE: [Spambayes-checkins] spambayes
	Options.py,1.19,1.20 
In-Reply-To: <BIEJKCLHCIOIHAGOKOLHEEMLDKAA.tim.one@comcast.net> 
Message-ID: <200209192315.g8JNF5o04034@localhost.localdomain>


>>> Tim Peters wrote
> That's a good change!  What would the testers find *most* convenient?  The
> unwieldy bayescustomize.ini is so named only because there *used* to be a
> bayes.ini too.  If someone wants to make option-processing heavenly instead
> of merely bearable, please do, without regard for backward compatibility --
> in real life, I eat the same thing for lunch every day, even though I don't
> like it <wink>.

Well, fwiw, the way I now do it is to have two files:
bonanza% cat original.ini

[Tokenizer]
basic_header_tokenize: True
mine_received_headers: True

[TestDriver]
show_false_negatives: True
bonanza% cat robinson.ini 
[Classifier]
use_robinson_probability: True

[TestDriver]
spam_cutoff: 0.50

to test with old-style scoring, I use BAYESCUSTOMIZE=original.ini
to test exactly the same thing with new-style scoring, I use 
BAYESCUSTOMIZE="original.ini robinson.ini" and it's all happy.

Course, we could just replace them all with getopt --longopt options,
but I'm not convinced anyone's going to type

  python timcv.py --basic_header_tokenize --mine_received_headers \
    --show_false_negatives --no-use_robinson_probability --spam_cutoff=0.50

more than once :)

usability-schmusability-so-long-as-I'm-rich'ly,
Anthony

From gward@python.net  Fri Sep 20 00:55:00 2002
From: gward@python.net (Greg Ward)
Date: Thu, 19 Sep 2002 19:55:00 -0400
Subject: [Spambayes] SpamBayes: announce mailing list?
In-Reply-To: <3D89E826.6050305@spamcop.net>
References: <3D89E826.6050305@spamcop.net>
Message-ID: <20020919235500.GA19867@cthulhu.gerg.ca>

On 19 September 2002, Alex Bischoff said:
> I heard about the project through Mozilla bug 163188 
> (http://bugzilla.mozilla.org/show_bug.cgi?id=163188). And, I see that 
> there's a discussion mailing list and a cvs-checkin mailing list for the 
> project.
> 
> However, how about an "announce" list for new versions or other news? 
> I'm just an ordinary user and, while a full discussion list may be more 
> than I need, an announce-list would be just right :).

Unfortunately, there's nothing to announce here.  For the time being,
this code is for hacking and testing and playing only.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Rules for Urban Cycling, #2:
When approaching an intersection with a "STOP" sign, you should step pedalling
until you are through the intersection.  At *least* stop accelerating.

From anthony@interlink.com.au  Fri Sep 20 01:34:55 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Fri, 20 Sep 2002 10:34:55 +1000
Subject: [Spambayes] SpamBayes: announce mailing list? 
In-Reply-To: <3D89E826.6050305@spamcop.net> 
Message-ID: <200209200035.g8K0Yt605736@localhost.localdomain>


>>> Alex Bischoff wrote
> However, how about an "announce" list for new versions or other news? 
> I'm just an ordinary user and, while a full discussion list may be more 
> than I need, an announce-list would be just right :).

There's not really anything to announce, as such - this is more an
exercise in algorithm research and tweaking... 

(not to say that there won't be stuff in the future, but dunno when...)

-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.


From anthony@interlink.com.au  Fri Sep 20 02:00:28 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Fri, 20 Sep 2002 11:00:28 +1000
Subject: Third result ... RE: [Spambayes] First result from Gary
	Robinson's ideas 
In-Reply-To: <BIEJKCLHCIOIHAGOKOLHAENFDKAA.tim.one@comcast.net> 
Message-ID: <200209200100.g8K10Ti06188@localhost.localdomain>


Let's try that again, this time with the current tokenizer.py

same settings as before.

false positive percentages
    0.223  0.334  lost   +49.78%
    0.278  0.278  tied
    0.167  0.167  tied
    0.278  0.278  tied
    0.334  0.389  lost   +16.47%

won   0 times
tied  3 times
lost  2 times

total unique fp went from 23 to 26 lost   +13.04%
mean fp % went from 0.255790439419 to 0.289173231033 lost   +13.05%

false negative percentages
    0.582  0.517  won    -11.17%
    0.388  0.388  tied
    0.581  0.516  won    -11.19%
    0.518  0.453  won    -12.55%
    0.712  0.712  tied

won   3 times
tied  2 times
lost  0 times

total unique fn went from 43 to 40 won     -6.98%
mean fn % went from 0.556199665243 to 0.517406493303 won     -6.97%

before:
Ham distribution for all runs:
* = 150 items
  0.00 8961 ************************************************************
  2.50    1 *
  5.00    0 
  7.50    2 *
 10.00    0 
 12.50    0 
 15.00    0 
 17.50    0 
 20.00    0 
 22.50    0 
 25.00    0 
 27.50    0 
 30.00    0 
 32.50    0 
 35.00    0 
 37.50    0 
 40.00    0 
 42.50    0 
 45.00    0 
 47.50    1 *
 50.00    1 *
 52.50    1 *
 55.00    0 
 57.50    0 
 60.00    0 
 62.50    0 
 65.00    0 
 67.50    0 
 70.00    0 
 72.50    0 
 75.00    0 
 77.50    0 
 80.00    0 
 82.50    0 
 85.00    1 *
 87.50    0 
 90.00    0 
 92.50    0 
 95.00    1 *
 97.50   22 *

Spam distribution for all runs:
* = 129 items
  0.00   35 *
  2.50    2 *
  5.00    0 
  7.50    2 *
 10.00    0 
 12.50    0 
 15.00    0 
 17.50    0 
 20.00    0 
 22.50    0 
 25.00    0 
 27.50    0 
 30.00    0 
 32.50    0 
 35.00    0 
 37.50    1 *
 40.00    0 
 42.50    0 
 45.00    0 
 47.50    0 
 50.00    0 
 52.50    0 
 55.00    0 
 57.50    1 *
 60.00    0 
 62.50    1 *
 65.00    0 
 67.50    1 *
 70.00    0 
 72.50    0 
 75.00    0 
 77.50    0 
 80.00    0 
 82.50    0 
 85.00    0 
 87.50    0 
 90.00    0 
 92.50    0 
 95.00    0 
 97.50 7688 ************************************************************


After:
Ham distribution for all runs:
* = 108 items
  0.00 6459 ************************************************************
  2.50  862 ********
  5.00  263 ***
  7.50   84 *
 10.00   32 *
 12.50  110 **
 15.00  379 ****
 17.50  241 ***
 20.00   80 *
 22.50   95 *
 25.00  116 **
 27.50   65 *
 30.00   48 *
 32.50   35 *
 35.00   23 *
 37.50   23 *
 40.00   16 *
 42.50   16 *
 45.00    9 *
 47.50    9 *
 50.00    7 *
 52.50    4 *
 55.00    7 *
 57.50    2 *
 60.00    1 *
 62.50    1 *
 65.00    0 
 67.50    0 
 70.00    0 
 72.50    1 *
 75.00    0 
 77.50    0 
 80.00    0 
 82.50    0 
 85.00    0 
 87.50    0 
 90.00    0 
 92.50    0 
 95.00    0 
 97.50    3 *

Spam distribution for all runs:
* = 123 items
  0.00    4 *
  2.50    0 
  5.00    0 
  7.50    0 
 10.00    0 
 12.50    0 
 15.00    1 *
 17.50    1 *
 20.00    0 
 22.50    1 *
 25.00    2 *
 27.50    2 *
 30.00    0 
 32.50    4 *
 35.00    3 *
 37.50    1 *
 40.00    3 *
 42.50    3 *
 45.00    5 *
 47.50   10 *
 50.00    7 *
 52.50   15 *
 55.00   15 *
 57.50   15 *
 60.00   16 *
 62.50   25 *
 65.00   27 *
 67.50   19 *
 70.00   20 *
 72.50   19 *
 75.00   21 *
 77.50   22 *
 80.00   44 *
 82.50   42 *
 85.00   26 *
 87.50    4 *
 90.00    0 
 92.50    1 *
 95.00   22 *
 97.50 7331 ************************************************************



From tim.one@comcast.net  Fri Sep 20 03:24:48 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 19 Sep 2002 22:24:48 -0400
Subject: [Spambayes] idea for tokenizer.crack_filename change
In-Reply-To: <20020919231450.GC19802@cthulhu.gerg.ca>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEFDBFAB.tim.one@comcast.net>

[Greg Ward]
> OK, when was the last time someone sent you an .exe or .vbs attachment
> that *wasn't* a virus?

Heh -- I *said* everyone would agree it was a fair characterization all
around <wink>.  In answer to the question, it was the last time Wise tried
to send me a new UNWISE32.EXE, to repair the mysterious "hey, the
uninstaller just plain vanished!" bug that started plaguing Python Windows
installations after we upgraded to Wise version 8.  Actually getting the
repaired uninstaller into my hands turned into a mini nightmare, because so
many intervening email packages were such friggin' nannies about .exe files.
In the end I got an email account just for this that didn't block them, and
then it turned out that a Windows Update patch had convinced Outlook 2000 to
refuse to let me even save an .exe attachment to disk, let alone run one.
All it would tell me is that there *was* an attachment.  This is still
considered to be "a feature"; at least Outlook 2002 has an undocumented
registry setting you can fiddle to turn off this dumbass "protection"(*).

> And if it happened tomorrow, would you actually run that attachment?
> Or delete it ASAP?

I'm not a typical user, Greg.  I somtimes run viruses just to see what they
do.  On my home box I have software to track and restore my hard drive, at
the physical block level, to what it was immediately prior to running them,
so it's not a big deal (well, at least not until a virus appears that knows
how to crack that subsystem).

> The code (and not much else!) is at elspy.sourceforge.net.

Yup!


(*) HKEY_CURRENT_USER\Software\Microsoft\Office\10.0\Outlook\Security

Create key Level1Remove, and set it to a semicolon-separated list of file
extensions (sans periods) you don't want Outlook 2002 to protect you from.
Outlook still won't let you run such attachments, but will let you save them
to disk then (and you can run them from there).


From tim.one@comcast.net  Fri Sep 20 04:40:27 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 19 Sep 2002 23:40:27 -0400
Subject: [Spambayes] RE: spam detection via probability - actual results!
In-Reply-To: <B9AF28EB.161F7%grobinson@transpose.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCMEFHBFAB.tim.one@comcast.net>

I have an interesting result from something that shouldn't have been tried
(ahem -- but I had to take a nap, and wanted to run something while I
snoozed that took no more setup than a one-line change from a tired brain).

In the lingo of our codebase, this compares

[Classifier]
use_robinson_probability: True
[TestDriver]
spam_cutoff: 0.50

to

[Classifier]
use_robinson_probability: True
max_discriminators: 150
[TestDriver]
spam_cutoff: 0.50

That is, it's Gary's combining scheme but looking at the top 150
discriminators (MAX_DISCRIMINATORS in classifier.py, 16 by default).  It's
limited to 150 to guarantee that nothing can silently underflow to 0.0 in
the combining code exactly as it is; it would be more interesting to rework
the code so that no bound were needed (and Gary has special reasons for
wanting us to try that, but it requires other changes first to give it a
trial under the intended preconditions).

Boosting MAX_DISCRIMINATORS was a disaster under Graham's combining scheme.
Neil Schemenauer bit the logarithm bullet on Yet Another Combining Scheme of
his own devising, and recently reported much better results than that (but
still not as good as the all-default code clamped to 16).

The results here were surprising in several respects (note that this means I
was surprised, not necessarily that they're in any way surprising -- this is
quite possibly a statement more about me than the results <wink>).

First the summary results.  It didn't hurt a bit, although as we get deeper
into this it's even more surprising that it didn't hurt:

"""
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spams
   [ditto 19 times]

false positive percentages
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.050  0.050  tied
    0.050  0.050  tied
    0.000  0.000  tied
    0.050  0.050  tied
    0.000  0.000  tied
    0.100  0.100  tied

won   0 times
tied 10 times
lost  0 times

total unique fp went from 5 to 5 tied
mean fp % went from 0.025 to 0.025 tied

false negative percentages
    0.218  0.145  won    -33.49%
    0.364  0.145  won    -60.16%
    0.000  0.000  tied
    0.218  0.218  tied
    0.218  0.218  tied
    0.291  0.291  tied
    0.218  0.218  tied
    0.145  0.218  lost   +50.34%
    0.291  0.291  tied
    0.073  0.000  won   -100.00%

won   3 times
tied  6 times
lost  1 times

total unique fn went from 28 to 24 won    -14.29%
mean fn % went from 0.203636363636 to 0.174545454546 won    -14.29%
"""

The real surprise is in the score histograms, which look much more like
normal distributions now, and the means for the ham and spam distributions
are much closer together now.  It's the combination of those two with that
the results *didn't* go to hell that's a surprise to me when taken
altogether:

Ham distribution for all runs:
* = 87 items
  0.00   81 *
  2.50   52 *
  5.00   67 *
  7.50  112 **
 10.00  165 **
 12.50  257 ***
 15.00  424 *****
 17.50  934 ***********
 20.00 2358 ****************************
 22.50 4705 *******************************************************
 25.00 5166 ************************************************************
 27.50 3238 **************************************
 30.00 1515 ******************
 32.50  546 *******
 35.00  221 ***
 37.50   91 **
 40.00   38 *
 42.50   14 *
 45.00    7 *
 47.50    4 *
 50.00    1 *
 52.50    0
 55.00    3 *
 57.50    0
 60.00    0
 62.50    1 *

That's another surprise:  the highest scoring ham is the fellow who added a
useless one-line comment to a quote of an entire Nigerian scam message.
That this *only* scores 0.625 is frightening.  Or encouraging, depending on
how you look at it <wink>.  It's getting actual benefit from containing
seemingly vanilla words like 'because' -- but so does a *real* Nigerian scam
now.  The few low-prob clues that this was sent by a real person are giving
it a *lot* of benefit:

prob('jeez,') = 0.01
prob('email addr:engineer.com') = 0.01
prob('header:X-Complaints-To:1') = 0.01
prob('wrote') = 0.01
prob('header:Organization:1') = 0.01
prob('flynn') = 0.01
prob('header:Errors-To:1') = 0.0194136

 65.00    0
 67.50    0
 70.00    0
 72.50    0
 75.00    0
 77.50    0
 80.00    0
 82.50    0
 85.00    0
 87.50    0
 90.00    0
 92.50    0
 95.00    0
 97.50    0

Spam distribution for all runs:
* = 37 items
  0.00    0
  2.50    0
  5.00    0
  7.50    0
 10.00    0
 12.50    0
 15.00    0
 17.50    0
 20.00    1 *
 22.50    0
 25.00    0
 27.50    0
 30.00    0
 32.50    0
 35.00    0
 37.50    0

Note that only one spam scored below 0.40.

 40.00    3 *
 42.50    5 *
 45.00    8 *
 47.50    7 *
 50.00   15 *
 52.50   32 *
 55.00   68 **
 57.50  127 ****
 60.00  291 ********
 62.50  542 ***************
 65.00 1035 ****************************
 67.50 1910 ****************************************************
 70.00 2180 ***********************************************************
 72.50 1926 *****************************************************
 75.00 1730 ***********************************************
 77.50 1278 ***********************************
 80.00  651 ******************
 82.50  302 *********
 85.00  349 **********
 87.50  189 ******
 90.00  140 ****
 92.50  158 *****
 95.00  238 *******
 97.50  565 ****************

Something without an explanation:  Gary had a report from someone else who
tried his combining scheme without bounding the number of words.  It did
substantially worse than when clamping to 15.  I know 150 isn't unbounded,
but I'm guessing there's not a heck of a lot of difference between 150 and
infinity here.  One side effect of boosting to 150 is that the list of top
discriminators becomes pretty much worthless; here are the top 10:

        'you' 16195 0.5
        'and' 17498 0.5
        'header:Errors-To:1' 18051 0.0200348
        'x-mailer:none' 19979 0.389823
        'the' 22084 0.5
        'header:Message-ID:1' 24311 0.364374
        'header:From:1' 26354 0.496583
        'header:Date:1' 26402 0.472034
        'header:To:1' 26463 0.489375
        'header:Subject:1' 26464 0.495639

IOW, when scoring 26,464 messages, the mere presence of a Subject line was
one of the 150 strongest discriminators.  This tells me that lots of times
we don't even have 150 distinct words to look at.  Lots of times we were
even reduced to looking at the 100% neutral "the", "and" and "you".

It's possible that the fellow who generated Gary's other result in this
direction wasn't aware of the potential underflow problems, and so got lots
of nonsense scores (the code is such that underflowing to 0 won't raise a
later exception; underflow can happen in either or both of P and Q; if they
both underflow, the score coming out in the end would be 0.5).


Now a question for Gary (hope you're still here <wink>), to help me
understand what's needed to do this right:  what, exactly, does it mean to
require that the spam probabilities be uniformly distributed?

Concrete and relevant example:  suppose I were to take the spamprobs exactly
as they are now, and merely round them to two significant decimal digits.
Then there would be exactly 99 distinct spamprobs in the system, uniformly
distributed in .01 through .99.  Is that all it takes to meet the formal
precondition?

If I normalize the existing probabilities instead based on rank (which I'm
happy to do), I have tens of thousands of words all with spamprob .01 now,
and also with spamprob .99 now.  Based on rank, then, assigning all ties to
a probability based on the median rank in an all-equal range would *still*
end up giving tens of thousands of words the same probabilities in the end.

So if simply rounding to two digits wouldn't satisfy the intended meaning of
"uniformly distributed", I'm wary that doing the ranked-based business
wouldn't actually satisfy it either.

Maybe the bottom-line question here is whether, to give "all words" a fair
trial, I really need also to change the way *initial* spamprob values are
computed too.


From tim.one@comcast.net  Fri Sep 20 04:57:21 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 19 Sep 2002 23:57:21 -0400
Subject: [Spambayes] RE: spam detection via probability - actual results!
In-Reply-To: <LNBBLJKPBEHFEDALKOLCMEFHBFAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEFJBFAB.tim.one@comcast.net>

BTW, in those last score histograms it's clear that the ham distribution has
lower variance than the spam distribution.  I've seen that before in a
different experiment that produced (what looked to be) normally distributed
ham and spam scores.  In that case, the bulk of the difference was explained
by that we still have hambias set to 2 (we still artificially double the
count on ham words).  When I tried that older experiment again after
removing the hambias (which is the only deliberate bias we haven't yet
removed from Graham's original scheme), it was the spam distribution that
turned out to be tighter.  I expect, but don't know, that the same would be
true here.


From tim.one@comcast.net  Fri Sep 20 05:23:42 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 20 Sep 2002 00:23:42 -0400
Subject: Third result ... RE: [Spambayes] First result from Gary
	Robinson'sideas
In-Reply-To: <200209200100.g8K10Ti06188@localhost.localdomain>
Message-ID: <LNBBLJKPBEHFEDALKOLCEEFKBFAB.tim.one@comcast.net>

[Anthony Baxter]
> Let's try that again, this time with the current tokenizer.py

Not a bad idea <wink> -- thanks.

> same settings as before.
>
> false positive percentages
>     0.223  0.334  lost   +49.78%
>     0.278  0.278  tied
>     0.167  0.167  tied
>     0.278  0.278  tied
>     0.334  0.389  lost   +16.47%
>
> won   0 times
> tied  3 times
> lost  2 times
>
> total unique fp went from 23 to 26 lost   +13.04%
> mean fp % went from 0.255790439419 to 0.289173231033 lost   +13.05%
>
> false negative percentages
>     0.582  0.517  won    -11.17%
>     0.388  0.388  tied
>     0.581  0.516  won    -11.19%
>     0.518  0.453  won    -12.55%
>     0.712  0.712  tied
>
> won   3 times
> tied  2 times
> lost  0 times
>
> total unique fn went from 43 to 40 won     -6.98%
> mean fn % went from 0.556199665243 to 0.517406493303 won     -6.97%

There isn't a compelling case for saying this is significantly different.
But, as before, it seems more eager to call things spam, and again shifting
spam_cutoff from 0.50 to 0.525 would tell a seemingly different story (fp
would *fall* from 23 to 19, fn would *rise* from 43 to 44).  (BTW, I did
those hastily <wink> -- double-checking my claims against your "after"
histograms would be a good exercise.)  It could be that, for this kind of
especially challenging corpus, raising spam_cutoff is a good idea for good
reasons (given the still-partial implementation of Gary's suggestions -- in
the time I can make for this, I'd rather push the rest of Gary's ideas along
than try to tune the part we've already done).


From anthony@interlink.com.au  Fri Sep 20 05:23:41 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Fri, 20 Sep 2002 14:23:41 +1000
Subject: [Spambayes] RE: spam detection via probability - actual results! 
In-Reply-To: <LNBBLJKPBEHFEDALKOLCMEFHBFAB.tim.one@comcast.net> 
Message-ID: <200209200423.g8K4Nie08893@localhost.localdomain>


>>> Tim Peters wrote
> max_discriminators: 150
 
For me, adding only this line to my previous run (the "after" in
http://mail.python.org/pipermail-21/spambayes/2002-September/000360.html
)

This is producing impressive results!


false positive percentages
    0.334  0.056  won    -83.23%
    0.278  0.000  won   -100.00%
    0.167  0.056  won    -66.47%
    0.278  0.333  lost   +19.78%
    0.389  0.222  won    -42.93%

won   4 times
tied  0 times
lost  1 times

total unique fp went from 26 to 12 won    -53.85%
mean fp % went from 0.289173231033 to 0.133419869849 won    -53.86%

false negative percentages
    0.517  0.840  lost   +62.48%
    0.388  0.647  lost   +66.75%
    0.516  0.710  lost   +37.60%
    0.453  0.712  lost   +57.17%
    0.712  1.165  lost   +63.62%

won   0 times
tied  0 times
lost  5 times

total unique fn went from 40 to 63 lost   +57.50%
mean fn % went from 0.517406493303 to 0.814957202608 lost   +57.51%

The new histograms:
Ham distribution for all runs:
* = 22 items
  0.00  468 **********************
  2.50  352 ****************
  5.00   66 ***
  7.50   83 ****
 10.00  123 ******
 12.50   62 ***
 15.00   66 ***
 17.50  157 ********
 20.00  532 *************************
 22.50  912 ******************************************
 25.00 1241 *********************************************************
 27.50 1235 *********************************************************
 30.00 1281 ***********************************************************
 32.50 1121 ***************************************************
 35.00  721 *********************************
 37.50  337 ****************
 40.00  133 *******
 42.50   53 ***
 45.00   21 *
 47.50   15 *
 50.00    7 *
 52.50    1 *
 55.00    2 *
 57.50    1 *
 60.00    1 *
 62.50    0 
 65.00    0 
 67.50    0 
 70.00    0 
 72.50    0 
 75.00    0 
 77.50    0 
 80.00    0 
 82.50    0 
 85.00    0 
 87.50    0 
 90.00    0 
 92.50    0 
 95.00    0 
 97.50    0 

Spam distribution for all runs:
* = 17 items
  0.00   0 
  2.50   0 
  5.00   0 
  7.50   0 
 10.00   0 
 12.50   0 
 15.00   0 
 17.50   0 
 20.00   2 *
 22.50   0 
 25.00   0 
 27.50   0 
 30.00   1 *
 32.50   0 
 35.00   2 *
 37.50   5 *
 40.00   5 *
 42.50   6 *
 45.00  17 *
 47.50  25 **
 50.00  33 **
 52.50  65 ****
 55.00 116 *******
 57.50 199 ************
 60.00 397 ************************
 62.50 725 *******************************************
 65.00 917 ******************************************************
 67.50 978 **********************************************************
 70.00 811 ************************************************
 72.50 554 *********************************
 75.00 397 ************************
 77.50 357 *********************
 80.00 327 ********************
 82.50 271 ****************
 85.00 180 ***********
 87.50 172 ***********
 90.00 101 ******
 92.50 114 *******
 95.00 102 ******
 97.50 852 ***************************************************



From tim.one@comcast.net  Fri Sep 20 05:30:53 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 20 Sep 2002 00:30:53 -0400
Subject: [Spambayes] idea for tokenizer.crack_filename change
In-Reply-To: <15754.19106.478876.417637@12-248-11-90.client.attbi.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEFLBFAB.tim.one@comcast.net>

[Skip Montanaro]
> It seems to me that base64-encoded, all DOS/Windows executables start
> with (reciting from memory, since I've deleted all viruses and haven't
> received any new ones in the last 15 minutes or so) "TPqAAA" or
> something similar.  Why rely on finding specific file extensions?  They
> can just change.

Well, not often, and the scheme we're working on is supposed to be able to
learn when they do <wink>.  Would you like to write some code to tokenize
this particular bit of Windows Lore?  We currently ignore 100% of the
*content* of MIME sections that don't have text/* type, although the MIME
metadata is tokenized for all MIME sections via

        # Content-{Type, Disposition} and their params, and charsets.
        for x in msg.walk():
            for w in crack_content_xyz(x):
                yield w

in tokenize_headers().


From tim.one@comcast.net  Fri Sep 20 06:04:43 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 20 Sep 2002 01:04:43 -0400
Subject: [Spambayes] RE: spam detection via probability - actual results!
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEFJBFAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEFNBFAB.tim.one@comcast.net>

[Tim, enjoying quality time with himself <wink>]
> BTW, in those last score histograms it's clear that the ham
> distribution has lower variance than the spam distribution.  I've
> seen that before in a different experiment that produced (what
> looked to be) normally distributed ham and spam scores.  In that
> case, the bulk of the difference was explained by that we still have
> hambias set to 2 (we still artificially double the count on ham
> words).  When I tried that older experiment again after removing
> the hambias (which is the only deliberate bias we haven't yet
> removed from Graham's original scheme), it was the spam distribution
> that turned out to be tighter.  I expect, but don't know, that the
> same would be true here.

So the most obvious simple experiment then was to also nuke hambias in this
run.  Here we're comparing

[Classifier]
use_robinson_probability: True
max_discriminators: 150
[TestDriver]
spam_cutoff: 0.50

to

[Classifier]
use_robinson_probability: True
max_discriminators: 150
hambias: 1.0
[TestDriver]
spam_cutoff: 0.50

and then I believe we have no deliberate biases remaining.  Alas, some other
things changed in the codebase between these runs, but let's ignore that (I
believe the only relevant change is that we're now falling back to the raw
email text when the email package just can't parse a string as a legitimate
email msg).

First the summary results:

"""
-> <stat> tested 2000 hams & 1375 spams against 18000 hams & 12375 spa
   [ditto 19 times]

false positive percentages
    0.000  0.150  lost  +(was 0)
    0.000  0.100  lost  +(was 0)
    0.000  0.100  lost  +(was 0)
    0.000  0.100  lost  +(was 0)
    0.050  0.250  lost  +400.00%
    0.050  0.200  lost  +300.00%
    0.000  0.000  tied
    0.050  0.100  lost  +100.00%
    0.000  0.050  lost  +(was 0)
    0.100  0.200  lost  +100.00%

won   0 times
tied  1 times
lost  9 times

total unique fp went from 5 to 25 lost  +400.00%
mean fp % went from 0.025 to 0.125 lost  +400.00%

false negative percentages
    0.218  0.000  won   -100.00%
    0.364  0.000  won   -100.00%
    0.000  0.000  tied
    0.218  0.073  won    -66.51%
    0.218  0.073  won    -66.51%
    0.291  0.000  won   -100.00%
    0.218  0.073  won    -66.51%
    0.145  0.000  won   -100.00%
    0.291  0.000  won   -100.00%
    0.073  0.000  won   -100.00%

won   9 times
tied  1 times
lost  0 times

total unique fn went from 28 to 3 won    -89.29%
mean fn % went from 0.203636363636 to 0.0218181818182 won    -89.29%
"""

Since we're no longer giving hamness artificial boosts, all things are more
likely to be called spam.  That's consistent with the jump in f-p and fall
in f-n rates observed.

"After" histograms:

Ham distribution for all runs:
* = 86 items
  0.00   43 *
  2.50    4 *
  5.00   15 *
  7.50   15 *
 10.00   14 *
 12.50   29 *
 15.00   39 *
 17.50   79 *
 20.00  213 ***
 22.50  652 ********
 25.00 1611 *******************
 27.50 3598 ******************************************
 30.00 5121 ************************************************************
 32.50 4312 ***************************************************
 35.00 2517 ******************************
 37.50 1034 *************
 40.00  426 *****
 42.50  166 **
 45.00   64 *
 47.50   23 *
 50.00   10 *
 52.50    8 *
 55.00    3 *
 57.50    0
 60.00    1 *
 62.50    2 *
 65.00    0
 67.50    0
 70.00    1 *
 72.50    0
 75.00    0
 77.50    0
 80.00    0
 82.50    0
 85.00    0
 87.50    0
 90.00    0
 92.50    0
 95.00    0
 97.50    0

Spam distribution for all runs:
* = 40 items
  0.00    0
  2.50    0
  5.00    0
  7.50    0
 10.00    0
 12.50    0
 15.00    0
 17.50    0
 20.00    0
 22.50    0
 25.00    0
 27.50    0
 30.00    0
 32.50    0
 35.00    1 *
 37.50    0
 40.00    0
 42.50    0
 45.00    1 *
 47.50    1 *
 50.00    3 *
 52.50    7 *
 55.00   11 *
 57.50    7 *
 60.00   32 *
 62.50   57 **
 65.00  133 ****
 67.50  309 ********
 70.00  563 ***************
 72.50 1191 ******************************
 75.00 2204 ********************************************************
 77.50 2381 ************************************************************
 80.00 2194 *******************************************************
 82.50 1544 ***************************************
 85.00  588 ***************
 87.50  418 ***********
 90.00  466 ************
 92.50  346 *********
 95.00  324 *********
 97.50  969 *************************

If we raised spam_cutoff to 0.575, then, we'd lose 10+8+3 = 21 of the false
positives, and gain 3+7+11=21 false negatives.  That would leave us with a
grand total of 4 f-p and 24 f-n, pretty much the same as before, but
*perhaps* slightly better.

This is where sticking to what the data tells you gets hard for me <0.5
wink>:  I *despise* hambias.  My previous lives tell me that deliberate
fudging of the data is always a poor idea in the end.  So I have high hopes
that following the rest of Gary's agenda will let us get a clear improvement
and toss the hambias crock.

OTOH, both error rates are already so low on my data that I can't make a
convincing case for real improvements anymore, I can only make a convincing
case that a bad idea is in fact a bad one.  That means all of you are on the
line to demonstrate that a thing is a real improvement now!  This is where
the public humiliation you've endured for posting less than perfectly
conducted experiments finally pays off <0.9 wink>.


From eppstein@ics.uci.edu  Fri Sep 20 07:53:07 2002
From: eppstein@ics.uci.edu (David Eppstein)
Date: Thu, 19 Sep 2002 23:53:07 -0700
Subject: [Spambayes] Re: we'll win them over yet...
References: <15752.45383.84504.165493@12-248-11-90.client.attbi.com>
Message-ID: <eppstein-EB39FB.23530619092002@main.gmane.org>

In article <15752.45383.84504.165493@12-248-11-90.client.attbi.com>,
 Skip Montanaro <skip@pobox.com> wrote:

>     (Excuse my use of the "ham" terminology, but I like it ;)
> 
> Maybe we can't get them to use a different language, but we'll get them to
> use our terminology. <wink>

Somehow the "ham" terminology is making me think you should change your 
w's to o's in the eye-gestures...

-- 
David Eppstein       UC Irvine Dept. of Information & Computer Science
eppstein@ics.uci.edu http://www.ics.uci.edu/~eppstein/



From sjoerd@acm.org  Fri Sep 20 10:29:19 2002
From: sjoerd@acm.org (Sjoerd Mullender)
Date: Fri, 20 Sep 2002 11:29:19 +0200
Subject: [Spambayes] RE: spam detection via probability - actual results!
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEFNBFAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCGEFNBFAB.tim.one@comcast.net> 
Message-ID: <200209200929.g8K9TJo11972@indus.ins.cwi.nl>

On Fri, Sep 20 2002 Tim Peters wrote:

> [Classifier]
> use_robinson_probability: True
> max_discriminators: 150
> hambias: 1.0
> [TestDriver]
> spam_cutoff: 0.50

Here are my results.  I also have
[Tokenizer]
count_all_header_lines: True
mine_received_headers: True
in both runs.

run1s -> run2s
-> <stat> tested 100 hams & 100 spams against 700 hams & 700 spams
-> <stat> tested 100 hams & 100 spams against 700 hams & 700 spams
-> <stat> tested 100 hams & 100 spams against 700 hams & 700 spams
-> <stat> tested 100 hams & 100 spams against 700 hams & 700 spams
-> <stat> tested 100 hams & 100 spams against 700 hams & 700 spams
-> <stat> tested 100 hams & 100 spams against 700 hams & 700 spams
-> <stat> tested 100 hams & 100 spams against 700 hams & 700 spams
-> <stat> tested 100 hams & 100 spams against 700 hams & 700 spams
-> <stat> tested 100 hams & 100 spams against 700 hams & 700 spams
-> <stat> tested 100 hams & 100 spams against 700 hams & 700 spams
-> <stat> tested 100 hams & 100 spams against 700 hams & 700 spams
-> <stat> tested 100 hams & 100 spams against 700 hams & 700 spams
-> <stat> tested 100 hams & 100 spams against 700 hams & 700 spams
-> <stat> tested 100 hams & 100 spams against 700 hams & 700 spams
-> <stat> tested 100 hams & 100 spams against 700 hams & 700 spams
-> <stat> tested 100 hams & 100 spams against 700 hams & 700 spams

false positive percentages
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  1.000  lost  +(was 0)
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied

won   0 times
tied  7 times
lost  1 times

total unique fp went from 0 to 1 lost  +(was 0)
mean fp % went from 0.0 to 0.125 lost  +(was 0)

false negative percentages
    2.000  1.000  won    -50.00%
    3.000  2.000  won    -33.33%
    0.000  0.000  tied
    1.000  1.000  tied
    4.000  2.000  won    -50.00%
    0.000  0.000  tied
    1.000  0.000  won   -100.00%
    1.000  1.000  tied

won   4 times
tied  4 times
lost  0 times

total unique fn went from 12 to 7 won    -41.67%
mean fn % went from 1.5 to 0.875 won    -41.67%

with histograms before:

Ham distribution for all runs:
* = 14 items
  0.00 800 **********************************************************
  2.50   0 
[ deleted because all 0 ]

Spam distribution for all runs:
* = 14 items
  0.00  11 *
  2.50   0 
[ deleted because all 0 ]
 82.50   0 
 85.00   1 *
 87.50   0 
 90.00   1 *
 92.50   0 
 95.00   1 *
 97.50 786 *********************************************************

and after:

Ham distribution for all runs:
* = 3 items
  0.00  68 ***********************
  2.50  16 ******
  5.00  13 *****
  7.50  20 *******
 10.00  32 ***********
 12.50  99 *********************************
 15.00  85 *****************************
 17.50 121 *****************************************
 20.00 107 ************************************
 22.50  81 ***************************
 25.00  47 ****************
 27.50  40 **************
 30.00  18 ******
 32.50  20 *******
 35.00  15 *****
 37.50   6 **
 40.00   4 **
 42.50   1 *
 45.00   1 *
 47.50   5 **
 50.00   0 
 52.50   1 *
 55.00   0 
 57.50   0 
 60.00   0 
 62.50   0 
 65.00   0 
 67.50   0 
 70.00   0 
 72.50   0 
 75.00   0 
 77.50   0 
 80.00   0 
 82.50   0 
 85.00   0 
 87.50   0 
 90.00   0 
 92.50   0 
 95.00   0 
 97.50   0 

Spam distribution for all runs:
* = 2 items
  0.00   0 
  2.50   0 
  5.00   0 
  7.50   0 
 10.00   0 
 12.50   0 
 15.00   0 
 17.50   0 
 20.00   0 
 22.50   0 
 25.00   0 
 27.50   0 
 30.00   0 
 32.50   0 
 35.00   0 
 37.50   1 *
 40.00   1 *
 42.50   0 
 45.00   2 *
 47.50   3 **
 50.00   5 ***
 52.50   4 **
 55.00   7 ****
 57.50   9 *****
 60.00  25 *************
 62.50  44 **********************
 65.00  57 *****************************
 67.50  74 *************************************
 70.00  69 ***********************************
 72.50  78 ***************************************
 75.00  52 **************************
 77.50  59 ******************************
 80.00  50 *************************
 82.50  40 ********************
 85.00  40 ********************
 87.50  30 ***************
 90.00  18 *********
 92.50  17 *********
 95.00  10 *****
 97.50 105 *****************************************************

Here are the results if I keep hambias at 2.0:

false positive percentages
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied

won   0 times
tied  8 times
lost  0 times

total unique fp went from 0 to 0 tied
mean fp % went from 0.0 to 0.0 tied

false negative percentages
    2.000  3.000  lost   +50.00%
    3.000  4.000  lost   +33.33%
    0.000  0.000  tied
    1.000  1.000  tied
    4.000  5.000  lost   +25.00%
    0.000  0.000  tied
    1.000  3.000  lost  +200.00%
    1.000  2.000  lost  +100.00%

won   0 times
tied  3 times
lost  5 times

total unique fn went from 12 to 18 lost   +50.00%
mean fn % went from 1.5 to 2.25 lost   +50.00%

Only the after histograms:

Ham distribution for all runs:
* = 2 items
  0.00  92 **********************************************
  2.50  47 ************************
  5.00  38 *******************
  7.50  59 ******************************
 10.00  93 ***********************************************
 12.50  90 *********************************************
 15.00 119 ************************************************************
 17.50 108 ******************************************************
 20.00  57 *****************************
 22.50  32 ****************
 25.00  24 ************
 27.50  15 ********
 30.00  11 ******
 32.50   6 ***
 35.00   3 **
 37.50   2 *
 40.00   2 *
 42.50   2 *
 45.00   0 
 47.50   0 
 50.00   0 
 52.50   0 
 55.00   0 
 57.50   0 
 60.00   0 
 62.50   0 
 65.00   0 
 67.50   0 
 70.00   0 
 72.50   0 
 75.00   0 
 77.50   0 
 80.00   0 
 82.50   0 
 85.00   0 
 87.50   0 
 90.00   0 
 92.50   0 
 95.00   0 
 97.50   0 

Spam distribution for all runs:
* = 2 items
  0.00   0 
  2.50   0 
  5.00   0 
  7.50   0 
 10.00   0 
 12.50   0 
 15.00   0 
 17.50   0 
 20.00   0 
 22.50   0 
 25.00   0 
 27.50   0 
 30.00   3 **
 32.50   0 
 35.00   0 
 37.50   1 *
 40.00   2 *
 42.50   2 *
 45.00   6 ***
 47.50   4 **
 50.00  13 *******
 52.50  17 *********
 55.00  29 ***************
 57.50  45 ***********************
 60.00  56 ****************************
 62.50  76 **************************************
 65.00  60 ******************************
 67.50  62 *******************************
 70.00  54 ***************************
 72.50  59 ******************************
 75.00  53 ***************************
 77.50  37 *******************
 80.00  44 **********************
 82.50  20 **********
 85.00  20 **********
 87.50  13 *******
 90.00  10 *****
 92.50   8 ****
 95.00   4 **
 97.50 102 ***************************************************

-- Sjoerd Mullender <sjoerd@acm.org>

From grobinson@transpose.com  Fri Sep 20 12:44:51 2002
From: grobinson@transpose.com (Gary Robinson)
Date: Fri, 20 Sep 2002 07:44:51 -0400
Subject: [Spambayes] RE: spam detection via probability - actual results!
Message-ID: <B9B08273.162EB%grobinson@transpose.com>

> Something without an explanation:  Gary had a report from someone else who
> tried his combining scheme without bounding the number of words.

I got an apology this morning, he said he had accidentally trained and
tested on the same data set. So maybe the explanation is in that somehow.

> It's possible that the fellow who generated Gary's other result in this
> direction wasn't aware of the potential underflow problems

That's the most likely thing, IMHO.

> Now a question for Gary (hope you're still here <wink>), to help me
> understand what's needed to do this right:  what, exactly, does it mean to
> require that the spam probabilities be uniformly distributed?
> 
> Concrete and relevant example:  suppose I were to take the spamprobs exactly
> as they are now, and merely round them to two significant decimal digits.
> Then there would be exactly 99 distinct spamprobs in the system, uniformly
> distributed in .01 through .99.  Is that all it takes to meet the formal
> precondition?

No, that would mean you would have 99 buckets, but a different number of
words in each bucket. So that wouldn't be uniform. Unless I'm
misunderstanding you.


> 
> If I normalize the existing probabilities instead based on rank (which I'm
> happy to do), I have tens of thousands of words all with spamprob .01 now,
> and also with spamprob .99 now.  Based on rank, then, assigning all ties to
> a probability based on the median rank in an all-equal range would *still*
> end up giving tens of thousands of words the same probabilities in the end.

I think you should get rid of the code that turns so many things into .01's
and .99's. There should be no need for those cut-offs under my scheme. Just
let everything be what it naturally is. Then you won't have the thousands of
equal probabilities.

BUT be sure you do "Further Improvement 1" first. (That is, the new and
improved FI1 that was written after we talked about Graham's way  of
generating his probabilities.) That will eliminate much of the reason for
the .01 and .99 cutoffs because it will be rarer to get such extreme values.

I think there's a really good chance that if you do FI1 AND FI2, the results
will be significantly better than we're seeing now.


--Gary


-- 
Gary Robinson
CEO
Transpose, LLC
grobinson@transpose.com
207-942-3463
http://www.emergentmusic.com
http://radio.weblogs.com/0101454



From tim.one@comcast.net  Fri Sep 20 15:51:14 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 20 Sep 2002 10:51:14 -0400
Subject: [Spambayes] RE: spam detection via probability - actual results!
In-Reply-To: <200209200929.g8K9TJo11972@indus.ins.cwi.nl>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEHDBFAB.tim.one@comcast.net>

[Sjoerd Mullender, using

> > [Classifier]
> > use_robinson_probability: True
> > max_discriminators: 150
> > hambias: 1.0
> > [TestDriver]
> > spam_cutoff: 0.50
>
> Here are my results.  I also have
> [Tokenizer]
> count_all_header_lines: True
> mine_received_headers: True
> in both runs.

running an 8-fold cv with 800 spam and 5600 ham, sees his first false
positive (for a ham scoring 0.525), and what may be a reduction in f-n rate
(4 runs tie, on 1 run 2 false negatives go away, on 3 runs 1 f-n goes away).
Trying again with the default hambias 2.0, f-p is unaffected, and f-n loses
on 5 runs of 8, ties on 3]

That's very helpful -- thanks.  At this point I think it's best to stop
trying to "hill climb" into Gary's approach one little change at a time --
I'll implement all his suggestions at once (he's got good reasons for why
they *should* be done together).


From crunch@shopip.com  Thu Sep 19 20:41:25 2002
From: crunch@shopip.com (John Draper)
Date: Thu, 19 Sep 2002 12:41:25 -0700
Subject: [Spambayes] Spamvolution
Message-ID: <v03110740b9afd8b680af@[192.168.0.2]>

My friend wanted me to post this to the spambayes group.

>If the SpamBabes project (and others to follow) becomes popular, we'll see
>Darwinian evolution show itself. Only the spams which taste like ham will
>survive, so spammers will adapt to make their spams read more like "plain
>old mail". Right now, there's no competition so you get these thrilling
>black and white correlations, but there's no stopping evolution. Please
>consider spamvolution.



From guido@python.org  Fri Sep 20 16:26:43 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 20 Sep 2002 11:26:43 -0400
Subject: [Spambayes] Spamvolution
In-Reply-To: Your message of "Thu, 19 Sep 2002 12:41:25 PDT."
             <v03110740b9afd8b680af@[192.168.0.2]> 
References: <v03110740b9afd8b680af@[192.168.0.2]> 
Message-ID: <200209201526.g8KFQhd08298@pcp02138704pcs.reston01.va.comcast.net>

> >If the SpamBabes project (and others to follow) becomes popular, we'll see
> >Darwinian evolution show itself. Only the spams which taste like ham will
> >survive, so spammers will adapt to make their spams read more like "plain
> >old mail". Right now, there's no competition so you get these thrilling
> >black and white correlations, but there's no stopping evolution. Please
> >consider spamvolution.

(1) I would welcome spam that reads more like plain old mail.

(2) It's an arms race.

(3) Paul Graham also predicts this.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.one@comcast.net  Fri Sep 20 16:37:36 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 20 Sep 2002 11:37:36 -0400
Subject: [Spambayes] RE: spam detection via probability - actual results!
In-Reply-To: <B9B08273.162EB%grobinson@transpose.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEHHBFAB.tim.one@comcast.net>

[Tim, wonders about the intended meaning of "uniformly distributed"
 in Gary's "Further Improvement 2"]

>> suppose I were to take the spamprobs exactly as they are now, and
>> merely round them to two significant decimal digits.  Then there
>> would be exactly 99 distinct spamprobs in the system, uniformly
>> distributed in .01 through .99.  Is that all it takes to meet
>> the formal precondition?

[Gary]
> No, that would mean you would have 99 buckets, but a different number of
> words in each bucket. So that wouldn't be uniform. Unless I'm
> misunderstanding you.

No, you got it.  You want that the number of words in each bucket be about
the same, not that the numerical values of the buckets (the midpoints of the
probability ranges attached to each bucket) be uniformly distributed --
unless I'm misunderstanding you now.  If I assume there aren't a lot of p(w)
or f(w) duplications going in to the ranking step, it produces a rescoring
that's "uniform" in both respects, and it wasn't clear to me whether which
(or whether both) were intended.

This is compounded because of the FI2:

    This improvement is worth trying ... whether or not the guesstimates
    are derived from Paul counting approach or from the Bayesian approach
    described above.

Something I know all too well, because I've implemented Paul's approach, is
that the p(w) values are massively duplicated at the endpoints under Paul's
approach, and the ranking scheme doesn't change that (it has a "duplicates
in, duplicates out" property).

>> If I normalize the existing probabilities instead based on rank
>> (which I'm happy to do), I have tens of thousands of words all with
>> spamprob .01 now, and also with spamprob .99 now.  Based on rank,
>> then, assigning all ties to a probability based on the median rank
>> in an all-equal range would *still* end up giving tens of thousands
>> of words the same probabilities in the end.

Which said the same thing.

> I think you should get rid of the code that turns so many things
> into .01's and .99's.

So do I wink <wink>.  To date, almost all attempts to widen the prob range
under Graham's combining scheme have clear losers.  I think there are clear
reasons for why widening the prob range interacts *badly* with clamping the
max # of discriminators at a low value, though; those reasons won't apply if
I grab all your ideas at once.

> There should be no need for those cut-offs under my scheme. Just
> let everything be what it naturally is. Then you won't have the
> thousands of equal probabilities.

Agreed -- and it's highly attractive to me too.

> BUT be sure you do "Further Improvement 1" first.

That's the crux of the matter, then:  FI2 should stop saying that Graham's
p(w) is an acceptable input to this step.  If people try that, it's not
going to deliver what you're after because of the above.

> (That is, the new and improved FI1 that was written after we talked
> about Graham's way of generating his probabilities.) That will eliminate
> much of the reason for the .01 and .99 cutoffs because it will be rarer
> to get such extreme values.

Gotcha.

> I think there's a really good chance that if you do FI1 AND FI2,
> the results will be significantly better than we're seeing now.

That's a plan, then!  I'm going to do all the rest of this in one gulp.
Thanks for the clarifications.


From grobinson@transpose.com  Fri Sep 20 16:43:04 2002
From: grobinson@transpose.com (Gary Robinson)
Date: Fri, 20 Sep 2002 11:43:04 -0400
Subject: [Spambayes] RE: spam detection via probability - actual
	results!
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCEHHBFAB.tim.one@comcast.net>
Message-ID: <B9B0BA48.16351%grobinson@transpose.com>


> 
> No, you got it.  You want that the number of words in each bucket be about
> the same, not that the numerical values of the buckets (the midpoints of the
> probability ranges attached to each bucket) be uniformly distributed --
> unless I'm misunderstanding you now.

No, you are not misunderstanding. Your statement describes a roughly uniform
distribution.

> If I assume there aren't a lot of p(w)
> or f(w) duplications going in to the ranking step, it produces a rescoring
> that's "uniform" in both respects, and it wasn't clear to me whether which
> (or whether both) were intended.
> 
> This is compounded because of the FI2:
> 
>   This improvement is worth trying ... whether or not the guesstimates
>   are derived from Paul counting approach or from the Bayesian approach
>   described above.
> 
> Something I know all too well, because I've implemented Paul's approach, is
> that the p(w) values are massively duplicated at the endpoints under Paul's
> approach, and the ranking scheme doesn't change that (it has a "duplicates
> in, duplicates out" property).

I was wrong, it would be better not to use Paul's if there are so many
duplications.


> 
>> I think you should get rid of the code that turns so many things
>> into .01's and .99's.
> 
> So do I wink <wink>.  To date, almost all attempts to widen the prob range
> under Graham's combining scheme have clear losers.  I think there are clear
> reasons for why widening the prob range interacts *badly* with clamping the
> max # of discriminators at a low value, though; those reasons won't apply if
> I grab all your ideas at once.

That sounds good to me! :)

> 
>> There should be no need for those cut-offs under my scheme. Just
>> let everything be what it naturally is. Then you won't have the
>> thousands of equal probabilities.
> 
> Agreed -- and it's highly attractive to me too.
> 
>> BUT be sure you do "Further Improvement 1" first.
> 
> That's the crux of the matter, then:  FI2 should stop saying that Graham's
> p(w) is an acceptable input to this step.  If people try that, it's not
> going to deliver what you're after because of the above.

I understand and think you're right. I will change that on the site.


> 
> That's a plan, then!  I'm going to do all the rest of this in one gulp.

Fantastic!


Gary


From tim.one@comcast.net  Fri Sep 20 17:02:43 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 20 Sep 2002 12:02:43 -0400
Subject: [Spambayes] Spamvolution
Message-ID: <LNBBLJKPBEHFEDALKOLCIEHJBFAB.tim.one@comcast.net>

> ...
> Please consider spamvolution.

In addition to Guido's points, I'd like to add that spam is overwhelmingly
advertising.  As such, it's constrained to the language and artifacts of
advertising.  In the end, maybe it will evolve to become as amusing as many
television commercials strive to be -- fine by me, if so.  There are some TV
ads I really enjoy (e.g., Apple's 1984 ad was a mini cinematic
masterpiece -- I've watched that in a tape loop for an hour, as if it *were*
a movie.  But most great TV commercials-- like that one --have a very hard
time getting around to *selling* you something specific.).

A spammer simply can't hope to sell you, say, a weight loss pill, without
mentioning several of weight, diet, exercise, and no unpleasant bending.
The words needed merely to describe the product are very likely to show up
as out of place against the background of your legit email, and especially
when several appear in a single message.

Perhaps spammers will do what many Asian spammers are doing now (but
seemingly doing so because of non-portable character set issues), and send
email consisting almost solely of a pointer to a giant .gif out on the web.
But then that's recognizable too:  my sisters are never going to send email
of that nature to me.

What we may be left with is spam like this (a persistent false negative in
my test data):

"""
Return-Path: <kba94@earthlink.net>
Delivered-To: bfsmedia-bluegoose.kennels@bfsmedia.com
Received: (qmail 16482 invoked from network); 11 Mar 2002 16:01:28 -0000
Received: from harrier.mail.pas.earthlink.net (HELO
harrier.prod.itd.earthlink.net) (207.217.120.12)
  by agamemnon.bfsmedia.com with SMTP; 11 Mar 2002 16:01:28 -0000
Received: from pool-209-128-140-231.gent.ipa.net ([209.128.140.231]
helo=13vys)
        by harrier.prod.itd.earthlink.net with smtp (Exim 3.33 #1)
        id 16kSN8-0004nj-00; Mon, 11 Mar 2002 08:10:04 -0800
From: "Keith Allison" <kba94@earthlink.net>
To: "Amy Kight" <amykight@surfari.net>,
        "Angel Baker" <piper1026@msn.com>,
        "Baker, Chris & Tammy" <bakert@ipa.net>,
        "Baker, Kerry" <kbaker@delta.sesc.k12.ar.us>,
        "Batagower, Holly" <hbatagower@acnielsen.com>,
        "Bo" <BoJessica@cs.com>,
        "Brad Hesgard" <Bheswad@aol.com>,
        "Brian Gilbert" <owl_45@hotmail.com>,
        "Brian Shinall" <shinall@1s.net>,
        "Briggs, A.J." <ATMABRIGGS@JUNO.COM>,
        "Bryant, Cindy" <cbryant@kraft.com>,
        "Burris, Jay" <jaybduck@ipa.net>,
        "Burturm, Jodi" <jodi.burtrum@satellink.net >,
        "Butler, Scott or Revonne" <butlers@hcil.net>,
        "Christina Clark" <christinarclark@hotmail.com>,
        "Clampit, Connie Joe" <clampitc@tyson.com>,
        "Clifford Toney" <cliffordtoney@cox-internet.com>,
        "Combs, Karrie" <karrie.combs@us.nestle.com>,
        "David Haffner" <dhaffner@ipa.net>,
        "Dennis Ditto" <dittohdditto@cs.com>,
        "Eddie Young" <ebfarms@seark.net>,
        "Eliza Bowman" <elbowman@monad.net>,
        "Froning, Sara" <sfroning@kraft.com>,
        "Gary McDonald Jr." <garymc@icnet.net>,
        "Glen Hemminger" <gj.hemminger@home.com>,
        "Good, Dennis" <Hogman9832@aol.com>,
        "Janet Thornhill \(E-mail\)" <Janet@thornhillauto.net>,
        "Jared Woodward" <Jdwood19@cs.com>,
        "Joel Strickland" <joel@aboweb.com>,
        "John Calhoun" <Calhoonj9@cs.com>,
        "John Metz" <JTMETZ1@aol.com>,
        "Jones, Shelley" <RTSRJ@aol>,
        "Lacy, Scott" <Slacey@Sofnet.com>,
        "Leonard, Tammy" <tndleonard@aol.com>,
        "Linda Cline" <cline@somedayretrievers.com>,
        "Lynn A. Yandell" <yandell@uark.edu>,
        "M. D. Baxter" <baxram@bellsouth.net>,
        "Mark So" <info@dogtra.com>,
        "McDonald, Gary" <gmcdonald1@mmcable.com>,
        "Michael Carter" <michaelraycarter@yahoo.com>,
        "Mike Anthony" <mikeanth@earthlink.net>,
        "Moon, Billy" <MoonPhase5@aol.com>,
        "Myers, Chuck" <cemyers@gte.net>,
        "Nancy Belknap" <nhah103@ipa.net>,
        "Nancy R Oldfather" <Nancy.Oldfather@kraft.com>,
        "Nathen C. Neal" <nneal@uark.edu>,
        "Nunn, Randy" <RandHNunn@aol.com>,
        "Owen Bybee" <owen@bomontana.com>,
        "Perry Cox" <perrycox@hotmail.com>,
        "Philip Hattaway" <phattaway@kc.rr.com>,
        "Richard Davis" <richard@gofit.net>,
        "Robert Clay Connor" <c2dg@midsouth.rr.com>,
        "Roberts, Karen" <karenr@ipa.net>,
        "Rose, Ernie & Pat" <bluegoose.kennels@bfsmedia.com>,
        "Shelley. Allison" <shelley.allison@us.pwcglobal.com>,
        "Stan Lewis" <slewis@iqcisp.com>,
        "Stephen Wright" <stephen.wright@leggett.com>,
        "Steve Vanduine" <vanduine@ukcdogs.com>,
        "Sutton, Wes" <medicus@medicusrg.com>,
        "Tami Baker" <cte87344@centurytel.net>,
        "Ted Cullen" <info@etch-marc.com>,
        "Teresa Fuller" <bsandtb@peoplepc.com>,
        "Todd Wilken" <tjwilken@premiernet.net>,
        "Tracy Johnston" <tracyjohnston@service-advantage.com>,
        "Whelan, Ken" <ken@UBH.com>
Subject: ESPN Great Outdoor Games Qualifier
Date: Mon, 11 Mar 2002 10:09:23 -0600
Message-ID: <MBBBIOHGNEDPIDOGJFMBMEDCCGAA.kba94@earthlink.net>
MIME-Version: 1.0
Content-Type: text/plain;
        charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0)
X-MIMEOLE: Produced By Microsoft MimeOLE V5.50.4133.2400
Importance: Normal
Content-Length: 1282
Lines: 33

Just thought I'd let you know that Cosmo and I finished 4th overall in the
ESPN Great Outdoor Games Qualifier this past weekend.  There were 18
outstanding retrievers that ran over the two day event.  You had to finish
in the top 6 on Saturday to go to Finals on Sunday and we finished 4th on
Sunday out of the finalists that ran.  The First and Second place dogs go to
the Great Outdoor Games in July to compete with the other dogs from around
the country that qualified as well.  There was some strong competition
there, all the dogs were Master Hunters, Hunting Retriever Champions and/or
Grand Hunting Retriever Champions that ran this weekend.

I think if Cosmo would have run as well as he normally does, we would have
been in the top 2 for the weekend, but we just didn't have enough luck that
day and that's the way it goes in any sport as well as dogs.

I think it will air several times on ESPN2 starting June 8th.  I believe
these are the air times... http://www.superretrieverseries.com/schedule.html

Take care,

Keith Allison
War Eagle Retrievers
http://www.WarEagleRetrievers.com/
Pea Ridge, AR
"""

What the hell was *that* all about <wink>?  It actually scores very near the
spam threshold, because bfsmedia.com shows up in the headers, and they're
the source of many surreal spams of this nature.  Maybe they're harvesting
attempts, trying to provoke you gently into replying.  Or perhaps this
really is ham!  Both URLs are legit, and still in business.


From guido@python.org  Fri Sep 20 17:09:38 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 20 Sep 2002 12:09:38 -0400
Subject: [Spambayes] Spamvolution
In-Reply-To: Your message of "Fri, 20 Sep 2002 12:02:43 EDT."
             <LNBBLJKPBEHFEDALKOLCIEHJBFAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCIEHJBFAB.tim.one@comcast.net> 
Message-ID: <200209201609.g8KG9cd09311@pcp02138704pcs.reston01.va.comcast.net>

> What we may be left with is spam like this (a persistent false negative in
> my test data):
> 
> """
> Return-Path: <kba94@earthlink.net>
> Delivered-To: bfsmedia-bluegoose.kennels@bfsmedia.com
> Received: (qmail 16482 invoked from network); 11 Mar 2002 16:01:28 -0000
> Received: from harrier.mail.pas.earthlink.net (HELO
> harrier.prod.itd.earthlink.net) (207.217.120.12)
>   by agamemnon.bfsmedia.com with SMTP; 11 Mar 2002 16:01:28 -0000
> Received: from pool-209-128-140-231.gent.ipa.net ([209.128.140.231]
> helo=13vys)
>         by harrier.prod.itd.earthlink.net with smtp (Exim 3.33 #1)
>         id 16kSN8-0004nj-00; Mon, 11 Mar 2002 08:10:04 -0800
> From: "Keith Allison" <kba94@earthlink.net>
> To: "Amy Kight" <amykight@surfari.net>,
>         "Angel Baker" <piper1026@msn.com>,
>         "Baker, Chris & Tammy" <bakert@ipa.net>,
>         "Baker, Kerry" <kbaker@delta.sesc.k12.ar.us>,
>         "Batagower, Holly" <hbatagower@acnielsen.com>,
>         "Bo" <BoJessica@cs.com>,
>         "Brad Hesgard" <Bheswad@aol.com>,
>         "Brian Gilbert" <owl_45@hotmail.com>,
>         "Brian Shinall" <shinall@1s.net>,
>         "Briggs, A.J." <ATMABRIGGS@JUNO.COM>,
>         "Bryant, Cindy" <cbryant@kraft.com>,
>         "Burris, Jay" <jaybduck@ipa.net>,
>         "Burturm, Jodi" <jodi.burtrum@satellink.net >,
>         "Butler, Scott or Revonne" <butlers@hcil.net>,
>         "Christina Clark" <christinarclark@hotmail.com>,
>         "Clampit, Connie Joe" <clampitc@tyson.com>,
>         "Clifford Toney" <cliffordtoney@cox-internet.com>,
>         "Combs, Karrie" <karrie.combs@us.nestle.com>,
>         "David Haffner" <dhaffner@ipa.net>,
>         "Dennis Ditto" <dittohdditto@cs.com>,
>         "Eddie Young" <ebfarms@seark.net>,
>         "Eliza Bowman" <elbowman@monad.net>,
>         "Froning, Sara" <sfroning@kraft.com>,
>         "Gary McDonald Jr." <garymc@icnet.net>,
>         "Glen Hemminger" <gj.hemminger@home.com>,
>         "Good, Dennis" <Hogman9832@aol.com>,
>         "Janet Thornhill \(E-mail\)" <Janet@thornhillauto.net>,
>         "Jared Woodward" <Jdwood19@cs.com>,
>         "Joel Strickland" <joel@aboweb.com>,
>         "John Calhoun" <Calhoonj9@cs.com>,
>         "John Metz" <JTMETZ1@aol.com>,
>         "Jones, Shelley" <RTSRJ@aol>,
>         "Lacy, Scott" <Slacey@Sofnet.com>,
>         "Leonard, Tammy" <tndleonard@aol.com>,
>         "Linda Cline" <cline@somedayretrievers.com>,
>         "Lynn A. Yandell" <yandell@uark.edu>,
>         "M. D. Baxter" <baxram@bellsouth.net>,
>         "Mark So" <info@dogtra.com>,
>         "McDonald, Gary" <gmcdonald1@mmcable.com>,
>         "Michael Carter" <michaelraycarter@yahoo.com>,
>         "Mike Anthony" <mikeanth@earthlink.net>,
>         "Moon, Billy" <MoonPhase5@aol.com>,
>         "Myers, Chuck" <cemyers@gte.net>,
>         "Nancy Belknap" <nhah103@ipa.net>,
>         "Nancy R Oldfather" <Nancy.Oldfather@kraft.com>,
>         "Nathen C. Neal" <nneal@uark.edu>,
>         "Nunn, Randy" <RandHNunn@aol.com>,
>         "Owen Bybee" <owen@bomontana.com>,
>         "Perry Cox" <perrycox@hotmail.com>,
>         "Philip Hattaway" <phattaway@kc.rr.com>,
>         "Richard Davis" <richard@gofit.net>,
>         "Robert Clay Connor" <c2dg@midsouth.rr.com>,
>         "Roberts, Karen" <karenr@ipa.net>,
>         "Rose, Ernie & Pat" <bluegoose.kennels@bfsmedia.com>,
>         "Shelley. Allison" <shelley.allison@us.pwcglobal.com>,
>         "Stan Lewis" <slewis@iqcisp.com>,
>         "Stephen Wright" <stephen.wright@leggett.com>,
>         "Steve Vanduine" <vanduine@ukcdogs.com>,
>         "Sutton, Wes" <medicus@medicusrg.com>,
>         "Tami Baker" <cte87344@centurytel.net>,
>         "Ted Cullen" <info@etch-marc.com>,
>         "Teresa Fuller" <bsandtb@peoplepc.com>,
>         "Todd Wilken" <tjwilken@premiernet.net>,
>         "Tracy Johnston" <tracyjohnston@service-advantage.com>,
>         "Whelan, Ken" <ken@UBH.com>

Stop right there.  You *are* counting the sheer length of the "To:"
header as a spam indicator, right?  Perhaps alphabetized lists should
be considered an additional spam indicator?  (AFAIK your current
tokenizer doesn't pick up on the alphabetization I'm sure so you'll
never know how much information there is in that until you try it.
I've seen many spams with slices of the alphabet in the To: line.)

(And yes, I still plan to run an experiment on my own mailbox.  None
of the answers on how to use MH folders were sufficient, but I think
it'll be trivial to augment splitndirs.py and others to support the
same kind of argument that hammie.py supports.)

--Guido van Rossum (home page: http://www.python.org/~guido/)

From python-spambayes@discworld.dyndns.org  Fri Sep 20 17:13:38 2002
From: python-spambayes@discworld.dyndns.org (Charles Cazabon)
Date: Fri, 20 Sep 2002 10:13:38 -0600
Subject: [Spambayes] Spamvolution
In-Reply-To: <LNBBLJKPBEHFEDALKOLCIEHJBFAB.tim.one@comcast.net>;
	from tim.one@comcast.net on Fri, Sep 20, 2002 at 12:02:43PM -0400
References: <LNBBLJKPBEHFEDALKOLCIEHJBFAB.tim.one@comcast.net>
Message-ID: <20020920101338.A16996@discworld.dyndns.org>

Tim Peters <tim.one@comcast.net> wrote:
> 
> What we may be left with is spam like this (a persistent false negative in
> my test data):
> Return-Path: <kba94@earthlink.net>
> Delivered-To: bfsmedia-bluegoose.kennels@bfsmedia.com
> Received: (qmail 16482 invoked from network); 11 Mar 2002 16:01:28 -0000
> Received: from harrier.mail.pas.earthlink.net (HELO
> harrier.prod.itd.earthlink.net) (207.217.120.12)
>   by agamemnon.bfsmedia.com with SMTP; 11 Mar 2002 16:01:28 -0000
> Received: from pool-209-128-140-231.gent.ipa.net ([209.128.140.231]
> helo=13vys)
>         by harrier.prod.itd.earthlink.net with smtp (Exim 3.33 #1)
>         id 16kSN8-0004nj-00; Mon, 11 Mar 2002 08:10:04 -0800
[...] 
> What the hell was *that* all about <wink>?  It actually scores very near the
> spam threshold, because bfsmedia.com shows up in the headers, and they're
> the source of many surreal spams of this nature.

Sorry, Tim, but that's not correct.  Bruce was my cubicle-mate while he was
collecting most of that spam, and "bfsmedia" is the company that owns the
machines that mail was collected on (they're about ten meters from where I'm
currently sitting).  The source of this message was actually
pool-209-128-140-231.gent.ipa.net ([209.128.140.231].

I'm not so sure this false negative isn't actually a true negative, or at
worst a misaddressed piece of mail.  It doesn't appear to have been trying to
sell anything.

> Or perhaps this really is ham!  Both URLs are legit, and still in business.

My thoughts.

munching-on-ham-'ly,

Charles
-- 
-----------------------------------------------------------------------
Charles Cazabon                 <python-spambayes@discworld.dyndns.org>
GPL'ed software available at:     http://www.qcc.ca/~charlesc/software/
-----------------------------------------------------------------------

From python-spambayes@discworld.dyndns.org  Fri Sep 20 17:25:09 2002
From: python-spambayes@discworld.dyndns.org (Charles Cazabon)
Date: Fri, 20 Sep 2002 10:25:09 -0600
Subject: [Spambayes] Spamvolution
In-Reply-To: 
	<200209201615.g8KGFK109828@pcp02138704pcs.reston01.va.comcast.net>; from
	guido@python.org on Fri, Sep 20, 2002 at 12:15:20PM -0400
References: <LNBBLJKPBEHFEDALKOLCIEHJBFAB.tim.one@comcast.net>
	<20020920101338.A16996@discworld.dyndns.org>
	<200209201615.g8KGFK109828@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <20020920102509.C16996@discworld.dyndns.org>

Guido van Rossum <guido@python.org> wrote:
> > Sorry, Tim, but that's not correct.  Bruce was my cubicle-mate while
> > he was collecting most of that spam, and "bfsmedia" is the company
> > that owns the machines that mail was collected on (they're about ten
> > meters from where I'm currently sitting).  The source of this
> > message was actually pool-209-128-140-231.gent.ipa.net
> > ([209.128.140.231].
> 
> Is any of the To: addresses amongst Bruce's set of bait addresses?

Nope.  This message was delivered to "Rose, Ernie & Pat"
<bluegoose.kennels@bfsmedia.com>, which is why
"bfsmedia-bluegood.kennels@bfsmedia.com" shows up in the Delivered-To: line at
the top of the message (it's a qmail-ism resulting from the bfsmedia.com being
a virtual domain -- i.e. the addresses in it /don't/ correspond to Unix
accounts on that server).

This looks like it doesn't belong in the spam corpus -- it's a legitimate
(albeit mass-addressed) piece of mail.  If you and your prize dog were going
to be on national TV for the first time, wouldn't you email everyone in your
addressbook, telling them to watch?  <wink>

Charles
-- 
-----------------------------------------------------------------------
Charles Cazabon                 <python-spambayes@discworld.dyndns.org>
GPL'ed software available at:     http://www.qcc.ca/~charlesc/software/
-----------------------------------------------------------------------

From guido@python.org  Fri Sep 20 17:15:20 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 20 Sep 2002 12:15:20 -0400
Subject: [Spambayes] Spamvolution
In-Reply-To: Your message of "Fri, 20 Sep 2002 10:13:38 MDT."
             <20020920101338.A16996@discworld.dyndns.org> 
References: <LNBBLJKPBEHFEDALKOLCIEHJBFAB.tim.one@comcast.net>  
            <20020920101338.A16996@discworld.dyndns.org> 
Message-ID: <200209201615.g8KGFK109828@pcp02138704pcs.reston01.va.comcast.net>

> Sorry, Tim, but that's not correct.  Bruce was my cubicle-mate while
> he was collecting most of that spam, and "bfsmedia" is the company
> that owns the machines that mail was collected on (they're about ten
> meters from where I'm currently sitting).  The source of this
> message was actually pool-209-128-140-231.gent.ipa.net
> ([209.128.140.231].

Is any of the To: addresses amongst Bruce's set of bait addresses?

> I'm not so sure this false negative isn't actually a true negative,
> or at worst a misaddressed piece of mail.

I've seen many spams trying to pose as a misaddressed piece of mail.

> It doesn't appear to have been trying to sell anything.

Oh yes it does: it tries to get you to watch a specific show on ESPN.

It's possible that it started out as a legit piece of mail that was
given large distribution by a spammer with above-average (for
spammers!) intelligence.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido@python.org  Fri Sep 20 17:27:40 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 20 Sep 2002 12:27:40 -0400
Subject: [Spambayes] Spamvolution
In-Reply-To: Your message of "Fri, 20 Sep 2002 10:25:09 MDT."
             <20020920102509.C16996@discworld.dyndns.org> 
References: <LNBBLJKPBEHFEDALKOLCIEHJBFAB.tim.one@comcast.net>
	<20020920101338.A16996@discworld.dyndns.org>
	<200209201615.g8KGFK109828@pcp02138704pcs.reston01.va.comcast.net>  
	<20020920102509.C16996@discworld.dyndns.org> 
Message-ID: <200209201627.g8KGReE09914@pcp02138704pcs.reston01.va.comcast.net>

> > Is any of the To: addresses amongst Bruce's set of bait addresses?
> 
> Nope.  This message was delivered to "Rose, Ernie & Pat"
> <bluegoose.kennels@bfsmedia.com>, which is why
> "bfsmedia-bluegood.kennels@bfsmedia.com" shows up in the
> Delivered-To: line at the top of the message (it's a qmail-ism
> resulting from the bfsmedia.com being a virtual domain -- i.e. the
> addresses in it /don't/ correspond to Unix accounts on that server).

Ah, and that's in the To: list as well.

> This looks like it doesn't belong in the spam corpus -- it's a
> legitimate (albeit mass-addressed) piece of mail.  If you and your
> prize dog were going to be on national TV for the first time,
> wouldn't you email everyone in your addressbook, telling them to
> watch?  <wink>

Looks like you're right -- Rose, Ernie & Pat must be colleagues of
Bruce at bfsmedia then.  But then how did it end up in Bruce's spam
collection?  Forwarded by a colleague?  Or did he look at all messages
coming in through that domain regardless of destination?

--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.one@comcast.net  Fri Sep 20 17:30:16 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 20 Sep 2002 12:30:16 -0400
Subject: [Spambayes] Spamvolution
In-Reply-To: <20020920101338.A16996@discworld.dyndns.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCMEHLBFAB.tim.one@comcast.net>

[Charles Cazabon]
> Sorry, Tim, but that's not correct.  Bruce was my cubicle-mate
> while he was collecting most of that spam, and "bfsmedia" is the company
> that owns the machines that mail was collected on (they're about ten
> meters  from where I'm currently sitting).  The source of this message
> was actually pool-209-128-140-231.gent.ipa.net ([209.128.140.231].

Oh, fudge.  I was afraid of that:  despite that I'm ignoring almost all the
header lines (and so running a crippled version of the algorithm), this
bogus clue about my mixed-source corpora is still sneaking in.

Oops!  It's actually not!  I'm actually ignoring the header lines in which
bfsmedia appears -- that's a correlation I had noticed by *eyeball*.  The
high-prob spam words in that were really

prob('competition') = 0.858033
prob('grand') = 0.863717
prob('qualified') = 0.869686
prob('air') = 0.869976
prob('outstanding') = 0.887064
prob('compete') = 0.928292
prob('country') = 0.941092
prob('subject:Great') = 0.945813
prob('outdoor') = 0.99
prob('finalists') = 0.99
prob('subject:Outdoor') = 0.99
prob('eagle') = 0.99
prob('retriever') = 0.99
prob('allison') = 0.99

> I'm not so sure this false negative isn't actually a true negative, or at
> worst a misaddressed piece of mail.  It doesn't appear to have
> been trying to sell anything.

It sure didn't.  The collection of To: addresses was spammish in its sheer
bulk, but very hammish in the wide variety of target domains, and in having
a "real name" to go along with each email address.  For the life of me I
can't find one of Bruce's bait addresses in it, either.

Tester's dilemma:  do I take this out of the spam set or not?  I removed two
others in the past, like output from a cron job Bruce apparently arranged to
mail to himself from one his spam collection machines.  This one is much
muddier than those, though.  Screw it:  common sense rules.  It's ham.


From guido@python.org  Fri Sep 20 17:29:28 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 20 Sep 2002 12:29:28 -0400
Subject: [Spambayes] Spamvolution
In-Reply-To: Your message of "Fri, 20 Sep 2002 12:27:40 EDT."
	<200209201627.g8KGReE09914@pcp02138704pcs.reston01.va.comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCIEHJBFAB.tim.one@comcast.net>
	<20020920101338.A16996@discworld.dyndns.org>
	<200209201615.g8KGFK109828@pcp02138704pcs.reston01.va.comcast.net>
	<20020920102509.C16996@discworld.dyndns.org>  
	<200209201627.g8KGReE09914@pcp02138704pcs.reston01.va.comcast.net> 
Message-ID: <200209201629.g8KGTSw09940@pcp02138704pcs.reston01.va.comcast.net>

[me]
> Looks like you're right -- Rose, Ernie & Pat must be colleagues of
> Bruce at bfsmedia then.  But then how did it end up in Bruce's spam
> collection?  Forwarded by a colleague?  Or did he look at all messages
> coming in through that domain regardless of destination?

Maybe we should ask Bruce?

--Guido van Rossum (home page: http://www.python.org/~guido/)

From python-spambayes@discworld.dyndns.org  Fri Sep 20 17:34:28 2002
From: python-spambayes@discworld.dyndns.org (Charles Cazabon)
Date: Fri, 20 Sep 2002 10:34:28 -0600
Subject: [Spambayes] Spamvolution
In-Reply-To: 
	<200209201627.g8KGReE09914@pcp02138704pcs.reston01.va.comcast.net>; from
	guido@python.org on Fri, Sep 20, 2002 at 12:27:40PM -0400
References: <LNBBLJKPBEHFEDALKOLCIEHJBFAB.tim.one@comcast.net>
	<20020920101338.A16996@discworld.dyndns.org>
	<200209201615.g8KGFK109828@pcp02138704pcs.reston01.va.comcast.net>
	<20020920102509.C16996@discworld.dyndns.org>
	<200209201627.g8KGReE09914@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <20020920103428.A17250@discworld.dyndns.org>

Guido van Rossum <guido@python.org> wrote:
> 
> > This looks like it doesn't belong in the spam corpus -- it's a
> > legitimate (albeit mass-addressed) piece of mail.  If you and your
> > prize dog were going to be on national TV for the first time,
> > wouldn't you email everyone in your addressbook, telling them to
> > watch?  <wink>
> 
> Looks like you're right -- Rose, Ernie & Pat must be colleagues of
> Bruce at bfsmedia then.  But then how did it end up in Bruce's spam
> collection?  Forwarded by a colleague?  Or did he look at all messages
> coming in through that domain regardless of destination?

Bruce's spam collection came from a /lot/ of sources.  Some of it was spam
received at his various legitimate addresses (i.e. manually filtered out and
saved to his spam Maildir).  Other sources were various bait addresses in
multiple domains he set up, which automatically filed all received mail to his
spam collection.  In this case, I suspect the message was received
legitimately and simply mis-filed by hand -- human error.

On the plus side, it means Tim can get rid of his most stubborn false negative
:).

Charles
-- 
-----------------------------------------------------------------------
Charles Cazabon                 <python-spambayes@discworld.dyndns.org>
GPL'ed software available at:     http://www.qcc.ca/~charlesc/software/
-----------------------------------------------------------------------

From python-spambayes@discworld.dyndns.org  Fri Sep 20 17:36:58 2002
From: python-spambayes@discworld.dyndns.org (Charles Cazabon)
Date: Fri, 20 Sep 2002 10:36:58 -0600
Subject: [Spambayes] Spamvolution
In-Reply-To: <LNBBLJKPBEHFEDALKOLCMEHLBFAB.tim.one@comcast.net>;
	from tim.one@comcast.net on Fri, Sep 20, 2002 at 12:30:16PM -0400
References: <20020920101338.A16996@discworld.dyndns.org>
	<LNBBLJKPBEHFEDALKOLCMEHLBFAB.tim.one@comcast.net>
Message-ID: <20020920103658.B17250@discworld.dyndns.org>

Tim Peters <tim.one@comcast.net> wrote:
> 
> Oops!  It's actually not!  I'm actually ignoring the header lines in which
> bfsmedia appears -- that's a correlation I had noticed by *eyeball*.  The
> high-prob spam words in that were really
> 
> prob('competition') = 0.858033
> prob('grand') = 0.863717
> prob('qualified') = 0.869686
> prob('air') = 0.869976
> prob('outstanding') = 0.887064
> prob('compete') = 0.928292
> prob('country') = 0.941092
> prob('subject:Great') = 0.945813
> prob('outdoor') = 0.99
> prob('finalists') = 0.99
> prob('subject:Outdoor') = 0.99
> prob('eagle') = 0.99
> prob('retriever') = 0.99
> prob('allison') = 0.99

So the author simply had a very spammy writing style.  Unfortunate for them,
if they still write that way when spambayes takes over the world :).

> bulk, but very hammish in the wide variety of target domains, and in having
> a "real name" to go along with each email address.  For the life of me I
> can't find one of Bruce's bait addresses in it, either.

See my exchange with Guido; it appears to have been mistakenly misfiled in
Bruce's spam corpus.

> Tester's dilemma:  do I take this out of the spam set or not?  I removed two
> others in the past, like output from a cron job Bruce apparently arranged to
> mail to himself from one his spam collection machines.  This one is much
> muddier than those, though.  Screw it:  common sense rules.  It's ham.

Excellent.

Charles
-- 
-----------------------------------------------------------------------
Charles Cazabon                 <python-spambayes@discworld.dyndns.org>
GPL'ed software available at:     http://www.qcc.ca/~charlesc/software/
-----------------------------------------------------------------------

From tim.one@comcast.net  Fri Sep 20 17:46:32 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 20 Sep 2002 12:46:32 -0400
Subject: [Spambayes] Spamvolution
In-Reply-To: <200209201609.g8KG9cd09311@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEHOBFAB.tim.one@comcast.net>

[Guido]
> Stop right there.  You *are* counting the sheer length of the "To:"
> header as a spam indicator, right?

Not yet, no.  In fact, I don't do *anything* with To: lines except count the
number of times a To header appears, and the case ("To" vs "TO" vs "to" vs
"tO" -- and you shouldn't either.  If I enable tokenizing To lines on my
corpus, then

    To: python-list@python.org

becomes a nearly 100% reliable spam vs non-spam discriminator on my
mixed-source corpora.  A similar effect will hold in your mixed-source
corpora.  Likewise I'm not counting the number of Xref targets either (as a
spammish cross-posting indicator), or, indeed, even counting whether an Xref
header *exists* (huge slices of my ham come from newsgroup postings, but
none of my spam does).  You should trust that I know what I'm doing by now
<wink>.

> Perhaps alphabetized lists should be considered an additional spam
> indicator?

I don't believe this is really spam.  It's also rare for the spam *I* get to
have "Tim Peters" in the To -- it's almost always just tim_one@msn.com, or
something like that.  The msg in question had a "real name" for everyone in
the To list.  Intelligent To tokenization should probably take that into
account too.

> (AFAIK your current tokenizer doesn't pick up on the alphabetization
> I'm sure so you'll never know how much information there is in that
> until you try it.

BOth my error rates are so low relative to my corpus size that I can no
longer demonstrate improvements; I can only demonstrate that an idea isn't a
disaster.  Until I get a new corpus, everyone else has to pitch in to
demonstrate improvements.

> I've seen many spams with slices of the alphabet in the To: line.)

Yup.  But the few email "groups" I have set up in my own mailer are
automatically alphabetized by Outlook.  The only way to know is to code it
and see what the data says, and on a corpus that's doing worse than mine is.

> (And yes, I still plan to run an experiment on my own mailbox.  None
> of the answers on how to use MH folders were sufficient, but I think
> it'll be trivial to augment splitndirs.py and others to support the
> same kind of argument that hammie.py supports.)

You should have reminded me yesterday!  I thought the symbolic-link idea had
real promise, at least until it got to a rebalancing step.


From python-spambayes@discworld.dyndns.org  Fri Sep 20 17:53:43 2002
From: python-spambayes@discworld.dyndns.org (Charles Cazabon)
Date: Fri, 20 Sep 2002 10:53:43 -0600
Subject: [Spambayes] Spamvolution
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCEHOBFAB.tim.one@comcast.net>;
	from tim.one@comcast.net on Fri, Sep 20, 2002 at 12:46:32PM -0400
References: <200209201609.g8KG9cd09311@pcp02138704pcs.reston01.va.comcast.net>
	<LNBBLJKPBEHFEDALKOLCCEHOBFAB.tim.one@comcast.net>
Message-ID: <20020920105343.A17418@discworld.dyndns.org>

Tim Peters <tim.one@comcast.net> wrote:
> 
> > I've seen many spams with slices of the alphabet in the To: line.)
> 
> Yup.  But the few email "groups" I have set up in my own mailer are
> automatically alphabetized by Outlook.  The only way to know is to code it
> and see what the data says, and on a corpus that's doing worse than mine is.

Mostly older spam -- as people implemented tarpitting and other spam filtering
methods that penalized messages arriving with large numbers of RCPT TO
commands, spammers started adapting and not putting so many recipients in the
To: headers of their messages.  Three years ago, virtually all the spam I
received had 20+ To/cc header recipients -- and today, over 90% of my
spam has a single address in the To: field (although it's normally not one of
mine :).

Charles
-- 
-----------------------------------------------------------------------
Charles Cazabon                 <python-spambayes@discworld.dyndns.org>
GPL'ed software available at:     http://www.qcc.ca/~charlesc/software/
-----------------------------------------------------------------------

From marklists@mceahern.com  Fri Sep 20 17:58:42 2002
From: marklists@mceahern.com (Mark McEahern)
Date: Fri, 20 Sep 2002 11:58:42 -0500
Subject: [Spambayes] Spamvolution
In-Reply-To: <LNBBLJKPBEHFEDALKOLCIEHJBFAB.tim.one@comcast.net>
Message-ID: <JHEOKEOOLIGLDHCMAHMOGEDHEAAA.marklists@mceahern.com>

[Tim Peters]
> What the hell was *that* all about <wink>?

You've seen Best In Show, of course?

// m

From guido@python.org  Fri Sep 20 18:01:04 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 20 Sep 2002 13:01:04 -0400
Subject: [Spambayes] Spamvolution
In-Reply-To: Your message of "Fri, 20 Sep 2002 11:58:42 CDT."
             <JHEOKEOOLIGLDHCMAHMOGEDHEAAA.marklists@mceahern.com> 
References: <JHEOKEOOLIGLDHCMAHMOGEDHEAAA.marklists@mceahern.com> 
Message-ID: <200209201701.g8KH14B11169@pcp02138704pcs.reston01.va.comcast.net>

> You've seen Best In Show, of course?

Yes!  Very funny movie, reminding me of the funniest show on Dutch tv
when I was still there (Koot & Bie, for Dutch folks reading this).

--Guido van Rossum (home page: http://www.python.org/~guido/)

From anthony@interlink.com.au  Fri Sep 20 18:25:06 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Sat, 21 Sep 2002 03:25:06 +1000
Subject: [Spambayes] Spamvolution 
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCEHOBFAB.tim.one@comcast.net> 
Message-ID: <200209201725.g8KHP6V17373@localhost.localdomain>

>>> Tim Peters wrote
> Not yet, no.  In fact, I don't do *anything* with To: lines except count the
> number of times a To header appears, and the case ("To" vs "TO" vs "to" vs
> "tO" -- and you shouldn't either.  If I enable tokenizing To lines on my
> corpus, then

Hm. Is it just me, or are
  To: a@b.c
  To: b@d.e
  To: d@f.g
and 
  To: a@b.c, b@d.e, d@f.g
of equal value in spamminess indication? What other headers have
multiple entries combined in this way? Just to: and cc: ?

Anthony

From tim.one@comcast.net  Fri Sep 20 18:30:09 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 20 Sep 2002 13:30:09 -0400
Subject: [Spambayes] Spamvolution
In-Reply-To: <20020920103428.A17250@discworld.dyndns.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCMEIBBFAB.tim.one@comcast.net>

[Charles Cazabon]
> ...
> In this case, I suspect the message was received legitimately and
> simply mis-filed by hand -- human error.

It bears repeating that I believe the error rates I'm seeing now (both of
them) are better than I could do by hand.  This isn't the first misfiling
the system has uncovered in the spam corpus either.

It's also worth mentioning that my chance of dying within the next year is
higher than the chance that my current classifier would deliver a false
positive on c.l.py traffic within the next year (assuming the nature of
c.l.py and spam in my test data is representative of what c.l.py will see
over the next year, and that the volume of c.l.py traffic doesn't
dramatically increase).

> On the plus side, it means Tim can get rid of his most stubborn
> false negative :).

'Fraid not -- that was just the most *mysterious* f-n.  It's all to the
credit of the scheme that this one never went away!  There are others that
have also never gone away, but are such blatant spam that, when you look at
one, you'd swear the system is worthless.


From tim.one@comcast.net  Fri Sep 20 18:37:18 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 20 Sep 2002 13:37:18 -0400
Subject: [Spambayes] Spamvolution
In-Reply-To: <20020920103658.B17250@discworld.dyndns.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCMEICBFAB.tim.one@comcast.net>

> > prob('competition') = 0.858033
> > prob('grand') = 0.863717
> > prob('qualified') = 0.869686
> > prob('air') = 0.869976
> > prob('outstanding') = 0.887064
> > prob('compete') = 0.928292
> > prob('country') = 0.941092
> > prob('subject:Great') = 0.945813
> > prob('outdoor') = 0.99
> > prob('finalists') = 0.99
> > prob('subject:Outdoor') = 0.99
> > prob('eagle') = 0.99
> > prob('retriever') = 0.99
> > prob('allison') = 0.99

[Charles Cazabon]
> So the author simply had a very spammy writing style.

I don't think so -- it's more that c.l.py just doesn't talk about eagles and
retrievers much, and never also in the context in competitions and
finalists.  We won't even mention that Allison bitch <wink>.

> Unfortunate for them, if they still write that way when spambayes takes
> over the world :).

If I were in the dog torturing business <wink>, I expect these very same
words would be *ham* indicators-- or neutral --in my database.

An interesting experiment:  do a full run, get the list of the best N
discriminators, *purge* those words from the database, and try predicting
again.  I've tried that a few times (but not recently), and found no
significant difference in outcomes.  You take away one set of "killer
strong" clues, and the system finds others seemingly just as good.  That's
one concrete (albeit indirect) reason to be optimistic about completing
Gary's agenda.


From tim.one@comcast.net  Fri Sep 20 18:41:47 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 20 Sep 2002 13:41:47 -0400
Subject: [Spambayes] Spamvolution
In-Reply-To: <200209201725.g8KHP6V17373@localhost.localdomain>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEIEBFAB.tim.one@comcast.net>

>> Not yet, no.  In fact, I don't do *anything* with To: lines
>> except count the number of times a To header appears, and the case
>> ("To" vs "TO" vs "to" vs "tO" -- and you shouldn't either.


[Anthony Baxter]
> Hm. Is it just me, or are
>   To: a@b.c
>   To: b@d.e
>   To: d@f.g
> and
>   To: a@b.c, b@d.e, d@f.g
> of equal value in spamminess indication?

It's the mere presence of *any* To header that's a ham indicator in my
database.  For whatever reason, a significant number of spams simply don't
have a To: header (maybe to hide the recipient list?).  Jeremy recently
added some code (that's not yet enabled by default) to get better value from
this (the current defaults infer ham from the presence of To, but don't
infer spam from its absence).

> What other headers have multiple entries combined in this way? Just
> to: and cc: ?

Barry's always eating lunch when you need him <wink>.


From tim.one@comcast.net  Fri Sep 20 20:55:06 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 20 Sep 2002 15:55:06 -0400
Subject: [Spambayes] Spamvolution
In-Reply-To: <20020920105343.A17418@discworld.dyndns.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCEEIOBFAB.tim.one@comcast.net>

Charles, while we've got your supernatural insight on the line here, can you
shed any light on this putative spam?

"""
Return-Path: <paige69@cranfield.ac.uk>
Delivered-To: bfsmedia-sales@bfsmedia.com
Received: (qmail 1012 invoked from network); 10 Mar 2002 18:06:08 -0000
Received: from unknown (HELO goforhost.com) (208.178.66.38)
  by agamemnon.bfsmedia.com with SMTP; 10 Mar 2002 18:06:08 -0000
Date: Sun, 10 Mar 2002 13:10:19 -0500
Message-Id: <200203101810.g2AI9sE21236@goforhost.com>
X-Mailer: Gnus/5.090001 (Oort Gnus v0.01) XEmacs/21.2 (Terspichore)
Reply-To: <paige69@cranfield.ac.uk>
From: <paige69@cranfield.ac.uk>
To: <sales@bg-ep.com>
Subject: Hello Tim
Content-Length: 106
Lines: 9

Tim,


 It was great to talk to you today I should have the propsal done by
tommorrow


Take Care,

Susan
"""

There's little about this that looks spammish to me (or to my classifier),
and it's another of my persistent false negatives.


From python-spambayes@discworld.dyndns.org  Fri Sep 20 21:15:26 2002
From: python-spambayes@discworld.dyndns.org (Charles Cazabon)
Date: Fri, 20 Sep 2002 14:15:26 -0600
Subject: [Spambayes] Spamvolution
In-Reply-To: <LNBBLJKPBEHFEDALKOLCEEIOBFAB.tim.one@comcast.net>;
	from tim.one@comcast.net on Fri, Sep 20, 2002 at 03:55:06PM -0400
References: <20020920105343.A17418@discworld.dyndns.org>
	<LNBBLJKPBEHFEDALKOLCEEIOBFAB.tim.one@comcast.net>
Message-ID: <20020920141526.A18892@discworld.dyndns.org>

Tim Peters <tim.one@comcast.net> wrote:
> Charles, while we've got your supernatural insight on the line here

Boggle.  I'm a pretty good mail administrator, but I'm not sure I qualify for
that adjective :).

> can you shed any light on this putative spam?

> Return-Path: <paige69@cranfield.ac.uk>
> Delivered-To: bfsmedia-sales@bfsmedia.com

The envelope.  Suspicious enough; the envelope sender's local-part is a common
name with a couple decimal digits tacked on, and some random domain.  The
envelope recipient was <sales@bfsmedia.com>, which was almost certainly
harvested off a web page.

> Received: (qmail 1012 invoked from network); 10 Mar 2002 18:06:08 -0000
> Received: from unknown (HELO goforhost.com) (208.178.66.38)
>   by agamemnon.bfsmedia.com with SMTP; 10 Mar 2002 18:06:08 -0000

Received from an MTA on an IP address without a valid PTR record, if Bruce had
DNS lookups enabled at the time.  I don't know if he did or not.  The machine
in question announced itself with the name goforhost.com, which appears to be
forged.

> Date: Sun, 10 Mar 2002 13:10:19 -0500
> Message-Id: <200203101810.g2AI9sE21236@goforhost.com>
> X-Mailer: Gnus/5.090001 (Oort Gnus v0.01) XEmacs/21.2 (Terspichore)

Older spam usually had the name of the spamming software in an X-Mailer or
similar header (programs like BlastAway and SuperMailer and crap like that).
Lately they've taken to using the names of real MUAs, but usually it's Outlook
they fake.  I've personally never seen a spam written with Gnus or claiming to
be written with Gnus, so that makes me think this is a ham.

> Reply-To: <paige69@cranfield.ac.uk>
> From: <paige69@cranfield.ac.uk>

The From: header and Reply-To: header match the envelope sender address, which
is just about never the case with spam.  Another clue it's a ham.

> To: <sales@bg-ep.com>
> Subject: Hello Tim
> Content-Length: 106
> Lines: 9
> 
> Tim,
> 
> 
>  It was great to talk to you today I should have the propsal done by
> tommorrow
> 
> 
> Take Care,
> 
> Susan

> There's little about this that looks spammish to me (or to my classifier),
> and it's another of my persistent false negatives.

I have to agree it's a ham, but the only suspicious bits about it are:

-the envelope
-the host it came from
-the presence of "Hello" in the subject line.  This is a definitive clue in my
corpus, but I don't know how you're tokenizing the subject.  I haven't built
Python2.3 out of cvs yet on my little P90 mail server at home.

And, of course, the fact that there's not enough context in the body for it to
mean anything to me.

Out of curiosity, how does "propsal" score in your corpus?  Misspellings of
simple English words are far more common in my spam than my ham.

Charles
-- 
-----------------------------------------------------------------------
Charles Cazabon                 <python-spambayes@discworld.dyndns.org>
GPL'ed software available at:     http://www.qcc.ca/~charlesc/software/
-----------------------------------------------------------------------

From gward@python.net  Fri Sep 20 21:26:20 2002
From: gward@python.net (Greg Ward)
Date: Fri, 20 Sep 2002 16:26:20 -0400
Subject: [Spambayes] Spamvolution
In-Reply-To: <LNBBLJKPBEHFEDALKOLCEEIOBFAB.tim.one@comcast.net>
References: <20020920105343.A17418@discworld.dyndns.org>
	<LNBBLJKPBEHFEDALKOLCEEIOBFAB.tim.one@comcast.net>
Message-ID: <20020920202620.GA25498@cthulhu.gerg.ca>

On 20 September 2002, Tim Peters said:
> Charles, while we've got your supernatural insight on the line here, can you
> shed any light on this putative spam?
> 
> """
[...headers skipped...]
> Tim,
> 
> 
>  It was great to talk to you today I should have the propsal done by
> tommorrow
> 
> 
> Take Care,
> 
> Susan
> """
> 
> There's little about this that looks spammish to me (or to my classifier),
> and it's another of my persistent false negatives.

I believe that's a new-ish virus that's been making the rounds this
summer -- it appears in that basic form, with various common English
male and female names in place of "Tim" and "Susan".

Sure there were no attachments, or funky "I removed a virus and
forwarded you this junk anyways" crap from a stupid virus scanner?

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
I haven't lost my mind; I know exactly where I left it.

From tim.one@comcast.net  Fri Sep 20 21:52:48 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 20 Sep 2002 16:52:48 -0400
Subject: [Spambayes] Spamvolution
In-Reply-To: <20020920141526.A18892@discworld.dyndns.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEJEBFAB.tim.one@comcast.net>

>> Charles, while we've got your supernatural insight on the line here

[Charles Cazabon]
> Boggle.  I'm a pretty good mail administrator, but I'm not sure I
> qualify for that adjective :).

I submit that you went on to prove that you do <0.6 wink -- I don't even
play a mail administrator on TV>.

>> can you shed any light on this putative spam?
>>
>> Return-Path: <paige69@cranfield.ac.uk>
>> Delivered-To: bfsmedia-sales@bfsmedia.com

> The envelope.  Suspicious enough; the envelope sender's
> local-part is a common name with a couple decimal digits tacked on,
> and some random domain.

Unless it actually came from Cranfield University.  A quick google search
suggests that Cranfield hands out email addresses like

    h.woodward@cranfield.ac.uk
    l.edwards@cranfield.ac.uk
    m.allen@cranfield.ac.uk
    f.steele@cranfield.ac.uk

though.  I'll add that knowledge to our classifier <wink>.  Ack,

    http://www.dei.unipd.it/~epitropo/spammers.txt

lists paige69@cranfield.ac.uk as someone who spammed them.  And--
oops! --google also found this very similar msg:

"""
Newsgroups: bionet.immunology
Subject: Hey Fred
Date: 7 Mar 2002 07:16:22 -0000
Organization: BIOSCI/MRC Human Genome Mapping Project Resource Centre
Lines: 11
Sender: nobody@hgmp.mrc.ac.uk
Message-ID: <200203070713.g277DEf07349@forserve.com>
NNTP-Posting-Host: mercury.hgmp.mrc.ac.uk
X-Trace: niobium.hgmp.mrc.ac.uk 1015485383 27678 193.62.192.80 (7 Mar 2002
07:16:23 GMT)
X-Complaints-To: news@net.bio.net
NNTP-Posting-Date: Thu, 7 Mar 2002 07:16:23 +0000 (UTC)
X-Received: from localhost (localhost [127.0.0.1])
	by mercury.hgmp.mrc.ac.uk (Postfix) with ESMTP id 54A6F17A96
	for <immuno@net.bio.net>; Thu,  7 Mar 2002 07:16:20 +0000 (GMT)
X-Received: from forserve.com (unknown [63.170.11.194])
	by mercury.hgmp.mrc.ac.uk (Postfix) with ESMTP id ED82517A7F
	for <immuno@net.bio.net>; Thu,  7 Mar 2002 07:16:16 +0000 (GMT)
X-Mailer: The Bat! (v1.53d) Educational
X-Reply-To: <paige69@cranfield.ac.uk>
X-To: <immunocise@bellsouth.net>
Xref: news.indiana.edu bionet.immunology:66

Fred,


  It was nice to talk to you today I will send the proposal tonight.



Thanks,
 Heidi
"""

At least Susan learned how to spell after changing his name to Heidi <wink>.

[... interesting analysis elided ...]
> ...
> I've personally never seen a spam written with Gnus or claiming to
> be written with Gnus, so that makes me think this is a ham.

That was the strongest thing that caught my eye, but that specific X-Mailer
name is prob 0.5 to my classifier.

> ...
> I have to agree it's a ham, but the only suspicious bits about it are:
>
> -the envelope
> -the host it came from
> -the presence of "Hello" in the subject line.  This is a
> definitive clue in my corpus, but I don't know how you're tokenizing
> the subject.

In the body, we fold case and just split on whitespace.  In the Subject
line, though, we preserve case and take alphanumeric runs, plus [$.%].

     prob('subject:Hello') = 0.521739
     prob('subject:Tim') = 0.01

in my database:  he got a lot of value out of abusing my name <snarl>.

> I haven't built Python2.3 out of cvs yet on my little P90 mail server
> at home.

Machines are cheap; Python is cheaper; no excuse <wink>.

> And, of course, the fact that there's not enough context in the
> body for it to mean anything to me.

Or to anyone else!  I'm going to keep it in the spam set.

> Out of curiosity, how does "propsal" score in your corpus?
> Misspellings of simple English words are far more common in my
> spam than my ham.

Misspellings in general, yes, "propsal" no.  That appears in exactly one of
my hams and one of my spams (this one).  Since the spam being scored was not
in the training set, the only trained occurrence was in ham; and because we
got rid of Graham's "minimum count" gimmick, the sole occurrence in a ham
made "propsal" a 0.01 clue.  I know, that sucks, but *overall* testing said
that getting rid of the minimum count gimmick was a clear win.  Gary
Robinson has a much more rational way to deal with words with small counts
(in this case, his scheme would make propsal about a 0.34 ham clue).  As is,
propsal's 0.01 exactly cancelled out the more damning

    prob('from:email addr:cranfield.ac.uk>') = 0.99

Thanks for the loan of your brain!


From tim.one@comcast.net  Fri Sep 20 22:02:52 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 20 Sep 2002 17:02:52 -0400
Subject: [Spambayes] Spamvolution
In-Reply-To: <20020920202620.GA25498@cthulhu.gerg.ca>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEJFBFAB.tim.one@comcast.net>

> ...
>>  It was great to talk to you today I should have the propsal done by
>> tommorrow

[Greg Ward]
> I believe that's a new-ish virus that's been making the rounds this
> summer -- it appears in that basic form, with various common English
> male and female names in place of "Tim" and "Susan".

I'm too fried -- I actually have a lot of these in my spam.  It's just that
*this* one is the only false negative remaining from that set -- it got
supernatural benefit out of putting "Tim" in the subject header, and from
misspelling proposal (someone on c.l.py made a propsal once too <wink>).
Under Robinson's combining scheme, it scores between 40 and 50 even so.

> Sure there were no attachments, or funky "I removed a virus and
> forwarded you this junk anyways" crap from a stupid virus scanner?

Believe it or not, that was the whole banana -- and they're all like that.
The

Content-Length: 106
Lines: 9

in the quoted headers was telling the truth, although one of the lines got
folded by mailer when I quoted the message (so you saw 10 lines in the body;
"tomorrow" belonged on the preceding line).

However, it *is* a worm, and a very smart one:  it's somehow managed to copy
itself all over this list of people who should know better <wink>.


From tim.one@comcast.net  Sat Sep 21 01:55:54 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 20 Sep 2002 20:55:54 -0400
Subject: [Spambayes] RE: spam detection via probability - actual results!
In-Reply-To: <200209200929.g8K9TJo11972@indus.ins.cwi.nl>
Message-ID: <LNBBLJKPBEHFEDALKOLCMEKABFAB.tim.one@comcast.net>

FYI, I'm stalled -- I screwed up my test data by mistakenly mixing the ham
reservoir at random into the spam sets (I intended to mix in from the spam
reservoir, of course).  This is proving a bitch to untangle again.

I discovered it while running a new all-default baseline in preparation for
testing more of Gary's ideas.  It looked a little suspicous <wink> that I
suddenly had hundreds of new false negatives.

OTOH, that's a darned interesting experiment in its own right!  Adding about
250 legit msgs into the spam set only caused 5 new false positives (out of
20,000 legit msgs), but a couple hundred new false negatives (which so far
look almost all to be the legit msgs I mixed in by mistake).


From nas@python.ca  Sat Sep 21 03:06:53 2002
From: nas@python.ca (Neil Schemenauer)
Date: Fri, 20 Sep 2002 19:06:53 -0700
Subject: [Spambayes] RE: spam detection via probability - actual results!
In-Reply-To: <LNBBLJKPBEHFEDALKOLCMEKABFAB.tim.one@comcast.net>
References: <200209200929.g8K9TJo11972@indus.ins.cwi.nl>
	<LNBBLJKPBEHFEDALKOLCMEKABFAB.tim.one@comcast.net>
Message-ID: <20020921020653.GA783@glacier.arctrix.com>

Tim Peters wrote:
> FYI, I'm stalled -- I screwed up my test data by mistakenly mixing the ham
> reservoir at random into the spam sets (I intended to mix in from the spam
> reservoir, of course).  This is proving a bitch to untangle again.

Time to use those clues that you where not letting the classifier use.
Looking at host and IP addresses in the "received" headers or Mailman
droppings in other headers should allow you to sort them out again.

  Neil

From tim.one@comcast.net  Sat Sep 21 03:29:39 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 20 Sep 2002 22:29:39 -0400
Subject: [Spambayes] RE: spam detection via probability - actual results!
In-Reply-To: <20020921020653.GA783@glacier.arctrix.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCOEKDBFAB.tim.one@comcast.net>

[Tim, screws up royally]

[Neil Schemenauer]
> Time to use those clues that you where not letting the classifier use.
> Looking at host and IP addresses in the "received" headers or Mailman
> droppings in other headers should allow you to sort them out again.

Damn, what a brilliant-- yet obvious <wink> --idea.  Thanks, Neil!  I'm
pretty fried.  It will be interesting to see whether it finds any more of
these.  The all-default scheme, and a few more steps in Gary's scheme (which
I haven't checked in yet -- more later) seem to zero in on these without
much trouble.  But it is an iterative process, and after 3 rounds of it I'm
ready to make the 4th the last ...


From tim.one@comcast.net  Sat Sep 21 04:29:19 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 20 Sep 2002 23:29:19 -0400
Subject: [Spambayes] Move closer to Gary's ideal
In-Reply-To: <B9B0BA48.16351%grobinson@transpose.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEKGBFAB.tim.one@comcast.net>

I've checked some new code in for the adventurous.  To try it, you can do

"""
[Classifier]
use_robinson_probability: True
use_robinson_combining: True
max_discriminators: 1500
[TestDriver]
spam_cutoff: 0.50
"""

"1500" was my lazy way of spelling infinity; for now, the code uses
math.frexp() to simulate unbounded dynamic float range instead of bothering
with logarithms; this also means the database entries are exactly the same
as they were before.  I left max_discriminators working because I suspect
we're going to want it again.

These options have no effect when the above is enabled:

"""
[Classifier]
hambias
spambias
min_spamprob
max_spamprob
"""

I hate all of those, so good riddance if they go <wink>.

Other options you may want to play with, but I don't recommend it unless
you've read the source material and think you know what you're doing:

"""
[Classifier]
# This one has no effect for now (it's easy to do, I just haven't gotten
# to it yet).
use_robinson_ranking: False

# The "a" parameter in Gary's prob adjustment.
robinson_probability_a = 1.0

# Likewise the "x" parameter -- it's like our current UNKNOWN_SPAMPROB.
robinson_probability_x = 0.5
"""

I'm still recovering from my corpus screwup and don't have a lot to say
about this yet.  Overall it seems to be doing as well as the all-default
scheme (our highly tuned and heavily fiddled Graham scheme)!  If it works
better than that, I won't be able to tell from my data (the all-default
scheme was working "too well" for me to demonstrate an improvement if one
were made).  I did notice it nail some difficult false negatives I don't
think the minprob/maxprob-hobbled Graham scheme would ever be able to nail.
So all signs are good so far, except maybe one:

There's one surprising/maybe-disturbing thing I've seen on all my little
random-subset runs (which are all I've run so far, interleaved with
re-cleaning my corpus):  there's not only "a middle ground" now, it's
essentially ALL "middle ground"!  Scores that's aren't due to my corpus
pollution are virtually all within 20 points of 50.  Here's a typical
histogram pair from a 10-fold c-v run on a random subset of 1000 ham and
1000 spam; the 6 lowest-scoring oddballs in the spam distro were in fact
bogus false negatives due to my corpus screwup (so picture those dots as
belong in the ham histogram instead):

Ham distribution for all runs:
* = 6 items
  0.00   0
  2.50   0
  5.00   0
  7.50   0
 10.00   0
 12.50   0
 15.00   0
 17.50   0
 20.00   0
 22.50   0
 25.00   0
 27.50   6 *
 30.00  35 ******
 32.50  98 *****************
 35.00 221 *************************************
 37.50 307 ****************************************************
 40.00 229 ***************************************
 42.50  78 *************
 45.00  20 ****
 47.50   5 *
 50.00   1 *
 52.50   0
 55.00   0
 57.50   0
 60.00   0
 62.50   0
 65.00   0
 67.50   0
 70.00   0
 72.50   0
 75.00   0
 77.50   0
 80.00   0
 82.50   0
 85.00   0
 87.50   0
 90.00   0
 92.50   0
 95.00   0
 97.50   0

Spam distribution for all runs:
* = 6 items
  0.00   0
  2.50   0
  5.00   0
  7.50   0
 10.00   0
 12.50   0
 15.00   0
 17.50   0
 20.00   0
 22.50   0
 25.00   0
 27.50   0
 30.00   0
 32.50   1 *
 35.00   1 *
 37.50   1 *
 40.00   1 *
 42.50   1 *
 45.00   1 *
 47.50   1 *
 50.00  28 *****
 52.50  64 ***********
 55.00 184 *******************************
 57.50 352 ***********************************************************
 60.00 295 **************************************************
 62.50  69 ************
 65.00   1 *
 67.50   0
 70.00   0
 72.50   0
 75.00   0
 77.50   0
 80.00   0
 82.50   0
 85.00   0
 87.50   0
 90.00   0
 92.50   0
 95.00   0
 97.50   0

I have to do other things now, but if anyone wants to play with what I
*would* do if I could <wink>, play with max_discriminators and see whether
reducing that helps spread this out.  A suspicion is that folding in endless
quantities of garbage words (spamprob so close to 0.5 that they're not
really clues at all) may be dragging everything toward 0.5 without giving a
real benefit.

The math in Graham's combining scheme is such that a prob 0.5 word has no
effect whatsoever on the outcome.  The math in Gary's combining scheme
doesn't appear to have the same property:  I believe adding a .5 prob word
there moves the outcome closer to neutrality.


From anthony@interlink.com.au  Sat Sep 21 04:40:14 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Sat, 21 Sep 2002 13:40:14 +1000
Subject: [Spambayes] RE: spam detection via probability - actual results! 
In-Reply-To: <LNBBLJKPBEHFEDALKOLCMEKABFAB.tim.one@comcast.net> 
Message-ID: <200209210340.g8L3eEA20570@localhost.localdomain>


>>> Tim Peters wrote
> FYI, I'm stalled -- I screwed up my test data by mistakenly mixing the ham
> reservoir at random into the spam sets (I intended to mix in from the spam
> reservoir, of course).  This is proving a bitch to untangle again.

I did this too - took me about 3 days to repeat the sorting. I now on a
nightly basis take a tarball of my spam/ham corpuses, and keep the last
7 around.

Anthony
-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.


From tim.one@comcast.net  Sat Sep 21 04:45:00 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 20 Sep 2002 23:45:00 -0400
Subject: [Spambayes] Move closer to Gary's ideal
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEKGBFAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCMEKHBFAB.tim.one@comcast.net>

[Tim]
> I've checked some new code in for the adventurous.

If anyone has already grabbed this, do a cvs up again before trying it.  I
fixed two cut 'n paste errors that will have an effect on the outcomes, but
I suspect minor effects.


From grobinson@transpose.com  Sat Sep 21 12:05:53 2002
From: grobinson@transpose.com (Gary Robinson)
Date: Sat, 21 Sep 2002 07:05:53 -0400
Subject: [Spambayes] Re: Move closer to Gary's ideal
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEKGBFAB.tim.one@comcast.net>
Message-ID: <B9B1CAD1.16427%grobinson@transpose.com>

Tim,

> The math in Graham's combining scheme is such that a prob 0.5 word has no
> effect whatsoever on the outcome.  The math in Gary's combining scheme
> doesn't appear to have the same property:  I believe adding a .5 prob word
> there moves the outcome closer to neutrality.

You are right, including everything does move the outcome closer to
neutrality BUT for classification purposes, the real question is simply...
is there more evidence for spam or more for ham?

Including everything lets us actually answer that question in a complete
way.

It lessen away that nice spread we've been looking at, but of course through
ranking or other means we could a nice spread back.

That is, the spread is still there, but the standard deviation is less, so
on an un-normalized graph everything looks compressed... but theoreticcally,
anything we could do with the spread before, we can still find a way to do
with the more compressed spread.

So... if we're doing well at classification this way, we will still be able
to do spread-based things like using sliders for cutoffs, or putting some
emails into a middle ground "you should look at this manually" group.

So I'll be very interested to see how well this does at binary
classification on smaller corpuses (and particularly on corpuses where the
testing and training data is different).


--Gary


-- 
Gary Robinson
CEO
Transpose, LLC
grobinson@transpose.com
207-942-3463
http://www.emergentmusic.com
http://radio.weblogs.com/0101454


> From: Tim Peters <tim.one@comcast.net>
> Date: Fri, 20 Sep 2002 23:29:19 -0400
> To: Gary Robinson <grobinson@transpose.com>, SpamBayes <spambayes@python.org>
> Cc: glouis@dynamicro.on.ca
> Subject: Move closer to Gary's ideal
> 
> I've checked some new code in for the adventurous.  To try it, you can do
> 
> """
> [Classifier]
> use_robinson_probability: True
> use_robinson_combining: True
> max_discriminators: 1500
> [TestDriver]
> spam_cutoff: 0.50
> """
> 
> "1500" was my lazy way of spelling infinity; for now, the code uses
> math.frexp() to simulate unbounded dynamic float range instead of bothering
> with logarithms; this also means the database entries are exactly the same
> as they were before.  I left max_discriminators working because I suspect
> we're going to want it again.
> 
> These options have no effect when the above is enabled:
> 
> """
> [Classifier]
> hambias
> spambias
> min_spamprob
> max_spamprob
> """
> 
> I hate all of those, so good riddance if they go <wink>.
> 
> Other options you may want to play with, but I don't recommend it unless
> you've read the source material and think you know what you're doing:
> 
> """
> [Classifier]
> # This one has no effect for now (it's easy to do, I just haven't gotten
> # to it yet).
> use_robinson_ranking: False
> 
> # The "a" parameter in Gary's prob adjustment.
> robinson_probability_a = 1.0
> 
> # Likewise the "x" parameter -- it's like our current UNKNOWN_SPAMPROB.
> robinson_probability_x = 0.5
> """
> 
> I'm still recovering from my corpus screwup and don't have a lot to say
> about this yet.  Overall it seems to be doing as well as the all-default
> scheme (our highly tuned and heavily fiddled Graham scheme)!  If it works
> better than that, I won't be able to tell from my data (the all-default
> scheme was working "too well" for me to demonstrate an improvement if one
> were made).  I did notice it nail some difficult false negatives I don't
> think the minprob/maxprob-hobbled Graham scheme would ever be able to nail.
> So all signs are good so far, except maybe one:
> 
> There's one surprising/maybe-disturbing thing I've seen on all my little
> random-subset runs (which are all I've run so far, interleaved with
> re-cleaning my corpus):  there's not only "a middle ground" now, it's
> essentially ALL "middle ground"!  Scores that's aren't due to my corpus
> pollution are virtually all within 20 points of 50.  Here's a typical
> histogram pair from a 10-fold c-v run on a random subset of 1000 ham and
> 1000 spam; the 6 lowest-scoring oddballs in the spam distro were in fact
> bogus false negatives due to my corpus screwup (so picture those dots as
> belong in the ham histogram instead):
> 
> Ham distribution for all runs:
> * = 6 items
> 0.00   0
> 2.50   0
> 5.00   0
> 7.50   0
> 10.00   0
> 12.50   0
> 15.00   0
> 17.50   0
> 20.00   0
> 22.50   0
> 25.00   0
> 27.50   6 *
> 30.00  35 ******
> 32.50  98 *****************
> 35.00 221 *************************************
> 37.50 307 ****************************************************
> 40.00 229 ***************************************
> 42.50  78 *************
> 45.00  20 ****
> 47.50   5 *
> 50.00   1 *
> 52.50   0
> 55.00   0
> 57.50   0
> 60.00   0
> 62.50   0
> 65.00   0
> 67.50   0
> 70.00   0
> 72.50   0
> 75.00   0
> 77.50   0
> 80.00   0
> 82.50   0
> 85.00   0
> 87.50   0
> 90.00   0
> 92.50   0
> 95.00   0
> 97.50   0
> 
> Spam distribution for all runs:
> * = 6 items
> 0.00   0
> 2.50   0
> 5.00   0
> 7.50   0
> 10.00   0
> 12.50   0
> 15.00   0
> 17.50   0
> 20.00   0
> 22.50   0
> 25.00   0
> 27.50   0
> 30.00   0
> 32.50   1 *
> 35.00   1 *
> 37.50   1 *
> 40.00   1 *
> 42.50   1 *
> 45.00   1 *
> 47.50   1 *
> 50.00  28 *****
> 52.50  64 ***********
> 55.00 184 *******************************
> 57.50 352 ***********************************************************
> 60.00 295 **************************************************
> 62.50  69 ************
> 65.00   1 *
> 67.50   0
> 70.00   0
> 72.50   0
> 75.00   0
> 77.50   0
> 80.00   0
> 82.50   0
> 85.00   0
> 87.50   0
> 90.00   0
> 92.50   0
> 95.00   0
> 97.50   0
> 
> I have to do other things now, but if anyone wants to play with what I
> *would* do if I could <wink>, play with max_discriminators and see whether
> reducing that helps spread this out.  A suspicion is that folding in endless
> quantities of garbage words (spamprob so close to 0.5 that they're not
> really clues at all) may be dragging everything toward 0.5 without giving a
> real benefit.
> 
> The math in Graham's combining scheme is such that a prob 0.5 word has no
> effect whatsoever on the outcome.  The math in Gary's combining scheme
> doesn't appear to have the same property:  I believe adding a .5 prob word
> there moves the outcome closer to neutrality.
> 


From tim.one@comcast.net  Sat Sep 21 12:16:37 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 21 Sep 2002 07:16:37 -0400
Subject: [Spambayes] Moving closer to Gary's ideal
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEKGBFAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCKELABFAB.tim.one@comcast.net>

Update.  My test data looks clean again.  Added 251 new spam, and purged a
ham that's been hiding in the spam forever.  Now using 20,000 ham and 14,000
spam.

There were two cut'n'paste typos in the code that affected results.  Fixing
those increased the separation between the ham and spam means over what I
reported last time.

The options used here are as reported last time:

"""
[Classifier]
use_robinson_probability: True
use_robinson_combining: True
max_discriminators: 1500

[TestDriver]
spam_cutoff: 0.50
"""

This implements everything we talked about, except for the disputed ranking
step.  All biases are gone, and no limits are placed on the probabilities.
For the probability adjustment step (FI1), I'm using a=1 and x=0.5 (I can
play with this, but doubt they're the most useful things to poke at; and we
moved to using 0.5 for the "unknown word probability" under Graham's scheme
long ago).

Here's a before-and-after 10-fold cross validation run, where before is the
default (our highly tweaked Graham scheme).  Each run trained on 18000 hams
& 12600 spams, then predicted 2000 disjoint hams & 1400 disjoint spams.  As
will be clear soon, the results aren't as bizarre as they look:

false positive percentages
    0.000  0.250  lost  +(was 0)
    0.000  0.200  lost  +(was 0)
    0.000  0.250  lost  +(was 0)
    0.000  0.100  lost  +(was 0)
    0.050  0.450  lost  +800.00%
    0.000  0.250  lost  +(was 0)
    0.000  0.150  lost  +(was 0)
    0.050  0.300  lost  +500.00%
    0.000  0.200  lost  +(was 0)
    0.100  0.350  lost  +250.00%

won   0 times
tied  0 times
lost 10 times

total unique fp went from 4 to 50 lost  +1150.00%
mean fp % went from 0.02 to 0.25 lost  +1150.00%

false negative percentages
    0.214  0.000  won   -100.00%
    0.286  0.000  won   -100.00%
    0.000  0.000  tied
    0.143  0.000  won   -100.00%
    0.143  0.000  won   -100.00%
    0.286  0.000  won   -100.00%
    0.143  0.071  won    -50.35%
    0.143  0.000  won   -100.00%
    0.286  0.000  won   -100.00%
    0.071  0.000  won   -100.00%

won   9 times
tied  1 times
lost  0 times

total unique fn went from 24 to 1 won    -95.83%
mean fn % went from 0.171428571428 to 0.00714285714286 won    -95.83%

So this test was a disaster for the false positive rate and a huge win for
the false negative rate.  This is because 0.50 is too low a cutoff now:

Ham distribution for all runs:
* = 86 items
  0.00    0
  2.50    0
  5.00    0
  7.50    0
 10.00    0
 12.50    0
 15.00    0
 17.50    0
 20.00   26 *
 22.50  155 **
 25.00  627 ********
 27.50 1859 **********************
 30.00 3780 ********************************************
 32.50 5108 ************************************************************
 35.00 4264 **************************************************
 37.50 2450 *****************************
 40.00 1056 *************
 42.50  395 *****
 45.00  178 ***
 47.50   52 *
 50.00   30 *
 52.50   13 *
 55.00    4 *
 57.50    1 *
 60.00    1 *
 62.50    1 *
 65.00    0
 67.50    0
 70.00    0
 72.50    0
 75.00    0
 77.50    0
 80.00    0
 82.50    0
 85.00    0
 87.50    0
 90.00    0
 92.50    0
 95.00    0
 97.50    0

Spam distribution for all runs:
* = 50 items
  0.00    0
  2.50    0
  5.00    0
  7.50    0
 10.00    0
 12.50    0
 15.00    0
 17.50    0
 20.00    0
 22.50    0
 25.00    0
 27.50    0
 30.00    0
 32.50    0
 35.00    0
 37.50    0
 40.00    0
 42.50    1 *
 45.00    0
 47.50    0
 50.00    3 *
 52.50    6 *
 55.00   17 *
 57.50   40 *
 60.00   76 **
 62.50  171 ****
 65.00  394 ********
 67.50  710 ***************
 70.00 1247 *************************
 72.50 2358 ************************************************
 75.00 2986 ************************************************************
 77.50 2659 ******************************************************
 80.00 1957 ****************************************
 82.50 1069 **********************
 85.00  192 ****
 87.50   31 *
 90.00   61 **
 92.50   22 *
 95.00    0
 97.50    0

So if the cutoff were boosted to 0.575, we'd lose 30+13+4 = 47 fp, and gain
3+6+17 = 26 fn, for a grand total of 3 fp and 27 fn.  That would leave it
essentially indistinguishable from the "before" run, but gets there without
artificial biases and limits.  Good show!  Against it, I have no idea how to
*predict* where to put the cutoff, and that simply wasn't an issue before
(0.90 "just worked", and on this particular large test only 5 of the 20,000
ham scored above 0.10, while only 24 of the 14,000 spam scored below 0.90).

The highest-scoring ham was again the fellow who added a one-line comment to
a quote of an entire Nigerian scam msg.  The second-highest was again the
lady looking for a Python course in the UK, damned by her employer's much
longer obnoxious sig.  These are all familiar.  Something we haven't seen
for a few weeks is the systematic reappearance of conference announcements
among the high-scoring ham; under the current scheme, tokenization
preserving case also hates those, and so does tokenization via word bigrams,
and ditto via character 5-grams; the tokenization we're using now
(case-folded unigrams but preserving punctuation) didn't hate them under the
Graham scheme (indeed, that's the only tokenization scheme I've tried that
*doesn't* hate them; they appear to benefit from Graham's low
max_discriminators because the really spammish "visit our website for more
information!" stuff doesn't show up until near the end, and by then enough
0.01 clues have been seen that the end doesn't matter).

The lowest-scoring spam is embarrassing:  it has

    Subject: HOW TO BECOME A MILLIONAIRE IN WEEKS!!

and the body consists solely of a uuencoded text file, which we don't
decipher.  That's the only spam to score below 0.5.  The tokenizer doesn't
give the inferencer much to go on here.  You'd think, e.g., that MILLIONAIRE
in the subject line is a spam clue; and it is, but it's *so* blatant that
few spam actually do that, leaving

    prob('subject:MILLIONAIRE') = 0.666667

Two other words in the subject were actually stronger clues:

    prob('subject:WEEKS') = 0.833333
    prob('subject:HOW') = 0.867101


Everyone who tests this (please do!  it looks very promising, although my
data only supports that it's not a regression -- I *expect* it will do
better for some of you), pay attention to your score histograms and figure
out the best value for spam_cutoff from them.  That would be a good number
to report.  I'd also appreciate it if you played with max_discriminators
here, and/or with some other gimmick aiming to keep nearly-neutral words out
of the scoring; if, e.g., the presence of "the" (a canonical <wink> spamprob
0.5 word) moves a score closer to 0.5, that's really not helping things (as
far as I can see).  Note that if you fiddle with both, they're most likely
not independent, so be sure to keep looking at the histograms (they reveal a
hell of a lot more than the raw error rates do).


From guido@python.org  Sat Sep 21 12:52:13 2002
From: guido@python.org (Guido van Rossum)
Date: Sat, 21 Sep 2002 07:52:13 -0400
Subject: [Spambayes] Re: Move closer to Gary's ideal
In-Reply-To: Your message of "Sat, 21 Sep 2002 07:05:53 EDT."
             <B9B1CAD1.16427%grobinson@transpose.com> 
References: <B9B1CAD1.16427%grobinson@transpose.com> 
Message-ID: <200209211152.g8LBqDo26013@pcp02138704pcs.reston01.va.comcast.net>

> So... if we're doing well at classification this way, we will still
> be able to do spread-based things like using sliders for cutoffs, or
> putting some emails into a middle ground "you should look at this
> manually" group.

The "manual review" group only makes sense in a system that actually
*throws away* spam rather than simply labeling it.  We run with this
setup at python.org, and the sysadmins who scan the 30 or so middle
ground message per day for false positives are my heros.

But this is a rather unrealistic deployment scheme -- the fear of
false positives will make almost everybody else use a setup where the
system only *labels* spam, and the end user's email program sorts
those into spam and ham folders based upon the labeling.  (In fact it
was argued here that evern allowing an *option* to throw away spam
would make us liable -- though that was before Tim's fenomonal
results.)

When all you do is labeling, I see no use for a middle category --
it's easier to merge that into the ham, which the end user will look
at anyway.

I guess what I'm saying is that producing a less bipolar distribution
lets us have a more usable control to weigh the importance of
f.p. against f.n. -- nothing more, nothing less.  And that's
important, so that individual users can decide how big exactly their
fear of f.p.'s is, relative to their hatred for spam.

I guess this would require an X-Spam-Probability: <float> header,
rather than a Yes/No header, either in addition or instead of the
Yes/No header.  If you also want to provide a Yes/No header, you'd
need a database of user preferences plus a UI for it.  But only a
numeric value is probably hard for the average mail program to filter
on...

--Guido van Rossum (home page: http://www.python.org/~guido/)

From grobinson@transpose.com  Sat Sep 21 13:54:25 2002
From: grobinson@transpose.com (Gary Robinson)
Date: Sat, 21 Sep 2002 08:54:25 -0400
Subject: [Spambayes] Moving closer to Gary's ideal
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKELABFAB.tim.one@comcast.net>
Message-ID: <B9B1E441.16436%grobinson@transpose.com>

>  I'd also appreciate it if you played with max_discriminators
> here, and/or with some other gimmick aiming to keep nearly-neutral words out
> of the scoring; if, e.g., the presence of "the" (a canonical <wink> spamprob
> 0.5 word) moves a score closer to 0.5, that's really not helping things (as
> far as I can see).

Keeping the '5's is just being REALLY purist. If everything else is right,
it shouldn't hurt.  It just falls out of really trying to be pure so that we
can do this in a mathematical way instead of a way of just constantly
finding little patches to little problems. In my experience being
"relentlessly" mathematical, as opposed to just continually finding little
"kludgy" solutions to each little aspect, generally pays off... that path
forces you to really get the basics EXACTLY right, and then various theorems
can be realistically invoked to maximize performance. OTOH sometimes the
data is just so messy to begin with that it just isn't possible to get
there.

IN ANY CASE, I think it would be pragmatic at this point to have more
experiments with small corpuses with FI1 (f(w)) and some kind of cutoff so
that only extreme values are processed... experimentally, you have accrued a
lot of evidence that, that does help under the current overall processing.

f(w) doesn't rely on esoteric theorems to be useful, as g(w) does. f(w) just
MAKES SENSE. It should work even if everything else isn't totally pure.

I would run MAX_DISCRIMINATOR testing with respect to f(w), not (p(w)), cuz
f(w) is more meaningful. Tim has tried both the 150 most extreme and the 15
most extreme... it would be interesting to see what happened with both of
those measures using abs(f(w)) as the indicator of extremeness -- maybe
that's what Tim is currently doing, I'm not sure now.

==>I would also experiment with the size of a. When a gets small (compared
to 1) it will make a very different  choice of words end up at the extremes.
a is really something that could meaningfully be tuned for performance.<==

---

For those of you who may be interested in my "purist" perspective (others
can stop reading now! ;)   ):

I want to be able to invoke a particular theorem via g(w). I have done so
before very effectively, but it may not be possible here.

I can give ideas to test to bring things further in that direction, but it
all depends on how far you guys want to go.

I would love to be able to offload experimental work with regard to that
idea from you guys and do it myself, but I can't now. I'm CEO of a company
that is in a critical period and just have no free time. I may have a bit
more free time beginning mid-october, but I'm not sure.

>From the purist perspective, I think it would be good to look at the case of
these conferences that are now being classified as spam, and see if we can
see some very fundamental reason why they are now coming out as spams. I
have some ideas which I will think about...

For instance, it is not really right that the original p's are computed by
counting ALL the occurrences of the words, even multiple occurrences in the
same email, but then we are NOT doing classification with that data but
rather just looking at presence vs. absence of a word. Until we are very
consistent about all that stuff, we may not have an underlying distribution
that enables theorems that rely on particular distributions to be invoked,
because everything just may be too off-kilter.

Also there may be other problems that have to be addressed in order to
really have a known distribution under a null hypothesis. I thought of some
possible ones last night and will continue thinking about them. Some of
these problems may be intrinsic to the data we're dealing with -- there may
be no way to get around them or we may have to do something clever to get
around them.

When I get clearer on the problems and possible solutions, I'll say so... or
if there's anyone here who wants to work with me on that perspective, that
would be great... just let me know.

But in the meantime, I think you guys should play with f(w) and tune a and
MAX_DISCRIMINATOR and see if you can optimise things for better performance
than you had before, and stay away from further work on g(w). That's
pragmatic.

--Gary

-- 
Gary Robinson
CEO
Transpose, LLC
grobinson@transpose.com
207-942-3463
http://www.emergentmusic.com
http://radio.weblogs.com/0101454


> From: Tim Peters <tim.one@comcast.net>
> Date: Sat, 21 Sep 2002 07:16:37 -0400
> To: Gary Robinson <grobinson@transpose.com>, SpamBayes <spambayes@python.org>
> Cc: glouis@dynamicro.on.ca
> Subject: RE: [Spambayes] Moving closer to Gary's ideal
> 
> Update.  My test data looks clean again.  Added 251 new spam, and purged a
> ham that's been hiding in the spam forever.  Now using 20,000 ham and 14,000
> spam.
> 
> There were two cut'n'paste typos in the code that affected results.  Fixing
> those increased the separation between the ham and spam means over what I
> reported last time.
> 
> The options used here are as reported last time:
> 
> """
> [Classifier]
> use_robinson_probability: True
> use_robinson_combining: True
> max_discriminators: 1500
> 
> [TestDriver]
> spam_cutoff: 0.50
> """
> 
> This implements everything we talked about, except for the disputed ranking
> step.  All biases are gone, and no limits are placed on the probabilities.
> For the probability adjustment step (FI1), I'm using a=1 and x=0.5 (I can
> play with this, but doubt they're the most useful things to poke at; and we
> moved to using 0.5 for the "unknown word probability" under Graham's scheme
> long ago).
> 
> Here's a before-and-after 10-fold cross validation run, where before is the
> default (our highly tweaked Graham scheme).  Each run trained on 18000 hams
> & 12600 spams, then predicted 2000 disjoint hams & 1400 disjoint spams.  As
> will be clear soon, the results aren't as bizarre as they look:
> 
> false positive percentages
>   0.000  0.250  lost  +(was 0)
>   0.000  0.200  lost  +(was 0)
>   0.000  0.250  lost  +(was 0)
>   0.000  0.100  lost  +(was 0)
>   0.050  0.450  lost  +800.00%
>   0.000  0.250  lost  +(was 0)
>   0.000  0.150  lost  +(was 0)
>   0.050  0.300  lost  +500.00%
>   0.000  0.200  lost  +(was 0)
>   0.100  0.350  lost  +250.00%
> 
> won   0 times
> tied  0 times
> lost 10 times
> 
> total unique fp went from 4 to 50 lost  +1150.00%
> mean fp % went from 0.02 to 0.25 lost  +1150.00%
> 
> false negative percentages
>   0.214  0.000  won   -100.00%
>   0.286  0.000  won   -100.00%
>   0.000  0.000  tied
>   0.143  0.000  won   -100.00%
>   0.143  0.000  won   -100.00%
>   0.286  0.000  won   -100.00%
>   0.143  0.071  won    -50.35%
>   0.143  0.000  won   -100.00%
>   0.286  0.000  won   -100.00%
>   0.071  0.000  won   -100.00%
> 
> won   9 times
> tied  1 times
> lost  0 times
> 
> total unique fn went from 24 to 1 won    -95.83%
> mean fn % went from 0.171428571428 to 0.00714285714286 won    -95.83%
> 
> So this test was a disaster for the false positive rate and a huge win for
> the false negative rate.  This is because 0.50 is too low a cutoff now:
> 
> Ham distribution for all runs:
> * = 86 items
> 0.00    0
> 2.50    0
> 5.00    0
> 7.50    0
> 10.00    0
> 12.50    0
> 15.00    0
> 17.50    0
> 20.00   26 *
> 22.50  155 **
> 25.00  627 ********
> 27.50 1859 **********************
> 30.00 3780 ********************************************
> 32.50 5108 ************************************************************
> 35.00 4264 **************************************************
> 37.50 2450 *****************************
> 40.00 1056 *************
> 42.50  395 *****
> 45.00  178 ***
> 47.50   52 *
> 50.00   30 *
> 52.50   13 *
> 55.00    4 *
> 57.50    1 *
> 60.00    1 *
> 62.50    1 *
> 65.00    0
> 67.50    0
> 70.00    0
> 72.50    0
> 75.00    0
> 77.50    0
> 80.00    0
> 82.50    0
> 85.00    0
> 87.50    0
> 90.00    0
> 92.50    0
> 95.00    0
> 97.50    0
> 
> Spam distribution for all runs:
> * = 50 items
> 0.00    0
> 2.50    0
> 5.00    0
> 7.50    0
> 10.00    0
> 12.50    0
> 15.00    0
> 17.50    0
> 20.00    0
> 22.50    0
> 25.00    0
> 27.50    0
> 30.00    0
> 32.50    0
> 35.00    0
> 37.50    0
> 40.00    0
> 42.50    1 *
> 45.00    0
> 47.50    0
> 50.00    3 *
> 52.50    6 *
> 55.00   17 *
> 57.50   40 *
> 60.00   76 **
> 62.50  171 ****
> 65.00  394 ********
> 67.50  710 ***************
> 70.00 1247 *************************
> 72.50 2358 ************************************************
> 75.00 2986 ************************************************************
> 77.50 2659 ******************************************************
> 80.00 1957 ****************************************
> 82.50 1069 **********************
> 85.00  192 ****
> 87.50   31 *
> 90.00   61 **
> 92.50   22 *
> 95.00    0
> 97.50    0
> 
> So if the cutoff were boosted to 0.575, we'd lose 30+13+4 = 47 fp, and gain
> 3+6+17 = 26 fn, for a grand total of 3 fp and 27 fn.  That would leave it
> essentially indistinguishable from the "before" run, but gets there without
> artificial biases and limits.  Good show!  Against it, I have no idea how to
> *predict* where to put the cutoff, and that simply wasn't an issue before
> (0.90 "just worked", and on this particular large test only 5 of the 20,000
> ham scored above 0.10, while only 24 of the 14,000 spam scored below 0.90).
> 
> The highest-scoring ham was again the fellow who added a one-line comment to
> a quote of an entire Nigerian scam msg.  The second-highest was again the
> lady looking for a Python course in the UK, damned by her employer's much
> longer obnoxious sig.  These are all familiar.  Something we haven't seen
> for a few weeks is the systematic reappearance of conference announcements
> among the high-scoring ham; under the current scheme, tokenization
> preserving case also hates those, and so does tokenization via word bigrams,
> and ditto via character 5-grams; the tokenization we're using now
> (case-folded unigrams but preserving punctuation) didn't hate them under the
> Graham scheme (indeed, that's the only tokenization scheme I've tried that
> *doesn't* hate them; they appear to benefit from Graham's low
> max_discriminators because the really spammish "visit our website for more
> information!" stuff doesn't show up until near the end, and by then enough
> 0.01 clues have been seen that the end doesn't matter).
> 
> The lowest-scoring spam is embarrassing:  it has
> 
>   Subject: HOW TO BECOME A MILLIONAIRE IN WEEKS!!
> 
> and the body consists solely of a uuencoded text file, which we don't
> decipher.  That's the only spam to score below 0.5.  The tokenizer doesn't
> give the inferencer much to go on here.  You'd think, e.g., that MILLIONAIRE
> in the subject line is a spam clue; and it is, but it's *so* blatant that
> few spam actually do that, leaving
> 
>   prob('subject:MILLIONAIRE') = 0.666667
> 
> Two other words in the subject were actually stronger clues:
> 
>   prob('subject:WEEKS') = 0.833333
>   prob('subject:HOW') = 0.867101
> 
> 
> Everyone who tests this (please do!  it looks very promising, although my
> data only supports that it's not a regression -- I *expect* it will do
> better for some of you), pay attention to your score histograms and figure
> out the best value for spam_cutoff from them.  That would be a good number
> to report.  I'd also appreciate it if you played with max_discriminators
> here, and/or with some other gimmick aiming to keep nearly-neutral words out
> of the scoring; if, e.g., the presence of "the" (a canonical <wink> spamprob
> 0.5 word) moves a score closer to 0.5, that's really not helping things (as
> far as I can see).  Note that if you fiddle with both, they're most likely
> not independent, so be sure to keep looking at the histograms (they reveal a
> hell of a lot more than the raw error rates do).
> 


From skip@pobox.com  Sat Sep 21 15:28:00 2002
From: skip@pobox.com (Skip Montanaro)
Date: Sat, 21 Sep 2002 09:28:00 -0500
Subject: [Spambayes] idea for tokenizer.crack_filename change
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEFLBFAB.tim.one@comcast.net>
References: <15754.19106.478876.417637@12-248-11-90.client.attbi.com>
        <LNBBLJKPBEHFEDALKOLCGEFLBFAB.tim.one@comcast.net>
Message-ID: <15756.33264.517811.330133@12-248-11-90.client.attbi.com>


    >> It seems to me that base64-encoded, all DOS/Windows executables start
    >> with (reciting from memory, since I've deleted all viruses and
    >> haven't received any new ones in the last 15 minutes or so) "TPqAAA"
    >> or something similar.  Why rely on finding specific file extensions?
    >> They can just change.

    Tim> Well, not often, and the scheme we're working on is supposed to be able to
    Tim> learn when they do <wink>.  Would you like to write some code to tokenize
    Tim> this particular bit of Windows Lore?

I gave it a try, but I'm still suffering with fp/fn rates around 15%, so
anything I see is suspect.  Also, I saw no change.  It's quite possible I
have a bug, but I've also cleaned out obvious viruses from my corpora.  True
spam may have enough indicators elsewhere that this scheme won't help.

Should I just go ahead and checkin my change (it is controlled by a couple
new options, and by default is not enabled) and let y'all point out my bugs?

Skip


From skip@pobox.com  Sat Sep 21 15:48:15 2002
From: skip@pobox.com (Skip Montanaro)
Date: Sat, 21 Sep 2002 09:48:15 -0500
Subject: [Spambayes] Re: Move closer to Gary's ideal
In-Reply-To: <200209211152.g8LBqDo26013@pcp02138704pcs.reston01.va.comcast.net>
References: <B9B1CAD1.16427%grobinson@transpose.com>
        <200209211152.g8LBqDo26013@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <15756.34479.643417.812741@12-248-11-90.client.attbi.com>


    Guido> I guess this would require an X-Spam-Probability: <float> header,
    Guido> rather than a Yes/No header, either in addition or instead of the
    Guido> Yes/No header.  If you also want to provide a Yes/No header,
    Guido> you'd need a database of user preferences plus a UI for it.  But
    Guido> only a numeric value is probably hard for the average mail
    Guido> program to filter on...

Yes, which is why SpamAssassin reports its findings several different ways.
Just to review for people who aren't familiar with it, here are the SA
headers added to a recent spam I received.  (The last two headers are not
score-related.  The X-Spam-Prev-Content-Type: header is used to defang
messages with complex MIME types.  They are delivered as text/plain.)

    X-Spam-Status: Yes, hits=16.4 required=5.0
            tests=FROM_NAME_NO_SPACES,NO_REAL_NAME,FRONTPAGE,BIG_FONT,
                  MIME_EXCESSIVE_QP,CHARSET_FARAWAY_HEADERS,NO_MX_FOR_FROM,
                  MSG_ID_ADDED_BY_MTA_3,MISSING_HEADERS
            version=2.31
    X-Spam-Flag: YES
    X-Spam-Level: ****************
    X-Spam-Checker-Version: SpamAssassin 2.31 (devel $Id: SpamAssassin.pm,v 1.94.2.2 2002/06/20 17:20:29 hughescr Exp $)
    X-Spam-Prev-Content-Type: multipart/related; boundary="a7e59999-cd9f-11d6-8d6b-00e04c6236e7"

Each "*" counts for 1, so since the computed value was 16.4, the
X-Spam-Level: header contains 16 stars.  That and the X-Spam-Flag: header or
the beginning of the X-Spam-Status: header are much easier to match on in
filters than numeric values and allow users to provide their own "middle
ground" if they want, e.g. in procmail, something like this:

    :0
    * ^X-Spam-Level: \*\*\*\*\*\*\*\*\*\*
    definitelyspam

    :0
    * ^X-Spam-Level: \*\*\*\*\*
    probablyspam

    :0
    * ^X-Spam-Level: \*\*\*
    probablynotspam

    :0
    * ^X-Spam-Level:
    goodstuff

You should be able to do the same in Outhouse or other mailers.

What this all boils down to is that assuming spambayes/hammie gets beyond
the "extremely cool experiment in algorithm design" stage, multiple ways to
eevaluate its results need to be provided to make it useful to as wide a
range of users as possible.

Which leads to another experiment.  You could scale its probabilities to
match the default scale used by SpamAssassin (>= 5.0 is considered spam) and
compare results from the two.

Skip


From skip@pobox.com  Sat Sep 21 16:11:45 2002
From: skip@pobox.com (Skip Montanaro)
Date: Sat, 21 Sep 2002 10:11:45 -0500
Subject: [Spambayes] RE: spam detection via probability - actual results!
In-Reply-To: <LNBBLJKPBEHFEDALKOLCMEKABFAB.tim.one@comcast.net>
References: <200209200929.g8K9TJo11972@indus.ins.cwi.nl>
        <LNBBLJKPBEHFEDALKOLCMEKABFAB.tim.one@comcast.net>
Message-ID: <15756.35889.250276.677544@12-248-11-90.client.attbi.com>


    Tim> FYI, I'm stalled -- I screwed up my test data by mistakenly mixing
    Tim> the ham reservoir at random into the spam sets (I intended to mix
    Tim> in from the spam reservoir, of course).  This is proving a bitch to
    Tim> untangle again.

I was thinking about this yesterday when I ran rebal because to switch from
rebalancing ham to rebalancing spam I hit Ctl-P then had to edit two places
in the line.  A bit late for Tim and Anthony, but perhaps the change I just
checked in will help others avoid this problem.

Skip

From grobinson@transpose.com  Sat Sep 21 16:28:58 2002
From: grobinson@transpose.com (Gary Robinson)
Date: Sat, 21 Sep 2002 11:28:58 -0400
Subject: [Spambayes] Practical applications
Message-ID: <B9B2087A.1644E%grobinson@transpose.com>

A couple of questions with regard to practical applications...

1) I notice that Python is an "Actice Scripting" language? Does that a mean
that a Python script can interact with Outlook to do things like add a
"Spam!" button, etc.?

2) I'm not up on the Python Software Foundation License. Does it allow a
company to make a proprietary product that incorporates spambayes code or
calls it as if from a library?


--Gary


-- 
Gary Robinson
CEO
Transpose, LLC
grobinson@transpose.com
207-942-3463
http://www.emergentmusic.com
http://radio.weblogs.com/0101454



From paul-python@svensson.org  Sat Sep 21 13:25:55 2002
From: paul-python@svensson.org (Paul Svensson)
Date: Sat, 21 Sep 2002 08:25:55 -0400 (EDT)
Subject: [Spambayes] Moving closer to Gary's ideal
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKELABFAB.tim.one@comcast.net>
Message-ID: <Pine.LNX.4.44.0209210823320.402-100000@familjen.svensson.org>

On Sat, 21 Sep 2002, Tim Peters wrote:

>Everyone who tests this (please do!  it looks very promising, although my
>data only supports that it's not a regression -- I *expect* it will do
>better for some of you), pay attention to your score histograms and figure
>out the best value for spam_cutoff from them.  That would be a good number
>to report.  I'd also appreciate it if you played with max_discriminators
>here, and/or with some other gimmick aiming to keep nearly-neutral words out
>of the scoring; if, e.g., the presence of "the" (a canonical <wink> spamprob
>0.5 word) moves a score closer to 0.5, that's really not helping things (as
>far as I can see).  Note that if you fiddle with both, they're most likely
>not independent, so be sure to keep looking at the histograms (they reveal a
>hell of a lot more than the raw error rates do).

Just a hunch:
For the 'indifferent' words to make no difference,
shouldn't x be set the same as the cutoff ?

	/Paul


From gward@python.net  Sat Sep 21 17:11:49 2002
From: gward@python.net (Greg Ward)
Date: Sat, 21 Sep 2002 12:11:49 -0400
Subject: [Spambayes] Re: Move closer to Gary's ideal
In-Reply-To: <15756.34479.643417.812741@12-248-11-90.client.attbi.com>
References: <B9B1CAD1.16427%grobinson@transpose.com>
	<200209211152.g8LBqDo26013@pcp02138704pcs.reston01.va.comcast.net>
	<15756.34479.643417.812741@12-248-11-90.client.attbi.com>
Message-ID: <20020921161149.GA26235@cthulhu.gerg.ca>

On 21 September 2002, Skip Montanaro said:
> Which leads to another experiment.  You could scale its probabilities to
> match the default scale used by SpamAssassin (>= 5.0 is considered spam) and
> compare results from the two.

I'm dubious about that -- SA's score is theoretically open-ended, while
spambayes always computes something between 0 and 1 (which is why we
like to think of it as a probability.)  (In reality, SA tops out around
50 -- I think the highest scoring spam I've seen was in the low 40s;
ISTR someone on one of the SA lists bragging about seeing one that
cracked 50.)

I guess you could play games with graphing the prob. distributions of SA
scores for a given corpus, and then come up with some correspondence
between the two scoring functions.  X-Spam-Level is *really* handy, and
not just for people using simplistic filtering schemes at the MDA
level.  Eg. I was just using it to clean up the python.org corpus
gathered last week -- using mutt to limit my view to messages matching
  ~h x-spam-level:\ [*]{4}$
showed me the "4-star" messages only.  Repeat with {3}, {2}, {1} to see
progressively less spammy subsets of the folder.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
All things are possible -- except skiing through a revolving door.

From tim.one@comcast.net  Sat Sep 21 17:55:05 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 21 Sep 2002 12:55:05 -0400
Subject: [Spambayes] Practical applications
In-Reply-To: <B9B2087A.1644E%grobinson@transpose.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCGELJBFAB.tim.one@comcast.net>

[Gary Robinson]
> A couple of questions with regard to practical applications...
>
> 1) I notice that Python is an "Actice Scripting" language? Does
> that a mean that a Python script can interact with Outlook to do
> things like add a "Spam!" button, etc.?

I'm afraid that's more a question about Outlook -- if you could do what
you're after using Visual Basic, or C++, or ... you could do it using
Python, and provided MS isn't hiding the necessary part(s) of the Outlook
API.  Trying to make sense of MS's object and programming models is always a
challenge, and the preferred way changed Yet Again for Outlook 2002:

    http://support.microsoft.com/default.aspx?scid=KB;EN-US;Q291163&


> 2) I'm not up on the Python Software Foundation License. Does it allow a
> company to make a proprietary product that incorporates spambayes code or
> calls it as if from a library?

Yes, to both.  The license

    grants Licensee a nonexclusive, royalty-free, world-wide
    license to reproduce, analyze, test, perform and/or display
    publicly, prepare derivative works, distribute, and otherwise
    use ... alone or in any derivative version

with some mild requirements for preserving the PSF copyright and license on
the PSF-licensed portions of your product.  It's not a "copyleft" kind of
license.  The full text of the Python license is here:

     http://www.python.org/2.2.1/license.html

I need to check a suitably simplified version of that into the spambayes
project.


From nas@python.ca  Sat Sep 21 18:23:28 2002
From: nas@python.ca (Neil Schemenauer)
Date: Sat, 21 Sep 2002 10:23:28 -0700
Subject: [Spambayes] Moving closer to Gary's ideal
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKELABFAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCKEKGBFAB.tim.one@comcast.net>
	<LNBBLJKPBEHFEDALKOLCKELABFAB.tim.one@comcast.net>
Message-ID: <20020921172327.GA1650@glacier.arctrix.com>

Tim Peters wrote:
> Everyone who tests this (please do!  it looks very promising, although my
> data only supports that it's not a regression -- I *expect* it will do
> better for some of you), pay attention to your score histograms and figure
> out the best value for spam_cutoff from them.

Here's my distributions:

    Ham distribution for all runs:
    * = 5 items
     17.50   0
     20.00   3 *
     22.50  12 ***
     25.00  57 ************
     27.50 127 **************************
     30.00 209 ******************************************
     32.50 270 ******************************************************
     35.00 292 ***********************************************************
     37.50 266 ******************************************************
     40.00 213 *******************************************
     42.50 130 **************************
     45.00  85 *****************
     47.50  52 ***********
     50.00  49 **********
     52.50  22 *****
     55.00  10 **
     57.50   3 *
     60.00   0

    Spam distribution for all runs:
    * = 5 items
     45.00   0
     47.50   1 *
     50.00   4 *
     52.50  10 **
     55.00  30 ******
     57.50  64 *************
     60.00  88 ******************
     62.50 151 *******************************
     65.00 192 ***************************************
     67.50 269 ******************************************************
     70.00 256 ****************************************************
     72.50 215 *******************************************
     75.00 164 *********************************
     77.50 115 ***********************
     80.00 107 **********************
     82.50  73 ***************
     85.00  42 *********
     87.50  18 ****
     90.00   1 *
     92.50   0

With "spam_cutoff: 0.56":

    false positive percentages
        0.667  0.667  tied
        0.000  0.333  lost  +(was 0)
        1.000  0.667  won    -33.30%
        0.333  0.667  lost  +100.30%
        0.000  0.333  lost  +(was 0)
        0.000  0.000  tied

    won   1 times
    tied  2 times
    lost  3 times

    total unique fp went from 6 to 8 lost   +33.33%
    mean fp % went from 0.333333333333 to 0.444444444444 lost   +33.33%

    false negative percentages
        0.333  1.333  lost  +300.30%
        1.333  1.667  lost   +25.06%
        1.667  1.667  tied
        0.333  0.000  won   -100.00%
        1.333  2.000  lost   +50.04%
        1.667  1.667  tied

    won   1 times
    tied  2 times
    lost  3 times

    total unique fn went from 20 to 25 lost   +25.00%
    mean fn % went from 1.11111111111 to 1.38888888889 lost   +25.00%

Reducing max_discriminators seems to make things worse.

  Neil

From crunch@shopip.com  Sat Sep 21 16:58:06 2002
From: crunch@shopip.com (John Draper)
Date: Sat, 21 Sep 2002 08:58:06 -0700
Subject: [Spambayes] Make your own spam
Message-ID: <v0311076eb9b2443c2819@[192.168.0.2]>

I believe a useful exercise would be to try to craft email messages which=
 would fool the classifier. Perhaps the goal should be to try to fabricate=
 (with minimal classifier feedback) messages which score 0.5 likelyhood.=
 Create a message (a 'upam' or 'shpam', for unknown ham/spam) and then guess=
 what it's S/H probability is beforehand, then submit it and see how close=
 your estimate was. I believe these exercises will suggest weaknesses/fortif=
ication methods, or at minimum give a better feel for the mechanism.



From tim.one@comcast.net  Sat Sep 21 18:45:05 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 21 Sep 2002 13:45:05 -0400
Subject: [Spambayes] Moving closer to Gary's ideal
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKELABFAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCOELKBFAB.tim.one@comcast.net>

[Tim]
> I'd also appreciate it if you played with max_discriminators
> here, and/or with some other gimmick aiming to keep nearly-neutral
> words out of the scoring; ...

Here's a 2-line patch that apparently pays for me:

"""
Index: classifier.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/classifier.py,v
retrieving revision 1.16
diff -c -r1.16 classifier.py
*** classifier.py       21 Sep 2002 03:43:13 -0000      1.16
--- classifier.py       21 Sep 2002 17:18:54 -0000
***************
*** 488,493 ****
--- 488,495 ----
                  prob = record.spamprob

              distance = abs(prob - 0.5)
+             if distance < 0.1:
+                 continue
              if distance > smallest_best:
                  heapreplace(nbest, (distance, prob, word, record))
                  smallest_best = nbest[0][0]
"""

IOW, ignore words with spamprobs in (0.4, 0.6).

Using (same as last time, except boosting spam_cutoff to 0.575):

"""
[Classifier]
use_robinson_probability: True
use_robinson_combining: True
max_discriminators: 1500

[TestDriver]
spam_cutoff: 0.575
"""

is a nearly pure win for the fn rate on my 34,000 message test, and gives a
random bump to the fp rate:

"""
false positive percentages
    0.000  0.050  lost  +(was 0)
    0.000  0.000  tied
    0.000  0.050  lost  +(was 0)
    0.000  0.000  tied
    0.050  0.050  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.050  0.050  tied
    0.000  0.000  tied
    0.100  0.050  won    -50.00%

won   1 times
tied  7 times
lost  2 times

total unique fp went from 4 to 5 lost   +25.00%
mean fp % went from 0.02 to 0.025 lost   +25.00%

false negative percentages
    0.214  0.000  won   -100.00%
    0.286  0.071  won    -75.17%
    0.000  0.000  tied
    0.143  0.143  tied
    0.143  0.143  tied
    0.286  0.143  won    -50.00%
    0.143  0.143  tied
    0.143  0.214  lost   +49.65%
    0.286  0.143  won    -50.00%
    0.071  0.000  won   -100.00%

won   5 times
tied  4 times
lost  1 times

total unique fn went from 24 to 14 won    -41.67%
mean fn % went from 0.171428571428 to 0.1 won    -41.67%
"""

The spam and ham histograms are more clearly separated this way, with the
ham mean going down and the spam mean going up:

Ham distribution for all runs:
* = 63 items
  0.00    0
  2.50    0
  5.00    0
  7.50    6 *
 10.00    6 *
 12.50   49 *
 15.00  140 ***
 17.50  337 ******
 20.00  773 *************
 22.50 1648 ***************************
 25.00 2684 *******************************************
 27.50 3738 ************************************************************
 30.00 3729 ************************************************************
 32.50 3043 *************************************************
 35.00 1930 *******************************
 37.50  998 ****************
 40.00  483 ********
 42.50  229 ****
 45.00  121 **
 47.50   37 *
 50.00   29 *
 52.50    9 *
 55.00    6 *
 57.50    3 *
 60.00    0
 62.50    1 *
 65.00    0
 67.50    1 *
 70.00    0
 72.50    0
 75.00    0
 77.50    0
 80.00    0
 82.50    0
 85.00    0
 87.50    0
 90.00    0
 92.50    0
 95.00    0
 97.50    0

Spam distribution for all runs:
* = 48 items
  0.00    0
  2.50    0
  5.00    0
  7.50    0
 10.00    0
 12.50    0
 15.00    0
 17.50    0
 20.00    0
 22.50    0
 25.00    0
 27.50    0
 30.00    0
 32.50    0
 35.00    0
 37.50    0
 40.00    1 *
 42.50    0
 45.00    0
 47.50    0
 50.00    2 *
 52.50    1 *
 55.00   10 *
 57.50    6 *
 60.00   20 *
 62.50   42 *
 65.00   87 **
 67.50  191 ****
 70.00  356 ********
 72.50  651 **************
 75.00 1058 ***********************
 77.50 1870 ***************************************
 80.00 2753 **********************************************************
 82.50 2825 ***********************************************************
 85.00 2078 ********************************************
 87.50 1576 *********************************
 90.00  332 *******
 92.50  127 ***
 95.00    0
 97.50   14 *

Note that if I had set spam_cutoff to 0.60, we would have lost 3(!) fp at
the cost of gaining 6 fn.  Note too that there's a useful "middle ground"
here that doesn't require 6-digit precision to express <wink>.

Here's "a new" fp (this is one of the about 10 that pop in and out of the fp
set routinely that I don't care about -- if this were an fp in real life, it
simply wouldn't matter):

"""
Data/Ham/Set1/64316.txt
prob = 0.585837796608
prob('header:Errors-To:1') = 0.0228749
prob('from:"nick') = 0.0333333
prob('hotmail') = 0.342294
prob('header:Message-ID:1') = 0.371758
prob('content-type:text/plain') = 0.38492
prob('private,') = 0.612645
prob('confirm') = 0.619434
prob('proto:http') = 0.623798
prob('from') = 0.635447
prob('get') = 0.678986
prob('your') = 0.701188
prob('url:www') = 0.719
prob('url:hotmail') = 0.764839
prob('header:Received:3') = 0.779864
prob('url:com') = 0.79222
prob('msn') = 0.810651
prob('from:email addr:hotmail.com>') = 0.818336
prob('header:Return-Path:1') = 0.919662
prob('free') = 0.952969
prob('e-mail') = 0.974521

Return-Path: <nicrayner@hotmail.com>
Delivered-To: mm+python-list@python.org
Received: from hotmail.com (law2-f27.hotmail.com [216.32.181.27])
        by mail.python.org (Postfix) with ESMTP id 2D4BDEAAF
        for <python-list@python.org>; Mon,  8 Jan 2001 07:51:06 -0500 (EST)
Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC;
         Mon, 8 Jan 2001 04:51:05 -0800
Received: from 194.35.93.8 by lw2fd.hotmail.msn.com with HTTP;
        Mon, 08 Jan 2001 12:50:57 GMT
X-Originating-IP: [194.35.93.8]
From: "Nick Rayner" <nicrayner@hotmail.com>
To: python-list@python.org
Subject: re: ok
Date: Mon, 08 Jan 2001 12:50:57 -0000
Mime-Version: 1.0
Content-Type: text/plain; format=flowed
Message-ID: <LAW2-F27H71yEXUIEAO0000d5ef@hotmail.com>
X-OriginalArrivalTime: 08 Jan 2001 12:51:05.0453 (UTC)
        FILETIME=[A9DDE1D0:01C07971]
Sender: python-list-admin@python.org
Errors-To: python-list-admin@python.org
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.0.1 (101270)
Precedence: bulk
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Post: <mailto:python-list@python.org>
List-Subscribe: <http://www.python.org/mailman/listinfo/python-list>,
        <mailto:python-list-request@python.org?subject=subscribe>
List-Id: General discussion list for the Python programming language
        <python-list.python.org>
List-Unsubscribe: <http://www.python.org/mailman/listinfo/python-list>,
        <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://www.python.org/pipermail/python-list/>

confirm 438765
_________________________________________________________________________
Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com.
""

The other "new fp" is in the same boat:

"""
Data/Ham/Set3/117539.txt
prob = 0.575783849872
prob('header:Errors-To:1') = 0.0225496
prob('from:"peoter') = 0.125
prob('from:email name:<peoter_veliki') = 0.125
prob('from:veliki"') = 0.142857
prob('thanks!') = 0.156368
prob('subject:please') = 0.206268
prob("haven't") = 0.332532
prob('subject:unsubscribe') = 0.333333
prob('header:Message-ID:1') = 0.371745
prob('content-type:text/plain') = 0.384223
prob('header:MIME-Version:1') = 0.627647
prob('been') = 0.662843
prob('from:email addr:hotmail.com>') = 0.808736
prob('charset:windows-1252') = 0.91759
prob('please') = 0.929855
prob('x-mailer:microsoft outlook express 6.00.2600.0000') = 0.931562
prob('content-type:multipart/alternative') = 0.938361
prob('skip:& 10') = 0.971706
prob('&nbsp;') = 0.987687
prob('unsubscribe.') = 0.991379
prob('content-type:text/html') = 0.991417

Received: from [64.4.14.196] (helo=hotmail.com)
        by mail.python.org with esmtp (Exim 3.21 #1)
        id 16Gn4M-0003wz-00
        for python-list@python.org; Wed, 19 Dec 2001 15:12:02 -0500
Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC;
         Wed, 19 Dec 2001 12:11:31 -0800
X-Originating-IP: [205.163.73.182]
From: "Peoter Veliki" <peoter_veliki@hotmail.com>
To: "Python-List@Python.Org" <python-list@python.org>
Subject: moderator: please unsubscribe me!
Date: Wed, 19 Dec 2001 12:14:33 -0800
MIME-Version: 1.0
Content-Type: multipart/alternative;
        boundary="----=_NextPart_000_0025_01C18886.B7DB43C0"
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2600.0000
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
Message-ID: <OE61Iqji8ThFuqpHPeh0000548d@hotmail.com>
X-OriginalArrivalTime: 19 Dec 2001 20:11:31.0429 (UTC)
        FILETIME=[597D9150:01C188C9]
Sender: python-list-admin@python.org
Errors-To: python-list-admin@python.org
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.0.8 (101270)
Precedence: bulk
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Post: <mailto:python-list@python.org>
List-Subscribe: <http://mail.python.org/mailman/listinfo/python-list>,
        <mailto:python-list-request@python.org?subject=subscribe>
List-Id: General discussion list for the Python programming language
        <python-list.python.org>
List-Unsubscribe: <http://mail.python.org/mailman/listinfo/python-list>,
        <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>

This is a multi-part message in MIME format.

------=_NextPart_000_0025_01C18886.B7DB43C0
Content-Type: text/plain;
        charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable

Moderator,=20

    I haven't been able to unsubscribe.  Please do it for me.  Thanks!

------=_NextPart_000_0025_01C18886.B7DB43C0
Content-Type: text/html;
        charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dwindows-1252">
<META content=3D"MSHTML 6.00.2600.0" name=3DGENERATOR>
<STYLE></STYLE>
</HEAD>
<BODY bgColor=3D#ffffff>
<DIV><FONT face=3DArial size=3D2>Moderator, </FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2>&nbsp;&nbsp;&nbsp; I haven't been able =
to=20
unsubscribe.&nbsp; Please do it for me.&nbsp; =
Thanks!</FONT></DIV></BODY></HTML>

------=_NextPart_000_0025_01C18886.B7DB43C0--"""

Looks like changing the tokenizer to replace &nbsp; with a space would save
this one.


From tim.one@comcast.net  Sat Sep 21 20:23:16 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 21 Sep 2002 15:23:16 -0400
Subject: [Spambayes] Moving closer to Gary's ideal
In-Reply-To: <B9B1E441.16436%grobinson@transpose.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCOELMBFAB.tim.one@comcast.net>

[Gary Robinson]
> Keeping the '5's is just being REALLY purist. If everything else is
> right, it shouldn't hurt.  It just falls out of really trying to be
> pure so that we can do this in a mathematical way instead of a way of
> just constantly finding little patches to little problems. In my
> experience being "relentlessly" mathematical, as opposed to just
> continually finding little "kludgy" solutions to each little aspect,
> generally pays off... that path forces you to really get the basics
> EXACTLY right, and then various theorems can be realistically invoked
> to maximize performance.

I couldn't agree more, Gary.  I ranted about the biases and "tricks" in
Paul's original formulation since the start, and have worked like hell to
eliminate them.  Every step of the way, results got demonstrably better by
doing so.  But I hit a wall when trying to out-maneuver the combination of
hambias + 2-digit prob bounds + low max_discriminators:  changing any one of
those demonstrably lost, while changing any two at a time produced mixed
results.  You've got a wonderfully clean way to nuke all 3 at once, my data
isn't arguing with it, and-- clean or not --I listen to my data most closely
of all <wink>.

> OTOH sometimes the data is just so messy to begin with that it just
> isn't possible to get there.

It's worse than that here, as this isn't passive data:  spam can change to
exploit quirks.  Simple example:  I change my spam to look like this:

    multipart-alternative
        text/html
            Here's my spam.
        text/plain
            Thousands of words' worth of utterly bland text.

The user sees the rendered HTML, but the scorer also sees the bland text,
which, if it's bland enough and there's enough of it, can pull the score
down as close to 0.5 as they care to make it.  Even simpler:

    Here's my spam.

    Here's thousands of words' worth of utterly bland text.

Some spam *already* looks like that, and even where the bland text is
randomly generated.  Spammers don't care:  they have to get your attention
at the very start; if they do, it doesn't matter what's at the end of the
message.  The combination of "smoking guns only" and "just a handful of
them" in Graham's formulation, and that 0.5-prob words make no difference at
all to the outcome (*any* pair of words whose probs sum to 1.0), seems
pretty robust against this kind of attack (to put "good" smoking guns in the
gibberish at the end requires that they know a great deal about the targets
they're sending the spam to, customizing it appropriately, and that's
expensive -- if spammers had the budget for that, they wouldn't be
bottom-feeding spammers <wink>).

> IN ANY CASE, I think it would be pragmatic at this point to have more
> experiments with small corpuses with FI1 (f(w))

Note to testers:  that's exactly what the currently checked-in code does, if
you use the suggested options.  I reviewed the code today after getting some
sleep, and found no more bugs in it -- it's computing what I intended it to
compute, which has a non-zero chance of also being what Gary wanted us to
try <wink>.

> and some kind of cutoff so that only extreme values are processed...
> experimentally, you have accrued a lot of evidence that, that does help
> under the current overall processing.

I earlier posted results from simply ignoring words with spamprob (f(w)) in
(0.4, 0.6), and that appeared to be a significant win for the f-n rate on my
large test set.  That's damned hard to achieve.  If there's a more
principled way to get there, I'm all ears, but my intuition remains that if
we try to draw subtle distinctions here, we're trying to solve the wrong
problem (e.g., if we were trying to decide whether a thing was written by
Shakespeare or Bacon, I'd be loathe to throw out the tiniest clue -- but
spam isn't subtle at all, and I think there's a danger of missing the bear's
jaws while staring at its feces with a magnifying glass <wink>).

> f(w) doesn't rely on esoteric theorems to be useful, as g(w)
> does. f(w) just MAKES SENSE.

It makes a lot of sense even from my ignorant viewpoint <wink>.  It's a
beautiful adjustment!

> It should work even if everything else isn't totally pure.
>
> I would run MAX_DISCRIMINATOR testing with respect to f(w), not
> (p(w)), cuz f(w) is more meaningful. Tim has tried both the 150 most
> extreme

Note that all my results from last night and today boosted that to 1,500 (it
was at 150 before only because I hadn't yet written code to prevent problems
with floating point underflow -- there's no limit on how high
max_discriminators can be set now).

> and the 15 most extreme...

I haven't tried that at all with f(w) yet.

> it would be interesting to see what happened with both of those
> measures using abs(f(w)) as the indicator of extremeness -- maybe
> that's what Tim is currently doing, I'm not sure now.

I'm unclear on what you mean by abs(f(w)).  f(w) is always > 0 in my
implementation; I don't see how it could become negative unless a negative
"a" or "x" parameter were fed into to it (which wouldn't seem to make any
sense).  The 2-line patch I posted earlier used

    abs(f(w) - 0.5)

as the measure of extremeness, in exactly the same way we measured "smoking
gun-ness" for Graham's p(w).

> ==>I would also experiment with the size of a. When a gets small
> (compared to 1) it will make a very different choice of words end up at
> the extremes.  a is really something that could meaningfully be tuned
> for performance.<==

If I had any performance left to gain, I'd try that <wink>.

> ---
> For those of you who may be interested in my "purist" perspective (others
> can stop reading now! ;)   ):
>
> I want to be able to invoke a particular theorem via g(w). I have done so
> before very effectively, but it may not be possible here.
>
> I can give ideas to test to bring things further in that direction, but
> it all depends on how far you guys want to go.
>
> I would love to be able to offload experimental work with regard to that
> idea from you guys and do it myself, but I can't now. I'm CEO of a
> company that is in a critical period and just have no free time. I may
> have a bit more free time beginning mid-october, but I'm not sure.

I think that, while it could be better, we've got very usable software in
place to automate almost all of a solid testing methodology.  It's quite
easy to run a test now, although there are several little pieces you have to
learn to string together in the right order, so there's definitely a
learning curve at the start.  What remains difficult for most people is
getting a good corpus of test data, and cleaning it, and classifying it,
and-- if it's a mixed source corpus --painfully learning which of the header
lines you dare not tokenize lest you get superb results for bogus reasons.

But several people here have endured that to the end, so I think we're up to
the challenge.  To test g(w), though, I first have to implement it <wink>.

> From the purist perspective, I think it would be good to look at
> the case of these conferences that are now being classified as spam,
> and see if we can see some very fundamental reason why they are now
> coming out as spams.

They're not, if I boost the cutoff to 0.575.  The "fundamental reason" is
really quite obvious:  they're trying to *sell* you on going to a
conference, and therefore use the language of advertising.  They pretty much
have to.  Tokenizing via word bigrams makes this screamingly obvious:  they
use word pairs that almost never show up in non-sales-oriented email (see
our project's TESTING.txt for many examples).  "You" and "receive" are bland
on their own, but "you receive" is a very spammish bigram.

It's an open question to me too whether or not they *should* be called ham!
Many of the conference announcements posted to comp.lang.python have no
particular relevance to that newsgroup (the ones that do are never called
spam -- they get strong brownie points for mentioning Python!), they are
unsolicited automated email, and they are trying to sell you something.

> I have some ideas which I will think about...
>
> For instance, it is not really right that the original p's are computed
> by counting ALL the occurrences of the words, even multiple occurrences
> in the same email, but then we are NOT doing classification with that
> data but rather just looking at presence vs. absence of a word.

Yes, that's a remaining asymmetry that bugs me too.  Two things I know from
running experiments before on our heavily tweaked Graham scheme;

1. It hurt to *score* a word multiple times in a msg.  The "handful of
   smoking guns" scoring meant that one unlucky word repeated multiple
   times could kill the whole message.  A classic example was the
   poor guy whose perfectly reasonable post was flagged as spam
   because he happened to include some Unix ls output, and his Unix
   name in the output-- repeated on every output line --just happened to
   have a very high spamprob.  Graham's scoring scheme latched on to
   that, and didn't look at anything else.

2. It also hurt to compute the spamprob *not* counting repetitions
   multiple times, but this was a milder effect and seemingly much
   subtler.  I may mention my penis offhandedly before this email
   is done, but the last spam I got mentioned somebody else's penis
   a dozen times.  Massive repetition is part of the "language of
   advertising" too.

Those experiments need to be redone now with f(w) and high
max_discriminators.

> Until we are very consistent about all that stuff, we may not have an
> underlying distribution that enables theorems that rely on particular
> distributions to be invoked, because everything just may be too off-
> kilter.

Understood and appreciated.

> Also there may be other problems that have to be addressed in order to
> really have a known distribution under a null hypothesis. I thought
> of some possible ones last night and will continue thinking about them.

Great!

> Some of these problems may be intrinsic to the data we're dealing
> with -- there may be no way to get around them or we may have to do
> something clever to get around them.

So far "stupid" has beat "clever" on almost every experiment, where I'm
careful not to confuse clever with principled.

> When I get clearer on the problems and possible solutions, I'll
> say so... or if there's anyone here who wants to work with me on that
> perspective, that would be great... just let me know.

Discussion is always welcome here.

> But in the meantime, I think you guys should play with f(w) and tune
> a and MAX_DISCRIMINATOR and see if you can optimise things for better
> performance than you had before, and stay away from further work
> on g(w). That's pragmatic.

Because g(w) is easy to implement, I'm going to try it.  Our source code
here is open for all to stare at, and that gives me much more confidence
that we're not suffering from "stupid bugs" when we try something -- and
it's very easy to make a mistake.  I won't believe that g(w) hurts or helps
until my tests tell me (no offense to GregL intended -- I simply have no
idea what his code does).


From bkc@murkworks.com  Sat Sep 21 20:38:56 2002
From: bkc@murkworks.com (Brad Clements)
Date: Sat, 21 Sep 2002 15:38:56 -0400
Subject: [Spambayes] Practical applications
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGELJBFAB.tim.one@comcast.net>
References: <B9B2087A.1644E%grobinson@transpose.com>
Message-ID: <3D8C92A4.6679.387A2695@localhost>

> > 2) I'm not up on the Python Software Foundation License. Does it allow a
> > company to make a proprietary product that incorporates spambayes code or
> > calls it as if from a library?
> 

I'm thinking along this route too.. I have done a fair bit of integration with Exchange, 
Lookout, Pegasus Mail, Eudora .. NIMS/NetMail, Groupwise. 

They're all in my sites.. I'm just so sick of the amount of spam I get.. Now reaching 
about 70% of my daily messages (I get > 200 inbox messages a day.. And all my "lists" 
are in digest mode.. so you see I get a lot of spam)

I'm a struggling small business with a passion for Python, I've never been able to afford 
a trip to IPC. Maybe productizing this could get me there along with a PSF sponsoring 
membership..  ;-)

Or, I could be sued for patent infringement   :-(

I should spend less time on the free stuff, Python CE, Python NetWare..  Or go out and 
get a real job.



Brad Clements,                bkc@murkworks.com   (315)268-1000
http://www.murkworks.com                          (315)268-9812 Fax
AOL-IM: BKClements


From nas@python.ca  Sat Sep 21 20:47:13 2002
From: nas@python.ca (Neil Schemenauer)
Date: Sat, 21 Sep 2002 12:47:13 -0700
Subject: [Spambayes] Moving closer to Gary's ideal
In-Reply-To: <LNBBLJKPBEHFEDALKOLCOELKBFAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCKELABFAB.tim.one@comcast.net>
	<LNBBLJKPBEHFEDALKOLCOELKBFAB.tim.one@comcast.net>
Message-ID: <20020921194713.GA1842@glacier.arctrix.com>

Tim Peters wrote:
>               distance = abs(prob - 0.5)
> +             if distance < 0.1:
> +                 continue
>               if distance > smallest_best:

For me, that's enough to match the performance default setup:

    [Classifier]
    use_robinson_probability: True
    use_robinson_combining: True
    max_discriminators: 1500

    [TestDriver]
    spam_cutoff: 0.6


    false positive percentages
        0.667  0.000  won   -100.00%
        0.000  0.000  tied
        1.000  0.667  won    -33.30%
        0.333  0.333  tied
        0.000  0.333  lost  +(was 0)
        0.000  0.000  tied

    won   2 times
    tied  3 times
    lost  1 times

    total unique fp went from 6 to 4 won    -33.33%
    mean fp % went from 0.333333333333 to 0.222222222222 won    -33.33%

    false negative percentages
        0.333  1.667  lost  +400.60%
        1.333  1.333  tied
        1.667  2.000  lost   +19.98%
        0.333  0.000  won   -100.00%
        1.333  1.333  tied
        1.667  1.333  won    -20.04%

    won   2 times
    tied  2 times
    lost  2 times

    total unique fn went from 20 to 23 lost   +15.00%
    mean fn % went from 1.11111111111 to 1.27777777778 lost   +15.00%

    Ham distribution for all runs:
    * = 4 items
      5.00   0
      7.50   1 *
     10.00   0
     12.50   3 *
     15.00  18 *****
     17.50  29 ********
     20.00  70 ******************
     22.50 155 ***************************************
     25.00 206 ****************************************************
     27.50 231 **********************************************************
     30.00 227 *********************************************************
     32.50 220 *******************************************************
     35.00 186 ***********************************************
     37.50 127 ********************************
     40.00  91 ***********************
     42.50  72 ******************
     45.00  51 *************
     47.50  31 ********
     50.00  36 *********
     52.50  23 ******
     55.00  13 ****
     57.50   6 **
     60.00   3 *
     62.50   1 *
     65.00   0

    Spam distribution for all runs:
    * = 5 items
     47.50   0
     50.00   4 *
     52.50   1 *
     55.00   7 **
     57.50  11 ***
     60.00  34 *******
     62.50  33 *******
     65.00  58 ************
     67.50 106 **********************
     70.00 169 **********************************
     72.50 210 ******************************************
     75.00 243 *************************************************
     77.50 264 *****************************************************
     80.00 217 ********************************************
     82.50 167 **********************************
     85.00 143 *****************************
     87.50  93 *******************
     90.00  37 ********
     92.50   3 *
     95.00   0


From grobinson@transpose.com  Sat Sep 21 20:56:35 2002
From: grobinson@transpose.com (Gary Robinson)
Date: Sat, 21 Sep 2002 15:56:35 -0400
Subject: [Spambayes] Further Improvement 2
Message-ID: <B9B24733.1646F%grobinson@transpose.com>

Tim said:

> Because g(w) is easy to implement, I'm going to try it.  Our source code
> here is open for all to stare at, and that gives me much more confidence
> that we're not suffering from "stupid bugs" when we try something -- and
> it's very easy to make a mistake.  I won't believe that g(w) hurts or helps
> until my tests tell me (no offense to GregL intended -- I simply have no
> idea what his code does).

[Tim: please see below!!!]



Guys,

I have spent a few hours today thinking about FI2, and I've come to the
conclusion that the nature of this data is such that we can't so easily do
what I was trying to do with it. I had an implicit simplifying assumption in
my thinking, and I now believe it was inappropriate to this real-world
application.

I DO have in mind a very very rough idea of how we might get there, but
unfortunately (for me, that is, because I really like thinking about this) I
don't have time to pursue it further right now. I'll probably keep thinking
about it in the background, and I should have a bit more time in late
october, so I am personally hopeful that the line of thought I have in mind
will bear fruit.

BUT for now, I am going to simply delete FI2 from the essay, and hope that
the ideas that have been explored so far will turn out to be useful. So far,
I'm happy that it looks like they may be! :)

I think playing with the "a" constant in FI1 will be useful, and playing
with max_discriminators will be useful. So I strongly encourage playing with
both of them... I know you're more used to playing with max_discrinators but
I think it's important to play with "a" too.

I'll stay subscribed to the spambayes list and will contribute any thoughts
I may happen have to have along the way, but I think now it's a mostly
matter of tuning the parameters against test setups, and I don't have a test
setup so I can't be of much use there!

All the best,

And thanks SO much for your willingness to play with my ideas, which I
REALLY appreciate,

Gary


-- 
Gary Robinson
CEO
Transpose, LLC
grobinson@transpose.com
207-942-3463
http://www.emergentmusic.com
http://radio.weblogs.com/0101454



From tim.one@comcast.net  Sat Sep 21 21:19:16 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 21 Sep 2002 16:19:16 -0400
Subject: [Spambayes] Moving closer to Gary's ideal
In-Reply-To: <20020921194713.GA1842@glacier.arctrix.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCOELOBFAB.tim.one@comcast.net>

[Neil Schemenauer, on fiddling the Robinson scheme to ignore words w/
 spamprob less than 0.1 away from 0.5]

> For me, that's enough to match the performance default setup:

Cool!
>     [Classifier]
>     use_robinson_probability: True
>     use_robinson_combining: True
>     max_discriminators: 1500
>
>     [TestDriver]
>     spam_cutoff: 0.6

Damn, I wish we had a better handle on guessing a good value for this a
priori.  It seems to get less sensitive the more the spam and ham means get
separated.

When a current batch of mini-tests finishes running, I'm going to check in a
new option:

[Classifier]
# When scoring a message, ignore all words with
# abs(word.spamprob - 0.5) < robinson_minimum_prob_strength.
# By default (0.0), nothing is ignored.
# Tim got a pretty clear improvement in f-n rate on his hasn't-improved-in-
# a-long-time large c.l.py test by using 0.1.  No other values have been
# tried yet.
# Neil Schemenauer also reported good results from 0.1, making the all-
# Robinson scheme match the all-default Graham-like scheme on a smaller
# and different corpus.
# NOTE:  Changing this may change the best spam_cutoff value for your
# corpus.  Since one effect is to separate the means more, you'll probably
# want a higher spam_cutoff.
robinson_minimum_prob_strength: 0.0


From crunch@shopip.com  Sat Sep 21 21:17:32 2002
From: crunch@shopip.com (John Draper)
Date: Sat, 21 Sep 2002 13:17:32 -0700
Subject: [Spambayes] Practical applications
In-Reply-To: <3D8C92A4.6679.387A2695@localhost>
References: <LNBBLJKPBEHFEDALKOLCGELJBFAB.tim.one@comcast.net>
 <B9B2087A.1644E%grobinson@transpose.com>
Message-ID: <v03110777b9b282f83602@[192.168.0.2]>

Has anyone thought of a local POP proxy (like that one that was submitted=
 last week),  instead of trying to "hook" into programs like Outlook or=
 Eudora.   On the Mac,  it's easy,  using AppleEvents,  but this is just Mac=
 related,  and does no good for the millions of winBlows users.

My understanding of how proxy servers work is really elementary,  but isn't=
 it possible for the SMTP server side to tag on some extra headers like the=
 spam probability numbers,   then let the local proxy "filter" this and sort=
 out the mail into local mailboxes?  Like "real spam",  "probable spam" and=
 the real messages in thier local inbox.   Well,   anyway,  I may be off=
 base here,  until I learn more about the POP server processes.   It's just=
 a thought.

John



From crunch@shopip.com  Sat Sep 21 21:24:05 2002
From: crunch@shopip.com (John Draper)
Date: Sat, 21 Sep 2002 13:24:05 -0700
Subject: [Spambayes] Practical applications
In-Reply-To: <3D8C92A4.6679.387A2695@localhost>
References: <LNBBLJKPBEHFEDALKOLCGELJBFAB.tim.one@comcast.net>
 <B9B2087A.1644E%grobinson@transpose.com>
Message-ID: <v03110779b9b28570ca78@[192.168.0.2]>

>They're all in my sites.. I'm just so sick of the amount of spam I get.. Now reaching 
>about 70% of my daily messages (I get > 200 inbox messages a day.. And all my "lists" 
>are in digest mode.. so you see I get a lot of spam)

HAH!   Don't feel bad - I'm pushing 85% spam.
>
>I'm a struggling small business with a passion for Python, I've never been able to afford 
>a trip to IPC. Maybe productizing this could get me there along with a PSF sponsoring 
>membership..  ;-)

What is IPC?
>
>Or, I could be sued for patent infringement   :-(
>
>I should spend less time on the free stuff, Python CE, Python NetWare..  Or go out and 
>get a real job.

BY the way,  are all of you aware of an anti-spam conference in Boston in Jan?
If not,  let me know,  and I'll send you all the info.

John



From tim.one@comcast.net  Sat Sep 21 22:10:43 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 21 Sep 2002 17:10:43 -0400
Subject: [Spambayes] RE: Further Improvement 2
In-Reply-To: <B9B24733.1646F%grobinson@transpose.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEMBBFAB.tim.one@comcast.net>

[Gary Robinson]
> I have spent a few hours today thinking about FI2, and I've come to the
> conclusion that the nature of this data is such that we can't so easily
> do what I was trying to do with it. I had an implicit simplifying
> assumption in my thinking, and I now believe it was inappropriate to
> this real-world application.

Fair enough -- we've got plenty to test without it <wink>.

> I DO have in mind a very very rough idea of how we might get there, but
> unfortunately (for me, that is, because I really like thinking
> about this) I don't have time to pursue it further right now. I'll
> probably keep thinking about it in the background, and I should have a
> bit more time in late october, so I am personally hopeful that the
> line of thought I have in mind will bear fruit.
>
> BUT for now, I am going to simply delete FI2 from the essay, and hope
> that the ideas that have been explored so far will turn out to be
> useful.  So far, I'm happy that it looks like they may be! :)

We're probably not going to get many test results over the weekend, but it
sure looks promising to me.

> I think playing with the "a" constant in FI1 will be useful, and
> playing with max_discriminators will be useful. So I strongly
> encourage playing with both of them... I know you're more used to
> playing with max_discrinators but I think it's important to play
> with "a" too.

Since I got a big win without any effort <wink> by introducing a brand new
"ignore probs that aren't at least this far from neutral" knob, that's the
one I'm most inclined to play with right now.  There isn't a knob in
existence that won't be played with, but especially large tests take
significant wall-clock time to complete, and there's only so much testing
one can do in a day.

Testers, "a" is already exposed via:

[Classifier]
robinson_probability_a: 1.0

I think values nearer to 0 are most likely to be most interesting.

Since nobody likes thrashing in the dark, I'll give you some motivation, but
I hope Gary can take just a little more time to flesh it out:

Graham computes probabilities in a simple and pretty obvious way (btw, this
part is *much* more obvious under Gary's scheme, because there aren't any
MINCOUNT and MAX_SPAMPROB and MIN_SPAMPROB and HAMBIAS and SPAMBIAS "mystery
knobs" complicating this part any more).

Gary then does a Bayesian adjustment to that probability.  From 10,000 feet,
we *assume* that the distribution of words comes from a particular
distribution family.  The "a" in the adjustment step is a parameter that
defines a specific member of that family.  The real-word evidence we collect
by counting words is more reliable the larger the corpus and the more often
we see a given word in the corpus.  The adjustment step blends our prior
assumption about word distribution with the actual data we see, giving more
weight to the latter the more actual data about the word we have.

So this has most effect on words that appear rarely.  When I got rid of
MINCOUNT, a word that appeared exactly once in the entire training set got a
"probability" of 0.01 or 0.99 (depending on which half of the corpus it was
in).  It was just as strong a clue to Graham's scoring scheme as a word that
appeared in every one of the spams and none of the hams.  That's nuts on the
face of it, but even so it made a real improvement.  I later thrashed with
various schemes to reduce this nonsense (that's what the now-gone
adjust_probs_by_evidence_mass option was all about, btw), but they just
weren't helping consistently.

Gary has a much better scheme for doing this.  What I can't tell you is
*which* prior distribution it's assuming, or how changing "a" affects its
shape.  But since we know for sure that getting rid of MINCOUNT made a real
difference before, we also know for sure that the treatment of rare words is
important, and that's what "a" is all about.  This is the point where Gary
unexpectedly discovers he has just enough free time to tell us more about
the specifics of this distribution <wink>.

> I'll stay subscribed to the spambayes list and will contribute
> any thoughts I may happen have to have along the way, but I think now
> it's a mostly matter of tuning the parameters against test setups,
> and I don't have a test setup so I can't be of much use there!

Just don't vanish entirely.

> All the best,
>
> And thanks SO much for your willingness to play with my ideas, which I
> REALLY appreciate,

We're going to get sooooo rich from giving your ideas away too, that's all
the thanks we need <wink>.  Thank you!


From grobinson@transpose.com  Sat Sep 21 22:16:38 2002
From: grobinson@transpose.com (Gary Robinson)
Date: Sat, 21 Sep 2002 17:16:38 -0400
Subject: [Spambayes] A little idea
Message-ID: <B9B259F6.16477%grobinson@transpose.com>

One small idea...

We all recognize that middling values are less important, but that is
implicitly recognized by the multiplicative math we are doing.

NEVERTHELESS, since we are not being purist, I think it would be fun to try
one simple modification.

We are trying different max_discriminator values which represent a binary
choice... a p is either IN or OUT. But wouldn't it be logical to be more
gentle about it, and just weight them less as they become more middling?

So the idea is to weight each p by c(w)=2*abs(.5-p(w)) (it's nice to have
weights that vary between 0 and 1).

When you're multiplying and taking the nth root, this is done by using the
weights as powers and using the sum of the weights as the root: So when we
have (p(w1)*p(w2)*...*p(wn))^(1/n), use instead

((p(w1)^c(w1))*(p(w2)^c(w2))*...*(pwn)^c(wn))^(1/(c(w1)+c(w2)+...+c(wn)).

This looks a little more natural when you do the calcs using logs, because
then you just multiply each term by the weight rather than using it as a
power... but the result is the same either way.

Just a thought!


--Gary


-- 
Gary Robinson
CEO
Transpose, LLC
grobinson@transpose.com
207-942-3463
http://www.emergentmusic.com
http://radio.weblogs.com/0101454



From tim.one@comcast.net  Sat Sep 21 22:29:20 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 21 Sep 2002 17:29:20 -0400
Subject: [Spambayes] RE: spam detection via probability - actual results!
In-Reply-To: <15756.35889.250276.677544@12-248-11-90.client.attbi.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEMCBFAB.tim.one@comcast.net>

[Skip Montanaro]
> I was thinking about this yesterday when I ran rebal because to
> switch from rebalancing ham to rebalancing spam I hit Ctl-P then
> had to edit two places in the line.  A bit late for Tim and
> Anthony, but perhaps the change I just checked in will help
> others avoid this problem.

Oddly enough, it wouldn't have helped me.  It's hard to reconstruct just how
screwed up I got <wink>.

Note that I've checked in another change:  -Q no longer implies -q.  That's
the part that would have helped me, across several steps of screwing up.  I
didn't want to confirm every move, but I had not said -q so the lack of any
output convinced me it wasn't doing anything.  So I kept doing it again and
again, fiddling the parameters each time in what turned out to be disastrous
ways.  The change you made probably would have prevented at least one of the
last screwups.

Heh:  as part of "fixing" this, I managed to move a whole bunch of ham into
my spam *reservoir* too.  So I'd clean out all the ham that a test found in
the spam Sets, ran rebal again, and time after time a new run kept finding
more new ham in the spam, but an ever-decreasing amount.  I thought that
this was just because it was getting trained better; it was really because
my rebals kept putting more ham into the spam.

In the end, I took Neil's suggestion one step further:  I stopped running
tests at all, and just did a grep for python-list over the spam sets.  That
caught 100% of the misclassified ham instantly.

So mixed-source corpora aren't entirely a bad thing either <wink>.


From popiel@wolfskeep.com  Sat Sep 21 22:37:24 2002
From: popiel@wolfskeep.com (T. Alexander Popiel)
Date: Sat, 21 Sep 2002 14:37:24 -0700
Subject: [Spambayes] RE: Further Improvement 2 
In-Reply-To: Message from Tim Peters <tim.one@comcast.net> 
	of "Sat, 21 Sep 2002 17:10:43 EDT."
	<LNBBLJKPBEHFEDALKOLCKEMBBFAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCKEMBBFAB.tim.one@comcast.net> 
Message-ID: <20020921213724.11380F589@cashew.wolfskeep.com>

In message:  <LNBBLJKPBEHFEDALKOLCKEMBBFAB.tim.one@comcast.net>
             Tim Peters <tim.one@comcast.net> writes:
>
>We're probably not going to get many test results over the weekend, but it
>sure looks promising to me.

Well, I was about to start doing some real tests using my corpora,
but immediately ran into the 'requires python 2.3a0' comment.  I'll
be honest: I don't use python much, and I'm not particularly inclined
to drag an alpha version of it out of CVS.  Is there any good reason
to make the filter depend on that version of python?  What is it that
would keep it from running using python 2.1.3 (the debian stable default)?

- Alex

From nas@python.ca  Sat Sep 21 23:56:45 2002
From: nas@python.ca (Neil Schemenauer)
Date: Sat, 21 Sep 2002 15:56:45 -0700
Subject: [Spambayes] Moving closer to Gary's ideal
In-Reply-To: <LNBBLJKPBEHFEDALKOLCOELMBFAB.tim.one@comcast.net>
References: <B9B1E441.16436%grobinson@transpose.com>
	<LNBBLJKPBEHFEDALKOLCOELMBFAB.tim.one@comcast.net>
Message-ID: <20020921225645.GA2130@glacier.arctrix.com>

Tim Peters wrote:
> [...] but spam isn't subtle at all, and I think there's a danger of
> missing the bear's jaws while staring at its feces with a magnifying
> glass <wink>).

<LOL> Are you sure there isn't any acid in _your_ water?  :-)

  Neil

From guido@python.org  Sun Sep 22 01:45:59 2002
From: guido@python.org (Guido van Rossum)
Date: Sat, 21 Sep 2002 20:45:59 -0400
Subject: [Spambayes] Practical applications
In-Reply-To: Your message of "Sat, 21 Sep 2002 11:28:58 EDT."
             <B9B2087A.1644E%grobinson@transpose.com> 
References: <B9B2087A.1644E%grobinson@transpose.com> 
Message-ID: <200209220046.g8M0k0J27042@pcp02138704pcs.reston01.va.comcast.net>

> 1) I notice that Python is an "Actice Scripting" language? Does that a mean
> that a Python script can interact with Outlook to do things like add
> a "Spam!" button, etc.?

Python can control any Windows application that supports a COM
interface.  You have to get the win32all set of additions (there's a
pointer to it on the download pages).  I've never done this, I suppose
you'd need docs for Outlook.

> 2) I'm not up on the Python Software Foundation License. Does it allow a
> company to make a proprietary product that incorporates spambayes code or
> calls it as if from a library?

Sure.  It's BSD-ish, not GPL-ish.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido@python.org  Sun Sep 22 02:15:04 2002
From: guido@python.org (Guido van Rossum)
Date: Sat, 21 Sep 2002 21:15:04 -0400
Subject: [Spambayes] Make your own spam
In-Reply-To: Your message of "Sat, 21 Sep 2002 08:58:06 PDT."
             <v0311076eb9b2443c2819@[192.168.0.2]> 
References: <v0311076eb9b2443c2819@[192.168.0.2]> 
Message-ID: <200209220115.g8M1F4I27108@pcp02138704pcs.reston01.va.comcast.net>

> I believe a useful exercise would be to try to craft email messages
> which would fool the classifier. Perhaps the goal should be to try
> to fabricate (with minimal classifier feedback) messages which score
> 0.5 likelyhood. Create a message (a 'upam' or 'shpam', for unknown
> ham/spam) and then guess what it's S/H probability is beforehand,
> then submit it and see how close your estimate was. I believe these
> exercises will suggest weaknesses/fortification methods, or at
> minimum give a better feel for the mechanism.

I think the term should be 'sham'.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido@python.org  Sun Sep 22 02:21:43 2002
From: guido@python.org (Guido van Rossum)
Date: Sat, 21 Sep 2002 21:21:43 -0400
Subject: [Spambayes] Moving closer to Gary's ideal
In-Reply-To: Your message of "Sat, 21 Sep 2002 13:45:05 EDT."
             <LNBBLJKPBEHFEDALKOLCOELKBFAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCOELKBFAB.tim.one@comcast.net> 
Message-ID: <200209220121.g8M1LhV27146@pcp02138704pcs.reston01.va.comcast.net>

> [TestDriver]
> spam_cutoff: 0.575

It's too soon for me to say for sure, but from a previous run that
very same cutoff also looks like it would be a winner for my corpus!
Neil's corpus likes a similar value (0.56).

What does this mean?  That the spam bell curve is narrower than the
ham bell curve?  (Hm, have you computed mean and standard deviation?)

--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido@python.org  Sun Sep 22 02:27:29 2002
From: guido@python.org (Guido van Rossum)
Date: Sat, 21 Sep 2002 21:27:29 -0400
Subject: [Spambayes] Practical applications
In-Reply-To: Your message of "Sat, 21 Sep 2002 13:24:05 PDT."
             <v03110779b9b28570ca78@[192.168.0.2]> 
References: <LNBBLJKPBEHFEDALKOLCGELJBFAB.tim.one@comcast.net>
	<B9B2087A.1644E%grobinson@transpose.com>  
	<v03110779b9b28570ca78@[192.168.0.2]> 
Message-ID: <200209220127.g8M1RUS27173@pcp02138704pcs.reston01.va.comcast.net>

> What is IPC?

International Python Conference.  See python.org/workshops/ for past
ones.  The next one is likely the week of March 23 next year, in
Washington DC, and *really* cheap (we're switching to YAS as the
organizer).  Probably $125.

> BY the way, are all of you aware of an anti-spam conference in
> Boston in Jan?  If not, let me know, and I'll send you all the info.

Please post here!

--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido@python.org  Sun Sep 22 02:35:48 2002
From: guido@python.org (Guido van Rossum)
Date: Sat, 21 Sep 2002 21:35:48 -0400
Subject: [Spambayes] Use for gray area in scoring range
In-Reply-To: Your message of "Sat, 21 Sep 2002 21:27:29 EDT."
	<200209220127.g8M1RUS27173@pcp02138704pcs.reston01.va.comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCGELJBFAB.tim.one@comcast.net>
	<B9B2087A.1644E%grobinson@transpose.com> <v03110779b9b28570ca78@[192.168.0.2]>
	<200209220127.g8M1RUS27173@pcp02138704pcs.reston01.va.comcast.net> 
Message-ID: <200209220135.g8M1Zn927221@pcp02138704pcs.reston01.va.comcast.net>

Yesterday I said I didn't think there was much use for a gray area in
the scoring range, where messages scoring somewhere in the middle
would have to be manually inspected.  I think I'm going to revert that
opinion, at least with the Gary Robinson formula.

It would be handy to have this if you're trying to tweak the cutoff.
You don't want the cutoff too low, or it'll give you too many false
negatives.  But even stronger you don't want it too high, or it'll
give too many false positives.  Specifying a gray area gives you a
useful tool to see if you have set your cutoff right.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.one@comcast.net  Sun Sep 22 04:28:13 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 21 Sep 2002 23:28:13 -0400
Subject: [Spambayes] Use for gray area in scoring range
In-Reply-To: <200209220135.g8M1Zn927221@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCOEMKBFAB.tim.one@comcast.net>

[Guido]
> Yesterday I said I didn't think there was much use for a gray area in
> the scoring range, where messages scoring somewhere in the middle
> would have to be manually inspected.  I think I'm going to revert that
> opinion, at least with the Gary Robinson formula.

That's mostly because it *has* a "middle ground" == a relatively small range
where a significant number of false negatives and false positives actually
live.  This wasn't true of the #s produced by the Graham combination
scheme -- numbers coming out of that were almost always very near 0 or very
near 1.  In contrast, Gary's combination scheme almost never produces
numbers near the endpoints.

> It would be handy to have this if you're trying to tweak the cutoff.
> You don't want the cutoff too low, or it'll give you too many false
> negatives.  But even stronger you don't want it too high, or it'll
> give too many false positives.

You've got those backwards, but I know what you mean <wnk>.

> Specifying a gray area gives you a useful tool to see if you have
> set your cutoff right.

The practical difficulty here is that the gray area we're observing is very
narrow:  the difference between setting the cutoff at 0.5 or 0.575 in my
large test was the difference between seeming disaster and "does just as
well as our Graham-like scheme".  Move it to 0.60, and it heads back to
disasterland again.  It's great to have the knob, but it's sensitive, and so
far we've no idea how to choose it short of trial and error (it's easy to
choose if you've got the score histograms to stare at, but end users won't).


From tim.one@comcast.net  Sun Sep 22 04:30:28 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 21 Sep 2002 23:30:28 -0400
Subject: [Spambayes] Practical applications
In-Reply-To: <200209220127.g8M1RUS27173@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCEEMLBFAB.tim.one@comcast.net>

[John Draper]
> BY the way, are all of you aware of an anti-spam conference in
> Boston in Jan?  If not, let me know, and I'll send you all the info.

{Guido]
> Please post here!

I assume this the conference Paul Graham is organizing:

    http://www.paulgraham.com/spamconference.html


From tim.one@comcast.net  Sun Sep 22 05:14:30 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 22 Sep 2002 00:14:30 -0400
Subject: [Spambayes] Moving closer to Gary's ideal
In-Reply-To: <200209220121.g8M1LhV27146@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEMMBFAB.tim.one@comcast.net>

[Guido van Rossum, on
    [TestDriver]
    spam_cutoff: 0.575
]

> It's too soon for me to say for sure, but from a previous run that
> very same cutoff also looks like it would be a winner for my corpus!

You can tell for sure just by looking at the score histograms and counting
the dots <wink>; there's no need to change spam_cutoff and then rerun the
test (spam_cutoff has no effect on the scores computed); I've walked through
that process in slow motion several times on the list now.

> Neil's corpus likes a similar value (0.56).

Until he used the new robinson_minimum_prob_strength option, and then the
best value appeared to creep up to 0.60 for him.

> What does this mean?

I don't know what this number means.  Paul said his outputs were
probabilities, but that wasn't so.  Gary has been suitably careful not to
claim anything about what his outputs "mean", beyond that

    the product of the probabilities is monotonic with the Fisher
    inverse chi-square combined probability technique from meta-analysis

The Gary-scores certainly don't act like probabilities either, although
we're seeing over and over that once you know the best cutoff point, a
significant number of the false positives and false negatives score within a
small distance of it.

> That the spam bell curve is narrower than the ham bell curve?

Become friends with the score histograms!  They were useless with the
Graham-like scheme, because the ham histogram approximated a solid bar at
0.0, and the spam histogram a solid bar at 1.0.  Under Gary's scheme, they
look normally distributed, but with relatively long lopsided tails dribbling
toward each other.  Here's the non-zero slice of my ham histogram:

 20.00   26 *
 22.50  155 **
 25.00  627 ********
 27.50 1859 **********************
 30.00 3780 ********************************************
 32.50 5108 ************************************************************
 35.00 4264 **************************************************
 37.50 2450 *****************************
 40.00 1056 *************
 42.50  395 *****
 45.00  178 ***
 47.50   52 *
 50.00   30 *
 52.50   13 *
 55.00    4 *
 57.50    1 *
 60.00    1 *
 62.50    1 *

By inspecition, the mean is clearly close to 35.  For the spam,

 42.50    1 *
 45.00    0
 47.50    0
 50.00    3 *
 52.50    6 *
 55.00   17 *
 57.50   40 *
 60.00   76 **
 62.50  171 ****
 65.00  394 ********
 67.50  710 ***************
 70.00 1247 *************************
 72.50 2358 ************************************************
 75.00 2986 ************************************************************
 77.50 2659 ******************************************************
 80.00 1957 ****************************************
 82.50 1069 **********************
 85.00  192 ****
 87.50   31 *
 90.00   61 **
 92.50   22 *

The mean is more like 77 there.  The ham distribution actually looks tighter
than the spam distribution, but the mean of the spam is about 25 points
above 50 while the mean of the ham only about 15 points below 50.  I expect
it's this lopsidedness (wrt the midpoint) that makes a cutoff above 50 more
suitable.

An observed effect of setting robinson_minimum_prob_strength is to increase
the separation of the ham and spam means:  the ham mean gets lower and the
spam mean gets higher.  This is what I expected, since, unlike as in
Graham's scheme, scoring words with neutral probability in Gary's scheme
drags a score closer to 0.5.  Now "drags" sounds pejorative, because that's
the way I feel about it -- I see no value in scoring neutral words at all in
this task.  Gary disagrees, but allows that it's more of a "purist" issue
than a pragmatic one.  However, something we agree 100% on is that measuring
the effects of *principled* changes gets much harder if pragmatic hacks
muddy the mathematical basis of a scheme.  If Gary's scheme proves to be as
good as, but no better than, our current scheme, I'd still switch to it for
this reason:  it has far fewer "mystery knobs" to confuse the underlying
issues.

> (Hm, have you computed mean and standard deviation?)

Nope.  What would you do with them if I did (they're easy enough to compute
and display if there's a point to it)?  You can get an excellent feel for
them by looking at the histograms (which reveal far more than a pair of
(mean, sdev) numbers anyway).


From tim.one@comcast.net  Sun Sep 22 05:36:14 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 22 Sep 2002 00:36:14 -0400
Subject: [Spambayes] Moving closer to Gary's ideal
In-Reply-To: <Pine.LNX.4.44.0209210823320.402-100000@familjen.svensson.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEMNBFAB.tim.one@comcast.net>

[Paul Svensson]
> Just a hunch:
> For the 'indifferent' words to make no difference,
> shouldn't x be set the same as the cutoff ?

You're welcome to test it <wink>, but I doubt it.  x is (in effect) the
spamprob given to a word during scoring when that word has never been seen
in the training data.  If the best cutoff value happens to be, say, 0.75, I
see no justification for saying that a msg with an unknown word has a three
quarters chance of being spam.  IOW, this seems to me to conflate unrelated
(or at best muddily related) issues.


From tim.one@comcast.net  Sun Sep 22 05:46:00 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 22 Sep 2002 00:46:00 -0400
Subject: [Spambayes] idea for tokenizer.crack_filename change
In-Reply-To: <15756.33264.517811.330133@12-248-11-90.client.attbi.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEMOBFAB.tim.one@comcast.net>

[Skip Montanaro]
>>> It seems to me that base64-encoded, all DOS/Windows executables start
>>> with (reciting from memory, since I've deleted all viruses and
>>> haven't received any new ones in the last 15 minutes or so) "TPqAAA"
>>> or something similar.  Why rely on finding specific file extensions?
>>> They can just change.

[Tim]
>> Well, not often, and the scheme we're working on is supposed to be
>> able tolearn when they do <wink>.  Would you like to write some
>> code to tokenize this particular bit of Windows Lore?

[Skip]
> I gave it a try, but I'm still suffering with fp/fn rates around 15%,

Yuck.  Weren't you going to share some detailed output?  Something isn't
right, but you've already guessed that <wink>.

> so anything I see is suspect.  Also, I saw no change.  It's quite
> possible I have a bug, but I've also cleaned out obvious viruses
> from my corpora.

Then is there any reason to hope that you *would* see a change?

> True spam may have enough indicators elsewhere that this scheme
> won't help.

Note that no single indicator carries much weight under this scheme (whether
Paul's or Gary's doesn't matter -- one clue is just one clue, and I expect
that makes a rule-based scheme (like Greg's) more effective for virus
detection, where a "smoking gun" is more like a "smoking hydrogen bomb").

> Should I just go ahead and checkin my change (it is controlled by a
> couple new options, and by default is not enabled) and let y'all point
> out my bugs?

Sure!  I'll be happy to run my tests, but note that I can no longer measure
improvements, and I know it will have no effect on my f-n rate (I know all
of my f-n intimately now, and none have an executable attachment).


From tim.one@comcast.net  Sun Sep 22 06:12:24 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 22 Sep 2002 01:12:24 -0400
Subject: [Spambayes] Practical applications
In-Reply-To: <3D8C92A4.6679.387A2695@localhost>
Message-ID: <LNBBLJKPBEHFEDALKOLCMEMPBFAB.tim.one@comcast.net>

[Brad Clements]
> I'm thinking along this route too.. I have done a fair bit of
> integration with Exchange, Lookout, Pegasus Mail, Eudora ..
> NIMS/NetMail, Groupwise.
>
> They're all in my sites.. I'm just so sick of the amount of spam
> I get.. Now reaching about 70% of my daily messages (I get > 200 inbox
> messages a day.. And all my "lists" are in digest mode.. so you see
> I get a lot of spam)
>
> I'm a struggling small business with a passion for Python, I've
> never been able to afford a trip to IPC. Maybe productizing this could
> get me there along with a PSF sponsoring membership..  ;-)

It seems I get a lot more spam than you, and an ongoing irony in this
project is that I don't particularly mind it <wink>.  I would *love* a
gimmick that could use this software to give my Outlook 2000 a way to
shuffle suspected spam into a different folder, where "love" means I would
pay for such a product if one were available.  I'd pay up to $150.00 for
that.  OTOH, I wouldn't spend an hour of my preciously tiny "free time"
trying to figure out how to hook up to Outlook myself.

> Or, I could be sued for patent infringement   :-(

Note that MS's patent is on Vector Support Machine anti-spam technology.
We're not anywhere close to that, and Bayesian learning techniques have been
in the open literature for more than 40 years.

> I should spend less time on the free stuff, Python CE, Python
> NetWare..  Or go out and get a real job.

The older I get (and I'm not going to get *much* older <wink>), the less I
think real jobs are worth the distraction from what's really important --
like sleeping past noon because passion kept you up past dawn <wink>.


From tim.one@comcast.net  Sun Sep 22 06:24:43 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 22 Sep 2002 01:24:43 -0400
Subject: [Spambayes] RE: Further Improvement 2
In-Reply-To: <20020921213724.11380F589@cashew.wolfskeep.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCOENABFAB.tim.one@comcast.net>

[T. Alexander Popiel]
> Well, I was about to start doing some real tests using my corpora,
> but immediately ran into the 'requires python 2.3a0' comment.  I'll
> be honest:

Please, anything but that <wink>.

> I don't use python much, and I'm not particularly inclined to drag an
> alpha version of it out of CVS.  Is there any good reason to make the
> filter depend on that version of python?

I'm a Python developer, and when I started this code I wanted it also to
serve as a stress test for several new features in Python 2.3.  Most of the
other coders on *this* project are also developers on the Python project.
It's just an historical quirk.

> What is it that would keep it from running using python 2.1.3
> (the debian stable default)?

Sorry, but I barely remember the 2.1 line anymore.  The current stable
release is 2.2.1.  The 2.3 features we're using over that are shallow:

+ Use of the new-in-2.3 "sets" module.
+ Use of the new-in-2.3 "heapq" module.

Those are pure Python modules and would probably work fine if just dropped
in as-is to a 2.2 Lib directory.

There are other minor 2.3-isms, like use of the new builtin "enumerate"
function (easily emulated with a Python function).

There's ubiquitous use of generators ("semi-coroutines"), though, and that's
never going to fly in the 2.1 line.  In the 2.2 line, it would just require
adding

    from __future__ import generators

to the top of a few modules.

So I expect it would be an easy project to get this code working under 2.2,
but for someone more conversant with Python than you are so far.  It's not
something I'm going to do, though -- anyone else?


From guido@python.org  Sun Sep 22 07:16:01 2002
From: guido@python.org (Guido van Rossum)
Date: Sun, 22 Sep 2002 02:16:01 -0400
Subject: [Spambayes] Use for gray area in scoring range
In-Reply-To: Your message of "Sat, 21 Sep 2002 23:28:13 EDT."
             <LNBBLJKPBEHFEDALKOLCOEMKBFAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCOEMKBFAB.tim.one@comcast.net> 
Message-ID: <200209220616.g8M6G2D27626@pcp02138704pcs.reston01.va.comcast.net>

> That's mostly because it *has* a "middle ground"

I was aware of that when I first said I didn't want a "maybe" scoring
category; what made me change was trying to tune the knob.

> > Specifying a gray area gives you a useful tool to see if you have
> > set your cutoff right.
> 
> The practical difficulty here is that the gray area we're observing
> is very narrow: the difference between setting the cutoff at 0.5 or
> 0.575 in my large test was the difference between seeming disaster
> and "does just as well as our Graham-like scheme".  Move it to 0.60,
> and it heads back to disasterland again.

I haven't found the right setting for me yet.  0.575 did better than
0.55 but still much worse on the fps than Graham.

> It's great to have the knob, but it's sensitive, and so far we've no
> idea how to choose it short of trial and error (it's easy to choose
> if you've got the score histograms to stare at, but end users
> won't).

Plus, we don't know why it's not 50, right?

Might that have to do with the spam/ham ration?  I've got 83 hams for
each 32 spams.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.one@comcast.net  Sun Sep 22 07:21:32 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 22 Sep 2002 02:21:32 -0400
Subject: [Spambayes] Another min prob strength result
In-Reply-To: <LNBBLJKPBEHFEDALKOLCEEMLBFAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCKENCBFAB.tim.one@comcast.net>

My usual big test.  "Before" here is

[Classifier]
use_robinson_probability: True
use_robinson_combining: True
max_discriminators: 1500
robinson_minimum_prob_strength: 0.1
[TestDriver]
spam_cutoff: 0.575

"After" is the same, but boosting 0.1 to 0.2.

"""
-> <stat> tested 2000 hams & 1400 spams against 18000 hams & 12600 spams
   [ditto 19 times]

false positive percentages
    0.050  0.050  tied
    0.000  0.000  tied
    0.050  0.050  tied
    0.000  0.000  tied
    0.050  0.100  lost  +100.00%
    0.000  0.000  tied
    0.000  0.000  tied
    0.050  0.050  tied
    0.000  0.000  tied
    0.050  0.050  tied

won   0 times
tied  9 times
lost  1 times

total unique fp went from 5 to 6 lost   +20.00%
mean fp % went from 0.025 to 0.03 lost   +20.00%

false negative percentages
    0.000  0.000  tied
    0.071  0.071  tied
    0.000  0.000  tied
    0.143  0.071  won    -50.35%
    0.143  0.071  won    -50.35%
    0.143  0.071  won    -50.35%
    0.143  0.143  tied
    0.214  0.071  won    -66.82%
    0.143  0.143  tied
    0.000  0.000  tied

won   4 times
tied  6 times
lost  0 times

total unique fn went from 14 to 9 won    -35.71%
mean fn % went from 0.1 to 0.0642857142857 won    -35.71%
"""

So the trend appears to continue:  ignoring low-value indicators decreases
the f-n rate, although 5 spam out of 14,000 isn't much to go on.

The ham mean again gets smaller, and spam mean higher:

Ham distribution for all runs:
* = 51 items
  0.00    1 *
  2.50   17 *
  5.00   72 **
  7.50  128 ***
 10.00  262 ******
 12.50  448 *********
 15.00  857 *****************
 17.50 1396 ****************************
 20.00 2038 ****************************************
 22.50 2726 ******************************************************
 25.00 3010 ************************************************************
 27.50 2862 *********************************************************
 30.00 2301 **********************************************
 32.50 1596 ********************************
 35.00 1027 *********************
 37.50  603 ************
 40.00  298 ******
 42.50  178 ****
 45.00   93 **
 47.50   37 *
 50.00   29 *
 52.50    9 *
 55.00    6 *
 57.50    3 *
 60.00    1 *
 62.50    1 *
 65.00    0
 67.50    1 *
 70.00    0
 ... [all 0 after this] ...

Spam distribution for all runs:
* = 48 items
 ... [all 0 before this] ...
 37.50    0
 40.00    1 *
 42.50    0
 45.00    0
 47.50    0
 50.00    3 *
 52.50    0
 55.00    5 *
 57.50    7 *
 60.00   11 *
 62.50   22 *
 65.00   37 *
 67.50   70 **
 70.00  149 ****
 72.50  206 *****
 75.00  404 *********
 77.50  640 **************
 80.00 1029 **********************
 82.50 1705 ************************************
 85.00 2836 ************************************************************
 87.50 2685 ********************************************************
 90.00 2779 **********************************************************
 92.50 1090 ***********************
 95.00  297 *******
 97.50   24 *

Boosting spam_cutoff to 0.585 would have saved us from the new false
positive; the historgram isn't fine enough to be sure about the effect on
the f-n rate, though.  The new fp was posted on 12 Sep 2001, and was a
relatively long message from a German fellow expressing sympathy for the
victims of the previous day's terrorist actions.  Of course there was no
relation to c.l.py whatsoever, but it wasn't spam.

Note that the more more-or-less bland words we ignore, the more the ham
distribution spreads out, and the tighter the spam distribution gets:  in
the limit, I expect spam will be reduced to one word <wink>.


From guido@python.org  Sun Sep 22 07:25:16 2002
From: guido@python.org (Guido van Rossum)
Date: Sun, 22 Sep 2002 02:25:16 -0400
Subject: [Spambayes] Moving closer to Gary's ideal
In-Reply-To: Your message of "Sun, 22 Sep 2002 00:14:30 EDT."
             <LNBBLJKPBEHFEDALKOLCGEMMBFAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCGEMMBFAB.tim.one@comcast.net> 
Message-ID: <200209220625.g8M6PI927651@pcp02138704pcs.reston01.va.comcast.net>

> You can tell for sure just by looking at the score histograms and counting
> the dots <wink>; there's no need to change spam_cutoff and then rerun the
> test (spam_cutoff has no effect on the scores computed); I've walked through
> that process in slow motion several times on the list now.

One thing isn't clear to me.  Does a dot at, say, 50.00 mean that
there are X items whose score is between 48.75 and 51.25, or does it
mean those items are between 47.50 and 50.00?

OK, here are my histograms (truncated):

Ham distribution for all in this training set:
* = 27 items
  5.00    1 *
  7.50    0 
 10.00    0 
 12.50    8 *
 15.00   35 **
 17.50   68 ***
 20.00  172 *******
 22.50  342 *************
 25.00  729 ***************************
 27.50 1305 *************************************************
 30.00 1570 ***********************************************************
 32.50 1466 *******************************************************
 35.00 1104 *****************************************
 37.50  634 ************************
 40.00  347 *************
 42.50  213 ********
 45.00  126 *****
 47.50   74 ***
 50.00   43 **
 52.50   14 *
 55.00   10 *
 57.50    8 *
 60.00    3 *
 62.50    1 *
 65.00    1 *
 67.50    0 
 70.00    0 
 72.50    0 
 75.00    2 *

Spam distribution for all in this training set:
* = 12 items
 45.00   1 *
 47.50   1 *
 50.00   2 *
 52.50   2 *
 55.00   4 *
 57.50  16 **
 60.00  40 ****
 62.50  62 ******
 65.00 107 *********
 67.50 241 *********************
 70.00 452 **************************************
 72.50 719 ************************************************************
 75.00 717 ************************************************************
 77.50 501 ******************************************
 80.00 261 **********************
 82.50  45 ****
 85.00  13 **
 87.50   7 *

What's the ideal cutoff here to compete with Graham?  The last 4
output lines from result.py for that set are:

total unique false pos 40
total unique false neg 204
average fp % 0.0480748377883
average fn % 0.634757552464

For my Robinson run with cutoff = 0.575, they are:

total unique false pos 101
total unique false neg 129
average fp % 0.121612480864
average fn % 0.401042537411

> An observed effect of setting robinson_minimum_prob_strength is to
> increase the separation of the ham and spam means: the ham mean gets
> lower and the spam mean gets higher.  This is what I expected,
> since, unlike as in Graham's scheme, scoring words with neutral
> probability in Gary's scheme drags a score closer to 0.5.  Now
> "drags" sounds pejorative, because that's the way I feel about it --
> I see no value in scoring neutral words at all in this task.  Gary
> disagrees, but allows that it's more of a "purist" issue than a
> pragmatic one.  However, something we agree 100% on is that
> measuring the effects of *principled* changes gets much harder if
> pragmatic hacks muddy the mathematical basis of a scheme.  If Gary's
> scheme proves to be as good as, but no better than, our current
> scheme, I'd still switch to it for this reason: it has far fewer
> "mystery knobs" to confuse the underlying issues.
> 
> > (Hm, have you computed mean and standard deviation?)
> 
> Nope.  What would you do with them if I did (they're easy enough to
> compute and display if there's a point to it)?  You can get an
> excellent feel for them by looking at the histograms (which reveal
> far more than a pair of (mean, sdev) numbers anyway).

Well, I for one, couldn't decide by staring at the two histograms
above which one to call "fatter".

--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido@python.org  Sun Sep 22 07:32:02 2002
From: guido@python.org (Guido van Rossum)
Date: Sun, 22 Sep 2002 02:32:02 -0400
Subject: [Spambayes] RE: Further Improvement 2
In-Reply-To: Your message of "Sun, 22 Sep 2002 01:24:43 EDT."
             <LNBBLJKPBEHFEDALKOLCOENABFAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCOENABFAB.tim.one@comcast.net> 
Message-ID: <200209220632.g8M6W3e27694@pcp02138704pcs.reston01.va.comcast.net>

> So I expect it would be an easy project to get this code working
> under 2.2, but for someone more conversant with Python than you are
> so far.  It's not something I'm going to do, though -- anyone else?

I can do this in my (and Orlijn's :-) sleep, as long as I can drop
copies of sets.py and heapq.py into the distribution, and as long as
the rest of the developers in the project from then on stop using
2.3-isms.

I think dropping heapq.py and sets.py should be fine w.r.t. the new
LICENSE.txt file -- they both are new code so the old license stack
doesn't apply.  I'm sure the original authors won't mind that their
code is being redistributed under a "different" license. :-)

--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.one@comcast.net  Sun Sep 22 07:39:51 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 22 Sep 2002 02:39:51 -0400
Subject: [Spambayes] Use for gray area in scoring range
In-Reply-To: <200209220616.g8M6G2D27626@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCGENDBFAB.tim.one@comcast.net>

[Guido]
> ...
> I haven't found the right setting for me yet.  0.575 did better than
> 0.55 but still much worse on the fps than Graham.

You don't have to run more tests for this:  the effect of any particular
setting can be determined from looking at your score histograms (provided
you stick to settings at the bucket boundaries; you can set nbuckets in the
options to a larger number to get finer-grained histograms).

Note too that setting the new robinson_minimum_prob_strength option has
given a strong reduction in f-n rate for the two people who have reported on
it.

>> It's great to have the knob, but it's sensitive, and so far we've no
>> idea how to choose it short of trial and error (it's easy to choose
>> if you've got the score histograms to stare at, but end users
>> won't).

> Plus, we don't know why it's not 50, right?

I covered that in a later message:  no, but the most likely reason seems
simply that the spam mean is farther from 50 than the ham mean.

> Might that have to do with the spam/ham ration?  I've got 83 hams for
> each 32 spams.

Run an experiment!  At least timcv.py has cmdline options to allow running
on random subsets, and allows specifying different counts for the ham
subsets than the spam subsets.  This allows testing on any ratio you like.

I expect that playing with robinson_probability_a will also affect this.
Setting that to 0 will make words with very low corpus counts act much more
like they did under the all-default scheme (i.e., it will give them extreme
probabilities, which can help if, e.g., you've only got one message from
your favorite porn vendor in your training set).


From eppstein@ics.uci.edu  Sun Sep 22 07:47:47 2002
From: eppstein@ics.uci.edu (David Eppstein)
Date: Sat, 21 Sep 2002 23:47:47 -0700
Subject: [Spambayes] Re: Practical applications
References: <3D8C92A4.6679.387A2695@localhost>
	<LNBBLJKPBEHFEDALKOLCMEMPBFAB.tim.one@comcast.net>
Message-ID: <eppstein-100359.23474721092002@main.gmane.org>

In article <LNBBLJKPBEHFEDALKOLCMEMPBFAB.tim.one@comcast.net>,
 Tim Peters <tim.one@comcast.net> wrote:

> > Or, I could be sued for patent infringement   :-(
> 
> Note that MS's patent is on Vector Support Machine anti-spam technology.
> We're not anywhere close to that, and Bayesian learning techniques have been
> in the open literature for more than 40 years.

That would be Support Vector Machines.  They are one of the standard 
techniques for classification problems such as this one.  I haven't 
tried looking at the patent, and I'm not a classification expert (we 
have other people in my dept. who are), but the only possible new idea 
(which I doubt is new to MS, and it's very obvious) would be the same 
one you're using here: try using mildly sophisticated but standard 
classification algorithms instead of ad-hoc pattern matching in the 
specific area of spam detection.

Speaking of which, has anyone tried boosting?  You should be able to 
plug it in on top of other methods such as the one you're doing here.

-- 
David Eppstein       UC Irvine Dept. of Information & Computer Science
eppstein@ics.uci.edu http://www.ics.uci.edu/~eppstein/



From guido@python.org  Sun Sep 22 08:06:01 2002
From: guido@python.org (Guido van Rossum)
Date: Sun, 22 Sep 2002 03:06:01 -0400
Subject: [Spambayes] RE: Further Improvement 2
In-Reply-To: Your message of "Sun, 22 Sep 2002 02:32:02 EDT."
	<200209220632.g8M6W3e27694@pcp02138704pcs.reston01.va.comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCOENABFAB.tim.one@comcast.net>  
	<200209220632.g8M6W3e27694@pcp02138704pcs.reston01.va.comcast.net> 
Message-ID: <200209220706.g8M765Q27830@pcp02138704pcs.reston01.va.comcast.net>

> > So I expect it would be an easy project to get this code working
> > under 2.2, but for someone more conversant with Python than you are
> > so far.  It's not something I'm going to do, though -- anyone else?
> 
> I can do this in my (and Orlijn's :-) sleep, as long as I can drop
> copies of sets.py and heapq.py into the distribution, and as long as
> the rest of the developers in the project from then on stop using
> 2.3-isms.

Done now.  The only thing that required being awake was figuring out
what to use instead of get_content_[main]type(), which are new in the
2.3 email package.

This requires Python 2.2.1 -- the original Python 2.2 won't do, as it
doesn't have True and False, and there are tons of these.  (I didn't
go the way of a "compatibility" module -- it didn't seem worth it,
since the majority of changes were adding "from __future__ import
generators" and you can't import that from another module.)

Caveat: I haven't tried every tool with Python 2.2.1.  But I grepped
every file for yield, enumerate and get_content_, and I ran timcv.py
and hammie.py, which are the ones I actually know how to use.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.one@comcast.net  Sun Sep 22 08:26:14 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 22 Sep 2002 03:26:14 -0400
Subject: [Spambayes] Moving closer to Gary's ideal
In-Reply-To: <200209220625.g8M6PI927651@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCIENFBFAB.tim.one@comcast.net>

[Guido]
> One thing isn't clear to me.  Does a dot at, say, 50.00 mean that
> there are X items whose score is between 48.75 and 51.25, or does it
> mean those items are between 47.50 and 50.00?

That the first bucket is labelled 0 and the last (if you're using 40
buckets) 97.5 is A Clue:  in your example, it means an item in the 50 bucket
has a score S satisfying

    50.00 <= S < 52.50

(again assuming you're using 40 buckets).

An exception is made for the final bucket, which includes all scores of 100
(I've never seen one of those under Gary's scheme, though).

Since you're a Python guy <wink>, except for that endcase the rest is easy
to remember via picturing

    bucketcount[int(score * nbuckets)] += 1

> OK, here are my histograms (truncated):
>
> Ham distribution for all in this training set:
> * = 27 items
>   5.00    1 *
>   7.50    0
>  10.00    0
>  12.50    8 *
>  15.00   35 **
>  17.50   68 ***
>  20.00  172 *******
>  22.50  342 *************
>  25.00  729 ***************************
>  27.50 1305 *************************************************
>  30.00 1570 ***********************************************************
>  32.50 1466 *******************************************************
>  35.00 1104 *****************************************
>  37.50  634 ************************
>  40.00  347 *************
>  42.50  213 ********
>  45.00  126 *****
>  47.50   74 ***
>  50.00   43 **
>  52.50   14 *
>  55.00   10 *
>  57.50    8 *
>  60.00    3 *
>  62.50    1 *
>  65.00    1 *
>  67.50    0
>  70.00    0
>  72.50    0
>  75.00    2 *

So if you set your spam_cutoff to 0.75, you would have 2 false positives.
Ditto for smaller settings until you get to 0.65, at which point you'd have
1+2 = 3 f-p; drop it to 0.625 and you'd pick up one more; drop to 0.6 and
you'd get 3 more; etc.


> Spam distribution for all in this training set:
> * = 12 items
>  45.00   1 *
>  47.50   1 *
>  50.00   2 *
>  52.50   2 *
>  55.00   4 *
>  57.50  16 **
>  60.00  40 ****
>  62.50  62 ******
>  65.00 107 *********
>  67.50 241 *********************
>  70.00 452 **************************************
>  72.50 719 ************************************************************
>  75.00 717 ************************************************************
>  77.50 501 ******************************************
>  80.00 261 **********************
>  82.50  45 ****
>  85.00  13 **
>  87.50   7 *
>
> What's the ideal cutoff here to compete with Graham?
>
> The last 4 output lines from result.py for that set are:
>
> total unique false pos 40
> total unique false neg 204
> average fp % 0.0480748377883
> average fn % 0.634757552464
>
> For my Robinson run with cutoff = 0.575, they are:
>
> total unique false pos 101
> total unique false neg 129
> average fp % 0.121612480864
> average fn % 0.401042537411

I can't answer the question:  you showed a historgram for a single training
set above ("Spam distribution for all in this training set").  To match the
figures you gave just above, you have to look at the aggregate "all runs"
histograms at the end of the output.

Here's the relevant slice of your ham histo:

>  57.50    8 *
>  60.00    3 *
>  62.50    1 *
>  65.00    1 *
>  67.50    0
>  70.00    0
>  72.50    0
>  75.00    2 *

You had 15 (8+3+1+1+0+0+0+2) ham total that scored at or above 0.575 *in
this particular sub-run*.  If you're running a 10-fold cross validation,
there are 9 other per-sub-run histogram pairs I haven't seen, plus 10 other
pairs that only make good sense when running timtest.py, and the final
histogram pair at the end that adds them all up.

> ...
> Well, I for one, couldn't decide by staring at the two histograms
> above which one to call "fatter".

It was your ham, but again I ask what use you would have for the mean and
sdev if I bothered to compute and display them?


From tim.one@comcast.net  Sun Sep 22 08:35:03 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 22 Sep 2002 03:35:03 -0400
Subject: [Spambayes] Re: Practical applications
In-Reply-To: <eppstein-100359.23474721092002@main.gmane.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCCENGBFAB.tim.one@comcast.net>

[David Eppstein]
> That would be Support Vector Machines.

So they are!

> They are one of the standard techniques for classification problems
> such as this one.  I haven't tried looking at the patent, and I'm not
> a classification expert (we  have other people in my dept. who are),

So why aren't they here helping <wink>?

> but the only possible new idea  (which I doubt is new to MS, and it's
> very obvious) would be the same one you're using here: try using mildly
> sophisticated but standard classification algorithms instead of ad-hoc
> pattern matching in the specific area of spam detection.

It's not new with MS; there are at least a dozen papers in the open
literature specifically applying Bayesian classification to the spam
classification task.

> Speaking of which, has anyone tried boosting?  You should be able to
> plug it in on top of other methods such as the one you're doing here.

Nobody in this specific project has -- give it a shot!


From guido@python.org  Sun Sep 22 08:35:07 2002
From: guido@python.org (Guido van Rossum)
Date: Sun, 22 Sep 2002 03:35:07 -0400
Subject: [Spambayes] Re: Practical applications
In-Reply-To: Your message of "Sun, 22 Sep 2002 03:35:03 EDT."
             <LNBBLJKPBEHFEDALKOLCCENGBFAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCCENGBFAB.tim.one@comcast.net> 
Message-ID: <200209220735.g8M7Z8B27892@pcp02138704pcs.reston01.va.comcast.net>

> > Speaking of which, has anyone tried boosting?  You should be able to
> > plug it in on top of other methods such as the one you're doing here.
> 
> Nobody in this specific project has -- give it a shot!

Excuse my ignorance, but what's boosting?

--Guido van Rossum (home page: http://www.python.org/~guido/)

From anthony@interlink.com.au  Sun Sep 22 08:42:40 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Sun, 22 Sep 2002 17:42:40 +1000
Subject: [Spambayes] Practical applications 
In-Reply-To: 
	<200209220046.g8M0k0J27042@pcp02138704pcs.reston01.va.comcast.net> 
Message-ID: <200209220742.g8M7gf025021@localhost.localdomain>


>>> Guido van Rossum wrote
> > 1) I notice that Python is an "Actice Scripting" language? Does that a mean
> > that a Python script can interact with Outlook to do things like add
> > a "Spam!" button, etc.?
> 
> Python can control any Windows application that supports a COM
> interface.  You have to get the win32all set of additions (there's a
> pointer to it on the download pages).  I've never done this, I suppose
> you'd need docs for Outlook.

I think he was asking for something the other way - a way to add
a button/command to Outbreak which runs a python script. I don't know if
that's possible - someone like MarkH may know.

Anthony
-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.


From guido@python.org  Sun Sep 22 08:45:50 2002
From: guido@python.org (Guido van Rossum)
Date: Sun, 22 Sep 2002 03:45:50 -0400
Subject: [Spambayes] Moving closer to Gary's ideal
In-Reply-To: Your message of "Sun, 22 Sep 2002 03:26:14 EDT."
             <LNBBLJKPBEHFEDALKOLCIENFBFAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCIENFBFAB.tim.one@comcast.net> 
Message-ID: <200209220745.g8M7joq27944@pcp02138704pcs.reston01.va.comcast.net>

I asked:
> > What's the ideal cutoff here to compete with Graham?
> >
> > The last 4 output lines from result.py for that set are:
> >
> > total unique false pos 40
> > total unique false neg 204
> > average fp % 0.0480748377883
> > average fn % 0.634757552464
> >
> > For my Robinson run with cutoff = 0.575, they are:
> >
> > total unique false pos 101
> > total unique false neg 129
> > average fp % 0.121612480864
> > average fn % 0.401042537411

but I forgot to include the "full set" histograms.  Here's the ham:

Ham distribution for all runs:
* = 268 items
  5.00     2 *
  7.50     0 
 10.00     4 *
 12.50    64 *
 15.00   262 *
 17.50   748 ***
 20.00  1734 *******
 22.50  3665 **************
 25.00  7662 *****************************
 27.50 12597 ************************************************
 30.00 16039 ************************************************************
 32.50 14821 ********************************************************
 35.00 10542 ****************************************
 37.50  6441 *************************
 40.00  3593 **************
 42.50  2073 ********
 45.00  1177 *****
 47.50   717 ***
 50.00   421 **
 52.50   197 *
 55.00    96 *
 57.50    60 *
 60.00    16 *
 62.50    10 *
 65.00     3 *
 67.50     1 *
 70.00     1 *
 72.50     2 *
 75.00     8 *

So to match the 40 fp's from Graham's scheme, I'd need to set the
cutoff to 0.60; that would give me 41 fp's here (16+10+3+1+1+2+8).
(If a message scores *exactly* the cutoff, is it spam or ham?)

Spam distribution for all runs:
* = 124 items
 37.50    1 *
 40.00    0 
 42.50    0 
 45.00    4 *
 47.50   12 *
 50.00   22 *
 52.50   20 *
 55.00   70 *
 57.50  153 **
 60.00  346 ***
 62.50  664 ******
 65.00 1334 ***********
 67.50 2503 *********************
 70.00 4303 ***********************************
 72.50 7136 **********************************************************
 75.00 7385 ************************************************************
 77.50 4918 ****************************************
 80.00 2481 *********************
 82.50  513 *****
 85.00  116 *
 87.50   73 *
 90.00   14 *

So with a cutoff of 0.60, this would give me 1+4+12+22+20+70+153 = 282
fn's.  That's still considerably worse than Graham's 204.

I'm going to have to look at the fp's and fn's to see if there are
real spams hiding in the ham, and vice versa.  I did notice that many
fp's were very spammish automated postings that I have specifically
signed up for, like our building's announcements, product newsletters,
and so on.  I haven't looked at the fn's.

> > Well, I for one, couldn't decide by staring at the two histograms
> > above which one to call "fatter".
> 
> It was your ham, but again I ask what use you would have for the
> mean and sdev if I bothered to compute and display them?

I was interested in which of the two bell curves was "fatter".  The
sdev tells me this.  But I'm not sure that the relative fatness of the
curves is a good measure -- it's the overlapping tails.  I suppose
there's a statistical measure for how "normal" a tail is, but I'm not
sure that's relevant given that we can easily see the overlap in the
histograms.

--Guido van Rossum (home page: http://www.python.org/~guido/)



From tim.one@comcast.net  Sun Sep 22 08:50:17 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 22 Sep 2002 03:50:17 -0400
Subject: [Spambayes] Re: Practical applications
In-Reply-To: <200209220735.g8M7Z8B27892@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCCENHBFAB.tim.one@comcast.net>

[Guido]
> Excuse my ignorance, but what's boosting?

google for

    boosting bagging

Searching for either alone won't yield happiness <wink>.

From tim.one@comcast.net  Sun Sep 22 10:15:51 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 22 Sep 2002 05:15:51 -0400
Subject: [Spambayes] Moving closer to Gary's ideal
In-Reply-To: <200209220745.g8M7joq27944@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCMENKBFAB.tim.one@comcast.net>

[Guido]
> ...
> Ham distribution for all runs:
> * = 268 items
>   5.00     2 *
>   7.50     0
>  10.00     4 *
>  12.50    64 *
>  15.00   262 *
>  17.50   748 ***
>  20.00  1734 *******
>  22.50  3665 **************
>  25.00  7662 *****************************
>  27.50 12597 ************************************************
>  30.00 16039 ************************************************************
>  32.50 14821 ********************************************************
>  35.00 10542 ****************************************
>  37.50  6441 *************************
>  40.00  3593 **************
>  42.50  2073 ********
>  45.00  1177 *****
>  47.50   717 ***
>  50.00   421 **
>  52.50   197 *
>  55.00    96 *
>  57.50    60 *
>  60.00    16 *
>  62.50    10 *
>  65.00     3 *
>  67.50     1 *
>  70.00     1 *
>  72.50     2 *
>  75.00     8 *
>
> So to match the 40 fp's from Graham's scheme, I'd need to set the
> cutoff to 0.60; that would give me 41 fp's here (16+10+3+1+1+2+8).

Right!  BTW, that's a huge test run -- certainly the largest to date.

> (If a message scores *exactly* the cutoff, is it spam or ham?)

>From Options.py:

    # A message is considered spam iff it scores greater than spam_cutoff.

Since I wrote that, you can trust that "greater than" does not mean "greater
than or equal to" <wink>.

Especially under Gary's scheme, the odds of scoring exactly at a cutoff
point are vanishingly small.  It *used* to be that a msg the email package
couldn't parse scored exactly 0.50, but we fall back to the raw text now.
If the raw text is empty, it will still get exactly 0.50.

> Spam distribution for all runs:
> * = 124 items
>  37.50    1 *
>  40.00    0
>  42.50    0
>  45.00    4 *
>  47.50   12 *
>  50.00   22 *
>  52.50   20 *
>  55.00   70 *
>  57.50  153 **
>  60.00  346 ***
>  62.50  664 ******
>  65.00 1334 ***********
>  67.50 2503 *********************
>  70.00 4303 ***********************************
>  72.50 7136 **********************************************************
>  75.00 7385 ************************************************************
>  77.50 4918 ****************************************
>  80.00 2481 *********************
>  82.50  513 *****
>  85.00  116 *
>  87.50   73 *
>  90.00   14 *
>
> So with a cutoff of 0.60, this would give me 1+4+12+22+20+70+153 = 282
> fn's.  That's still considerably worse than Graham's 204.

Indeed it is.  Note that I spent an enormous amount of time tweaking an
enormous number of things to get supernatural results out of Graham's
scheme, though, and none of those things have been revisited under Gary's
scheme yet.  The first thing I thought to try didn't make sense to try under
Graham:  leaving close-to-0.5 guys out of the scoring, and that's been
reported to alleviate the very problem you're seeing (higher f-n rate).
There may well be a better way to get there, but we won't know until people
*try* other things.  In particular, the treatment of rare corpus words is
very different under the default "a" parameter of 1.0 in Gary's scheme than
it is under the default all-Graham scheme, and as explained recently in a
different msg, we already know for certain that the treatment of rare words
has a significant effect on error rates.

> I'm going to have to look at the fp's and fn's to see if there are
> real spams hiding in the ham, and vice versa.

If there aren't, you'll be the first tester ever not to discover some.  For
example, I've found 3 hams in BruceG's spam collection so far, and I believe
you're using that too (but much more of it).

BTW, it's my belief that this all works *best* if the ratio of ham to spam
trained on matches your real-life inbox ratio.  But it's a peculiar property
of the cross-fold validation framework that the ratio of ham to spam
predicted on exactly matches the ratio of ham to spam trained on, so
inbalance in ham-vs-spam numbers can't be blamed for poor *test* results
(although *both* your error rates are under 1%, and I doubt any project save
ours would call that "poor" <wink>).

> I did notice that many fp's were very spammish automated postings
> that I have specifically signed up for, like our building's
> announcements, product newsletters, and so on.  I haven't looked at
> the fn's.

I expect these are your moral equivalents to the conference announcements in
my c.l.py ham, except worse.  However, I expect you have more cause for
optimism about those:  you (like me) are running a crippled version of the
algorithm because of your mixed-source corpora.  The headers we're ignoring
are bound to have strong clues about the *senders* of the spammish stuff
you've signed up for.

> I was interested in which of the two bell curves was "fatter".  The
> sdev tells me this.

Enough already <wink>.  Next time you do a cvs up, you'll get a nice
printout of total data points, mean, and sdev with each histogram.

> But I'm not sure that the relative fatness of the curves is a good
> measure

It depends on what you're trying to figure out, of course.

> -- it's the overlapping tails.

For "middle ground" purposes, absolutely.  Wherever they overlap, changing
spam_cutoff necessarily hurts one of the error rates and necessarily helps
the other.

> I suppose there's a statistical measure for how "normal" a tail is,
> but I'm not sure that's relevant given that we can easily see the
> overlap in the histograms.

It would be nicer to display them side-by-side, but you'd have to learn how
to use more of your available screen area then <wink>.


From tim.one@comcast.net  Sun Sep 22 10:49:01 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 22 Sep 2002 05:49:01 -0400
Subject: [Spambayes] A curious max_discrimators result
In-Reply-To: <LNBBLJKPBEHFEDALKOLCIENFBFAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCIENMBFAB.tim.one@comcast.net>

My usual large test here.  "Before":

"""
[Classifier]
use_robinson_probability: True
use_robinson_combining: True
max_discriminators: 1500

[TestDriver]
spam_cutoff: 0.575
"""

"After" is identical except for changing max_discriminator to 16 (this had
never been tried before (by me) with both use_robinson_xyz options True):

"""
false positive percentages
    0.000  0.050  lost  +(was 0)
    0.000  0.050  lost  +(was 0)
    0.000  0.050  lost  +(was 0)
    0.000  0.000  tied
    0.050  0.100  lost  +100.00%
    0.000  0.000  tied
    0.000  0.000  tied
    0.050  0.050  tied
    0.000  0.000  tied
    0.050  0.050  tied

won   0 times
tied  6 times
lost  4 times

total unique fp went from 3 to 7 lost  +133.33%
mean fp % went from 0.015 to 0.035 lost  +133.33%

false negative percentages
    0.071  0.143  lost  +101.41%
    0.286  0.214  won    -25.17%
    0.071  0.000  won   -100.00%
    0.214  0.143  won    -33.18%
    0.143  0.214  lost   +49.65%
    0.214  0.071  won    -66.82%
    0.214  0.143  won    -33.18%
    0.357  0.071  won    -80.11%
    0.357  0.071  won    -80.11%
    0.000  0.000  tied

won   7 times
tied  1 times
lost  2 times

total unique fn went from 27 to 15 won    -44.44%
mean fn % went from 0.192857142857 to 0.107142857143 won    -44.44%
"""

I'm starting to believe that any tweak whatsoever will reduce the f-n rate.
BTW, this also tells me that I'm getting almost all the good there is to get
(in my data) out of the 16 most extreme clues.  OTOH, Neil reported that
reducing max_discriminators (from 1,500) hurt him, but didn't quantify it
(like how much it hurt, or how much he reduced it).

Further result:  if I raised spam_cutoff to 0.60 in the "after" run, 6(!) of
the fps would go away, and 15 f-n would be gained, leaving a grand total of
1 fp and 30 fn.  Bizarre:  this would let both the Nigerian scam quote(!)
and the lady with the obnoxious sig through.  The only ham it would stop is
this, which I've never posted before:

"""
Data/Ham/Set2/4059.txt
prob = 0.609747546095
prob('header:Organization:1') = 0.0124308
prob('script') = 0.0167547
prob('thanks.') = 0.119632
prob('mark') = 0.133815
prob('notification') = 0.885737
prob('protection') = 0.897292
prob('traffic') = 0.909737
prob('anti') = 0.909836
prob('information') = 0.911857
prob('please') = 0.929992
prob('address') = 0.946832
prob('age') = 0.951843
prob('email') = 0.960494
prob('davidson') = 0.964517
prob('verification') = 0.975702
prob('webmasters') = 0.981481

From: huy4huy@cs.com (HUY4HUY)
Newsgroups: comp.lang.python
Subject: need scripts...help
Lines: 13
NNTP-Posting-Host: ladder06.news.cs.com
X-Admin: news@cs.com
Date: 2 Jun 1999 18:47:40 GMT
Organization: CompuServe (http://www.compuserve.com/)
Message-ID: <19990602144740.17875.00000120@ng-cf1.news.cs.com>
Path:
news!uunet!ffx.uu.net!newsfeed.us.ibm.net!ibm.net!news-spur1.maxwell.syr.edu
!news.maxwell.syr.edu!newsfeed.cwix.com!152.163.199.19!portc03.blue.aol.com!
portc03.blue.cs.com!audrey03.news.cs.com!not-for-mail
Xref: news comp.lang.python:64176
To: python-list@python.org

I need a program that would automate an age verification system, I need the
script to handle all aspects.

It should have easy sign up for webmasters and surfers, anti cheater devices
as
well as the real time monitoring of all system activities, email
notification
to the webmaster joining to the admin, logs all traffic trough the system
and
gathers email addresses, built in password protection for admin area, no
SSL's,
no htaccess...this is an example of the format I need.  I will appreciate
it,
if anyone can help me out......thanks
--------------------------------------------------------------------
Mark Davidson             inert28@hotmail.com
please inform with the information to the email address above, thanks.
"""


From glouis@dynamicro.on.ca  Sun Sep 22 13:36:48 2002
From: glouis@dynamicro.on.ca (Greg Louis)
Date: Sun, 22 Sep 2002 08:36:48 -0400
Subject: [Spambayes] Moving closer to Gary's ideal
In-Reply-To: <LNBBLJKPBEHFEDALKOLCOELMBFAB.tim.one@comcast.net>
References: <B9B1E441.16436%grobinson@transpose.com>
	<LNBBLJKPBEHFEDALKOLCOELMBFAB.tim.one@comcast.net>
Message-ID: <20020922123648.GB1274@athame.dynamicro.on.ca>

On 20020921 (Sat) at 1523:16 -0400, Tim Peters wrote:

> Because g(w) is easy to implement, I'm going to try it.  Our source code
> here is open for all to stare at, and that gives me much more confidence
> that we're not suffering from "stupid bugs" when we try something -- and
> it's very easy to make a mistake.  I won't believe that g(w) hurts or helps
> until my tests tell me (no offense to GregL intended -- I simply have no
> idea what his code does).

No offense taken.  The more independent tests, the better, IMHO, until
we have a clear outcome.  In my hands (after taking lots of advice from
Gary) g(w) seemed, in one trial, not to improve discrimination
significantly, but that doesn't constitute a final conclusion.

The code I wrote to try g(w) is a quick ugly hack that runs
ultra-slowly, and I don't want to spend time to polish it unless and
until it actually holds out some promise of better discrimination.  But
it does rank the words in the right order, and it does pick the right
rank out of the list for each word; the rest of the calculations are
the same as in the patch I posted on the bogofilter mailing list
yesterday (the second version of the patch).

What I have done is a close approximation to g(w) with all nonspam
words ranked in the lower half of the scale and all spam words ranked
in the upper half (even though there are more nonspam words than spam
ones) -- this is, I think, what Gary advised in an exchange of emails
on Friday.  I am making no claim that it works better or less well than
f(w), just that a very preliminary test didn't seem to make a lot of
difference.  I certainly don't want anyone to suppose that I've
"proven" anything about g(w) at this point.

-- 
| G r e g  L o u i s          | gpg public key:      |
|   http://www.bgl.nu/~glouis |   finger greg@bgl.nu |

From barry@python.org  Sun Sep 22 15:21:47 2002
From: barry@python.org (Barry A. Warsaw)
Date: Sun, 22 Sep 2002 10:21:47 -0400
Subject: [Spambayes] RE: Further Improvement 2
References: <LNBBLJKPBEHFEDALKOLCOENABFAB.tim.one@comcast.net>
	<200209220632.g8M6W3e27694@pcp02138704pcs.reston01.va.comcast.net>
	<200209220706.g8M765Q27830@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <15757.53755.736942.134794@anthem.wooz.org>


>>>>> "GvR" == Guido van Rossum <guido@python.org> writes:

    GvR> Done now.  The only thing that required being awake was
    GvR> figuring out what to use instead of get_content_[main]type(),
    GvR> which are new in the 2.3 email package.

Note that Python 2.2.1's email package is different than Python 2.3's
in many ways, some of them perhaps significant to spambayes.  For
example, 2.3's parser has a lax mode so that malformed messages won't
just crap out with an exception.

get_content_type() and friends is another useful change.  According to
the RFCs, all messages/parts have a content type, which may be an
implicit default.  Python 2.2's email package didn't support this by
2.3's does.

You might want to grab email 2.3.1 from the mimelib project on
SourceForge.  It's identical to what's in Python 2.3's cvs right now.

http://sourceforge.net/project/showfiles.php?group_id=25568&release_id=110035

-Barry

From bkc@murkworks.com  Sun Sep 22 16:08:12 2002
From: bkc@murkworks.com (Brad Clements)
Date: Sun, 22 Sep 2002 11:08:12 -0400
Subject: [Spambayes] My first run (long)
Message-ID: <3D8DA4AC.23050.3CA8A79A@localhost>

Question: I seem to have \r\n line endings.. does that matter or should I strip them?


Description of my corpus:

All email has been received by me directly, either through my primary email address, or 
sent to various aliases (info@murkworks.com) or from email sent to employees I used 
to have.

Spam has been collected over the past year or two.. Although I have over 33,000 spam 
messages, I only took the "most recent" 13,000 to balance my ham corpus.

HAM messages are those currently in my inbox, messages requesting tech support for 
our various products, messages to mailing lists such as ftp, ntp, smtp protocol 
discussion lists, various IETF lists, "commercial-like" email from Ingram Micro, Novell, 
HP, confirmation messages from any company I purchased software from, etc. Some 
HAM is rather old .. 1995 or so.


With a cvs update from 2 hours ago..:

[Classifier]
use_robinson_probability: True
use_robinson_combining: True
max_discriminators: 1500
robinson_minimum_prob_strength: 0.1
[TestDriver]
spam_cutoff: 0.575

checking the false negatives, I have a few spams that were bounced to postmaster.. 

My false positives all seem to be "marketing" type stuff from Ingram-Micro, Novell, HP, 
E-Trade, etc.

I need to tune up my corpus...

Ham distribution for all runs:
13000 items; mean 30.27; sample sdev 7.58
* = 34 items
  0.00    0
  2.50    7 *
  5.00    7 *
  7.50   13 *
 10.00   54 **
 12.50   87 ***
 15.00  217 *******
 17.50  369 ***********
 20.00  798 ************************ 
 22.50 1367 *****************************************
 25.00 1949 **********************************************************
 27.50 2029 ************************************************************
 30.00 1861 *******************************************************
 32.50 1378 *****************************************
 35.00  916 ***************************
 37.50  733 **********************
 40.00  462 **************
 42.50  219 *******
 45.00  162 *****
 47.50  138 *****
 50.00   75 ***
 52.50   61 **
 55.00   35 **
 57.50   30 *
 60.00   12 *
 62.50    9 *
 65.00    4 *
 67.50    3 *
 70.00    0
 72.50    3 *
 75.00    2 * <- is spam
 77.50    0
 80.00    0

The .6250 spams seem to be like this:

subject: BRAD CLEMENTS, it's time to renew your subscription to EDN - Second
	Notice

(hey, that really is ham)

My Mother scored a .58, so did Borland with:

> Time is running out - Save 15% on New Delphi 7

Verisign/Netsol scores 0.589

>  Confirmation of murkworks.com renewal order

United Airlines, 0.67

> Welcome to United Connection



Spam distribution for all runs:
13000 items; mean 79.08; sample sdev 8.12
* = 38 items
  0.00    0
  2.50    0
  5.00    0
  7.50    0
 10.00    0
 12.50    0
 15.00    0
 17.50    2 *
 20.00    0
 22.50    2 *
 25.00    1 *
 27.50    2 *
 30.00    2 *
 32.50    2 *
 35.00    4 *
 37.50   49 **
 40.00   27 *
 42.50    6 *
 45.00   16 *
 47.50   19 *
 50.00   23 *
 52.50   33 *
 55.00   67 **
 57.50   92 ***
 60.00  169 *****
 62.50  232 *******
 65.00  349 **********
 67.50  502 **************
 70.00  695 *******************
 72.50  714 *******************
 75.00 1149 *******************************
 77.50 1572 ******************************************
 80.00 2179 **********************************************************
 82.50 2271 ************************************************************
 85.00 1722 **********************************************
 87.50  816 **********************
 90.00  278 ********
 92.50    5 *
 95.00    0
 97.50    0

How do I get timcv.py to dump the score of false negatives?

Here's a sample:

----

From: James <pmchuong0806@webmail.winona.edu>
To: Undisclosed.Recipients@anvil.murkworks.com
Subject: I've been so lazy
Sender: James <pmchuong0806@webmail.winona.edu>
Mime-Version: 1.0
Content-Type: text/plain; charset="iso-8859-1"
Date: Sun, 11 Aug 2002 23:04:49 -0500
X-Mailer: Microsoft Outlook Express 5.50.4133.2400

Hey how are you doing?

I've been so lazy this summer, I just got back from a fishing trip (3rd time this month).
 
And my girlfriend is wondering how I can afford to vacation all the time. "Where is all 
your money is coming from?  Are you with the mob?"  She asks.  "I'm not doing 
anything illegal", I said.  I just work about 30 minutes a week and within 60 days my 
check was $1,143!  

I told her it's this automated thing on the internet that works worldwide, I don't really 
know much about it, I just know when people sign up for free with no obligation, some 
how the system does all the work of signing people and paying me.  

It's duplication that makes it successful, they tell me.  And the best part it didn't cost me 
anything for me to be in the system.  Once there are people under me thru spill-overs, I 
upgraded.  I can't believe it cost me only $10 a month to earn $2,000 a week.

Well, if you're interested, let me know by replying to me with the subject: 
INTERESTED.  I check my messages 2 -3 times a week, so you should get something 
from me within 72 hours.  If you're not interested, ignore this message or reply with the 
subject: REMOVE to remove yourself.

Either way, I really don't care.  I know the system works because I get paid every week 
on time.  And that's good enough for me.


James
-----

    best discriminators:
        'only' 3048 0.63766
        'skip:r 10' 3057 0.765613
        'content-type:multipart/alternative' 3081 0.8886
        'offer' 3086 0.961576
        'what' 3253 0.366849
        'message' 3274 0.638722
        'now' 3370 0.676764
        'because' 3641 0.804518
        'but' 3711 0.315054
        'unsubscribe' 3737 0.960647
        'more' 3802 0.645036
        'receive' 3861 0.928872
        'new' 3958 0.603993
        'free' 4467 0.891097
        'header:Mime-Version:1' 4519 0.663075
        'url:gif' 4836 0.95745
        'one' 5274 0.697623
        'header:Reply-To:1' 5291 0.795601
        'url:murkworks' 5302 0.77701
        'get' 5611 0.638269
        'our' 5782 0.6929
        'please' 5845 0.708737
        'all' 6030 0.613003
        'here' 6347 0.879113
        'email' 6373 0.811837
        'click' 6628 0.943325
        'content-type:text/html' 6988 0.944668
        'header:Received:4' 8159 0.639928
        'content-type:text/plain' 8750 0.36458
        'x-mailer:none' 9011 0.631889


-> Training on Data/Ham/Set2-10 & Data/Spam/Set2-10 ... 11700 hams & 11700 
spams
-> Predicting Data/Ham/Set1 & Data/Spam/Set1 ...
-> <stat> tested 1300 hams & 1300 spams against 11700 hams & 11700 spams
-> <stat> false positive %: 0.692307692308
-> <stat> false negative %: 1.69230769231
      0.692   1.692
-> <stat> 9 new false positives
-> <stat> 22 new false negatives
-> Training on Data/Ham/Set1 & Data/Spam/Set1 ... 1300 hams & 1300 spams
-> Forgetting Data/Ham/Set2 & Data/Spam/Set2 ... 1300 hams & 1300 spams
-> Predicting Data/Ham/Set2 & Data/Spam/Set2 ...
-> <stat> tested 1300 hams & 1300 spams against 11700 hams & 11700 spams
-> <stat> false positive %: 0.384615384615
-> <stat> false negative %: 2.46153846154
      0.385   2.462
-> <stat> 5 new false positives
-> <stat> 32 new false negatives
-> Training on Data/Ham/Set2 & Data/Spam/Set2 ... 1300 hams & 1300 spams
-> Forgetting Data/Ham/Set3 & Data/Spam/Set3 ... 1300 hams & 1300 spams
-> Predicting Data/Ham/Set3 & Data/Spam/Set3 ...
-> <stat> tested 1300 hams & 1300 spams against 11700 hams & 11700 spams
-> <stat> false positive %: 0.384615384615
-> <stat> false negative %: 2.15384615385
      0.385   2.154
-> <stat> 5 new false positives
-> <stat> 28 new false negatives
-> Training on Data/Ham/Set3 & Data/Spam/Set3 ... 1300 hams & 1300 spams
-> Forgetting Data/Ham/Set4 & Data/Spam/Set4 ... 1300 hams & 1300 spams
-> Predicting Data/Ham/Set4 & Data/Spam/Set4 ...
-> <stat> tested 1300 hams & 1300 spams against 11700 hams & 11700 spams
-> <stat> false positive %: 0.692307692308
-> <stat> false negative %: 1.92307692308
      0.692   1.923
-> <stat> 9 new false positives
-> <stat> 25 new false negatives
-> Training on Data/Ham/Set4 & Data/Spam/Set4 ... 1300 hams & 1300 spams
-> Forgetting Data/Ham/Set5 & Data/Spam/Set5 ... 1300 hams & 1300 spams
-> Predicting Data/Ham/Set5 & Data/Spam/Set5 ...
-> <stat> tested 1300 hams & 1300 spams against 11700 hams & 11700 spams
-> <stat> false positive %: 0.461538461538
-> <stat> false negative %: 1.84615384615
      0.462   1.846
-> <stat> 6 new false positives
-> <stat> 24 new false negatives
-> Training on Data/Ham/Set5 & Data/Spam/Set5 ... 1300 hams & 1300 spams
-> Forgetting Data/Ham/Set6 & Data/Spam/Set6 ... 1300 hams & 1300 spams
-> Predicting Data/Ham/Set6 & Data/Spam/Set6 ...
-> <stat> tested 1300 hams & 1300 spams against 11700 hams & 11700 spams
-> <stat> false positive %: 0.153846153846
-> <stat> false negative %: 1.69230769231
      0.154   1.692
-> <stat> 2 new false positives
-> <stat> 22 new false negatives
-> Training on Data/Ham/Set6 & Data/Spam/Set6 ... 1300 hams & 1300 spams
-> Forgetting Data/Ham/Set7 & Data/Spam/Set7 ... 1300 hams & 1300 spams
-> Predicting Data/Ham/Set7 & Data/Spam/Set7 ...
-> <stat> tested 1300 hams & 1300 spams against 11700 hams & 11700 spams
-> <stat> false positive %: 0.461538461538
-> <stat> false negative %: 1.53846153846
      0.462   1.538
-> <stat> 6 new false positives
-> <stat> 20 new false negatives
-> Training on Data/Ham/Set7 & Data/Spam/Set7 ... 1300 hams & 1300 spams
-> Forgetting Data/Ham/Set8 & Data/Spam/Set8 ... 1300 hams & 1300 spams
-> Predicting Data/Ham/Set8 & Data/Spam/Set8 ...
-> <stat> tested 1300 hams & 1300 spams against 11700 hams & 11700 spams
-> <stat> false positive %: 0.384615384615
-> <stat> false negative %: 2.07692307692
      0.385   2.077
-> <stat> 5 new false positives
-> <stat> 27 new false negatives
-> Training on Data/Ham/Set8 & Data/Spam/Set8 ... 1300 hams & 1300 spams
-> Forgetting Data/Ham/Set9 & Data/Spam/Set9 ... 1300 hams & 1300 spams
-> Predicting Data/Ham/Set9 & Data/Spam/Set9 ...
-> <stat> tested 1300 hams & 1300 spams against 11700 hams & 11700 spams
-> <stat> false positive %: 0.769230769231
-> <stat> false negative %: 2.07692307692
      0.769   2.077
-> <stat> 10 new false positives
-> <stat> 27 new false negatives
-> Training on Data/Ham/Set9 & Data/Spam/Set9 ... 1300 hams & 1300 spams
-> Forgetting Data/Ham/Set10 & Data/Spam/Set10 ... 1300 hams & 1300 spams
-> Predicting Data/Ham/Set10 & Data/Spam/Set10 ...
-> <stat> tested 1300 hams & 1300 spams against 11700 hams & 11700 spams
-> <stat> false positive %: 0.461538461538
-> <stat> false negative %: 2.15384615385
      0.462   2.154
-> <stat> 6 new false positives
-> <stat> 28 new false negatives
total unique false pos 63
total unique false neg 255
average fp % 0.484615384615
average fn % 1.96153846154


Brad Clements,                bkc@murkworks.com   (315)268-1000
http://www.murkworks.com                          (315)268-9812 Fax
AOL-IM: BKClements


From bkc@murkworks.com  Sun Sep 22 16:50:15 2002
From: bkc@murkworks.com (Brad Clements)
Date: Sun, 22 Sep 2002 11:50:15 -0400
Subject: [Spambayes] 2nd run with received lines
Message-ID: <3D8DAE87.8134.3CCF2A5A@localhost>

I added this option:

[Tokenizer]
mine_received_headers: True

and did the run again.

I would like to see the change in stdev per run from one run to the next. I think reducing 
the stdev is also a win. I'll see if I can tweak rates.py and cmp.py to extract this info.

false positive percentages
    0.692  0.538  won    -22.25%
    0.385  0.308  won    -20.00%
    0.385  0.308  won    -20.00%
    0.692  0.538  won    -22.25%
    0.462  0.385  won    -16.67%
    0.154  0.154  tied
    0.462  0.462  tied
    0.385  0.231  won    -40.00%
    0.769  0.538  won    -30.04%
    0.462  0.385  won    -16.67%

won   8 times
tied  2 times
lost  0 times

total unique fp went from 63 to 50 won    -20.63%
mean fp % went from 0.484615384615 to 0.384615384615 won    -20.63%

false negative percentages
    1.692  1.615  won     -4.55%
    2.462  2.385  won     -3.13%
    2.154  2.000  won     -7.15%
    1.923  1.615  won    -16.02%
    1.846  1.769  won     -4.17%
    1.692  1.615  won     -4.55%
    1.538  1.385  won     -9.95%
    2.077  1.846  won    -11.12%
    2.077  2.000  won     -3.71%
    2.154  2.077  won     -3.57%

won  10 times
tied  0 times
lost  0 times

total unique fn went from 255 to 238 won     -6.67%
mean fn % went from 1.96153846154 to 1.83076923077 won     -6.67%



Brad Clements,                bkc@murkworks.com   (315)268-1000
http://www.murkworks.com                          (315)268-9812 Fax
AOL-IM: BKClements


From grobinson@transpose.com  Sun Sep 22 17:34:53 2002
From: grobinson@transpose.com (Gary Robinson)
Date: Sun, 22 Sep 2002 12:34:53 -0400
Subject: [Spambayes] Re: Further Improvement 2
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEMBBFAB.tim.one@comcast.net>
Message-ID: <B9B3696D.164DA%grobinson@transpose.com>

With regard to the

          a + y
f(w) = ------------
       (a / x) + n

calc, Tim Peters says:
 
> What I can't tell you is
> *which* prior distribution it's assuming, or how changing "a" affects its
> shape.  But since we know for sure that getting rid of MINCOUNT made a real
> difference before, we also know for sure that the treatment of rare words is
> important, and that's what "a" is all about.  This is the point where Gary
> unexpectedly discovers he has just enough free time to tell us more about
> the specifics of this distribution <wink>.
> 

OK. 

I started to try to explain it in a way that people without Bayesian stats
in their background could understand but it quickly look like it would take
me more time than I can afford.  So I had to let that goal go -- but I think
that's OK because I will point to another source where a reader can go to
get the parts of the explanation I will leave out.

Note also that I want to change the form of the equation from the one
currently in the essay to a slightly simpler equivalent one. I will change
it in the essay too. The version we've been working with says:

          a + y
f(w) = ------------
       (a / x) + n

but

       (s * x) + y
f(w) = -----------
         s + n

is equivalent and more intuitive.

Moreover, it's hard to use a as a variable name when communicating by text
because it looks like a part of speech! ;)   And "s" has another appropriate
meaning, as we will see below.


If you want to understand this in detail and haven't studied Bayesian stats
please refer to this paper: http://citeseer.nj.nec.com/135897.html, which is
a tuturial by Microsoftee David Heckerman on Bayesian networks. You only
need to read through page 7.

>From this point on I take it for granted that the reader understands as much
Bayesian theory as is communicated by Heckerman in those 7 pages. But even
without that understanding the reader will probably be able to get a very
rough high-level view out of this.

Let us start in an imaginary world where there are exactly as many spams as
nonspams in the training corpus. Then I'll bring it back to the real world
at the end. For now, there are the same number of spams and hams.

We are going to do a test that is the equivalent of the "experiment" of
tossing a coin multiple times to see whether it appears to be biased. There
are n trials. If we were tossing a coin, each coin toss would be a "trial"
and we'd be counting the number of heads. But in our case, the trial is
looking at the next occurrence of "Nigeria" in the training corpus and
seeing whether the email it contains is a spam.

Clearly, it's a binomial experiment: there are two values, yes or no. And
they are independent, at least if you only consider the first appearance of
Nigeria in a given email and don't use the other occurrences in that email
as trials. (So we'll assume we're doing that and bring it back to Grahamian
reality at the end.)

Now, it happens that if you have a binomial experiment and assume a beta
distribution for the prior, you can express the expectation that the nth + 1
trial will come out "yes" as


(s * x) + y
-----------.  (See Heckerman.)
   s + n


A beta distribution is determined by 2 parameters. Heckerman calls them
alpha-h and alpha-t but let's just call them u and v since we can't send an
alpha character in a text email reliably (as far as I know).

The relationship between s, x, u, and v is as follows:

s = u + v
s * x = u

If you want to see the affects of changing s on the beta distribution, just
hold x constant and recalculate u and v, then graph it.

But actually looking at the beta distribution is just esoterica for our
application. The beauty of using s and x rather than u and v in our calcs is
that intuitively, we can see s as the "strength" (hence the use of "s") of
an original assumed expectation for y relative to how strongly we want to
consider our actual collected data. And the x represents that expectation.
And when n is 0, the expression directly gives our assumed expectation. So,
while u and v more directly describe the particular beta distribution, s and
x are much more helpful for our actual use of the calculation. In particular
s can be "tuned" according to whatever strength of the prior assumption
leads to the best performance, and x can be tuned for whatever assumed
Grahamian probability leads to the best performance. If desired, a
reasonable value for x can be calculated from the training data by averaging
Graham's probability, which we shall call p(w), for all words w.

So, suppose we have a corpus with the same number of spams as hams, and we
want to calculate an expectation for the word "Nigeria." We start looking
through emails, one by one, incrementing n for each email that contains
"Nigeria," and incrementing y if that email is a spam.

Before we look at our first email, our Bayesian calc gives us our assumed
expectation, x:

    (s + x) + 0
x = -----------.
      s + 0

As we get more data, the expression moves gradually farther away from x,
asymptotically approaching the actual probability that an email containing
"Nigeria" is a spam.

This is all well and good, except for one thing: we don't have the same
number of spams and hams in our training data.

The question is, do we want to give more weight to evidence in favour of
spamminess or hamminess because of the fact that the particular individual
using a system built on this technique might happen to receive more ham than
spam (or vice versa)?

Graham implicitly assumes not. And I tend to agree. And most importantly, in
testing, this technique, with Graham's assumption, seems to "work."

Here I will quote from Tim Peters' description of the Grahamian method (by
personal email):

> suppose Nigeria appears in 50 ham and 50 spam...  In [Graham's]
> scheme, it also depends on the total number of hams and of spams.  Say there
> are 100 spams and 200 hams.  Then Graham effectively says "OK, it appears in
> half of the spams, but only a quarter of the hams, so if I see Nigeria it's
> twice as likely to be a spam, so the spamprob is really 2/3"

(If you're familiar with Graham's technique, note that we're ignoring the
fact that he multiplies the ham counts by 2, which testing is showing is not
helpful when certain other changes are made and which we have made.)

Now, when we're using this Grahamian calculation, that's different from the
simple count described above for the binomial experiment in order to derive
an expectation. However:

Again, let p(w) be the result of the Grahamian calculation described by Tim
Peters. It is easy to see that the expression (n * p(w)), where n is the
number of "Nigeria" instances looked at, approximates the value the counter
I describe above WOULD have had if we actually had the same number of spams
and hams (of course this becomes more true the larger the number of spams
and hams we have).

For instance, in Tim's example, let's assume that instead of 100 spams and
200 hams, there were 150 spams and 150 hams, leaving the total number of
emails at the same 300. Then "Nigeria" would have appeared in approximately
75 spams and 37.5 hams. So, our counters would have arrived at approximately
the values:

n= 112.5
y = 75

(And of course 75/112.5 = 2/3 = .6666.)

Let p(w) be the Grahamian probability, .666666.

Then our formula would be:

       (s * x) + 75
f(w) = ------------
        s + 112.5  


  (s * x) + (112.5*.666666)
= -------------------------
        s + 112.5 


  (s * x) + (n * p(w))
= --------------------
        s + n     


And we have arrived at the desired expression.

Note 1:  We could be more rigorous about the derivation of our expression
from Graham's. The expectation of ham or spam for Nigeria could both be
separately calculated using a binomial random variable with a beta prior.
But for corpuses of any size at all, the s and x terms would be completely
overwhelmed by the actual data, so there is really no point in going through
the exercise of doing that calculation. However, I believe this procedure
could be used to derive a fairly rigorous justification of the Grahamiam
assumption, which I am not taking the time to do here.

Note 2: In calculating p(w) Graham counts every instance of word w in every
email in which it appears. Obviously, if a word appears once in an email,
there is a greater probability that it will appear again in that same email
than if it hadn't already appeared at all. So, the random variable is not
really independent under Graham's technique which is one reason why, in the
description above, we only count the first occurrence. However, we are
pragmatic and whatever works best in practice is what we should do. There is
some evidence at this point that using the Graham counting technique leads
to slightly better results than using the "pure" technique above. This may
because it is not ignoring any of the data. So, p(w) and n should simply be
computed the way that gives the best results.

Note 3: One could invoke the "naive Bayesian assumption" of independence
even where the variables are known to not be independent. That has been
proven to be an acceptible assumption in certain contexts usually having to
do with classification. I don't think this application meets the
requirements for invoking those proofs but I haven't studied that question
in detail.






From eppstein@ics.uci.edu  Sun Sep 22 17:11:05 2002
From: eppstein@ics.uci.edu (David Eppstein)
Date: Sun, 22 Sep 2002 09:11:05 -0700
Subject: [Spambayes] Re: Practical applications
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCENHBFAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCCENHBFAB.tim.one@comcast.net>
Message-ID: <6995229.1032685864@[192.168.1.102]>

On 9/22/02 3:50 AM -0400, Tim Peters wrote:
> [Guido]
>> Excuse my ignorance, but what's boosting?
>
> google for
>
>     boosting bagging
>
> Searching for either alone won't yield happiness <wink>.

Google for adaboost also works pretty well.

-- 
David Eppstein       UC Irvine Dept. of Information & Computer Science
eppstein@ics.uci.edu http://www.ics.uci.edu/~eppstein/


From popiel@wolfskeep.com  Sun Sep 22 18:28:01 2002
From: popiel@wolfskeep.com (T. Alexander Popiel)
Date: Sun, 22 Sep 2002 10:28:01 -0700
Subject: [Spambayes] RE: Further Improvement 2 
In-Reply-To: Message from Guido van Rossum <guido@python.org> 
	of "Sun, 22 Sep 2002 03:06:01 EDT."
	<200209220706.g8M765Q27830@pcp02138704pcs.reston01.va.comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCOENABFAB.tim.one@comcast.net>
	<200209220632.g8M6W3e27694@pcp02138704pcs.reston01.va.comcast.net>
	<200209220706.g8M765Q27830@pcp02138704pcs.reston01.va.comcast.net> 
Message-ID: <20020922172801.62242F4CD@cashew.wolfskeep.com>

In message:  <200209220706.g8M765Q27830@pcp02138704pcs.reston01.va.comcast.net>
             Guido van Rossum <guido@python.org> writes:
>> > So I expect it would be an easy project to get this code working
>> > under 2.2
>
>Done now.

Thanks.  I'll grab the updated copies and see to providing more data
for you folks to chew on. :-)

- Alex

From bkc@murkworks.com  Sun Sep 22 18:41:56 2002
From: bkc@murkworks.com (Brad Clements)
Date: Sun, 22 Sep 2002 13:41:56 -0400
Subject: [Spambayes] cmp.py with mean and dev comparison
Message-ID: <3D8DC8B4.16147.3D35699B@localhost>

I hacked rates.py to extract the ham/spam mean and sdev from each run

then changed cmp.py to output the means and changes, sdev and changes for each 
run.

Note that the mean/sdev and diffs are output under "false positive/false negative", but 
the numbers aren't for the "false.." part, just lined up with the run.

Its ugly, but in my case is shows a decrease in ham mean (I like), but also a decrease 
in spam mean (don't like).

But for both ham and spam, enabling recv lines has also reduced the sdev, which I 
think is a good step.

Or .. is this over-analyzing?

false positive percentages            ham mean                 ham sdev
    0.692  0.538  won    -22.25%      30.30 29.46 (-2.77%)     7.75 7.32 (-5.55%)
    0.385  0.308  won    -20.00%      30.04 29.31 (-2.43%)     7.66 7.24 (-5.48%)
    0.385  0.308  won    -20.00%      30.77 29.90 (-2.83%)     7.60 7.11 (-6.45%)
    0.692  0.538  won    -22.25%      30.71 29.83 (-2.87%)     7.49 7.13 (-4.81%)
    0.462  0.385  won    -16.67%      30.26 29.47 (-2.61%)     7.72 7.23 (-6.35%)
    0.154  0.154  tied                30.47 29.62 (-2.79%)     7.44 7.10 (-4.57%)
    0.462  0.462  tied                30.16 29.36 (-2.65%)     7.49 7.10 (-5.21%)
    0.385  0.231  won    -40.00%      30.07 29.23 (-2.79%)     7.43 6.91 (-7.00%)
    0.769  0.538  won    -30.04%      29.88 29.13 (-2.51%)     7.70 7.25 (-5.84%)
    0.462  0.385  won    -16.67%      30.09 29.32 (-2.56%)     7.49 6.97 (-6.94%)

won   8 times
tied  2 times
lost  0 times

total unique fp went from 63 to 50 won    -20.63%
mean fp % went from 0.484615384615 to 0.384615384615 won    -20.63%

false negative percentages            spam mean                spam sdev
    1.692  1.615  won     -4.55%      79.05 78.63 (-0.53%)     7.90 7.79 (-1.39%)
    2.462  2.385  won     -3.13%      78.99 78.67 (-0.41%)     8.44 8.25 (-2.25%)
    2.154  2.000  won     -7.15%      79.15 78.85 (-0.38%)     8.31 8.15 (-1.93%)
    1.923  1.615  won    -16.02%      79.31 79.00 (-0.39%)     8.00 7.81 (-2.38%)
    1.846  1.769  won     -4.17%      79.07 78.79 (-0.35%)     8.02 7.77 (-3.12%)
    1.692  1.615  won     -4.55%      79.36 79.05 (-0.39%)     7.65 7.37 (-3.66%)
    1.538  1.385  won     -9.95%      78.73 78.52 (-0.27%)     7.87 7.60 (-3.43%)
    2.077  1.846  won    -11.12%      78.99 78.71 (-0.35%)     8.24 7.99 (-3.03%)
    2.077  2.000  won     -3.71%      79.33 79.02 (-0.39%)     8.28 8.05 (-2.78%)
    2.154  2.077  won     -3.57%      78.79 78.46 (-0.42%)     8.47 8.37 (-1.18%)

won  10 times
tied  0 times
lost  0 times

total unique fn went from 255 to 238 won     -6.67%
mean fn % went from 1.96153846154 to 1.83076923077 won     -6.67%

[bkc@marimba spambayes]$ cvs diff rates.py 
Index: rates.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/rates.py,v
retrieving revision 1.6
diff -r1.6 rates.py
41c41,42
<     interesting = filter(lambda line: line.startswith('-> '), ifile)
---
>     interesting = filter(lambda line: line.startswith('-> ') or (
>         line.find('sample sdev') != -1), ifile)


[bkc@marimba spambayes]$ cvs diff cmp.py
Index: cmp.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/cmp.py,v
retrieving revision 1.9
diff -r1.9 cmp.py
24c24,28
<     fps = []
---
>     fps = []
>     hamdev = []
>     spamdev = []
>     devcount = 0
>     
28c32,33
<         if line.startswith('-> <stat> tested'):
---
>         if line.startswith('-> <stat> tested'):
>             devcount = 0
33c38,50
<             break
---
>             break
>         if line.find('sample sdev') != -1:
>             if devcount > 1:
>                 continue
>             vals = line.split(';')
>             mean = float(vals[1].split(' ')[-1])
>             sdev = float(vals[2].split(' ')[-1])
>             if not devcount:
>                 hamdev.append((mean,sdev))
>             else:
>                 spamdev.append((mean,sdev))
>             devcount += 1
>             continue
47c64
<     return fps, fns, fptot, fntot, fpmean, fnmean
---
>     return fps, fns, fptot, fntot, fpmean, fnmean, hamdev, spamdev
51c68
<             t = "tied"
---
>             t = "tied          "
60,61c77,89
< 
< def dump(p1s, p2s):
---
> 
> def mtag(m1,m2):
>     mean1,dev1 = m1
>     mean2,dev2 = m2
>     mp = (mean2 - mean1) * 100.0 / mean1
>     dp = (dev2 - dev1) * 100.0 / dev1
> 
>     return "     %2.2f %2.2f (%+2.2f%%)     %2.2f %2.2f (%+2.2f%%)" %  (
>             mean1,mean2,mp,
>             dev1,dev2,dp
>         )
>     
> def dump(p1s, p2s,meandev1,meandev2):
63c91
<     for p1, p2 in zip(p1s, p2s):
---
>     for p1, p2,m1,m2 in zip(p1s, p2s,meandev1,meandev2):
65c93
<         print "    %5.3f  %5.3f  %s" % (p1, p2, t)
---
>         print "    %5.3f  %5.3f  %s %s" % (p1, p2, t,mtag(m1,m2))
85,86c113,114
< fp1, fn1, fptot1, fntot1, fpmean1, fnmean1 = suck(file(f1n))
< fp2, fn2, fptot2, fntot2, fpmean2, fnmean2 = suck(file(f2n))
---
> fp1, fn1, fptot1, fntot1, fpmean1, fnmean1,hamdev1,spamdev1 = suck(file(f1n))
> fp2, fn2, fptot2, fntot2, fpmean2, fnmean2,hamdev2,spamdev2 = suck(file(f2n))
89,90c117,118
< print "false positive percentages"
< dump(fp1, fp2)
---
> print "false positive percentages            ham mean                 ham sdev"
> dump(fp1, fp2,hamdev1,hamdev2)
95,96c123,124
< print "false negative percentages"
< dump(fn1, fn2)
---
> print "false negative percentages            spam mean                spam sdev"
> dump(fn1, fn2,spamdev1,spamdev2)



Brad Clements,                bkc@murkworks.com   (315)268-1000
http://www.murkworks.com                          (315)268-9812 Fax
AOL-IM: BKClements


From crunch@shopip.com  Sun Sep 22 10:15:57 2002
From: crunch@shopip.com (John Draper)
Date: Sun, 22 Sep 2002 02:15:57 -0700
Subject: [Spambayes] Spam Conference in Boston.
In-Reply-To: 
 <200209220127.g8M1RUS27173@pcp02138704pcs.reston01.va.comcast.net>
References: Your message of "Sat, 21 Sep 2002 13:24:05 PDT."            
 <v03110779b9b28570ca78@[192.168.0.2]>
 <LNBBLJKPBEHFEDALKOLCGELJBFAB.tim.one@comcast.net>
 <B9B2087A.1644E%grobinson@transpose.com>             
 <v03110779b9b28570ca78@[192.168.0.2]>
Message-ID: <v03110781b9b3398519ea@[192.168.0.2]>

http://www.paulgraham.com/spamconference.html

Gee - I thought you guys would already know about it.   
Talk to Gilberte Houbart <gilberte@alum.mit.edu> for details on 
arrangement for speaking or showing off some products or whatever.

Paul Graham turned me onto it.  He also turned me onto the Spambayes
project,  and I'm amazed you guys haven't herd about it.

Gilberte is the one you should talk to,  about details,  and it would
be great if we all can meet there.   

January 17, 2003
Bartos Theater, MIT Media Laboratory
20 Ames Street, Cambridge, MA
Interested in spam filters?  Come join us at a Conference on Spam Filtering.
While anyone will be welcome, we're hoping most of all to
make this conference an opportunity for hackers working
on spam filters to get together and compare notes.
Arrangements will be very informal: no fees, sponsorships, proceedings,
luncheons, contests, etc.  Just a stream of quick, concentrated talks, and then we all go somewhere and get Chinese food.  (It worked well at LL1.)
We already have some good speakers informally lined up.  If you'd like to
give a talk at this conference, please send an email directly to 
gh@archub.org.  Talks will be 20 minutes, but if you want to do
something longer we're flexible.
There is a possibility of money towards plane
tickets for speakers, especially those who are working on 
open-source spam filtering projects. 






From crunch@shopip.com  Sun Sep 22 10:38:33 2002
From: crunch@shopip.com (John Draper)
Date: Sun, 22 Sep 2002 02:38:33 -0700
Subject: [Spambayes] Practical applications
In-Reply-To: <LNBBLJKPBEHFEDALKOLCEEMLBFAB.tim.one@comcast.net>
References: 
 <200209220127.g8M1RUS27173@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <v03110787b9b33fec9b32@[192.168.0.2]>

>[John Draper]
>> BY the way, are all of you aware of an anti-spam conference in
>> Boston in Jan?  If not, let me know, and I'll send you all the info.
>
>{Guido]
>> Please post here!
>
[Tim Peters]
>I assume this the conference Paul Graham is organizing:
>
>    http://www.paulgraham.com/spamconference.html

That's right...




From crunch@shopip.com  Sun Sep 22 19:02:16 2002
From: crunch@shopip.com (John Draper)
Date: Sun, 22 Sep 2002 11:02:16 -0700
Subject: [Spambayes] RE: Further Improvement 2
In-Reply-To: <15757.53755.736942.134794@anthem.wooz.org>
References: <LNBBLJKPBEHFEDALKOLCOENABFAB.tim.one@comcast.net>
 <200209220632.g8M6W3e27694@pcp02138704pcs.reston01.va.comcast.net>
 <200209220706.g8M765Q27830@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <v0311078bb9b3b588f0d7@[192.168.0.2]>

>>>>>> "GvR" == Guido van Rossum <guido@python.org> writes:
>
>    GvR> Done now.  The only thing that required being awake was
>    GvR> figuring out what to use instead of get_content_[main]type(),
>    GvR> which are new in the 2.3 email package.
>
>Note that Python 2.2.1's email package is different than Python 2.3's
>in many ways, some of them perhaps significant to spambayes.  For
>example, 2.3's parser has a lax mode so that malformed messages won't
>just crap out with an exception.

So,  what does it take to use the new email package with the older 2.2.1.  Is there a HUGE dependancy involved?
>
>get_content_type() and friends is another useful change.  According to
>the RFCs, all messages/parts have a content type, which may be an
>implicit default.  Python 2.2's email package didn't support this by
>2.3's does.

How hard would it be to add just the new email module (and other modules) to the older 2.2.1?
>
>You might want to grab email 2.3.1 from the mimelib project on
>SourceForge.  It's identical to what's in Python 2.3's cvs right now.

JOhn



From tim.one@comcast.net  Sun Sep 22 19:05:14 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 22 Sep 2002 14:05:14 -0400
Subject: [Spambayes] A perverse prob strength experiment
In-Reply-To: <LNBBLJKPBEHFEDALKOLCIENMBFAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEOFBFAB.tim.one@comcast.net>

What if we ignored all words with spamprob *greater* than 0.1 away from 0.5?
That is, only look at the blandest words.  That's an indicator of how much
relevant information is in the bland words.  Apart from that, the same as
"the usual" default untweaked Robinson setup, although I had the foresight
<wink> to ask for 1000 histogram buckets:

[Classifier]
use_robinson_probability: True
use_robinson_combining: True
max_discriminators: 1500

[TestDriver]
spam_cutoff: 0.5
nbuckets: 1000

There is *some* information in the bland words; with the cutoff at 0.5,

total unique false pos 7372
total unique false neg 4345
average fp % 36.86
average fn % 31.0357142857

The variance in f-p rates across runs was very high:

      f-p %   f-n %
     38.850  29.214
     32.500  33.071
     33.950  34.929
     38.450  30.000
     44.050  26.786
     38.700  29.071
     30.800  34.214
     36.750  31.000
     36.450  32.286
     38.100  29.786

If there were no exploitable information, both error rates would be near
50%.  A question is whether the bits of information in the bland words vote
"in the same direction" as the non-bland words.  The idea that it should do
no *harm* to include the bland words assumes that they tend to vote in the
same direction as the non-bland words (which we know very strongly vote in
"the right" direction).  I'm not sure that they do!  The mean of the spam
distribution is a little higher than the mean of the ham distribution, but
both are far below the values of spam_cutoff that have worked best overall
by all reports so far:

Ham distribution for all runs:
20000 items; mean 49.76; sample sdev 0.70
* = 21 items
 45.90    0
 46.00    1 *
 46.10    0
 46.20    0
 46.30    0
 46.40    0
 46.50    1 *
 46.60    0
 46.70    3 *
 46.80    2 *
 46.90    1 *
 47.00    2 *
 47.10    4 *
 47.20    9 *
 47.30   10 *
 47.40   15 *
 47.50   25 **
 47.60   21 *
 47.70   28 **
 47.80   40 **
 47.90   55 ***
 48.00   60 ***
 48.10   98 *****
 48.20  117 ******
 48.30  131 *******
 48.40  182 *********
 48.50  238 ************
 48.60  298 ***************
 48.70  340 *****************
 48.80  404 ********************
 48.90  521 *************************
 49.00  617 ******************************
 49.10  706 **********************************
 49.20  833 ****************************************
 49.30  892 *******************************************
 49.40 1056 ***************************************************
 49.50 1134 ******************************************************
 49.60 1176 ********************************************************
 49.70 1171 ********************************************************
 49.80 1250 ************************************************************
 49.90 1187 *********************************************************
 50.00 1095 *****************************************************
 50.10 1009 *************************************************
 50.20  951 **********************************************
 50.30  875 ******************************************
 50.40  708 **********************************
 50.50  600 *****************************
 50.60  514 *************************
 50.70  358 ******************
 50.80  317 ****************
 50.90  260 *************
 51.00  188 *********
 51.10  163 ********
 51.20   89 *****
 51.30   64 ****
 51.40   52 ***
 51.50   41 **
 51.60   21 *
 51.70   21 *
 51.80   21 *
 51.90    7 *
 52.00    6 *
 52.10    3 *
 52.20    2 *
 52.30    1 *
 52.40    2 *
 52.50    2 *
 52.60    1 *
 52.70    0
 52.80    0
 52.90    1 *
 53.00    0

Spam distribution for all runs:
14000 items; mean 50.37; sample sdev 0.81
* = 14 items
 46.90   0
 47.00   8 *
 47.10   3 *
 47.20   4 *
 47.30   0
 47.40   1 *
 47.50   2 *
 47.60   8 *
 47.70   7 *
 47.80  10 *
 47.90  13 *
 48.00  18 **
 48.10  21 **
 48.20  21 **
 48.30  30 ***
 48.40  56 ****
 48.50  92 *******
 48.60  54 ****
 48.70  66 *****
 48.80 126 *********
 48.90 146 ***********
 49.00 196 **************
 49.10 194 **************
 49.20 212 ****************
 49.30 296 **********************
 49.40 311 ***********************
 49.50 376 ***************************
 49.60 412 ******************************
 49.70 464 **********************************
 49.80 552 ****************************************
 49.90 646 ***********************************************
 50.00 698 **************************************************
 50.10 780 ********************************************************
 50.20 736 *****************************************************
 50.30 788 *********************************************************
 50.40 684 *************************************************
 50.50 672 ************************************************
 50.60 629 *********************************************
 50.70 557 ****************************************
 50.80 543 ***************************************
 50.90 528 **************************************
 51.00 449 *********************************
 51.10 470 **********************************
 51.20 437 ********************************
 51.30 343 *************************
 51.40 253 *******************
 51.50 254 *******************
 51.60 194 **************
 51.70 121 *********
 51.80 125 *********
 51.90  81 ******
 52.00  69 *****
 52.10  68 *****
 52.20  43 ****
 52.30  32 ***
 52.40  21 **
 52.50  15 **
 52.60  19 **
 52.70  18 **
 52.80   9 *
 52.90   2 *
 53.00  14 *
 53.10   3 *
 53.20   0

To minimize the total # of misclassifications, spam_cutoff should be raised
to 0.505.  The we would lose 4638 fp:

[slice of ham]
 50.00 1095
 50.10 1009
 50.20  951
 50.30  875
 50.40  708

and gain 3686 fn:

[slice of spam]
 50.00  698
 50.10  780
 50.20  736
 50.30  788
 50.40  684

Going beyond that would start raising the total number again, because the #
of ham in the .505 bucket is larger than the number of spam:

[slice of ham]
 50.50  600

[slice of spam]
 50.50  672

So, at 0.505, we'd see

total unique false pos 7372-4638 = 2734
total unique false neg 4345+3686 = 8031
average fp % 2734/20000 = 13.67%
average fn % 8031/14000 = 57.36%

However, at that point, rating spam is worse than flipping a coin!
Seriously, if it is in fact the case that the bland words are such that
reducing the f-p rate wrt them requires making the f-n rate wrt them worse
than flipping a coin, it's small surprise that we see a reduction in f-n
rate when leaving the bland words out.


From tim.one@comcast.net  Sun Sep 22 19:20:38 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 22 Sep 2002 14:20:38 -0400
Subject: [Spambayes] RE: Further Improvement 2
In-Reply-To: <15757.53755.736942.134794@anthem.wooz.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCAEOHBFAB.tim.one@comcast.net>

[Barry A. Warsaw]
> Note that Python 2.2.1's email package is different than Python 2.3's
> in many ways, some of them perhaps significant to spambayes.  For
> example, 2.3's parser has a lax mode so that malformed messages won't
> just crap out with an exception.

Barry, can you please fix this?  I knew nothing about the email package
before this project, and I don't understand my own code anymore now that
Guido has changed it to work with some older version of the package.  We
"fixed" the problems with the use of 2.3's sets and heapq libraries just by
checking their source into the spambayes project.  Can you do the same for
your email pkg?

> get_content_type() and friends is another useful change.  According to
> the RFCs, all messages/parts have a content type, which may be an
> implicit default.  Python 2.2's email package didn't support this by
> 2.3's does.

And I *was* relying on that (I think <wink>) to generate assorted metadata
tokens for all the MIME sections in a msg.


From tim.one@comcast.net  Sun Sep 22 19:30:11 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 22 Sep 2002 14:30:11 -0400
Subject: [Spambayes] cmp.py with mean and dev comparison
In-Reply-To: <3D8DC8B4.16147.3D35699B@localhost>
Message-ID: <LNBBLJKPBEHFEDALKOLCAEOIBFAB.tim.one@comcast.net>

[Brad Clements]
> I hacked rates.py to extract the ham/spam mean and sdev from each run

Cool!

> then changed cmp.py to output the means and changes, sdev and
> changes for each run.
>
> Note that the mean/sdev and diffs are output under "false
> positive/false negative", but the numbers aren't for the "false.." part,
> just lined up with the run.
>
> Its ugly,

It's not ugly, but it's too wide.  Print the new stuff on new lines of their
own instead.

> but in my case is shows a decrease in ham mean (I like), but also a
> decrease in spam mean (don't like).

If the former decrease is larger than the latter decrease, the overall
separation increases, and that's the important thing (provided you believe
increasing separation is a good thing <wink>).

> But for both ham and spam, enabling recv lines has also reduced
> the sdev, which I think is a good step.

Me too.

> Or .. is this over-analyzing?

You must be joking <wink>.

[horribly line-wrapped too-wide output elided]

> ===================================================================
> RCS file: /cvsroot/spambayes/spambayes/rates.py,v
> retrieving revision 1.6
> diff -r1.6 rates.py
> 41c41,42
> <     interesting = filter(lambda line: line.startswith('-> '), ifile)
> ---
> >     interesting = filter(lambda line: line.startswith('-> ') or (
> >         line.find('sample sdev') != -1), ifile)

Don't do that:  change the lines you want to suck out to begin with '-> '
instead, like all the other lines this script processes.  Else it becomes an
unmaintainable mass of ad hoc and error-prone "pattern matching" rules.  It
*used* to do that, and I spent a fair amount of time changing all the
"interesting lines" to start with "-> " instead.  More, I changed the
statistics lines to start with "-> <stat> ".

IOW, just make the new code look as much as possible like the code that's
already there.  I think this is a good change if you do that.


From bkc@murkworks.com  Sun Sep 22 19:38:01 2002
From: bkc@murkworks.com (Brad Clements)
Date: Sun, 22 Sep 2002 14:38:01 -0400
Subject: [Spambayes] cmp.py with mean and dev comparison
In-Reply-To: <LNBBLJKPBEHFEDALKOLCAEOIBFAB.tim.one@comcast.net>
References: <3D8DC8B4.16147.3D35699B@localhost>
Message-ID: <3D8DD5D8.11901.3D68C07D@localhost>

On 22 Sep 2002 at 14:30, Tim Peters wrote:

> Don't do that:  change the lines you want to suck out to begin with '-> '
> instead, like all the other lines this script processes.  Else it becomes
> an unmaintainable mass of ad hoc and error-prone "pattern matching" rules. 
> It *used* to do that, and I spent a fair amount of time changing all the
> "interesting lines" to start with "-> " instead.  More, I changed the
> statistics lines to start with "-> <stat> ".

But then I'd have to change timcv.py and re-run my tests.. :-(

PIII-933 isn't very fast.. 

But .. since I'll be spelunking the timcv.py code, I think I'll also work on a "end-user 
emulation" module. Where I can simulate receiving messages and training as I go.

I want to determine the daily error rate as I train the system. I suppose I'll have to be 
able to specify a ratio of ham/spam (< 1 in my case) and a "daily message count". 
What's typical numbers to use? (also need a mis-classification rate too for lazy folks)

I suspect most users will be diligent about feeding spam into the trainer, but will be lazy 
about feeding it ham.


Brad Clements,                bkc@murkworks.com   (315)268-1000
http://www.murkworks.com                          (315)268-9812 Fax
AOL-IM: BKClements


From grobinson@transpose.com  Sun Sep 22 19:42:16 2002
From: grobinson@transpose.com (Gary Robinson)
Date: Sun, 22 Sep 2002 14:42:16 -0400
Subject: [Spambayes] Central Limit Theorem??!!     :)
Message-ID: <B9B38748.164F0%grobinson@transpose.com>

Guys, 

I happened to have an idea while leaving McDonald's today that, after toying
with it a bit, seems so clearly right,  and is so exciting to me, that I
just have to mention it.

For theoretical reasons that you will see, and which are so straightforward
there can really be little doubt about them, it should perform really,
really well.

Here goes:

First, see this description of the Central Limit Theorem:

> 
> The central limit theorem states that given a distribution with a mean m and
> variance s^2, the sampling distribution of the mean approaches a  normal
> distribution with a mean (m) and a variance (s^2)/N as N, the  sample size,
> increases.

(edited slightly from http://davidmlane.com/hyperstat/A14043.html for
compatibility with this text medium.)

The great thing about this is that the original variables don't have to be
normally distributed. They can be ANY distribution... including the
distribution we happen to have in our f(w)'s. Even if we only take the most
extreme f(w)'s!

This gives us  a means to be totally rigorous, even based on the very
application-specific, not-mathematically-rigorous data we are starting with.

Say we have decided only to look at the 30 most extreme f(w)'s in every
email. I am choosing 30 because it is usually enough for the central limit
theorem to really kick in, but we may get better performance with a greater
number of extrema.

Now let's generate a population of f(w)'s as follows.

We go through all the spam emails one by one, adding the 30 most extreme
f(w)'s to our population (only making one addition for each unique
occurrence of a word in the population).

There are certainly enough data points in a training corpus of > 1,000
emails that this would give us a nice normal distribution.

So, from this population, we calculate the population mean M, and the
population variance S^2.

Now we look at a particular email, and consider the 30 most extreme f(w)'s
in that email to be a sample from the larger population (again counting each
word only once). The sample mean m (calculated by simply averaging the
f(w)'s) has a mean M and variance (s^2)/n. Normally n will be 30, or course,
but for emails with less than 30 unique words it will obviously be less.

Now calculate the z-score of m, using M as the mean and (s^2)/n as the
variance.

Now do the all the same things, but from the perspective of considering the
ham corpus to be the population. (Obviously we would compute the ham and
spam population M's and S^2's up front and only do them once.)

So we have two z-scores: zs and zh for spam and ham respectively.

Let R = abs(zh) - abs(zs).

If R > 0, we have spam; otherwise we have ham.

That is, we accept the ham/spam hypothesis that is less violated by the
data.

This really should work very, very well, because it incorporates much more
data than something like S does (or Graham's combining mechod) and uses it
in a much more meaningful way.

It will work less well as the number of words in an email approach 0 because
the central limit theorem becomes progressively less applicable in such
cases, but that's no big surprise, is it? ;)  And most such emails are
probably so clearly spam or ham that this degradation will not be an issue.

It is a little complicated to implement but really not that complicated. Any
elementary statistics book contains straightforward arithmetic techniques
for computing these numbers.

Moreover, Python probably already has libraries that can compute them
effortlessly.


I would be extremely interested in the results of any testing on this.





--Gary


-- 
Gary Robinson
CEO
Transpose, LLC
grobinson@transpose.com
207-942-3463
http://www.emergentmusic.com
http://radio.weblogs.com/0101454



From tim.one@comcast.net  Sun Sep 22 20:19:45 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 22 Sep 2002 15:19:45 -0400
Subject: [Spambayes] cmp.py with mean and dev comparison
In-Reply-To: <3D8DD5D8.11901.3D68C07D@localhost>
Message-ID: <LNBBLJKPBEHFEDALKOLCAEOLBFAB.tim.one@comcast.net>

[Brad Clements]
> But then I'd have to change timcv.py

Why?  The means and sdevs are computed and displayed by the Hist(ogram)
class in TestDriver.py.  timcv.py is just a tiny wrapper that adapts
TestDriver to "the standard" Ham/Spam directory structure, and drives
TestDriver in a cross-validation way.  timcv.py doesn't compute or print
anything directly.

> and re-run my tests.. :-(
>
> PIII-933 isn't very fast..

When you run 50x as many tests on a slower machine, I'll have some sympathy
based on comparable experience <wink>.

> But .. since I'll be spelunking the timcv.py code, I think I'll
> also work on a "end-user emulation" module. Where I can simulate
> receiving messages and training as I go.

Such stuff probably belongs in new classes.

> I want to determine the daily error rate as I train the system. I
> suppose I'll have to be able to specify a ratio of ham/spam (< 1 in my
> case) and a "daily message count".  What's typical numbers to use?

For whom?  I get about 600 emails per day.  Is that typical?  It certainly
is for me <wink>.  Some people get very little spam, John Draper reported
something like 70% spam.

> ...
> I suspect most users will be diligent about feeding spam into the
> trainer, but will be lazy about feeding it ham.

Me too.  Gary fiddled the formulas in his approach to try to be robust in
the face of this.  No experiments have been run to test it, though, neither
under Gary's nor Paul's schemes.


From bkc@murkworks.com  Sun Sep 22 20:36:12 2002
From: bkc@murkworks.com (Brad Clements)
Date: Sun, 22 Sep 2002 15:36:12 -0400
Subject: [Spambayes] cmp.py with mean and dev comparison
In-Reply-To: <LNBBLJKPBEHFEDALKOLCAEOLBFAB.tim.one@comcast.net>
References: <3D8DD5D8.11901.3D68C07D@localhost>
Message-ID: <3D8DE37B.27712.3D9E0643@localhost>

On 22 Sep 2002 at 15:19, Tim Peters wrote:

> [Brad Clements]
> > But then I'd have to change timcv.py
> 
> Why?  

Because I wasn't paying attention when I wrote that ;-)

I've found the goodies and just waiting for my first test re-run to complete.

Meanwhile, I'm going to move Hist into it's own module and allow it to keep all the 
numbers from add() and save itself in a pickle.

Then, I'm planning on working up a Pythoncard app that will plot a density (weighted 
moving average) of the values, instead of a bucket'd histogram.

By loading two pickled Hists (spam/ham), I'll throw in a slider control or two to show 
how many fp, fn's and "grey-n's" fall out based on spam-prob .. probably need another 
constant if we want a middle ground, though overlaying the graphs and playing with a 
slider might point out we don't need a middle ground.

Is extending Hist this way acceptable. Using it as a collection point should be illegal, but 
it's a good spot to do it without having to change a lot of stuff in TestDriver.

> > But .. since I'll be spelunking the timcv.py code, I think I'll
> > also work on a "end-user emulation" module. Where I can simulate
> > receiving messages and training as I go.
> 
> Such stuff probably belongs in new classes.

new module with a new class.

> > I suspect most users will be diligent about feeding spam into the
> > trainer, but will be lazy about feeding it ham.
> 
> Me too.  Gary fiddled the formulas in his approach to try to be robust in
> the face of this.  No experiments have been run to test it, though, neither
> under Gary's nor Paul's schemes.

I intend to find out..



Brad Clements,                bkc@murkworks.com   (315)268-1000
http://www.murkworks.com                          (315)268-9812 Fax
AOL-IM: BKClements


From tim.one@comcast.net  Sun Sep 22 20:37:00 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 22 Sep 2002 15:37:00 -0400
Subject: [Spambayes] RE: Further Improvement 2
In-Reply-To: <B9B3696D.164DA%grobinson@transpose.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCOEOLBFAB.tim.one@comcast.net>

[Gary Robinson]
> With regard to the
>
>           a + y
> f(w) = ------------
>        (a / x) + n
>
> calc ...

Thank you, Gary!  This is very helpful.  May I check this explanation into
the project (with attribution, of course, and a link to your web page)?
We're very low on docs explaining the underpinnings of our code, and this
was wonderfully lucid.

> ...
> Note 3: One could invoke the "naive Bayesian assumption" of independence
> even where the variables are known to not be independent. That has been
> proven to be an acceptible assumption in certain contexts usually
> having to do with classification. I don't think this application meets
> the requirements for invoking those proofs but I haven't studied that
> question in detail.

We wrestled a lot with what to do about HTML markup in this project.
Whether a msg contains, e.g., "<html>", and whether it contains "<br>", are
nearly 100% correlated in the real world.  Both are strong spam indicators
simply because so much spam comes in HTML form.  This makes naive Bayes
*grossly* overestimate the probability that a doc containing both is spam
(P(Spam|<html> and <br>) ~= P(Spam|<html>) in real life, but naive Bayes
thinks it's much more likely if it contains both).  Multiply that by a
hundred other indicators nearly unique to HTML markup, and, in the end, we
had to resort to stripping HTML markup entirely, else no HTML message could
escape being called spam.  We still seem to get bad effects from that
P(Spam|content-type=text/html) is not independent of P(Spam|"&nbsp;") in
real life.


From bkc@murkworks.com  Sun Sep 22 21:36:28 2002
From: bkc@murkworks.com (Brad Clements)
Date: Sun, 22 Sep 2002 16:36:28 -0400
Subject: [Spambayes] 2nd try at mean/dev cmp.py
Message-ID: <3D8DF19B.22946.3DD532D4@localhost>


---------------------- multipart/mixed attachment
Produces this output:

ham mean                 ham sdev
30.30 29.46 (-2.77%)     7.75 7.32 (-5.55%)
30.04 29.31 (-2.43%)     7.66 7.24 (-5.48%)
30.77 29.90 (-2.83%)     7.60 7.11 (-6.45%)
30.71 29.83 (-2.87%)     7.49 7.13 (-4.81%)
30.26 29.47 (-2.61%)     7.72 7.23 (-6.35%)
30.47 29.62 (-2.79%)     7.44 7.10 (-4.57%)
30.16 29.36 (-2.65%)     7.49 7.10 (-5.21%)
30.07 29.23 (-2.79%)     7.43 6.91 (-7.00%)
29.88 29.13 (-2.51%)     7.70 7.25 (-5.84%)
30.09 29.32 (-2.56%)     7.49 6.97 (-6.94%)

ham mean and sdev for all runs
30.27 29.46 (-2.68%)     7.58 7.14 (-5.80%)

spam mean                spam sdev
79.05 78.63 (-0.53%)     7.90 7.79 (-1.39%)
78.99 78.67 (-0.41%)     8.44 8.25 (-2.25%)
79.15 78.85 (-0.38%)     8.31 8.15 (-1.93%)
79.31 79.00 (-0.39%)     8.00 7.81 (-2.38%)
79.07 78.79 (-0.35%)     8.02 7.77 (-3.12%)
79.36 79.05 (-0.39%)     7.65 7.37 (-3.66%)
78.73 78.52 (-0.27%)     7.87 7.60 (-3.43%)
78.99 78.71 (-0.35%)     8.24 7.99 (-3.03%)
79.33 79.02 (-0.39%)     8.28 8.05 (-2.78%)
78.79 78.46 (-0.42%)     8.47 8.37 (-1.18%)

spam mean and sdev for all runs
79.08 78.77 (-0.39%)     8.12 7.92 (-2.46%)

ham/spam mean difference: 48.81 49.31 +0.50

[bkc@marimba spambayes]$ cvs diff TestDriver.py
Index: TestDriver.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/TestDriver.py,v
retrieving revision 1.6
diff -r1.6 TestDriver.py
92c92
<     print "Ham distribution for", tag
---
>     print "-> <stat> Ham distribution for", tag,
96c96
<     print "Spam distribution for", tag
---
>     print "-> <stat> Spam distribution for", tag,

[bkc@marimba spambayes]$ cvs diff cmp.py
(attached)





Brad Clements,                bkc@murkworks.com   (315)268-1000
http://www.murkworks.com                          (315)268-9812 Fax
AOL-IM: BKClements



---------------------- multipart/mixed attachment
Index: cmp.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/cmp.py,v
retrieving revision 1.9
diff -r1.9 cmp.py
24c24,27
<     fps = []
---
>     fps = []
>     hamdev = []
>     spamdev = []
>     
28c31
<         if line.startswith('-> <stat> tested'):
---
>         if line.startswith('-> <stat> tested'):
29a33,49
>         if line.find('sample sdev') != -1:
>             vals = line.split(';')
>             mean = float(vals[1].split(' ')[-1])
>             sdev = float(vals[2].split(' ')[-1])
>             val = (mean,sdev)
>             typ = vals[0].split(' ')[2]
>             if line.find('for all runs') != -1:
>                 if typ == 'Ham':
>                     hamdevall = val
>                 else:
>                     spamdevall = val
>             elif line.find('all in this') != -1:
>                 if typ == 'Ham':
>                     hamdev.append(val)
>                 else:
>                     spamdev.append(val)
>             continue
33c53
<             break
---
>             break
47c67
<     return fps, fns, fptot, fntot, fpmean, fnmean
---
>     return fps, fns, fptot, fntot, fpmean, fnmean, hamdev, spamdev,hamdevall,spamdevall
51c71
<             t = "tied"
---
>             t = "tied          "
60c80,91
< 
---
> 
> def mtag(m1,m2):
>     mean1,dev1 = m1
>     mean2,dev2 = m2
>     mp = (mean2 - mean1) * 100.0 / mean1
>     dp = (dev2 - dev1) * 100.0 / dev1
> 
>     return "%2.2f %2.2f (%+2.2f%%)     %2.2f %2.2f (%+2.2f%%)" %  (
>             mean1,mean2,mp,
>             dev1,dev2,dp
>         )
>     
70a102,105
> 
> def dumpdev(meandev1,meandev2):
>     for m1,m2 in zip(meandev1,meandev2):
>         print mtag(m1, m2)
85,86c120,121
< fp1, fn1, fptot1, fntot1, fpmean1, fnmean1 = suck(file(f1n))
< fp2, fn2, fptot2, fntot2, fpmean2, fnmean2 = suck(file(f2n))
---
> fp1, fn1, fptot1, fntot1, fpmean1, fnmean1,hamdev1,spamdev1,hamdevall1,spamdevall1 = suck(file(f1n))
> fp2, fn2, fptot2, fntot2, fpmean2, fnmean2,hamdev2,spamdev2,hamdevall2,spamdevall2 = suck(file(f2n))
98a134,151
> 
> print
> print "ham mean                 ham sdev"
> dumpdev(hamdev1,hamdev2)
> print
> print "ham mean and sdev for all runs"
> dumpdev([hamdevall1],[hamdevall2])
> 
> print
> print "spam mean                spam sdev"
> dumpdev(spamdev1,spamdev2)
> print
> print "spam mean and sdev for all runs"
> dumpdev([spamdevall1],[spamdevall2])
> print
> diff1 = spamdevall1[0] - hamdevall1[0]
> diff2 = spamdevall2[0] - hamdevall2[0]
> print "ham/spam mean difference: %2.2f %2.2f %+2.2f" % (diff1,diff2,(diff2-diff1))

---------------------- multipart/mixed attachment--

From grobinson@transpose.com  Sun Sep 22 21:46:26 2002
From: grobinson@transpose.com (Gary Robinson)
Date: Sun, 22 Sep 2002 16:46:26 -0400
Subject: [Spambayes] Re: Central Limit Theorem??!!     :)
In-Reply-To: <B9B38748.164F0%grobinson@transpose.com>
Message-ID: <B9B3A462.164FF%grobinson@transpose.com>

One mistake: I said:

> There are certainly enough data points in a training corpus of > 1,000 emails
> that this would give us a nice normal distribution.

That's not actually a normal distribution... the whole point is that it
doesn't have to be! :)


--Gary


-- 
Gary Robinson
CEO
Transpose, LLC
grobinson@transpose.com
207-942-3463
http://www.emergentmusic.com
http://radio.weblogs.com/0101454


> From: Gary Robinson <grobinson@transpose.com>
> Date: Sun, 22 Sep 2002 14:42:16 -0400
> To: SpamBayes <spambayes@python.org>, Tim Peters <tim.one@comcast.net>, Greg
> Louis <glouis@dynamicro.on.ca>
> Subject: Central Limit Theorem??!!     :)
> 
> Guys, 
> 
> I happened to have an idea while leaving McDonald's today that, after toying
> with it a bit, seems so clearly right,  and is so exciting to me, that I just
> have to mention it.
> 
> For theoretical reasons that you will see, and which are so straightforward
> there can really be little doubt about them, it should perform really, really
> well.
> 
> Here goes:
> 
> First, see this description of the Central Limit Theorem:
> 
>> 
>> The central limit theorem states that given a distribution with a mean m and
>> variance s^2, the sampling distribution of the mean approaches a  normal
>> distribution with a mean (m) and a variance (s^2)/N as N, the  sample size,
>> increases.
> 
> (edited slightly from http://davidmlane.com/hyperstat/A14043.html for
> compatibility with this text medium.)
> 
> The great thing about this is that the original variables don't have to be
> normally distributed. They can be ANY distribution... including the
> distribution we happen to have in our f(w)'s. Even if we only take the most
> extreme f(w)'s!
> 
> This gives us  a means to be totally rigorous, even based on the very
> application-specific, not-mathematically-rigorous data we are starting with.
> 
> Say we have decided only to look at the 30 most extreme f(w)'s in every email.
> I am choosing 30 because it is usually enough for the central limit theorem to
> really kick in, but we may get better performance with a greater number of
> extrema.
> 
> Now let's generate a population of f(w)'s as follows.
> 
> We go through all the spam emails one by one, adding the 30 most extreme
> f(w)'s to our population (only making one addition for each unique occurrence
> of a word in the population).
> 
> There are certainly enough data points in a training corpus of > 1,000 emails
> that this would give us a nice normal distribution.
> 
> So, from this population, we calculate the population mean M, and the
> population variance S^2.
> 
> Now we look at a particular email, and consider the 30 most extreme f(w)'s in
> that email to be a sample from the larger population (again counting each word
> only once). The sample mean m (calculated by simply averaging the f(w)'s) has
> a mean M and variance (s^2)/n. Normally n will be 30, or course, but for
> emails with less than 30 unique words it will obviously be less.
> 
> Now calculate the z-score of m, using M as the mean and (s^2)/n as the
> variance.
> 
> Now do the all the same things, but from the perspective of considering the
> ham corpus to be the population. (Obviously we would compute the ham and spam
> population M's and S^2's up front and only do them once.)
> 
> So we have two z-scores: zs and zh for spam and ham respectively.
> 
> Let R = abs(zh) - abs(zs).
> 
> If R > 0, we have spam; otherwise we have ham.
> 
> That is, we accept the ham/spam hypothesis that is less violated by the data.
> 
> This really should work very, very well, because it incorporates much more
> data than something like S does (or Graham's combining mechod) and uses it in
> a much more meaningful way.
> 
> It will work less well as the number of words in an email approach 0 because
> the central limit theorem becomes progressively less applicable in such cases,
> but that's no big surprise, is it? ;)  And most such emails are probably so
> clearly spam or ham that this degradation will not be an issue.
> 
> It is a little complicated to implement but really not that complicated. Any
> elementary statistics book contains straightforward arithmetic techniques for
> computing these numbers.
> 
> Moreover, Python probably already has libraries that can compute them
> effortlessly.
> 
> 
> I would be extremely interested in the results of any testing on this.
> 
> 
> 
> 
> 
> --Gary
> 
> 
> -- 
> Gary Robinson
> CEO
> Transpose, LLC
> grobinson@transpose.com
> 207-942-3463
> http://www.emergentmusic.com
> http://radio.weblogs.com/0101454
> 


From tim.one@comcast.net  Sun Sep 22 22:02:46 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 22 Sep 2002 17:02:46 -0400
Subject: [Spambayes] 2nd try at mean/dev cmp.py
In-Reply-To: <3D8DF19B.22946.3DD532D4@localhost>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEPABFAB.tim.one@comcast.net>

[Brad Clements]
> Produces this output:
>
> ham mean                 ham sdev
> 30.30 29.46 (-2.77%)     7.75 7.32 (-5.55%)
> 30.04 29.31 (-2.43%)     7.66 7.24 (-5.48%)
> 30.77 29.90 (-2.83%)     7.60 7.11 (-6.45%)
> 30.71 29.83 (-2.87%)     7.49 7.13 (-4.81%)
> 30.26 29.47 (-2.61%)     7.72 7.23 (-6.35%)
> 30.47 29.62 (-2.79%)     7.44 7.10 (-4.57%)
> 30.16 29.36 (-2.65%)     7.49 7.10 (-5.21%)
> 30.07 29.23 (-2.79%)     7.43 6.91 (-7.00%)
> 29.88 29.13 (-2.51%)     7.70 7.25 (-5.84%)
> 30.09 29.32 (-2.56%)     7.49 6.97 (-6.94%)
> 
> ham mean and sdev for all runs
> 30.27 29.46 (-2.68%)     7.58 7.14 (-5.80%)
> 
> spam mean                spam sdev
> 79.05 78.63 (-0.53%)     7.90 7.79 (-1.39%)
> 78.99 78.67 (-0.41%)     8.44 8.25 (-2.25%)
> 79.15 78.85 (-0.38%)     8.31 8.15 (-1.93%)
> 79.31 79.00 (-0.39%)     8.00 7.81 (-2.38%)
> 79.07 78.79 (-0.35%)     8.02 7.77 (-3.12%)
> 79.36 79.05 (-0.39%)     7.65 7.37 (-3.66%)
> 78.73 78.52 (-0.27%)     7.87 7.60 (-3.43%)
> 78.99 78.71 (-0.35%)     8.24 7.99 (-3.03%)
> 79.33 79.02 (-0.39%)     8.28 8.05 (-2.78%)
> 78.79 78.46 (-0.42%)     8.47 8.37 (-1.18%)
> 
> spam mean and sdev for all runs
> 79.08 78.77 (-0.39%)     8.12 7.92 (-2.46%)
> 
> ham/spam mean difference: 48.81 49.31 +0.50

Cool!  Looks good -- check it in.

From grobinson@transpose.com  Sun Sep 22 22:32:40 2002
From: grobinson@transpose.com (Gary Robinson)
Date: Sun, 22 Sep 2002 17:32:40 -0400
Subject: [Spambayes] Practical applications 
In-Reply-To: <200209220742.g8M7gf025021@localhost.localdomain>
Message-ID: <B9B3AF38.16504%grobinson@transpose.com>


>>>> Guido van Rossum wrote
>>> 1) I notice that Python is an "Actice Scripting" language? Does that a mean
>>> that a Python script can interact with Outlook to do things like add
>>> a "Spam!" button, etc.?
>> 
>> Python can control any Windows application that supports a COM
>> interface.  You have to get the win32all set of additions (there's a
>> pointer to it on the download pages).  I've never done this, I suppose
>> you'd need docs for Outlook.
> 
> I think he was asking for something the other way - a way to add
> a button/command to Outbreak which runs a python script. I don't know if
> that's possible - someone like MarkH may know.
> 


Right... for instance, say a new email comes into Outlook. It should be
checked for spammness before going into the user's In box. I'm pretty sure
this could be done in Visual Basic for Applications.

And I read somewhere that anything a VBA script can do, an  Active Scripting
language can do. And I read somewhere that Python is an Active Scripting
language.

But... can it actually do it?

Gary


From grobinson@transpose.com  Sun Sep 22 22:33:59 2002
From: grobinson@transpose.com (Gary Robinson)
Date: Sun, 22 Sep 2002 17:33:59 -0400
Subject: [Spambayes] Re: Central Limit Theorem??!!     :)
In-Reply-To: <B9B3A462.164FF%grobinson@transpose.com>
Message-ID: <B9B3AF87.16507%grobinson@transpose.com>

Ooops, I was behind on my reading on the list, and just saw all the stuff
you guys are doing with sdev etc. :)


--Gary


-- 
Gary Robinson
CEO
Transpose, LLC
grobinson@transpose.com
207-942-3463
http://www.emergentmusic.com
http://radio.weblogs.com/0101454




From bkc@murkworks.com  Sun Sep 22 22:34:32 2002
From: bkc@murkworks.com (Brad Clements)
Date: Sun, 22 Sep 2002 17:34:32 -0400
Subject: [Spambayes] 2nd try at mean/dev cmp.py
In-Reply-To: <LNBBLJKPBEHFEDALKOLCIEPABFAB.tim.one@comcast.net>
References: <3D8DF19B.22946.3DD532D4@localhost>
Message-ID: <3D8DFF37.2332.3E0A5DED@localhost>

On 22 Sep 2002 at 17:02, Tim Peters wrote:

> > 
> > ham/spam mean difference: 48.81 49.31 +0.50
> 
> Cool!  Looks good -- check it in.

uh-oh, now I've been found out. Even though I stayed in a Holiday Inn last night, I can't 
fake this one.

I can check out, but I can't check in. I'm just a visual source-safe user in disguise.

Also, I'm at home now with 3 little kids crawling on me, and checking this via a very 
slow VNC.

:-( can't try checkin until tomorrow. I did post the patch.. (hopeful)..


Brad Clements,                bkc@murkworks.com   (315)268-1000
http://www.murkworks.com                          (315)268-9812 Fax
AOL-IM: BKClements


From marklists@mceahern.com  Sun Sep 22 23:46:56 2002
From: marklists@mceahern.com (Mark McEahern)
Date: Sun, 22 Sep 2002 17:46:56 -0500
Subject: [Spambayes] Practical applications 
In-Reply-To: <B9B3AF38.16504%grobinson@transpose.com>
Message-ID: <JHEOKEOOLIGLDHCMAHMOEEIPEAAA.marklists@mceahern.com>

[Gary Robinson]
> Right... for instance, say a new email comes into Outlook. It should be
> checked for spammness before going into the user's In box. I'm pretty sure
> this could be done in Visual Basic for Applications.
>
> And I read somewhere that anything a VBA script can do, an
> Active Scripting language can do. And I read somewhere that Python is an
> Active Scripting language.
>
> But... can it actually do it?

One way to do this in Outlook would be to create a Custom Action to be used
by the Rules engine, which according to the following cannot be done with
VB*:

  http://www.slipstick.com/emo/1997/up970804.htm#custom

Also, (as I think someone else pointed out previously), apparently rules in
Outlook 2002 are different.

Cheers,

// m


From tim.one@comcast.net  Mon Sep 23 03:16:12 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 22 Sep 2002 22:16:12 -0400
Subject: [Spambayes] RE: Central Limit Theorem??!!     :)
In-Reply-To: <B9B38748.164F0%grobinson@transpose.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEPHBFAB.tim.one@comcast.net>

[Gary Robinson]
> I happened to have an idea while leaving McDonald's today that,
> after toying with it a bit, seems so clearly right,  and is so exciting
> to me, that I just have to mention it.

Glad you did!  It's intriguing.

[... intro snipped ...]
>
> Say we have decided only to look at the 30 most extreme f(w)'s in every
> email. I am choosing 30 because it is usually enough for the central
> limit theorem to really kick in, but we may get better performance with
> a greater number of extrema.
>
> Now let's generate a population of f(w)'s as follows.
>
> We go through all the spam emails one by one, adding the 30 most extreme
> f(w)'s to our population (only making one addition for each unique
> occurrence of a word in the population).

Just noting that this is a two-pass process:  we have to go through all the
spam and all the ham in the training set first, in order to compute f(w).
Then we go though them again to tally the extremes (which ones are extreme
can't be known before the first pass completes).

> ...
> So, from this population, we calculate the population mean M, and the
> population variance S^2.

S^2 isn't used below; I believe it should be.

> Now we look at a particular email, and consider the 30 most extreme
> f(w)'s in that email to be a sample from the larger population (again
> counting each  word only once).  The sample mean m (calculated by simply
> averaging the f(w)'s) has a mean M and variance (s^2)/n.

I believe that last was meant to be S^2/n.

> Normally n will be 30, or course, but for emails with less than 30
> unique words it will obviously be less.
>
> Now calculate the z-score of m, using M as the mean and (s^2)/n as the
> variance.

Ditto wrt s^2 vs S^2.

> Now do the all the same things, but from the perspective of
> considering the ham corpus to be the population. (Obviously we would
> compute the ham and spam population M's and S^2's up front and only
> do them once.)

Therein lies a tale <wink -- see below>.

> So we have two z-scores: zs and zh for spam and ham respectively.
>
> Let R = abs(zh) - abs(zs).
>
> If R > 0, we have spam; otherwise we have ham.

I scaled this more-or-less arbitrarily, like so:

        if R < -20.0:
            R = -20.0
        elif R > 20.0:
            R = 20.0
        R = 0.5 + R / 40.0

That is, I'm bashing R into [0, 1], with 0.5 corresponding to R=0.  Once I
can get this to run thru a whole test set, I'm sure better clipping bounds
will be apparent.

> That is, we accept the ham/spam hypothesis that is less violated by the
> data.
>
> This really should work very, very well, because it incorporates much
> more data than something like S does (or Graham's combining mechod) and
> uses it in a much more meaningful way.
>
> It will work less well as the number of words in an email
> approach 0 because the central limit theorem becomes progressively less
> applicable in such cases, but that's no big surprise, is it? ;)  And
> most such emails are probably so clearly spam or ham that this
> degradation will not be an issue.

To date, very short emails have given us a lot of trouble; or at least those
of us who have to stay away from most of the header lines.  For example, a
message consisting of a single URL may be a legit response to a question, or
a come-on for porn.  There are a handful of clues in URLs that have usually
been enough to nail it, though.

> It is a little complicated to implement but really not that
> complicated.

The code is actually easier this way, although the two-pass nature is quite
a change, and my code doesn't actually work <wink>:

> Any elementary statistics book contains straightforward arithmetic
> techniques for computing these numbers.

That turns out to be a problem.  Over time, we want to incrementally train
the classifier, and we sure don't want the user to have to save all the
training data they've ever used.  Just keeping counts was easy this way.
Our test drivers also gain massive efficiency by learning *and* unlearning
msgs as a test run proceeds (e.g., learn on everything, unlearn a small
subset, predict on that subset, learn on that subset again, then move on to
another small subset).  For the test drivers, keeping all the msgs around
for multiple passes isn't a problem.

In order to support "easy" incrementalism, for ham and for spam I saved the
f(w) counts, the sum of the f(w) values, and the sum of their squares.
Means and variances are easily (heh) computed from those, and it's just as
easy to subtract stuff away when unlearning an old msg as to add it in when
learning a new msg.

Unfortunately, this isn't a numerically stable process when using
double-precision floats:  in particular, the sums across all training msgs
are large compared to the sums for a single training msg.  As a result, so
many low-order bits are lost when incrementally adding or subtracting a
message that the "clever textbook"

    variance = sum_of_squares/n - mean**2

suffers catastrophic cancellation and can easily produce a result consisting
of pure noise bits (previous rounding errors "shifted to the left" by
cancelling of high-order bits).  When I'm lucky, this comes out as negative
trash, and then a subsequent attempt to take a square root blows up with an
exception.

> Moreover, Python probably already has libraries that can compute them
> effortlessly.

The numeric pitfalls here are too extreme to use doubles in a
straightforward way.  I'm going to have to fake extra precision, or save
giant lists of the most-extreme f(w)'s so that the variance can be computed
via a more robust method (e.g., the utterly straightforward

    variance = (sum over i of (x[i] - mean)**2)/n

is numerically very robust -- but it requires saving an explicit vector of
data points).

> I would be extremely interested in the results of any testing on this.

I'm afraid I can't get thru a large test until the numeric disasters get
fixed.  Numeric disasters in small tests are more prevalent when using
Graham's p(w) than your f(w), because they're more extreme in small tests
(we got rid of the minimum-count gimmick, so a word appearing just once in
the training data will get p(w) of 0.01 or 0.99).

When part of a small test run actually completes <sigh>, this approach
appears to work very well with p(w), and less so with f(w) (a=1.0 is "too
high" on my data when using small training sets).

Here's a typical p(w) run training on 900 of each and predicting on 100 of
each:  no errors, and R distribution (scaled as above):

Ham distribution for all in this training set:
100 items; mean 0.18; sample sdev 1.36
* = 2 items
  0.00 98 *************************************************
  2.50  0
  5.00  1 *
  7.50  0
 10.00  1 *
 12.50  0
...

Spam distribution for all in this training set:
100 items; mean 99.02; sample sdev 5.64
* = 2 items
...
 52.50  0
 55.00  1 *
 57.50  0
 60.00  0
 62.50  0
 65.00  0
 67.50  1 *
 70.00  0
 72.50  0
 75.00  0
 77.50  0
 80.00  0
 82.50  1 *
 85.00  0
 87.50  0
 90.00  0
 92.50  1 *
 95.00  0
 97.50 96 ************************************************

On the same test data using f(w), there are 6 false negatives, and the R
histos change to

Ham distribution for this pair:
100 items; mean 0.17; sample sdev 1.20
* = 2 items
  0.00 98 *************************************************
  2.50  0
  5.00  0
  7.50  2 *
 10.00  0
...

Spam distribution for this pair:
100 items; mean 78.97; sample sdev 12.37
* = 1 items
...
 30.00  0
 32.50  1 *
 35.00  0
 37.50  1 *
 40.00  1 *
 42.50  1 *
 45.00  0
 47.50  2 **
 50.00  1 *
 52.50  2 **
 55.00  2 **
 57.50  1 *
 60.00  0
 62.50  1 *
 65.00  3 ***
 67.50  1 *
 70.00  3 ***
 72.50  2 **
 75.00  2 **
 77.50  3 ***
 80.00  3 ***
 82.50 30 ******************************
 85.00 38 **************************************
 87.50  2 **
 90.00  0
 92.50  0
 95.00  0
 97.50  0

It seems typical that spam R scores are *very* much more spread out (than
ham scores, or than when using p(w)) when using f(w).  This seems to persist
as training data gets larger too.  It may because "a" is too large, or may
be because there's another bug in this new code, or may be because spam
sometimes has very hammish elements more often than ham has very spammish
elements, or ...


From guido@python.org  Mon Sep 23 03:48:46 2002
From: guido@python.org (Guido van Rossum)
Date: Sun, 22 Sep 2002 22:48:46 -0400
Subject: [Spambayes] RE: Further Improvement 2
In-Reply-To: Your message of "Sun, 22 Sep 2002 11:02:16 PDT."
             <v0311078bb9b3b588f0d7@[192.168.0.2]> 
References: <LNBBLJKPBEHFEDALKOLCOENABFAB.tim.one@comcast.net>
	<200209220632.g8M6W3e27694@pcp02138704pcs.reston01.va.comcast.net>
	<200209220706.g8M765Q27830@pcp02138704pcs.reston01.va.comcast.net>  
	<v0311078bb9b3b588f0d7@[192.168.0.2]> 
Message-ID: <200209230248.g8N2mkK04947@pcp02138704pcs.reston01.va.comcast.net>

> So, what does it take to use the new email package with the older
> 2.2.1.  Is there a HUGE dependancy involved?

I think the new email package should work without change (or with very
little change) with 2.2.1.  In fact I believe the intention is to
backport the new email package to 2.2.2.  But I don't want to import
the entire email package into the spambayes CVS tree.

> >get_content_type() and friends is another useful change.  According to
> >the RFCs, all messages/parts have a content type, which may be an
> >implicit default.  Python 2.2's email package didn't support this by
> >2.3's does.

I've checked the uses of get_type() (all in tokenizer.py), and I
believe that they all do the right thing except possibly one.  This
code:

            # Remove HTML/XML tags.
            if (part.get_type() == "text/plain" or
                    not options.retain_pure_html_tags):
                text = html_re.sub(' ', text)

should become:

            # Remove HTML/XML tags.
            if (part.get_type('text/plain') == "text/plain" or
                    not options.retain_pure_html_tags):
                text = html_re.sub(' ', text)

--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido@python.org  Mon Sep 23 04:27:30 2002
From: guido@python.org (Guido van Rossum)
Date: Sun, 22 Sep 2002 23:27:30 -0400
Subject: [Spambayes] Moving closer to Gary's ideal
In-Reply-To: Your message of "Sun, 22 Sep 2002 05:15:51 EDT."
             <LNBBLJKPBEHFEDALKOLCMENKBFAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCMENKBFAB.tim.one@comcast.net> 
Message-ID: <200209230327.g8N3RYo05107@pcp02138704pcs.reston01.va.comcast.net>

> > So to match the 40 fp's from Graham's scheme, I'd need to set the
> > cutoff to 0.60; that would give me 41 fp's here (16+10+3+1+1+2+8).
> 
> Right!  BTW, that's a huge test run -- certainly the largest to date.

I finally looked at my fp's in detail, which I should have done right
away.  There were 4 empty files (MH refiling accidents probably) and
11 spams that I had somehow saved.  I'm rerunning everything with
these removed now.

Among the remaining fp's were a few forwarded spams (several by you,
Tim!); a bunch of automated responses from web sites where I ordered
stuff; a bunch of product newsletters that I like to get because I use
the product; two pieces of legit email (largely) in Spanish; a
legitimate job posting sent to jobs@python.org; a list of diet tips
forwarded by my wife; a message from a publisher asking where to send
my check (I'd hate to have misfiled that one!); a bunch of very short
messages (some with, some without ISP-added trailers); a few messages
with HTML alternatives; and one question about HTML style sheets that
quoted a few lines of typical stylesheet gibberish (not HTML, but this
stuff is often inlined in HTML).  One of the brief questions used
charset=GB2312 (whatever that is); there were a lot of bogus 8-bit
characters, but in the middle it said loud and clear 'Where can I get
a "wxpython"?'.

> > I'm going to have to look at the fp's and fn's to see if there are
> > real spams hiding in the ham, and vice versa.
> 
> If there aren't, you'll be the first tester ever not to discover
> some.

See above. :-)

> For example, I've found 3 hams in BruceG's spam collection so
> far, and I believe you're using that too (but much more of it).

I've not found the courage to look at the hundreds of fn's.

Suggestion: rather than showing the content of the fn's and fp's (the
filenames are enough for me), would it be possible to show the
filenames corresponding to the outliers in the ham/spam distributions?
E.g. there's 1 message in my spam collection that scores 37.50
acfording to the overall histogram.  How to find that one?

> BTW, it's my belief that this all works *best* if the ratio of ham
> to spam trained on matches your real-life inbox ratio.

That's impossible to know in my case.  Almost all of my mail goes
through the SpamAssassin setup at python.org, which throws all spam
away.  As a result I see maybe 1 spam for every 50 hams -- but that's
not the spam/ham ratio seen by the MTA for guido@python.org.

> > I did notice that many fp's were very spammish automated postings
> > that I have specifically signed up for, like our building's
> > announcements, product newsletters, and so on.  I haven't looked at
> > the fn's.
> 
> I expect these are your moral equivalents to the conference
> announcements in my c.l.py ham, except worse.  However, I expect you
> have more cause for optimism about those: you (like me) are running
> a crippled version of the algorithm because of your mixed-source
> corpora.  The headers we're ignoring are bound to have strong clues
> about the *senders* of the spammish stuff you've signed up for.

Only if I saved enough of these, right?  Any clue as to what option to
try?

> It would be nicer to display them side-by-side, but you'd have to learn how
> to use more of your available screen area then <wink>.

Jeremy would suggest to generate gnuplot input so we can draw them in
multiple colors. :-)

--Guido van Rossum (home page: http://www.python.org/~guido/)

From anthony@interlink.com.au  Mon Sep 23 01:58:05 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Mon, 23 Sep 2002 10:58:05 +1000
Subject: [Spambayes] dealing with non-english data.
Message-ID: <200209230058.g8N0w5b29718@localhost.localdomain>

Hm. So I've just about finished going through the new messages I've dumped
into my corpus, and I'm trying to narrow down the fp's and fn's. There's a
_lot_ of stuff in these mailboxes that are non-english and non-ascii. At the
moment, the tokenizer doesn't do a fabulous job on this stuff. I'm wondering
about doing conversion into the given character set, or else tagging the 
words with the character set (if it's non-english). 

Unfortunately my knowledge of character set issues is up there with my
knowledge of high-altitude yak milking, but I'd love to know if we've got
anyone on this list who knows more about this - for instance, tokenizing
koi-8r, or euc-kr...

Anthony
--
Anthony Baxter     <anthony@interlink.com.au>
It's never too late to have a happy childhood.

From guido@python.org  Mon Sep 23 04:53:27 2002
From: guido@python.org (Guido van Rossum)
Date: Sun, 22 Sep 2002 23:53:27 -0400
Subject: [Spambayes] dealing with non-english data.
In-Reply-To: Your message of "Mon, 23 Sep 2002 10:58:05 +1000."
             <200209230058.g8N0w5b29718@localhost.localdomain> 
References: <200209230058.g8N0w5b29718@localhost.localdomain> 
Message-ID: <200209230353.g8N3rRh05238@pcp02138704pcs.reston01.va.comcast.net>

> Hm. So I've just about finished going through the new messages I've dumped
> into my corpus, and I'm trying to narrow down the fp's and fn's. There's a
> _lot_ of stuff in these mailboxes that are non-english and non-ascii. At the
> moment, the tokenizer doesn't do a fabulous job on this stuff. I'm wondering
> about doing conversion into the given character set, or else tagging the 
> words with the character set (if it's non-english). 
> 
> Unfortunately my knowledge of character set issues is up there with my
> knowledge of high-altitude yak milking, but I'd love to know if we've got
> anyone on this list who knows more about this - for instance, tokenizing
> koi-8r, or euc-kr...

Me neither.  But here's something any schmuck with a recent Python
version can try: use the regular expression \w+ compiled with the re.U
flag to find maximal strings of word characters according to the
Unicode locale.  This should return to strings of characters for each
of which u.isalnum() or u == '_'.  Then all we need to assume in
addition is that the Unicode standard defines letter-ness in a useful
way for Korean and Chinese...

--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.one@comcast.net  Mon Sep 23 05:27:08 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 23 Sep 2002 00:27:08 -0400
Subject: [Spambayes] dealing with non-english data.
In-Reply-To: <200209230058.g8N0w5b29718@localhost.localdomain>
Message-ID: <LNBBLJKPBEHFEDALKOLCAEPNBFAB.tim.one@comcast.net>

[Anthony Baxter]
> Hm. So I've just about finished going through the new messages I've
> dumped into my corpus, and I'm trying to narrow down the fp's and fn's.
> There's a _lot_ of stuff in these mailboxes that are non-english and
> non-ascii.  At the moment, the tokenizer doesn't do a fabulous job on
> this stuff.

I gave up trying.  charset identifiers generate tokens, but everything else
goes thru the same path as English.  Note that every "long word" containing
at least 1 high-bit character generates an "8bit%:nn" token, where nn is the
percentage of characters within the word with the sign bit lit.

That much was enough so that I have no Asian-language false negatives (and
therefore there's nothing I can do to test possible improvements -- it's
just not an issue in my test data anymore).

Beyond that, I'm clueless.  Greg (Ward) says python.org gets tons of Asian
spam, so when I move to that corpus I'll doubtless be inspired to do better
on it.


From tim.one@comcast.net  Mon Sep 23 05:29:56 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 23 Sep 2002 00:29:56 -0400
Subject: [Spambayes] dealing with non-english data.
In-Reply-To: <200209230353.g8N3rRh05238@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEPNBFAB.tim.one@comcast.net>

[Guido]
> Me neither.  But here's something any schmuck with a recent Python
> version can try: use the regular expression \w+ compiled with the re.U
> flag to find maximal strings of word characters according to the
> Unicode locale.  This should return to strings of characters for each
> of which u.isalnum() or u == '_'.  Then all we need to assume in
> addition is that the Unicode standard defines letter-ness in a useful
> way for Korean and Chinese...

Does part.get_payload(decode=True) produce Unicode when appropriate?  I've
no idea.  Note that some Asian languages don't use whitespace, and a large
pile of 60-character "words" isn't going to do much good.


From tim.one@comcast.net  Mon Sep 23 07:07:34 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 23 Sep 2002 02:07:34 -0400
Subject: [Spambayes] Moving closer to Gary's ideal
In-Reply-To: <200209230327.g8N3RYo05107@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEAABGAB.tim.one@comcast.net>

[Guido]
> ...
> One of the brief questions used charset=GB2312 (whatever that is);

One of many Chinese character sets.

> ...
> Suggestion: rather than showing the content of the fn's and fp's (the
> filenames are enough for me), would it be possible to show the
> filenames corresponding to the outliers in the ham/spam distributions?

[TestDriver]
show_charlimit: 0

in conjuction with show_false_positives and/or show_false_negatives and/or
the following.

> E.g. there's 1 message in my spam collection that scores 37.50
> acfording to the overall histogram.  How to find that one?

As above, or

[TestDriver]
show_spam_lo: .375
show_spam_hi: .4   (or whatever your next bucket boundary is)


>> BTW, it's my belief that this all works *best* if the ratio of ham
>> to spam trained on matches your real-life inbox ratio.

> That's impossible to know in my case.

By "your inbox" I mean the one you see.

> Almost all of my mail goes through the SpamAssassin setup at
> python.org, which throws all spam away.  As a result I see maybe
> 1 spam for every 50 hams -- but that's not the spam/ham ratio
> seen by the MTA for guido@python.org.

Training for your MTA is different than training for the inbox you see.  If
your interest is to help python.org do a better job, then you need to get
the spam that python.org currently rejects; if your interest is in the spam
you see despite python.org, then you should train on the email you actually
get.

>>> I did notice that many fp's were very spammish automated postings
>>> that I have specifically signed up for, like our building's
>>> announcements, product newsletters, and so on.  I haven't looked at
>>> the fn's.

>> I expect these are your moral equivalents to the conference
>> announcements in my c.l.py ham, except worse.  However, I expect you
>> have more cause for optimism about those: you (like me) are running
>> a crippled version of the algorithm because of your mixed-source
>> corpora.  The headers we're ignoring are bound to have strong clues
>> about the *senders* of the spammish stuff you've signed up for.

> Only if I saved enough of these, right?

Right.  I don't remember which scheme you're using.  Under the all-default
Graham scheme, a single instance of a word in a ham or spam training set
gives that word maximum probability strength.  In the all-default Robinson
scheme, it does not, and the "a" parameter still needs tuning.

> Any clue as to what option to try?

Have you read the "defaults" string in Options.py?  That's an embedded .ini
file explaining every option in its comments, and supplying their default
values.  mine_received_headers has been powerful for those who can use it; I
doubt that you can (I cannot; it picks up way too many clues about BruceG).
We've gotten mixed reports on count_all_header_lines (it helps or it
doesn't; I don't recall anyone saying it hurt); I can't use that either, but
that's mostly because my ham is full of Mailman list headers that appear
almost nowhere in my spam; you may have better luck with that.  Jeremy added
some options:

    basic_header_tokenize
    basic_header_skip

that are also disabled by default.  I haven't tried them.  I've been burned
so often by trying new headers that I'm not going to spend more time on that
until I have a single-source corpus.


From tim.one@comcast.net  Mon Sep 23 07:20:47 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 23 Sep 2002 02:20:47 -0400
Subject: [Spambayes] RE: Central Limit Theorem??!!     :)
In-Reply-To: <B9B38748.164F0%grobinson@transpose.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEABBGAB.tim.one@comcast.net>

[Gary Robinson]
> ...
> Now let's generate a population of f(w)'s as follows.
>
> We go through all the spam emails one by one, adding the 30 most extreme
> f(w)'s to our population (only making one addition for each unique
> occurrence of a word in the population).

I'm unclear on the intent of the parenthetical comment.  Suppose I have 1
spam with 500 occurrences of Nigera, and 99 spam with 1 occurrence of
Nigeria each, and no other spam contains Nigeria.  Is f('Nigeria') to be
added in 1 time total, or 100 times total?  The code I have does the latter.


From tim.one@comcast.net  Mon Sep 23 08:46:56 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 23 Sep 2002 03:46:56 -0400
Subject: [Spambayes] RE: Central Limit Theorem??!!     :)
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEPHBFAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEAEBGAB.tim.one@comcast.net>

My numerical woes are repaired, by saving sums of probs and sums of squares
of probs as scaled and unbounded Python ints.  This way no (significant)
information is lost until the very final conversion to floating-point; the
cancellation in subtraction is still there, but is harmless now because no
significant low-order bits are ever lost in intermediate steps.  It's
numerical overkill, but better safe than sorry at the start!

Turns out not all negative sqrts were due to catastrophic cancellation,
though:  some were due to inconsistent views of the world across testing
steps.  That is, "my usual" test starts like so:

1. Train on the last 9 sets of 10 ham/spam pairs, numbered 2 through 9.
2. Predict against pair 1.
3. Add pair 1 to the training data.
4. Untrain ("forget") pair 2.
5. Predict against pair 2.

Step #4 is A Problem:  step #3 changes f(w) values, which in turn can change
the set of extremes, and then the extremes found in step #4 while looking at
pair 2 messages aren't necessarily the same as were found when pair 2 was
originally trained on in step #1.  Neither are the f(w) values associated
with the extremes necessarily the same.  This can lead to step #4
subtracting more from the sum-of-squares and/or sum-of-probs accumulators
than were originally added for pair 2 in step #1.  Disaster ensues.

A different test driver here now trains on one pair, predicts on another,
and doesn't try to save time by reusing classifier state:  each pair of
pairs starts with a brand new classifier.

Here are 5 runs, each training on 1000 of each and predicting against 1000
of each.  These are random and disjoint subsets of my full test data.

       fp %    fn %
      0.000   0.400
0 new false positives
4 new false negatives

      0.000   0.100
0 new false positives
1 new false negatives

      0.000   0.500
0 new false positives
5 new false negatives

      0.300   0.100
3 new false positives
1 new false negatives

      0.400   0.200
4 new false positives
2 new false negatives

total unique false pos 7
total unique false neg 13
average fp % 0.14
average fn % 0.26

This was using 30 extremes, and using Graham's p(w) (complete with hambias
2, minprob .01 and maxprob .99).  The f-n rate was more than 10x worse using
f(w) with a=x=0.5, and I have no idea why yet (and we're *generally* having
problems with f-n rates on smaller training sets when using f(w), whether
using the central-limit scoring, or Gary's previous scoring; perhaps 'a'
needs to be much smaller than 0.5 -- there's too much to test here).

Here are the aggregate scaled R values (clamped to [-20, 20], and then
scaled linearly into [0, 1]):

Ham distribution for all runs:
5000 items; mean 0.35; sample sdev 3.62
* = 82 items
  0.00 4907 ************************************************************
  2.50   13 *
  5.00   14 *
  7.50   12 *
 10.00    4 *
 12.50   11 *
 15.00    6 *
 17.50    7 *
 20.00    4 *
 22.50    2 *
 25.00    2 *
 27.50    3 *
 30.00    0
 32.50    1 *
 35.00    3 *
 37.50    0
 40.00    1 *
 42.50    1 *
 45.00    1 *
 47.50    1 *
 50.00    1 *
 52.50    0
 55.00    1 *
 57.50    0
 60.00    0
 62.50    1 *
 65.00    2 *
 67.50    0
 70.00    0
 72.50    0
 75.00    0
 77.50    0
 80.00    0
 82.50    0
 85.00    0
 87.50    0
 90.00    0
 92.50    0
 95.00    0
 97.50    2 *

Spam distribution for all runs:
5000 items; mean 98.97; sample sdev 5.87
* = 80 items
  0.00    0
  2.50    0
  5.00    0
  7.50    0
 10.00    0
 12.50    0
 15.00    0
 17.50    0
 20.00    1 *
 22.50    0
 25.00    0
 27.50    3 *
 30.00    1 *
 32.50    1 *
 35.00    1 *
 37.50    1 *
 40.00    1 *
 42.50    0
 45.00    1 *
 47.50    3 *
 50.00    6 *
 52.50    5 *
 55.00    5 *
 57.50    3 *
 60.00    7 *
 62.50    6 *
 65.00    9 *
 67.50   11 *
 70.00   19 *
 72.50   11 *
 75.00    8 *
 77.50   10 *
 80.00   13 *
 82.50    9 *
 85.00   11 *
 87.50   16 *
 90.00   31 *
 92.50    8 *
 95.00   15 *
 97.50 4784 ************************************************************

All of the false positives had significant numbers of both 0.01 and 0.99
clues.  This seems to be a reappearance of the p(w) "cancellation disease"
that we wormed around before by adding gobs of special-case code to Graham
scoring.  Several of the fns also had this problem.  The outcome is like
flipping a coin when this happens.  Note that f(w) doesn't have this problem
(it's mostly an artifact of that p(w) artificially clamps probabilities, and
so many words end up with probs at the extreme values).

Here are the means and variances of the training data scaled R values:

hammean  0.0315194110198 hamvar  0.0102392908745
spammean 0.977596060549  spamvar 0.00629493144389

hammean  0.0289322128628 hamvar  0.00860263576484
spammean 0.976784463455  spamvar 0.00703754635535

hammean  0.0292168061706 hamvar  0.00850922282341
spammean 0.977330456163  spamvar 0.00656386045426

hammean  0.0292418489626 hamvar  0.00869327431102
spammean 0.972324957985  spamvar 0.00968199258783

hammean  0.0266295579103 hamvar  0.00745682458391
spammean 0.974432833096  spamvar 0.00865812701944


Finally, here's the same thing (including exactly the same messages in the
training and prediction sets) all over again, *except* using f(w) with a=0.1
and x=0.5 (I mentioned a=0.5 above; I lowered it again for this run, and
that did help the f-n rate, but not much):

      0.000   3.700
0 new false positives
37 new false negatives

      0.000   2.500
0 new false positives
25 new false negatives

      0.000   4.800
0 new false positives
48 new false negatives

      0.100   2.900
1 new false positives
29 new false negatives

      0.100   3.400
1 new false positives
34 new false negatives

total unique false pos 2
total unique false neg 173
average fp % 0.04
average fn % 3.46

Ham distribution for all runs:
5000 items; mean 0.05; sample sdev 1.57
* = 84 items
  0.00 4991 ************************************************************
  2.50    2 *
  5.00    0
  7.50    1 *
 10.00    0
 12.50    0
 15.00    0
 17.50    0
 20.00    1 *
 22.50    1 *
 25.00    1 *
 27.50    1 *
 30.00    0
 32.50    0
 35.00    0
 37.50    0
 40.00    0
 42.50    0
 45.00    0
 47.50    0
 50.00    0
 52.50    0
 55.00    0
 57.50    1 *
 60.00    0
 62.50    0
 65.00    0
 67.50    0
 70.00    0
 72.50    0
 75.00    0
 77.50    1 *
 80.00    0
 82.50    0
 85.00    0
 87.50    0
 90.00    0
 92.50    0
 95.00    0
 97.50    0

Spam distribution for all runs:
5000 items; mean 94.82; sample sdev 15.16
* = 69 items
  0.00    5 *
  2.50    2 *
  5.00    1 *
  7.50    5 *
 10.00    8 *
 12.50    6 *
 15.00   10 *
 17.50    8 *
 20.00    6 *
 22.50   14 *
 25.00   10 *
 27.50   11 *
 30.00    9 *
 32.50    4 *
 35.00   12 *
 37.50   12 *
 40.00   11 *
 42.50    8 *
 45.00   16 *
 47.50   15 *
 50.00   21 *
 52.50   21 *
 55.00   20 *
 57.50   27 *
 60.00   21 *
 62.50   19 *
 65.00   31 *
 67.50   18 *
 70.00   18 *
 72.50   22 *
 75.00   22 *
 77.50   37 *
 80.00   36 *
 82.50   50 *
 85.00   59 *
 87.50   50 *
 90.00   67 *
 92.50   74 **
 95.00   76 **
 97.50 4138 ************************************************************

Too bizarre for me -- there may be a gross bug here, but the central-limit
code is exactly the same in both cases, and the f(w) code is exactly the
same as I've been using with good results for a few days.

hammean  0.083575863227  hamvar 0.0384359627039
spammean 0.978952388668 spamvar 0.00372443106446

hammean  0.075515986459  hamvar 0.0331126798862
spammean 0.97742506528  spamvar 0.00420869219347

hammean  0.0776481612081 hamvar 0.0338362239317
spammean 0.978462136207 spamvar 0.00341505065601

hammean  0.07972071882   hamvar 0.0355833143776
spammean 0.974508870296 spamvar 0.00530574730638

hammean  0.0713015705881 hamvar 0.0303987810665
spammean 0.976734052416 spamvar 0.00468795224229

For whatever reason(s), I note that the ham variances are higher here than
when using p(w), and the spam variances lower.  Perhaps that's just due to
that f(w) doesn't have an artificial ham bias.  OTOH, the prediction set ham
distribution is much tighter when using the unbiased f(w), while the spam
distribution is much looser.


From sjoerd@acm.org  Mon Sep 23 09:32:21 2002
From: sjoerd@acm.org (Sjoerd Mullender)
Date: Mon, 23 Sep 2002 10:32:21 +0200
Subject: [Spambayes] Moving closer to Gary's ideal
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKELABFAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCKELABFAB.tim.one@comcast.net> 
Message-ID: <200209230832.g8N8WL613221@indus.ins.cwi.nl>

On Sat, Sep 21 2002 Tim Peters wrote:

> """
> [Classifier]
> use_robinson_probability: True
> use_robinson_combining: True
> max_discriminators: 1500
> 
> [TestDriver]
> spam_cutoff: 0.50
> """

I tested this against the default options (except I have
count_all_header_lines: True and mine_received_headers: True
permanently) and got these results:

false positive percentages
    0.524  1.047  lost   +99.81%
    0.000  0.524  lost  +(was 0)
    0.524  0.524  tied
    0.524  1.047  lost   +99.81%
    0.524  1.571  lost  +199.81%

won   0 times
tied  1 times
lost  4 times

total unique fp went from 4 to 9 lost  +125.00%
mean fp % went from 0.418848167539 to 0.942408376964 lost  +125.00%

false negative percentages
    1.571  0.000  won   -100.00%
    2.618  2.094  won    -20.02%
    1.571  0.524  won    -66.65%
    0.524  0.524  tied
    1.571  1.047  won    -33.35%

won   4 times
tied  1 times
lost  0 times

total unique fn went from 15 to 8 won    -46.67%
mean fn % went from 1.57068062827 to 0.83769633508 won    -46.67%

The histograms in the default scheme show the usual pattern, but the
histograms with the changed parameters is like this:


Ham distribution for all runs:
955 items; mean 26.28; sample sdev 8.12
* = 3 items
  0.00   0 
  2.50   0 
  5.00   0 
  7.50  40 **************
 10.00   0 
 12.50  61 *********************
 15.00  27 *********
 17.50  47 ****************
 20.00  96 ********************************
 22.50 127 *******************************************
 25.00 155 ****************************************************
 27.50 127 *******************************************
 30.00  96 ********************************
 32.50  65 **********************
 35.00  44 ***************
 37.50  24 ********
 40.00  13 *****
 42.50  13 *****
 45.00   5 **
 47.50   6 **
 50.00   8 ***
 52.50   1 *
 55.00   0 

Spam distribution for all runs:
955 items; mean 68.60; sample sdev 8.43
* = 2 items
 32.50   0 
 35.00   1 *
 37.50   2 *
 40.00   0 
 42.50   0 
 45.00   3 **
 47.50   2 *
 50.00  10 *****
 52.50  15 ********
 55.00  31 ****************
 57.50  70 ***********************************
 60.00  93 ***********************************************
 62.50 117 ***********************************************************
 65.00 109 *******************************************************
 67.50 117 ***********************************************************
 70.00 103 ****************************************************
 72.50  58 *****************************
 75.00  78 ***************************************
 77.50  59 ******************************
 80.00  34 *****************
 82.50  24 ************
 85.00   8 ****
 87.50   6 ***
 90.00  15 ********
 92.50   0 
 95.00   0 
 97.50   0 

-- Sjoerd Mullender <sjoerd@acm.org>

From grobinson@transpose.com  Mon Sep 23 13:46:10 2002
From: grobinson@transpose.com (Gary Robinson)
Date: Mon, 23 Sep 2002 08:46:10 -0400
Subject: [Spambayes] RE: Central Limit Theorem??!!     :)
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCEAEBGAB.tim.one@comcast.net>
Message-ID: <B9B48552.16549%grobinson@transpose.com>

OK. That's fascinating.

Remember that the multiplicative method in S, which calculates the geometric
mean of the f(w)'s, stresses the MOST extreme values more than the less
extreme ones. The more extreme the value is, the more it is stressed. Very
extreme ones are stressed very highly, in an exponentially compounding way
if there are several really extreme ones.

That's why it's the basis for that 1971 optimality theorem that I kept
trying to invoke more strongly, but which is still invoked to a degree in S.

That's the reasoning behind S in the first place, and why it works so well,
and it also happens in Graham's original approach (but only on one side),
but we are completely losing it in R.

In R, we are trading that very powerful multiplicative effect away in order
to get the benefit of real parametric statistics. ALSO a very powerful
technique but apparently slightly less powerful in this application -- at
least when used alone.

If there is a performance loss in R (and there are no remaining coding
errors), I am confident that that's why.

THERE IS A POTENTIAL FIX FOR THIS LOSSAGE, so that we can theoretically get
the best of both totally different techniques.

When training on the spam side, don't use f(w), use: ln (1-f(w)).

When training on the ham side, don't use f(w), use: ln f(w).

Same when testing. Don't add the f(w)'s increating the sample mean; add the
expressions above, and divide by n. So the spam side uses ln (1-f(w)) both
for training and calculating the sample means, and the ham side uses ln f(w)
for both.

As we've discussed, averaging the ln's is the same thing as a geometric mean
if you then subsequently raise e to the power of that computed average. But
we don't do that last step here. We just feed the arithmetic mean of the
ln's into the z-score calcs.

This *should* bring us the benefits of the multiplicative approach and of
the parametric stats approach at the same time.

The downside of this is that the ln's make such a skewed distribution that
it should take a bigger n to make the central limit theorem kick in. BUT,
OTOH, it WILL kick in, and something like 30 may really still be enough (it
usually kicks in at significantly smaller numbers). And you've also
successfully used n=150 with f(w) and that is DEFINITELY enough.

The other downside is that it just seems like a bit  of a wild thing to do
and sometimes when you do wild things, strange reasons emerge why they won't
work. But I really can't see any at this point as long as n is big enough
that the sample means take on a normal distribution.

THANKS for doing all the coding work to test this idea!!!!  :)

Gary




 
> 
> This was using 30 extremes, and using Graham's p(w) (complete with hambias
> 2, minprob .01 and maxprob .99).  The f-n rate was more than 10x worse using
> f(w) with a=x=0.5, and I have no idea why yet (and we're *generally* having
> problems with f-n rates on smaller training sets when using f(w), whether
> using the central-limit scoring, or Gary's previous scoring; perhaps 'a'
> needs to be much smaller than 0.5 -- there's too much to test here).
> 
> Here are the aggregate scaled R values (clamped to [-20, 20], and then
> scaled linearly into [0, 1]):
> 
> Ham distribution for all runs:
> 5000 items; mean 0.35; sample sdev 3.62
> * = 82 items
> 0.00 4907 ************************************************************
> 2.50   13 *
> 5.00   14 *
> 7.50   12 *
> 10.00    4 *
> 12.50   11 *
> 15.00    6 *
> 17.50    7 *
> 20.00    4 *
> 22.50    2 *
> 25.00    2 *
> 27.50    3 *
> 30.00    0
> 32.50    1 *
> 35.00    3 *
> 37.50    0
> 40.00    1 *
> 42.50    1 *
> 45.00    1 *
> 47.50    1 *
> 50.00    1 *
> 52.50    0
> 55.00    1 *
> 57.50    0
> 60.00    0
> 62.50    1 *
> 65.00    2 *
> 67.50    0
> 70.00    0
> 72.50    0
> 75.00    0
> 77.50    0
> 80.00    0
> 82.50    0
> 85.00    0
> 87.50    0
> 90.00    0
> 92.50    0
> 95.00    0
> 97.50    2 *
> 
> Spam distribution for all runs:
> 5000 items; mean 98.97; sample sdev 5.87
> * = 80 items
> 0.00    0
> 2.50    0
> 5.00    0
> 7.50    0
> 10.00    0
> 12.50    0
> 15.00    0
> 17.50    0
> 20.00    1 *
> 22.50    0
> 25.00    0
> 27.50    3 *
> 30.00    1 *
> 32.50    1 *
> 35.00    1 *
> 37.50    1 *
> 40.00    1 *
> 42.50    0
> 45.00    1 *
> 47.50    3 *
> 50.00    6 *
> 52.50    5 *
> 55.00    5 *
> 57.50    3 *
> 60.00    7 *
> 62.50    6 *
> 65.00    9 *
> 67.50   11 *
> 70.00   19 *
> 72.50   11 *
> 75.00    8 *
> 77.50   10 *
> 80.00   13 *
> 82.50    9 *
> 85.00   11 *
> 87.50   16 *
> 90.00   31 *
> 92.50    8 *
> 95.00   15 *
> 97.50 4784 ************************************************************
> 
> All of the false positives had significant numbers of both 0.01 and 0.99
> clues.  This seems to be a reappearance of the p(w) "cancellation disease"
> that we wormed around before by adding gobs of special-case code to Graham
> scoring.  Several of the fns also had this problem.  The outcome is like
> flipping a coin when this happens.  Note that f(w) doesn't have this problem
> (it's mostly an artifact of that p(w) artificially clamps probabilities, and
> so many words end up with probs at the extreme values).
> 
> Here are the means and variances of the training data scaled R values:
> 
> hammean  0.0315194110198 hamvar  0.0102392908745
> spammean 0.977596060549  spamvar 0.00629493144389
> 
> hammean  0.0289322128628 hamvar  0.00860263576484
> spammean 0.976784463455  spamvar 0.00703754635535
> 
> hammean  0.0292168061706 hamvar  0.00850922282341
> spammean 0.977330456163  spamvar 0.00656386045426
> 
> hammean  0.0292418489626 hamvar  0.00869327431102
> spammean 0.972324957985  spamvar 0.00968199258783
> 
> hammean  0.0266295579103 hamvar  0.00745682458391
> spammean 0.974432833096  spamvar 0.00865812701944
> 
> 
> Finally, here's the same thing (including exactly the same messages in the
> training and prediction sets) all over again, *except* using f(w) with a=0.1
> and x=0.5 (I mentioned a=0.5 above; I lowered it again for this run, and
> that did help the f-n rate, but not much):
> 
>     0.000   3.700
> 0 new false positives
> 37 new false negatives
> 
>     0.000   2.500
> 0 new false positives
> 25 new false negatives
> 
>     0.000   4.800
> 0 new false positives
> 48 new false negatives
> 
>     0.100   2.900
> 1 new false positives
> 29 new false negatives
> 
>     0.100   3.400
> 1 new false positives
> 34 new false negatives
> 
> total unique false pos 2
> total unique false neg 173
> average fp % 0.04
> average fn % 3.46
> 
> Ham distribution for all runs:
> 5000 items; mean 0.05; sample sdev 1.57
> * = 84 items
> 0.00 4991 ************************************************************
> 2.50    2 *
> 5.00    0
> 7.50    1 *
> 10.00    0
> 12.50    0
> 15.00    0
> 17.50    0
> 20.00    1 *
> 22.50    1 *
> 25.00    1 *
> 27.50    1 *
> 30.00    0
> 32.50    0
> 35.00    0
> 37.50    0
> 40.00    0
> 42.50    0
> 45.00    0
> 47.50    0
> 50.00    0
> 52.50    0
> 55.00    0
> 57.50    1 *
> 60.00    0
> 62.50    0
> 65.00    0
> 67.50    0
> 70.00    0
> 72.50    0
> 75.00    0
> 77.50    1 *
> 80.00    0
> 82.50    0
> 85.00    0
> 87.50    0
> 90.00    0
> 92.50    0
> 95.00    0
> 97.50    0
> 
> Spam distribution for all runs:
> 5000 items; mean 94.82; sample sdev 15.16
> * = 69 items
> 0.00    5 *
> 2.50    2 *
> 5.00    1 *
> 7.50    5 *
> 10.00    8 *
> 12.50    6 *
> 15.00   10 *
> 17.50    8 *
> 20.00    6 *
> 22.50   14 *
> 25.00   10 *
> 27.50   11 *
> 30.00    9 *
> 32.50    4 *
> 35.00   12 *
> 37.50   12 *
> 40.00   11 *
> 42.50    8 *
> 45.00   16 *
> 47.50   15 *
> 50.00   21 *
> 52.50   21 *
> 55.00   20 *
> 57.50   27 *
> 60.00   21 *
> 62.50   19 *
> 65.00   31 *
> 67.50   18 *
> 70.00   18 *
> 72.50   22 *
> 75.00   22 *
> 77.50   37 *
> 80.00   36 *
> 82.50   50 *
> 85.00   59 *
> 87.50   50 *
> 90.00   67 *
> 92.50   74 **
> 95.00   76 **
> 97.50 4138 ************************************************************
> 
> Too bizarre for me -- there may be a gross bug here, but the central-limit
> code is exactly the same in both cases, and the f(w) code is exactly the
> same as I've been using with good results for a few days.
> 
> hammean  0.083575863227  hamvar 0.0384359627039
> spammean 0.978952388668 spamvar 0.00372443106446
> 
> hammean  0.075515986459  hamvar 0.0331126798862
> spammean 0.97742506528  spamvar 0.00420869219347
> 
> hammean  0.0776481612081 hamvar 0.0338362239317
> spammean 0.978462136207 spamvar 0.00341505065601
> 
> hammean  0.07972071882   hamvar 0.0355833143776
> spammean 0.974508870296 spamvar 0.00530574730638
> 
> hammean  0.0713015705881 hamvar 0.0303987810665
> spammean 0.976734052416 spamvar 0.00468795224229
> 
> For whatever reason(s), I note that the ham variances are higher here than
> when using p(w), and the spam variances lower.  Perhaps that's just due to
> that f(w) doesn't have an artificial ham bias.  OTOH, the prediction set ham
> distribution is much tighter when using the unbiased f(w), while the spam
> distribution is much looser.
> 


From barry@python.org  Mon Sep 23 14:30:35 2002
From: barry@python.org (Barry A. Warsaw)
Date: Mon, 23 Sep 2002 09:30:35 -0400
Subject: [Spambayes] RE: Further Improvement 2
References: <15757.53755.736942.134794@anthem.wooz.org>
	<LNBBLJKPBEHFEDALKOLCAEOHBFAB.tim.one@comcast.net>
Message-ID: <15759.6011.711849.279645@anthem.wooz.org>


>>>>> "TP" == Tim Peters <tim.one@comcast.net> writes:

    TP> Barry, can you please fix this?  I knew nothing about the
    TP> email package before this project, and I don't understand my
    TP> own code anymore now that Guido has changed it to work with
    TP> some older version of the package.  We "fixed" the problems
    TP> with the use of 2.3's sets and heapq libraries just by
    TP> checking their source into the spambayes project.  Can you do
    TP> the same for your email pkg?

Ok, I just imported the email package into the spambayes directory.
I'll now go and update the various uses of the old API.  Of course,
this relies on that the spambayes directory is on sys.path, so that it
can find our imported email package.

(Please double check my edits.)

-Barry

From grobinson@transpose.com  Mon Sep 23 14:36:56 2002
From: grobinson@transpose.com (Gary Robinson)
Date: Mon, 23 Sep 2002 09:36:56 -0400
Subject: [Spambayes] RE: Central Limit Theorem??!!     :)
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCEAEBGAB.tim.one@comcast.net>
Message-ID: <B9B49138.16557%grobinson@transpose.com>

You noted that when the training set is smaller smaller values for a (what I
am now calling s ;)  ) work better.

The denominator of f(w) is (s+n). Clearly when there are smaller n's, s has
a bigger impact.

In an ideal Bayesian world, s would represent the strength of our a priori
beliefs and would be independent of n. But  here we are tuning s strictly
for performance, and there are all sorts of subtle interactions that depend
on n because of the generally non-pure approach we are taking.

So it doesn't surprise me that in this application s would be sensitive to
n.

It may be a reason to test on a constant n for now and derive a table for
ideal s values later...

Still seems better than p(w), which  is assuming s is 0! That's just another
decision for a particular constant s... there's no getting getting away from
the issue.

Gary


From guido@python.org  Mon Sep 23 14:43:18 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 23 Sep 2002 09:43:18 -0400
Subject: [Spambayes] RE: Further Improvement 2
In-Reply-To: Your message of "Mon, 23 Sep 2002 09:30:35 EDT."
             <15759.6011.711849.279645@anthem.wooz.org> 
References: <15757.53755.736942.134794@anthem.wooz.org>
	<LNBBLJKPBEHFEDALKOLCAEOHBFAB.tim.one@comcast.net>  
	<15759.6011.711849.279645@anthem.wooz.org> 
Message-ID: <200209231343.g8NDhJp06771@pcp02138704pcs.reston01.va.comcast.net>

> Ok, I just imported the email package into the spambayes directory.
> I'll now go and update the various uses of the old API.  Of course,
> this relies on that the spambayes directory is on sys.path, so that it
> can find our imported email package.

Um, is this the right approach?  I know I did that for heapq and sets,
because they're not available in any other way.  But the 2.3 email
package is available separately, and (I presume) will also be in
Python 2.2.2 when it comes out in a few weeks.  I'm uncomfortable with
including such a large amount of code that isn't really part of the
project -- tracking the changes in the email package will be more work
than I care for.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From gward@python.net  Mon Sep 23 14:45:08 2002
From: gward@python.net (Greg Ward)
Date: Mon, 23 Sep 2002 09:45:08 -0400
Subject: [Spambayes] Moving closer to Gary's ideal
In-Reply-To: <200209230327.g8N3RYo05107@pcp02138704pcs.reston01.va.comcast.net>
References: <LNBBLJKPBEHFEDALKOLCMENKBFAB.tim.one@comcast.net>
	<200209230327.g8N3RYo05107@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <20020923134508.GA27558@cthulhu.gerg.ca>

On 22 September 2002, Guido van Rossum said:
> That's impossible to know in my case.  Almost all of my mail goes
> through the SpamAssassin setup at python.org, which throws all spam
> away.  As a result I see maybe 1 spam for every 50 hams -- but that's
> not the spam/ham ratio seen by the MTA for guido@python.org.

One possibility is to use the spam half of the corpus I gathered last
week.  You'll still have the "multiple sources" problem after a fashion,
because the "Received" headers for the spam I gathered stop at
mail.python.org, but the ham from your inbox winds its way through your
ISP to your inbox.

Maybe we need a tool to strip off "Received" headers up to a
certain point -- eg. if Guido wants to train spambayes for his traffic,
but run it at the python.org MTA, then it shouldn't train on the
Received headers added by his ISP.  Or maybe there should be an option
to ignore "Received" headers up to server X.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Are we THERE yet?

From guido@python.org  Mon Sep 23 14:46:56 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 23 Sep 2002 09:46:56 -0400
Subject: [Spambayes] Moving closer to Gary's ideal
In-Reply-To: Your message of "Mon, 23 Sep 2002 09:45:08 EDT."
             <20020923134508.GA27558@cthulhu.gerg.ca> 
References: <LNBBLJKPBEHFEDALKOLCMENKBFAB.tim.one@comcast.net>
	<200209230327.g8N3RYo05107@pcp02138704pcs.reston01.va.comcast.net>  
	<20020923134508.GA27558@cthulhu.gerg.ca> 
Message-ID: <200209231346.g8NDkvB06828@pcp02138704pcs.reston01.va.comcast.net>

> One possibility is to use the spam half of the corpus I gathered last
> week.

How many msgs by now?

> You'll still have the "multiple sources" problem after a fashion,
> because the "Received" headers for the spam I gathered stop at
> mail.python.org, but the ham from your inbox winds its way through your
> ISP to your inbox.
> 
> Maybe we need a tool to strip off "Received" headers up to a
> certain point -- eg. if Guido wants to train spambayes for his traffic,
> but run it at the python.org MTA, then it shouldn't train on the
> Received headers added by his ISP.  Or maybe there should be an option
> to ignore "Received" headers up to server X.

Neat.  I'll think about it when I get my hands on your spam corpus.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From bkc@murkworks.com  Mon Sep 23 14:58:34 2002
From: bkc@murkworks.com (Brad Clements)
Date: Mon, 23 Sep 2002 09:58:34 -0400
Subject: [Spambayes] 2nd try at mean/dev cmp.py
In-Reply-To: <LNBBLJKPBEHFEDALKOLCIEPABFAB.tim.one@comcast.net>
References: <3D8DF19B.22946.3DD532D4@localhost>
Message-ID: <3D8EE5D5.29647.418F4779@localhost>

On 22 Sep 2002 at 17:02, Tim Peters wrote:

> > ham/spam mean difference: 48.81 49.31 +0.50
> 
> Cool!  Looks good -- check it in.

Checked in.. gee, that was easy.

Brad Clements,                bkc@murkworks.com   (315)268-1000
http://www.murkworks.com                          (315)268-9812 Fax
AOL-IM: BKClements


From sjoerd@acm.org  Mon Sep 23 15:04:58 2002
From: sjoerd@acm.org (Sjoerd Mullender)
Date: Mon, 23 Sep 2002 16:04:58 +0200
Subject: [Spambayes] RE: Further Improvement 2
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEMBBFAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCKEMBBFAB.tim.one@comcast.net> 
Message-ID: <200209231404.g8NE4wD01999@indus.ins.cwi.nl>

On Sat, Sep 21 2002 Tim Peters wrote:

> Since I got a big win without any effort <wink> by introducing a brand new
> "ignore probs that aren't at least this far from neutral" knob, that's the
> one I'm most inclined to play with right now.  There isn't a knob in
> existence that won't be played with, but especially large tests take
> significant wall-clock time to complete, and there's only so much testing
> one can do in a day.
> 
> Testers, "a" is already exposed via:
> 
> [Classifier]
> robinson_probability_a: 1.0
> 
> I think values nearer to 0 are most likely to be most interesting.

Here are my results.  The before run has options
"""
[Classifier]
use_robinson_probability: True
use_robinson_combining: True
max_discriminators: 1500
[TestDriver]
spam_cutoff: 0.50
"""
and the after run adds
"""
robinson_probability_a: 0.1
"""
to the set.

false positive percentages
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.405  0.540  lost   +33.33%
    0.270  0.135  won    -50.00%

won   1 times
tied  3 times
lost  1 times

total unique fp went from 5 to 5 tied
mean fp % went from 0.134952766532 to 0.134952766532 tied

false negative percentages
    1.571  1.571  tied
    2.618  2.094  won    -20.02%
    0.524  1.047  lost   +99.81%
    1.571  0.000  won   -100.00%
    2.094  1.571  won    -24.98%

won   3 times
tied  1 times
lost  1 times

total unique fn went from 16 to 12 won    -25.00%
mean fn % went from 1.67539267016 to 1.25654450262 won    -25.00%

And the before histograms:

Ham distribution for all runs:
3705 items; mean 23.12; sample sdev 7.44
* = 12 items
  0.00   0 
  2.50   0 
  5.00 105 *********
  7.50  69 ******
 10.00  98 *********
 12.50 306 **************************
 15.00 177 ***************
 17.50 352 ******************************
 20.00 497 ******************************************
 22.50 701 ***********************************************************
 25.00 506 *******************************************
 27.50 354 ******************************
 30.00 220 *******************
 32.50  99 *********
 35.00  93 ********
 37.50  49 *****
 40.00  25 ***
 42.50  26 ***
 45.00  16 **
 47.50   7 *
 50.00   1 *
 52.50   4 *
 55.00   0 

Spam distribution for all runs:
955 items; mean 68.33; sample sdev 8.74
* = 2 items
 32.50   0 
 35.00   1 *
 37.50   2 *
 40.00   0 
 42.50   0 
 45.00   4 **
 47.50   9 *****
 50.00   9 *****
 52.50  18 *********
 55.00  42 *********************
 57.50  73 *************************************
 60.00  88 ********************************************
 62.50 104 ****************************************************
 65.00 111 ********************************************************
 67.50 105 *****************************************************
 70.00 109 *******************************************************
 72.50  53 ***************************
 75.00  76 **************************************
 77.50  62 *******************************
 80.00  37 *******************
 82.50  23 ************
 85.00   8 ****
 87.50   6 ***
 90.00  15 ********
 92.50   0 
 95.00   0 
 97.50   0 

And finally the after histograms:

Ham distribution for all runs:
3705 items; mean 21.33; sample sdev 6.94
* = 12 items
  0.00   0 
  2.50   0 
  5.00 152 *************
  7.50  24 **
 10.00 197 *****************
 12.50 328 ****************************
 15.00 261 **********************
 17.50 498 ******************************************
 20.00 689 **********************************************************
 22.50 610 ***************************************************
 25.00 393 *********************************
 27.50 218 *******************
 30.00 117 **********
 32.50  79 *******
 35.00  71 ******
 37.50  25 ***
 40.00  11 *
 42.50  19 **
 45.00   6 *
 47.50   2 *
 50.00   4 *
 52.50   1 *
 55.00   0 

Spam distribution for all runs:
955 items; mean 72.04; sample sdev 10.31
* = 3 items
 35.00   0 
 37.50   1 *
 40.00   1 *
 42.50   1 *
 45.00   3 *
 47.50   6 **
 50.00  14 *****
 52.50  15 *****
 55.00  30 **********
 57.50  42 **************
 60.00  63 *********************
 62.50  61 *********************
 65.00  74 *************************
 67.50  91 *******************************
 70.00 102 **********************************
 72.50 123 *****************************************
 75.00  69 ***********************
 77.50  49 *****************
 80.00  45 ***************
 82.50  38 *************
 85.00  44 ***************
 87.50  42 **************
 90.00  17 ******
 92.50  12 ****
 95.00  12 ****
 97.50   0 


-- Sjoerd Mullender <sjoerd@acm.org>

From grobinson@transpose.com  Mon Sep 23 15:06:47 2002
From: grobinson@transpose.com (Gary Robinson)
Date: Mon, 23 Sep 2002 10:06:47 -0400
Subject: [Spambayes] RE: Central Limit Theorem??!!     :)
In-Reply-To: <B9B48552.16549%grobinson@transpose.com>
Message-ID: <B9B49837.16560%grobinson@transpose.com>

Error:

> When training on the spam side, don't use f(w), use: ln (1-f(w)).
> 
> When training on the ham side, don't use f(w), use: ln f(w).

is backwards. Do exactly the  reverse:

When training on the spam side, don't use f(w), use: ln f(w).

When training on the ham side, don't use f(w), use: ln (1-f(w)).

Also, those numbers will be negative. If you want positive numbers, use -(ln
f(w)) and -(ln (1-f(w))). The minus sign will make no difference whatsoever
to the results.




--Gary


-- 
Gary Robinson
CEO
Transpose, LLC
grobinson@transpose.com
207-942-3463
http://www.emergentmusic.com
http://radio.weblogs.com/0101454


> From: Gary Robinson <grobinson@transpose.com>
> Date: Mon, 23 Sep 2002 08:46:10 -0400
> To: Tim Peters <tim.one@comcast.net>, SpamBayes <spambayes@python.org>, Greg
> Louis <glouis@dynamicro.on.ca>
> Subject: Re: [Spambayes] RE: Central Limit Theorem??!!     :)
> 
> OK. That's fascinating.
> 
> Remember that the multiplicative method in S, which calculates the geometric
> mean of the f(w)'s, stresses the MOST extreme values more than the less
> extreme ones. The more extreme the value is, the more it is stressed. Very
> extreme ones are stressed very highly, in an exponentially compounding way if
> there are several really extreme ones.
> 
> That's why it's the basis for that 1971 optimality theorem that I kept trying
> to invoke more strongly, but which is still invoked to a degree in S.
> 
> That's the reasoning behind S in the first place, and why it works so well,
> and it also happens in Graham's original approach (but only on one side), but
> we are completely losing it in R.
> 
> In R, we are trading that very powerful multiplicative effect away in order to
> get the benefit of real parametric statistics. ALSO a very powerful technique
> but apparently slightly less powerful in this application -- at least when
> used alone.
> 
> If there is a performance loss in R (and there are no remaining coding
> errors), I am confident that that's why.
> 
> THERE IS A POTENTIAL FIX FOR THIS LOSSAGE, so that we can theoretically get
> the best of both totally different techniques.
> 
> When training on the spam side, don't use f(w), use: ln (1-f(w)).
> 
> When training on the ham side, don't use f(w), use: ln f(w).
> 
> Same when testing. Don't add the f(w)'s increating the sample mean; add the
> expressions above, and divide by n. So the spam side uses ln (1-f(w)) both for
> training and calculating the sample means, and the ham side uses ln f(w) for
> both.
> 
> As we've discussed, averaging the ln's is the same thing as a geometric mean
> if you then subsequently raise e to the power of that computed average. But we
> don't do that last step here. We just feed the arithmetic mean of the ln's
> into the z-score calcs.
> 
> This *should* bring us the benefits of the multiplicative approach and of the
> parametric stats approach at the same time.
> 
> The downside of this is that the ln's make such a skewed distribution that it
> should take a bigger n to make the central limit theorem kick in. BUT, OTOH,
> it WILL kick in, and something like 30 may really still be enough (it usually
> kicks in at significantly smaller numbers). And you've also successfully used
> n=150 with f(w) and that is DEFINITELY enough.
> 
> The other downside is that it just seems like a bit  of a wild thing to do and
> sometimes when you do wild things, strange reasons emerge why they won't work.
> But I really can't see any at this point as long as n is big enough that the
> sample means take on a normal distribution.
> 
> THANKS for doing all the coding work to test this idea!!!!  :)
> 
> Gary
> 
> 
> 
> 
> 
>> 
>> This was using 30 extremes, and using Graham's p(w) (complete with hambias
>> 2, minprob .01 and maxprob .99).  The f-n rate was more than 10x worse using
>> f(w) with a=x=0.5, and I have no idea why yet (and we're *generally* having
>> problems with f-n rates on smaller training sets when using f(w), whether
>> using the central-limit scoring, or Gary's previous scoring; perhaps 'a'
>> needs to be much smaller than 0.5 -- there's too much to test here).
>> 
>> Here are the aggregate scaled R values (clamped to [-20, 20], and then
>> scaled linearly into [0, 1]):
>> 
>> Ham distribution for all runs:
>> 5000 items; mean 0.35; sample sdev 3.62
>> * = 82 items
>> 0.00 4907 ************************************************************
>> 2.50   13 *
>> 5.00   14 *
>> 7.50   12 *
>> 10.00    4 *
>> 12.50   11 *
>> 15.00    6 *
>> 17.50    7 *
>> 20.00    4 *
>> 22.50    2 *
>> 25.00    2 *
>> 27.50    3 *
>> 30.00    0
>> 32.50    1 *
>> 35.00    3 *
>> 37.50    0
>> 40.00    1 *
>> 42.50    1 *
>> 45.00    1 *
>> 47.50    1 *
>> 50.00    1 *
>> 52.50    0
>> 55.00    1 *
>> 57.50    0
>> 60.00    0
>> 62.50    1 *
>> 65.00    2 *
>> 67.50    0
>> 70.00    0
>> 72.50    0
>> 75.00    0
>> 77.50    0
>> 80.00    0
>> 82.50    0
>> 85.00    0
>> 87.50    0
>> 90.00    0
>> 92.50    0
>> 95.00    0
>> 97.50    2 *
>> 
>> Spam distribution for all runs:
>> 5000 items; mean 98.97; sample sdev 5.87
>> * = 80 items
>> 0.00    0
>> 2.50    0
>> 5.00    0
>> 7.50    0
>> 10.00    0
>> 12.50    0
>> 15.00    0
>> 17.50    0
>> 20.00    1 *
>> 22.50    0
>> 25.00    0
>> 27.50    3 *
>> 30.00    1 *
>> 32.50    1 *
>> 35.00    1 *
>> 37.50    1 *
>> 40.00    1 *
>> 42.50    0
>> 45.00    1 *
>> 47.50    3 *
>> 50.00    6 *
>> 52.50    5 *
>> 55.00    5 *
>> 57.50    3 *
>> 60.00    7 *
>> 62.50    6 *
>> 65.00    9 *
>> 67.50   11 *
>> 70.00   19 *
>> 72.50   11 *
>> 75.00    8 *
>> 77.50   10 *
>> 80.00   13 *
>> 82.50    9 *
>> 85.00   11 *
>> 87.50   16 *
>> 90.00   31 *
>> 92.50    8 *
>> 95.00   15 *
>> 97.50 4784 ************************************************************
>> 
>> All of the false positives had significant numbers of both 0.01 and 0.99
>> clues.  This seems to be a reappearance of the p(w) "cancellation disease"
>> that we wormed around before by adding gobs of special-case code to Graham
>> scoring.  Several of the fns also had this problem.  The outcome is like
>> flipping a coin when this happens.  Note that f(w) doesn't have this problem
>> (it's mostly an artifact of that p(w) artificially clamps probabilities, and
>> so many words end up with probs at the extreme values).
>> 
>> Here are the means and variances of the training data scaled R values:
>> 
>> hammean  0.0315194110198 hamvar  0.0102392908745
>> spammean 0.977596060549  spamvar 0.00629493144389
>> 
>> hammean  0.0289322128628 hamvar  0.00860263576484
>> spammean 0.976784463455  spamvar 0.00703754635535
>> 
>> hammean  0.0292168061706 hamvar  0.00850922282341
>> spammean 0.977330456163  spamvar 0.00656386045426
>> 
>> hammean  0.0292418489626 hamvar  0.00869327431102
>> spammean 0.972324957985  spamvar 0.00968199258783
>> 
>> hammean  0.0266295579103 hamvar  0.00745682458391
>> spammean 0.974432833096  spamvar 0.00865812701944
>> 
>> 
>> Finally, here's the same thing (including exactly the same messages in the
>> training and prediction sets) all over again, *except* using f(w) with a=0.1
>> and x=0.5 (I mentioned a=0.5 above; I lowered it again for this run, and
>> that did help the f-n rate, but not much):
>> 
>>     0.000   3.700
>> 0 new false positives
>> 37 new false negatives
>> 
>>     0.000   2.500
>> 0 new false positives
>> 25 new false negatives
>> 
>>     0.000   4.800
>> 0 new false positives
>> 48 new false negatives
>> 
>>     0.100   2.900
>> 1 new false positives
>> 29 new false negatives
>> 
>>     0.100   3.400
>> 1 new false positives
>> 34 new false negatives
>> 
>> total unique false pos 2
>> total unique false neg 173
>> average fp % 0.04
>> average fn % 3.46
>> 
>> Ham distribution for all runs:
>> 5000 items; mean 0.05; sample sdev 1.57
>> * = 84 items
>> 0.00 4991 ************************************************************
>> 2.50    2 *
>> 5.00    0
>> 7.50    1 *
>> 10.00    0
>> 12.50    0
>> 15.00    0
>> 17.50    0
>> 20.00    1 *
>> 22.50    1 *
>> 25.00    1 *
>> 27.50    1 *
>> 30.00    0
>> 32.50    0
>> 35.00    0
>> 37.50    0
>> 40.00    0
>> 42.50    0
>> 45.00    0
>> 47.50    0
>> 50.00    0
>> 52.50    0
>> 55.00    0
>> 57.50    1 *
>> 60.00    0
>> 62.50    0
>> 65.00    0
>> 67.50    0
>> 70.00    0
>> 72.50    0
>> 75.00    0
>> 77.50    1 *
>> 80.00    0
>> 82.50    0
>> 85.00    0
>> 87.50    0
>> 90.00    0
>> 92.50    0
>> 95.00    0
>> 97.50    0
>> 
>> Spam distribution for all runs:
>> 5000 items; mean 94.82; sample sdev 15.16
>> * = 69 items
>> 0.00    5 *
>> 2.50    2 *
>> 5.00    1 *
>> 7.50    5 *
>> 10.00    8 *
>> 12.50    6 *
>> 15.00   10 *
>> 17.50    8 *
>> 20.00    6 *
>> 22.50   14 *
>> 25.00   10 *
>> 27.50   11 *
>> 30.00    9 *
>> 32.50    4 *
>> 35.00   12 *
>> 37.50   12 *
>> 40.00   11 *
>> 42.50    8 *
>> 45.00   16 *
>> 47.50   15 *
>> 50.00   21 *
>> 52.50   21 *
>> 55.00   20 *
>> 57.50   27 *
>> 60.00   21 *
>> 62.50   19 *
>> 65.00   31 *
>> 67.50   18 *
>> 70.00   18 *
>> 72.50   22 *
>> 75.00   22 *
>> 77.50   37 *
>> 80.00   36 *
>> 82.50   50 *
>> 85.00   59 *
>> 87.50   50 *
>> 90.00   67 *
>> 92.50   74 **
>> 95.00   76 **
>> 97.50 4138 ************************************************************
>> 
>> Too bizarre for me -- there may be a gross bug here, but the central-limit
>> code is exactly the same in both cases, and the f(w) code is exactly the
>> same as I've been using with good results for a few days.
>> 
>> hammean  0.083575863227  hamvar 0.0384359627039
>> spammean 0.978952388668 spamvar 0.00372443106446
>> 
>> hammean  0.075515986459  hamvar 0.0331126798862
>> spammean 0.97742506528  spamvar 0.00420869219347
>> 
>> hammean  0.0776481612081 hamvar 0.0338362239317
>> spammean 0.978462136207 spamvar 0.00341505065601
>> 
>> hammean  0.07972071882   hamvar 0.0355833143776
>> spammean 0.974508870296 spamvar 0.00530574730638
>> 
>> hammean  0.0713015705881 hamvar 0.0303987810665
>> spammean 0.976734052416 spamvar 0.00468795224229
>> 
>> For whatever reason(s), I note that the ham variances are higher here than
>> when using p(w), and the spam variances lower.  Perhaps that's just due to
>> that f(w) doesn't have an artificial ham bias.  OTOH, the prediction set ham
>> distribution is much tighter when using the unbiased f(w), while the spam
>> distribution is much looser.
>> 


From gward@python.net  Mon Sep 23 15:15:19 2002
From: gward@python.net (Greg Ward)
Date: Mon, 23 Sep 2002 10:15:19 -0400
Subject: [Spambayes] Moving closer to Gary's ideal
In-Reply-To: <200209231346.g8NDkvB06828@pcp02138704pcs.reston01.va.comcast.net>
References: <LNBBLJKPBEHFEDALKOLCMENKBFAB.tim.one@comcast.net>
	<200209230327.g8N3RYo05107@pcp02138704pcs.reston01.va.comcast.net>
	<20020923134508.GA27558@cthulhu.gerg.ca>
	<200209231346.g8NDkvB06828@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <20020923141519.GA27648@cthulhu.gerg.ca>

On 23 September 2002, Guido van Rossum said:
> > One possibility is to use the spam half of the corpus I gathered last
> > week.
> 
> How many msgs by now?

Here are the all-but-final stats:

  dsn                   1658 messages      8654 kB
  spam                  1887 messages     16640 kB
  ham                   3840 messages     11342 kB
  virus                  988 messages    120754 kB

(dsn = "delivery status notification", aka bounces and delay notifications)

This is from harvesting for one week (2002-09-11 to 2002-09-18), with
some manual sorting and cleaning.  Also, I deleted 90% of the DSNs at
random, because they were by far the largest source of email -- so the
1658 there is just 10% of what was actually received.  No doubt test
runs will reveal more FPs and FNs, but I've done all the manual
cleaning/sorting I can stand to do for now.  (Hence "all-but-final"
stats.)

Those of you with logins on mail.python.org can find the corpus in
/usr/local/var/harvest; there are four Maildir folders there.  (Hint:
look in spam/cur for the spam.  Or just run "mutt -f spam" -- mutt groks
Maildir fine.)

The original, untouched, uncleaned harvest is in
/usr/local/var/harvest-20020911-20020918.tar.gz -- that's just under 300
MB of email.

I'd like to put some tarballs on the mail.python.org web server, but
password-protected -- I want to know who has access to this corpus.
This is not so important if I only put the spam and viruses on the web,
but the ham folder contains some semi-private stuff (eg. a bit of
postmaster traffic, posts to yours and Barry's "extracurricular
activity" lists [dc-ci, dc-bass, etc.]).  And the dsn folder reveals all
sorts of stuff about who's subscribed to python.org lists.  There also
seems to be some personal email in the ham -- not sure how that snuck
in, since I was careful to detect and *not* save it.  Haven't
investigated yet.  Anyways, at least take a look at the ham and spam
folders and lemme know what you think.  If you spot FPs or FNs let me
know.  Ditto if you want to delete anything.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
All the world's a stage and most of us are desperately unrehearsed.

From barry@python.org  Mon Sep 23 15:19:44 2002
From: barry@python.org (Barry A. Warsaw)
Date: Mon, 23 Sep 2002 10:19:44 -0400
Subject: [Spambayes] RE: Further Improvement 2
References: <LNBBLJKPBEHFEDALKOLCOENABFAB.tim.one@comcast.net>
	<200209220632.g8M6W3e27694@pcp02138704pcs.reston01.va.comcast.net>
	<200209220706.g8M765Q27830@pcp02138704pcs.reston01.va.comcast.net>
	<v0311078bb9b3b588f0d7@[192.168.0.2]>
	<200209230248.g8N2mkK04947@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <15759.8960.681900.927322@anthem.wooz.org>


>>>>> "GvR" == Guido van Rossum <guido@python.org> writes:

    GvR> I think the new email package should work without change (or
    GvR> with very little change) with 2.2.1.  In fact I believe the
    GvR> intention is to backport the new email package to 2.2.2.  But
    GvR> I don't want to import the entire email package into the
    GvR> spambayes CVS tree.

Wauugh!  I just did this. :(

I will follow up in a separate message about backporting.

-Barry

From skip@pobox.com  Mon Sep 23 15:42:08 2002
From: skip@pobox.com (Skip Montanaro)
Date: Mon, 23 Sep 2002 09:42:08 -0500
Subject: [Spambayes] Re: [Spambayes-checkins] spambayes tokenizer.py,1.31,1.32
In-Reply-To: <E17tTIT-0006bQ-00@usw-pr-cvs1.sourceforge.net>
References: <E17tTIT-0006bQ-00@usw-pr-cvs1.sourceforge.net>
Message-ID: <15759.10304.883189.899985@12-248-11-90.client.attbi.com>

    Barry> Use the email 2.3 API, get_type() and friends ->
    Barry> get_content_type() and friends.  The latter always returns a
    Barry> content type string, never None.

Now I'm confused.  I just checked in a change to tokenizer.py that changed
a get_content_type() call to get_type().  The comparison is against
"application/octet-stream".  Should I have used something else?

Skip

From skip@pobox.com  Mon Sep 23 15:52:39 2002
From: skip@pobox.com (Skip Montanaro)
Date: Mon, 23 Sep 2002 09:52:39 -0500
Subject: [Spambayes] RE: Further Improvement 2
In-Reply-To: <200209231404.g8NE4wD01999@indus.ins.cwi.nl>
References: <LNBBLJKPBEHFEDALKOLCKEMBBFAB.tim.one@comcast.net>
        <200209231404.g8NE4wD01999@indus.ins.cwi.nl>
Message-ID: <15759.10935.189018.319822@12-248-11-90.client.attbi.com>


    Sjoerd> On Sat, Sep 21 2002 Tim Peters wrote:

    >> [Classifier]
    >> robinson_probability_a: 1.0
    >> 
    >> I think values nearer to 0 are most likely to be most interesting.

    Sjoerd> Here are my results.

We've been doing a lot of manual knob turning trying to find the best
settings for various options.  It seems like we need an optimization tool of
some sort to automate this process.  I know by suggesting it, I'm implicitly
volunteering to create it, but I don't have the time to do this.

Skip


From barry@python.org  Mon Sep 23 15:58:02 2002
From: barry@python.org (Barry A. Warsaw)
Date: Mon, 23 Sep 2002 10:58:02 -0400
Subject: [Spambayes] RE: Further Improvement 2
References: <15757.53755.736942.134794@anthem.wooz.org>
	<LNBBLJKPBEHFEDALKOLCAEOHBFAB.tim.one@comcast.net>
	<15759.6011.711849.279645@anthem.wooz.org>
	<200209231343.g8NDhJp06771@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <15759.11258.301930.985013@anthem.wooz.org>


>>>>> "GvR" == Guido van Rossum <guido@python.org> writes:

    >> Ok, I just imported the email package into the spambayes
    >> directory.  I'll now go and update the various uses of the old
    >> API.  Of course, this relies on that the spambayes directory is
    >> on sys.path, so that it can find our imported email package.

    GvR> Um, is this the right approach?

Indeed, it may not be.  I was reacting to Tim's message but it's taken
me several hours this morning to catch up on email, so I missed your
anti-recommendation until it was "too late".
    
    GvR> I know I did that for heapq and sets, because they're not
    GvR> available in any other way.  But the 2.3 email package is
    GvR> available separately, and (I presume) will also be in Python
    GvR> 2.2.2 when it comes out in a few weeks.  I'm uncomfortable
    GvR> with including such a large amount of code that isn't really
    GvR> part of the project -- tracking the changes in the email
    GvR> package will be more work than I care for.

If we decide to backport email to Python 2.2.2 (see a separate thread
to mimelib-devel on this subject), then I agree we should cvsrm the
email package from spambayes.

As it is, we could still do that and just require spambayesers to
distutils install email 2.3.1, although you have to do some
contortions to get that to override the distributed email package in
Python 2.2.1 :(

-Barry

From grobinson@transpose.com  Mon Sep 23 16:02:45 2002
From: grobinson@transpose.com (Gary Robinson)
Date: Mon, 23 Sep 2002 11:02:45 -0400
Subject: [Spambayes] Re: Practical Applications
Message-ID: <B9B4A555.16572%grobinson@transpose.com>

Suppose I wanted to build a practical application on spambayes that
integrated with Outlook but it just couldn't be done as elegantly with
separate Python code communicating with Outlook via COM.

Would the open source license spambayes is using allow me to translate
spambayes into Visual Basic for Applications?????

I love Python and hate the idea of VBA but whatever gets the job done
best...  

also, if translation is allowable, about how many lines of code does
spambayes have?

Just playing with an interesting idea...


--Gary


-- 
Gary Robinson
CEO
Transpose, LLC
grobinson@transpose.com
207-942-3463
http://www.emergentmusic.com
http://radio.weblogs.com/0101454



From tim.one@comcast.net  Mon Sep 23 16:04:18 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 23 Sep 2002 11:04:18 -0400
Subject: [Spambayes] RE: Central Limit Theorem??!!     :)
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCEABBGAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCEEBKBGAB.tim.one@comcast.net>

[Gary Robinson]
>> ...
>> Now let's generate a population of f(w)'s as follows.
>>
>> We go through all the spam emails one by one, adding the 30 most
>> extreme f(w)'s to our population (only making one addition for each
>> unique occurrence of a word in the population).

[Tim]
> I'm unclear on the intent of the parenthetical comment.  Suppose I have 1
> spam with 500 occurrences of Nigeria, and 99 spam with 1 occurrence of
> Nigeria each, and no other spam contains Nigeria.  Is f('Nigeria') to be
> added in 1 time total, or 100 times total?  The code I have does
> the latter.

When I woke up, it occurred to me that the former (add f('Nigeria') only
once) made a lot more sense.  Whether it makes more sense or not <wink>,
changing the code to add it only once clearly reduced the false negative
rate in test runs.


From guido@python.org  Mon Sep 23 16:11:17 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 23 Sep 2002 11:11:17 -0400
Subject: [Spambayes] Re: Practical Applications
In-Reply-To: Your message of "Mon, 23 Sep 2002 11:02:45 EDT."
             <B9B4A555.16572%grobinson@transpose.com> 
References: <B9B4A555.16572%grobinson@transpose.com> 
Message-ID: <200209231511.g8NFBHQ09856@pcp02138704pcs.reston01.va.comcast.net>

> Suppose I wanted to build a practical application on spambayes that
> integrated with Outlook but it just couldn't be done as elegantly with
> separate Python code communicating with Outlook via COM.
> 
> Would the open source license spambayes is using allow me to translate
> spambayes into Visual Basic for Applications?????

Sure.  I'd consider it a derivative work so it's still covered by the
PSF license.  Which allows you to distribute it as a binary without
source, though I'd encourage you to also distribute the source, as a
way of giving back to the open source community (not that you haven't
given back already by your math insights).

> I love Python and hate the idea of VBA but whatever gets the job done
> best...  
> 
> also, if translation is allowable, about how many lines of code does
> spambayes have?

Check it out and do a wordcount.  I count 6231 lines in the current
code base, but the tokenizer and classifier together are under 1700
lines total, and the rest is test drivers and deployment code.  OTOH
I don't know what kind of email message parsing VB has, and here I'm
not counting the Python email package.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From barry@python.org  Mon Sep 23 16:15:20 2002
From: barry@python.org (Barry A. Warsaw)
Date: Mon, 23 Sep 2002 11:15:20 -0400
Subject: [Spambayes] Re: [Spambayes-checkins] spambayes
	tokenizer.py,1.31,1.32
References: <E17tTIT-0006bQ-00@usw-pr-cvs1.sourceforge.net>
	<15759.10304.883189.899985@12-248-11-90.client.attbi.com>
Message-ID: <15759.12296.535226.109372@anthem.wooz.org>


>>>>> "SM" == Skip Montanaro <skip@pobox.com> writes:

    Barry> Use the email 2.3 API, get_type() and friends ->
    Barry> get_content_type() and friends.  The latter always returns
    Barry> a content type string, never None.

    SM> Now I'm confused.  I just checked in a change to tokenizer.py
    SM> that changed a get_content_type() call to get_type().  The
    SM> comparison is against "application/octet-stream".  Should I
    SM> have used something else?

get_type(), get_main_type(), and get_subtype() are (silently)
deprecated in favor of get_content_type(), get_content_maintype(), and
get_content_subtype().

I was behind in reading the messages to this list, so if you want to
revert my cvs import of the email package, and until/unless we
backport email to Python 2.2.2, then yes, you should use get_type().

FWIW, get_type() is broken because when looking only at a single
subpart, you can't know what the default type is (so can't pass in a
proper failobj).  It's only by knowing that a subpart is inside a
multipart/digest for instance, that the subpart's default type is
message/rfc822 instead of text/plain.  The Parser handles this
properly when creating the message structure from a string/file, but
then every subpart will always have a content type, so in that case
get_type()'s failobj would never be used.  Thus the API change.

-Barry

From tim.one@comcast.net  Mon Sep 23 16:25:25 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 23 Sep 2002 11:25:25 -0400
Subject: [Spambayes] Re: Practical Applications
In-Reply-To: <B9B4A555.16572%grobinson@transpose.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEBMBGAB.tim.one@comcast.net>

[Gary Robinson]
> Suppose I wanted to build a practical application on spambayes that
> integrated with Outlook but it just couldn't be done as elegantly with
> separate Python code communicating with Outlook via COM.
>
> Would the open source license spambayes is using allow me to translate
> spambayes into Visual Basic for Applications?????

Sure.

> I love Python and hate the idea of VBA but whatever gets the job done
> best...
>
> also, if translation is allowable, about how many lines of code does
> spambayes have?

Note that it would take a great many more lines of Visual Basic.

C:\Code\spambayes>wc *.py
    292    1296   11204 Options.py
    243     862    8603 TestDriver.py
    173     639    6355 Tester.py
    221     597    5874 cdb.py
    733    3939   30411 classifier.py
     98     346    2527 cmp.py
    314    1072    9089 hammie.py
    142     438    3977 hammiesrv.py
    253    1853   11434 heapq.py
     12      43     297 hmm.py
    101     448    3404 loosecksum.py
     89     257    2227 mboxcount.py
    174     457    4209 mboxtest.py
     91     285    2729 mboxutils.py
    172     736    6177 neilfilter.py
     59     156    1490 neiltrain.py
     10      29     250 pik.py
    582    2286   22269 pop3proxy.py
     72     166    1712 randomtrain.py
     96     361    2777 rates.py
    171     643    5094 rebal.py
    498    1954   16656 sets.py
      8      11     156 setup.py
    100     303    2445 split.py
    110     367    2909 splitn.py
    117     386    3203 splitndirs.py
     23      49     575 temp.py
    213     685    5980 timcv.py
    122     334    3141 timtest.py
   1053    5754   40349 tokenizer.py
     99     272    2794 unheader.py
   6441   27024  220317 total

C:\Code\spambayes>

The only "core code" there is:

   1053    5754   40349 tokenizer.py
    733    3939   30411 classifier.py

These have huge comment-line to code ratios, because comment blocks discuss
dozens of alternatives, and even include some full "this way and that way"
comparative test-run output.

The other files are in support of data massaging, test driving, and
analyzing test output.

Oops:  sets.py and heapq.py implement set and priority queue datatypes.

My classifier.py is bigger than anyone else's at the moment, because it
contains full implementations of Graham's scheme, the scheme on your
webpage, and the central-limit scheme.

pop3proxy.py is a very cool way to hook up any mail agent to the classifier
(you point your mail reader at *it*, and it sucks down the email from your
real pop server, adds a header line with the classifier's spam judgement,
and passes the messages so augmented onto your mail reader).


From guido@python.org  Mon Sep 23 16:24:41 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 23 Sep 2002 11:24:41 -0400
Subject: [Spambayes] Re: [Spambayes-checkins] spambayes
	tokenizer.py,1.31,1.32
In-Reply-To: Your message of "Mon, 23 Sep 2002 11:15:20 EDT."
             <15759.12296.535226.109372@anthem.wooz.org> 
References: <E17tTIT-0006bQ-00@usw-pr-cvs1.sourceforge.net>
	<15759.10304.883189.899985@12-248-11-90.client.attbi.com>  
	<15759.12296.535226.109372@anthem.wooz.org> 
Message-ID: <200209231524.g8NFOfL09933@pcp02138704pcs.reston01.va.comcast.net>

> get_type(), get_main_type(), and get_subtype() are (silently)
> deprecated in favor of get_content_type(), get_content_maintype(), and
> get_content_subtype().
> 
> I was behind in reading the messages to this list, so if you want to
> revert my cvs import of the email package, and until/unless we
> backport email to Python 2.2.2, then yes, you should use get_type().

Let's wait until it's decided that email will be backported to 2.2.2.

> FWIW, get_type() is broken because when looking only at a single
> subpart, you can't know what the default type is (so can't pass in a
> proper failobj).  It's only by knowing that a subpart is inside a
> multipart/digest for instance, that the subpart's default type is
> message/rfc822 instead of text/plain.  The Parser handles this
> properly when creating the message structure from a string/file, but
> then every subpart will always have a content type, so in that case
> get_type()'s failobj would never be used.  Thus the API change.

I don't understand the last two sentences.  "The Parser handles
this..." -- does that refer to the fixed (in 2.3) Parser or to the old
Parser in a certain context?

AFAICT the *only* place where the default isn't text/plain is inside
multipart/digest, and then it's message/rfc822.  So code that asks
whether part.get_type()=='multipart/alternative' is in fact not
broken.  Code that asks whether
part.get_type('text/plain')=='text/plain' but where it is known that
part is either the top-level or inside a multipart/alternative is also
safe.

Someone who knows tokenizer.py better than me needs to do an audit of
these.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From barry@python.org  Mon Sep 23 16:35:05 2002
From: barry@python.org (Barry A. Warsaw)
Date: Mon, 23 Sep 2002 11:35:05 -0400
Subject: [Spambayes] Re: [Spambayes-checkins] spambayes
	tokenizer.py,1.31,1.32
References: <E17tTIT-0006bQ-00@usw-pr-cvs1.sourceforge.net>
	<15759.10304.883189.899985@12-248-11-90.client.attbi.com>
	<15759.12296.535226.109372@anthem.wooz.org>
	<200209231524.g8NFOfL09933@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <15759.13481.712243.399398@anthem.wooz.org>


>>>>> "GvR" == Guido van Rossum <guido@python.org> writes:

    GvR> Let's wait until it's decided that email will be backported
    GvR> to 2.2.2.

Cool.

    >> FWIW, get_type() is broken because when looking only at a
    >> single subpart, you can't know what the default type is (so
    >> can't pass in a proper failobj).  It's only by knowing that a
    >> subpart is inside a multipart/digest for instance, that the
    >> subpart's default type is message/rfc822 instead of text/plain.
    >> The Parser handles this properly when creating the message
    >> structure from a string/file, but then every subpart will
    >> always have a content type, so in that case get_type()'s
    >> failobj would never be used.  Thus the API change.

    GvR> I don't understand the last two sentences.  "The Parser
    GvR> handles this..." -- does that refer to the fixed (in 2.3)
    GvR> Parser or to the old Parser in a certain context?

The new Parser, because it'll make a set_default_type() call when it
finds a subpart inside a multipart/digest.

    GvR> AFAICT the *only* place where the default isn't text/plain is
    GvR> inside multipart/digest, and then it's message/rfc822.  So
    GvR> code that asks whether
    GvR> part.get_type()=='multipart/alternative' is in fact not
    GvR> broken.

That comparison will work.
    
    GvR> Code that asks whether
    GvR> part.get_type('text/plain')=='text/plain' but where it is
    GvR> known that part is either the top-level or inside a
    GvR> multipart/alternative is also safe.

Right.  If you /know/ where the subpart lives in the hierarchy, then
you can pass the right default into failobj.

-Barry

From popiel@wolfskeep.com  Mon Sep 23 16:47:03 2002
From: popiel@wolfskeep.com (T. Alexander Popiel)
Date: Mon, 23 Sep 2002 08:47:03 -0700
Subject: [Spambayes] RE: Central Limit Theorem??!! :) 
In-Reply-To: Message from Tim Peters <tim.one@comcast.net> 
	of "Sun, 22 Sep 2002 22:16:12 EDT."
	<LNBBLJKPBEHFEDALKOLCKEPHBFAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCKEPHBFAB.tim.one@comcast.net> 
Message-ID: <20020923154703.2C816F57F@cashew.wolfskeep.com>

In message:  <LNBBLJKPBEHFEDALKOLCKEPHBFAB.tim.one@comcast.net>
             Tim Peters <tim.one@comcast.net> writes:
>[Gary Robinson]
>> Say we have decided only to look at the 30 most extreme f(w)'s in every
>> email. I am choosing 30 because it is usually enough for the central
>> limit theorem to really kick in, but we may get better performance with
>> a greater number of extrema.
>>
>> Now let's generate a population of f(w)'s as follows.
>>
>> We go through all the spam emails one by one, adding the 30 most extreme
>> f(w)'s to our population (only making one addition for each unique
>> occurrence of a word in the population).
>
>Just noting that this is a two-pass process:  we have to go through all the
>spam and all the ham in the training set first, in order to compute f(w).
>Then we go though them again to tally the extremes (which ones are extreme
>can't be known before the first pass completes).

This can still be done in one pass.  While calculating mean and stddev,
keep a running list of the 30 highest and 30 lowest f(w), then at the end
figure out which of those 60 are most extreme.  (Assuming, of course, that
I'm properly interpreting what is meant by 'extreme'.)

>> Any elementary statistics book contains straightforward arithmetic
>> techniques for computing these numbers.
>
>That turns out to be a problem.  Over time, we want to incrementally train
>the classifier, and we sure don't want the user to have to save all the
>training data they've ever used.

The formulas in elementary statistics books certainly aren't good for
this, but the recurrence relations in Knuth's TAOCP vol 2 (Seminumerical
Algorithms) are quite good for the incremental approach.  Quoting from
4.2.2, equations 15 & 16:

  (15)  M[1] = x[1],  M[k] = M[k-1] + ((x[k] - M[k-1]) / k)
  (16)  S[1] = 0,     S[k] = S[k-1] + ((x[k] - M[k-1]) * (x[k] - M[k]))

  for a population x of size n, 2 <= k <= n, where
  sigma = sqrt(S[n]/(n-1)).

These formulas can be run forwards or backwards with equal grace, and
(if coded as written without 'simplification') handle round-off errors
well, too.  (If greater precision is needed, the precision of the sums
can be doubled through use of the formulas in exercise 19 of the same
chapter.)

IANAS, but I trust Knuth. ;-)

- Alex

From popiel@wolfskeep.com  Mon Sep 23 16:53:47 2002
From: popiel@wolfskeep.com (T. Alexander Popiel)
Date: Mon, 23 Sep 2002 08:53:47 -0700
Subject: [Spambayes] RE: Central Limit Theorem??!! :) 
In-Reply-To: Message from "T. Alexander Popiel" <popiel@wolfskeep.com> 
	of "Mon, 23 Sep 2002 08:47:03 PDT."
	<20020923154703.2C816F57F@cashew.wolfskeep.com> 
References: <LNBBLJKPBEHFEDALKOLCKEPHBFAB.tim.one@comcast.net>
	<20020923154703.2C816F57F@cashew.wolfskeep.com> 
Message-ID: <20020923155347.626D2F57F@cashew.wolfskeep.com>

In message:  <20020923154703.2C816F57F@cashew.wolfskeep.com>
             "T. Alexander Popiel" <popiel@wolfskeep.com> writes:
>
>  (15)  M[1] = x[1],  M[k] = M[k-1] + ((x[k] - M[k-1]) / k)
>  (16)  S[1] = 0,     S[k] = S[k-1] + ((x[k] - M[k-1]) * (x[k] - M[k]))
>
>  for a population x of size n, 2 <= k <= n, where
>  sigma = sqrt(S[n]/(n-1)).

It just occured to me that I neglected to mention that Knuth's S and
Gary Robinson's S are different; Gary Robinson seems to be using S to
be standard deviation (listed as sigma, above).

Just a warning for those who may use the formulas I quoted without
looking too closely at the definitions.

- Alex

From tim.one@comcast.net  Mon Sep 23 17:30:45 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 23 Sep 2002 12:30:45 -0400
Subject: [Spambayes] RE: Central Limit Theorem??!! :)
In-Reply-To: <20020923155347.626D2F57F@cashew.wolfskeep.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCOECCBGAB.tim.one@comcast.net>

[T. Alexander Popiel]
> In message:  <20020923154703.2C816F57F@cashew.wolfskeep.com>
>              "T. Alexander Popiel" <popiel@wolfskeep.com> writes:
>>
>>  (15)  M[1] = x[1],  M[k] = M[k-1] + ((x[k] - M[k-1]) / k)
>>  (16)  S[1] = 0,     S[k] = S[k-1] + ((x[k] - M[k-1]) * (x[k] - M[k]))
>>
>>  for a population x of size n, 2 <= k <= n, where
>>  sigma = sqrt(S[n]/(n-1)).

Note that n can be 1 too.  That's why S[1] starts out at 0.

> It just occured to me that I neglected to mention that Knuth's S and
> Gary Robinson's S are different; Gary Robinson seems to be using S to
> be standard deviation (listed as sigma, above).

Yes, but in addition Gary is using population sdev rather than sample sdev.
Getting that from the Knuth recurrence is just a matter of dividing by n
rather than n-1:

    sigma = sqrt(S[n]/n)

If the difference between n and n-1 really matters, though, the scheme is in
big trouble <wink>.  A minor advantage of using n here instead is that you
don't have special-case n=1 to avoid division by 0 when computing sigma.


From gward@python.net  Mon Sep 23 17:36:32 2002
From: gward@python.net (Greg Ward)
Date: Mon, 23 Sep 2002 12:36:32 -0400
Subject: python.org corpus (was Re: [Spambayes] Moving closer to Gary's ideal)
In-Reply-To: <20020923141519.GA27648@cthulhu.gerg.ca>
References: <LNBBLJKPBEHFEDALKOLCMENKBFAB.tim.one@comcast.net>
	<200209230327.g8N3RYo05107@pcp02138704pcs.reston01.va.comcast.net>
	<20020923134508.GA27558@cthulhu.gerg.ca>
	<200209231346.g8NDkvB06828@pcp02138704pcs.reston01.va.comcast.net>
	<20020923141519.GA27648@cthulhu.gerg.ca>
Message-ID: <20020923163632.GA28690@cthulhu.gerg.ca>

On 23 September 2002, To SpamBayes said:
> There also
> seems to be some personal email in the ham -- not sure how that snuck
> in, since I was careful to detect and *not* save it.  Haven't
> investigated yet.

Ahh, figured it out.  There are a couple of ways personal mail can get
into the 'ham' folder of the email harvest:

  * it was flagged as junk mail by Exim on receipt, but was manually
    moved by me from the 'misc-junk' folder to the 'ham' folder on
    inspection

  * it was flagged by SpamAssassin on receipt, but again was manually
    moved by me from 'caught-spam' to 'ham'

  * it was sent to eg. Guido@python.org instead of guido@python.org

The last one is obviously a silly bug in the harvesting code; I'll fix
that before the next harvesting run.  I'm going to delete those messages
from the ham folder now; there's no reason to keep them since there's no
other genuinely personal email in there.

The other two are trickier.  We really want to harvest spam that, if
ham, would be considered personal email -- not doing so is failing to
utilize all available resources!  But once I look at that so-called spam
and realize that it's an FP, then all of a sudden it's personal email.
*BUT* it's also email that some junk-detection system (my ad-hoc Exim
ACLs, or SpamAssassin) flagged as junk... so it's valuable to keep in
the training data.  Hmmm.  Opinions?

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
No man is an island, but some of us are long peninsulas.

From tim.one@comcast.net  Mon Sep 23 18:11:17 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 23 Sep 2002 13:11:17 -0400
Subject: [Spambayes] RE: Central Limit Theorem??!! :)
In-Reply-To: <20020923154703.2C816F57F@cashew.wolfskeep.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKECGBGAB.tim.one@comcast.net>

[T. Alexander Popiel]
> This can still be done in one pass.  While calculating mean and stddev,
> keep a running list of the 30 highest and 30 lowest f(w), then at the end
> figure out which of those 60 are most extreme.  (Assuming, of course,
> that I'm properly interpreting what is meant by 'extreme'.)

OK, it's really a three-pass process.

You don't know what any of the f(w) are until the first pass is complete.
Computing f(w) depends on first having seen every message that contains w.

Computing f(w) is then a second pass, but doesn't require the source of
messages, it just needs the accumulated counts from the first pass.

The third pass is needed to find the 30 extremes for *each* training message
(as Gary said).  This again requires knowing the set of words contained in
each message.

If you only put a grand total of 30 extremes in each of the ham and spam
populations, then, e.g., under Graham's p(w) all of the ham extremes will
almost certainly be 0.01, and all of the spam extremes 0.99, and the
population variances will be 0, rendering the scheme useless (the z-score
for any message whose mean isn't exactly 0.01 or 0.99 will be infinite).
Under Gary's f(w), it will approach that the more messages you train on.

> ...
> The formulas in elementary statistics books certainly aren't good for
> this, but the recurrence relations in Knuth's TAOCP vol 2 (Seminumerical
> Algorithms) are quite good for the incremental approach.  Quoting from
> 4.2.2, equations 15 & 16:
>
>   (15)  M[1] = x[1],  M[k] = M[k-1] + ((x[k] - M[k-1]) / k)
>   (16)  S[1] = 0,     S[k] = S[k-1] + ((x[k] - M[k-1]) * (x[k] - M[k]))
>
>   for a population x of size n, 2 <= k <= n, where
>   sigma = sqrt(S[n]/(n-1)).
>
> These formulas can be run forwards or backwards with equal grace,

[16] can be run backwards gracefully, but I'm not sure about [15].  In the
forward direction, [15] wins because the rounding error in the subtraction
is effectively divided by k.  I believe

    M[k] = M[k+1] + (M[k+1] - x[k+1]) / k

is a numerically robust way to run [15] backwards, though.

> and (if coded as written without 'simplification') handle round-off
> errors well, too.  (If greater precision is needed, the precision of the
> sums can be doubled through use of the formulas in exercise 19 of the
> same chapter.)
>
> IANAS, but I trust Knuth. ;-)

For exercise 19, you trust Kahan, and he's more trustworthy here than Knuth
by a long shot <wink>.  I'm not going to bother until we know better whether
this scheme works (Python's unbounded ints are builtin, and fast, and using
them instead obviously loses no significant information).


From tim.one@comcast.net  Mon Sep 23 18:24:30 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 23 Sep 2002 13:24:30 -0400
Subject: [Spambayes] Moving closer to Gary's ideal
In-Reply-To: <200209230832.g8N8WL613221@indus.ins.cwi.nl>
Message-ID: <LNBBLJKPBEHFEDALKOLCCECIBGAB.tim.one@comcast.net>

[Sjoerd Mullender]
> > """
> > [Classifier]
> > use_robinson_probability: True
> > use_robinson_combining: True
> > max_discriminators: 1500
> >
> > [TestDriver]
> > spam_cutoff: 0.50
> > """
>
> I tested this against the default options (except I have
> count_all_header_lines: True and mine_received_headers: True
> permanently)

I'm jealous!  If you can use those, you can probably enable Jeremy's
basic_header_tokenize too.

> and got these results:
>
> false positive percentages
>     0.524  1.047  lost   +99.81%
>     0.000  0.524  lost  +(was 0)
>     0.524  0.524  tied
>     0.524  1.047  lost   +99.81%
>     0.524  1.571  lost  +199.81%
>
> won   0 times
> tied  1 times
> lost  4 times
>
> total unique fp went from 4 to 9 lost  +125.00%
> mean fp % went from 0.418848167539 to 0.942408376964 lost  +125.00%
>
> false negative percentages
>     1.571  0.000  won   -100.00%
>     2.618  2.094  won    -20.02%
>     1.571  0.524  won    -66.65%
>     0.524  0.524  tied
>     1.571  1.047  won    -33.35%
>
> won   4 times
> tied  1 times
> lost  0 times
>
> total unique fn went from 15 to 8 won    -46.67%
> mean fn % went from 1.57068062827 to 0.83769633508 won    -46.67%
>
> The histograms in the default scheme show the usual pattern, but the
> histograms with the changed parameters is like this:

Note that most people reported needing to boost spam_cutoff above 0.5 for
best results in this scheme.  For example, your histograms show that
boosting it to 0.525 would cut 8 fp and add 10 fn, leaving a grand total of
9-8 = 1 fp and 8+10 = 18 fn.  Boosting it to 0.55 would eliminate all your
fp, but add yet anoter 15 fn.  Setting nbuckets higher than the default 40
would allow prediction of finer-grained changes.

Passive testing is helpful, but you have to *play* some if you want to help
tune a new approach.  Like playing with spam_cutoff, trying

robinson_minimum_prob_strength: 0.1

or even higher values, and trying different values for
robinson_probability_a.


From popiel@wolfskeep.com  Mon Sep 23 18:35:46 2002
From: popiel@wolfskeep.com (T. Alexander Popiel)
Date: Mon, 23 Sep 2002 10:35:46 -0700
Subject: [Spambayes] RE: Central Limit Theorem??!! :) 
In-Reply-To: Message from Tim Peters <tim.one@comcast.net> 
	of "Mon, 23 Sep 2002 13:11:17 EDT."
	<LNBBLJKPBEHFEDALKOLCKECGBGAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCKECGBGAB.tim.one@comcast.net> 
Message-ID: <20020923173546.E3D7CF57F@cashew.wolfskeep.com>

In message:  <LNBBLJKPBEHFEDALKOLCKECGBGAB.tim.one@comcast.net>
             Tim Peters <tim.one@comcast.net> writes:
>[T. Alexander Popiel]
>> This can still be done in one pass.  While calculating mean and stddev,
>> keep a running list of the 30 highest and 30 lowest f(w), then at the end
>> figure out which of those 60 are most extreme.  (Assuming, of course,
>> that I'm properly interpreting what is meant by 'extreme'.)
>
>OK, it's really a three-pass process.
>
>You don't know what any of the f(w) are until the first pass is complete.
>Computing f(w) depends on first having seen every message that contains w.
>
>Computing f(w) is then a second pass, but doesn't require the source of
>messages, it just needs the accumulated counts from the first pass.
>
>The third pass is needed to find the 30 extremes for *each* training message
>(as Gary said).  This again requires knowing the set of words contained in
>each message.

Right.  Okay, I'll go hide my head in the sand, now...

- Alex

From tim.one@comcast.net  Mon Sep 23 18:36:37 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 23 Sep 2002 13:36:37 -0400
Subject: [Spambayes] RE: Central Limit Theorem??!!     :)
In-Reply-To: <B9B49138.16557%grobinson@transpose.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCAECKBGAB.tim.one@comcast.net>

[Gary Robinson]
> ...
> Still seems better than p(w), which  is assuming s is 0!

It approaches that, but not quite:  p(w) is still bounded by 0.01 and 0.99.
In small training sets, something I've seen several times is that the
training data ends up with an original message or a reply to it, and the
prediction data contains the other half, and sometimes there's a word
common to that pair but otherwise unique across the entire corpus.  p(w)
assigns that word prob 0.01 (well, our p(w) does; Graham's gives it the
"unknown word prob" due to "not appearing often enough"), and that really
helps the prediction.  Of course it works out in the other direction too --
for example, a spam containing the misspelling "propsal" (for "proposal")
became an fn just because somebody once made the same misspelling in my ham.
Overall it seems much more *likely* that unique typos are shared between a
message and a reply, though, and "message + reply containing some quote" are
virtually always both ham.  OTOH, spam seems to contain common wild
misspellings too.

> That's just another decision for a particular constant s... there's no
> getting getting away from the issue.

Indeed there isn't.


From tim.one@comcast.net  Mon Sep 23 18:41:21 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 23 Sep 2002 13:41:21 -0400
Subject: [Spambayes] Moving closer to Gary's ideal
In-Reply-To: <20020923134508.GA27558@cthulhu.gerg.ca>
Message-ID: <LNBBLJKPBEHFEDALKOLCAECLBGAB.tim.one@comcast.net>

[Greg Ward]
> ...
> Maybe we need a tool to strip off "Received" headers up to a
> certain point -- eg. if Guido wants to train spambayes for his traffic,
> but run it at the python.org MTA, then it shouldn't train on the
> Received headers added by his ISP.  Or maybe there should be an option
> to ignore "Received" headers up to server X.

While fudging the data is a cardinal sin, if nobody tells me they're
indulging in this particular vice, I won't ask <wink>.


From tim.one@comcast.net  Mon Sep 23 18:43:36 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 23 Sep 2002 13:43:36 -0400
Subject: [Spambayes] 2nd try at mean/dev cmp.py
In-Reply-To: <3D8EE5D5.29647.418F4779@localhost>
Message-ID: <LNBBLJKPBEHFEDALKOLCGECLBGAB.tim.one@comcast.net>

[Brad Clements]
>>> ham/spam mean difference: 48.81 49.31 +0.50

[Tim]
>> Cool!  Looks good -- check it in.

[Brad]
> Checked in.. gee, that was easy.

Thanks!  The msg in which you tried to shame someone else into checking it
in would have worked <wink>, except the harder I try to keep up with all the
traffic on this list, the more I fall behind.


From tim.one@comcast.net  Mon Sep 23 18:52:15 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 23 Sep 2002 13:52:15 -0400
Subject: [Spambayes] RE: Further Improvement 2
In-Reply-To: <15759.10935.189018.319822@12-248-11-90.client.attbi.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKECMBGAB.tim.one@comcast.net>

[ Skip Montanaro]
> We've been doing a lot of manual knob turning trying to find the best
> settings for various options.  It seems like we need an
> optimization tool of some sort to automate this process.  I know by
> suggesting it, I'm implicitly volunteering to create it, but I don't
> have the time to do this.

It's difficult, as the effects of various option settings are generally not
independent.  I've been using what amounts to a "deepest descent" strategy
by hand:  fiddle a whole bunch of parameters, one a time, on a fixed and
small test corpus.  Whichever one of those gave the biggest improvement then
gets loving and exceedingly wall-clock-time consuming attention on my full
test suite.  Go back to square one.  On the second iteration of this, you
discover that the first thing you tuned is no longer optimally tuned after
you tune the second thing you found.  The effects of various options are
generally not independent of corpus size either.  It's fun <wink>!


From tim.one@comcast.net  Mon Sep 23 18:54:37 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 23 Sep 2002 13:54:37 -0400
Subject: [Spambayes] RE: Further Improvement 2
In-Reply-To: <15759.11258.301930.985013@anthem.wooz.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCEECNBGAB.tim.one@comcast.net>

[about the email pkg]

Someone should run their test data with the current email pkg, and again
with the 2.2 email pkg.  If it doesn't make any difference to results,
there's no point to forcing use of the new pkg.  If it does make a
difference to results, there is a point.


From tim.one@comcast.net  Mon Sep 23 19:02:36 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 23 Sep 2002 14:02:36 -0400
Subject: [Spambayes] RE: Central Limit Theorem??!! :)
In-Reply-To: <20020923173546.E3D7CF57F@cashew.wolfskeep.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCOECNBGAB.tim.one@comcast.net>

[T. Alexander Popiel]
> ...
> Right.  Okay, I'll go hide my head in the sand, now...

Don't hide too deeply!  I had forgotten all about the recurrences Knuth gave
for robust sdev computation, and that was helpful.  Everyone on this list
has their own personal false positive and false negative rates when
speculating <wink>.


From guido@python.org  Mon Sep 23 19:19:43 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 23 Sep 2002 14:19:43 -0400
Subject: [Spambayes] RE: Further Improvement 2
In-Reply-To: Your message of "Mon, 23 Sep 2002 13:54:37 EDT."
             <LNBBLJKPBEHFEDALKOLCEECNBGAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCEECNBGAB.tim.one@comcast.net> 
Message-ID: <200209231819.g8NIJhN11310@pcp02138704pcs.reston01.va.comcast.net>

> Someone should run their test data with the current email pkg, and again
> with the 2.2 email pkg.  If it doesn't make any difference to results,
> there's no point to forcing use of the new pkg.  If it does make a
> difference to results, there is a point.

OK, will do.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.one@comcast.net  Mon Sep 23 20:46:27 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 23 Sep 2002 15:46:27 -0400
Subject: [Spambayes] RE: Central Limit Theorem??!!     :)
In-Reply-To: <B9B49138.16557%grobinson@transpose.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEDEBGAB.tim.one@comcast.net>

Ran into another cute numeric pitfall when using a very small training set,
20 ham and 20 spam, and g(w).

It so happened that every one of the most 520 extreme probs in one 20-ham
set had spamprob 0.01 (520 < 600 = 20*30 because not all the hams had 30
not-seen-before-as-an-extreme words).  The ham variance was 0 then, so
computing the ham z-score when predicting died with ZeroDivisionError.


From guido@python.org  Mon Sep 23 22:40:32 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 23 Sep 2002 17:40:32 -0400
Subject: [Spambayes] email pkg
In-Reply-To: Your message of "Mon, 23 Sep 2002 13:54:37 EDT."
             <LNBBLJKPBEHFEDALKOLCEECNBGAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCEECNBGAB.tim.one@comcast.net> 
Message-ID: <200209232140.g8NLeWs19659@pcp02138704pcs.reston01.va.comcast.net>

> Someone should run their test data with the current email pkg, and again
> with the 2.2 email pkg.  If it doesn't make any difference to results,
> there's no point to forcing use of the new pkg.  If it does make a
> difference to results, there is a point.

I said I'd run tests.  The output is vastly different.  But I don't
know if that's due to the email package or due to all the other
changes in the code.

I'm tired of this so I'm not going to bother again.  Sorry.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From bkc@murkworks.com  Mon Sep 23 22:58:40 2002
From: bkc@murkworks.com (Brad Clements)
Date: Mon, 23 Sep 2002 17:58:40 -0400
Subject: [Spambayes] Saving Hist
Message-ID: <3D8F565A.4555.4346D5BB@localhost>

I want to use gnu plot to plot histograms of global histograms, so ..

I'd like to check in these changes:


Index: TestDriver.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/TestDriver.py,v
retrieving revision 1.7
diff -r1.7 TestDriver.py
160a161,168
> 
>         if options.save_histogram_pickles:
>             for f,h in (('ham',self.global_ham_hist),('spam',self.global_spam_hist)):
>                 fname = "%s%s.pik" % (options.pickle_basename,f)
>                 print "    saving %s histogram pickle to %s" %(f, fname)
>                 fp = file(fname, 'wb')
>                 pickle.dump(h, fp, 1)
>                 fp.close()


Index: Options.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Options.py,v
retrieving revision 1.23
diff -r1.23 Options.py
137a138,142
> 
> # if save_histogram_pickles is true, Driver.train() saves a binary
> # pickle of the spam and ham histogram for "all test runs". The file
> # basename is given by pickle_basename, the suffix _spamhist.pik
> # or _hamhist_pik is appended  to the basename.
140c145,146
< pickle_basename: class
---
> pickle_basename: class
> save_histogram_pickles: False
208a215
>                    'save_histogram_pickles': boolean_cracker,



Brad Clements,                bkc@murkworks.com   (315)268-1000
http://www.murkworks.com                          (315)268-9812 Fax
AOL-IM: BKClements


From tim.one@comcast.net  Mon Sep 23 23:00:46 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 23 Sep 2002 18:00:46 -0400
Subject: [Spambayes] RE: Central Limit Theorem??!!     :)
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCEAEBGAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCAEDOBGAB.tim.one@comcast.net>

Gary, let me pick your brain here.  Here are the score histos repeated from
a central limit test (I've since added each unique "extreme word" only once
to the spam and ham population distributions, and that reduced the f-n rate,
but the same *kinds* of things are still going on):

> Ham distribution for all runs:
> 5000 items; mean 0.05; sample sdev 1.57
> * = 84 items
>   0.00 4991 ***********************************************************
>   2.50    2 *
>   5.00    0
>   7.50    1 *
>  10.00    0
>  12.50    0
>  15.00    0
>  17.50    0
>  20.00    1 *
>  22.50    1 *
>  25.00    1 *
>  27.50    1 *
>  30.00    0
>  32.50    0
>  35.00    0
>  37.50    0
>  40.00    0
>  42.50    0
>  45.00    0
>  47.50    0
>  50.00    0
>  52.50    0
>  55.00    0
>  57.50    1 *
>  60.00    0
>  62.50    0
>  65.00    0
>  67.50    0
>  70.00    0
>  72.50    0
>  75.00    0
>  77.50    1 *
>  80.00    0
>  82.50    0
>  85.00    0
>  87.50    0
>  90.00    0
>  92.50    0
>  95.00    0
>  97.50    0
>
> Spam distribution for all runs:
> 5000 items; mean 94.82; sample sdev 15.16
> * = 69 items
>   0.00    5 *
>   2.50    2 *
>   5.00    1 *
>   7.50    5 *
>  10.00    8 *
>  12.50    6 *
>  15.00   10 *
>  17.50    8 *
>  20.00    6 *
>  22.50   14 *
>  25.00   10 *
>  27.50   11 *
>  30.00    9 *
>  32.50    4 *
>  35.00   12 *
>  37.50   12 *
>  40.00   11 *
>  42.50    8 *
>  45.00   16 *
>  47.50   15 *
>  50.00   21 *
>  52.50   21 *
>  55.00   20 *
>  57.50   27 *
>  60.00   21 *
>  62.50   19 *
>  65.00   31 *
>  67.50   18 *
>  70.00   18 *
>  72.50   22 *
>  75.00   22 *
>  77.50   37 *
>  80.00   36 *
>  82.50   50 *
>  85.00   59 *
>  87.50   50 *
>  90.00   67 *
>  92.50   74 **
>  95.00   76 **
>  97.50 4138 ************************************************************

So the ham scores are very much tighter than the spam scores.  OTOH, in the
"extreme word" ham and spam populations, the ham probability distribution
has much higher variance than the spam probability distribution (sometimes
by a factor of 10!):

> hammean  0.083575863227  hamvar 0.0384359627039
> spammean 0.978952388668 spamvar 0.00372443106446
>
> hammean  0.075515986459  hamvar 0.0331126798862
> spammean 0.97742506528  spamvar 0.00420869219347
>
> hammean  0.0776481612081 hamvar 0.0338362239317
> spammean 0.978462136207 spamvar 0.00341505065601
>
> hammean  0.07972071882   hamvar 0.0355833143776
> spammean 0.974508870296 spamvar 0.00530574730638
>
> hammean  0.0713015705881 hamvar 0.0303987810665
> spammean 0.976734052416 spamvar 0.00468795224229

I'm starting to suspect that these are deeply related, and because of my "R
score normalization" gimmick.  Here's the code for that:

        mean = sum / n

        # If this sample is drawn from the spam population, its mean is
        # distributed around spammean with variance spamvar/n.  Likewise
        # for if it's drawn from the ham population.  Compute a normalized
        # z-score (how many stddevs is it away from the population mean?)
        # against both populations, and then it's ham or spam depending
        # on which population it matches better.
        zham = (mean - self.hammean) / sqrt(self.hamvar / n)
        zspam = (mean - self.spammean) / sqrt(self.spamvar / n)
        stat = abs(zham) - abs(zspam)  # > 0 for spam, < 0 for ham

        # Normalize into [0, 1].  I'm arbitrarily clipping it to fit in
        # [-20, 20] first.
        if stat < -20.0:
            stat = -20.0
        elif stat > 20.0:
            stat = 20.0
        stat = 0.5 + stat / 40.0

Now suppose something is close to being ham (mean near about 0.08).  Its
zspam is going to be huge, becase spamvar is small.  Therefore R (the first
"stat" above) is going to be a very large negative number ("very" might
mean -100, it's all relative <wink>).

OTOH, suppose something is close to being spam (mean near about 0.97).  Its
zham is going to be large, but not nearly as large as zspam is when a thing
is close to being spam, because hamvar is much larger than spamvar, so R
will again a be large number, but not *as* large (in magnitude) as when a
thing is close to being ham.

IOW, the distribution of R around 0 can't be symmetric (unless the world is
much stranger than even I think it is <wink>), so scaling it linearly into
[0, 1] compresses the ham-favoring side while leaving the spam-favoring side
more spread out.

OK, printing R out in a test run shows that R is usually around -50 when
scoring a ham, and hovering around +20 when scoring a spam.  So most of the
ham scores get clipped to -20 and from there to 0, while the spam scores
spread out.

This explains the weird appearance of the score histograms, but none of this
changes whether R is > 0 or < 0 (== whether the scaled value is > or < than
0.5).

Do you know offhand of a better way to normalize R into [0, 1] anyway?  The
histograms are pain *disturbing* right now <wink>.


From tim.one@comcast.net  Mon Sep 23 23:10:13 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 23 Sep 2002 18:10:13 -0400
Subject: [Spambayes] RE: email pkg
In-Reply-To: <200209232140.g8NLeWs19659@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEDPBGAB.tim.one@comcast.net>

[TIm]
>> Someone should run their test data with the current email pkg, and
>> again with the 2.2 email pkg.  If it doesn't make any difference to
>> results, there's no point to forcing use of the new pkg.  If it does
>> make a difference to results, there is a point.

[Guido]
> I said I'd run tests.  The output is vastly different.  But I don't
> know if that's due to the email package or due to all the other
> changes in the code.
>
> I'm tired of this so I'm not going to bother again.  Sorry.

I don't know what Guido did, but if someone else will try this, it's a safe
bet you shouldn't do what he did <wink>:

1. Run a baseline test.
2. Arrange to use the other email package.  Change nothing else.
3. Run the same test again.

If there are any differences, they're due to the email package.


From guido@python.org  Mon Sep 23 23:16:31 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 23 Sep 2002 18:16:31 -0400
Subject: [Spambayes] RE: email pkg
In-Reply-To: Your message of "Mon, 23 Sep 2002 18:10:13 EDT."
             <LNBBLJKPBEHFEDALKOLCGEDPBGAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCGEDPBGAB.tim.one@comcast.net> 
Message-ID: <200209232216.g8NMGVJ19822@pcp02138704pcs.reston01.va.comcast.net>

> 1. Run a baseline test.
> 2. Arrange to use the other email package.  Change nothing else.
> 3. Run the same test again.
> 
> If there are any differences, they're due to the email package.

What about fixing the random seed?

--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.one@comcast.net  Mon Sep 23 23:26:00 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 23 Sep 2002 18:26:00 -0400
Subject: [Spambayes] Saving Hist
In-Reply-To: <3D8F565A.4555.4346D5BB@localhost>
Message-ID: <LNBBLJKPBEHFEDALKOLCOEEABGAB.tim.one@comcast.net>

[Brad Clements]
> I want to use gnu plot to plot histograms of global histograms, so ..
>
> I'd like to check in these changes:

Fine by me.

> Index: TestDriver.py
> ===================================================================
> RCS file: /cvsroot/spambayes/spambayes/TestDriver.py,v
> retrieving revision 1.7
> diff -r1.7 TestDriver.py
> 160a161,168
>>
>>         if options.save_histogram_pickles:
>>             for f,h in

A space after the comma, please.

> (('ham',self.global_ham_hist),('spam',self.global_spam_hist)):

Ditto twice.

>>                 fname = "%s%s.pik" % (options.pickle_basename,f)

Ditto.  Note this doesn't match what the docs say wrt the name.  Perhaps you
wanted "%s_%shist.pik" as the format?


> ...
>
> Index: Options.py
> ===================================================================
> RCS file: /cvsroot/spambayes/spambayes/Options.py,v
> retrieving revision 1.23
> diff -r1.23 Options.py
> 137a138,142
> >
> > # if save_histogram_pickles is true, Driver.train() saves a binary
> > # pickle of the spam and ham histogram for "all test runs". The file
> > # basename is given by pickle_basename, the suffix _spamhist.pik

The code addes the suffix "spam.pik".

> > # or _hamhist_pik is appended  to the basename.

Likewise.


From tim.one@comcast.net  Mon Sep 23 23:36:17 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 23 Sep 2002 18:36:17 -0400
Subject: [Spambayes] RE: email pkg
In-Reply-To: <200209232216.g8NMGVJ19822@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCMEEBBGAB.tim.one@comcast.net>

[Guido]
> What about fixing the random seed?

I'm not sure what you mean.  timcv and timtest make no use of randomness
unless you run on a subset of your data via the --ham_keep
and/or --spam_keep cmdline options.  In that case, they do pick random
subsets, and the -s option can then be used to force reproducibility
(although it can't if the set of msgs changes between runs).

Or if this is about mboxtest, I'm not familiar with what that does ... OK,
it uses seed 101 by default.  My test drivers don't, because sometimes I
want random subsets that I can't predict -- setting a seed by hand is poor
practice because people (knowingly or not, and me included) tend to pick the
same numbers repeatedly.

I don't think there's any behavior here you can't get easily, although
perhaps you'd prefer a different default behavior.  I'd be willing to add a
new option for that, and even to default it to the behavior I don't want.


From dan.mick@sun.com  Tue Sep 24 00:09:53 2002
From: dan.mick@sun.com (Dan Mick)
Date: Mon, 23 Sep 2002 16:09:53 -0700
Subject: [Spambayes] What to do first?
Message-ID: <3D8F9F41.4050209@sun.com>


OK, I've been half-reading about half the messages here.

I have a wad of spam and a wad of ham.

What would be most useful to you all at the moment?  (And is there
something besides the presence of True and False that makes this
code depend on bleeding-edge Python, or can I use my trusty
2.2?


From tim.one@comcast.net  Tue Sep 24 00:19:26 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 23 Sep 2002 19:19:26 -0400
Subject: [Spambayes] What to do first?
In-Reply-To: <3D8F9F41.4050209@sun.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEEFBGAB.tim.one@comcast.net>

[Dan Mick]
> OK, I've been half-reading about half the messages here.
>
> I have a wad of spam and a wad of ham.

Cool!  Welcome.

> What would be most useful to you all at the moment?

People running tests 30 hours a day is always helpful.  If that doesn't grab
you, there hasn't yet been nearly enough thought given to deployment:  how
do we package this so people can *use* it?  What are the target groups
(individuals, sysadmins, ...)?  Is it possible to seed a classifier with
words that will do "an average" user any good?  Or will a new user need to
train one from scratch?  How does this work with languages other than
English?  Etc.  You can spend happy hours just listing all the things we
can't answer <wink>.

> (And is there something besides the presence of True and False that
> makes this code depend on bleeding-edge Python, or can I use my trusty
> 2.2?

It requires 2.2.1, but as of yesterday should no longer require 2.3
(although I haven't tried this with 2.2.1 yet).


From marklists@mceahern.com  Tue Sep 24 00:28:39 2002
From: marklists@mceahern.com (Mark McEahern)
Date: Mon, 23 Sep 2002 18:28:39 -0500
Subject: [Spambayes] What to do first?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEEFBGAB.tim.one@comcast.net>
Message-ID: <JHEOKEOOLIGLDHCMAHMOKEOBEAAA.marklists@mceahern.com>

[Dan Mick]
> (And is there something besides the presence of True and False that
> makes this code depend on bleeding-edge Python, or can I use my trusty
> 2.2?

[Tim Peters]
> It requires 2.2.1, but as of yesterday should no longer require 2.3
> (although I haven't tried this with 2.2.1 yet).

Somewhat OT:  Is there an easy way to "backport" True/False to Python 2.2;
e.g., adding them to __builtins__ in sitecustomize?  Or is that braindead?

// m

-


From tim.one@comcast.net  Tue Sep 24 00:43:02 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 23 Sep 2002 19:43:02 -0400
Subject: [Spambayes] What to do first?
In-Reply-To: <JHEOKEOOLIGLDHCMAHMOKEOBEAAA.marklists@mceahern.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEEHBGAB.tim.one@comcast.net>

[Mark McEahern]
> Somewhat OT:  Is there an easy way to "backport" True/False to Python
> 2.2; e.g., adding them to __builtins__ in sitecustomize?  Or is that
> braindead?

Everyone using the 2.2 line *should* be using 2.2.1, and 2.2.1 already has
True and False built in.  That should be fine for this project.  In 2.2.1
they're just predefined names for the ints 0 and 1, but I don't believe this
project is making any use of that bool is a distinct subclass of int in 2.3.


From dan.mick@sun.com  Tue Sep 24 01:14:02 2002
From: dan.mick@sun.com (Dan Mick)
Date: Mon, 23 Sep 2002 17:14:02 -0700
Subject: [Spambayes] What to do first?
References: <LNBBLJKPBEHFEDALKOLCIEEHBGAB.tim.one@comcast.net>
Message-ID: <3D8FAE4A.5010702@sun.com>

Tim Peters wrote:
> [Mark McEahern]
> 
>>Somewhat OT:  Is there an easy way to "backport" True/False to Python
>>2.2; e.g., adding them to __builtins__ in sitecustomize?  Or is that
>>braindead?
> 
> 
> Everyone using the 2.2 line *should* be using 2.2.1, and 2.2.1 already has
> True and False built in.  That should be fine for this project.  In 2.2.1
> they're just predefined names for the ints 0 and 1, but I don't believe this
> project is making any use of that bool is a distinct subclass of int in 2.3.

Well, it's a question of whether I build my own or not, and then getting it 
deployed on servers I don't control, which are both pains.  If it's just 
True/False I'd like to beg for relief.  (I've already hacked in "True = 1 
False = 0" into several of the scripts since that was the only immediate 
complaint.)



From dan.mick@sun.com  Tue Sep 24 01:15:31 2002
From: dan.mick@sun.com (Dan Mick)
Date: Mon, 23 Sep 2002 17:15:31 -0700
Subject: [Spambayes] What to do first?
References: <LNBBLJKPBEHFEDALKOLCKEEFBGAB.tim.one@comcast.net>
Message-ID: <3D8FAEA3.4080809@sun.com>

>>What would be most useful to you all at the moment?
> 
> People running tests 30 hours a day is always helpful.  

All of the test frameworks?  Some?


From tim.one@comcast.net  Tue Sep 24 01:31:04 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 23 Sep 2002 20:31:04 -0400
Subject: [Spambayes] What to do first?
In-Reply-To: <3D8FAEA3.4080809@sun.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEELBGAB.tim.one@comcast.net>

>> People running tests 30 hours a day is always helpful.

[Dan Mick]
> All of the test frameworks?  Some?

Whichever is easiest for you to set up.  One of {mboxtest, timcv} should be
sufficient.


From tim.one@comcast.net  Tue Sep 24 01:35:45 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 23 Sep 2002 20:35:45 -0400
Subject: [Spambayes] What to do first?
In-Reply-To: <3D8FAE4A.5010702@sun.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCOEELBGAB.tim.one@comcast.net>

[Dan Mick]
> Well, it's a question of whether I build my own or not, and then
> getting it deployed on servers I don't control, which are both pains.
> If it's just True/False I'd like to beg for relief.  (I've already
> hacked in "True = 1 False = 0" into several of the scripts since that
> was the only immediate complaint.)

This project requires 2.2.1, and that's the bottom line.  I don't want us
wasting time rediscovering subtle bugs in 2.2, either; a lot of stuff got
fixed for 2.2.1, and this project has lots of challenges without revisiting
problems that have already been solved.

BTW, which platform are you on?  There are 2.2.1 RPMs for Linux, and
precompiled versions for Macs and Windows.  Most everyone else seems to do
"make install" and get a cup of coffee <wink>.


From anthony@interlink.com.au  Tue Sep 24 01:47:26 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Tue, 24 Sep 2002 10:47:26 +1000
Subject: [Spambayes] Re: Practical Applications 
In-Reply-To: <B9B4A555.16572%grobinson@transpose.com> 
Message-ID: <200209240047.g8O0lR720595@localhost.localdomain>


>>> Gary Robinson wrote
> Suppose I wanted to build a practical application on spambayes that
> integrated with Outlook but it just couldn't be done as elegantly with
> separate Python code communicating with Outlook via COM.
> 
> Would the open source license spambayes is using allow me to translate
> spambayes into Visual Basic for Applications?????

I'd suggest an alternate approach - 
- for the inbound, something like pop3proxy.py, but it should also
keep a copy of (some/all) messages as 'ham' samples.

- for marking as spam, a small application (maybe in VBA) that
copies the current message into a magic location on disk.

The classifier should pick up the (some/all) ham, and also the marked
spam. First off, remove any of the spam from the new ham samples 
(message-id, or whatever), then load the data up into the classifier.

If you wanted to be extra-clever, you might try something like 
"expiring" trained data after 3 months - on the grounds that your
mail reading habits (and the spammer's habits) will have adapted/
changed.

Random thoughts before the first coffee.

Anthony
-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.


From guido@python.org  Tue Sep 24 01:50:10 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 23 Sep 2002 20:50:10 -0400
Subject: [Spambayes] What to do first?
In-Reply-To: Your message of "Mon, 23 Sep 2002 18:28:39 CDT."
             <JHEOKEOOLIGLDHCMAHMOKEOBEAAA.marklists@mceahern.com> 
References: <JHEOKEOOLIGLDHCMAHMOKEOBEAAA.marklists@mceahern.com> 
Message-ID: <200209240050.g8O0oFn20192@pcp02138704pcs.reston01.va.comcast.net>

> Somewhat OT:  Is there an easy way to "backport" True/False to Python 2.2;
> e.g., adding them to __builtins__ in sitecustomize?  Or is that braindead?

Please don't.  If you have to use the old 2.2 (I don't understand
why), write a compat.py that exports True and False and add "from
compat import *" at the top of your modules.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From dan.mick@sun.com  Tue Sep 24 01:51:20 2002
From: dan.mick@sun.com (Dan Mick)
Date: Mon, 23 Sep 2002 17:51:20 -0700
Subject: [Spambayes] What to do first?
References: <LNBBLJKPBEHFEDALKOLCOEELBGAB.tim.one@comcast.net>
Message-ID: <3D8FB708.7040704@sun.com>

Tim Peters wrote:
> [Dan Mick]
> 
>>Well, it's a question of whether I build my own or not, and then
>>getting it deployed on servers I don't control, which are both pains.
>>If it's just True/False I'd like to beg for relief.  (I've already
>>hacked in "True = 1 False = 0" into several of the scripts since that
>>was the only immediate complaint.)
> 
> 
> This project requires 2.2.1, and that's the bottom line.  I don't want us
> wasting time rediscovering subtle bugs in 2.2, either; a lot of stuff got
> fixed for 2.2.1, and this project has lots of challenges without revisiting
> problems that have already been solved.

sigh.

> BTW, which platform are you on?  There are 2.2.1 RPMs for Linux, and
> precompiled versions for Macs and Windows.  Most everyone else seems to do
> "make install" and get a cup of coffee <wink>.

Solaris, as the email might suggest.  And it's not the compilation, it's 
the deployment.  As I said, I don't control the servers where it would need 
to be deployed.

I'll work on it; I'm not going to actually be using spambayes for actual 
mail filtering for a while yet, I'm guessing, anyway.



From guido@python.org  Tue Sep 24 02:04:25 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 23 Sep 2002 21:04:25 -0400
Subject: [Spambayes] What to do first?
In-Reply-To: Your message of "Mon, 23 Sep 2002 17:51:20 PDT."
             <3D8FB708.7040704@sun.com> 
References: <LNBBLJKPBEHFEDALKOLCOEELBGAB.tim.one@comcast.net>  
            <3D8FB708.7040704@sun.com> 
Message-ID: <200209240104.g8O14PL20326@pcp02138704pcs.reston01.va.comcast.net>

> Solaris, as the email might suggest.  And it's not the compilation, it's 
> the deployment.  As I said, I don't control the servers where it would need 
> to be deployed.
> 
> I'll work on it; I'm not going to actually be using spambayes for actual 
> mail filtering for a while yet, I'm guessing, anyway.

Our emails crossed.  For our convenience, the official CVS version of
the project will always (well, at least as long as there's active
development on the algorithm development side) require 2.2.1 or newer,
but it should be trivial to make a derived version that works with
2.2, by adding a line per module, as I explained.  In fact, since the
email package is designed to work with 2.1 too, it shouldn't be too
hard to get it to work there either; but I'm not going to try that
myself.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.one@comcast.net  Tue Sep 24 02:25:53 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 23 Sep 2002 21:25:53 -0400
Subject: [Spambayes] What to do first?
In-Reply-To: <200209240104.g8O14PL20326@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCOEFABGAB.tim.one@comcast.net>

[Guido]
> ...
> In fact, since the email package is designed to work with 2.1 too, it
> shouldn't be too hard to get it to work there either; but I'm not
> going to try that myself.

It would take a fair amount of semi-mechanical pain to purge the codebase of
generators and __iter__ for 2.1.  And without __slots__ in the WordInfo
records, memory use would zoom (although I expect they'll evenutally become
tuples anyway).


From tim.one@comcast.net  Tue Sep 24 04:22:13 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 23 Sep 2002 23:22:13 -0400
Subject: [Spambayes] RE: Central Limit Theorem??!!     :)
In-Reply-To: <B9B48552.16549%grobinson@transpose.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEFJBGAB.tim.one@comcast.net>

[Gary Robinson, outlines another central limit scheme, but using
 ln(f(w)) for spam scores and ln(1-f(w)) for ham scores when
 building the ham and spam "extreme word" populations, then computing
 two corresponding log sums when scoring.
]

> ...
> This *should* bring us the benefits of the multiplicative approach
> and of the parametric stats approach at the same time.

My first "real test" says it's a winner!  See below.

> The downside of this is that the ln's make such a skewed distribution
> that it should take a bigger n to make the central limit theorem kick
> in.

This isn't a good time to mention it <wink>, but the original central-limit
code worked better when I reduced the # of extremes from 30 to 15.  Note
that these are the kinds of oddities I spent countless hours "tuning away"
on the Graham scheme, though, and I;ve still done almost nothing of that
nature for your schemes.

> BUT, OTOH, it WILL kick in, and something like 30 may really still
> be enough (it usually kicks in at significantly smaller numbers).

I predict more experiments on the horizon.

> And you've also successfully used n=150 with f(w)

I've also used n=15 and n=1500, BTW.  150->1500 neither helped nor hurt.
150->15 gave mixed results, too close to call across the few small tests I
ran.

> and that is DEFINITELY enough.
>
> The other downside is that it just seems like a bit  of a wild thing
< to do and sometimes when you do wild things, strange reasons emerge
> why they won't work.

It's an extreme problem, though.  I ran experiments long ago training on a
single ham and a single ham:  that was enough to get about 30% fp rate and
about 20% fn rate.  *Much* better than flipping a coin!  Training on two of
each did even better <wink>.

 But I really can't see any at this point as long as n is big enough
> that the sample means take on a normal distribution.
>
> THANKS for doing all the coding work to test this idea!!!!  :)

Thank me for the testing, if you have to thank me at all -- the coding is
fun, the testing is mind-numbing <drool>.

Speaking of which, the 3-pass nature of training under this scheme doesn't
work well with my cross-validation driver.  Instead the test here is on a
5x5 test grid, skipping the diagonal.  I picked 5 disjoint sets of 1000 ham
each at random, and 5 disjoint sets of 1000 spam each at random.  These were
then paired into 5 collections, 1 thru 5.  First I trained on pair 1, then
predicted on pairs 2, 3, 4, and 5.  Then I trained a new classifier on pair
2, and predicted on pairs 1, 3, 4, and 5.  Etc.  In all we get 20 test runs
out of this, and each message is predicted 4 times.

In both runs, 30 extremes were used, and f(w) with a=0.1.  The original
central limit code is on the left, and the logarithmic version on the right.
I haven't done anything to do a more sensible scaling of scores into [0, 1],
so I'll skip histograms.

-> <stat> tested 1000 hams & 1000 spams against 1000 hams & 1000 spams
   [ditto 39 times]

false positive percentages
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.200  0.200  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.100  0.100  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.100  0.100  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.100  0.000  won   -100.00%
    0.100  0.100  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied

won   1 times
tied 19 times
lost  0 times

total unique fp went from 3 to 2 won    -33.33%
mean fp % went from 0.03 to 0.025 won    -16.67%

false negative percentages
    2.200  0.800  won    -63.64%
    2.100  0.900  won    -57.14%
    1.700  0.600  won    -64.71%
    1.900  0.400  won    -78.95%
    1.500  1.100  won    -26.67%
    1.900  0.700  won    -63.16%
    1.600  0.600  won    -62.50%
    1.600  0.700  won    -56.25%
    1.400  0.600  won    -57.14%
    1.900  0.400  won    -78.95%
    1.600  0.700  won    -56.25%
    1.600  0.400  won    -75.00%
    1.900  0.600  won    -68.42%
    1.600  0.500  won    -68.75%
    1.300  0.500  won    -61.54%
    1.600  0.300  won    -81.25%
    1.300  0.400  won    -69.23%
    1.900  0.400  won    -78.95%
    1.300  0.400  won    -69.23%
    1.400  0.300  won    -78.57%

won  20 times
tied  0 times
lost  0 times

total unique fn went from 125 to 54 won    -56.80%
mean fn % went from 1.665 to 0.565 won    -66.07%

So the logarithmic version was indeed a huge and pure win for the fn rate!

There are two other comparisons I need to make here:  against your webpage
scheme (f(w) with "S"), and against our highly fiddled Graham scheme.

First the same thing (same test msgs, etc), with log central-limit on the
left, and f(w) with a=0.1, 1500 discrimators, and
robinson_minimum_prob_strength=0.1 (a gimmick that simply ignores all words
with spamprob in (0.4, 0.6)) on the right:

[Classifier]
use_robinson_probability: True
use_robinson_combining: True
max_discriminators: 1500
robinson_minimum_prob_strength: 0.1
robinson_probability_a: 0.1
use_central_limit: False

[TestDriver]
spam_cutoff: 0.575

false positive percentages
    0.000  0.000  tied
    0.000  0.100  lost  +(was 0)
    0.000  0.000  tied
    0.200  0.200  tied
    0.000  0.000  tied
    0.000  0.100  lost  +(was 0)
    0.000  0.000  tied
    0.100  0.300  lost  +200.00%
    0.000  0.000  tied
    0.000  0.100  lost  +(was 0)
    0.000  0.000  tied
    0.100  0.200  lost  +100.00%
    0.000  0.000  tied
    0.000  0.100  lost  +(was 0)
    0.000  0.100  lost  +(was 0)
    0.100  0.200  lost  +100.00%
    0.000  0.000  tied
    0.000  0.100  lost  +(was 0)
    0.000  0.100  lost  +(was 0)
    0.000  0.000  tied

won   0 times
tied 10 times
lost 10 times

total unique fp went from 2 to 6 lost  +200.00%
mean fp % went from 0.025 to 0.08 lost  +220.00%

false negative percentages
    0.800  0.100  won    -87.50%
    0.900  0.000  won   -100.00%
    0.600  0.200  won    -66.67%
    0.400  0.200  won    -50.00%
    1.100  0.100  won    -90.91%
    0.700  0.100  won    -85.71%
    0.600  0.200  won    -66.67%
    0.700  0.100  won    -85.71%
    0.600  0.000  won   -100.00%
    0.400  0.000  won   -100.00%
    0.700  0.200  won    -71.43%
    0.400  0.200  won    -50.00%
    0.600  0.100  won    -83.33%
    0.500  0.100  won    -80.00%
    0.500  0.000  won   -100.00%
    0.300  0.100  won    -66.67%
    0.400  0.100  won    -75.00%
    0.400  0.100  won    -75.00%
    0.400  0.000  won   -100.00%
    0.300  0.300  tied

won  19 times
tied  1 times
lost  0 times

total unique fn went from 54 to 11 won    -79.63%
mean fn % went from 0.565 to 0.11 won    -80.53%

Alas, it turns out it would have been better to set spam_cutoff to 0.6 on
this run.  That would have cut 9 instances of fp and added 33 instances of
fn (note that since each msg is scored 4 times, it's *not* necessarily the
case that these deltas would reflect directly in the "total unique" counts),
leaving the two roughly comparable on this test.

I predict (but haven't yet tried it) that I'd get an improvement in the
central-limit scheme by systematically ignoring low-value words too.

Finally, the same thing with log central limit on the left, and our Graham
scheme on the right (16 discriminators, spam cutoff 0.90).

false positive percentages
    0.000  0.000  tied
    0.000  0.300  lost  +(was 0)
    0.000  0.000  tied
    0.200  0.200  tied
    0.000  0.000  tied
    0.000  0.100  lost  +(was 0)
    0.000  0.000  tied
    0.100  0.200  lost  +100.00%
    0.000  0.000  tied
    0.000  0.100  lost  +(was 0)
    0.000  0.000  tied
    0.100  0.200  lost  +100.00%
    0.000  0.100  lost  +(was 0)
    0.000  0.100  lost  +(was 0)
    0.000  0.100  lost  +(was 0)
    0.100  0.300  lost  +200.00%
    0.000  0.100  lost  +(was 0)
    0.000  0.000  tied
    0.000  0.300  lost  +(was 0)
    0.000  0.000  tied

won   0 times
tied  9 times
lost 11 times

total unique fp went from 2 to 10 lost  +400.00%
mean fp % went from 0.025 to 0.105 lost  +320.00%

false negative percentages
    0.800  0.100  won    -87.50%
    0.900  0.100  won    -88.89%
    0.600  0.300  won    -50.00%
    0.400  0.200  won    -50.00%
    1.100  0.300  won    -72.73%
    0.700  0.400  won    -42.86%
    0.600  0.300  won    -50.00%
    0.700  0.100  won    -85.71%
    0.600  0.100  won    -83.33%
    0.400  0.000  won   -100.00%
    0.700  0.200  won    -71.43%
    0.400  0.200  won    -50.00%
    0.600  0.100  won    -83.33%
    0.500  0.100  won    -80.00%
    0.500  0.100  won    -80.00%
    0.300  0.200  won    -33.33%
    0.400  0.100  won    -75.00%
    0.400  0.100  won    -75.00%
    0.400  0.200  won    -50.00%
    0.300  0.200  won    -33.33%

won  20 times
tied  0 times
lost  0 times

total unique fn went from 54 to 16 won    -70.37%
mean fn % went from 0.565 to 0.17 won    -69.91%

All of Gary's schemes beat the fiddled Graham scheme on fp rate here, but
the Graham scheme remains unbeaten on the fn rate.  (I should add, of
course, "as implemented by Tim" to all of those.)


From anthony@interlink.com.au  Tue Sep 24 04:39:59 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Tue, 24 Sep 2002 13:39:59 +1000
Subject: [Spambayes] more testing: spam/ham proportions?
Message-ID: <200209240339.g8O3dxw22573@localhost.localdomain>


Ok, as well as my humongous test set, I've now got a largish collection
of spams from the last couple of weeks, and I'm about to pair them up 
with randomly selected hams (but selected from messages from the same
time window, obviously). 

What's the best spam/ham proportion to go with on the current set of 
things to be tested? As close to the real proportions as possible? 
1:1? 2:1? 10:1?

Anthony

From tim.one@comcast.net  Tue Sep 24 05:03:03 2002
From: tim.one@comcast.net (Tim Peters)
Date: Tue, 24 Sep 2002 00:03:03 -0400
Subject: [Spambayes] RE: Central Limit Theorem??!!     :)
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEFJBGAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCOEFNBGAB.tim.one@comcast.net>

[Tim]
> ...
> I predict (but haven't yet tried it) that I'd get an improvement in
> the central-limit scheme by systematically ignoring low-value words
> too.

Yup.  Same test setup and data, both runs using the logarithmic central
limit scheme, with 30 extremes, f(w), a=0.1; the only difference is that the
run on the right pretends that words with spamprob in (0.4, 0.6) don't exist
(the code that extracts the 30 highest-value words (spamprob farthest from
0.5) from a msg simply skips over words that aren't at least 0.1 away from
0.5):

-> <stat> tested 1000 hams & 1000 spams against 1000 hams & 1000 spams
   [ditto 39 times]

false positive percentages
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.200  0.200  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.100  0.100  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.100  0.100  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.100  0.100  tied
    0.000  0.000  tied
    0.000  0.100  lost  +(was 0)
    0.000  0.000  tied
    0.000  0.000  tied

won   0 times
tied 19 times
lost  1 times

total unique fp went from 2 to 3 lost   +50.00%
mean fp % went from 0.025 to 0.03 lost   +20.00%

false negative percentages
    0.800  0.500  won    -37.50%
    0.900  0.400  won    -55.56%
    0.600  0.300  won    -50.00%
    0.400  0.300  won    -25.00%
    1.100  0.600  won    -45.45%
    0.700  0.600  won    -14.29%
    0.600  0.400  won    -33.33%
    0.700  0.400  won    -42.86%
    0.600  0.300  won    -50.00%
    0.400  0.000  won   -100.00%
    0.700  0.200  won    -71.43%
    0.400  0.200  won    -50.00%
    0.600  0.300  won    -50.00%
    0.500  0.200  won    -60.00%
    0.500  0.300  won    -40.00%
    0.300  0.200  won    -33.33%
    0.400  0.000  won   -100.00%
    0.400  0.100  won    -75.00%
    0.400  0.100  won    -75.00%
    0.300  0.300  tied

won  19 times
tied  1 times
lost  0 times

total unique fn went from 54 to 29 won    -46.30%
mean fn % went from 0.565 to 0.285 won    -49.56%

So ignoring near-neutral words is a huge and pure win for the fn rate here
too.  I'm still at a loss to articulate why I think that's necessarily true,
regardless of scoring scheme, on this specific task -- but if you think it's
just a hack, argue with my data <wink>.

The scaled "scores" don't really make any sense under this scheme, but they
do have distributions <ahem>, and even working with senseless scores, the
variance of the spam scores decreases when ignoring near-neutral words:

ham mean                 ham sdev
0.09 0.09 (+0.00%)     1.89 1.93 (+2.12%)
0.12 0.12 (+0.00%)     2.20 2.22 (+0.91%)
0.15 0.15 (+0.00%)     2.13 2.18 (+2.35%)
0.15 0.15 (+0.00%)     2.39 2.44 (+2.09%)
0.16 0.15 (-6.25%)     1.97 2.01 (+2.03%)

ham mean and sdev for all runs
0.14 0.13 (-7.14%)     2.13 2.16 (+1.41%)

spam mean                spam sdev
97.90 98.32 (+0.43%)     8.35 7.02 (-15.93%)
97.88 98.26 (+0.39%)     8.42 7.21 (-14.37%)
98.41 98.85 (+0.45%)     7.12 5.58 (-21.63%)
98.22 98.67 (+0.46%)     7.48 6.01 (-19.65%)
98.44 98.85 (+0.42%)     7.03 5.74 (-18.35%)

spam mean and sdev for all runs
98.17 98.59 (+0.43%)     7.71 6.35 (-17.64%)

ham/spam mean difference: 98.03 98.46 +0.43


From tim.one@comcast.net  Tue Sep 24 05:19:57 2002
From: tim.one@comcast.net (Tim Peters)
Date: Tue, 24 Sep 2002 00:19:57 -0400
Subject: [Spambayes] more testing: spam/ham proportions?
In-Reply-To: <200209240339.g8O3dxw22573@localhost.localdomain>
Message-ID: <LNBBLJKPBEHFEDALKOLCMEFOBGAB.tim.one@comcast.net>

[Anthony Baxter]
> Ok, as well as my humongous test set, I've now got a largish collection
> of spams from the last couple of weeks, and I'm about to pair them up
> with randomly selected hams (but selected from messages from the same
> time window, obviously).

Excellent!

> What's the best spam/ham proportion to go with on the current set of
> things to be tested? As close to the real proportions as possible?
> 1:1? 2:1? 10:1?

My advice is get as close to real life as you can.  Every deviation just
opens the door for a hundred gremlins to sneak in.  I can't say that I know
of any specific gremlin here, but nobody has run controlled experiments to
see whether or how much this matters, and in my view that means we don't
know anything about it yet -- we can only guess.

You could do us a great service by picking one of the better schemes on the
table, use a real-life ham-to-spam ratio, tune the relevant parameters until
you're tired of that, and then stop fiddling.  After that point, change only
one thing:  the ham-to-spam ratio you test on.  Keep records and stick to
the agenda.  Report what you find out.  Note that at least timcv and timtest
have cmdline arguments allowing to use random, but reproducible across runs,
subsets of your ham and your spam, and allow to specify different numbers
for the ham and the spam sets to use.  This lets you test on any ratio you
like.  I'd do three full runs of the test at each ratio of interest, forcing
a different random seed (-s option) on each full run.

I'd first aim to keep the total number of ham+spam msgs fixed as you vary
the ratio, as some schemes on the table have some parameters whose best
settings seem to vary with corpus size (there are no good reports of
controlled experiments on that either, alas).  If the ratio makes a
significant difference, you'll soon find out.  Else remind me in 6 months,
and I'll remind you that there's no point continuing to test something that
doesn't make any difference <win>.


From anthony@interlink.com.au  Tue Sep 24 05:33:05 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Tue, 24 Sep 2002 14:33:05 +1000
Subject: [Spambayes] more testing: spam/ham proportions? 
In-Reply-To: <LNBBLJKPBEHFEDALKOLCMEFOBGAB.tim.one@comcast.net> 
Message-ID: <200209240433.g8O4X7123123@localhost.localdomain>



> > What's the best spam/ham proportion to go with on the current set of
> > things to be tested? As close to the real proportions as possible?
> > 1:1? 2:1? 10:1?
> My advice is get as close to real life as you can.  Every deviation just
> opens the door for a hundred gremlins to sneak in.  I can't say that I know
> of any specific gremlin here, but nobody has run controlled experiments to
> see whether or how much this matters, and in my view that means we don't
> know anything about it yet -- we can only guess.

Hm. I haven't kept all ham for the last couple of weeks, but I can certainly
grab a fair slab of it.

> You could do us a great service by picking one of the better schemes on the
> table,

Suggestions? I'm starting to feel a little dizzy at the number of options 
at the moment - what do you want tested? For that matter, what should I 
be feeding my mungo-set into? Courtesy of several hours of boredom waiting 
for my partner to come out of surgery, I _think_ I've now got the complete 
set of info@ekit.com email integrated and most of the badly filed stuff out 
of the way. There's 10 sets each of 1000 spam and 1950 ham. Unfortunately,
while the laptop has a 1.something gig cpu, it's only got 256M of RAM, so
running the big tests takes a fair while...

Anthony

-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.


From anthony@interlink.com.au  Tue Sep 24 06:39:58 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Tue, 24 Sep 2002 15:39:58 +1000
Subject: [Spambayes] memory consumption...
Message-ID: <200209240540.g8O5e2J23940@localhost.localdomain>


For my standard test, using the graham scheme, memory consumption's blown
out considerably in the last couple of days. timcv -n 10 used to consume
about 50M - it's now about 75M. This unfortunately pushes my system into
swap :-( is there anything obvious to change to make it happy again?



From richie@entrian.com  Tue Sep 24 08:31:23 2002
From: richie@entrian.com (Richie Hindle)
Date: Tue, 24 Sep 2002 08:31:23 +0100
Subject: [Spambayes] Re: Practical Applications 
In-Reply-To: <200209240047.g8O0lR720595@localhost.localdomain>
References: <B9B4A555.16572%grobinson@transpose.com>
	<200209240047.g8O0lR720595@localhost.localdomain>
Message-ID: <5c40pu0btq2qmifl27nq2037fjqcpvms88@4ax.com>

Anthony,

> I'd suggest an alternate approach - 
> - for the inbound, something like pop3proxy.py, but it should also
> keep a copy of (some/all) messages as 'ham' samples.
> 
> - for marking as spam, a small application (maybe in VBA) that
> copies the current message into a magic location on disk.

Also in the "Random thoughts before the first coffee" category is this:
for email clients to which you can't add such a function, we could make
pop3proxy into an SMTP proxy for outgoing mail and define an email address
'spam' to which you bulk-forward spams.  The proxy would retrain on those
messages, and not forward them on to the mail server.

My email client (not sure about others) lets me forward a batch of
messages together, so it would only be a couple of keystrokes and would
work on all platforms without need to plug into the email clients.  It
also means you don't need to explicitly run the classifier, or schedule it
to run regularly, to retrain it on new spams.

The biggest problem with either scheme is which mails you can assume are
ham - you should really have to hit a copy-to-ham-folder button or forward
them to a 'ham' address in order to guarantee that no spam gets filed as
ham (looking through the past messages I can't see a previous discussion
on this, but I'm sure there was one - the phrase 'auto-ham' is buzzing
round my head.  Probably a pre-coffee problem.)

-- 
Richie Hindle
richie@entrian.com


From grobinson@transpose.com  Tue Sep 24 12:31:20 2002
From: grobinson@transpose.com (Gary Robinson)
Date: Tue, 24 Sep 2002 07:31:20 -0400
Subject: [Spambayes] RE: Central Limit Theorem??!!     :)
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEFJBGAB.tim.one@comcast.net>
Message-ID: <B9B5C548.166B0%grobinson@transpose.com>

> All of Gary's schemes beat the fiddled Graham scheme on fp rate here, but
> the Graham scheme remains unbeaten on the fn rate.  (I should add, of
> course, "as implemented by Tim" to all of those.)

Summarizing all the test results, I get

         fn  fp
f(w):    11  6
Graham:  16  10
lcm:     29  3

where f(w) is pure f(w) with a=0.1, 1500 discrimators, and
robinson_minimum_prob_strength=0.1, Graham is tweaked Graham, and lcm is
log-central-mean ignoring the middling values.

So doesn't the f(w) technique beat Graham all-around? At least in this
latest round of testing??? Or am I misreading the  results?

I wonder if the errors for lcm are mostly in the region where there are a
small number of data points, such that the central mean theorem isn't really
kicking in.

That is, it may be that there is some number n for which f(w) gives the best
results if the number of non-middling words is < n and and lcm gives the
best results if the number of non-middling words is > n.

That WOULD make a lot of theoretical sense, because for small enough n, the
central mean theorem is meaningless and can only make trouble for us.

Something else for you to test in your copious free time. ;)


--Gary


-- 
Gary Robinson
CEO
Transpose, LLC
grobinson@transpose.com
207-942-3463
http://www.emergentmusic.com
http://radio.weblogs.com/0101454


> From: Tim Peters <tim.one@comcast.net>
> Date: Mon, 23 Sep 2002 23:22:13 -0400
> To: Gary Robinson <grobinson@transpose.com>
> Cc: SpamBayes <spambayes@python.org>, Greg Louis <glouis@dynamicro.on.ca>
> Subject: RE: [Spambayes] RE: Central Limit Theorem??!!     :)
> 
> [Gary Robinson, outlines another central limit scheme, but using
> ln(f(w)) for spam scores and ln(1-f(w)) for ham scores when
> building the ham and spam "extreme word" populations, then computing
> two corresponding log sums when scoring.
> ]
> 
>> ...
>> This *should* bring us the benefits of the multiplicative approach
>> and of the parametric stats approach at the same time.
> 
> My first "real test" says it's a winner!  See below.
> 
>> The downside of this is that the ln's make such a skewed distribution
>> that it should take a bigger n to make the central limit theorem kick
>> in.
> 
> This isn't a good time to mention it <wink>, but the original central-limit
> code worked better when I reduced the # of extremes from 30 to 15.  Note
> that these are the kinds of oddities I spent countless hours "tuning away"
> on the Graham scheme, though, and I;ve still done almost nothing of that
> nature for your schemes.
> 
>> BUT, OTOH, it WILL kick in, and something like 30 may really still
>> be enough (it usually kicks in at significantly smaller numbers).
> 
> I predict more experiments on the horizon.
> 
>> And you've also successfully used n=150 with f(w)
> 
> I've also used n=15 and n=1500, BTW.  150->1500 neither helped nor hurt.
> 150->15 gave mixed results, too close to call across the few small tests I
> ran.
> 
>> and that is DEFINITELY enough.
>> 
>> The other downside is that it just seems like a bit  of a wild thing
> < to do and sometimes when you do wild things, strange reasons emerge
>> why they won't work.
> 
> It's an extreme problem, though.  I ran experiments long ago training on a
> single ham and a single ham:  that was enough to get about 30% fp rate and
> about 20% fn rate.  *Much* better than flipping a coin!  Training on two of
> each did even better <wink>.
> 
> But I really can't see any at this point as long as n is big enough
>> that the sample means take on a normal distribution.
>> 
>> THANKS for doing all the coding work to test this idea!!!!  :)
> 
> Thank me for the testing, if you have to thank me at all -- the coding is
> fun, the testing is mind-numbing <drool>.
> 
> Speaking of which, the 3-pass nature of training under this scheme doesn't
> work well with my cross-validation driver.  Instead the test here is on a
> 5x5 test grid, skipping the diagonal.  I picked 5 disjoint sets of 1000 ham
> each at random, and 5 disjoint sets of 1000 spam each at random.  These were
> then paired into 5 collections, 1 thru 5.  First I trained on pair 1, then
> predicted on pairs 2, 3, 4, and 5.  Then I trained a new classifier on pair
> 2, and predicted on pairs 1, 3, 4, and 5.  Etc.  In all we get 20 test runs
> out of this, and each message is predicted 4 times.
> 
> In both runs, 30 extremes were used, and f(w) with a=0.1.  The original
> central limit code is on the left, and the logarithmic version on the right.
> I haven't done anything to do a more sensible scaling of scores into [0, 1],
> so I'll skip histograms.
> 
> -> <stat> tested 1000 hams & 1000 spams against 1000 hams & 1000 spams
>  [ditto 39 times]
> 
> false positive percentages
>   0.000  0.000  tied
>   0.000  0.000  tied
>   0.000  0.000  tied
>   0.200  0.200  tied
>   0.000  0.000  tied
>   0.000  0.000  tied
>   0.000  0.000  tied
>   0.100  0.100  tied
>   0.000  0.000  tied
>   0.000  0.000  tied
>   0.000  0.000  tied
>   0.100  0.100  tied
>   0.000  0.000  tied
>   0.000  0.000  tied
>   0.100  0.000  won   -100.00%
>   0.100  0.100  tied
>   0.000  0.000  tied
>   0.000  0.000  tied
>   0.000  0.000  tied
>   0.000  0.000  tied
> 
> won   1 times
> tied 19 times
> lost  0 times
> 
> total unique fp went from 3 to 2 won    -33.33%
> mean fp % went from 0.03 to 0.025 won    -16.67%
> 
> false negative percentages
>   2.200  0.800  won    -63.64%
>   2.100  0.900  won    -57.14%
>   1.700  0.600  won    -64.71%
>   1.900  0.400  won    -78.95%
>   1.500  1.100  won    -26.67%
>   1.900  0.700  won    -63.16%
>   1.600  0.600  won    -62.50%
>   1.600  0.700  won    -56.25%
>   1.400  0.600  won    -57.14%
>   1.900  0.400  won    -78.95%
>   1.600  0.700  won    -56.25%
>   1.600  0.400  won    -75.00%
>   1.900  0.600  won    -68.42%
>   1.600  0.500  won    -68.75%
>   1.300  0.500  won    -61.54%
>   1.600  0.300  won    -81.25%
>   1.300  0.400  won    -69.23%
>   1.900  0.400  won    -78.95%
>   1.300  0.400  won    -69.23%
>   1.400  0.300  won    -78.57%
> 
> won  20 times
> tied  0 times
> lost  0 times
> 
> total unique fn went from 125 to 54 won    -56.80%
> mean fn % went from 1.665 to 0.565 won    -66.07%
> 
> So the logarithmic version was indeed a huge and pure win for the fn rate!
> 
> There are two other comparisons I need to make here:  against your webpage
> scheme (f(w) with "S"), and against our highly fiddled Graham scheme.
> 
> First the same thing (same test msgs, etc), with log central-limit on the
> left, and f(w) with a=0.1, 1500 discrimators, and
> robinson_minimum_prob_strength=0.1 (a gimmick that simply ignores all words
> with spamprob in (0.4, 0.6)) on the right:
> 
> [Classifier]
> use_robinson_probability: True
> use_robinson_combining: True
> max_discriminators: 1500
> robinson_minimum_prob_strength: 0.1
> robinson_probability_a: 0.1
> use_central_limit: False
> 
> [TestDriver]
> spam_cutoff: 0.575
> 
> false positive percentages
>   0.000  0.000  tied
>   0.000  0.100  lost  +(was 0)
>   0.000  0.000  tied
>   0.200  0.200  tied
>   0.000  0.000  tied
>   0.000  0.100  lost  +(was 0)
>   0.000  0.000  tied
>   0.100  0.300  lost  +200.00%
>   0.000  0.000  tied
>   0.000  0.100  lost  +(was 0)
>   0.000  0.000  tied
>   0.100  0.200  lost  +100.00%
>   0.000  0.000  tied
>   0.000  0.100  lost  +(was 0)
>   0.000  0.100  lost  +(was 0)
>   0.100  0.200  lost  +100.00%
>   0.000  0.000  tied
>   0.000  0.100  lost  +(was 0)
>   0.000  0.100  lost  +(was 0)
>   0.000  0.000  tied
> 
> won   0 times
> tied 10 times
> lost 10 times
> 
> total unique fp went from 2 to 6 lost  +200.00%
> mean fp % went from 0.025 to 0.08 lost  +220.00%
> 
> false negative percentages
>   0.800  0.100  won    -87.50%
>   0.900  0.000  won   -100.00%
>   0.600  0.200  won    -66.67%
>   0.400  0.200  won    -50.00%
>   1.100  0.100  won    -90.91%
>   0.700  0.100  won    -85.71%
>   0.600  0.200  won    -66.67%
>   0.700  0.100  won    -85.71%
>   0.600  0.000  won   -100.00%
>   0.400  0.000  won   -100.00%
>   0.700  0.200  won    -71.43%
>   0.400  0.200  won    -50.00%
>   0.600  0.100  won    -83.33%
>   0.500  0.100  won    -80.00%
>   0.500  0.000  won   -100.00%
>   0.300  0.100  won    -66.67%
>   0.400  0.100  won    -75.00%
>   0.400  0.100  won    -75.00%
>   0.400  0.000  won   -100.00%
>   0.300  0.300  tied
> 
> won  19 times
> tied  1 times
> lost  0 times
> 
> total unique fn went from 54 to 11 won    -79.63%
> mean fn % went from 0.565 to 0.11 won    -80.53%
> 
> Alas, it turns out it would have been better to set spam_cutoff to 0.6 on
> this run.  That would have cut 9 instances of fp and added 33 instances of
> fn (note that since each msg is scored 4 times, it's *not* necessarily the
> case that these deltas would reflect directly in the "total unique" counts),
> leaving the two roughly comparable on this test.
> 
> I predict (but haven't yet tried it) that I'd get an improvement in the
> central-limit scheme by systematically ignoring low-value words too.
> 
> Finally, the same thing with log central limit on the left, and our Graham
> scheme on the right (16 discriminators, spam cutoff 0.90).
> 
> false positive percentages
>   0.000  0.000  tied
>   0.000  0.300  lost  +(was 0)
>   0.000  0.000  tied
>   0.200  0.200  tied
>   0.000  0.000  tied
>   0.000  0.100  lost  +(was 0)
>   0.000  0.000  tied
>   0.100  0.200  lost  +100.00%
>   0.000  0.000  tied
>   0.000  0.100  lost  +(was 0)
>   0.000  0.000  tied
>   0.100  0.200  lost  +100.00%
>   0.000  0.100  lost  +(was 0)
>   0.000  0.100  lost  +(was 0)
>   0.000  0.100  lost  +(was 0)
>   0.100  0.300  lost  +200.00%
>   0.000  0.100  lost  +(was 0)
>   0.000  0.000  tied
>   0.000  0.300  lost  +(was 0)
>   0.000  0.000  tied
> 
> won   0 times
> tied  9 times
> lost 11 times
> 
> total unique fp went from 2 to 10 lost  +400.00%
> mean fp % went from 0.025 to 0.105 lost  +320.00%
> 
> false negative percentages
>   0.800  0.100  won    -87.50%
>   0.900  0.100  won    -88.89%
>   0.600  0.300  won    -50.00%
>   0.400  0.200  won    -50.00%
>   1.100  0.300  won    -72.73%
>   0.700  0.400  won    -42.86%
>   0.600  0.300  won    -50.00%
>   0.700  0.100  won    -85.71%
>   0.600  0.100  won    -83.33%
>   0.400  0.000  won   -100.00%
>   0.700  0.200  won    -71.43%
>   0.400  0.200  won    -50.00%
>   0.600  0.100  won    -83.33%
>   0.500  0.100  won    -80.00%
>   0.500  0.100  won    -80.00%
>   0.300  0.200  won    -33.33%
>   0.400  0.100  won    -75.00%
>   0.400  0.100  won    -75.00%
>   0.400  0.200  won    -50.00%
>   0.300  0.200  won    -33.33%
> 
> won  20 times
> tied  0 times
> lost  0 times
> 
> total unique fn went from 54 to 16 won    -70.37%
> mean fn % went from 0.565 to 0.17 won    -69.91%
> 
> All of Gary's schemes beat the fiddled Graham scheme on fp rate here, but
> the Graham scheme remains unbeaten on the fn rate.  (I should add, of
> course, "as implemented by Tim" to all of those.)
> 


From sjoerd@acm.org  Tue Sep 24 16:00:29 2002
From: sjoerd@acm.org (Sjoerd Mullender)
Date: Tue, 24 Sep 2002 17:00:29 +0200
Subject: [Spambayes] Moving closer to Gary's ideal
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCECIBGAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCCECIBGAB.tim.one@comcast.net> 
Message-ID: <200209241500.g8OF0TK17719@indus.ins.cwi.nl>

On Mon, Sep 23 2002 Tim Peters wrote:

> [Sjoerd Mullender]
> > > """
> > > [Classifier]
> > > use_robinson_probability: True
> > > use_robinson_combining: True
> > > max_discriminators: 1500
> > >
> > > [TestDriver]
> > > spam_cutoff: 0.50
> > > """
> >
> > I tested this against the default options (except I have
> > count_all_header_lines: True and mine_received_headers: True
> > permanently)
> 
> I'm jealous!  If you can use those, you can probably enable Jeremy's
> basic_header_tokenize too.

I tried two tests to test this hypothesis.
In the first I added
"""
[Tokenize]
basic_header_tokenize: True
"""
to the options above and in the second I added
"basic_header_tokenize_only: True" as well.
They were both loosers compared to the above settings, although I
don't know how significant these results are.

First comparison:

false positive percentages
    0.000  0.250  lost  +(was 0)
    0.000  0.250  lost  +(was 0)
    0.250  0.250  tied          
    0.250  0.375  lost   +50.00%
    0.250  0.500  lost  +100.00%

won   0 times
tied  1 times
lost  4 times

total unique fp went from 6 to 13 lost  +116.67%
mean fp % went from 0.15 to 0.325 lost  +116.67%

false negative percentages
    1.000  1.500  lost   +50.00%
    0.500  1.000  lost  +100.00%
    0.500  1.500  lost  +200.00%
    1.500  2.000  lost   +33.33%
    1.500  0.500  won    -66.67%

won   1 times
tied  0 times
lost  4 times

total unique fn went from 10 to 13 lost   +30.00%
mean fn % went from 1.0 to 1.3 lost   +30.00%

(Moving the spam_cutoff to 0.525 I get 4 fp and 22 fn.)

Second comparison:

false positive percentages
    0.000  0.250  lost  +(was 0)
    0.000  0.000  tied          
    0.250  0.500  lost  +100.00%
    0.250  0.625  lost  +150.00%
    0.250  0.750  lost  +200.00%

won   0 times
tied  1 times
lost  4 times

total unique fp went from 6 to 17 lost  +183.33%
mean fp % went from 0.15 to 0.425 lost  +183.33%

false negative percentages
    1.000  1.000  tied          
    0.500  0.500  tied          
    0.500  1.500  lost  +200.00%
    1.500  1.500  tied          
    1.500  0.500  won    -66.67%

won   1 times
tied  3 times
lost  1 times

total unique fn went from 10 to 10 tied          
mean fn % went from 1.0 to 1.0 tied

(Moving the spam_cutoff to 0.525 I get 12 fp and 20 fn.)

-- Sjoerd Mullender <sjoerd@acm.org>

From tim@zope.com  Tue Sep 24 18:19:23 2002
From: tim@zope.com (Tim Peters)
Date: Tue, 24 Sep 2002 13:19:23 -0400
Subject: [Spambayes] memory consumption...
In-Reply-To: <200209240540.g8O5e2J23940@localhost.localdomain>
Message-ID: <BIEJKCLHCIOIHAGOKOLHCEOMDKAA.tim@zope.com>

[Anthony Baxter]
> For my standard test, using the graham scheme, memory consumption's blown
> out considerably in the last couple of days. timcv -n 10 used to consume
> about 50M - it's now about 75M. This unfortunately pushes my system into
> swap :-( is there anything obvious to change to make it happy again?

If you haven't changed the set of msgs you're using, and neither have you
changed the set of options you're using, then that's a mystery.

Nothing has changed in the codebase in "the last couple of days" that would
account for this, unless you have a great many messages that the email pkg
just can't parse.  If that's the case, then a little over 4 days ago changes
were checked in to fall back to the raw text instead of giving up.

BTW, I also have 256MB RAM on my test machine, and of course it goes to disk
a lot when running my test:  it's opening 34,000 files multiple times.
timcv is carefully arranged to minimize that burden after the initial pain
of initial training on 9/10ths of all msgs.

Whether 50-75M "is excessive" can't be guessed without knowing how many
messages you're using and the full set of options you're running with.  If,
e.g., you didn't used to save binary pickles, but started to, memory use is
bound to blow up (the under-the-covers pickle memo dict is huge).


From neale@woozle.org  Tue Sep 24 18:50:53 2002
From: neale@woozle.org (Neale Pickett)
Date: 24 Sep 2002 10:50:53 -0700
Subject: [Spambayes] memory consumption...
In-Reply-To: <200209240540.g8O5e2J23940@localhost.localdomain>
References: <200209240540.g8O5e2J23940@localhost.localdomain>
Message-ID: <w531y7j5laa.fsf@woozle.org>

So then, Anthony Baxter <anthony@interlink.com.au> is all like:

> For my standard test, using the graham scheme, memory consumption's blown
> out considerably in the last couple of days. timcv -n 10 used to consume
> about 50M - it's now about 75M. This unfortunately pushes my system into
> swap :-( is there anything obvious to change to make it happy again?

At the risk of being chased off the list, we could start using a
database instead of a giant in-memory dict...  *duck*

Or you could always head down to your local Geeks-R'-Us and pick up more
RAM.  I hear it's pretty cheap these days.

Neale

From skip@pobox.com  Tue Sep 24 19:06:37 2002
From: skip@pobox.com (Skip Montanaro)
Date: Tue, 24 Sep 2002 13:06:37 -0500
Subject: [Spambayes] memory consumption...
In-Reply-To: <w531y7j5laa.fsf@woozle.org>
References: <200209240540.g8O5e2J23940@localhost.localdomain>
        <w531y7j5laa.fsf@woozle.org>
Message-ID: <15760.43437.700635.201087@12-248-11-90.client.attbi.com>


    Neale> Or you could always head down to your local Geeks-R'-Us and pick
    Neale> up more RAM.  I hear it's pretty cheap these days.

Yes indeed.  Just bought 1gig for my new Powerbook for < $200 including
shipping.

Skip


From Alexander@Leidinger.net  Tue Sep 24 18:24:45 2002
From: Alexander@Leidinger.net (Alexander Leidinger)
Date: Tue, 24 Sep 2002 19:24:45 +0200
Subject: [Spambayes] Traceback with mboxcount
Message-ID: <20020924192445.471168fd.Alexander@Leidinger.net>

This is a multi-part message in MIME format.

---------------------- multipart/mixed attachment
Hi,

I get this with mboxcount of today (my modification to it to print a
total of all good and bad mails is attached):
---snip---
Traceback (most recent call last):
  File "../../mboxcount.py", line 97, in ?
    main()
  File "../../mboxcount.py", line 88, in main
    goodn, badn = count(fname)
  File "../../mboxcount.py", line 57, in count
    badcount = 0
  File "/usr/local/lib/python2.2/mailbox.py", line 34, in next
    return self.factory(_Subfile(self.fp, start, stop))
  File "../../mboxcount.py", line 48, in _factory
    try:
  File "/home/netchild/work/src/spambayes/email/__init__.py", line 39, in message_from_file
    return _Parser(_class, strict=strict).parse(fp)
  File "/home/netchild/work/src/spambayes/email/Parser.py", line 48, in parse
    self._parsebody(root, fp)
  File "/home/netchild/work/src/spambayes/email/Parser.py", line 118, in _parsebody
    boundary = container.get_boundary()
  File "/home/netchild/work/src/spambayes/email/Message.py", line 677, in get_boundary
    return _unquotevalue(boundary.strip())
AttributeError: 'tuple' object has no attribute 'strip'
---snip---

This is with python 2.2.1 from the FreeBSD ports collection.

Bye,
Alexander.

-- 
Failure is not an option. It comes bundled with your Microsoft product.

http://www.Leidinger.net                       Alexander @ Leidinger.net
  GPG fingerprint = C518 BC70 E67F 143F BE91  3365 79E2 9C60 B006 3FE7

---------------------- multipart/mixed attachment
A non-text attachment was scrubbed...
Name: mboxcount.py.diff
Type: application/octet-stream
Size: 1512 bytes
Desc: not available
Url : http://mail.python.org/pipermail-21/spambayes/attachments/20020924/071ff7bd/mboxcount.py.exe

---------------------- multipart/mixed attachment--

From Alexander@Leidinger.net  Tue Sep 24 18:47:39 2002
From: Alexander@Leidinger.net (Alexander Leidinger)
Date: Tue, 24 Sep 2002 19:47:39 +0200
Subject: [Spambayes] Patch: Add globbing to splitndirs.py
Message-ID: <20020924194739.5461ca01.Alexander@Leidinger.net>

This is a multi-part message in MIME format.

---------------------- multipart/mixed attachment
Hi,

the attached patch adds globbing to splitndirs.py. This is if the length
of your command line with every mbox file you want to split exceeds the
max. possible command line length of the shell.

Bye,
Alexander.

-- 
            0 and 1. Now what could be so hard about that?

http://www.Leidinger.net                       Alexander @ Leidinger.net
  GPG fingerprint = C518 BC70 E67F 143F BE91  3365 79E2 9C60 B006 3FE7

---------------------- multipart/mixed attachment
A non-text attachment was scrubbed...
Name: splitndirs.py.diff
Type: application/octet-stream
Size: 4134 bytes
Desc: not available
Url : http://mail.python.org/pipermail-21/spambayes/attachments/20020924/6b64032a/splitndirs.py.exe

---------------------- multipart/mixed attachment--

From skip@pobox.com  Tue Sep 24 19:19:30 2002
From: skip@pobox.com (Skip Montanaro)
Date: Tue, 24 Sep 2002 13:19:30 -0500
Subject: [Spambayes] ini files in CVS?
Message-ID: <15760.44210.522958.711607@12-248-11-90.client.attbi.com>

For easier (and less error-prone) comparison of results between different
people I think it would be a good idea to add some ini files to CVS which
setup various key configurations (graham, robinson, lcm, etc).  

Skip


From guido@python.org  Tue Sep 24 19:30:13 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 24 Sep 2002 14:30:13 -0400
Subject: [Spambayes] Patch: Add globbing to splitndirs.py
In-Reply-To: Your message of "Tue, 24 Sep 2002 19:47:39 +0200."
             <20020924194739.5461ca01.Alexander@Leidinger.net> 
References: <20020924194739.5461ca01.Alexander@Leidinger.net> 
Message-ID: <200209241830.g8OIUDT28426@odiug.zope.com>

> the attached patch adds globbing to splitndirs.py. This is if the
> length of your command line with every mbox file you want to split
> exceeds the max. possible command line length of the shell.

Cute.  I've checked this in.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From Alexander@Leidinger.net  Tue Sep 24 19:39:15 2002
From: Alexander@Leidinger.net (Alexander Leidinger)
Date: Tue, 24 Sep 2002 20:39:15 +0200
Subject: [Spambayes] Patch: Add globbing to splitndirs.py
In-Reply-To: <200209241830.g8OIUDT28426@odiug.zope.com>
References: <20020924194739.5461ca01.Alexander@Leidinger.net>
	<200209241830.g8OIUDT28426@odiug.zope.com>
Message-ID: <20020924203915.66811368.Alexander@Leidinger.net>

On Tue, 24 Sep 2002 14:30:13 -0400 Guido van Rossum <guido@python.org>
wrote:

> Cute.  I've checked this in.

I've the same problem with splitndirs.py as with mboxcount.py (see other
mail with the traceback). There seems to be something wrong eith the
email class... at least with 2.2.1.

mboxcount also says "0 mails" for 115 files here. I haven't looked at
everyone of them, but the first 5 I looked at had some content, and I
haven't seen something in this content, which looked like a bug in the
mbox file.

Is someone interested in looking at those mails?

Bye,
Alexander.

-- 
         The computer revolution is over. The computers won.

http://www.Leidinger.net                       Alexander @ Leidinger.net
  GPG fingerprint = C518 BC70 E67F 143F BE91  3365 79E2 9C60 B006 3FE7

From guido@python.org  Tue Sep 24 19:50:00 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 24 Sep 2002 14:50:00 -0400
Subject: [Spambayes] Patch: Add globbing to splitndirs.py
In-Reply-To: Your message of "Tue, 24 Sep 2002 20:39:15 +0200."
             <20020924203915.66811368.Alexander@Leidinger.net> 
References: <20020924194739.5461ca01.Alexander@Leidinger.net>
	<200209241830.g8OIUDT28426@odiug.zope.com>  
	<20020924203915.66811368.Alexander@Leidinger.net> 
Message-ID: <200209241850.g8OIo0d28489@odiug.zope.com>

Are you perhaps using individual files (like those created by MH or
GNUS) that aren't in mbox format???

Mbox format requires each message to be preceded by a "Unix From"
line, of the form

>From guido@python.org  Tue Sep 24 14:49:21 2002

(Note that there's a space, no colon after "From".)

--Guido van Rossum (home page: http://www.python.org/~guido/)

From Alexander@Leidinger.net  Tue Sep 24 20:01:14 2002
From: Alexander@Leidinger.net (Alexander Leidinger)
Date: Tue, 24 Sep 2002 21:01:14 +0200
Subject: [Spambayes] Patch: Add globbing to splitndirs.py
In-Reply-To: <200209241850.g8OIo0d28489@odiug.zope.com>
References: <20020924194739.5461ca01.Alexander@Leidinger.net>
	<200209241830.g8OIUDT28426@odiug.zope.com>
	<20020924203915.66811368.Alexander@Leidinger.net>
	<200209241850.g8OIo0d28489@odiug.zope.com>
Message-ID: <20020924210114.25d924a1.Alexander@Leidinger.net>

On Tue, 24 Sep 2002 14:50:00 -0400 Guido van Rossum <guido@python.org>
wrote:

> Are you perhaps using individual files (like those created by MH or
> GNUS) that aren't in mbox format???

No.

> Mbox format requires each message to be preceded by a "Unix From"
> line, of the form
> 
> >From guido@python.org  Tue Sep 24 14:49:21 2002
> 
> (Note that there's a space, no colon after "From".)

(275) netchild@ttyp0 % for i in $(grep ' 0 (' count.txt | cut -f 1 -d ' '); do
head -1 $i          
done
>From owner-freebsd-advocacy  Fri Apr 13  7:29: 6 2001
>From owner-freebsd-advocacy  Wed Aug 29 15:42: 0 2001
>From owner-freebsd-advocacy  Thu Nov 15 13: 6:40 2001
>From owner-freebsd-advocacy  Fri Nov 23  7:34: 9 2001
>From owner-freebsd-advocacy  Sun Dec 30 18:25: 4 2001
[...]

BTW.: splitndirs.py with -n 1000 is a lot slower than with -n 100, but I
don't see the reason. I already added an '-r'-Switch (round robin
instead of random distribution), but this doesn't help. Any ideas?

Bye,
Alexander.

-- 
           Intel: where Quality is job number 0.9998782345!

http://www.Leidinger.net                       Alexander @ Leidinger.net
  GPG fingerprint = C518 BC70 E67F 143F BE91  3365 79E2 9C60 B006 3FE7

From Alexander@Leidinger.net  Tue Sep 24 20:06:43 2002
From: Alexander@Leidinger.net (Alexander Leidinger)
Date: Tue, 24 Sep 2002 21:06:43 +0200
Subject: [Spambayes] Patch: Add globbing to splitndirs.py
In-Reply-To: <200209241850.g8OIo0d28489@odiug.zope.com>
References: <20020924194739.5461ca01.Alexander@Leidinger.net>
	<200209241830.g8OIUDT28426@odiug.zope.com>
	<20020924203915.66811368.Alexander@Leidinger.net>
	<200209241850.g8OIo0d28489@odiug.zope.com>
Message-ID: <20020924210643.61467ecd.Alexander@Leidinger.net>

This is a multi-part message in MIME format.

---------------------- multipart/mixed attachment
On Tue, 24 Sep 2002 14:50:00 -0400 Guido van Rossum <guido@python.org>
wrote:

> Are you perhaps using individual files (like those created by MH or
> GNUS) that aren't in mbox format???

Actually it's just one message in some of those mails, but it looks like
mbox format for me. Those messages are from the same source as every
other message wich gets counted correctly. I attach one of those message
with more than one mail and which didn't gets counted here.

Bye,
Alexander.

-- 
                   It's not a bug, it's tradition!

http://www.Leidinger.net                       Alexander @ Leidinger.net
  GPG fingerprint = C518 BC70 E67F 143F BE91  3365 79E2 9C60 B006 3FE7

---------------------- multipart/mixed attachment
A non-text attachment was scrubbed...
Name: 20010819.freebsd-afs.txt
Type: application/octet-stream
Size: 6712 bytes
Desc: not available
Url : http://mail.python.org/pipermail-21/spambayes/attachments/20020924/9b3825eb/20010819.freebsd-afs.exe

---------------------- multipart/mixed attachment--

From gward@python.net  Tue Sep 24 21:08:38 2002
From: gward@python.net (Greg Ward)
Date: Tue, 24 Sep 2002 16:08:38 -0400
Subject: [Spambayes] Patch: Add globbing to splitndirs.py
In-Reply-To: <20020924210643.61467ecd.Alexander@Leidinger.net>
References: <20020924194739.5461ca01.Alexander@Leidinger.net>
	<200209241830.g8OIUDT28426@odiug.zope.com>
	<20020924203915.66811368.Alexander@Leidinger.net>
	<200209241850.g8OIo0d28489@odiug.zope.com>
	<20020924210643.61467ecd.Alexander@Leidinger.net>
Message-ID: <20020924200838.GA5764@cthulhu.gerg.ca>

On 24 September 2002, Alexander Leidinger said:
> Actually it's just one message in some of those mails, but it looks like
> mbox format for me. Those messages are from the same source as every
> other message wich gets counted correctly. I attach one of those message
> with more than one mail and which didn't gets counted here.

The first line of the file you attached started with the string
">From ".  If that's really what the file on disk looks like, that's
wrong -- it should start with "From ".

See http://www.qmail.org/man/man5/mbox.html for a description of the
mbox format, such as it is.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Do radioactive cats have 18 half-lives?

From tim.one@comcast.net  Tue Sep 24 21:38:11 2002
From: tim.one@comcast.net (Tim Peters)
Date: Tue, 24 Sep 2002 16:38:11 -0400
Subject: [Spambayes] RE: Central Limit Theorem??!!     :)
In-Reply-To: <B9B5C548.166B0%grobinson@transpose.com>
Message-ID: <BIEJKCLHCIOIHAGOKOLHOEPBDKAA.tim.one@comcast.net>

[Gary Robinson]
> Summarizing all the test results, I get
>
>          fn  fp
> f(w):    11  6
> Graham:  16  10
> lcm:     29  3
>
> where f(w) is pure f(w) with a=0.1, 1500 discrimators, and
> robinson_minimum_prob_strength=0.1, Graham is tweaked Graham, and lcm is
> log-central-mean ignoring the middling values.

Good summary.  Note that of the first (f(w)) result, I said:

>> Alas, it turns out it would have been better to set spam_cutoff to 0.6
>> on this run.  That would have cut 9 instances of fp and added 33
>> instances of fn (note that since each msg is scored 4 times, it's *not*
>> necessarily the vase that these deltas would reflect directly in the
>> "total unique" counts), leaving the two roughly comparable on this test.

I don't know what the "total unique" counts would have been then.  The
biggest practical problem we've got with the f(w) scheme is that each person
who tests it seems to need a slightly different value of spam_cutoff to work
best (all reported "best values" have been in [0.5, 0.6] to date, and a
change of 0.01 is sometimes enough to change a "looks like it lost" outcome
to a "looks like it won" outcome).

> So doesn't the f(w) technique beat Graham all-around? At least in this
> latest round of testing??? Or am I misreading the  results?

Leaving spam_cutoff at 0.575 in the f(w) run did beat both Graham numbers on
this, yes.  I wanted to boost it to 0.6 to combat the f-p rate regression
f(w) showed compared to the lcm scheme.  The absolute difference is 4 (10-6)
or 5 (16-11) messages out of 20000 predictions, though, and so I call them
approximately equal -- the delta is small compared to total predictions
made.

> I wonder if the errors for lcm are mostly in the region where there are a
> small number of data points, such that the central mean theorem
> isn't really kicking in.
>
> That is, it may be that there is some number n for which f(w)
> gives the best results if the number of non-middling words is < n and
> lcm gives the best results if the number of non-middling words is > n.
>
> That WOULD make a lot of theoretical sense, because for small
> enough n, the central mean theorem is meaningless and can only make
> trouble for us.
>
> Something else for you to test in your copious free time. ;)

I have complete listings of all misclassifications for all tests reported,
but not available to me here and now.  I'll check it out later.  The
historical problems I had with the f-n rate under the Graham scheme involved
very long and very short msgs.  One effect of ignoring middling words is
actually to decrease n on short msgs, and sometimes to decrease n a lot.
Whatever damage is done to the central limit theorem then seems more than
compensated for by not sucking in words that try to drag the z-values 15
stddevs away from both the ham and spam means.  Only words with conviction
should vote <wink>.


From tim.one@comcast.net  Wed Sep 25 01:26:09 2002
From: tim.one@comcast.net (Tim Peters)
Date: Tue, 24 Sep 2002 20:26:09 -0400
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEFJBGAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCAEICBGAB.tim.one@comcast.net>

I'm detecting subtle <snort> signs that people mistake the chaos on this
list for chaos in the project.  It is getting out of control, but it's not
there quite yet.

The idea here was always to take competing core ideas, test them against
each other, and kill the loser.  Unfortunately, I had more ideas than I
could possibly make time to test, and Gary generates more ideas than I can
possibly make time to test too.  As a result, we've got 4 core scoring
schemes now, and more options to control them than even I can keep track of.

So, it's time to kill one!  I need your help.  I have no interest in
absolute error rates here, just in which of two specific schemes does better
on your data.  If you have error rates of 30%, and believe your ham and spam
are clean, fine, your input is important (not mentioning Skip by name ...);
indeed, it's *more* important than ever-finer hair-splitting from people
with error rates under 1%.

To avoid distractions, there's only one kind of test run I'll look at for
this:  a 10-fold cross-validation run with exactly 200 ham and 200 spam in
each set.  If you don't have at least 2,000 ham and 2,000 spam, you can't
run this test, and reporting results won't help.  If you have more than 200
in each set, that's fine, run timcv like so

    timcv -n10 --ham=200 --spam=200 -s12345

"12345" is a magic number, and you can pick any you like, but you have to
use the same int each time.  It's folded into a randomization scheme that
picks a reproducible set of 200 messages "at random" from each of your Set
directories.  For those running mboxtest.py, I believe it's capable of doing
something similar already; Jeremy can say for sure.

Don't add new messages to your Sets while running these tests.  Indeed,
don't change anything apart from what's detailed below.  If you find ham in
your spam, or vice versa, do clean it out, and then start over again.

The two combatants in this test are G0 and G1:  our last stab at Graham's
scheme, and Gary's f(w) scheme.  The losing code will be purged from the
codebase and the options controlling it will disappear.  In case of a tie,
Gary's scheme wins, because the code is cleaner and it's much easier to
reason about what it's doing.


How To Run the Graham Scheme
============================
This may be a surprise to you <wink>:  Graham's scheme is still there, and
hasn't changed at all lately.  The way to get it is simply to leave your

[Classifier]

section entirely empty, and force

[TestDriver]
spam_cutoff: 0.90

That's also the default, so if you don't use a customization file at all,
you get the Graham scheme.

What else you put in the [Tokenizer] and [TestDriver] sections doesn't
matter for the purposes of this test -- just pick something, and then leave
it alone.  Whatever evidence you're generating, we just want to see which
scheme does better with it.


How To Run The f(w) Scheme
==========================
This is harder, because we've spent almost no time systematically tuning the
parameters that matter to this scheme.  Part of this test is for you to
figure out and report back on which parameter values work best on your data
in this test.  Start like so:

"""
[Classifier]
use_robinson_combining: True
use_robinson_probability: True
robinson_probability_x: 0.5
robinson_probability_a: 0.5
max_discriminators: 150
robinson_minimum_prob_strength: 0.1

[TestDriver]
spam_cutoff: 0.550
"""

You'll never change use_robinson_combining or use_robinson_probability in
this part of the test.

robinson_probability_x is related to Graham's UNKNOWN_WORD_SPAMPROB, and
I've seen no evidence to suggest that moving it away from 1/2 is going to do
anyone any good.  So if you never get around to fiddling this one, that's
fine by me.  If you can make time to fiddle it, great.

robinson_probability_a is important, and no good tests have been reported on
it.  Potentially interesting values range from 0.0 to infinity.  Try to find
the optimal value for your data (I suspect it will be on the low end of [0,
1], but don't know; but exactly 0.0 may well be a disaster).  The lower it
is, the more extreme the spamprobs given to words that appear in few of your
training msgs.

max_discriminators controls how many "extreme words" the scheme scores.  I
saw no real difference between 150 and 1500 in one brief test, but then I
saw no real difference between 150 and 15 in another brief test.  We need
more results on this.  Again try to find an optimal value for your data.

When robinson_minimum_prob_strength is greater than 0, "bland" words are
ignored.  Calling the value p, all words with spamprob in 0.5-p to 0.5+p
(exclusive of the endpoints) are ignored.  At 0.0, no words are ignored;  at
0.5, all words are ignored.  I conjecture that the best value is below 0.5
<wink>.  The suggested starting point of 0.1 ignores only the really bland
words, and has been a winner according to all people reporting on it so far.
Try to find the *best* value for you.

Finally, and most irritatingly, spam_cutoff is crucial in this scheme --
changing its value by 0.02 can make a run that looks like a loser look like
a winner, or vice versa.  You can stare at your score histograms to figure
out whether boosting this or dropping it would help a run do better.  We
simply don't know anything useful about this setting yet.

Caution #1:  These parameters almost certainly interact with each other.
Tuning all at one time is difficult.  (The Graham scheme has more
interacting parameters, btw; you weren't nearly as aware of that because I
stayed up for a week <0.7 wink> tuning them while you slept -- I can't do
that anymore.)

Caution #2:  If your error rates are already low, it's quite possible that
this test isn't big enough to reveal anything.  But that's interesting too,
in its own charmingly useless way.


Extra Credit
============
After you're done, do it all over again with a different magic int.  Do the
relative results change?  If so, do you need significantly different
parameter settings for best results?


Extra Extra Credit
==================
Repeat the above until you drop from boredom.


FAQ
===
Q. This is a ham/spam ratio of 1.  Is that realistic?
A. We can't test everything at once.

Q. 1800 each of ham & spam is a very small training set.
   Wouldn't it be better to use more training data?
A. We can't test everything at once.

Q. 1800 each of ham & spam is a very large training set.
   Wouldn't it be better to use less training data?
A. We can't test everything at once.

Q. I thought Gary's central-limit ideas were more interesting.
   Why don't we test them instead?
A. We can't test everything at once -- and the little we know
   about tuning f(w) is already much more than we know about
   tuning the c-l schemes.

Q. It occurs to me that our tokenization scheme may get benefit
   from leaving punctuation attached to words *mostly* because
   Graham's scheme only looks at a handful of words, and then,
   e.g., counting "Python" and "Python." and "Python," etc as
   distinct words really helps.  But the f(w) scheme can look
   at any number of words, so maybe a different tokenization
   scheme would do better there.
A. You apparently figured out what I was really thinking about
   while tedioulsy repeating "We can't test everything at once."
   Congratulations!  It's irrelevant to this test, though --
   bring it up later.

Q. Each of the 10 cross-validation runs predicts against only
   200 ham and 200 spam.  That's not very much!  In fact, the
   smallest possible change in an error rate will be 0.5% (1
   message).  That isn't a very sensitive test.  In addition,
A. Let me know when you're ready to stop whining and run the
   tests <wink>.

shake-hands-and-come-out-fighting-ly y'rs  - tim


From gward@python.net  Wed Sep 25 02:44:12 2002
From: gward@python.net (Greg Ward)
Date: Tue, 24 Sep 2002 21:44:12 -0400
Subject: [Spambayes] Problems with unheader.py
Message-ID: <20020925014412.GA8349@cthulhu.gerg.ca>

I've been playing around with unheader.py -- it looks like it's just the
ticket for cleaning up some of the spam I've gathered.  Apart from
adding Maildir support to it, I think I've found some bugs:

  1) if it tries to read from stdin, it crashes with:
     [...]
       File "/scratch/src/spambayes/unheader.py", line 68, in process_mailbox
         for msg in mailbox.PortableUnixMailbox(f, Parser().parse):
       File "/www/plat/python2.2.1/lib/python2.2/mailbox.py", line 23, in next
         self.fp.seek(self.seekp)
     IOError: [Errno 29] Illegal seek

  2) deSA() removes the first and last line of the body

  3) deSA() crashes on MIME messages -- assumes the result of
     get_payload() is a string

#1 is easy to fix -- just remove the ability to read from stdin.  Anyone
care?  It'll make my patch to add Maildir simpler.

#2 is *probably* an easy/silly bug, but I haven't looked into it yet.

#3 makes it look like no one has used this code (the attempt to remove
SA's "SPAM: " lines in particular) on a real spam corpus.  Really?

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Just because you're paranoid doesn't mean they *aren't* out to get you.

From anthony@interlink.com.au  Wed Sep 25 02:48:24 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Wed, 25 Sep 2002 11:48:24 +1000
Subject: [Spambayes] Problems with unheader.py 
In-Reply-To: <20020925014412.GA8349@cthulhu.gerg.ca> 
Message-ID: <200209250148.g8P1mPf04457@localhost.localdomain>


>>> Greg Ward wrote
> I've been playing around with unheader.py -- it looks like it's just the
> ticket for cleaning up some of the spam I've gathered.  Apart from
> adding Maildir support to it, I think I've found some bugs:

Quite probably. It's a quick hack :)


>   2) deSA() removes the first and last line of the body
> 
>   3) deSA() crashes on MIME messages -- assumes the result of
>      get_payload() is a string
> 
> #1 is easy to fix -- just remove the ability to read from stdin.  Anyone
> care?  It'll make my patch to add Maildir simpler.

I'd say kill it.

> #2 is *probably* an easy/silly bug, but I haven't looked into it yet.

Really? Hadn't seen that one.

> #3 makes it look like no one has used this code (the attempt to remove
> SA's "SPAM: " lines in particular) on a real spam corpus.  Really?

but SA turns the message into a text/plain, anyway... well, it does the
way I run it here... 

Anthony

-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.


From gward@python.net  Wed Sep 25 02:50:02 2002
From: gward@python.net (Greg Ward)
Date: Tue, 24 Sep 2002 21:50:02 -0400
Subject: [Spambayes] Problems with unheader.py
In-Reply-To: <20020925014412.GA8349@cthulhu.gerg.ca>
References: <20020925014412.GA8349@cthulhu.gerg.ca>
Message-ID: <20020925015002.GA6550@cthulhu.gerg.ca>

On 24 September 2002, I said:
> #1 is easy to fix -- just remove the ability to read from stdin.  Anyone
> care?  It'll make my patch to add Maildir simpler.

Here's my patch to add Maildir support to unheader.py, and remove
read-from-stdin support at the same time.  I'll probably check it in
once I'm convinced it works, unless someone objects.

----------------------------------------------------------------------
--- unheader.py	24 Sep 2002 17:59:58 -0000	1.4
+++ unheader.py	25 Sep 2002 01:47:53 -0000
@@ -2,9 +2,12 @@
 
 import re
 import sys
+import os
+import glob
 import mailbox
 import email.Parser
 import email.Message
+import email.Generator
 import getopt
 
 def unheader(msg, pat):
@@ -55,14 +58,32 @@
             msg.set_payload("\n".join(newbody))
     unheader(msg, "X-Spam-")
 
+def process_message(msg, dosa, pats):
+    if pats is not None:
+        unheader(msg, pats)
+    if dosa:
+        deSA(msg)
+
 def process_mailbox(f, dosa=1, pats=None):
     for msg in mailbox.PortableUnixMailbox(f, Parser().parse):
-        if pats is not None:
-            unheader(msg, pats)
-        if dosa:
-            deSA(msg)
+        process_message(msg, dosa, pats)
         print msg
 
+def process_maildir(d, dosa=1, pats=None):
+    parser = Parser()
+    for fn in glob.glob(os.path.join(d, "cur", "*")):
+        print ("reading from %s..." % fn),
+        file = open(fn)
+        msg = parser.parse(file)
+        process_message(msg, dosa, pats)
+
+        tmpfn = os.path.join(d, "tmp", os.path.basename(fn))
+        tmpfile = open(tmpfn, "w")
+        print "writing to %s" % tmpfn
+        email.Generator.Generator(tmpfile, maxheaderlen=0)(msg, unixfrom=0)
+
+        #os.rename(tmpfn, fn)
+
 def usage():
     print >> sys.stderr, "usage: unheader.py [ -p pat ... ] [ -s ]"
     print >> sys.stderr, "-p pat gives a regex pattern used to eliminate unwanted headers"
@@ -72,8 +93,9 @@
 def main(args):
     headerpats = []
     dosa = 1
+    ismbox = 1
     try:
-        opts, args = getopt.getopt(args, "p:sh")
+        opts, args = getopt.getopt(args, "p:shd")
     except getopt.GetoptError:
         usage()
         sys.exit(1)
@@ -86,15 +108,19 @@
                 headerpats.append(arg)
             elif opt == "-s":
                 dosa = 0
+            elif opt == "-d":
+                ismbox = 0
         pats = headerpats and "|".join(headerpats) or None
-        if not args:
-            f = sys.stdin
-        elif len(args) == 1:
-            f = file(args[0])
-        else:
+
+        if len(args) != 1:
             usage()
             sys.exit(1)
-        process_mailbox(f, dosa, pats)
+
+        if ismbox:
+            f = file(args[0])
+            process_mailbox(f, dosa, pats)
+        else:
+            process_maildir(args[0], dosa, pats)
 
 if __name__ == "__main__":
     main(sys.argv[1:])

------------------------------------------------------------------------

-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Support bacteria -- it's the only culture some people have!

From gward@python.net  Wed Sep 25 02:51:25 2002
From: gward@python.net (Greg Ward)
Date: Tue, 24 Sep 2002 21:51:25 -0400
Subject: [Spambayes] Problems with unheader.py
In-Reply-To: <200209250148.g8P1mPf04457@localhost.localdomain>
References: <20020925014412.GA8349@cthulhu.gerg.ca>
	<200209250148.g8P1mPf04457@localhost.localdomain>
Message-ID: <20020925015125.GB6550@cthulhu.gerg.ca>

On 25 September 2002, Anthony Baxter said:
> > #3 makes it look like no one has used this code (the attempt to remove
> > SA's "SPAM: " lines in particular) on a real spam corpus.  Really?
> 
> but SA turns the message into a text/plain, anyway... well, it does the
> way I run it here... 

That's an SA option -- which I happen not to use.  No wonder no one else
has seen this bug.

OK: anyone know how to get the raw message body from an email.Message
object?  Or am I gonna have to RTFM?

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
I haven't lost my mind; I know exactly where I left it.

From matt@mondoinfo.com  Wed Sep 25 03:02:11 2002
From: matt@mondoinfo.com (Matthew Dixon Cowles)
Date: Tue, 24 Sep 2002 21:02:11 -0500 (CDT)
Subject: [Spambayes] Problems with unheader.py
In-Reply-To: <20020925015125.GB6550@cthulhu.gerg.ca>
References: <20020925014412.GA8349@cthulhu.gerg.ca>
	<200209250148.g8P1mPf04457@localhost.localdomain>
	<20020925015125.GB6550@cthulhu.gerg.ca>
Message-ID: <1032919002.81.1352@sake.mondoinfo.com>

Dear Greg,

> OK: anyone know how to get the raw message body from an
> email.Message object?

If the message is text/plain, you can use the Message's get_payload()
method. If the message is multipart, there isn't an easy way unless
the message has been parsed with the HeaderParser class. In that case
the body isn't parsed and get_payload() will return the raw body text.

Regards,
Matt


From guido@python.org  Wed Sep 25 03:06:27 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 24 Sep 2002 22:06:27 -0400
Subject: [Spambayes] There Can Be Only One
In-Reply-To: Your message of "Tue, 24 Sep 2002 20:26:09 EDT."
             <LNBBLJKPBEHFEDALKOLCAEICBGAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCAEICBGAB.tim.one@comcast.net> 
Message-ID: <200209250206.g8P26RV26472@pcp02138704pcs.reston01.va.comcast.net>

Here's a first result.  I compared baseline Graham (empty .ini) with
Gary according to your suggestion, but with spam_cutoff set to 0.625
(after looking at the histograms of a trial run).

Net result:

Graham: 8 fp's, 13 fn's.
Gary:   7 fp's, 13 fn's.

I ran the same two tests with a different random number:

Graham: 8 fp's, 11 fn's.
Gary:   6 fp's, 16 fn's.

According to the histogram, Gary would have given 8 fp's and 10 fn's
with a cutoff of 0.6, again beating Graham with the smallest margin.

I found one more spam in my ham (but didn't remove it between these
runs).  I also found 10 empty messages in Bruce Guenter's spam
archives!  (1 in 2002/01, 2 in 2002/05, 7 in 2002/06.)  Tim, I presume
you cleaned these out long ago?

I'm going to clean these out, start over, and try to tweak Gary's
parameters.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.one@comcast.net  Wed Sep 25 03:09:33 2002
From: tim.one@comcast.net (Tim Peters)
Date: Tue, 24 Sep 2002 22:09:33 -0400
Subject: [Spambayes] Problems with unheader.py
In-Reply-To: <20020925015125.GB6550@cthulhu.gerg.ca>
Message-ID: <LNBBLJKPBEHFEDALKOLCOEIJBGAB.tim.one@comcast.net>

[Greg Ward]
> ...
> OK: anyone know how to get the raw message body from an email.Message
> object?  Or am I gonna have to RTFM?

Barry knows, but he usually won't tell you anything unless you buy him lunch
first.

str(msg) returns the whole msg as a string, including the headers.  I'm not
aware of a way to get just the body (unless you luck out and it's not
multipart, in which case get_payload() does the trick).


From gward@python.net  Wed Sep 25 03:14:11 2002
From: gward@python.net (Greg Ward)
Date: Tue, 24 Sep 2002 22:14:11 -0400
Subject: [Spambayes] Problems with unheader.py
In-Reply-To: <1032919002.81.1352@sake.mondoinfo.com>
References: <20020925014412.GA8349@cthulhu.gerg.ca>
	<200209250148.g8P1mPf04457@localhost.localdomain>
	<20020925015125.GB6550@cthulhu.gerg.ca>
	<1032919002.81.1352@sake.mondoinfo.com>
Message-ID: <20020925021411.GC6550@cthulhu.gerg.ca>

On 24 September 2002, Matthew Dixon Cowles said:
> If the message is text/plain, you can use the Message's get_payload()
> method. If the message is multipart, there isn't an easy way unless
> the message has been parsed with the HeaderParser class. In that case
> the body isn't parsed and get_payload() will return the raw body text.

Thanks, that was exactly it -- all I had to do was change Parser to
subclass HeaderParser instead of the original Parser class.  Neat!

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Ward's Rule of Gender Identification:
If I can't tell at a glance, I don't want to know.

From tim.one@comcast.net  Wed Sep 25 03:14:31 2002
From: tim.one@comcast.net (Tim Peters)
Date: Tue, 24 Sep 2002 22:14:31 -0400
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <LNBBLJKPBEHFEDALKOLCAEICBGAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCAEILBGAB.tim.one@comcast.net>

Here are my results on the test setup exactly as suggested, Graham on the
left and f(w) on the right:

false positive percentages
    0.500  0.500  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied

won   0 times
tied 10 times
lost  0 times

total unique fp went from 1 to 1 tied
mean fp % went from 0.05 to 0.05 tied

false negative percentages
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.500  0.000  won   -100.00%
    0.000  0.000  tied

won   1 times
tied  9 times
lost  0 times

total unique fn went from 1 to 0 won   -100.00%
mean fn % went from 0.05 to 0.0 won   -100.00%


From tim.one@comcast.net  Wed Sep 25 03:21:44 2002
From: tim.one@comcast.net (Tim Peters)
Date: Tue, 24 Sep 2002 22:21:44 -0400
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <200209250206.g8P26RV26472@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEINBGAB.tim.one@comcast.net>

[Guido]
> Here's a first result.  I compared baseline Graham (empty .ini) with
> Gary according to your suggestion, but with spam_cutoff set to 0.625

Wow!  0.625 is the largest reported "best value" to date (Neil and I both
needed 0.6 once; the suggested 0.55 was optimal for my run -- which is why I
suggested it <wink>).

> (after looking at the histograms of a trial run).
>
> Net result:
>
> Graham: 8 fp's, 13 fn's.
> Gary:   7 fp's, 13 fn's.
>
> I ran the same two tests with a different random number:
>
> Graham: 8 fp's, 11 fn's.
> Gary:   6 fp's, 16 fn's.
>
> According to the histogram, Gary would have given 8 fp's and 10 fn's
> with a cutoff of 0.6, again beating Graham with the smallest margin.

Have you used cmp.py?  It generates side-by-side listings comparing two
runs, and it's extremely important to know how often each scheme beat the
other; the summary numbers reveal nothing about that.  For this particular
test, the tail end of the cmp.py output (with the changes in ham and spam
score means and sdevs) is useless.  When tweaking paramters for a single
scheme, though, all the cmp.py output is dripping with valuable clues.

> I found one more spam in my ham (but didn't remove it between these
> runs).  I also found 10 empty messages in Bruce Guenter's spam
> archives!  (1 in 2002/01, 2 in 2002/05, 7 in 2002/06.)  Tim, I presume
> you cleaned these out long ago?

No, I've left empty messages in both my corpora.  Although I'm unclear on
what you mean by "empty".  I mean they have no body, but do have message
headers.  Sometimes a c.l.py ham consists solely of a question in the
Subject line!

> I'm going to clean these out, start over, and try to tweak Gary's
> parameters.

Bless you.


From guido@python.org  Wed Sep 25 03:50:44 2002
From: guido@python.org (Guido van Rossum)
Date: Tue, 24 Sep 2002 22:50:44 -0400
Subject: [Spambayes] There Can Be Only One
In-Reply-To: Your message of "Tue, 24 Sep 2002 22:21:44 EDT."
             <LNBBLJKPBEHFEDALKOLCCEINBGAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCCEINBGAB.tim.one@comcast.net> 
Message-ID: <200209250250.g8P2oiU26801@pcp02138704pcs.reston01.va.comcast.net>

> Wow!  0.625 is the largest reported "best value" to date (Neil and I
> both needed 0.6 once; the suggested 0.55 was optimal for my run --
> which is why I suggested it <wink>).

0.6 is the *lowest* that works for me -- any lower and Graham
wins on both fp's and fn's.

> > (after looking at the histograms of a trial run).
> >
> > Net result:
> >
> > Graham: 8 fp's, 13 fn's.
> > Gary:   7 fp's, 13 fn's.
> >
> > I ran the same two tests with a different random number:
> >
> > Graham: 8 fp's, 11 fn's.
> > Gary:   6 fp's, 16 fn's.
> >
> > According to the histogram, Gary would have given 8 fp's and 10 fn's
> > with a cutoff of 0.6, again beating Graham with the smallest margin.
> 
> Have you used cmp.py?  It generates side-by-side listings comparing
> two runs, and it's extremely important to know how often each scheme
> beat the other; the summary numbers reveal nothing about that.  For
> this particular test, the tail end of the cmp.py output (with the
> changes in ham and spam score means and sdevs) is useless.  When
> tweaking paramters for a single scheme, though, all the cmp.py
> output is dripping with valuable clues.

Ok.  For the first pair:

false positive percentages
    0.500  1.000  lost  +100.00%
    0.000  0.500  lost  +(was 0)
    0.500  0.500  tied          
    1.000  0.500  won    -50.00%
    1.000  0.500  won    -50.00%
    0.000  0.000  tied          
    0.500  0.000  won   -100.00%
    0.000  0.000  tied          
    0.500  0.500  tied          
    0.000  0.000  tied          

won   3 times
tied  5 times
lost  2 times

total unique fp went from 8 to 7 won    -12.50%
mean fp % went from 0.4 to 0.35 won    -12.50%

false negative percentages
    1.000  0.000  won   -100.00%
    1.500  1.500  tied          
    0.500  0.500  tied          
    0.500  0.000  won   -100.00%
    0.500  0.000  won   -100.00%
    0.500  0.500  tied          
    1.000  1.000  tied          
    0.500  1.000  lost  +100.00%
    0.500  2.000  lost  +300.00%
    0.000  0.000  tied          

won   3 times
tied  5 times
lost  2 times

total unique fn went from 13 to 13 tied          
mean fn % went from 0.65 to 0.65 tied          

For the second pair:

false positive percentages
    0.500  0.500  tied          
    0.000  0.000  tied          
    0.000  0.000  tied          
    0.500  0.000  won   -100.00%
    0.500  0.500  tied          
    0.500  0.500  tied          
    0.000  0.000  tied          
    0.500  0.500  tied          
    0.000  0.000  tied          
    1.500  1.000  won    -33.33%

won   2 times
tied  8 times
lost  0 times

total unique fp went from 8 to 6 won    -25.00%
mean fp % went from 0.4 to 0.3 won    -25.00%

false negative percentages
    0.500  0.500  tied          
    1.000  0.500  won    -50.00%
    0.000  0.500  lost  +(was 0)
    1.000  1.000  tied          
    0.000  1.000  lost  +(was 0)
    1.000  1.000  tied          
    0.500  1.000  lost  +100.00%
    0.000  0.000  tied          
    0.500  0.500  tied          
    1.000  2.000  lost  +100.00%

won   1 times
tied  5 times
lost  4 times

total unique fn went from 11 to 16 lost   +45.45%
mean fn % went from 0.55 to 0.8 lost   +45.45%

> > I found one more spam in my ham (but didn't remove it between these
> > runs).  I also found 10 empty messages in Bruce Guenter's spam
> > archives!  (1 in 2002/01, 2 in 2002/05, 7 in 2002/06.)  Tim, I presume
> > you cleaned these out long ago?
> 
> No, I've left empty messages in both my corpora.  Although I'm unclear on
> what you mean by "empty".  I mean they have no body, but do have message
> headers.  Sometimes a c.l.py ham consists solely of a question in the
> Subject line!

No, I have 10 files that have length 0 (i.e. no headers and no body)
in BruceG's original bz2 files.  I checked, and the tar listing has
these too.

Here are the results for my first Gary variation: changing
max_discriminators to 1500, while keeping the cutoff at 0.60:

false positive percentages
    0.500  0.500  tied          
    0.000  0.000  tied          
    0.000  0.000  tied          
    0.000  0.000  tied          
    0.500  0.500  tied          
    0.500  0.500  tied          
    0.000  0.500  lost  +(was 0)
    0.500  0.500  tied          
    0.000  0.000  tied          
    1.000  1.500  lost   +50.00%

won   0 times
tied  8 times
lost  2 times

total unique fp went from 6 to 8 lost   +33.33%
mean fp % went from 0.3 to 0.4 lost   +33.33%

false negative percentages
    0.500  0.000  won   -100.00%
    0.500  0.500  tied          
    0.500  0.500  tied          
    1.000  0.500  won    -50.00%
    1.000  0.000  won   -100.00%
    1.000  1.000  tied          
    1.000  0.500  won    -50.00%
    0.000  0.000  tied          
    0.500  0.500  tied          
    2.000  1.500  won    -25.00%

won   5 times
tied  5 times
lost  0 times

total unique fn went from 16 to 10 won    -37.50%
mean fn % went from 0.8 to 0.5 won    -37.50%

The 60.0 bins in the histogram have 2 hams and 7 spams, so moving the
cutoff to 0.625 would have made it a tie for fps and a loss by 1 for
fns.

Ah, and here are the results for md=15 (left == md=150, right == md=15):

false positive percentages
    0.500  0.000  won   -100.00%
    0.000  0.000  tied          
    0.000  0.500  lost  +(was 0)
    0.000  1.000  lost  +(was 0)
    0.500  1.000  lost  +100.00%
    0.500  0.500  tied          
    0.000  0.500  lost  +(was 0)
    0.500  1.500  lost  +200.00%
    0.000  0.000  tied          
    1.000  1.000  tied          

won   1 times
tied  4 times
lost  5 times

total unique fp went from 6 to 12 lost  +100.00%
mean fp % went from 0.3 to 0.6 lost  +100.00%

false negative percentages
    0.500  0.000  won   -100.00%
    0.500  0.500  tied          
    0.500  0.500  tied          
    1.000  0.500  won    -50.00%
    1.000  0.000  won   -100.00%
    1.000  1.000  tied          
    1.000  0.500  won    -50.00%
    0.000  0.000  tied          
    0.500  0.500  tied          
    2.000  1.500  won    -25.00%

won   5 times
tied  5 times
lost  0 times

total unique fn went from 16 to 10 won    -37.50%
mean fn % went from 0.8 to 0.5 won    -37.50%

The histograms look totally different here though!

-> <stat> Ham scores for all runs: 2000 items; mean 11.01; sample sdev 15.30
* = 21 items
  0.00 1201 **********************************************************
  2.50   53 ***
  5.00    9 *
  7.50    7 *
 10.00    2 *
 12.50   29 **
 15.00   87 *****
 17.50  100 *****
 20.00   67 ****
 22.50   48 ***
 25.00   54 ***
 27.50   54 ***
 30.00   41 **
 32.50   49 ***
 35.00   46 ***
 37.50   29 **
 40.00   23 **
 42.50   30 **
 45.00   19 *
 47.50   16 *
 50.00    9 *
 52.50    7 *
 55.00    2 *
 57.50    6 *
 60.00    5 *
 62.50    1 *
 65.00    1 *
 67.50    1 *
 70.00    0 
 72.50    0 
 75.00    0 
 77.50    2 *
 80.00    0 
 82.50    0 
 85.00    1 *
 87.50    0 
 90.00    0 
 92.50    0 
 95.00    1 *
 97.50    0 

-> <stat> Spam scores for all runs: 2000 items; mean 95.00; sample sdev 7.86
* = 21 items
[...]
 47.50    1 *
 50.00    1 *
 52.50    1 *
 55.00    3 *
 57.50    4 *
 60.00    8 *
 62.50    9 *
 65.00   11 *
 67.50   15 *
 70.00   13 *
 72.50   15 *
 75.00   23 **
 77.50   51 ***
 80.00   65 ****
 82.50   41 **
 85.00    2 *
 87.50    9 *
 90.00   12 *
 92.50   66 ****
 95.00  425 *********************
 97.50 1225 ***********************************************************

Note the hams scoring all the way in the 90s.  There are no spams
here!  I think the highest scoring ham was Kim's message with diet
tips (also implicated in the Graham run) -- typical diet terms don't
occur much in my ham, but I guess spams for magic diets are common.
Kim sent this with almost no happy-talk; there were only 3 low-scoring
words amongst the 15.

Given this, I'm going to keep this parameter at 150 and vary other
things.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From nas@python.ca  Wed Sep 25 04:09:28 2002
From: nas@python.ca (Neil Schemenauer)
Date: Tue, 24 Sep 2002 20:09:28 -0700
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <LNBBLJKPBEHFEDALKOLCAEILBGAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCAEICBGAB.tim.one@comcast.net>
	<LNBBLJKPBEHFEDALKOLCAEILBGAB.tim.one@comcast.net>
Message-ID: <20020925030928.GA12308@glacier.arctrix.com>

Tim Peters wrote:
> Here are my results on the test setup exactly as suggested, Graham on the
> left and f(w) on the right:

> total unique fp went from 1 to 1 tied
> mean fp % went from 0.05 to 0.05 tied

> total unique fn went from 1 to 0 won   -100.00%
> mean fn % went from 0.05 to 0.0 won   -100.00%

You're a victim of your own success. :-)  Could you try sabotaging the
classifier to make it work harder and produce higher error rates?  For
example, don't let it look at any header information.

  Neil

From nas@python.ca  Wed Sep 25 04:55:48 2002
From: nas@python.ca (Neil Schemenauer)
Date: Tue, 24 Sep 2002 20:55:48 -0700
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <LNBBLJKPBEHFEDALKOLCAEICBGAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCKEFJBGAB.tim.one@comcast.net>
	<LNBBLJKPBEHFEDALKOLCAEICBGAB.tim.one@comcast.net>
Message-ID: <20020925035548.GA12077@glacier.arctrix.com>

Here's my first result (Graham on left):

    false positive percentages
        0.000  0.500  lost  +(was 0)
        1.500  1.500  tied          
        0.000  0.000  tied          
        0.000  0.000  tied          
        0.500  0.500  tied          
        0.500  1.000  lost  +100.00%
        0.500  0.500  tied          
        0.500  0.000  won   -100.00%
        0.000  0.500  lost  +(was 0)
        0.000  0.500  lost  +(was 0)

    won   1 times
    tied  5 times
    lost  4 times

    total unique fp went from 7 to 10 lost   +42.86%
    mean fp % went from 0.35 to 0.5 lost   +42.86%

    false negative percentages
        0.500  0.000  won   -100.00%
        2.000  2.000  tied          
        1.000  0.500  won    -50.00%
        1.000  0.500  won    -50.00%
        2.000  1.500  won    -25.00%
        1.000  1.000  tied          
        0.500  0.500  tied          
        2.000  0.000  won   -100.00%
        1.500  0.500  won    -66.67%
        0.000  0.000  tied          

    won   6 times
    tied  4 times
    lost  0 times

    total unique fn went from 23 to 13 won    -43.48%
    mean fn % went from 1.15 to 0.65 won    -43.48%

Already Robinson seems to be doing better.  The only tweak I made was to
set spam_cutoff to 0.580.  Next, I tried setting robinson_probability_a
to 0.2 (compared to last Robinson result):

    total unique fp went from 10 to 6 won    -40.00%
    mean fp % went from 0.5 to 0.3 won    -40.00%

    total unique fn went from 13 to 14 lost    +7.69%
    mean fn % went from 0.65 to 0.7 lost    +7.69%

That's another win in overall error rate.  Setting
robinson_probability_a to 0.05 made things worse (again compared best
result):

    total unique fp went from 6 to 5 won    -16.67%
    mean fp % went from 0.3 to 0.25 won    -16.67%

    total unique fn went from 14 to 25 lost   +78.57%
    mean fn % went from 0.7 to 1.25 lost   +78.57%

I could probably do better by adjusting the cutoff but not better than
a = 0.2.  Now, a = 0.1:

    total unique fp went from 6 to 5 won    -16.67%
    mean fp % went from 0.3 to 0.25 won    -16.67%

    total unique fn went from 14 to 17 lost   +21.43%
    mean fn % went from 0.7 to 0.85 lost   +21.43%

Again, I could probably do a little better by lowering the cutoff but I
think a = 0.2 is still better.  Just for kicks, a = 0.8:

    total unique fp went from 6 to 11 lost   +83.33%
    mean fp % went from 0.3 to 0.55 lost   +83.33%

    total unique fn went from 14 to 18 lost   +28.57%
    mean fn % went from 0.7 to 0.9 lost   +28.57%

So 'a' between 0.1 and 0.2 seems best.  Sticking with a = 0.2 and
lowering robinson_minimum_prob_strength to 0.0 (lowering cutoff to 0.56
to hit the sweet spot of the distribution):

    total unique fp went from 6 to 12 lost  +100.00%
    mean fp % went from 0.3 to 0.6 lost  +100.00%

    total unique fn went from 14 to 16 lost   +14.29%
    mean fn % went from 0.7 to 0.8 lost   +14.29%

It seems robinson_minimum_prob_strength > 0 is useful.  Now, increase it
to 0.2:

    false positive percentages
        0.000  0.500  lost  +(was 0)
        1.000  2.000  lost  +100.00%
        0.000  0.000  tied          
        0.000  0.000  tied          
        0.500  0.500  tied          
        0.500  1.000  lost  +100.00%
        0.500  0.500  tied          
        0.000  0.500  lost  +(was 0)
        0.500  1.500  lost  +200.00%
        0.000  0.500  lost  +(was 0)

    won   0 times
    tied  4 times
    lost  6 times

    total unique fp went from 6 to 14 lost  +133.33%
    mean fp % went from 0.3 to 0.7 lost  +133.33%

    false negative percentages
        0.000  0.000  tied          
        1.000  1.000  tied          
        1.000  0.000  won   -100.00%
        0.500  0.500  tied          
        2.000  1.500  won    -25.00%
        1.500  1.000  won    -33.33%
        0.000  0.000  tied          
        0.500  0.000  won   -100.00%
        0.500  0.000  won   -100.00%
        0.000  0.000  tied          

    won   5 times
    tied  5 times
    lost  0 times

    total unique fn went from 14 to 8 won    -42.86%
    mean fn % went from 0.7 to 0.4 won    -42.86%

I could probably do slightly better by adjusting the cuttoff but I think
0.1 is still better.  I tried increasing max_discriminators to 1500
(with robinson_minimum_prob_strength back to 0.1 and cuttoff back to
0.58).  This is another close one:

    false positive percentages
        0.000  0.000  tied          
        1.000  1.000  tied          
        0.000  0.000  tied          
        0.000  0.000  tied          
        0.500  0.500  tied          
        0.500  0.500  tied          
        0.500  0.500  tied          
        0.000  0.000  tied          
        0.500  0.500  tied          
        0.000  0.000  tied          

    won   0 times
    tied 10 times
    lost  0 times

    total unique fp went from 6 to 6 tied          
    mean fp % went from 0.3 to 0.3 tied          

    false negative percentages
        0.000  0.000  tied          
        1.000  1.000  tied          
        1.000  1.000  tied          
        0.500  0.500  tied          
        2.000  2.000  tied          
        1.500  1.500  tied          
        0.000  0.500  lost  +(was 0)
        0.500  0.500  tied          
        0.500  0.500  tied          
        0.000  0.000  tied          

    won   0 times
    tied  9 times
    lost  1 times

    total unique fn went from 14 to 15 lost    +7.14%
    mean fn % went from 0.7 to 0.75 lost    +7.14%

I'd say it's a toss-up.  Now, max_discriminators = 70 (another close
one):

    false positive percentages
        0.000  0.000  tied          
        1.000  0.500  won    -50.00%
        0.000  0.000  tied          
        0.000  0.000  tied          
        0.500  0.500  tied          
        0.500  0.500  tied          
        0.500  0.500  tied          
        0.000  0.000  tied          
        0.500  0.500  tied          
        0.000  0.500  lost  +(was 0)

    won   1 times
    tied  8 times
    lost  1 times

    total unique fp went from 6 to 6 tied          
    mean fp % went from 0.3 to 0.3 tied          

    false negative percentages
        0.000  0.000  tied          
        1.000  1.000  tied          
        1.000  1.000  tied          
        0.500  0.500  tied          
        2.000  2.000  tied          
        1.500  1.000  won    -33.33%
        0.000  0.000  tied          
        0.500  0.500  tied          
        0.500  0.500  tied          
        0.000  0.000  tied          

    won   1 times
    tied  9 times
    lost  0 times

    total unique fn went from 14 to 13 won     -7.14%
    mean fn % went from 0.7 to 0.65 won     -7.14%

Hmmm, a little better perhaps?  Hard to say.  Let's try 50 and compare
it to 150:

    false positive percentages
        0.000  0.000  tied          
        1.000  1.000  tied          
        0.000  0.000  tied          
        0.000  0.000  tied          
        0.500  0.500  tied          
        0.500  0.500  tied          
        0.500  0.500  tied          
        0.000  0.500  lost  +(was 0)
        0.500  0.500  tied          
        0.000  0.000  tied          

    won   0 times
    tied  9 times
    lost  1 times

    total unique fp went from 6 to 7 lost   +16.67%
    mean fp % went from 0.3 to 0.35 lost   +16.67%

    false negative percentages
        0.000  0.000  tied          
        1.000  1.000  tied          
        1.000  0.500  won    -50.00%
        0.500  0.500  tied          
        2.000  2.000  tied          
        1.500  1.500  tied          
        0.000  0.000  tied          
        0.500  0.500  tied          
        0.500  1.000  lost  +100.00%
        0.000  0.000  tied          

    won   1 times
    tied  8 times
    lost  1 times

    total unique fn went from 14 to 14 tied          
    mean fn % went from 0.7 to 0.7 tied          

It seems to be no better.  Finally, I tried changing the random seed and
retesting with an empty bayescustomize.ini and with the settings that
gave me the best result with "Robinson":

    false positive percentages
        0.000  0.000  tied          
        1.500  1.000  won    -33.33%
        0.000  0.000  tied          
        0.000  0.000  tied          
        0.500  0.500  tied          
        0.500  0.500  tied          
        0.500  0.500  tied          
        0.500  0.500  tied          
        0.000  0.500  lost  +(was 0)
        0.000  0.000  tied          

    won   1 times
    tied  8 times
    lost  1 times

    total unique fp went from 7 to 7 tied          
    mean fp % went from 0.35 to 0.35 tied          

    false negative percentages
        0.500  0.000  won   -100.00%
        2.000  1.000  won    -50.00%
        1.000  0.500  won    -50.00%
        1.000  0.500  won    -50.00%
        2.000  2.000  tied          
        1.000  1.500  lost   +50.00%
        0.500  0.000  won   -100.00%
        2.000  0.500  won    -75.00%
        1.500  1.000  won    -33.33%
        0.000  0.000  tied          

    won   7 times
    tied  2 times
    lost  1 times

    total unique fn went from 23 to 14 won    -39.13%
    mean fn % went from 1.15 to 0.7 won    -39.13%


Well, that's all for me for now.  In summary, the best
result was with:

"""
[Classifier]
use_robinson_combining: True
use_robinson_probability: True
robinson_probability_x: 0.5
robinson_probability_a: 0.2
max_discriminators: 150
robinson_minimum_prob_strength: 0.1

[TestDriver]
SPAM_cutoff: 0.580
"""    

HTH,

  Neil

From tim.one@comcast.net  Wed Sep 25 04:54:50 2002
From: tim.one@comcast.net (Tim Peters)
Date: Tue, 24 Sep 2002 23:54:50 -0400
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <20020925030928.GA12308@glacier.arctrix.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCAEJEBGAB.tim.one@comcast.net>

[Tim report his usual pathetic results]
> total unique fp went from 1 to 1 tied
> mean fp % went from 0.05 to 0.05 tied
>
> total unique fn went from 1 to 0 won   -100.00%
> mean fn % went from 0.05 to 0.0 won   -100.00%

[Neil Schemenauer]
> You're a victim of your own success. :-)  Could you try sabotaging the
> classifier to make it work harder and produce higher error rates?  For
> example, don't let it look at any header information.

I'm not sure it would help enough (meaning I'm not sure it would hurt enough
<wink>).  Recall that I started by ignoring the headers completely, and
adding the few in that I did made only fraction-of-percent differences to
me.

I've said before that I'm suspicious the *tokenizer* is overtuned to my test
data:  I pretty much tokenized all and only the things that made a
significant difference for it (without picking up trivially strong clues
about newsgroup postings vs bruceg), and even tokenized different parts in
different ways likewise.  As TESTING.txt has prophetically said all along,

+ Any sufficiently general scheme with enough free parameters can
  eventually be trained to recognize any specific dataset exactly.
  It's wonderful if other people test your changes against other
  datasets too.  That's hard to arrange, so at least change your own
  data periodically.  I'm suspicious that some of the weirder "proven
  winner" changes I've made are really specific to statistical
  anomalies in my test data; and as the error rates get closer to 0%,
  the chance that a winning change helped only a few specific msgs
  zooms (of course sometimes that's intentional!  I haven't been shy
  about adding changes specifically geared toward squahsing very
  narrow classes of false positives).

What would really be interesting, then, is if Skip gave me his corpus --
he's reported by far the worst error rates of anyone, and I have an amply
demonstrated ability to change 1000 things at once and end up fitting the
data exactly <0.8 wink>.

I definitely need *some* new corpus to strain against.  GregW has made great
progress on collecting a new python.org corpus, with oodles of Asian spam,
and ham msgs other than just c.l.py traffic.  I look forward to watching my
error rates go down the toilet again.

But, for purposes of the test at hand, it's actually helpful:  the question
is which scheme does better, Guido's rates are much worse than mine, and
it's important to know whether one scheme dominates consistently, or whether
it's a mixed bag.


From barry@python.org  Wed Sep 25 04:58:07 2002
From: barry@python.org (Barry A. Warsaw)
Date: Tue, 24 Sep 2002 23:58:07 -0400
Subject: [Spambayes] Problems with unheader.py
References: <20020925015125.GB6550@cthulhu.gerg.ca>
	<LNBBLJKPBEHFEDALKOLCOEIJBGAB.tim.one@comcast.net>
Message-ID: <15761.13391.274132.989695@anthem.wooz.org>


>>>>> "TP" == Tim Peters <tim.one@comcast.net> writes:

    TP> [Greg Ward]
    >> ...  OK: anyone know how to get the raw message body from an
    >> email.Message object?  Or am I gonna have to RTFM?

    TP> Barry knows, but he usually won't tell you anything unless you
    TP> buy him lunch first.

    TP> str(msg) returns the whole msg as a string, including the
    TP> headers.  I'm not aware of a way to get just the body (unless
    TP> you luck out and it's not multipart, in which case
    TP> get_payload() does the trick).

There's always a way, and (ssh! don't tell Guido!) sometimes more than
one way.

Looks like you got what you wanted with HeaderParser, so we'll leave
it at that <wink>.

-Barry

From chk@pobox.com  Wed Sep 25 05:51:59 2002
From: chk@pobox.com (Harald Koch)
Date: Wed, 25 Sep 2002 00:51:59 -0400
Subject: [Spambayes] Re: There Can Be Only One 
In-Reply-To: Your message of "Tue, 24 Sep 2002 20:26:09 -0400".
	 <LNBBLJKPBEHFEDALKOLCAEICBGAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCAEICBGAB.tim.one@comcast.net>
Message-ID: <18221.1032929519@elisabeth.cfrq.net>

> To avoid distractions, there's only one kind of test run I'll look at for
> this:  a 10-fold cross-validation run with exactly 200 ham and 200 spam in
> each set.  If you don't have at least 2,000 ham and 2,000 spam, you can't
> run this test, and reporting results won't help.

I don't know about you, but I receive about 250 spam a *month*. It would
take me 8 months to collect 2000 spams. However, the spam I was
receiving eight months ago is generally quite different from the spam
I'm receiving now, so even if I *did* have an archive going back that
far it wouldn't be terribly useful.

One of the purported strengths of Paul's original idea was that it was
*adaptive*.  Based on my recent reading of this list, I believe two
important facts about spam in the wild are being downplayed:

- mail headers (I understand *why* you're doing it, but the
  discriminators on *my* spam vs. ham *do* often come from the headers.
  Spammers seem to all find the same open relays at the same time ;-)

- *time*. my spam is self-similar over short periods of time, and (with
  some exceptions) changes and evolves over longer periods. Token
  statistics collected eight months ago wouldn't eliminate much of my
  current spam. Heck, if i train the classifier on only the old spam and
  then run it against all of the new spam, the f-n rate is abysmal.

I'm running a very simple perl version of the algorithm right now. The
thing that most aggressively lowers my f-n rate is my daily inbox cull;
I feed f-n spam into the classifier whenever I find it.

> Q. This is a ham/spam ratio of 1.  Is that realistic?
> A. We can't test everything at once.

It can be, especially at certain times of the day; I get most of my spam
at night and most of my regular email during the day (North America time).

> Q. 1800 each of ham & spam is a very large training set.
>    Wouldn't it be better to use less training data?
> A. We can't test everything at once.

<laughter>

-- 
Harald Koch     <chk@pobox.com>

"It takes a child to raze a village."
		-Michael T. Fry

From tim.one@comcast.net  Wed Sep 25 06:25:51 2002
From: tim.one@comcast.net (Tim Peters)
Date: Wed, 25 Sep 2002 01:25:51 -0400
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <200209250250.g8P2oiU26801@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCAEJHBGAB.tim.one@comcast.net>

[Guido]
> ...
> The 60.0 bins in the histogram have 2 hams and 7 spams, so moving the
> cutoff to 0.625 would have made it a tie for fps and a loss by 1 for
> fns.

This is too painful.  There's a new option now, enabled by default:

"""
[TestDriver]
# When compute_best_cutoffs_from_histograms is enabled, after the
# display of a ham+spam histogram pair, a listing is given of all
# the cutoff scores (coinciding with a histogram boundary) that
# minimize the total number of misclassified messages (false
# positives + false negatives).
compute_best_cutoffs_from_histograms: True
"""

Other definitions of "best" are certainly possible.  You may wish to
increase nbuckets too (to increase the resolution of the automated
analysis).

The output looks like this(*):

-> best cutoff for this pair: 0.425
->     with 1 fp + 0 fn = 1 mistakes
->     matched at 0.45 (0 fp + 1 fn)
->     matched at 0.475 (0 fp + 1 fn)
->     matched at 0.5 (0 fp + 1 fn)

You can be sure then that the total errors are higher at all other histogram
boundary points.

> ...
> I guess spams for magic diets are common.

Exceedingly!  Remember the time one of my persistent false positives was a
long message from Alex Martelli, discussing the quality of water needed in
the preparation of pasta.  He got nailed more by the "water retention" parts
of diet spams than by the "water sports" parts of porn spams.  None of these
have much to do with Python, either <wink>.


(*) So nobody else bothers, here's comparing my f(w) run compared to the
same thing but setting robinson_probability_x to 0.1 (the "unknown word"
prob -- I set it to a silly value just to make sure it mattered -- if you
consider unknown words to be strong ham indicators, then there should be a
push toward calling all things ham, so the f-p rate should go down and the
f-n rate up; and so it is):

false positive percentages
    0.500  0.000  won   -100.00%
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied

won   1 times
tied  9 times
lost  0 times

total unique fp went from 1 to 0 won   -100.00%
mean fp % went from 0.05 to 0.0 won   -100.00%

false negative percentages
    0.000  1.500  lost  +(was 0)
    0.000  2.000  lost  +(was 0)
    0.000  3.000  lost  +(was 0)
    0.000  1.000  lost  +(was 0)
    0.000  1.500  lost  +(was 0)
    0.000  1.500  lost  +(was 0)
    0.000  1.500  lost  +(was 0)
    0.000  2.000  lost  +(was 0)
    0.000  1.000  lost  +(was 0)
    0.000  2.000  lost  +(was 0)

won   0 times
tied  0 times
lost 10 times

total unique fn went from 0 to 34 lost  +(was 0)
mean fn % went from 0.0 to 1.7 lost  +(was 0)

ham mean                     ham sdev
  33.01   28.66  -13.18%        6.26    5.49  -12.30%
  32.19   27.90  -13.33%        5.38    5.02   -6.69%
  32.99   28.62  -13.25%        5.60    5.16   -7.86%
  33.46   29.14  -12.91%        5.77    5.30   -8.15%
  33.16   28.70  -13.45%        5.56    5.11   -8.09%
  32.81   28.69  -12.56%        5.72    5.20   -9.09%
  33.38   29.15  -12.67%        5.76    5.30   -7.99%
  32.55   28.24  -13.24%        5.70    5.25   -7.89%
  33.11   28.73  -13.23%        5.52    5.08   -7.97%
  34.21   29.74  -13.07%        5.84    5.39   -7.71%

ham mean and sdev for all runs
  33.09   28.76  -13.09%        5.73    5.25   -8.38%

spam mean                    spam sdev
  82.95   71.96  -13.25%        6.82    7.26   +6.45%
  82.17   71.80  -12.62%        6.34    7.63  +20.35%
  82.06   71.37  -13.03%        6.14    7.34  +19.54%
  82.39   71.95  -12.67%        5.93    6.72  +13.32%
  82.53   72.43  -12.24%        7.00    7.56   +8.00%
  82.76   71.97  -13.04%        6.56    7.30  +11.28%
  82.06   71.55  -12.81%        5.73    6.90  +20.42%
  82.26   72.49  -11.88%        5.97    6.96  +16.58%
  82.65   72.90  -11.80%        6.71    7.54  +12.37%
  83.43   73.03  -12.47%        6.37    7.95  +24.80%

spam mean and sdev for all runs
  82.53   72.14  -12.59%        6.37    7.34  +15.23%

ham/spam mean difference: 49.44 43.38 -6.06

Now if I *were* to keep such a silly change, the last histogram analysis at
least tells me how to minimize the damage:

-> best cutoff for all runs: 0.475
->     with 2 fp + 6 fn = 8 mistakes

Since the change was in the direction of calling more things ham, this makes
good sense:  reducing spam_cutoff works in the direction of calling more
things spam.


From tim.one@comcast.net  Wed Sep 25 07:03:56 2002
From: tim.one@comcast.net (Tim Peters)
Date: Wed, 25 Sep 2002 02:03:56 -0400
Subject: [Spambayes] Re: There Can Be Only One
In-Reply-To: <18221.1032929519@elisabeth.cfrq.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCAEJJBGAB.tim.one@comcast.net>

[Harald Koch]
> I don't know about you, but I receive about 250 spam a *month*.

Now you know about me too <wink>:  I get more than 100 a day, not counting
my hotmail account.

> It would take me 8 months to collect 2000 spams. However, the spam
> I was receiving eight months ago is generally quite different from
> the spam I'm receiving now, so even if I *did* have an archive going
> back that far it wouldn't be terribly useful.

I'm not sure what "quite different" means.  I've been getting porn spam, and
get-rich-quick spam, and human-growth-hormone spam (etc), for a loooooong
time.  Ooh!  A new one just came while I was typing:  "Dear Valued Friend,
Great news! Your new 50% Hotel Discount Card is ready to be downloaded!
[yadda yadda yadda]" ... there's nothing unique about that.

As to whether an old archive would or wouldn't be useful, that's something
that could be tested (and should be).

Another one just came in:

    Subject: Change your life!!! Last time you spent $25 did u make
        $100,000,s? Well this time you will. pvh

    Hello

    You may have seen this business before and
    ignored it. I know I did - many times! However,
    please take a few moments to read this letter.
    I was amazed when the profit potential of this
    business finally sunk in... and it works!

    [yadda yadda yadda]

I've been seeing that one for years too (it's the usual chain letter).

Are you using an email account that already has some sort of spam filtering
in place?  If so, that would explain why only unusual spam gets through to
you.  I do nothing to try to stop spam.

Another one just came in:  the usual Viagra pitch.

> One of the purported strengths of Paul's original idea was that it was
> *adaptive*.

Indeed yes!  We didn't take that part out, and I've run experiments starting
by training on a single ham and a single spam.  With just that much, the fp
rate was about 33% and the fn rate about 20%, predicting against msgs at
random from my test data.  Add another ham and spam to the training, and it
does better.  Etc -- it certainly learns.  Something I suspect but haven't
had time to test:  there's a critical mass of spam messages such that once
you've trained on that many, you've basically seen them all.  4,000 isn't
enough.  10,000 may be.

> Based on my recent reading of this list, I believe two important facts
> about spam in the wild are being downplayed:
>
> - mail headers (I understand *why* you're doing it, but the
>   discriminators on *my* spam vs. ham *do* often come from the headers.
>   Spammers seem to all find the same open relays at the same time ;-)

Other people on this list are mining the headers.  I can't, until I get a
single-source corpus.  This has also been a strength for us:  I started
without looking at *any* headers (they were chopped off the msg first
thing), only at content, and both error rates from the original scheme got
chopped by at least a factor of 10 each since then.  It's hard to imagine
that adding rich clues from the headers is going to make that worse,
although only time and testing will tell.

> - *time*. my spam is self-similar over short periods of time, and (with
>   some exceptions) changes and evolves over longer periods. Token
>   statistics collected eight months ago wouldn't eliminate much of my
>   current spam. Heck, if i train the classifier on only the old spam and
>   then run it against all of the new spam, the f-n rate is abysmal.

It would be interesting to try the same experiment with our classifier; I
expect our content tokenization is more powerful, simply because ours
started without the headers all.  For example, the false negative rate was
cut in half just by special parsing and tagging of embedded URLs; e.g.,
"remove" in a message body isn't particularly incriminating, but "remove"
embedded in a URL specifically is plain damning.

> I'm running a very simple perl version of the algorithm right now. The
> thing that most aggressively lowers my f-n rate is my daily inbox cull;
> I feed f-n spam into the classifier whenever I find it.

You didn't say how many total spam you've trained it on, but you only get
250 a month and this scheme isn't much older than that.  If you have
relatively few data points, then it's not surprising that feeding it more
helps a lot.

>> Q. This is a ham/spam ratio of 1.  Is that realistic?
>> A. We can't test everything at once.

> It can be, especially at certain times of the day; I get most of my
> spam at night and most of my regular email during the day
> (North America time).

I believe it!  I expect there are great clues in the headers about this that
I'm missing too.

>> Q. 1800 each of ham & spam is a very large training set.
>>    Wouldn't it be better to use less training data?
>> A. We can't test everything at once.
>
> <laughter>

My usual test runs on 20,000 ham and 14,000 spam.  An original purpose of
this project was to investigate whether this scheme would be suitable for
filtering the email traffic going through python.org.  Compared to that
purpose, the test sizes I'm using are a drop in the bucket.  Even with the
limited header analysis I'm doing, on that large test I have a total of 4
false positives and 24 false negatives.  How's the Perl script doing <wink>?

Another spam just came in:

    Subject: Look At the Quality of our cars

    Dear Sir / Madam

    We are a Singapore based company dealing in Quality Used Cars and
    looking for Dealers in countries that is Right Hand Drive.

    Take a look at the quality of our cars below, click on the Number
    Plate beside the price to see photo fo the car.

    [yadda yadda yadda]

BTW, all the spams that came in while I typed this were nailed by the
classifier -- they're simply timeless in their relentless spammishness
<wink>.


From anthony@interlink.com.au  Wed Sep 25 07:30:45 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Wed, 25 Sep 2002 16:30:45 +1000
Subject: [Spambayes] Re: There Can Be Only One 
In-Reply-To: <LNBBLJKPBEHFEDALKOLCAEJJBGAB.tim.one@comcast.net> 
Message-ID: <200209250630.g8P6UkN08684@localhost.localdomain>


>>> Tim Peters wrote
> Other people on this list are mining the headers.  I can't, until I get a
> single-source corpus.  This has also been a strength for us:  I started
> without looking at *any* headers (they were chopped off the msg first
> thing), only at content, and both error rates from the original scheme got
> chopped by at least a factor of 10 each since then.  It's hard to imagine
> that adding rich clues from the headers is going to make that worse,
> although only time and testing will tell.

On my personal corpus, which is around 10k ham and 700 spam now, I'm 
actually seeing the full headers causing harm, not benefit. Analyzing
it some, it looks like the problem is that _most_ of my spam is delivered
directly to me, while most ham is via mailing lists. This means that the
occasional spam via mailing lists nearly always gets fn'd. 

More on that later - I'm currently processing the scheme-deathmatch.

Anthony

-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.


From anthony@interlink.com.au  Wed Sep 25 07:50:37 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Wed, 25 Sep 2002 16:50:37 +1000
Subject: [Spambayes] Practical applications 
Message-ID: <200209250650.g8P6odp08869@localhost.localdomain>


>>> Anthony Baxter wrote
> I think he was asking for something the other way - a way to add
> a button/command to Outbreak which runs a python script. I don't know if
> that's possible - someone like MarkH may know.

Something similar came up on Python-list, so I followed up with Mark. 
Here's a summary:

>>> Mark wrote
> You would need to use the new Universal Gateway code (see 
> win32com\test\test_pycomtest.py) and create a server that implements 
> IDTExtensibility2.  From there, you should be able to use any of the VB 
> samples to move forward - all the params passed are standard COM 
> objects, so you should be able to do whatever you want.  You would also 
> need to get your head around COM events and how to connect to them from 
> Python, as addins are very much event based.
> 
> I'm doing some work on a plugin now - but not using Python :(  And I 
> would *love* to see this work, especially in the face of the Bayes work 
> being done in Python.  Mail me if you need a hand.

I asked about hooking into "get new messages" to filter the message, and
also adding 'delete as ham/delete as spam' buttons. Mark's response was
that "Both are trivial once the [outlook-python] framework is in place".
(This is with full Outlook, not Outlook Express)

So, anyone who runs windows and Outlook feel like taking this on? 

Anthony, who is not at all dismayed that MH doesn't have a COM interface...

From anthony@interlink.com.au  Wed Sep 25 09:09:04 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Wed, 25 Sep 2002 18:09:04 +1000
Subject: [Spambayes] Two Scheme Enter, One Scheme Leave.
In-Reply-To: <LNBBLJKPBEHFEDALKOLCAEICBGAB.tim.one@comcast.net>
Message-ID: <200209250809.g8P894H09766@localhost.localdomain>


This is on my mungo-corpus, (which, after it's most recent update, is
now 11300 spam and 20200 ham), but selecting only 2000/2000. I chose
a seed of 12346, just to do one better than Tim :)

Robinson defaults as supplied by Tim

  (Graham on left)

    false positive percentages
        0.500  0.500  tied
        2.500  3.500  lost   +40.00%
        1.500  2.500  lost   +66.67%
        2.500  3.000  lost   +20.00%
        2.500  3.500  lost   +40.00%
        1.000  0.500  won    -50.00%
        0.500  2.500  lost  +400.00%
        1.000  2.500  lost  +150.00%
        1.500  2.500  lost   +66.67%
        2.500  2.500  tied

    won   1 times
    tied  2 times
    lost  7 times

    total unique fp went from 32 to 47 lost   +46.88%
    mean fp % went from 1.6 to 2.35 lost   +46.88%

    false negative percentages
        0.000  0.000  tied
        1.500  0.500  won    -66.67%
        0.500  0.000  won   -100.00%
        0.500  0.000  won   -100.00%
        1.000  1.000  tied
        0.500  0.500  tied
        0.500  0.500  tied
        1.000  0.000  won   -100.00%
        0.000  0.000  tied
        0.000  0.500  lost  +(was 0)

    won   4 times
    tied  5 times
    lost  1 times

    total unique fn went from 11 to 6 won    -45.45%
    mean fn % went from 0.55 to 0.3 won    -45.45%

Raising spam_cutoff to 0.6 (the optimal value for minimum fn+fp) gives us
  (Graham on left)

    total unique fp went from 32 to 11 won    -65.62%
    mean fp % went from 1.6 to 0.55 won    -65.62%
    total unique fn went from 11 to 21 lost   +90.91%
    mean fn % went from 0.55 to 1.05 lost   +90.91%

  (default Robinson (spam_cutoff 0.550) on left)

    total unique fp went from 47 to 11 won    -76.60%
    mean fp % went from 2.35 to 0.55 won    -76.60%
    total unique fn went from 6 to 21 lost  +250.00%
    mean fn % went from 0.3 to 1.05 lost  +250.00%

So let's leave spam_cutoff at 0.6 from now on (rather than trying to
juggle 15 different parameters at once).

Summarising values tried for robinson_probability_a
(using the 0.6 cutoff)

     a      fp       fn    fp+fn
    0.0     18      650     668
    0.001   13       36      49
    0.01    13       28      41
    0.025   12       24      36
    0.05    11       23      34
    0.075   10       21      31
    0.1      9       21      30
    0.125   10       21      31
    0.15    10       21      31
    0.2      9       22      31
    0.25     9       22      31
    0.35    10       21      31
    0.45    10       22      32
    0.5     11       21      33     (tim's default)
    1.0     13       29      42
    2.0     12       42      54
    10.0    11       96      107

I have the raw run data for these, if anyone cares. They're rather large :)

So it looks like a=0.1,cutoff=0.6 is the winning combo of these two. 
  (Graham on left)
    total unique fp went from 32 to 9 won    -71.88%
    mean fp % went from 1.6 to 0.45 won    -71.88%
    total unique fn went from 11 to 21 lost   +90.91%
    mean fn % went from 0.55 to 1.05 lost   +90.91%

  (default Robinson on left)
    total unique fp went from 47 to 9 won    -80.85%
    mean fp % went from 2.35 to 0.45 won    -80.85%
    total unique fn went from 6 to 21 lost  +250.00%
    mean fn % went from 0.3 to 1.05 lost  +250.00%

Next up, the other knobs and dials!

Anthony

From marklists@mceahern.com  Wed Sep 25 13:17:54 2002
From: marklists@mceahern.com (Mark McEahern)
Date: Wed, 25 Sep 2002 07:17:54 -0500
Subject: [Spambayes] Practical applications 
In-Reply-To: <200209250650.g8P6odp08869@localhost.localdomain>
Message-ID: <JHEOKEOOLIGLDHCMAHMOIEFMEBAA.marklists@mceahern.com>

[Anthony Baxter]
[snip Mark Hammond's analysis]
> So, anyone who runs windows and Outlook feel like taking this on?

I heard that Outlook 2002 has a different way to do this than previous
versions, so the method described--is that only for Outlook 2002?

Thanks,

// mark


From Alexander@Leidinger.net  Wed Sep 25 09:17:33 2002
From: Alexander@Leidinger.net (Alexander Leidinger)
Date: Wed, 25 Sep 2002 10:17:33 +0200
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <LNBBLJKPBEHFEDALKOLCAEJEBGAB.tim.one@comcast.net>
References: <20020925030928.GA12308@glacier.arctrix.com>
	<LNBBLJKPBEHFEDALKOLCAEJEBGAB.tim.one@comcast.net>
Message-ID: <20020925101733.6a6c188a.Alexander@Leidinger.net>

On Tue, 24 Sep 2002 23:54:50 -0400 Tim Peters <tim.one@comcast.net>
wrote:

> I definitely need *some* new corpus to strain against.  GregW has made
> great progress on collecting a new python.org corpus, with oodles of
> Asian spam, and ham msgs other than just c.l.py traffic.  I look
> forward to watching my error rates go down the toilet again.

I'm working on splitting up the FreeBSD mailinglist archive (1997-2001)
from ~9k mbox files into your preferred test set layout. After that I
try to seperate the spam from the ham. I expect it to be x*10^6 messages
(spam+ham). I you are interested I will make (parts of it) available for
download somewhere (I don't know when I have them ready, I'm fighting
with some problems at the moment, see other messages to the list from
me).

Bye,
Alexander.

-- 
            Secret hacker rule #11: hackers read manuals.

http://www.Leidinger.net                       Alexander @ Leidinger.net
  GPG fingerprint = C518 BC70 E67F 143F BE91  3365 79E2 9C60 B006 3FE7

From Alexander@Leidinger.net  Wed Sep 25 12:28:58 2002
From: Alexander@Leidinger.net (Alexander Leidinger)
Date: Wed, 25 Sep 2002 13:28:58 +0200
Subject: [Spambayes] Mailbox class in the spambayes project & python 2.2.1
Message-ID: <20020925132858.2880c885.Alexander@Leidinger.net>

Hi,

at http://www.leidinger.net/20001001.freebsd-isp.txt.bz2 (40k) I have a
mbox which is responsible for this error:
---snip---
Traceback (most recent call last):
  File "../../mboxcount.py", line 100, in ?
    main()
  File "../../mboxcount.py", line 90, in main
    goodn, badn = count(fname)
  File "../../mboxcount.py", line 58, in count
    for msg in mbox:
  File "/usr/local/lib/python2.2/mailbox.py", line 34, in next
    return self.factory(_Subfile(self.fp, start, stop))
  File "../../mboxcount.py", line 49, in _factory
    return email.message_from_file(fp)
  File "/home/netchild/work/src/spambayes/email/__init__.py", line 39, in message_from_file
    return _Parser(_class, strict=strict).parse(fp)
  File "/home/netchild/work/src/spambayes/email/Parser.py", line 48, in parse
    self._parsebody(root, fp)
  File "/home/netchild/work/src/spambayes/email/Parser.py", line 118, in _parsebody
    boundary = container.get_boundary()
  File "/home/netchild/work/src/spambayes/email/Message.py", line 677, in get_boundary
    return _unquotevalue(boundary.strip())
AttributeError: 'tuple' object has no attribute 'strip'
---snip---

I haven't tried to reduce the unpacked ~200k file into a smaller one
which still produces this traceback because I think it should be easy
(with intimate knowledge of Message.py) to just add some debug output to
Message.py to find the problem.

Bye,
Alexander.

-- 
     The three Rs of Microsoft support: Retry, Reboot, Reinstall.

http://www.Leidinger.net                       Alexander @ Leidinger.net
  GPG fingerprint = C518 BC70 E67F 143F BE91  3365 79E2 9C60 B006 3FE7

From anthony@interlink.com.au  Wed Sep 25 15:23:39 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Thu, 26 Sep 2002 00:23:39 +1000
Subject: [Spambayes] Practical applications 
In-Reply-To: <JHEOKEOOLIGLDHCMAHMOIEFMEBAA.marklists@mceahern.com> 
Message-ID: <200209251423.g8PENd603392@localhost.localdomain>


>>> "Mark McEahern" wrote
> I heard that Outlook 2002 has a different way to do this than previous
> versions, so the method described--is that only for Outlook 2002?


Don't know. The only time I boot windows is to play warcraft3. :)

Drop MarkH an email, ask him...




From anthony@interlink.com.au  Wed Sep 25 16:18:11 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Thu, 26 Sep 2002 01:18:11 +1000
Subject: [Spambayes] Two Scheme Enter, One Scheme Leave. 
In-Reply-To: <200209250809.g8P894H09766@localhost.localdomain> 
Message-ID: <200209251518.g8PFIBP04211@localhost.localdomain>


Part the second...

A brief sidetrip into fiddling robinson_probability_x showed that
setting it to 0.4 and 0.6 (instead of the default 0.5) had no real
affect on fp/fn numbers, but resulted in average ham and spam numbers
being around 1% lower and higher, respectively. 


min_prob_strength is next. Carrying over best so far, 
(cutoff=0.6, a=0.1, x=0.5)

        fp      fn      fp+fn
0.00     7      50       57
0.05     8      23       31
0.08     9      21       30
0.09     9      21       30
0.10     9      21       30
0.11    12      20       32
0.12    12      20       32
0.15    13      19       32
0.20    23      19       42
0.25    23      18       41
0.30    28      15       43
0.35    29      17       46
0.40    36      17       53
0.45    51      17       68
0.49    75      32      107

The "best" cutoff numbers for the different min_prob_strength settings:

        fp      fn      fp+fn   cutoff
0.00    13      24       37     0.575
0.05     8      23       31     0.6
0.08     9      21       30     0.6
0.09     9      21       30     0.6
0.10     9      21       30     0.6
0.11    12      20       32	0.6
0.12    12      20       32	0.6
0.15    13      19       32     0.6
0.20    13      26       39     0.625
0.25    23      18       41     0.6
0.30    28      15       43     0.6
0.35    23      21       44     0.625
0.40    27      18       45     0.625
0.45    40      27       67     0.625
0.49    77      26      103     0.575

That's it for tonight. If people (well, ok, Tim) want more detail, 
let me know, and let me know what you want to see. All up, just the 
test_foo_2s.txt summary files alone are about 4M of data (about 35 
test runs). If the tram ride to work tomorrow is slow, I might write 
something to run through all the data files and try to load it all 
up into some sort of 4d array or something, see if it sees anything 
interesting...

Tomorrow, I'll try the current "best settings"
(cutoff=0.6, a=0.1, x=0.5, min_prob_strength=0.09) 
with a few different seeds, compared to Graham, and also try with
different spam/ham corpus sizes.

Anthony

From guido@python.org  Wed Sep 25 16:43:52 2002
From: guido@python.org (Guido van Rossum)
Date: Wed, 25 Sep 2002 11:43:52 -0400
Subject: [Spambayes] Mailbox class in the spambayes project & python 2.2.1
In-Reply-To: Your message of "Wed, 25 Sep 2002 13:28:58 +0200."
             <20020925132858.2880c885.Alexander@Leidinger.net> 
References: <20020925132858.2880c885.Alexander@Leidinger.net> 
Message-ID: <200209251543.g8PFhqE02663@odiug.zope.com>

> at http://www.leidinger.net/20001001.freebsd-isp.txt.bz2 (40k) I have a
> mbox which is responsible for this error:
> ---snip---
> Traceback (most recent call last):
>   File "../../mboxcount.py", line 100, in ?
>     main()
>   File "../../mboxcount.py", line 90, in main
>     goodn, badn = count(fname)
>   File "../../mboxcount.py", line 58, in count
>     for msg in mbox:
>   File "/usr/local/lib/python2.2/mailbox.py", line 34, in next
>     return self.factory(_Subfile(self.fp, start, stop))
>   File "../../mboxcount.py", line 49, in _factory
>     return email.message_from_file(fp)
>   File "/home/netchild/work/src/spambayes/email/__init__.py", line 39, in message_from_file
>     return _Parser(_class, strict=strict).parse(fp)
>   File "/home/netchild/work/src/spambayes/email/Parser.py", line 48, in parse
>     self._parsebody(root, fp)
>   File "/home/netchild/work/src/spambayes/email/Parser.py", line 118, in _parsebody
>     boundary = container.get_boundary()
>   File "/home/netchild/work/src/spambayes/email/Message.py", line 677, in get_boundary
>     return _unquotevalue(boundary.strip())
> AttributeError: 'tuple' object has no attribute 'strip'
> ---snip---
> 
> I haven't tried to reduce the unpacked ~200k file into a smaller one
> which still produces this traceback because I think it should be easy
> (with intimate knowledge of Message.py) to just add some debug output to
> Message.py to find the problem.

I think I've found a message that could trigger a problem with this
mailbox.  It's either a bug in the email package, or it violates a
simplifying assumption made by Tim's tokenizer.  More when I can get
some of Barry's time.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido@python.org  Wed Sep 25 16:58:36 2002
From: guido@python.org (Guido van Rossum)
Date: Wed, 25 Sep 2002 11:58:36 -0400
Subject: [Spambayes] Mailbox class in the spambayes project & python 2.2.1
In-Reply-To: Your message of "Wed, 25 Sep 2002 13:28:58 +0200."
             <20020925132858.2880c885.Alexander@Leidinger.net> 
References: <20020925132858.2880c885.Alexander@Leidinger.net> 
Message-ID: <200209251558.g8PFwaB02740@odiug.zope.com>

> at http://www.leidinger.net/20001001.freebsd-isp.txt.bz2 (40k) I have a
> mbox which is responsible for this error:

[...]
>   File "/home/netchild/work/src/spambayes/email/Message.py", line 677, in get_boundary
>     return _unquotevalue(boundary.strip())
> AttributeError: 'tuple' object has no attribute 'strip'

There are two messages in that mailbox from Goblin <h0b_g0blin@yahoo.com>.

These messages use a rare new header feature, encoded parameters.  The
first one has this:

Content-Type: text/plain; charset*=ansi-x3-4-1968''us-ascii

and the second one has this:

Content-Type: multipart/signed; micalg*=ansi-x3-4-1968''pgp-md5;
	protocol*=ansi-x3-4-1968''application%2Fpgp-signature;
	boundary*="ansi-x3-4-1968''EeQfGwPcQSOJBaQU"

as well as this (in one of the body parts):

Content-Type: text/plain; charset*=ansi-x3-4-1968''us-ascii

I get the above failure on the second message.  Here's the code:

    def get_boundary(self, failobj=None):
        """Return the boundary associated with the payload if present.

        The boundary is extracted from the Content-Type: header's `boundary'
        parameter, and it is unquoted.
        """
        missing = []
        boundary = self.get_param('boundary', missing)
        if boundary is missing:
            return failobj
        return _unquotevalue(boundary.strip())

This fails because in this case, boundary is the tuple
('ansi-x3-4-1968', '', 'EeQfGwPcQSOJBaQU') where the actual boundary
is the third element.  (The first is an encoding, a fancy name for
ASCII AFAICT; the second is an optional natural language.)

Barry isn't sure whether get_boundary() should expect such a tuple
here or whether get_param() should return the 3rd item of the tuple.

I get other failures in Hammie on the content-type; the code (part of
crach_content_xyz() in spambayes/tokenizer.py) is this:

    for x in msg.get_charsets(None):
        if x is not None:
            yield 'charset:' + x.lower()

where the code doesn't expect a triple of the form ('ansi-x3-4-1968',
'', 'us-ascii').  I don't know yet whether get_charsets() should have
returned the third tuple item or whether crack_content_xyz() should be
fixed to expect a tuple.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From Alexander@Leidinger.net  Wed Sep 25 16:57:51 2002
From: Alexander@Leidinger.net (Alexander Leidinger)
Date: Wed, 25 Sep 2002 17:57:51 +0200
Subject: [Spambayes] Mailbox class in the spambayes project & python 2.2.1
In-Reply-To: <200209251543.g8PFhqE02663@odiug.zope.com>
References: <20020925132858.2880c885.Alexander@Leidinger.net>
	<200209251543.g8PFhqE02663@odiug.zope.com>
Message-ID: <20020925175751.15d1b00d.Alexander@Leidinger.net>

On Wed, 25 Sep 2002 11:43:52 -0400 Guido van Rossum <guido@python.org>
wrote:

> I think I've found a message that could trigger a problem with this
> mailbox.  It's either a bug in the email package, or it violates a
> simplifying assumption made by Tim's tokenizer.  More when I can get
> some of Barry's time.

The tokenizer isn't involved here. mboxcount.py is complaining here.

Bye,
Alexander.

-- 
   "One world, one web, one program"  -- Microsoft promotional ad
         "Ein Volk, ein Reich, ein Fuehrer"  -- Adolf Hitler

http://www.Leidinger.net                       Alexander @ Leidinger.net
  GPG fingerprint = C518 BC70 E67F 143F BE91  3365 79E2 9C60 B006 3FE7

From Alexander@Leidinger.net  Wed Sep 25 17:14:33 2002
From: Alexander@Leidinger.net (Alexander Leidinger)
Date: Wed, 25 Sep 2002 18:14:33 +0200
Subject: [Spambayes] Mailbox class in the spambayes project & python 2.2.1
In-Reply-To: <200209251558.g8PFwaB02740@odiug.zope.com>
References: <20020925132858.2880c885.Alexander@Leidinger.net>
	<200209251558.g8PFwaB02740@odiug.zope.com>
Message-ID: <20020925181433.60cd382b.Alexander@Leidinger.net>

On Wed, 25 Sep 2002 11:58:36 -0400 Guido van Rossum <guido@python.org>
wrote:

> Content-Type: multipart/signed; micalg*=ansi-x3-4-1968''pgp-md5;
> 	protocol*=ansi-x3-4-1968''application%2Fpgp-signature;
> 	boundary*="ansi-x3-4-1968''EeQfGwPcQSOJBaQU"

[...]

>     def get_boundary(self, failobj=None):
>         """Return the boundary associated with the payload if present.
> 
>         The boundary is extracted from the Content-Type: header's
>         `boundary' parameter, and it is unquoted.
>         """
>         missing = []
>         boundary = self.get_param('boundary', missing)
>         if boundary is missing:
>             return failobj
>         return _unquotevalue(boundary.strip())
> 
> This fails because in this case, boundary is the tuple
> ('ansi-x3-4-1968', '', 'EeQfGwPcQSOJBaQU') where the actual boundary
> is the third element.  (The first is an encoding, a fancy name for
> ASCII AFAICT; the second is an optional natural language.)
> 
> Barry isn't sure whether get_boundary() should expect such a tuple
> here or whether get_param() should return the 3rd item of the tuple.

IMHO expecting such a tuple in get_boundary() (and perhaps propagating
it to upper layers) seems logical, you may be able to get useful
additional information out of it (compared to trowing it away in
get_baram()).

> I get other failures in Hammie on the content-type; the code (part of
> crach_content_xyz() in spambayes/tokenizer.py) is this:
> 
>     for x in msg.get_charsets(None):
>         if x is not None:
>             yield 'charset:' + x.lower()
> 
> where the code doesn't expect a triple of the form ('ansi-x3-4-1968',
> '', 'us-ascii').  I don't know yet whether get_charsets() should have
> returned the third tuple item or whether crack_content_xyz() should be
> fixed to expect a tuple.

This depends. Do you want to add get_charset_encoding() and
get_charset_fancy_name()?
 no: crack_content_xyz() should understand the triple.
 yes: get_charset() should return the third element.

Bye,
Alexander.

-- 
              The best things in life are free, but the
                expensive ones are still worth a look.

http://www.Leidinger.net                       Alexander @ Leidinger.net
  GPG fingerprint = C518 BC70 E67F 143F BE91  3365 79E2 9C60 B006 3FE7

From tim.one@comcast.net  Wed Sep 25 17:28:10 2002
From: tim.one@comcast.net (Tim Peters)
Date: Wed, 25 Sep 2002 12:28:10 -0400
Subject: [Spambayes] Re: There Can Be Only One
In-Reply-To: <200209250630.g8P6UkN08684@localhost.localdomain>
Message-ID: <BIEJKCLHCIOIHAGOKOLHMEADDLAA.tim.one@comcast.net>

[Anthony Baxter]
> On my personal corpus, which is around 10k ham and 700 spam now, I'm
> actually seeing the full headers causing harm, not benefit. Analyzing
> it some, it looks like the problem is that _most_ of my spam is delivered
> directly to me, while most ham is via mailing lists. This means that the
> occasional spam via mailing lists nearly always gets fn'd.

Good point.  I noticed that in your msg here, python.org + Mailman added all
this cruft to the headers:

Sender: spambayes-bounces@python.org
Errors-to: spambayes-bounces@python.org
Precedence: list
X-BeenThere: spambayes@python.org
X-Spam-Status: No,
 hits=-12.7 required=5.0
tests=X_AUTH_WARNING,TO_LOCALPART_EQ_REAL,BODY_PYTHON_ZOPE,
      USER_IN_WHITELIST_TO
X-Spam-Level:
X-Mailman-Version: 2.1b3+
List-Post: <mailto:spambayes@python.org>
List-Subscribe: <http://mail.python.org/mailman-21/listinfo/spambayes>,
	<mailto:spambayes-request@python.org?subject=subscribe>
List-Unsubscribe: <http://mail.python.org/mailman-21/listinfo/spambayes>,
	<mailto:spambayes-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail-21/spambayes>
List-Help: <mailto:spambayes-request@python.org?subject=help>
List-Id: Discussion list for Pythonic Bayesian classifier
<spambayes.python.org>

That's likely to generate at least a dozen strong ham tokens, and it's going
to be very hard to overcome the hamminess of that many clues (perhaps even
impossible under the max_discriminators=16 Graham scheme).

Does enabling mine_received_headers alone also hurt you?

> More on that later - I'm currently processing the scheme-deathmatch.

And doing a fine job too -- thank you.


From Alexander@Leidinger.net  Wed Sep 25 17:31:03 2002
From: Alexander@Leidinger.net (Alexander Leidinger)
Date: Wed, 25 Sep 2002 18:31:03 +0200
Subject: [Spambayes] Mailbox class in the spambayes project & python 2.2.1
In-Reply-To: <200209251558.g8PFwaB02740@odiug.zope.com>
References: <20020925132858.2880c885.Alexander@Leidinger.net>
	<200209251558.g8PFwaB02740@odiug.zope.com>
Message-ID: <20020925183103.384cfc74.Alexander@Leidinger.net>

On Wed, 25 Sep 2002 11:58:36 -0400 Guido van Rossum <guido@python.org>
wrote:

> Content-Type: multipart/signed; micalg*=ansi-x3-4-1968''pgp-md5;
> 	protocol*=ansi-x3-4-1968''application%2Fpgp-signature;
> 	boundary*="ansi-x3-4-1968''EeQfGwPcQSOJBaQU"

> This fails because in this case, boundary is the tuple
> ('ansi-x3-4-1968', '', 'EeQfGwPcQSOJBaQU') where the actual boundary
> is the third element.  (The first is an encoding, a fancy name for
> ASCII AFAICT; the second is an optional natural language.)

This means the boundary is encoded with ansi-x3-4-1968? So get_boundary()
is supposed to return ansi_x3_4_1968.decode(boundary) with the actual API?

Do I hear something like iconv awakening?

Bye,
Alexander.

P.S.: s/fancy_name\(\)/natural_language()/ in my last response to this mail.
-- 
            The dark ages were caused by the Y1K problem.

http://www.Leidinger.net                       Alexander @ Leidinger.net
  GPG fingerprint = C518 BC70 E67F 143F BE91  3365 79E2 9C60 B006 3FE7

From barry@zope.com  Wed Sep 25 17:46:54 2002
From: barry@zope.com (Barry A. Warsaw)
Date: Wed, 25 Sep 2002 12:46:54 -0400
Subject: [Spambayes] Mailbox class in the spambayes project & python 2.2.1
References: <20020925132858.2880c885.Alexander@Leidinger.net>
	<200209251558.g8PFwaB02740@odiug.zope.com>
	<20020925181433.60cd382b.Alexander@Leidinger.net>
Message-ID: <15761.59518.386970.451680@anthem.wooz.org>


>>>>> "AL" == Alexander Leidinger <Alexander@leidinger.net> writes:

    AL> IMHO expecting such a tuple in get_boundary() (and perhaps
    AL> propagating it to upper layers) seems logical, you may be able
    AL> to get useful additional information out of it (compared to
    AL> trowing it away in get_baram()).

Or, should get_param() return the decoded parameter as a string?  Note
that AFAIK, the email package doesn't provide any utilities to do this
decoding yet.

Also Python doesn't know anything about charset "ansi-x3-4-1968" but
it does know about "ansi-x3.4-1968".  So you can't do a non-kludge
decode anyway.  That should probably be fixed in Python somehow.  For
now, adding a hack to just return the third element of the triple
should keep you going, but I doubt that's the right long term solution.

We'll thrash out the right fix on the mimelib-devel@lists.sf.net list.

-Barry

From guido@python.org  Wed Sep 25 17:51:26 2002
From: guido@python.org (Guido van Rossum)
Date: Wed, 25 Sep 2002 12:51:26 -0400
Subject: [Spambayes] Mailbox class in the spambayes project & python 2.2.1
In-Reply-To: Your message of "Wed, 25 Sep 2002 18:31:03 +0200."
             <20020925183103.384cfc74.Alexander@Leidinger.net> 
References: <20020925132858.2880c885.Alexander@Leidinger.net>
	<200209251558.g8PFwaB02740@odiug.zope.com>  
	<20020925183103.384cfc74.Alexander@Leidinger.net> 
Message-ID: <200209251651.g8PGpQj11912@odiug.zope.com>

> On Wed, 25 Sep 2002 11:58:36 -0400 Guido van Rossum <guido@python.org>
> wrote:
> 
> > Content-Type: multipart/signed; micalg*=ansi-x3-4-1968''pgp-md5;
> > 	protocol*=ansi-x3-4-1968''application%2Fpgp-signature;
> > 	boundary*="ansi-x3-4-1968''EeQfGwPcQSOJBaQU"
> 
> > This fails because in this case, boundary is the tuple
> > ('ansi-x3-4-1968', '', 'EeQfGwPcQSOJBaQU') where the actual boundary
> > is the third element.  (The first is an encoding, a fancy name for
> > ASCII AFAICT; the second is an optional natural language.)
> 
> This means the boundary is encoded with ansi-x3-4-1968? So get_boundary()
> is supposed to return ansi_x3_4_1968.decode(boundary) with the actual API?

I tried this:

  if isinstance(x, tuple):
     assert len(x) == 3
     x = unicode(x[2], x[0]).encode('ascii')

but this didn't work because while Python's encodings package knows
'ansi_x3.4_1968' and 'ansi_x3.4_1986' (sic) as aliases for 'ascii', it
doesn't know 'ansi-x3-4-1968'. :-(

I've added this to Lib/encodings/aliases.py in the 2.3 trunk and 2.2.2
branch, but that wouldn't help current deployment using 2.2.1 or even
2.2, which we're currently aiming for.

I consider this a buglet in Goblin's mailer: things like charset names
and boundaries are supposed to be ASCII according to the MIME
standard, and he is unnecessarily explicit about this encoding.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.one@comcast.net  Wed Sep 25 17:52:06 2002
From: tim.one@comcast.net (Tim Peters)
Date: Wed, 25 Sep 2002 12:52:06 -0400
Subject: [Spambayes] Advantages of S?
In-Reply-To: <20020919145314.GA8615@athame.dynamicro.on.ca>
Message-ID: <BIEJKCLHCIOIHAGOKOLHOEAEDLAA.tim.one@comcast.net>

[Greg Louis, reports on an experiment using f(w) in a modified bogofilter]
> ...
> Yup.  In the test I reported earler, the one false positive had an S
> value of 0.93 or thereabouts.  It was a message to a bogofilter mailing
> list and looked extremely innocuous except for a couple of artefacts in
> the training database (words that ought to have been seen in many
> messages but actually showed up predominantly in spam).  Ironically,
> one of these was "fetchmail-5.9.14" -- I'd been adding a lot of spam
> to the corpus since that version came out :)  Shows how easy it is to
> bias this kind of classification; you really need to feed it similar
> numbers of similarly-dated messages in order to get good training, and
> shortcuts can be dangerous...

They sure can be -- many of us here learned this the hard way too.  It was
so bad for me at first (mixed-source data) that I ignored headers entirely;
but that forced us to work harder on other things, and that appears to have
given us some huge wins.

One thing that really helped me:  Our datadase has an integer "kill count"
attached to each word.  Whenever scoring a new message, the words that
survive to the end of the Graham scheme (the "most extreme" words in the
message) each get their killcount bumped by 1.  Whenever I've gotten great
results for bogus reasons, looking at the words with the highest killcounts
has instantly pinpointed the cause (for example, when I started tokenizing
"To" headers, "bruceg" became one of the two most frequent killer clues in
the whole database).  This kind of thing is easy for us to do so long as
we're doing research; I imagine Eric might balk at the additional database
burden in bogofilter, but he shouldn't <wink>.

The killcount is much less revealing under the S ("f(w)") scheme, btw -- it
tends then to reveal merely which words appear most often.


From gward@python.net  Wed Sep 25 18:41:23 2002
From: gward@python.net (Greg Ward)
Date: Wed, 25 Sep 2002 13:41:23 -0400
Subject: [Spambayes] The python.org email corpus
Message-ID: <20020925174123.GA9191@cthulhu.gerg.ca>

Hi all --

The seed of the python.org email corpus -- that is, the email gathered
from 2002-09-11 to 2002-09-18, sorted and cleaned up -- is now available
at http://mail.python.org/corpus/ .

But you need a username and password to access it.  And I'm not going to
give access to just anyone -- if I have met you in person, *or*
Guido/Barry/Tim say it's OK, then I'll give you the password.  Otherwise
not likely.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
All programmers are playwrights and all computers are lousy actors.

From tim.one@comcast.net  Wed Sep 25 19:53:10 2002
From: tim.one@comcast.net (Tim Peters)
Date: Wed, 25 Sep 2002 14:53:10 -0400
Subject: [Spambayes] Moving closer to Gary's ideal
In-Reply-To: <200209241500.g8OF0TK17719@indus.ins.cwi.nl>
Message-ID: <BIEJKCLHCIOIHAGOKOLHMEAHDLAA.tim.one@comcast.net>

FYI, there's a new display option best_cutoff_fp_weight.  The histogram
analysis code now finds the buckets that minimize

    best_cutoff_fp_weight * (# false positives) + (# false negatives)

By default it's 1 (minimize total # of misclassified msgs, favoring neither
kind of mistake).  If, e.g., you're happy to endure 100 false negatives to
save 1 false positive, set to 100.  Don't be surprised if your f-n rate
zooms, though!


From skip@pobox.com  Wed Sep 25 19:54:08 2002
From: skip@pobox.com (Skip Montanaro)
Date: Wed, 25 Sep 2002 13:54:08 -0500
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <20020925035548.GA12077@glacier.arctrix.com>
Message-ID: <15762.1616.879712.110668@12-248-11-90.client.attbi.com>


An now, coming in from faaaar out in right field we have Skip, the weirdo
with the wacky results:

    grahams -> fws
    -> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
    ...

    false positive percentages
        0.000  0.000  tied          
        0.000  0.500  lost  +(was 0)
        0.000  0.000  tied          
        0.000  0.000  tied          
        0.000  0.000  tied          
        0.000  0.000  tied          
        0.500  0.500  tied          
        0.000  0.000  tied          
        0.500  0.000  won   -100.00%
        0.000  0.000  tied          

    won   1 times
    tied  8 times
    lost  1 times

    total unique fp went from 2 to 2 tied          
    mean fp % went from 0.1 to 0.1 tied          

    false negative percentages
        9.000  10.500  lost   +16.67%
        11.000  14.000  lost   +27.27%
        10.000  12.000  lost   +20.00%
        7.000  8.500  lost   +21.43%
        14.000  19.500  lost   +39.29%
        12.000  12.000  tied          
        13.000  17.000  lost   +30.77%
        9.000  12.500  lost   +38.89%
        9.000  13.000  lost   +44.44%
        9.500  12.000  lost   +26.32%

    won   0 times
    tied  1 times
    lost  9 times

    total unique fn went from 207 to 262 lost   +26.57%
    mean fn % went from 10.35 to 13.1 lost   +26.57%

    ham mean                     ham sdev
       0.00   21.97 +(was 0)        0.00    6.82 +(was 0)
       0.58   23.40 +3934.48%        5.62    8.78  +56.23%
       0.00   22.83 +(was 0)        0.00    7.58 +(was 0)
       0.00   22.52 +(was 0)        0.00    7.65 +(was 0)
       0.00   22.96 +(was 0)        0.00    7.78 +(was 0)
       0.03   22.27 +74133.33%        0.42    7.70 +1733.33%
       0.50   22.79 +4458.00%        7.07    7.67   +8.49%
       0.00   22.77 +(was 0)        0.00    6.88 +(was 0)
       0.50   22.44 +4388.00%        7.07    7.61   +7.64%
       0.00   22.76 +(was 0)        0.00    7.13 +(was 0)

    ham mean and sdev for all runs
       0.16   22.67 +14068.75%        3.63    7.57 +108.54%

    spam mean                    spam sdev
      91.02   69.87  -23.24%       28.49   12.08  -57.60%
      89.95   67.92  -24.49%       29.71   12.32  -58.53%
      90.62   69.54  -23.26%       28.80   11.87  -58.78%
      93.70   71.21  -24.00%       23.90   10.83  -54.69%
      86.81   66.41  -23.50%       33.55   13.55  -59.61%
      88.40   69.92  -20.90%       31.84   12.23  -61.59%
      88.89   68.79  -22.61%       30.41   13.18  -56.66%
      91.15   69.36  -23.91%       28.24   12.28  -56.52%
      91.38   69.29  -24.17%       27.94   12.55  -55.08%
      90.73   70.44  -22.36%       28.83   12.34  -57.20%

    spam mean and sdev for all runs
      90.27   69.27  -23.26%       29.26   12.38  -57.69%

    ham/spam mean difference: 90.11 46.60 -43.51

Here are the overall graphs from the f(w) run:

    -> <stat> Ham scores for all runs: 2000 items; mean 22.67; sample sdev 7.57
    * = 5 items
      0.00   5 *
      2.50   6 **
      5.00  16 ****
      7.50  48 **********
     10.00  80 ****************
     12.50 121 *************************
     15.00 195 ***************************************
     17.50 277 ********************************************************
     20.00 278 ********************************************************
     22.50 260 ****************************************************
     25.00 257 ****************************************************
     27.50 168 **********************************
     30.00 125 *************************
     32.50  55 ***********
     35.00  40 ********
     37.50  20 ****
     40.00  18 ****
     42.50  12 ***
     45.00   7 **
     47.50   4 *
     50.00   1 *
     52.50   5 *
     55.00   0 
     57.50   0 
     60.00   2 *
     62.50   0 
     65.00   0 
     67.50   0 
     70.00   0 
     72.50   0 
     75.00   0 
     77.50   0 
     80.00   0 
     82.50   0 
     85.00   0 
     87.50   0 
     90.00   0 
     92.50   0 
     95.00   0 
     97.50   0 

    -> <stat> Spam scores for all runs: 2000 items; mean 69.27; sample sdev 12.38
    * = 3 items
      0.00   0 
      2.50   0 
      5.00   0 
      7.50   0 
     10.00   0 
     12.50   0 
     15.00   0 
     17.50   0 
     20.00   1 *
     22.50   0 
     25.00   1 *
     27.50   2 *
     30.00   1 *
     32.50   6 **
     35.00   4 **
     37.50  10 ****
     40.00  23 ********
     42.50  20 *******
     45.00  37 *************
     47.50  41 **************
     50.00  60 ********************
     52.50  56 *******************
     55.00  70 ************************
     57.50 107 ************************************
     60.00 124 ******************************************
     62.50 121 *****************************************
     65.00 151 ***************************************************
     67.50 163 *******************************************************
     70.00 166 ********************************************************
     72.50 169 *********************************************************
     75.00 139 ***********************************************
     77.50 126 ******************************************
     80.00 102 **********************************
     82.50  95 ********************************
     85.00  79 ***************************
     87.50  53 ******************
     90.00  46 ****************
     92.50  23 ********
     95.00   4 **
     97.50   0 

It looks to me like the problem (whatever it is) with my data is that it
causes the standard deviation to get quite large, though the mean spam score
seems to be a bit lower than other stuff I've seen.  What would cause that?
More variability in my spam?

Skip

From eppstein@ics.uci.edu  Wed Sep 25 19:55:21 2002
From: eppstein@ics.uci.edu (David Eppstein)
Date: Wed, 25 Sep 2002 11:55:21 -0700
Subject: [Spambayes] Re: Mailbox class in the spambayes project & python 2.2.1
References: <20020925132858.2880c885.Alexander@Leidinger.net>
	<200209251558.g8PFwaB02740@odiug.zope.com>
	<20020925183103.384cfc74.Alexander@Leidinger.net>
	<200209251651.g8PGpQj11912@odiug.zope.com>
Message-ID: <eppstein-AAFDF5.11552125092002@main.gmane.org>

In article <200209251651.g8PGpQj11912@odiug.zope.com>,
 Guido van Rossum <guido@python.org> wrote:

> I tried this:
> 
>   if isinstance(x, tuple):
>      assert len(x) == 3
>      x = unicode(x[2], x[0]).encode('ascii')
> 
> but this didn't work because while Python's encodings package knows
> 'ansi_x3.4_1968' and 'ansi_x3.4_1986' (sic) as aliases for 'ascii', it
> doesn't know 'ansi-x3-4-1968'. :-(

How about:

  if isinstance(x, tuple):
     assert len(x) == 3
     try:
        x = unicode(x[2], x[0]).encode('ascii')
     except:
        x = x[2]

(probably the except should be more careful about what exceptions it 
watches...)

-- 
David Eppstein       UC Irvine Dept. of Information & Computer Science
eppstein@ics.uci.edu http://www.ics.uci.edu/~eppstein/



From gward@python.net  Wed Sep 25 20:28:49 2002
From: gward@python.net (Greg Ward)
Date: Wed, 25 Sep 2002 15:28:49 -0400
Subject: [Spambayes] Anyone know Italian?
Message-ID: <20020925192849.GA10687@cthulhu.gerg.ca>

Trying to figure out if this is spam.  Any guesses?

"""
[...skip headers...]
Message-ID: <000801c25e75$f9a01e00$dca41c97@it>
From: "Coviello Giuseppe" <giuseppecoviello@libero.it>
To: <webmaster@python.org>
Subject: e-mail
Date: Tue, 17 Sep 2002 20:13:44 +0200
MIME-Version: 1.0
[...skip headers...]

[...skip MIME junk...]
vorrei informazioni riguardo la sintesi e la reattivita' degli =
ossirani,in special modo,pubblicazioni e ricerche del prof.RUGGERO =
CURCI.vi ringrazio sin d'ora.arrivederci
"""

-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
This quote intentionally left blank.

From gward@python.net  Wed Sep 25 20:31:22 2002
From: gward@python.net (Greg Ward)
Date: Wed, 25 Sep 2002 15:31:22 -0400
Subject: [Spambayes] How about German?
In-Reply-To: <20020925192849.GA10687@cthulhu.gerg.ca>
References: <20020925192849.GA10687@cthulhu.gerg.ca>
Message-ID: <20020925193122.GA10553@cthulhu.gerg.ca>

Another so-called false positive; I'm guessing this one belongs in the
dsn folder (bounces, delay notifications, and the like) -- it was sent
to mailman-users-admin and sure looks like an autoresponse.  Can anyone
confirm?

"""
Date: Wed, 18 Sep 2002 09:48:28 +0200 (MET DST)
Message-Id: <200209180748.JAA04388@tomahawk.inf.ethz.ch>
From: Frank.Thommen@inf.ethz.ch
Subject: Bin in den Ferien
Apparently-To: mailman-users-admin@python.org
Bcc:

---

Betrifft: Ihre Mail bezueglich 


Ich bin vom 6. bis am 23. September in den Ferien.

Bitte wenden Sie sich bei Fragen und Problemen bezueglich der IT-Infrastruktur am D-INFK an su
pport@inf.ethz.ch oder an die ISG-Telephonnummer (01) 63 27210.

Mit freundlichen Gruessen

     Frank Thommen
[...]
"""

-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Condense soup, not books!

From nas@python.ca  Wed Sep 25 20:41:52 2002
From: nas@python.ca (Neil Schemenauer)
Date: Wed, 25 Sep 2002 12:41:52 -0700
Subject: [Spambayes] Anyone know Italian?
In-Reply-To: <20020925192849.GA10687@cthulhu.gerg.ca>
References: <20020925192849.GA10687@cthulhu.gerg.ca>
Message-ID: <20020925194152.GA14414@glacier.arctrix.com>

Greg Ward wrote:
> Trying to figure out if this is spam.  Any guesses?

>From http://www.freetranslation.com/:

    I would want information I pertain the synthesis and the reattivit of
    the ossirani, in special manner, publications and search for the prof.
    US. I thank yourselves sin of now. goodbye

Awful translation but I think he's asking for some information and
thanking us in advance.

  Neil

From nas@python.ca  Wed Sep 25 20:44:35 2002
From: nas@python.ca (Neil Schemenauer)
Date: Wed, 25 Sep 2002 12:44:35 -0700
Subject: [Spambayes] How about German?
In-Reply-To: <20020925193122.GA10553@cthulhu.gerg.ca>
References: <20020925192849.GA10687@cthulhu.gerg.ca>
	<20020925193122.GA10553@cthulhu.gerg.ca>
Message-ID: <20020925194435.GB14414@glacier.arctrix.com>

Greg Ward wrote:
> Another so-called false positive; I'm guessing this one belongs in the
> dsn folder (bounces, delay notifications, and the like) -- it was sent
> to mailman-users-admin and sure looks like an autoresponse.  Can anyone
> confirm?

freetranslation.com does a little better on this one:

    Concerns: your mails regarding

    I am of that 6th until on the 23 September in the vacation.  

    Please you address in questions and problems regarding the
    IT-infrastructure to the D-INFK at su pport@inf.ethz.ch
    or at the ISG-telephone number (01) 63  27210.  

    With friendly greeting

    Frank Thommen

From guido@python.org  Wed Sep 25 20:46:35 2002
From: guido@python.org (Guido van Rossum)
Date: Wed, 25 Sep 2002 15:46:35 -0400
Subject: [Spambayes] How about German?
In-Reply-To: Your message of "Wed, 25 Sep 2002 15:31:22 EDT."
             <20020925193122.GA10553@cthulhu.gerg.ca> 
References: <20020925192849.GA10687@cthulhu.gerg.ca>  
            <20020925193122.GA10553@cthulhu.gerg.ca> 
Message-ID: <200209251946.g8PJkag23077@odiug.zope.com>

> Another so-called false positive; I'm guessing this one belongs in the
> dsn folder (bounces, delay notifications, and the like) -- it was sent
> to mailman-users-admin and sure looks like an autoresponse.  Can anyone
> confirm?
> 
> """
> Date: Wed, 18 Sep 2002 09:48:28 +0200 (MET DST)
> Message-Id: <200209180748.JAA04388@tomahawk.inf.ethz.ch>
> From: Frank.Thommen@inf.ethz.ch
> Subject: Bin in den Ferien
> Apparently-To: mailman-users-admin@python.org
> Bcc:
> 
> ---
> 
> Betrifft: Ihre Mail bezueglich 
> 
> 
> Ich bin vom 6. bis am 23. September in den Ferien.
> 
> Bitte wenden Sie sich bei Fragen und Problemen bezueglich der IT-Infrastruktur am D-INFK an su
> pport@inf.ethz.ch oder an die ISG-Telephonnummer (01) 63 27210.
> 
> Mit freundlichen Gruessen
> 
>      Frank Thommen
> [...]
> """

Yes, that's a German vacation message.

The other one was Italian but Babelfish didn't make enough sense out
of it to give me a clue.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From matt@mondoinfo.com  Wed Sep 25 20:48:00 2002
From: matt@mondoinfo.com (Matthew Dixon Cowles)
Date: Wed, 25 Sep 2002 14:48:00 -0500 (CDT)
Subject: [Spambayes] Anyone know Italian?
In-Reply-To: <20020925192849.GA10687@cthulhu.gerg.ca>
References: <20020925192849.GA10687@cthulhu.gerg.ca>
Message-ID: <1032982637.64.1387@sake.mondoinfo.com>

Dear Greg,

> Trying to figure out if this is spam.  Any guesses?

> vorrei informazioni riguardo la sintesi e la reattivita' degli
> ossirani,in special modo,pubblicazioni e ricerche del prof.RUGGERO
> CURCI.vi ringrazio sin d'ora.arrivederci

Looks like badly misdirected ham to me:

I'm looking for information regarding the synthesis and the
reactivity of [chemical term], especially publications and research
of professor Ruggero Curchi. Thanks in advance. See you later.

(Harder parts courtesy of a buddy of mine.)

Regards,
Matt


From tim.one@comcast.net  Wed Sep 25 22:22:06 2002
From: tim.one@comcast.net (Tim Peters)
Date: Wed, 25 Sep 2002 17:22:06 -0400
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <15762.1616.879712.110668@12-248-11-90.client.attbi.com>
Message-ID: <BIEJKCLHCIOIHAGOKOLHGEAMDLAA.tim.one@comcast.net>

[Skip Montanaro]
> An now, coming in from faaaar out in right field we have Skip, the weirdo
> with the wacky results:

Speaking of which, are you going to follow up on

http://mail.python.org/pipermail-21/spambayes/2002-September/000211.html

?

>     grahams -> fws
>     -> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
>     ...
>
>     false positive percentages
>         0.000  0.000  tied
>         0.000  0.500  lost  +(was 0)
>         0.000  0.000  tied
>         0.000  0.000  tied
>         0.000  0.000  tied
>         0.000  0.000  tied
>         0.500  0.500  tied
>         0.000  0.000  tied
>         0.500  0.000  won   -100.00%
>         0.000  0.000  tied
>
>     won   1 times
>     tied  8 times
>     lost  1 times
>
>     total unique fp went from 2 to 2 tied
>     mean fp % went from 0.1 to 0.1 tied

So 0.1% is very low with this little training data.

>     false negative percentages
>         9.000  10.500  lost   +16.67%
>         11.000  14.000  lost   +27.27%
>         10.000  12.000  lost   +20.00%
>         7.000  8.500  lost   +21.43%
>         14.000  19.500  lost   +39.29%
>         12.000  12.000  tied
>         13.000  17.000  lost   +30.77%
>         9.000  12.500  lost   +38.89%
>         9.000  13.000  lost   +44.44%
>         9.500  12.000  lost   +26.32%
>
>     won   0 times
>     tied  1 times
>     lost  9 times
>
>     total unique fn went from 207 to 262 lost   +26.57%
>     mean fn % went from 10.35 to 13.1 lost   +26.57%

And this is a supernaturally high increase.  When the f-p rate is
supernaturally low, and the f-n rate supernaturally high, the system is too
willing to call everything ham.  You histograms show that you have 56 false
negatives living in the 0.525-0.550 bucket, so dropping spam_cutoff to 0.525
would get your f-n rate near to where it was.  Yours is the only report so
far where dropping spam_cutoff would help; others have reported that
increasing it would help; I suggested the value that was optimal for my run.

Also read the parent msg in this thread for other things you should try (the
f(w) scheme hasn't been tuned -- tuning it for your data ia part of the job
here).

If you cvs up, the histogram pairs are now followed by an account of the
spam_cutoff rates that minimize fp+fn.  You can set another new option to
make that analysis look more kindly upon false negatives than false
positives (or vice versa).  I recommend that you also set nbuckets to 100 or
200 (the default is 40), as you've got much denser overlap than other people
are seeing, and a change of 0.025 in spam_cutoff for you in the overlap
range is going to reclassify 50-100 messages.  More buckets would allow
finer-grained analysis.

> ...
> Here are the overall graphs from the f(w) run:
>
>     -> <stat> Ham scores for all runs: 2000 items; mean 22.67;
> sample sdev 7.57
>     * = 5 items
>       0.00   5 *
>       2.50   6 **
>       5.00  16 ****
>       7.50  48 **********
>      10.00  80 ****************
>      12.50 121 *************************
>      15.00 195 ***************************************
>      17.50 277 ********************************************************
>      20.00 278 ********************************************************
>      22.50 260 ****************************************************
>      25.00 257 ****************************************************
>      27.50 168 **********************************
>      30.00 125 *************************
>      32.50  55 ***********
>      35.00  40 ********
>      37.50  20 ****
>      40.00  18 ****
>      42.50  12 ***
>      45.00   7 **
>      47.50   4 *
>      50.00   1 *
>      52.50   5 *
>      55.00   0
>      57.50   0
>      60.00   2 *
>      62.50   0
>      65.00   0
>      67.50   0
>      70.00   0
>      72.50   0
>      75.00   0
>      77.50   0
>      80.00   0
>      82.50   0
>      85.00   0
>      87.50   0
>      90.00   0
>      92.50   0
>      95.00   0
>      97.50   0
>
>     -> <stat> Spam scores for all runs: 2000 items; mean 69.27;
> sample sdev 12.38
>     * = 3 items
>       0.00   0
>       2.50   0
>       5.00   0
>       7.50   0
>      10.00   0
>      12.50   0
>      15.00   0
>      17.50   0
>      20.00   1 *
>      22.50   0
>      25.00   1 *
>      27.50   2 *
>      30.00   1 *
>      32.50   6 **
>      35.00   4 **
>      37.50  10 ****
>      40.00  23 ********
>      42.50  20 *******
>      45.00  37 *************
>      47.50  41 **************
>      50.00  60 ********************
>      52.50  56 *******************
>      55.00  70 ************************
>      57.50 107 ************************************
>      60.00 124 ******************************************
>      62.50 121 *****************************************
>      65.00 151 ***************************************************
>      67.50 163 *******************************************************
>      70.00 166 ********************************************************
>      72.50 169 *********************************************************
>      75.00 139 ***********************************************
>      77.50 126 ******************************************
>      80.00 102 **********************************
>      82.50  95 ********************************
>      85.00  79 ***************************
>      87.50  53 ******************
>      90.00  46 ****************
>      92.50  23 ********
>      95.00   4 **
>      97.50   0
>
> It looks to me like the problem (whatever it is) with my data is that it
> causes the standard deviation to get quite large, though the mean
> spam score seems to be a bit lower than other stuff I've seen.  What would
> cause that?  More variability in my spam?

Possibly, although that's hard to swallow since spam seems so much alike in
so many ways.  Why would the spam you get be more variable than the spam
everyone else gets?

Another possibility is that your ham contains popular spam words.  For
example, if you have significant ham that talks about diet and exercise,
Nigeria, or the size of assorted sexual organs, that would spread out your
spam scores.  As mentioned at the top, the way to get an experienced eyeball
to look at this needs to be followed up on.  You have an amazing percentage
of low-scoring spam!

Just for fun, try setting robinson_minimum_prob_strength to 0.4.


From tim.one@comcast.net  Thu Sep 26 00:21:15 2002
From: tim.one@comcast.net (Tim Peters)
Date: Wed, 25 Sep 2002 19:21:15 -0400
Subject: [Spambayes] Two Scheme Enter, One Scheme Leave.
In-Reply-To: <200209250809.g8P894H09766@localhost.localdomain>
Message-ID: <LNBBLJKPBEHFEDALKOLCKELJBGAB.tim.one@comcast.net>

[Anthony Baxter, with lots of good experiments]
> ...
> Summarising values tried for robinson_probability_a
> (using the 0.6 cutoff)
>
>      a      fp       fn    fp+fn
>     0.0     18      650     668

Heh -- I thought 0.0 would be a disaster.

>     0.001   13       36      49
>     0.01    13       28      41
>     0.025   12       24      36
>     0.05    11       23      34
>     0.075   10       21      31
>     0.1      9       21      30
>     0.125   10       21      31
>     0.15    10       21      31
>     0.2      9       22      31
>     0.25     9       22      31
>     0.35    10       21      31
>     0.45    10       22      32
>     0.5     11       21      33     (tim's default)
>     1.0     13       29      42
>     2.0     12       42      54
>     10.0    11       96      107

I'll explain the 0.0 thing here.  The lower a, the more the system
"believes" the probability estimates gotten from just counting how many
times each word appears in ham and spam, and no matter how few (but
non-zero) times it sees a word in the training data.  At a=0, it believes
them totally, even to the extent of giving words spamprobs of exactly 0 (if
a word has been seen in at least one ham but in no spam) and 1 (flip flop).

Probabilities of 0 and 1 are certainties, and the math is such that if a
prob 0 and prob 1 word both appear, the score comes out to exactly 0.5.  If
you look at your histograms, I bet you'll find a lot of both ham and spam in
the 0.5 bucket.  But since your cutoff is above 0.5, they're all called ham,
and that's a disaster for the f-n rate.  If you cutoff were 0.49999999
instead, it would have been a disaster for the f-p rate.

Cute:  with a=0.0, I had one spam with all these "certain spam" words:

prob('subject:loan') = 1
prob('subject:Find') = 1
prob('from:email addr:link2buy.com>') = 1
prob('subject:rates') = 1
prob('url:o') = 1
prob('url:e') = 1
prob('url:u') = 1
prob('originator.') = 1
prob('url:r') = 1
prob('url:training') = 1
prob('mailing.') = 1
prob('unsubscribe') = 1
prob('url:img') = 1
prob('url:esf') = 1
prob('refinance') = 1
prob('rates') = 1
prob('url:4') = 1
prob('cash') = 1
prob('loan') = 1
prob('email name:training') = 1
prob('url:link2buy') = 1
prob('homeowners') = 1
prob('header:Received:8') = 1
prob('url:101700546') = 1
prob('from:skip:e 10') = 1

OTOH, it had one "certain ham" word too:

prob('clearing') = 0

That was enough to score it exactly 0.5.

A moral of the story is that allowing certainty to creep into an inherently
uncertain scheme is certain to screw up.

In the other direction, the higher a is, the more it pushes computed
probabilities toward x (the "unknown word" spamprob).

Perhaps revealing:  at a=0.0, I had 665 spam that scored 0.5, but 1235 ham
that scored 0.5.  So nearly half of all messages (more than half of my ham,
+ about 1/3 of my spam) contained at least one word that had only appeared
in the same kind of message in the training data, *and* at least one word
that had only appeared in the other kind of message.  That's a pretty
dramatic statement of just how flaky the probability estimates obtained from
counting alone can be.


From tim.one@comcast.net  Thu Sep 26 00:29:47 2002
From: tim.one@comcast.net (Tim Peters)
Date: Wed, 25 Sep 2002 19:29:47 -0400
Subject: [Spambayes] Re: Mailbox class in the spambayes project & python
 2.2.1
In-Reply-To: <eppstein-AAFDF5.11552125092002@main.gmane.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCCELLBGAB.tim.one@comcast.net>

[Guido]
> I tried this:
>   if isinstance(x, tuple):
>      assert len(x) == 3
>      x = unicode(x[2], x[0]).encode('ascii')
>
> but this didn't work because while Python's encodings package knows
> 'ansi_x3.4_1968' and 'ansi_x3.4_1986' (sic) as aliases for 'ascii', it
> doesn't know 'ansi-x3-4-1968'. :-(

[David Eppstein]
> How about:
>
>   if isinstance(x, tuple):
>      assert len(x) == 3
>      try:
>         x = unicode(x[2], x[0]).encode('ascii')
>      except:
>         x = x[2]
>
> (probably the except should be more careful about what exceptions it
> watches...)

Goodness, you fellows are working hard!  Stupid beats smart almost all the
time here; the first thing I would have tried was

    for piece in repr(x).split():
        yield 'charset:' + piece

whether it's a tuple or a string or whatever.  If there's a goofy encoding
spec in the charset parameter, that may be a good clue too!  We've seen many
times that spammers share software with peculiar ways of spelling things;
for example, Mime-Version is vanilla, but MiME-Version is a killer-strong
spam indicator.


From tim.one@comcast.net  Thu Sep 26 00:45:54 2002
From: tim.one@comcast.net (Tim Peters)
Date: Wed, 25 Sep 2002 19:45:54 -0400
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <LNBBLJKPBEHFEDALKOLCAEJEBGAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCKELMBGAB.tim.one@comcast.net>

Inspired by Neil and Anthony, I tried

robinson_probability_a: 0.225

in my test case.  It made no difference to results, but separated the ham
and spam means at the cost of increasing their variances (a=0.5 vs a=0.225
here):


ham mean                     ham sdev
  33.01   32.00   -3.06%        6.26    6.39   +2.08%
  32.19   31.14   -3.26%        5.38    5.51   +2.42%
  32.99   31.96   -3.12%        5.60    5.71   +1.96%
  33.46   32.46   -2.99%        5.77    5.90   +2.25%
  33.16   32.20   -2.90%        5.56    5.74   +3.24%
  32.81   31.85   -2.93%        5.72    5.81   +1.57%
  33.38   32.36   -3.06%        5.76    5.89   +2.26%
  32.55   31.51   -3.20%        5.70    5.91   +3.68%
  33.11   32.14   -2.93%        5.52    5.63   +1.99%
  34.21   33.22   -2.89%        5.84    5.95   +1.88%

ham mean and sdev for all runs
  33.09   32.09   -3.02%        5.73    5.87   +2.44%

spam mean                    spam sdev
  82.95   83.89   +1.13%        6.82    7.16   +4.99%
  82.17   83.11   +1.14%        6.34    6.85   +8.04%
  82.06   83.08   +1.24%        6.14    6.58   +7.17%
  82.39   83.20   +0.98%        5.93    6.31   +6.41%
  82.53   83.31   +0.95%        7.00    7.33   +4.71%
  82.76   83.76   +1.21%        6.56    7.01   +6.86%
  82.06   82.97   +1.11%        5.73    6.02   +5.06%
  82.26   83.09   +1.01%        5.97    6.17   +3.35%
  82.65   83.55   +1.09%        6.71    7.07   +5.37%
  83.43   84.37   +1.13%        6.37    6.71   +5.34%

spam mean and sdev for all runs
  82.53   83.43   +1.09%        6.37    6.75   +5.97%

ham/spam mean difference: 49.44 51.34 +1.90

I can live with that.

In contrast, Skip reported a ham mean of 22.67 (much lower) with sdev 7.57
(higher), and a spam mean of 69.27 (much lower) with sdev 12.38 (much
higher).  His spam and his ham are hammier, but less certain of themselves.
I expect talk therapy would do them some good <wink>.


From tim.one@comcast.net  Thu Sep 26 01:04:31 2002
From: tim.one@comcast.net (Tim Peters)
Date: Wed, 25 Sep 2002 20:04:31 -0400
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKELMBGAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCKELOBGAB.tim.one@comcast.net>

Here's an interesting experiment:  max_discriminators=1.  That is, only look
at the single strongest clue in a message.  Unsurprisingly, this gives a
very Graham-like bipolar distribution.  But it does surprisingly well for me
(max_discriminators 150 vs 1):

false positive percentages
    0.500  1.500  lost  +200.00%
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.500  lost  +(was 0)
    0.000  0.500  lost  +(was 0)
    0.000  0.000  tied
    0.000  0.500  lost  +(was 0)
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.500  lost  +(was 0)

won   0 times
tied  5 times
lost  5 times

total unique fp went from 1 to 7 lost  +600.00%
mean fp % went from 0.05 to 0.35 lost  +600.00%

false negative percentages
    0.000  0.000  tied
    0.000  0.500  lost  +(was 0)
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  1.000  lost  +(was 0)
    0.000  0.000  tied
    0.000  0.500  lost  +(was 0)
    0.000  0.000  tied

won   0 times
tied  7 times
lost  3 times

total unique fn went from 0 to 4 lost  +(was 0)
mean fn % went from 0.0 to 0.2 lost  +(was 0)

ham mean                     ham sdev
  33.01    1.57  -95.24%        6.26   12.13  +93.77%
  32.19    0.05  -99.84%        5.38    0.16  -97.03%
  32.99    0.04  -99.88%        5.60    0.11  -98.04%
  33.46    0.54  -98.39%        5.77    7.03  +21.84%
  33.16    0.57  -98.28%        5.56    7.02  +26.26%
  32.81    0.06  -99.82%        5.72    0.15  -97.38%
  33.38    0.55  -98.35%        5.76    7.02  +21.87%
  32.55    0.07  -99.78%        5.70    0.35  -93.86%
  33.11    0.07  -99.79%        5.52    0.25  -95.47%
  34.21    0.55  -98.39%        5.84    7.01  +20.03%

ham mean and sdev for all runs
  33.09    0.41  -98.76%        5.73    5.89   +2.79%

spam mean                    spam sdev
  82.95   99.90  +20.43%        6.82    0.15  -97.80%
  82.17   99.36  +20.92%        6.34    7.04  +11.04%
  82.06   99.88  +21.72%        6.14    0.28  -95.44%
  82.39   99.91  +21.26%        5.93    0.10  -98.31%
  82.53   99.89  +21.03%        7.00    0.14  -98.00%
  82.76   99.91  +20.72%        6.56    0.17  -97.41%
  82.06   98.91  +20.53%        5.73    9.82  +71.38%
  82.26   99.87  +21.41%        5.97    0.28  -95.31%
  82.65   99.38  +20.24%        6.71    6.60   -1.64%
  83.43   99.88  +19.72%        6.37    0.32  -94.98%

spam mean and sdev for all runs
  82.53   99.69  +20.79%        6.37    4.37  -31.40%

ham/spam mean difference: 49.44 99.28 +49.84

The wild swings across runs in the ham and spam sdevs suggest it's not a
very stable approach <heh>.



From tim.one@comcast.net  Thu Sep 26 02:38:45 2002
From: tim.one@comcast.net (Tim Peters)
Date: Wed, 25 Sep 2002 21:38:45 -0400
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <BIEJKCLHCIOIHAGOKOLHGEAMDLAA.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCMEMDBGAB.tim.one@comcast.net>

One more run, starting over from scratch with a new seed, but setting

robinson_probability_a: 0.225

Graham vs that:

grahams -> fofwbases
-> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
   ...

false positive percentages
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.500  0.500  tied
    0.000  0.500  lost  +(was 0)
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.500  0.500  tied

won   0 times
tied  9 times
lost  1 times

total unique fp went from 2 to 3 lost   +50.00%
mean fp % went from 0.1 to 0.15 lost   +50.00%

false negative percentages
    0.500  0.000  won   -100.00%
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    1.000  0.000  won   -100.00%
    0.500  0.000  won   -100.00%

won   3 times
tied  7 times
lost  0 times

total unique fn went from 4 to 0 won   -100.00%
mean fn % went from 0.2 to 0.0 won   -100.00%

Boosting cutoff to 0.56 (it was 0.55) would have been a pure win, chopping
one fp.  Boosting it to 0.585 would have lost another fp, but gained 3 fn.
That would have left it with 1 fp and 3 fn, vs 2 and 4 for Graham.

It's not really fair to keep fiddling spam_cutoff after the fact, so it
remains disturbing that "one size fits all" doesn't here.  OTOH, the range
across which people fiddle spam_cutoff spans about 0.10, and relatively few
messages live there, so if you believe in middle grounds, this has a useful
one.

Does anyone else intend to participate in this death match?  If not, I'll
tally results tomorrow.  My general *impression* (subject to change) is that
Gary's scheme is doing at least as well, and often slightly better.  At
least two people have said (offline) that they're more comfortable with the
false positive mistakes it makes -- that they make "more sense" than the
ones the Graham scheme produces.  I haven't experienced that in this
particular test; e.g., the two fp under Graham in the run above were also fp
under Gary's scheme.  In the first run here I reported, both had a single
fp, and again it was the same one.  It seems that very brief msgs are hard
to score for any scheme (but my hope remains that mining more header lines
will help that -- we may even want to *presume* that a very brief msg is
ham, in the absence of strong evidence to the contrary?).


From tim.one@comcast.net  Thu Sep 26 03:01:10 2002
From: tim.one@comcast.net (Tim Peters)
Date: Wed, 25 Sep 2002 22:01:10 -0400
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <200209250250.g8P2oiU26801@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEMEBGAB.tim.one@comcast.net>

[Guido, tries Gary's scheme with max_discrimators at 150, and then at 15]
> ...
> Ah, and here are the results for md=15 (left == md=150, right == md=15):
>
> false positive percentages
>     0.500  0.000  won   -100.00%
>     0.000  0.000  tied
>     0.000  0.500  lost  +(was 0)
>     0.000  1.000  lost  +(was 0)
>     0.500  1.000  lost  +100.00%
>     0.500  0.500  tied
>     0.000  0.500  lost  +(was 0)
>     0.500  1.500  lost  +200.00%
>     0.000  0.000  tied
>     1.000  1.000  tied
>
> won   1 times
> tied  4 times
> lost  5 times
>
> total unique fp went from 6 to 12 lost  +100.00%
> mean fp % went from 0.3 to 0.6 lost  +100.00%
>
> false negative percentages
>     0.500  0.000  won   -100.00%
>     0.500  0.500  tied
>     0.500  0.500  tied
>     1.000  0.500  won    -50.00%
>     1.000  0.000  won   -100.00%
>     1.000  1.000  tied
>     1.000  0.500  won    -50.00%
>     0.000  0.000  tied
>     0.500  0.500  tied
>     2.000  1.500  won    -25.00%
>
> won   5 times
> tied  5 times
> lost  0 times
>
> total unique fn went from 16 to 10 won    -37.50%
> mean fn % went from 0.8 to 0.5 won    -37.50%
>
> The histograms look totally different here though!

That part isn't surprising -- this is making it look at only two handfuls of
*the* most extreme words in a msg, just like the Graham scheme does.  "Only
extremes in, only extremes out" applies here too, although not as viciously
under Gary's combining scheme as under Graham's.

> -> <stat> Ham scores for all runs: 2000 items; mean 11.01; sample
> sdev 15.30
> * = 21 items
>   0.00 1201 **********************************************************
>   2.50   53 ***
>   5.00    9 *
>   7.50    7 *
>  10.00    2 *
>  12.50   29 **
>  15.00   87 *****
>  17.50  100 *****
>  20.00   67 ****
>  22.50   48 ***
>  25.00   54 ***
>  27.50   54 ***
>  30.00   41 **
>  32.50   49 ***
>  35.00   46 ***
>  37.50   29 **
>  40.00   23 **
>  42.50   30 **
>  45.00   19 *
>  47.50   16 *
>  50.00    9 *
>  52.50    7 *
>  55.00    2 *
>  57.50    6 *
>  60.00    5 *
>  62.50    1 *
>  65.00    1 *
>  67.50    1 *
>  70.00    0
>  72.50    0
>  75.00    0
>  77.50    2 *
>  80.00    0
>  82.50    0
>  85.00    1 *
>  87.50    0
>  90.00    0
>  92.50    0
>  95.00    1 *
>  97.50    0
>
> -> <stat> Spam scores for all runs: 2000 items; mean 95.00;
> sample sdev 7.86
> * = 21 items
> [...]
>  47.50    1 *
>  50.00    1 *
>  52.50    1 *
>  55.00    3 *
>  57.50    4 *
>  60.00    8 *
>  62.50    9 *
>  65.00   11 *
>  67.50   15 *
>  70.00   13 *
>  72.50   15 *
>  75.00   23 **
>  77.50   51 ***
>  80.00   65 ****
>  82.50   41 **
>  85.00    2 *
>  87.50    9 *
>  90.00   12 *
>  92.50   66 ****
>  95.00  425 *********************
>  97.50 1225 ***********************************************************
>
> Note the hams scoring all the way in the 90s.

We see that under Graham's scheme too (else there would never be false
positives there -- we use spam_cutoff 0.90 there); extremes in, etc.

> There are no spams here!

I didn't catch the meaning.  You had 1728 (12+66+425+1225) spam "scoring all
the way in the 90s", so "here" must refer to something else?


From gward@python.net  Thu Sep 26 03:27:40 2002
From: gward@python.net (Greg Ward)
Date: Wed, 25 Sep 2002 22:27:40 -0400
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <LNBBLJKPBEHFEDALKOLCMEMDBGAB.tim.one@comcast.net>
References: <BIEJKCLHCIOIHAGOKOLHGEAMDLAA.tim.one@comcast.net>
	<LNBBLJKPBEHFEDALKOLCMEMDBGAB.tim.one@comcast.net>
Message-ID: <20020926022740.GA24111@cthulhu.gerg.ca>

On 25 September 2002, Tim Peters said:
> Does anyone else intend to participate in this death match?

Yes, I've been running tests all afternoon and evening.  Vague,
hand-wavey results:

  * my histograms are not terribly normal -- not as weird as Guido's,
    but not nearly as nice as Tim's
  * I think my peaks are better separated though -- there's a pretty
    wide range for spam_cutoff
  * I'm one of the few who seems to win by setting set spam_cutoff < 0.5

Oh, my corpus is the python.org Sep 2002 harvest + all spam sent to
gward@python.net from Feb 2002 to Aug 2002 and caught by SpamAssassin +
everything sitting in my personal inboxes at around noon today.  The
stuff from my inboxes was cleaned of "Received" headers that are clear
artifacts of the various ISPs I have used over the 2 years that stuff
has been piling up in those inboxes.

Here's the bottom line for Graham vs. Robinson f(w):

  total unique fp went from 6 to 4 won    -33.33%
  mean fp % went from 0.3 to 0.2 won    -33.33%
  total unique fn went from 25 to 31 lost   +24.00%
  mean fn % went from 1.25 to 1.55 lost   +24.00%

I'll post more complete results and a description of my corpus in the
morning.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
If you can read this, thank a programmer.

From anthony@interlink.com.au  Thu Sep 26 00:43:00 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Thu, 26 Sep 2002 09:43:00 +1000
Subject: [Spambayes] Anyone know Italian? 
In-Reply-To: <20020925192849.GA10687@cthulhu.gerg.ca> 
Message-ID: <200209252343.g8PNh0U08144@localhost.localdomain>

\
>>> Greg Ward wrote
> Trying to figure out if this is spam.  Any guesses?
> 
> """
> [...skip headers...]
> Message-ID: <000801c25e75$f9a01e00$dca41c97@it>
> From: "Coviello Giuseppe" <giuseppecoviello@libero.it>
> To: <webmaster@python.org>
> Subject: e-mail
> Date: Tue, 17 Sep 2002 20:13:44 +0200
> MIME-Version: 1.0
> [...skip headers...]
> 
> [...skip MIME junk...]
> vorrei informazioni riguardo la sintesi e la reattivita' degli =
> ossirani,in special modo,pubblicazioni e ricerche del prof.RUGGERO =
> CURCI.vi ringrazio sin d'ora.arrivederci
> """

babelfish sez:

 I would want information care the synthesis and the reattivita' of 
 the ossirani, in special way, banns and searches of prof.RUGGERO the 
 CURCI.vi ringrazio sin of ora.arrivederci

Which, I think, is pretty clear <wink>

Anthony
-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.


From anthony@interlink.com.au  Thu Sep 26 00:46:10 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Thu, 26 Sep 2002 09:46:10 +1000
Subject: [Spambayes] Re: There Can Be Only One 
In-Reply-To: <BIEJKCLHCIOIHAGOKOLHMEADDLAA.tim.one@comcast.net> 
Message-ID: <200209252346.g8PNkAi08164@localhost.localdomain>


>>> Tim Peters wrote
> 
> That's likely to generate at least a dozen strong ham tokens, and it's going
> to be very hard to overcome the hamminess of that many clues (perhaps even
> impossible under the max_discriminators=16 Graham scheme).

Pretty much, yeah.

> Does enabling mine_received_headers alone also hurt you?

Yep. I'm thinking about finding my older received processing code from
where-ever it ended up - all it pulled out were hostnames and IP addresses.

> > More on that later - I'm currently processing the scheme-deathmatch.
> And doing a fine job too -- thank you.

Bah. The laptop's the one doing the work.

Anthony

-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.


From tim.one@comcast.net  Thu Sep 26 03:45:30 2002
From: tim.one@comcast.net (Tim Peters)
Date: Wed, 25 Sep 2002 22:45:30 -0400
Subject: [Spambayes] memory consumption...
In-Reply-To: <BIEJKCLHCIOIHAGOKOLHCEOMDKAA.tim@zope.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCEEMHBGAB.tim.one@comcast.net>

[Anthony Baxter]
> For my standard test, using the graham scheme, memory
> consumption's blown out considerably in the last couple of days.
> timcv -n 10 used to consume about 50M - it's now about 75M.
> ...

You didn't follow up with more info (unless I missed it -- sorry if so!), so
I can just tell you what I'm seeing on my "large test", which is running
right now.  It trains on 18,000 ham and 12,600 spam per run; this
understates the burden though, since after predicting on a set it also
*trains* on it again, so the peak message burden in the dict is 20,000 ham
and 14,000 spam, at the end of each run.  I'm using CVS Python, and this
customization file:

"""
[Classifier]
use_robinson_combining: True
use_robinson_probability: True
robinson_probability_x: 0.5
robinson_probability_a: 0.225
max_discriminators: 150
robinson_minimum_prob_strength: 0.1

[TestDriver]
spam_cutoff: 0.570
nbuckets: 200
best_cutoff_fp_weight: 10
"""

The process size is holding rock steady, at a bit under 50MB.  Each time it
unlearns a set (i.e., at the start of each run), the disk goes crazy, while
it reads up the 3400 files containing the msgs to forget.  It reads them
again while predicting the set, but the disk is quiet then.  It reads them
yet a third time while training on the set again, and the disk is quiet then
too.

I didn't find it plausible that a 75MB process would cause swapping to disk
on a 256MB box (unless you're running an OS much feebler than Win98 <wink>),
but it was very plausible that it's going to disk to read thousands of
files.

BTW, the scoring scheme being used shouldn't have a significant effect on
process size; the wordinfo dict is overwhelmingly "it".  For purposes of
testing, I think it's certain you don't want a database instead:  we touch
every element of the dict at least twice per run (updating all the
probabilities after forgetting a set, and again after retraining on it).


From tim.one@comcast.net  Thu Sep 26 04:22:16 2002
From: tim.one@comcast.net (Tim Peters)
Date: Wed, 25 Sep 2002 23:22:16 -0400
Subject: [Spambayes] Re: There Can Be Only One
In-Reply-To: <200209252346.g8PNkAi08164@localhost.localdomain>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEMJBGAB.tim.one@comcast.net>

[Tim]
>> Does enabling mine_received_headers alone also hurt you?

[Anthony Baxter]
> Yep. I'm thinking about finding my older received processing code from
> where-ever it ended up - all it pulled out were hostnames and IP
> addresses.

Be sure to read the existing code before you bother:  unless I'm horribly
mistaken, that's all Neil's mine_received_headers pulls out:

received_host_re = re.compile(r'from (\S+)\s')
received_ip_re = re.compile(r'\s[[(]((\d{1,3}\.?){4})[\])]')

def breakdown_host(host):
    parts = host.split('.')
    for i in range(1, len(parts) + 1):
        yield '.'.join(parts[-i:])

def breakdown_ipaddr(ipaddr):
    parts = ipaddr.split('.')
    for i in range(1, 5):
        yield '.'.join(parts[:i])

if options.mine_received_headers:
    for header in msg.get_all("received", ()):
        for pat, breakdown in [(received_host_re, breakdown_host),
                               (received_ip_re, breakdown_ipaddr)]:
            m = pat.search(header)
            if m:
                for tok in breakdown(m.group(1).lower()):
                    yield 'received:' + tok

So it's tokenizing hosts and IPs, and suffixes of hosts, and prefixes of
IPs, and that's it.


From mhammond@skippinet.com.au  Thu Sep 26 05:19:08 2002
From: mhammond@skippinet.com.au (Mark Hammond)
Date: Thu, 26 Sep 2002 14:19:08 +1000
Subject: [Spambayes] Practical applications 
In-Reply-To: <200209250650.g8P6odp08869@localhost.localdomain>
Message-ID: <LCEPIIGDJPKCOIHOBJEPCEDPGKAA.mhammond@skippinet.com.au>

> Something similar came up on Python-list, so I followed up with Mark.
> Here's a summary:

I followed up too :)

I have a working Outlook Addin in place.  This used the generic "Office
Plugin" architecture, so should work with all of Office 2000 (but outlook is
the target).  I assume it will work with Outlook 2002/XP, I'm not sure.  I
doubt it will work with Office 98, but again I am not sure.  It will not
work with Outlook Express.

There are some bugs/incompleteness in the new universal gateway code I am
fixing, so this will rely on yet-to-be-released win32all versions.

I'll keep you updated.

Mark.


From tim.one@comcast.net  Thu Sep 26 05:19:36 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 26 Sep 2002 00:19:36 -0400
Subject: [Spambayes] Re: There Can Be Only One
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCEMJBGAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCAEMMBGAB.tim.one@comcast.net>

I did one last run on my full large (c.l.py vs bruceg_spam) test.  The
options are as I originally suggested for the f(w) test, except that I've
boosted "a" to 0.57, and increased the # of buckets (that has no effect on
scores) to allow pickier distinctions to get drawn.

"""
[Classifier]
use_robinson_combining: True
use_robinson_probability: True
robinson_probability_x: 0.5
robinson_probability_a: 0.225
max_discriminators: 150
robinson_minimum_prob_strength: 0.1

[TestDriver]
spam_cutoff: 0.570
nbuckets: 200
best_cutoff_fp_weight: 10
"""

Graham vs f(w):

-> <stat> tested 2000 hams & 1400 spams against 18000 hams & 12600 spams
   ...

false positive percentages
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.050  0.050  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.050  0.050  tied
    0.000  0.000  tied
    0.100  0.050  won    -50.00%

won   1 times
tied  9 times
lost  0 times

total unique fp went from 4 to 3 won    -25.00%
mean fp % went from 0.02 to 0.015 won    -25.00%

false negative percentages
    0.214  0.071  won    -66.82%
    0.286  0.071  won    -75.17%
    0.000  0.000  tied
    0.143  0.143  tied
    0.143  0.143  tied
    0.286  0.143  won    -50.00%
    0.143  0.143  tied
    0.143  0.214  lost   +49.65%
    0.286  0.214  won    -25.17%
    0.071  0.000  won   -100.00%

won   5 times
tied  4 times
lost  1 times

total unique fn went from 24 to 16 won    -33.33%
mean fn % went from 0.171428571428 to 0.114285714286 won    -33.33%


The histogram code told me:

-> best cutoff for all runs: 0.58
->     with weighted total 10*2 fp + 18 fn = 38
->     fp rate 0.01%  fn rate 0.129%

So boosting spam_cutoff to 0.58 would have saved 1 fp and added 2 fn, for
2 fp + 18 fn vs 4 fp + 24 fn for Graham.


For the obsessed <wink>:

-> <stat> Ham scores for all runs: 20000 items; mean 28.57; sdev 5.65
* = 14 items
...
  3.00   1 *
  3.50   1 *
  4.00   1 *
  4.50   1 *
  5.00   0
  5.50   1 *
  6.00   1 *
  6.50   0
  7.00   3 *
  7.50   3 *
  8.00   0
  8.50   2 *
  9.00   4 *
  9.50   2 *
 10.00   8 *
 10.50   6 *
 11.00   9 *
 11.50  11 *
 12.00  21 **
 12.50  12 *
 13.00  17 **
 13.50  26 **
 14.00  26 **
 14.50  27 **
 15.00  37 ***
 15.50  63 *****
 16.00  54 ****
 16.50  78 ******
 17.00  84 ******
 17.50 119 *********
 18.00 105 ********
 18.50 121 *********
 19.00 147 ***********
 19.50 191 **************
 20.00 193 **************
 20.50 253 *******************
 21.00 276 ********************
 21.50 321 ***********************
 22.00 323 ************************
 22.50 359 **************************
 23.00 444 ********************************
 23.50 551 ****************************************
 24.00 537 ***************************************
 24.50 547 ****************************************
 25.00 616 ********************************************
 25.50 672 ************************************************
 26.00 716 ****************************************************
 26.50 714 ***************************************************
 27.00 782 ********************************************************
 27.50 776 ********************************************************
 28.00 798 *********************************************************
 28.50 779 ********************************************************
 29.00 761 *******************************************************
 29.50 697 **************************************************
 30.00 718 ****************************************************
 30.50 705 ***************************************************
 31.00 623 *********************************************
 31.50 672 ************************************************
 32.00 532 **************************************
 32.50 503 ************************************
 33.00 480 ***********************************
 33.50 422 *******************************
 34.00 382 ****************************
 34.50 347 *************************
 35.00 337 *************************
 35.50 246 ******************
 36.00 242 ******************
 36.50 187 **************
 37.00 174 *************
 37.50 156 ************
 38.00 144 ***********
 38.50 116 *********
 39.00 107 ********
 39.50  75 ******
 40.00  82 ******
 40.50  57 *****
 41.00  52 ****
 41.50  45 ****
 42.00  34 ***
 42.50  38 ***
 43.00  27 **
 43.50  19 **
 44.00  33 ***
 44.50  30 ***
 45.00  19 **
 45.50  13 *
 46.00   8 *
 46.50  17 **
 47.00  10 *
 47.50   4 *
 48.00   6 *
 48.50   6 *
 49.00   3 *
 49.50   8 *
 50.00   2 *
 50.50   3 *
 51.00   1 *
 51.50   3 *
 52.00   1 *
 52.50   3 *
 53.00   0
 53.50   2 *
 54.00   1 *
 54.50   1 *
 55.00   2 *
 55.50   0
 56.00   1 *
 56.50   1 *
---------------------- my cutoff
 57.00   0
 57.50   1 *
---------------------- best cutoff at best_cutoff_fp_weight=10
 58.00   0
 58.50   0
 59.00   0
 59.50   0
 60.00   0
 60.50   0
 61.00   0
 61.50   0
 62.00   1 *
 62.50   0
 63.00   0
 63.50   0
 64.00   0
 64.50   0
 65.00   0
 65.50   0
 66.00   0
 66.50   0
 67.00   0
 67.50   0
 68.00   1 *

-> <stat> Spam scores for all runs: 14000 items; mean 85.02; sdev 6.72
* = 9 items
 41.50   1 *
 42.00   0
 42.50   0
 43.00   0
 43.50   0
 44.00   0
 44.50   0
 45.00   0
 45.50   0
 46.00   1 *
 46.50   0
 47.00   0
 47.50   0
 48.00   0
 48.50   0
 49.00   0
 49.50   1 *
 50.00   0
 50.50   0
 51.00   0
 51.50   0
 52.00   0
 52.50   1 *
 53.00   1 *
 53.50   0
 54.00   0
 54.50   3 *
 55.00   2 *
 55.50   3 *
 56.00   1 *
 56.50   2 *
---------------------- my cutoff
 57.00   1 *
 57.50   1 *
---------------------- best cutoff at best_cutoff_fp_weight=10
 58.00   1 *
 58.50   0
 59.00   3 *
 59.50   1 *
 60.00   6 *
 60.50   3 *
 61.00   8 *
 61.50   3 *
 62.00   4 *
 62.50   4 *
 63.00   6 *
 63.50   7 *
 64.00   7 *
 64.50   8 *
 65.00  20 ***
 65.50  10 **
 66.00  10 **
 66.50  28 ****
 67.00  12 **
 67.50  16 **
 68.00  24 ***
 68.50  23 ***
 69.00  24 ***
 69.50  41 *****
 70.00  44 *****
 70.50  47 ******
 71.00  50 ******
 71.50  39 *****
 72.00  60 *******
 72.50  66 ********
 73.00  63 *******
 73.50  70 ********
 74.00  84 **********
 74.50  99 ***********
 75.00 114 *************
 75.50 101 ************
 76.00 125 **************
 76.50 126 **************
 77.00 150 *****************
 77.50 154 ******************
 78.00 204 ***********************
 78.50 218 *************************
 79.00 236 ***************************
 79.50 264 ******************************
 80.00 330 *************************************
 80.50 403 *********************************************
 81.00 394 ********************************************
 81.50 422 ***********************************************
 82.00 451 ***************************************************
 82.50 521 **********************************************************
 83.00 464 ****************************************************
 83.50 512 *********************************************************
 84.00 536 ************************************************************
 84.50 495 *******************************************************
 85.00 478 ******************************************************
 85.50 442 **************************************************
 86.00 469 *****************************************************
 86.50 439 *************************************************
 87.00 416 ***********************************************
 87.50 430 ************************************************
 88.00 405 *********************************************
 88.50 455 ***************************************************
 89.00 357 ****************************************
 89.50 254 *****************************
 90.00 239 ***************************
 90.50 216 ************************
 91.00 184 *********************
 91.50 138 ****************
 92.00 187 *********************
 92.50 134 ***************
 93.00  82 **********
 93.50 115 *************
 94.00 127 ***************
 94.50 117 *************
 95.00 115 *************
 95.50  97 ***********
 96.00  64 ********
 96.50  75 *********
 97.00  83 **********
 97.50 167 *******************
 98.00 120 **************
 98.50 108 ************
 99.00 188 *********************
 99.50 170 *******************
-> best cutoff for all runs: 0.58
->     with weighted total 10*2 fp + 18 fn = 38
->     fp rate 0.01%  fn rate 0.129%


The fp and fn remaining are tough (like that wasn't obvious from their
number <wink>).

You may recall the "Hello, my Name is BlackIntrepid" spam, which, under the
Graham scheme, didn't have a single word among the 16 most extreme with
spamprob even as high as 0.05(!).  It's still too hard for f(w) to crack,
but it scores 0.4997 instead of 1.0e-29 now.  Indeed, all the false
negatives score above 0.41 now.

The two tough fps are still the Nigerian scam quote, and the lady with the
brief question and the obnoxious long SIG.  The fp on the edge lost because
my HTML-stipping code didn't remove a very long HTML comment whose body
contained a bunch of spammish stuff.  Since there are no other fps, this
section ends here <wink>.

The most embarrassing fn remaining is probably this one:  it's been a
persistent fn under all schemes, and is ragingly obvious spam to the human
judge:

"""
Data/Spam/Set6/11936.txt
prob = 0.553481764018
prob('these?') = 0.0180723
prob('"set') = 0.0215311
prob('tho') = 0.0238095
prob('oh,') = 0.0269842
prob('tangible') = 0.0302013
prob('(but') = 0.0447642
prob('(so') = 0.0452536
prob("'em") = 0.0488866
prob("i'd") = 0.0504009
prob('"sell"') = 0.0505618
prob('anybody') = 0.0581792
prob('worthwhile,') = 0.0652174
prob('itself"') = 0.0652174
prob('(it') = 0.0657555
prob('obviously') = 0.0741598
prob('quite') = 0.0809572
prob('"systems"') = 0.0918367
prob('far),') = 0.0918367
prob('heck?') = 0.0918367
prob("i've") = 0.107173
prob('probably') = 0.110252
prob('slower') = 0.114656
prob('"real"') = 0.141942
prob('looked') = 0.151258
prob('lots)') = 0.155172
prob('change"') = 0.155172
prob('"pocket') = 0.155172
prob('first):') = 0.155172
prob('deleting,') = 0.155172
prob('subject:heck') = 0.155172
prob('(using') = 0.157513
prob("doesn't") = 0.166024
prob("they'd") = 0.169304
prob('run') = 0.176378
prob('maybe') = 0.179954
prob('what?') = 0.180883
prob('front,') = 0.189495
prob('handle') = 0.220975
prob("didn't") = 0.236259
prob('but') = 0.237198
prob("i'll") = 0.241053
prob('much.') = 0.242396
prob('some') = 0.271794
prob('subject:What') = 0.272233
prob('(with') = 0.303739
prob('content-type:text/plain') = 0.306858
prob('so,') = 0.31475
prob('doing') = 0.343253
prob('actually') = 0.35998
prob("you're") = 0.361893
prob('set') = 0.362257
prob('....') = 0.371527
prob('header:Message-ID:1') = 0.372054
prob('criteria') = 0.380642
prob('personally') = 0.382671
prob('missing') = 0.383963
prob("that's") = 0.384922
prob('doing,') = 0.386306
prob('up"') = 0.38683
prob('skip:o 40') = 0.388831
prob('really') = 0.388969
prob('same') = 0.394487
prob('when') = 0.396404
prob("don't") = 0.397979

-- note that there's nothing from 0.4 to 0.6 because the
-- "minimum prob strength" gimmick is set to 0.1

prob('any') = 0.605843
prob('put') = 0.608789
prob('they') = 0.612345
prob('skip:i 10') = 0.627048
prob('shape') = 0.628314
prob('(on') = 0.631372
prob('even') = 0.634185
prob('few') = 0.642143
prob('after') = 0.64864
prob('keep') = 0.650868
prob('been') = 0.658237
prob('more') = 0.659455
prob('talk') = 0.663404
prob('honestly,') = 0.666728
prob('please"') = 0.666806
prob('only') = 0.67491
prob('get') = 0.677922
prob('skip:t 10') = 0.679148
prob('make') = 0.682176
prob('start') = 0.68534
prob('each') = 0.690608
prob('based') = 0.699793
prob('again') = 0.70178
prob('less') = 0.717412
prob('now.') = 0.718559
prob('others') = 0.722678
prob('take') = 0.730888
prob('out') = 0.747482
prob('worst') = 0.755264
prob('course') = 0.759866
prob('.....') = 0.760887
prob('next') = 0.76219
prob('required') = 0.771335
prob('people') = 0.774769
prob('to)') = 0.776412
prob('junk') = 0.781171
prob('header:Received:3') = 0.781465
prob('sent') = 0.787053
prob('skip:s 40') = 0.794153
prob('over') = 0.811581
prob('every') = 0.812297
prob('kept') = 0.817143
prob('started,') = 0.825477
prob('commit') = 0.830104
prob('2-3') = 0.833942
prob('boat?') = 0.844828
prob('spending') = 0.855089
prob('set-up') = 0.860469
prob('link') = 0.876754
prob('year') = 0.87787
prob('mailto:') = 0.885713
prob('email addr:yahoo.com') = 0.891006
prob('virtually') = 0.892485
prob('mail') = 0.89875
prob('week') = 0.899703
prob('potential') = 0.906919
prob('"plans",') = 0.908163
prob('legal,') = 0.911014
prob('months') = 0.915806
prob('100') = 0.919184
prob('header:Return-Path:1') = 0.920177
prob('can:') = 0.921418
prob('potential,') = 0.924261
prob('send') = 0.925941
prob('please') = 0.929334
prob('internet') = 0.951275
prob('sell') = 0.951484
prob('email') = 0.960241
prob('products') = 0.962115
prob('ads') = 0.965349
prob('lucrative') = 0.96985
prob('subject:Well') = 0.973373
prob('subject:.....') = 0.973373
prob('removed,') = 0.973392
prob('marketing') = 0.974393
prob('skip:* 40') = 0.975068
prob('buy,') = 0.97709
prob('profit') = 0.977388
prob('money') = 0.98195
prob('business.') = 0.987567
prob('email addr:yahoo.com?subject=remove') = 0.992611
prob('click') = 0.992703
prob('financial') = 0.995297
prob('header:FROM:1') = 0.997691
prob('header:SUBJECT:1') = 0.998314
prob('header:DATE:1') = 0.998327

Return-Path: <>
Delivered-To: em-ca-bruceg@em.ca
Received: (qmail 29934 invoked from network); 1 Mar 2002 01:12:45 -0000
Received: from unknown (HELO mail.caironet.net) (217.52.90.4)
  by agamemnon.bfsmedia.com with SMTP; 1 Mar 2002 01:12:45 -0000
Received: from ([])
        by mail.caironet.net (MERAK 2.10.330) with ESMTP id 247B994E;
        Fri, 01 Mar 2002 02:47:45 -0000
DATE: 28 Feb 02 5:43:30 PM
FROM:
Message-ID: <GxA9XKwkMCHz964iQaE>
SUBJECT: Well,..... What the heck?
Content-Length: 2408

What the heck?

Was I the one missing the boat?

If I just kept doing what I've been doing, I'll probably be in the same
financial shape next year that I am now.

I've been deleting, deleting, deleting, ..... what? maybe 50 to a 100
"plans", "programs", "systems" sent to me via email each and every week (I
am OBVIOUSLY on a few junk email lists),........

Do people actually make any MONEY with these?  Is it like what they say
about even the worst TV ads or Infomercials?  They'd only keep spending the
money to run 'em over & over again because they actually ARE effective?

So,.... what the heck?  I investigated.

I've looked at quite a few email offers/businesses.  Some for about,.... oh,
2 seconds.  Others a little longer.  But only two (using my criteria), are
worthwhile, .... that's right, just TWO (so far), and I thought,.........

What the heck?  I doesn't take much.  "Pocket Change" as compared to the
"real" world, when you start a "real" business.

So, I set some criteria before I'd put any of my money in an internet based
business:

* Very little money required
* The return had to be lucrative
* I could pull out at any time!!!!
  (but I'd commit for 2-3 months for an "honest try")
* No tangible products to buy, handle or sell
* Didn't have to personally talk to or "sell" anybody
* Could be as annonimous as I wanted to be
* Had to be legal, of course
* Virtually "ran itself" after set-up
   (on "auto-pilot" for marketing AND monies)

*^*^*^*^*^*^*^*^*^*^*^*^*^*^*^*^*^*^*^*^
So here's the first one,....... (it takes less to get started, is "simpler",
less profit potential (but still lots) and maybe slower to grow):

mail to (with "info please" as the subject):
oucinfo12002@yahoo.com

or just click this link & send if you can:
oucinfo12002@yahoo.com?subject=info_please

*^*^*^*^*^*^*^*^*^*^*^*^*^*^*^*^*^*^*^*^
And the second,....... (a little more up front, tho still cheap, has more
profit potential, and honestly, if you want "auto-pilot" - you don't have to
do ANYTHING (if you don't want to) after you're "set up" - I did this one
first):

mail to (with "info please" as the subject):
sfinfo12002@yahoo.com

or just click this link & send if you can:
sfinfo12002@yahoo.com?subject=info_please

*^*^*^*^*^*^*^*^*^*^*^*^*^*^*^*^*^*^*^*^
so,....... WHAT THE HECK?  Can it really hurt to take a peak?

To be removed, please mailto: not4me12002@yahoo.com?subject=REMOVE
"""

God bless America <wink>.


From tim.one@comcast.net  Thu Sep 26 05:29:21 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 26 Sep 2002 00:29:21 -0400
Subject: [Spambayes] Re: There Can Be Only One
In-Reply-To: <LNBBLJKPBEHFEDALKOLCAEMMBGAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEMNBGAB.tim.one@comcast.net>

[Tim]
> The options are as I originally suggested for the f(w) test, except
> that I've boosted "a" to 0.57, and increased the # of buckets (that
> has no effect on scores) to allow pickier distinctions to get drawn.

The pasted .ini file was more accurate, of course:  I boosted spam_cutoff to
0.67, and dropped "a" to 0.225.


From anthony@interlink.com.au  Thu Sep 26 10:00:38 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Thu, 26 Sep 2002 19:00:38 +1000
Subject: [Spambayes] More HTML strippage.
Message-ID: <200209260900.g8Q90e613628@localhost.localdomain>

I'm seeing email with stuff like:

    <TD 
    style="PADDING-RIGHT: 0.75pt; PADDING-LEFT: 0.75pt; 
    PADDING-BOTTOM: 0.75pt; WIDTH: 15.75pt; PADDING-TOP: 0.75pt" 
    width=21 rowSpan=4>

in it - the current HTML stripping RE doesn't pick this up,
as it's limited to 128 characters, and doesn't span lines. 

I propose we change it from 

    html_re = re.compile(r"""
        <
        [^\s<>]     # e.g., don't match 'a < b' or '<<<' or 'i << 5' or 'a<>b'
        [^>]{0,128} # search for the end '>', but don't run wild
        >
    """, re.VERBOSE)

to 

    html_re = re.compile(r"""
        <
        [^\s<>]      # e.g., don't match 'a < b' or '<<<' or 'i << 5' or 'a<>b'
        [^>]{0,256}? # search for the next '>', but with non-greedy RE
        >
    """, re.VERBOSE|re.DOTALL)

Will this cause people pain? Is there a better way?

Anthony

From anthony@interlink.com.au  Thu Sep 26 10:14:21 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Thu, 26 Sep 2002 19:14:21 +1000
Subject: [Spambayes] memory consumption... 
In-Reply-To: <LNBBLJKPBEHFEDALKOLCEEMHBGAB.tim.one@comcast.net> 
Message-ID: <200209260914.g8Q9EML15849@localhost.localdomain>


>>> Tim Peters wrote
> I didn't find it plausible that a 75MB process would cause swapping to disk
> on a 256MB box (unless you're running an OS much feebler than Win98 <wink>),
> but it was very plausible that it's going to disk to read thousands of
> files.

Well, it looks like the problem is that this wierd little "linux" OS 
wants to eat half the RAM in the laptop for "Cached" memory. Gods 
only know why it's doing that, or how to make it stop doing it. 

bah-i-can't-see-this-linux-thing-taking-off,
Anthony

From Alexander@Leidinger.net  Thu Sep 26 11:21:56 2002
From: Alexander@Leidinger.net (Alexander Leidinger)
Date: Thu, 26 Sep 2002 12:21:56 +0200
Subject: [Spambayes] Re: Mailbox class in the spambayes project & python
	2.2.1
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCELLBGAB.tim.one@comcast.net>
References: <eppstein-AAFDF5.11552125092002@main.gmane.org>
	<LNBBLJKPBEHFEDALKOLCCELLBGAB.tim.one@comcast.net>
Message-ID: <20020926122156.2021e446.Alexander@Leidinger.net>

On Wed, 25 Sep 2002 19:29:47 -0400 Tim Peters <tim.one@comcast.net>
wrote:

> [David Eppstein]
> > How about:
> >
> >   if isinstance(x, tuple):
> >      assert len(x) == 3
> >      try:
> >         x = unicode(x[2], x[0]).encode('ascii')
> >      except:
> >         x = x[2]
> >
> > (probably the except should be more careful about what exceptions it
> > watches...)

I've added this to email/Message.py and mboxcount.py now doesn't fail
anymore.

Now I want to clean up some unparseable messages (mostly some missing
'>' in front of some "From " lines). Is there a way to get the line
number on the file of such an unparseable message? It's a little bit
time consuming to wade through:
---snip---
reservoir/20001119.freebsd-stable.txt     529 (+ unparseable: 1)
reservoir/20010211.freebsd-questions.txt    1220 (+ unparseable: 1)
reservoir/20010225.freebsd-questions.txt    1028 (+ unparseable: 1)
reservoir/20010408.freebsd-questions.txt    1011 (+ unparseable: 1)
reservoir/20010415.freebsd-questions.txt    1080 (+ unparseable: 1)
reservoir/20010422.freebsd-questions.txt    1211 (+ unparseable: 1)
reservoir/20010513.freebsd-questions.txt    1179 (+ unparseable: 3)
reservoir/20010527.freebsd-questions.txt    1036 (+ unparseable: 1)
reservoir/20010603.freebsd-questions.txt    1031 (+ unparseable: 2)
reservoir/20011014.freebsd-questions.txt    1008 (+ unparseable: 5)
reservoir/20010805.freebsd-questions.txt    1054 (+ unparseable: 4)
reservoir/20010902.freebsd-questions.txt     995 (+ unparseable: 1)
reservoir/20010909.freebsd-questions.txt    1000 (+ unparseable: 1)
reservoir/20011021.freebsd-questions.txt    1278 (+ unparseable: 1)
reservoir/20011028.freebsd-questions.txt     948 (+ unparseable: 2)
reservoir/20011104.freebsd-questions.txt    1071 (+ unparseable: 1)
reservoir/20011118.freebsd-questions.txt    1112 (+ unparseable: 2)
reservoir/20011125.freebsd-questions.txt    1235 (+ unparseable: 1)
reservoir/20011209.freebsd-questions.txt    1069 (+ unparseable: 1)
reservoir/20011223.freebsd-questions.txt     956 (+ unparseable: 1)
---snip---

Searching for "^From " doesn't work here, most of these lines are valid
>From lines as part of the message body.

> Goodness, you fellows are working hard!  Stupid beats smart almost all
> the time here; the first thing I would have tried was
> 
>     for piece in repr(x).split():
>         yield 'charset:' + piece

I assume this should be a part of tokenize.py, but as long as Message.py
raises an exception, this doesn't work.

Bye,
Alexander.


-- 
                   Press every key to continue.

http://www.Leidinger.net                       Alexander @ Leidinger.net
  GPG fingerprint = C518 BC70 E67F 143F BE91  3365 79E2 9C60 B006 3FE7

From anthony@interlink.com.au  Thu Sep 26 11:29:01 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Thu, 26 Sep 2002 20:29:01 +1000
Subject: [Spambayes] Two Scheme Enter, One Scheme Leave. 
Message-ID: <200209261029.g8QAT2K17340@localhost.localdomain>

Quick bit on max_discriminators. 
Using the settings from before, I get

md      fp      fn      total
5       69      17      86      (cutoff 0.55)
15      11      30      41      (cutoff 0.625)
100     10      21      31
150     9       21      30
400     9       21      30

--

Next is Graham vs. best settings for Robinson, with 
a bunch of different seeds. Letting the best cutoff 
win in all cases for Graham, to compensate for the work 
put into tuning Robinson in this trial (cutoff for graham's 
nearly always 0.95, it seems)

(format is fp+fn=total)

seed	   Graham  	Robinson
1010101    41+12=53     19+21=40
12346	   30+12=42      9+21=30
271170     29+16=45     25+16=41
432104	   48+19=67	18+24=42
56743	   29+13=42     13+17=30
774213	   44+17=61	21+19=40
999111     39+21=60     20+26=46

mean       37+16=53     18+21=39
mean fp%   1.85%        0.9%
mean fn%   0.80%        1.05%

The final test, of course, is to feed the full corpus into the
two tweaked schemes. That's 11,000 spam and 19,000 ham split into 
10 sets. Note that I was a _bit_ naughty here - these two tests
are done with the version of tokenizer.py that strips out and 
discards style sheets. 

Graham:

  -> <stat> Ham scores for all runs: 19224 items; mean 0.78; sdev 8.59
  -> <stat> Spam scores for all runs: 11303 items; mean 99.38; sdev 7.61
  -> best cutoff for all runs: 0.8
  ->     with 139 fp + 74 fn = 213 mistakes
  ->     matched at 0.825 (139 fp + 74 fn)
  ->     matched at 0.85 (137 fp + 76 fn)
  ->     matched at 0.9 (135 fp + 78 fn)
  total unique false pos 135
  total unique false neg 78
  average fp % 0.702258059105
  average fn % 0.690069873164

Robinson, with cutoff to 0.65 (because I care more about fp than fn):
  -> <stat> Ham scores for all runs: 19224 items; mean 31.30; sdev 13.09
  -> <stat> Spam scores for all runs: 11303 items; mean 83.51; sdev 9.47
  -> best cutoff for all runs: 0.6
  ->     with 76 fp + 108 fn = 184 mistakes
  total unique false pos 10
  total unique false neg 258
  average fp % 0.0520210194464
  average fn % 2.28248163189


This is a completely _spooky_ level of accuracy, given the sheer
unpleasantness of the corpus (and how badly the default Graham did
on it)

If the change to make the HTML re a bit greedier was in the code
it would fix 2 of my 10 fps. 

I've appended the histograms from this run to the email.

Comparing the final two versions...
---- 
graham->robinson

test1_alls -> test2_alls

false positive percentages
    0.572  0.000  won   -100.00%
    0.676  0.104  won    -84.62%
    0.780  0.052  won    -93.33%
    0.884  0.052  won    -94.12%
    0.728  0.052  won    -92.86%
    0.832  0.000  won   -100.00%
    0.676  0.104  won    -84.62%
    0.676  0.104  won    -84.62%
    0.572  0.000  won   -100.00%
    0.624  0.052  won    -91.67%

won  10 times
tied  0 times
lost  0 times

total unique fp went from 135 to 10 won    -92.59%
mean fp % went from 0.702258059105 to 0.0520210194464 won    -92.59%

false negative percentages
    0.088  1.062  lost  +1106.82%
    0.619  2.653  lost  +328.59%
    1.150  2.389  lost  +107.74%
    0.796  2.829  lost  +255.40%
    0.973  3.363  lost  +245.63%
    0.531  2.124  lost  +300.00%
    0.796  2.476  lost  +211.06%
    0.531  1.858  lost  +249.91%
    0.885  1.770  lost  +100.00%
    0.531  2.301  lost  +333.33%

won   0 times
tied  0 times
lost 10 times

total unique fn went from 78 to 258 lost  +230.77%
mean fn % went from 0.690069873164 to 2.28248163189 lost  +230.76%

ham mean                     ham sdev
   0.63   31.50 +4900.00%        7.79   12.94  +66.11%
   0.67   30.95 +4519.40%        8.16   13.05  +59.93%
   0.79   31.35 +3868.35%        8.75   13.24  +51.31%
   1.00   31.47 +3047.00%        9.72   13.11  +34.88%
   0.84   31.30 +3626.19%        8.99   13.13  +46.05%
   0.91   31.12 +3319.78%        9.24   13.17  +42.53%
   0.71   31.09 +4278.87%        8.19   13.11  +60.07%
   0.85   31.90 +3652.94%        8.71   12.88  +47.88%
   0.63   31.20 +4852.38%        7.67   13.10  +70.80%
   0.78   31.14 +3892.31%        8.43   13.19  +56.47%

ham mean and sdev for all runs
   0.78   31.30 +3912.82%        8.59   13.09  +52.39%

spam mean                    spam sdev
  99.91   83.86  -16.06%        2.97    8.83 +197.31%
  99.38   83.59  -15.89%        7.80    9.57  +22.69%
  98.86   83.62  -15.42%       10.53    9.75   -7.41%
  99.40   82.76  -16.74%        7.07    9.56  +35.22%
  99.19   82.82  -16.50%        8.64    9.60  +11.11%
  99.52   83.31  -16.29%        6.69    9.43  +40.96%
  99.22   83.61  -15.73%        8.58    9.46  +10.26%
  99.62   83.94  -15.74%        5.74    9.41  +63.94%
  99.17   83.57  -15.73%        8.93    9.44   +5.71%
  99.53   83.98  -15.62%        6.52    9.58  +46.93%

spam mean and sdev for all runs
  99.38   83.51  -15.97%        7.61    9.47  +24.44%

ham/spam mean difference: 98.60 52.21 -46.39
----

I'd say that Robinson scheme is a clear winner for this data set.

Anthony

.........

Histogram for the final full run of Robinson scheme:

-> <stat> Ham scores for all runs: 19224 items; mean 31.30; sdev 13.09
* = 33 items
  0.00  448 **************
  2.50  843 **************************
  5.00  385 ************
  7.50  283 *********
 10.00  349 ***********
 12.50  416 *************
 15.00  484 ***************
 17.50  512 ****************
 20.00  519 ****************
 22.50  610 *******************
 25.00  803 *************************
 27.50 1389 *******************************************
 30.00 1590 *************************************************
 32.50 1768 ******************************************************
 35.00 1963 ************************************************************
 37.50 1885 **********************************************************
 40.00 1591 *************************************************
 42.50 1245 **************************************
 45.00  871 ***************************
 47.50  533 *****************
 50.00  325 **********
 52.50  177 ******
 55.00  106 ****
 57.50   53 **
 60.00   43 **
 62.50   23 *
 65.00    7 *
 67.50    2 *
 70.00    0 
 72.50    1 *
 75.00    0 
 77.50    0 
 80.00    0 
 82.50    0 
 85.00    0 
 87.50    0 
 90.00    0 
 92.50    0 
 95.00    0 
 97.50    0 

-> <stat> Spam scores for all runs: 11303 items; mean 83.51; sdev 9.47
* = 26 items
  0.00    0 
  2.50    0 
  5.00    0 
  7.50    0 
 10.00    0 
 12.50    0 
 15.00    0 
 17.50    0 
 20.00    0 
 22.50    0 
 25.00    0 
 27.50    0 
 30.00    0 
 32.50    0 
 35.00    0 
 37.50    1 *
 40.00    0 
 42.50    2 *
 45.00    6 *
 47.50   10 *
 50.00   10 *
 52.50   13 *
 55.00   29 **
 57.50   37 **
 60.00   59 ***
 62.50   91 ****
 65.00  159 *******
 67.50  260 **********
 70.00  428 *****************
 72.50  741 *****************************
 75.00 1113 *******************************************
 77.50 1463 *********************************************************
 80.00 1521 ***********************************************************
 82.50 1044 *****************************************
 85.00  624 ************************
 87.50  507 ********************
 90.00  618 ************************
 92.50  668 **************************
 95.00  870 **********************************
 97.50 1029 ****************************************
-> best cutoff for all runs: 0.6
->     with 76 fp + 108 fn = 184 mistakes


Final bonus note: The false positive that scored over 72.5 (with
names obscured). It's a false positive, because it wasn't spam, but
at the same time, I don't think anyone would've missed it had it 
"gone astray". :)

Subject: Inquiry

I'm looking for the right person at your company to contact regarding our
services. Frankly, I don't know if we can increase your sales as we have for
our current partners, but you do have a GREAT site and I'd like to see if a
relationship would make sense.

Would you put me in touch with the person who is responsible for driving
your online sales? Any help would be most appreciated.

Thanks!
Uuu

 ==============================
Uuu Uuuuuuu ......... Vice President, NNNNNNNN Inc.
Tel:  3x0-6x0-xxxx ...... NNNN S. SS SSSSSSS, Suite 706
eFax: (xxx) 843-xxxx...... PPPPPPPPP, CA ppppp
uuuuuuu@NNNNNNNN.com .... http://www.NNNNNNNN.com/inc/solutions/

NNNNNNNN, Inc's Product Rating, E-Mail Sharing, Gift Registry and Shopping
List services are cost effective, simple to integrate, and GUARANTEED to
increase sales:
http://www.NNNNNNNN.com/inc/demo/index.htm?m=solutions




From anthony@interlink.com.au  Thu Sep 26 11:33:21 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Thu, 26 Sep 2002 20:33:21 +1000
Subject: [Spambayes] Two Scheme Enter, One Scheme Leave. 
In-Reply-To: <200209261029.g8QAT2K17340@localhost.localdomain> 
Message-ID: <200209261033.g8QAXMZ17591@localhost.localdomain>


Amusingly, the message I just sent through got picked up by my own
spamassassin as spam. Oh well. :)

Anthony

>>> Anthony Baxter wrote
> SPAM: -------------------- Start SpamAssassin results ----------------------
> SPAM: This mail is probably spam.  The original message has been altered
> SPAM: so you can recognise or block similar unwanted mail in future.
> SPAM: See http://spamassassin.org/tag/ for more details.
> SPAM: 
> SPAM: Content analysis details:   (5.4 hits, 5 required)
> SPAM: X_AUTH_WARNING     (-0.7 points) Found a X-Authentication-Warning heade
r
> SPAM: TO_LOCALPART_EQ_REAL (0.5 points)  To: repeats local-part as real name
> SPAM: DOUBLE_CAPSWORD    (1.1 points)  BODY: A word in all caps repeated on t
he line
> SPAM: PORN_10            (0.6 points)  BODY: Uses words and phrases which ind
icate porn (10)
> SPAM: GUARANTEE          (1.5 points)  BODY: Contains word 'guarantee' in all
-caps
> SPAM: INCREASE_SOMETHING (0.4 points)  BODY: Instructions on how to increase 
something
> SPAM: PORN_3             (2.0 points)  Uses words and phrases which indicate 
porn (3)
> SPAM: 
> SPAM: -------------------- End of SpamAssassin results ---------------------


-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.


From skip@pobox.com  Thu Sep 26 13:12:07 2002
From: skip@pobox.com (Skip Montanaro)
Date: Thu, 26 Sep 2002 07:12:07 -0500
Subject: [Spambayes] There Can Be Only One
Message-ID: <15762.63895.199782.41444@12-248-11-90.client.attbi.com>

    Tim> Speaking of which, are you going to follow up on

    Tim> http://mail.python.org/pipermail-21/spambayes/2002-September/000211.html

I forgot about that.  Thanks for reminding me.

Okay, I just created fpfn.ini with just these options:

    [TestDriver]
    show_false_positives = False
    show_false_negatives = True
    show_charlimit = 500000

and ran

    BAYESCUSTOMIZE=fpfn.ini timcv.py -n10 --ham=200 --spam=200 -s12345

The rather large results are available at either of

    http://manatee.mojam.com/~skip/public_html/fpfn.zip
    http://manatee.mojam.com/~skip/public_html/fpfn.out

The .out file is about 1.5MB.  The .zip file about 450KB.

I haven't scanned the output yet myself.  Got a kid to get off to school and
plumbers coming at 8...

Skip


From gward@python.net  Thu Sep 26 14:54:38 2002
From: gward@python.net (Greg Ward)
Date: Thu, 26 Sep 2002 09:54:38 -0400
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <20020926022740.GA24111@cthulhu.gerg.ca>
References: <BIEJKCLHCIOIHAGOKOLHGEAMDLAA.tim.one@comcast.net>
	<LNBBLJKPBEHFEDALKOLCMEMDBGAB.tim.one@comcast.net>
	<20020926022740.GA24111@cthulhu.gerg.ca>
Message-ID: <20020926135437.GA24403@cthulhu.gerg.ca>

On 25 September 2002, I said:
> On 25 September 2002, Tim Peters said:
> Yes, I've been running tests all afternoon and evening.  Vague,
> hand-wavey results:
> 
>   * my histograms are not terribly normal -- not as weird as Guido's,
>     but not nearly as nice as Tim's
>   * I think my peaks are better separated though -- there's a pretty
>     wide range for spam_cutoff
>   * I'm one of the few who seems to win by setting set spam_cutoff < 0.5

OK, here's some more detail.  First, my corpus: I had to cobble together
a few sources of email in order make it into the Big Leagues, ie. have
2000 spam to play with.  Here are my sources:

  * python.org Sept 2002 harvest:
      1895 spam
      3821 normal ham
      1662 dsn
      5483 total ham

  * spam destined for gward@python.net, but detected by SpamAssassin
    and set aside by my .procmailrc on starship.python.net, from
    2002-02-11 to 2002-08-07 (ie. from when I started using SA
    on starship until I replaced qmail with Exim).  That's 1580
    spams.

    There are two obvious artifacts of this spam collection: lots of
    messages have "To: gward@python.net", and all of them have
    "Received: ... by starship.python.net".  So I counterbalanced it
    with...

  * the contents of all my personal inboxes as of about noon yesterday
    -- 1333 hams.  These have the same two artifacts as the
    gward@python.net spam collection, which seems to have prevented
    eg. "To: gward@python.net" being a very good clue either way.
    But since they were sent on from starship to my various ISP
    accounts (3 or 4 variations over the time that this mail has
    been received), they were covered with "Received" headers
    giving that away.  So I removed all "Received" headers up to
    the "Received: ... by starship.python.net", which seems to
    have done the trick.  There are a few messages that were sent
    straight to one of my ISP addresses, but not many since I never
    use them publicly.

The biggest artifact remaining is that all of the gward-spam was
received by qmail, whereas most of the other mail was received by Exim.
This leaks into the "best discriminators" list:

        'received:0000' 87 0.655303
        'received:HELO' 96 0.725595
        'received:unknown' 96 0.746786

Almost every initial "Received" line in my gward-spam collection looks
like

  Received: from unknown (HELO yahoo.com) (218.232.230.20)
    by starship.python.net with SMTP; 28 Apr 2002 19:12:41 -0000

If Exim had received the same message, it would slap on something like
this instead:

  Received: from [218.232.230.20] (helo=yahoo.com)
          by starship.python.net with smtp (Exim 4.05)
          id 17sNGM-0004Ch-00
          for gward@python.net; Sun 28 Apr 2002 15:12:41 -0400

so the above three clues mean that "qmail receives more spam than ham"
-- because 100% (1580 msgs) of gward-spam is spam received by qmail, and
1077/1333 messages in gward-ham were received by qmail.

The flip side of the coin:

        'received:helo' 133 0.38154
        'received:for' 163 0.393567
        'received:4.05' 166 0.389087
        'received:0400' 167 0.397625
        'received:esmtp' 177 0.163725
        'received:Exim' 183 0.364903

mean "Exim receives a bit more ham than spam" -- unless the protocol is
ESMTP, which is a fairly good ham indicator.  That probably *is* a valid
clue, rather than a qmail-vs-Exim artifact.  Interesting.

Anyways, that little exploration has me wondering just how valid my data
is.  I should probably rerun everything without looking at "Received"
headers at all (except to count them -- for the most part, they stop at
either mail.python.org or starship.python.net, which are the front-line
servers for these two collections).

Right, onto the results.  First, I'm a little unclear on how everyone
else has been generating the results they've been posting.  I did this:

  [...several test runs...]
  timcv.py -n10 --ham=200 --spam=200 -s54321 > timcv-run5.log
  [...tweak .ini file a couple of times...]
  timcv.py -n10 --ham=200 --spam=200 -s54321 > timcv-run7.log

run5 is Graham, run7 is f(w) with spam_cutoff=0.475 (based on the
results of run6, which I'm not showing here).

Then I ran
  rates.py timcv-run5.log
  rates.py timcv-run7.log

and then
  cmp.py timcv-run5.logs.txt timcv-run7.logs.txt

which gives me this:

"""
timcv-run5.logs.txt -> timcv-run7.logs.txt
[...]

false positive percentages
    0.000  0.500  lost  +(was 0)
    0.000  0.000  tied          
    0.000  0.000  tied          
    0.500  0.500  tied          
    0.000  0.000  tied          
    1.000  0.000  won   -100.00%
    0.000  0.000  tied          
    0.000  0.000  tied          
    0.500  0.000  won   -100.00%
    1.000  1.000  tied          

won   2 times
tied  7 times
lost  1 times

total unique fp went from 6 to 4 won    -33.33%
mean fp % went from 0.3 to 0.2 won    -33.33%

false negative percentages
    2.000  2.000  tied          
    1.500  2.500  lost   +66.67%
    1.000  2.000  lost  +100.00%
    1.500  2.000  lost   +33.33%
    1.000  0.500  won    -50.00%
    0.500  1.500  lost  +200.00%
    1.000  1.000  tied          
    2.000  1.500  won    -25.00%
    1.000  1.000  tied          
    1.000  1.500  lost   +50.00%

won   2 times
tied  3 times
lost  5 times

total unique fn went from 25 to 31 lost   +24.00%
mean fn % went from 1.25 to 1.55 lost   +24.00%

ham mean                     ham sdev
   0.23   20.68 +8891.30%        3.20    8.87 +177.19%
   0.00   20.99 +(was 0)        0.00    9.40 +(was 0)
   0.00   20.02 +(was 0)        0.00    8.17 +(was 0)
   0.50   21.70 +4240.00%        7.03    9.84  +39.97%
   0.00   20.59 +(was 0)        0.00    8.91 +(was 0)
   0.99   20.70 +1990.91%        9.89   10.06   +1.72%
   0.00   19.02 +(was 0)        0.00    8.47 +(was 0)
   0.00   19.41 +(was 0)        0.00    8.65 +(was 0)
   0.50   18.99 +3698.00%        7.05    7.96  +12.91%
   1.00   21.73 +2073.00%        9.95   10.64   +6.93%

ham mean and sdev for all runs
   0.32   20.38 +6268.75%        5.55    9.18  +65.41%

spam mean                    spam sdev
  98.41   80.22  -18.48%       11.97   11.30   -5.60%
  98.48   79.66  -19.11%       12.15   11.51   -5.27%
  99.00   79.41  -19.79%        9.95   12.24  +23.02%
  98.50   79.70  -19.09%       12.16   11.44   -5.92%
  99.00   79.93  -19.26%        9.95   11.49  +15.48%
  99.50   79.48  -20.12%        7.05   10.94  +55.18%
  99.00   80.18  -19.01%        9.95   11.01  +10.65%
  98.00   78.66  -19.73%       14.00   11.04  -21.14%
  99.00   78.52  -20.69%        9.95   11.55  +16.08%
  99.00   79.80  -19.39%        9.95   11.25  +13.07%

spam mean and sdev for all runs
  98.79   79.56  -19.47%       10.87   11.40   +4.88%

ham/spam mean difference: 98.47 59.18 -39.29
"""

Here are the histograms for run5 (Graham):

* = 34 items
  0.00 1993 ***********************************************************
  2.50    0 
  5.00    0 
  7.50    0 
 10.00    0 
 12.50    0 
 15.00    0 
 17.50    0 
 20.00    0 
 22.50    0 
 25.00    0 
 27.50    0 
 30.00    0 
 32.50    0 
 35.00    0 
 37.50    0 
 40.00    0 
 42.50    0 
 45.00    1 *
 47.50    0 
 50.00    0 
 52.50    0 
 55.00    0 
 57.50    0 
 60.00    0 
 62.50    0 
 65.00    0 
 67.50    0 
 70.00    0 
 72.50    0 
 75.00    0 
 77.50    0 
 80.00    0 
 82.50    0 
 85.00    0 
 87.50    0 
 90.00    0 
 92.50    0 
 95.00    0 
 97.50    6 *

-> <stat> Spam scores for all runs: 2000 items; mean 98.79; sdev 10.87
* = 33 items
  0.00   23 *
  2.50    0 
  5.00    1 *
  7.50    0 
 10.00    0 
 12.50    0 
 15.00    0 
 17.50    0 
 20.00    0 
 22.50    0 
 25.00    0 
 27.50    0 
 30.00    0 
 32.50    0 
 35.00    0 
 37.50    0 
 40.00    0 
 42.50    0 
 45.00    0 
 47.50    0 
 50.00    0 
 52.50    0 
 55.00    0 
 57.50    0 
 60.00    0 
 62.50    0 
 65.00    0 
 67.50    0 
 70.00    0 
 72.50    0 
 75.00    0 
 77.50    0 
 80.00    0 
 82.50    1 *
 85.00    0 
 87.50    0 
 90.00    1 *
 92.50    0 
 95.00    1 *
 97.50 1973 ************************************************************

and for run7 (Robinson f(w)):

-> <stat> Ham scores for all runs: 2000 items; mean 20.38; sdev 9.18
* = 5 items
  0.00   0 
  2.50  41 *********
  5.00  96 ********************
  7.50 144 *****************************
 10.00 148 ******************************
 12.50 146 ******************************
 15.00 206 ******************************************
 17.50 202 *****************************************
 20.00 253 ***************************************************
 22.50 212 *******************************************
 25.00 173 ***********************************
 27.50 124 *************************
 30.00  77 ****************
 32.50  51 ***********
 35.00  41 *********
 37.50  26 ******
 40.00  13 ***
 42.50  13 ***
 45.00  11 ***
 47.50  11 ***
 50.00   4 *
 52.50   4 *
 55.00   1 *
 57.50   3 *
 60.00   0 
 62.50   0 
 65.00   0 
 67.50   0 
 70.00   0 
 72.50   0 
 75.00   0 
 77.50   0 
 80.00   0 
 82.50   0 
 85.00   0 
 87.50   0 
 90.00   0 
 92.50   0 
 95.00   0 
 97.50   0 

-> <stat> Spam scores for all runs: 2000 items; mean 79.56; sdev 11.40
* = 3 items
  0.00   0 
  2.50   0 
  5.00   0 
  7.50   0 
 10.00   0 
 12.50   0 
 15.00   0 
 17.50   0 
 20.00   0 
 22.50   0 
 25.00   0 
 27.50   0 
 30.00   0 
 32.50   0 
 35.00   0 
 37.50   2 *
 40.00   1 *
 42.50   5 **
 45.00   4 **
 47.50   6 **
 50.00   6 **
 52.50   7 ***
 55.00  18 ******
 57.50  41 **************
 60.00  47 ****************
 62.50  68 ***********************
 65.00 105 ***********************************
 67.50 115 ***************************************
 70.00 160 ******************************************************
 72.50 141 ***********************************************
 75.00 149 **************************************************
 77.50 128 *******************************************
 80.00 118 ****************************************
 82.50 154 ****************************************************
 85.00 159 *****************************************************
 87.50 171 *********************************************************
 90.00 119 ****************************************
 92.50  85 *****************************
 95.00  75 *************************
 97.50 116 ***************************************
-> best cutoff for all runs: 0.5
->     with 12 fp + 18 fn = 30 mistakes

Oops, just noticed the "best cutoff for all runs" thing.  I must have
misinterpreted the run6 output -- picking 0.475 was an eyeball average.
D'ohh.

Off to re-run things, without mine_received and with a better
spam_cutoff.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
A closed mouth gathers no foot.

From popiel@wolfskeep.com  Thu Sep 26 15:00:52 2002
From: popiel@wolfskeep.com (T. Alexander Popiel)
Date: Thu, 26 Sep 2002 07:00:52 -0700
Subject: [Spambayes] There Can Be Only One 
In-Reply-To: Message from Tim Peters <tim.one@comcast.net> 
	of "Wed, 25 Sep 2002 21:38:45 EDT."
	<LNBBLJKPBEHFEDALKOLCMEMDBGAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCMEMDBGAB.tim.one@comcast.net> 
Message-ID: <20020926140052.C01D6F5A7@cashew.wolfskeep.com>

In message:  <LNBBLJKPBEHFEDALKOLCMEMDBGAB.tim.one@comcast.net>
             Tim Peters <tim.one@comcast.net> writes:
>
>Does anyone else intend to participate in this death match?

Yes, I do... but I'm having trouble getting enough time to do so.
I may not be able to do anything on it until Saturday.

On the other hand, I have a corpus of 1425 ham and 2279 spam,
single source, single time-frame, single user, representing
_everything_ than came to the address.  Probably a bit more
'pure' than what many are working with.  I can augment the ham
number with selected slightly older ham (to get up to the 2000
ham you want), with the caveat that it will introduce time and
subject distortions.  In another week I should have 2000 ham
of the pure variety.

>If not, I'll tally results tomorrow.

Feel free... as I said, while I want to participate, I may not
be able to until Saturday (or later, if closing on the house
hits any more snags).

- Alex

From anthony@interlink.com.au  Thu Sep 26 15:05:56 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Fri, 27 Sep 2002 00:05:56 +1000
Subject: [Spambayes] There Can Be Only One 
In-Reply-To: <20020926135437.GA24403@cthulhu.gerg.ca> 
Message-ID: <200209261405.g8QE5w619370@localhost.localdomain>


>>> Greg Ward wrote
> Anyways, that little exploration has me wondering just how valid my data
> is.  I should probably rerun everything without looking at "Received"
> headers at all (except to count them -- for the most part, they stop at
> either mail.python.org or starship.python.net, which are the front-line
> servers for these two collections).

The data's fine, you'll just need to be careful about the headers you
look at - it's distressing _how_ good this stuff is at spotting patterns.

Delivery-date is another header to watch out for, if the ham/spam comes
from different places or times. It's not clear to me what puts that 
header in - it might be an MH thing.

I tend to do multiple runs by having multiple ini files. Say, a 
common.ini with the options that are constant, then test1.ini, test2.ini
or whatever, that have the options that vary. I can then put 

BAYESCUSTOMIZE="common.ini test1.ini" python2.3 timcv.py ...... > test1.txt
BAYESCUSTOMIZE="common.ini test2.ini" python2.3 timcv.py ...... > test2.txt
BAYESCUSTOMIZE="common.ini test3.ini" python2.3 timcv.py ...... > test3.txt

in a shell script, run it and go get lunch (or coffee, or sleep, or
whatever)

(aside - I tend to name the changing ini files more like x01.ini x02.ini 
mindisc15.ini, makes life more sane...)

Anthony

From anthony@ekit-inc.com  Thu Sep 26 05:49:36 2002
From: anthony@ekit-inc.com (Anthony Baxter)
Date: Thu, 26 Sep 2002 14:49:36 +1000
Subject: [Spambayes] Two Scheme Enter, One Scheme Leave. 
Message-ID: <200209260449.g8Q4nav10534@localhost.localdomain>

---------------------- multipart/mixed attachment
My normal email's off the air at the moment, but here's some more data, 
plus a dinky little hack.

I wrote a quickie script to trawl through the test*.txt full output and
try and find the best results overall. It's attached. For me, when run
against all the robinson tests, I get:

best results
got   9fp +  21fn =  30 tot, at a:0.100 x:0.500 min: 0.080 cutoff: 60.000
got   9fp +  21fn =  30 tot, at a:0.100 x:0.500 min: 0.090 cutoff: 60.000
got   9fp +  21fn =  30 tot, at a:0.100 x:0.500 min: 0.100 cutoff: 60.000
got   8fp +  23fn =  31 tot, at a:0.100 x:0.500 min: 0.050 cutoff: 60.000
got   9fp +  22fn =  31 tot, at a:0.200 x:0.500 min: 0.100 cutoff: 60.000
got   9fp +  22fn =  31 tot, at a:0.250 x:0.500 min: 0.100 cutoff: 60.000
got  10fp +  21fn =  31 tot, at a:0.075 x:0.500 min: 0.100 cutoff: 60.000
got  10fp +  21fn =  31 tot, at a:0.125 x:0.500 min: 0.100 cutoff: 60.000
got  10fp +  21fn =  31 tot, at a:0.150 x:0.500 min: 0.100 cutoff: 60.000
got  10fp +  21fn =  31 tot, at a:0.350 x:0.500 min: 0.100 cutoff: 60.000
got  12fp +  19fn =  31 tot, at a:0.100 x:0.600 min: 0.100 cutoff: 60.000
got  10fp +  22fn =  32 tot, at a:0.450 x:0.500 min: 0.100 cutoff: 60.000
got  11fp +  21fn =  32 tot, at a:0.500 x:0.500 min: 0.100 cutoff: 60.000
got  12fp +  20fn =  32 tot, at a:0.100 x:0.500 min: 0.110 cutoff: 60.000
got  12fp +  20fn =  32 tot, at a:0.100 x:0.500 min: 0.120 cutoff: 60.000
got  13fp +  19fn =  32 tot, at a:0.100 x:0.500 min: 0.150 cutoff: 60.000
got   8fp +  25fn =  33 tot, at a:0.100 x:0.400 min: 0.100 cutoff: 60.000
got  12fp +  24fn =  36 tot, at a:0.025 x:0.500 min: 0.100 cutoff: 60.000
got  20fp +  16fn =  36 tot, at a:0.100 x:0.400 min: 0.100 cutoff: 57.500
got  22fp +  14fn =  36 tot, at a:0.010 x:0.500 min: 0.100 cutoff: 57.500
got  23fp +  13fn =  36 tot, at a:0.025 x:0.500 min: 0.100 cutoff: 57.500
got   9fp +  28fn =  37 tot, at a:0.100 x:0.500 min: 0.150 cutoff: 62.500

Anyway, the script's attached. It could do with being generalised (in
some way... not sure what's best), but hopefully it's useful.

You could easily enough pull the data out of the Storage class at the
end and plot it with your favourite graphing tool that's capable of 
plotting 3 data sets on a 4-dimensional graph...

Anthony


---------------------- multipart/mixed attachment
A non-text attachment was scrubbed...
Name: not available
Type: text/x-python
Size: 3613 bytes
Desc: spamnalysis.py
Url : http://mail.python.org/pipermail-21/spambayes/attachments/20020926/570601f0/attachment.py

---------------------- multipart/mixed attachment--

From guido@python.org  Thu Sep 26 15:25:59 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 26 Sep 2002 10:25:59 -0400
Subject: [Spambayes] \r\n endings
In-Reply-To: Your message of "Thu, 26 Sep 2002 01:46:13 PDT."
             <E17uUHl-0004lh-00@usw-pr-cvs1.sourceforge.net> 
References: <E17uUHl-0004lh-00@usw-pr-cvs1.sourceforge.net> 
Message-ID: <200209261425.g8QEPxW25892@odiug.zope.com>

Sjoerd just checked this in:
> Modified Files:
> 	HistToGNU.py 
> Log Message:
> Converted \r\n line endings to \n.

*Someone* out there is using a broken editor or CVS client that checks
files in with \r\n line endings.  I've cleaned one of these up, and
I've seen several others do this on occasion.  I think it may be the
editor because it appeared that only *some* line endings (those of
changed lines?) used \r\n, while others used \n.

I think it's bkc.

Whoever you are, can you try not to do this?

--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido@python.org  Thu Sep 26 15:42:03 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 26 Sep 2002 10:42:03 -0400
Subject: [Spambayes] There Can Be Only One
In-Reply-To: Your message of "Fri, 27 Sep 2002 00:05:56 +1000."
             <200209261405.g8QE5w619370@localhost.localdomain> 
References: <200209261405.g8QE5w619370@localhost.localdomain> 
Message-ID: <200209261442.g8QEg3826059@odiug.zope.com>

> Delivery-date is another header to watch out for, if the ham/spam comes
> from different places or times. It's not clear to me what puts that 
> header in - it might be an MH thing.

Yes, that's how the new MH on Linux keeps the info from the "Unix From
line".  It inserts these at the very top.  When you use packf to convert
an MH folder to a mailbox file it uses these to regenerate the Unix
>From line -- but only if they are still at the very top.  Mine often
aren't, e.g. when another MH tool has inserted a Replied-To header in
front. :-) / :-(

--Guido van Rossum (home page: http://www.python.org/~guido/)

From gward@python.net  Thu Sep 26 16:09:08 2002
From: gward@python.net (Greg Ward)
Date: Thu, 26 Sep 2002 11:09:08 -0400
Subject: [Spambayes] Re: There Can Be Only One
In-Reply-To: <LNBBLJKPBEHFEDALKOLCAEMMBGAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCCEMJBGAB.tim.one@comcast.net>
	<LNBBLJKPBEHFEDALKOLCAEMMBGAB.tim.one@comcast.net>
Message-ID: <20020926150908.GB24403@cthulhu.gerg.ca>

On 26 September 2002, Tim Peters said:
> The most embarrassing fn remaining is probably this one:  it's been a
> persistent fn under all schemes, and is ragingly obvious spam to the human
> judge:

Just for grins, I ran this one through SpamAssassin 2.41 (the latest,
but not yet on mail.python.org).  SA had no trouble calling it spam:

SPAM: ---- Start SpamAssassin results
SPAM: 20.30 hits, 5 required;
SPAM: *  2.2 -- From: has a malformed address
SPAM: *  1.7 -- Message-Id has no @ sign
SPAM: *  1.6 -- Invalid Date: header (not RFC 2822)
SPAM: *  1.2 -- Message-Id is not valid, according to RFC 2822
SPAM: *  1.1 -- Header with all capitals found
SPAM: *  4.3 -- BODY: Claims you can be removed from the list
SPAM: *  1.0 -- BODY: List removal information
SPAM: * -0.1 -- BODY: Spam phrases score is 08 to 13 (medium)
SPAM:           [score: 10]
SPAM: *  1.0 -- URI: Includes a link to a likely spammer email address
SPAM: *  2.7 -- Date: is 24 to 48 hours before Received: date
SPAM: *  1.4 -- Missing To: header
SPAM: *  2.2 -- RBL: Received via a relay in orbs.dorkslayers.com
SPAM:           [RBL check: found 4.90.52.217.orbs.dorkslayers.com.]
SPAM: 
SPAM: ---- End of SpamAssassin results

That give you any ideas for tokenization hacks?

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
"Question authority!"  "Oh yeah?  Says who?"

From gward@python.net  Thu Sep 26 16:11:44 2002
From: gward@python.net (Greg Ward)
Date: Thu, 26 Sep 2002 11:11:44 -0400
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <200209261405.g8QE5w619370@localhost.localdomain>
References: <20020926135437.GA24403@cthulhu.gerg.ca>
	<200209261405.g8QE5w619370@localhost.localdomain>
Message-ID: <20020926151144.GC24403@cthulhu.gerg.ca>

On 27 September 2002, Anthony Baxter said:
> Delivery-date is another header to watch out for, if the ham/spam comes
> from different places or times. It's not clear to me what puts that 
> header in - it might be an MH thing.

Delivery-date is added by the MTA at delivery time, at least if your MTA
is qmail or Exim using a router with the add_delivery_date option.

IMHO it's also perfectly valid to take the date in the "From " line and
convert it to Delivery-date -- I wrote an mbox to Maildir script
(because all the existing ones SUCK), and it does that.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Are we THERE yet?

From python-spambayes@discworld.dyndns.org  Thu Sep 26 16:31:47 2002
From: python-spambayes@discworld.dyndns.org (Charles Cazabon)
Date: Thu, 26 Sep 2002 09:31:47 -0600
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <20020926151144.GC24403@cthulhu.gerg.ca>;
	from gward@python.net on Thu, Sep 26, 2002 at 11:11:44AM -0400
References: <20020926135437.GA24403@cthulhu.gerg.ca>
	<200209261405.g8QE5w619370@localhost.localdomain>
	<20020926151144.GC24403@cthulhu.gerg.ca>
Message-ID: <20020926093147.D17707@discworld.dyndns.org>

Greg Ward <gward@python.net> wrote:

> Delivery-date is added by the MTA at delivery time, at least if your MTA
> is qmail or Exim using a router with the add_delivery_date option.

Something not quite accurate there.  qmail-local (qmail's included MDA)
doesn't speak MH and doesn't add a Delivery-Date: header field; it speaks
Maildir, in which case it adds Return-Path: and Delivered-To:, and it speaks
mbox format, which adds those two plus the standard "From " line.  I can't
speak for procmail or other MDAs you might be invoking.

Charles
-- 
-----------------------------------------------------------------------
Charles Cazabon                 <python-spambayes@discworld.dyndns.org>
GPL'ed software available at:     http://www.qcc.ca/~charlesc/software/
-----------------------------------------------------------------------

From Alexander@Leidinger.net  Thu Sep 26 16:39:32 2002
From: Alexander@Leidinger.net (Alexander Leidinger)
Date: Thu, 26 Sep 2002 17:39:32 +0200
Subject: [Spambayes] Traceback from hammie.py
Message-ID: <20020926173932.37ba8a3b.Alexander@Leidinger.net>

Hi,

am I doing something wrong?

---snip---
(83) netchild@ttyp3 % ./hammie.py -d -g Data/Ham/Set1 -s Data/Spam/Set1 -u Data/Ham/Set10
Training ham:
   958
Training spam:
   174
Traceback (most recent call last):
  File "./hammie.py", line 313, in ?
    main()
  File "./hammie.py", line 310, in main
    score(bayes, unknown)
  File "./hammie.py", line 214, in score
    prob, clues = bayes.spamprob(tokenize(msg), True)
  File "./classifier.py", line 550, in robinson_spamprob
    distance = abs(prob - 0.5)
TypeError: unsupported operand type(s) for -: 'NoneType' and 'float'

(84) netchild@ttyp3 % <bayescustomize.ini 
[Classifier]
use_robinson_probability: True
use_robinson_combining: True
use_robinson_ranking: True
max_discriminators: 1500
robinson_minimum_prob_strength: 0.1
[TestDriver]
spam_cutoff: 0.575
---snip---

Bye,
Alexander.

-- 
                   It's not a bug, it's tradition!

http://www.Leidinger.net                       Alexander @ Leidinger.net
  GPG fingerprint = C518 BC70 E67F 143F BE91  3365 79E2 9C60 B006 3FE7

From guido@python.org  Thu Sep 26 16:46:25 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 26 Sep 2002 11:46:25 -0400
Subject: [Spambayes] There Can Be Only One
In-Reply-To: Your message of "Thu, 26 Sep 2002 09:31:47 MDT."
             <20020926093147.D17707@discworld.dyndns.org> 
References: <20020926135437.GA24403@cthulhu.gerg.ca>
	<200209261405.g8QE5w619370@localhost.localdomain>
	<20020926151144.GC24403@cthulhu.gerg.ca>  
	<20020926093147.D17707@discworld.dyndns.org> 
Message-ID: <200209261546.g8QFkPS29452@odiug.zope.com>

> Greg Ward <gward@python.net> wrote:
> 
> > Delivery-date is added by the MTA at delivery time, at least if your MTA
> > is qmail or Exim using a router with the add_delivery_date option.
> 
> Something not quite accurate there.  qmail-local (qmail's included MDA)
> doesn't speak MH and doesn't add a Delivery-Date: header field; it speaks
> Maildir, in which case it adds Return-Path: and Delivered-To:, and it speaks
> mbox format, which adds those two plus the standard "From " line.  I can't
> speak for procmail or other MDAs you might be invoking.

I ran a test.  The mailbox started as follows:

>From guido@odiug.zope.com  Thu Sep 26 11:42:31 2002
Return-Path: <guido@odiug.zope.com>
Received: from localhost (localhost.localdomain [127.0.0.1])
	by odiug.zope.com (8.11.0/8.11.0) with ESMTP id g8QFgV529429
	for <guido@localhost>; Thu, 26 Sep 2002 11:42:31 -0400

Then I ran MH's "inc".  The MH file started as follows:

Return-Path: guido@odiug.zope.com
Delivery-Date: Thu Sep 26 11:42:31 2002
Return-Path: <guido@odiug.zope.com>
Received: from localhost (localhost.localdomain [127.0.0.1])
	by odiug.zope.com (8.11.0/8.11.0) with ESMTP id g8QFgV529429
	for <guido@localhost>; Thu, 26 Sep 2002 11:42:31 -0400

I.e. the address and timestamp were copied exactly from the Unix From
line.

Of course, other programs that touch mail can add these headers too,
as witnessed by the double Return-Path.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.one@comcast.net  Thu Sep 26 17:28:35 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 26 Sep 2002 12:28:35 -0400
Subject: [Spambayes] Re: There Can Be Only One
In-Reply-To: <20020926150908.GB24403@cthulhu.gerg.ca>
Message-ID: <BIEJKCLHCIOIHAGOKOLHOEBPDLAA.tim.one@comcast.net>

[Greg Ward, on the "most embarrassing" fn in my last full-test run]

> Just for grins, I ran this one through SpamAssassin 2.41 (the latest,
> but not yet on mail.python.org).  SA had no trouble calling it spam:
> ...
> That give you any ideas for tokenization hacks?

This is an excellent idea.  Note that many of the clues came from the
headers, and by default most of our header tokenization is turned off.  I
hope other people jump on the things where we could be mining more clues --
I'm spead too thin.

> SPAM: ---- Start SpamAssassin results
> SPAM: 20.30 hits, 5 required;
> SPAM: *  2.2 -- From: has a malformed address

We have no code to catch that.

> SPAM: *  1.7 -- Message-Id has no @ sign

Ditto.

> SPAM: *  1.6 -- Invalid Date: header (not RFC 2822)

Ditto.

> SPAM: *  1.2 -- Message-Id is not valid, according to RFC 2822

Ditto.

> SPAM: *  1.1 -- Header with all capitals found

We caught several of those in this msg, via Anthony's case-sensitive
header-counting code, restricted (by default) to the list of header fields
in option safe_headers.  Note that we don't say they're ham or spam:  as
with all things, we merely produce "words", and the system learns on its own
which tokens are hammish, which spammish, and which neutral.

> SPAM: *  4.3 -- BODY: Claims you can be removed from the list

We caught clues related to this.

> SPAM: *  1.0 -- BODY: List removal information

Ditto.

> SPAM: * -0.1 -- BODY: Spam phrases score is 08 to 13 (medium)
> SPAM:           [score: 10]

This scheme doesn't look for phrases, although it could be taught to.  Note
that I've gotten *worse* results via using word bigrams (closer to
"phrases"), although a comment in tokenizer.py details a scheme that, at
least at the time, gave a major improvement in f-n rate via producing
bigrams and unigrams, and doing a tiny bit of lemmatization (just stripping
a trailing 's'); that also used runs of alphanumerics instead of splitting
on whitespace.  It hurt the f-p rate, though.  The tradeoffs here may be
different under Gary's scheme, so "every time you change anything all
previous decisions should be revisited" applies, as always.

> SPAM: *  1.0 -- URI: Includes a link to a likely spammer email address

We do a lot of special tokenization of URIs already, and I expect the system
has already learned more about spammish things in URIs than people will ever
figure out by thinking about it.

> SPAM: *  2.7 -- Date: is 24 to 48 hours before Received: date

We have no handle on that.

> SPAM: *  1.4 -- Missing To: header

We currently infer hammishness by the presence of a To: header, but don't
generate a token indicating the absence of a To: header.  Jeremy has
suggested before that we do both; I thought he had checked in an option to
do so, but I can't seem to find it.

> SPAM: *  2.2 -- RBL: Received via a relay in orbs.dorkslayers.com
> SPAM:           [RBL check: found 4.90.52.217.orbs.dorkslayers.com.]

Neil's mine_received_headers option (off by default) tokenizes IPs and their
prefixes, and will learn about networks associated with spam on its own.
However, the algorithm isn't going to learn *swiftly* when a new source
appears.  Given the error rates I'm seeing without looking at received
headers, though, I'm not sure I care <wink>.

> SPAM:
> SPAM: ---- End of SpamAssassin results

That's where we win in the end:  it's far from the end of the stuff this
scheme looks at, and this scheme learns about stuff nobody would ever dream
of teaching it.


From tim.one@comcast.net  Thu Sep 26 17:55:39 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 26 Sep 2002 12:55:39 -0400
Subject: [Spambayes] Re: Mailbox class in the spambayes project &
 python2.2.1
In-Reply-To: <20020926122156.2021e446.Alexander@Leidinger.net>
Message-ID: <BIEJKCLHCIOIHAGOKOLHCECBDLAA.tim.one@comcast.net>

[Alexander Leidinger]
> ...
> Now I want to clean up some unparseable messages (mostly some missing
> '>' in front of some "From " lines).

See the 'cleanarch' script, already in the project.

From gward@python.net  Thu Sep 26 17:59:11 2002
From: gward@python.net (Greg Ward)
Date: Thu, 26 Sep 2002 12:59:11 -0400
Subject: [Spambayes] Re: There Can Be Only One
In-Reply-To: <BIEJKCLHCIOIHAGOKOLHOEBPDLAA.tim.one@comcast.net>
References: <20020926150908.GB24403@cthulhu.gerg.ca>
	<BIEJKCLHCIOIHAGOKOLHOEBPDLAA.tim.one@comcast.net>
Message-ID: <20020926165911.GA25963@cthulhu.gerg.ca>

On 26 September 2002, Tim Peters said:
> > SPAM: *  2.2 -- RBL: Received via a relay in orbs.dorkslayers.com
> > SPAM:           [RBL check: found 4.90.52.217.orbs.dorkslayers.com.]
> 
> Neil's mine_received_headers option (off by default) tokenizes IPs and their
> prefixes, and will learn about networks associated with spam on its own.
> However, the algorithm isn't going to learn *swiftly* when a new source
> appears.  Given the error rates I'm seeing without looking at received
> headers, though, I'm not sure I care <wink>.

I suspect the best thing to do here, at least for mail.python.org, is
have Exim do a bunch of RBL lookups, and add X-RBL-Warning headers for
each hit.  Count the number of X-RBL-Warning headers and you've probably
got a pretty good clue.  I'm going to enable that before the next
harvesting run.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
God is real, unless declared integer.

From gward@python.net  Thu Sep 26 18:11:16 2002
From: gward@python.net (Greg Ward)
Date: Thu, 26 Sep 2002 13:11:16 -0400
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <20020926135437.GA24403@cthulhu.gerg.ca>
References: <BIEJKCLHCIOIHAGOKOLHGEAMDLAA.tim.one@comcast.net>
	<LNBBLJKPBEHFEDALKOLCMEMDBGAB.tim.one@comcast.net>
	<20020926022740.GA24111@cthulhu.gerg.ca>
	<20020926135437.GA24403@cthulhu.gerg.ca>
Message-ID: <20020926171116.GA26049@cthulhu.gerg.ca>

On 26 September 2002, I said:
> Off to re-run things, without mine_received and with a better
> spam_cutoff.

Right, as promised.  Corpus as before, with some late-discovered FPs/FNs
moved around.

Here are the tokenization options used for both runs:

[Tokenizer]
basic_header_tokenize: True
basic_header_skip: received envelope-to delivered-to delivery-date
mine_received_headers: True

And here are the additional options for my Robinson run:

[Classifier]
use_robinson_combining: True
use_robinson_probability: True
robinson_probability_x: 0.5
robinson_probability_a: 0.5
max_discriminators: 150
robinson_minimum_prob_strength: 0.1

[TestDriver]
spam_cutoff: 0.525

...ie. nothing special there.  Turns out that I don't benefit from
spam_cutoff < 0.5 after all, as I initially thought.  Oh well.

Comparison stats, Graham vs. Robinson:

"""
run1.logs.txt -> run2.logs.txt
[...]
false positive percentages
    0.000  0.500  lost  +(was 0)
    0.000  0.000  tied          
    0.000  0.000  tied          
    0.500  1.500  lost  +200.00%
    0.000  0.000  tied          
    1.000  1.000  tied          
    0.000  0.000  tied          
    0.000  0.000  tied          
    0.500  0.500  tied          
    1.000  2.500  lost  +150.00%

won   0 times
tied  7 times
lost  3 times

total unique fp went from 6 to 12 lost  +100.00%
mean fp % went from 0.3 to 0.6 lost  +100.00%

false negative percentages
    2.000  1.000  won    -50.00%
    2.000  1.500  won    -25.00%
    1.000  0.500  won    -50.00%
    1.500  1.000  won    -33.33%
    1.000  0.500  won    -50.00%
    0.000  0.500  lost  +(was 0)
    1.000  1.000  tied          
    1.500  1.000  won    -33.33%
    1.000  1.000  tied          
    1.000  1.000  tied          

won   6 times
tied  3 times
lost  1 times

total unique fn went from 24 to 18 won    -25.00%
mean fn % went from 1.2 to 0.9 won    -25.00%

ham mean                     ham sdev
   0.26   20.89 +7934.62%        3.61    8.87 +145.71%
   0.00   21.18 +(was 0)        0.00    9.41 +(was 0)
   0.00   20.31 +(was 0)        0.00    8.06 +(was 0)
   0.50   21.89 +4278.00%        7.05   10.04  +42.41%
   0.00   20.76 +(was 0)        0.00    8.86 +(was 0)
   1.07   20.76 +1840.19%        9.99   10.02   +0.30%
   0.00   19.45 +(was 0)        0.01    8.44 +84300.00%
   0.00   19.71 +(was 0)        0.00    8.59 +(was 0)
   0.50   19.29 +3758.00%        7.05    8.01  +13.62%
   1.09   21.95 +1913.76%       10.03   10.91   +8.77%

ham mean and sdev for all runs
   0.34   20.62 +5964.71%        5.61    9.21  +64.17%

spam mean                    spam sdev
  98.42   82.43  -16.25%       11.93   10.95   -8.21%
  98.22   81.67  -16.85%       12.75   11.33  -11.14%
  99.00   81.33  -17.85%        9.94   11.72  +17.91%
  98.50   81.86  -16.89%       12.15   11.23   -7.57%
  99.00   81.69  -17.48%        9.95   11.41  +14.67%
  99.99   81.67  -18.32%        0.08   10.42 +12925.00%
  99.00   81.90  -17.27%        9.95   10.60   +6.53%
  98.50   81.07  -17.70%       12.16   10.82  -11.02%
  99.00   80.47  -18.72%        9.94   11.33  +13.98%
  99.00   81.96  -17.21%        9.95   11.14  +11.96%

spam mean and sdev for all runs
  98.86   81.60  -17.46%       10.48   11.11   +6.01%

ham/spam mean difference: 98.52 60.98 -37.54
"""

Bottom line: Graham is better at avoiding FPs, Robinson is better at
avoiding FNs.  Great.  At least there's an obvious middle ground with
Robinson.  Here are my histograms, Graham first (nothing unusual here):

-> <stat> Ham scores for all runs: 2000 items; mean 0.34; sdev 5.61
* = 34 items
  0.00 1991 ***********************************************************
  2.50    0 
  5.00    0 
  7.50    0 
 10.00    0 
 12.50    1 *
 15.00    0 
 17.50    1 *
 20.00    0 
 22.50    0 
 25.00    0 
 27.50    0 
 30.00    0 
 32.50    0 
 35.00    0 
 37.50    0 
 40.00    0 
 42.50    0 
 45.00    0 
 47.50    0 
 50.00    1 *
 52.50    0 
 55.00    0 
 57.50    0 
 60.00    0 
 62.50    0 
 65.00    0 
 67.50    0 
 70.00    0 
 72.50    0 
 75.00    0 
 77.50    0 
 80.00    0 
 82.50    0 
 85.00    0 
 87.50    0 
 90.00    0 
 92.50    0 
 95.00    0 
 97.50    6 *

-> <stat> Spam scores for all runs: 2000 items; mean 98.86; sdev 10.48
* = 33 items
  0.00   21 *
  2.50    0 
  5.00    1 *
  7.50    0 
 10.00    0 
 12.50    0 
 15.00    0 
 17.50    0 
 20.00    0 
 22.50    0 
 25.00    0 
 27.50    0 
 30.00    0 
 32.50    0 
 35.00    0 
 37.50    0 
 40.00    0 
 42.50    1 *
 45.00    0 
 47.50    0 
 50.00    0 
 52.50    0 
 55.00    0 
 57.50    0 
 60.00    0 
 62.50    0 
 65.00    0 
 67.50    0 
 70.00    0 
 72.50    0 
 75.00    0 
 77.50    0 
 80.00    0 
 82.50    0 
 85.00    1 *
 87.50    0 
 90.00    1 *
 92.50    0 
 95.00    0 
 97.50 1975 ************************************************************
-> best cutoff for all runs: 0.2
->     with 7 fp + 22 fn = 29 mistakes

and now Robinson (not nearly as pretty as Tim's, but still quite
tolerable):

-> <stat> Ham scores for all runs: 2000 items; mean 20.62; sdev 9.21
* = 5 items
  0.00   0 
  2.50  24 *****
  5.00  79 ****************
  7.50 125 *************************
 10.00 175 ***********************************
 12.50 162 *********************************
 15.00 199 ****************************************
 17.50 245 *************************************************
 20.00 239 ************************************************
 22.50 219 ********************************************
 25.00 147 ******************************
 27.50 114 ***********************
 30.00  85 *****************
 32.50  52 ***********
 35.00  42 *********
 37.50  27 ******
 40.00  15 ***
 42.50  11 ***
 45.00  12 ***
 47.50   9 **
 50.00   7 **
 52.50   5 *
 55.00   2 *
 57.50   4 *
 60.00   0 
 62.50   1 *
 65.00   0 
 67.50   0 
 70.00   0 
 72.50   0 
 75.00   0 
 77.50   0 
 80.00   0 
 82.50   0 
 85.00   0 
 87.50   0 
 90.00   0 
 92.50   0 
 95.00   0 
 97.50   0 

-> <stat> Spam scores for all runs: 2000 items; mean 81.60; sdev 11.11
* = 4 items
  0.00   0 
  2.50   0 
  5.00   0 
  7.50   0 
 10.00   0 
 12.50   0 
 15.00   0 
 17.50   0 
 20.00   0 
 22.50   0 
 25.00   0 
 27.50   0 
 30.00   0 
 32.50   0 
 35.00   0 
 37.50   0 
 40.00   2 *
 42.50   4 *
 45.00   3 *
 47.50   5 **
 50.00   4 *
 52.50   8 **
 55.00  15 ****
 57.50  23 ******
 60.00  31 ********
 62.50  49 *************
 65.00  78 ********************
 67.50 114 *****************************
 70.00 137 ***********************************
 72.50 139 ***********************************
 75.00 114 *****************************
 77.50 112 ****************************
 80.00 117 ******************************
 82.50 139 ***********************************
 85.00 162 *****************************************
 87.50 190 ************************************************
 90.00 213 ******************************************************
 92.50 140 ***********************************
 95.00  88 **********************
 97.50 113 *****************************
-> best cutoff for all runs: 0.525
->     with 12 fp + 18 fn = 30 mistakes

Chew on that, stat-boy!

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Do radioactive cats have 18 half-lives?

From tim.one@comcast.net  Thu Sep 26 18:16:06 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 26 Sep 2002 13:16:06 -0400
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <15762.63895.199782.41444@12-248-11-90.client.attbi.com>
Message-ID: <BIEJKCLHCIOIHAGOKOLHOECBDLAA.tim.one@comcast.net>

[Skip Montanaro]
> Okay, I just created fpfn.ini with just these options:
>
>     [TestDriver]
>     show_false_positives = False
>     show_false_negatives = True
>     show_charlimit = 500000
>
> and ran
> 
>     BAYESCUSTOMIZE=fpfn.ini timcv.py -n10 --ham=200 --spam=200 -s12345
>
> The rather large results are available at either of
>
>     http://manatee.mojam.com/~skip/public_html/fpfn.zip
>     http://manatee.mojam.com/~skip/public_html/fpfn.out
>
> The .out file is about 1.5MB.  The .zip file about 450KB.

No joy:

    Not Found
The requested URL /~skip/public_html/fpfn.zip was not found on this server.

Likewise for the .out file.

From bkc@murkworks.com  Thu Sep 26 18:25:51 2002
From: bkc@murkworks.com (Brad Clements)
Date: Thu, 26 Sep 2002 13:25:51 -0400
Subject: [Spambayes] \r\n endings
In-Reply-To: <200209261425.g8QEPxW25892@odiug.zope.com>
References: Your message of "Thu, 26 Sep 2002 01:46:13 PDT."
	<E17uUHl-0004lh-00@usw-pr-cvs1.sourceforge.net> 
Message-ID: <3D930ADB.26536.99FDFB3@localhost>

Guilty, sorry.

I'm using Pythonwin via Samba, but checking in using cvs on linux. Hence, I never saw 
the \r's, and cvs diff didn't show me adding \r's when I modified TestDriver.py

Anyone know the best way to keep this from happening in the future? Pythonwin 
doesn't seem to have an option to control line endings.





On 26 Sep 2002 at 10:25, Guido van Rossum wrote:

> > 	HistToGNU.py 
> > Log Message:
> > Converted \r\n line endings to \n.
> 
> *Someone* out there is using a broken editor or CVS client that checks
> files in with \r\n line endings.  I've cleaned one of these up, and I've
> seen several others do this on occasion.  I think it may be the editor
> because it appeared that only *some* line endings (those of changed lines?)
> used \r\n, while others used \n.
> 
> I think it's bkc.
> 
> Whoever you are, can you try not to do this?
> 
> --Guido van Rossum (home page: http://www.python.org/~guido/)
> 
> 


Brad Clements,                bkc@murkworks.com   (315)268-1000
http://www.murkworks.com                          (315)268-9812 Fax
AOL-IM: BKClements


From tim.one@comcast.net  Thu Sep 26 18:29:37 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 26 Sep 2002 13:29:37 -0400
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <20020926135437.GA24403@cthulhu.gerg.ca>
Message-ID: <BIEJKCLHCIOIHAGOKOLHMECCDLAA.tim.one@comcast.net>

[Greg Ward]
> ...
> Here are the histograms for run5 (Graham):

Please don't chop off the header line on the histogram.  There's useful info
there.  Also, the individual run histograms aren't *usually* interesting.
The final histogram (which says "all runs" in its header line) is often very
interesting (I think you're actually showing that, but *calling* in "run5").

> ...
> and for run7 (Robinson f(w)):
>
> -> <stat> Ham scores for all runs: 2000 items; mean 20.38; sdev 9.18
> * = 5 items
>   0.00   0
>   2.50  41 *********
>   5.00  96 ********************
>   7.50 144 *****************************
>  10.00 148 ******************************
>  12.50 146 ******************************
>  15.00 206 ******************************************
>  17.50 202 *****************************************
>  20.00 253 ***************************************************
>  22.50 212 *******************************************
>  25.00 173 ***********************************
>  27.50 124 *************************
>  30.00  77 ****************
>  32.50  51 ***********
>  35.00  41 *********
>  37.50  26 ******
>  40.00  13 ***
>  42.50  13 ***
>  45.00  11 ***
>  47.50  11 ***
>  50.00   4 *
>  52.50   4 *
>  55.00   1 *
>  57.50   3 *
>  60.00   0
>  62.50   0
>  65.00   0
>  67.50   0
>  70.00   0
>  72.50   0
>  75.00   0
>  77.50   0
>  80.00   0
>  82.50   0
>  85.00   0
>  87.50   0
>  90.00   0
>  92.50   0
>  95.00   0
>  97.50   0
>
> -> <stat> Spam scores for all runs: 2000 items; mean 79.56; sdev 11.40
> * = 3 items
>   0.00   0
>   2.50   0
>   5.00   0
>   7.50   0
>  10.00   0
>  12.50   0
>  15.00   0
>  17.50   0
>  20.00   0
>  22.50   0
>  25.00   0
>  27.50   0
>  30.00   0
>  32.50   0
>  35.00   0
>  37.50   2 *
>  40.00   1 *
>  42.50   5 **
>  45.00   4 **
>  47.50   6 **
>  50.00   6 **
>  52.50   7 ***
>  55.00  18 ******
>  57.50  41 **************
>  60.00  47 ****************
>  62.50  68 ***********************
>  65.00 105 ***********************************
>  67.50 115 ***************************************
>  70.00 160 ******************************************************
>  72.50 141 ***********************************************
>  75.00 149 **************************************************
>  77.50 128 *******************************************
>  80.00 118 ****************************************
>  82.50 154 ****************************************************
>  85.00 159 *****************************************************
>  87.50 171 *********************************************************
>  90.00 119 ****************************************
>  92.50  85 *****************************
>  95.00  75 *************************
>  97.50 116 ***************************************
> -> best cutoff for all runs: 0.5
> ->     with 12 fp + 18 fn = 30 mistakes
>
> Oops, just noticed the "best cutoff for all runs" thing.  I must have
> misinterpreted the run6 output -- picking 0.475 was an eyeball average.
> D'ohh.

Note that you can also set best_cutoff_fp_weight to tell this histogram
analysis that, e.g., you hate a false positive 10x as much as a false
negative (but you're running older code there; do cvs up first).  Setting
nbuckets higher than the default 40 is helpful here to give the histogram a
finer-grained view of the world.  For example, you have 11 ham and 6 spam in
the .475-.500 bucket, so the finest-grained decision we can make here is
that boosting spam_cutoff to 0.5 would eliminate 11 fp and gain 6 fn.


From skip@pobox.com  Thu Sep 26 18:37:39 2002
From: skip@pobox.com (Skip Montanaro)
Date: Thu, 26 Sep 2002 12:37:39 -0500
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <BIEJKCLHCIOIHAGOKOLHOECBDLAA.tim.one@comcast.net>
References: <15762.63895.199782.41444@12-248-11-90.client.attbi.com>
        <BIEJKCLHCIOIHAGOKOLHOECBDLAA.tim.one@comcast.net>
Message-ID: <15763.17891.382438.372451@12-248-11-90.client.attbi.com>


    >> http://manatee.mojam.com/~skip/public_html/fpfn.zip
    >> http://manatee.mojam.com/~skip/public_html/fpfn.out

    Tim> No joy:

Ack...  Delete the "/public_html" part:

    http://manatee.mojam.com/~skip/fpfn.zip
    http://manatee.mojam.com/~skip/fpfn.out

Sorry for the bogus reference.

Skip

From mwh@python.net  Thu Sep 26 18:28:07 2002
From: mwh@python.net (Michael Hudson)
Date: 26 Sep 2002 18:28:07 +0100
Subject: [Spambayes] Re: There Can Be Only One
References: <20020926150908.GB24403@cthulhu.gerg.ca>
	<BIEJKCLHCIOIHAGOKOLHOEBPDLAA.tim.one@comcast.net>
Message-ID: <lkhegcwti0.fsf@pc150.maths.bris.ac.uk>

Tim Peters <tim.one@comcast.net> writes:

> > SPAM: *  4.3 -- BODY: Claims you can be removed from the list
> 
> We caught clues related to this.

This spamassassin rule is the source of most of my false positives on
the starship.  Occasionally a claim that you can be removed is true!

I'd learn how to customize SA enough to get rid of this rule for my
mail, but I don't think that works on the starship (Greg?).

M.

-- 
  Programming languages should be designed not by piling feature on
  top of feature, but by removing the weaknesses and restrictions
  that make the additional features appear necessary.
               -- Revised(5) Report on the Algorithmic Language Scheme



From Alexander@Leidinger.net  Thu Sep 26 18:45:04 2002
From: Alexander@Leidinger.net (Alexander Leidinger)
Date: Thu, 26 Sep 2002 19:45:04 +0200
Subject: [Spambayes] Re: Mailbox class in the spambayes project &
	python2.2.1
In-Reply-To: <BIEJKCLHCIOIHAGOKOLHCECBDLAA.tim.one@comcast.net>
References: <20020926122156.2021e446.Alexander@Leidinger.net>
	<BIEJKCLHCIOIHAGOKOLHCECBDLAA.tim.one@comcast.net>
Message-ID: <20020926194504.1617cc6d.Alexander@Leidinger.net>

On Thu, 26 Sep 2002 12:55:39 -0400 Tim Peters <tim.one@comcast.net>
wrote:

> [Alexander Leidinger]
> > ...
> > Now I want to clean up some unparseable messages (mostly some
> > missing'>' in front of some "From " lines).
> 
> See the 'cleanarch' script, already in the project.

No, cleanarch damaged the files in the first run. After cleaning them up
(removing the '>' in front of every line which begins with ">From " and
has a date at the end) I now have to find those lines, which need the
'>' now. I already found some in mboxes with upto 90 messages (all of
them had at least one unparseable message) and fixed it.

Here you can see what happens when I use cleanarch (the second one is
filtered with cleanarch):

(43) netchild@ttyp0 % ../../mboxcount.py 20001119.freebsd-stable.txt 20001119.freebsd-stable.txt_ 
20001119.freebsd-stable.txt             529 (+ unparseable: 1)
20001119.freebsd-stable.txt_            368 (+ unparseable: 1)
Total                                   897 (+ unparseable: 2)

And the diff:
--- 20001119.freebsd-stable.txt Thu Sep 26 19:34:36 2002
+++ 20001119.freebsd-stable.txt_        Thu Sep 26 19:35:06 2002
@@ -1,4 +1,4 @@
-From owner-freebsd-stable  Sun Nov 12  0: 6:26 2000
+>From owner-freebsd-stable  Sun Nov 12  0: 6:26 2000
 Delivered-To: freebsd-stable@freebsd.org
 Received: from sol.cc.u-szeged.hu (sol.cc.u-szeged.hu [160.114.8.24])
        by hub.freebsd.org (Postfix) with ESMTP id 720D537B4D7
@@ -124,7 +124,7 @@
 with "unsubscribe freebsd-stable" in the body of the message
 
 
-From owner-freebsd-stable  Sun Nov 12  4: 7:19 2000
+>From owner-freebsd-stable  Sun Nov 12  4: 7:19 2000
 Delivered-To: freebsd-stable@freebsd.org
 Received: from Awfulhak.org (tun.AwfulHak.org [194.242.139.173])
        by hub.freebsd.org (Postfix) with ESMTP id 929C837B479
@@ -319,7 +319,7 @@
 with "unsubscribe freebsd-stable" in the body of the message
 
 
-From owner-freebsd-stable  Sun Nov 12  8: 7:58 2000
+>From owner-freebsd-stable  Sun Nov 12  8: 7:58 2000
 Delivered-To: freebsd-stable@freebsd.org
 Received: from sfinx.lasting.ro (sfinx.lasting.ro [193.230.239.254])
        by hub.freebsd.org (Postfix) with ESMTP id 129E637B479
@@ -403,7 +403,7 @@

All of those falsely "corrected" messages are beginnings of a new
message.

Bye,
Alexander.

-- 
Failure is not an option. It comes bundled with your Microsoft product.

http://www.Leidinger.net                       Alexander @ Leidinger.net
  GPG fingerprint = C518 BC70 E67F 143F BE91  3365 79E2 9C60 B006 3FE7

From gward@python.net  Thu Sep 26 18:54:22 2002
From: gward@python.net (Greg Ward)
Date: Thu, 26 Sep 2002 13:54:22 -0400
Subject: [Spambayes] Re: Mailbox class in the spambayes project &
	python2.2.1
In-Reply-To: <20020926194504.1617cc6d.Alexander@Leidinger.net>
References: <20020926122156.2021e446.Alexander@Leidinger.net>
	<BIEJKCLHCIOIHAGOKOLHCECBDLAA.tim.one@comcast.net>
	<20020926194504.1617cc6d.Alexander@Leidinger.net>
Message-ID: <20020926175422.GB26120@cthulhu.gerg.ca>


---------------------- multipart/mixed attachment
On 26 September 2002, Alexander Leidinger said:
> No, cleanarch damaged the files in the first run. After cleaning them up
> (removing the '>' in front of every line which begins with ">From " and
> has a date at the end) I now have to find those lines, which need the
> '>' now. I already found some in mboxes with upto 90 messages (all of
> them had at least one unparseable message) and fixed it.

The mbox format sucks.  All tools that parse mbox suck; Python's
mailbox.py, however, sucks slightly more than most.  formail is a tool
bundled with procmail; it does a pretty good job of splitting up an mbox
file.  The best use for it is to convert that mbox to a Maildir.  I'll
attach my scripts for converting an mbox to a Maildir.

(BTW, looking at the code, it looks like the attached addtomaildir
script does *not* convert the date in "From " to a Delivery-date header.
It probably should.)

(Oh, Alexander, I think you'll have problems because of the weird 
"From " lines that you showed in your mail.  You'll need to tweak the
regex used to parse "From " lines in addtomaildir if you want to use
it on that mbox file.  Good luck!)

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Eschew obfuscation!

---------------------- multipart/mixed attachment
#!/bin/sh

# Convert an mbox mail file to a maildir.
# Requires formail (from procmail) and my addtomaildir script.
# Idea stolen from
#   http://www.nb.net/~lbudney/linux/software/safecat/one-liners.html
#
# by Greg Ward, 2002/01/22
#
# Usage:
#   mb2md mbox maildir
# where mbox must exist, and maildir must not exist. 

if [ "$#" -ne 2 ] ; then
  echo "usage: $0 mbox maildir" >&2
  exit 1
fi

mbox=$1
maildir=$2

if [ -e "$maildir" ] ; then
  echo "error: $maildir already exists" >&2
  exit 1
fi

if [ ! -e "$mbox" ] ; then
  echo "error: $mbox does not exist" >&2
  exit 1
fi

mkdir -p $maildir/{cur,new,tmp}
formail -s addtomaildir $maildir < $mbox

---------------------- multipart/mixed attachment
#!/usr/bin/env python

"""addtomaildir

Reads an RFC 822 message (possibly with leading "From " line) on stdin
and adds it to a Maildir.  The exact details of where it lands and what
it's called in the Maildir depend on various header values in the input
message:

  * if no "Status" header, the message goes in "new", otherwise in "cur"

  * if "Status" is "O" (old), the filename has no info field

  * if "Status" is "RO" (read old), the filename has ":2,S" appended
    as its info field

  * the mtime of the file will be the delivery time of the message,
    if we can figure out the delivery time.  Tries the "Delivery-date"
    header first, then the "From " line; if neither exists or can
    be parsed, leaves the mtime alone.
"""

import sys, os, re
import socket, errno
from time import time, mktime, strptime, ctime, sleep
from rfc822 import Message, parsedate_tz, mktime_tz



class Error (Exception):
    pass


def warn (msg):
    sys.stderr.write("warning: %s\n" % msg)


def maildir_open (maildir):
    # Assumes we're already chdir'd into maildir

    hostname = socket.gethostname()
    pid = os.getpid()
    num_tries = 0
    max_tries = 5
    while 1:
        name = "tmp/%.6f%05d.%s" % (time(), pid, hostname)
        ok = 0                          # assume the worst
        num_tries += 1
        try:
            os.stat(name)
        except OSError, err:
            # Good: file called 'name' doesn't already exist.
            if err.errno == errno.ENOENT:
                ok = 1

        if ok:
            break
        else:
            if num_tries > max_tries:
                raise Error("error: could not create temporary file in %s/tmp"
                            % maildir)
            sleep(2)               # and try again

    fd = os.open(name, os.O_WRONLY|os.O_EXCL|os.O_CREAT, 0600)
    return (name, fd)


def grok_status (msg):
    # Figure out if this is a new message, an "old" message
    # (seen by MUA, but not read by user), or a read message.
    status = msg.get("Status")
    if status == "O":                   # seen by MUA, but not read by user
        dir = "cur"
        info = ""
    elif status == "RO":                # read by user
        dir = "cur"
        info = ":2,S"
    else:                               # not there, empty, or unknown value
        dir = "new"
        info = ""

    return (dir, info)

def get_delivery_time (msg):
    # Figure out the delivery time.
    dtime = None
    if msg.has_key("Delivery-date"):
        # eg. "Thu, 12 Jul 2001 08:47:20 -0400" to 994942040 (seconds
        # since epoch in UTC)
        dtime = mktime_tz(parsedate_tz(msg["Delivery-date"]))
    elif msg.unixfrom:
        # Parse eg.
        #   "From python-dev-admin@python.org Thu Jul 12 08:47:20 2001"
        # -- this is the "From " line format used by Exim; hopefully other
        # MTAs do the same!
        m = re.match(r'^From (\S+) (\w{3} \w{3}\s+\d\d? \d\d:\d\d:\d\d \d{4})$',
                     msg.unixfrom)
        if not m:
            warn("warning: could not parse \"From \" line: %s" % msg.unixfrom)
        else:
            (return_path, dtime_str) = m.groups()
            # Eg. "Thu Jul 12 08:47:20 2001" -> 994945640 -- note that
            # this might be different from what we get parsing the same
            # date string above, because this one doesn't include the
            # timezone.  Sigh.
            dtime = mktime(strptime(dtime_str, "%c"))

            # Attempt to detect and correct for DST differences.
            # (This works if we parsed a summer time during the winter;
            # what about the inverse?)
            dtime_str_curtz = ctime(dtime)
            if dtime_str_curtz != dtime_str:
                dtime_curtz = mktime(strptime(dtime_str_curtz, "%c"))
                diff = dtime_curtz - dtime
                dtime -= diff

    return dtime

def write_message (msg, msg_file, out_fd):
    # Write the headers to the temp file.
    headers = str(msg) + "\n"
    n = os.write(out_fd, headers)
    if n != len(headers):
        raise Error("failed to write headers (%d/%d bytes written)"
                    % (n, len(headers)))
    
    # Copy the body from msg_file to the temp file.
    chunk = 16*1024
    while 1:
        data = msg_file.read(chunk)
        if not data:
            break
        n = os.write(out_fd, data)
        if n != len(data):
            raise Error("failed to write chunk of body (%d/%d bytes written)"
                        % (n, len(data)))

    # Sync and close the temp file.
    try:
        os.fsync(out_fd)
        os.close(out_fd)
    except OSError, err:
        os.unlink(tmp_name)
        raise Error("unable to fsync() or close() temp file: %s" % err)


def finish_message (tmp_name, dir, info, dtime):

    # Link the temp file to its ultimate destination (in either "new" or
    # "cur", with info appended to the name), and remove the temp name.
    base_name = os.path.basename(tmp_name)
    dst_name = os.path.join(dir, base_name + info)
    os.link(tmp_name, dst_name)

    # Set the modification time to the delivery time, if known.
    if dtime is not None:
        atime = os.stat(dst_name).st_atime
        os.utime(dst_name, (atime, dtime))

    return dst_name

def add (msg_file, maildir):
    # First reserve a place in the maildir (ie. open the file in tmp).
    start_dir = os.getcwd()
    os.chdir(maildir)
    (tmp_name, out_fd) = maildir_open(maildir)

    try:
        msg = Message(msg_file)
        (dir, info) = grok_status(msg)
        dtime = get_delivery_time(msg)
        write_message(msg, msg_file, out_fd)
        dst_name = finish_message(tmp_name, dir, info, dtime)
    finally:
        os.unlink(tmp_name)
        os.chdir(start_dir)

    print dst_name

# add ()


def main ():
    prog = os.path.basename(sys.argv[0])
    args = sys.argv[1:]
    if len(args) == 1:
        maildir = args[0]
        msg_file = sys.stdin
    elif len(args) == 2:
        (msg_filename, maildir) = args
        msg_file = open(msg_filename)
    else:
        sys.exit("usage: %s maildir\n"
                 "       %s msg_file maildir\n"
                 "\n"
                 "error: incorrect number of arguments\n")

    if not (os.path.isdir(maildir) and
            os.path.isdir(os.path.join(maildir, "tmp")) and
            os.path.isdir(os.path.join(maildir, "cur")) and
            os.path.isdir(os.path.join(maildir, "new"))):
        sys.exit("error: not a maildir: %s" % maildir)

    try:
        add(msg_file, maildir)
    except Error, err:
        sys.exit(str(err))


main()

---------------------- multipart/mixed attachment--

From skip@pobox.com  Thu Sep 26 19:13:33 2002
From: skip@pobox.com (Skip Montanaro)
Date: Thu, 26 Sep 2002 13:13:33 -0500
Subject: [Spambayes] Re: There Can Be Only One
In-Reply-To: <lkhegcwti0.fsf@pc150.maths.bris.ac.uk>
References: <20020926150908.GB24403@cthulhu.gerg.ca>
        <BIEJKCLHCIOIHAGOKOLHOEBPDLAA.tim.one@comcast.net>
        <lkhegcwti0.fsf@pc150.maths.bris.ac.uk>
Message-ID: <15763.20045.603953.609496@12-248-11-90.client.attbi.com>

    >> > SPAM: *  4.3 -- BODY: Claims you can be removed from the list

    Michael> This spamassassin rule is the source of most of my false
    Michael> positives on the starship.  Occasionally a claim that you can
    Michael> be removed is true!

    Michael> I'd learn how to customize SA enough to get rid of this rule
    Michael> for my mail, but I don't think that works on the starship
    Michael> (Greg?).

Not sure how things are set up on the starship, but you might try creating a
~/.spamassassin/user_prefs file that contains this line:

    score EXCUSE_8 1.0

There are several other EXCUSE_* rules which match that message, but only
EXCUSE_8 has a default score near 4.0.

Skip

From Alexander@Leidinger.net  Thu Sep 26 19:28:23 2002
From: Alexander@Leidinger.net (Alexander Leidinger)
Date: Thu, 26 Sep 2002 20:28:23 +0200
Subject: [Spambayes] Re: Mailbox class in the spambayes project &
	python2.2.1
In-Reply-To: <20020926175422.GB26120@cthulhu.gerg.ca>
References: <20020926122156.2021e446.Alexander@Leidinger.net>
	<BIEJKCLHCIOIHAGOKOLHCECBDLAA.tim.one@comcast.net>
	<20020926194504.1617cc6d.Alexander@Leidinger.net>
	<20020926175422.GB26120@cthulhu.gerg.ca>
Message-ID: <20020926202823.11f1755e.Alexander@Leidinger.net>

On Thu, 26 Sep 2002 13:54:22 -0400 Greg Ward <gward@python.net> wrote:

> The mbox format sucks.  All tools that parse mbox suck; Python's
> mailbox.py, however, sucks slightly more than most.  formail is a tool
> bundled with procmail; it does a pretty good job of splitting up an
> mbox file.  The best use for it is to convert that mbox to a Maildir. 
> I'll attach my scripts for converting an mbox to a Maildir.

Shouldn't I first fix the From lines (those which need to have a '>' in
front of them)?

> (Oh, Alexander, I think you'll have problems because of the weird 
> "From " lines that you showed in your mail.  You'll need to tweak the
> regex used to parse "From " lines in addtomaildir if you want to use
> it on that mbox file.  Good luck!)

I already have all of the mboxes converted into the standard test setup
layout with splitndirs.py (957750 messages in 1000 directories after a
3h run). At the moment I'm having fun as a spam-hunter...

What I want to do now: fix the original mboxes (~200 unparseable
messages total), split them up into the test layout again, automagically
find those messages which I already identified as spam (either with
hammie.py or by some grep magic) and move them into a spam directory.

Bye,
Alexander.

-- 
                      Loose bits sink chips.

http://www.Leidinger.net                       Alexander @ Leidinger.net
  GPG fingerprint = C518 BC70 E67F 143F BE91  3365 79E2 9C60 B006 3FE7

From nas@python.ca  Thu Sep 26 19:35:18 2002
From: nas@python.ca (Neil Schemenauer)
Date: Thu, 26 Sep 2002 11:35:18 -0700
Subject: [Spambayes] Re: There Can Be Only One
In-Reply-To: <BIEJKCLHCIOIHAGOKOLHOEBPDLAA.tim.one@comcast.net>
References: <20020926150908.GB24403@cthulhu.gerg.ca>
	<BIEJKCLHCIOIHAGOKOLHOEBPDLAA.tim.one@comcast.net>
Message-ID: <20020926183517.GB17272@glacier.arctrix.com>

Tim Peters wrote:
> > SPAM: *  2.2 -- From: has a malformed address
> 
> We have no code to catch that.

Perhaps we could generate a token based on the form of the To and From
addresses (e.g. if they include a real name and are valid based on
RFCs).

> > SPAM: *  1.7 -- Message-Id has no @ sign
> 
> Ditto.

Again, maybe we could generate tokens based on the form of the
Message-Ids.  Common MTAs and MTUs have specific ways of generating
message ids.

> > SPAM: *  1.6 -- Invalid Date: header (not RFC 2822)
> 
> Ditto.
> 
> > SPAM: *  1.2 -- Message-Id is not valid, according to RFC 2822
> 
> Ditto.

Same as above.

  Neil

From barry@zope.com  Thu Sep 26 19:54:14 2002
From: barry@zope.com (Barry A. Warsaw)
Date: Thu, 26 Sep 2002 14:54:14 -0400
Subject: [Spambayes] Re: Mailbox class in the spambayes project &
	python2.2.1
References: <20020926122156.2021e446.Alexander@Leidinger.net>
	<BIEJKCLHCIOIHAGOKOLHCECBDLAA.tim.one@comcast.net>
	<20020926194504.1617cc6d.Alexander@Leidinger.net>
Message-ID: <15763.22486.386646.291054@anthem.wooz.org>


    From owner-freebsd-stable  Sun Nov 12  0: 6:26 2000
-------------------------------------------^^^

This From_ line doesn't match mailbox.UnixMailbox._fromlinepattern so
it looks like a non-delimiting From_ line, thus it gets escaped.

-Barry

From carel.fellinger@chello.nl  Thu Sep 26 20:27:04 2002
From: carel.fellinger@chello.nl (Carel Fellinger)
Date: Thu, 26 Sep 2002 21:27:04 +0200
Subject: [Spambayes] memory consumption...
In-Reply-To: <200209260914.g8Q9EML15849@localhost.localdomain>
References: <LNBBLJKPBEHFEDALKOLCEEMHBGAB.tim.one@comcast.net>
	<200209260914.g8Q9EML15849@localhost.localdomain>
Message-ID: <20020926192704.GA3931@schaduw.felnet>

On Thu, Sep 26, 2002 at 07:14:21PM +1000, Anthony Baxter wrote:
> 
> >>> Tim Peters wrote
> > I didn't find it plausible that a 75MB process would cause swapping to disk
> > on a 256MB box (unless you're running an OS much feebler than Win98 <wink>),
> > but it was very plausible that it's going to disk to read thousands of
> > files.
> 
> Well, it looks like the problem is that this wierd little "linux" OS 
> wants to eat half the RAM in the laptop for "Cached" memory. Gods 
> only know why it's doing that, or how to make it stop doing it. 

I take it that your new to linux? Otherwise ignore my rambling.

Linux uses all its free memory for caching, but only trully free
memory.  So before any swapping starts the cache will shrink to its
bare minimum first.

As Tim noted, it's more likely that you suffer from the many files
that need to be accessed, although that's were the cache should help.
Maybe, just maybe, you've setup your mail-drive to do synchronous
updates?  Use the lsattr command to check it.


-- 
groetjes, carel


From guido@python.org  Thu Sep 26 20:15:05 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 26 Sep 2002 15:15:05 -0400
Subject: [Spambayes] \r\n endings
In-Reply-To: Your message of "Thu, 26 Sep 2002 13:25:51 EDT."
             <3D930ADB.26536.99FDFB3@localhost> 
References: "Your message of Thu, 26 Sep 2002 01:46:13 PDT."
	<E17uUHl-0004lh-00@usw-pr-cvs1.sourceforge.net>  
	<3D930ADB.26536.99FDFB3@localhost> 
Message-ID: <200209261915.g8QJF5q01591@odiug.zope.com>

> Guilty, sorry.
> 
> I'm using Pythonwin via Samba, but checking in using cvs on
> linux. Hence, I never saw the \r's, and cvs diff didn't show me
> adding \r's when I modified TestDriver.py
> 
> Anyone know the best way to keep this from happening in the future?
> Pythonwin doesn't seem to have an option to control line endings.

You could write a script that you run on Linux before you check in.
Here's what I use; I call it "crlf":

#! /usr/bin/env python
import sys, re, os
for file in sys.argv[1:]:
    if os.path.isdir(file):
        print file, "Directory!"
        continue
    data = open(file, "rb").read()
    if '\0' in data:
        print file, "Binary!"
        continue
    newdata = re.sub(r"\r\n", r"\n", data)
    if newdata != data:
        print file
        f = open(file, "wb")
        f.write(newdata)
        f.close()

--Guido van Rossum (home page: http://www.python.org/~guido/)

From Alexander@Leidinger.net  Thu Sep 26 20:19:11 2002
From: Alexander@Leidinger.net (Alexander Leidinger)
Date: Thu, 26 Sep 2002 21:19:11 +0200
Subject: [Spambayes] Re: Mailbox class in the spambayes project &
	python2.2.1
In-Reply-To: <15763.22486.386646.291054@anthem.wooz.org>
References: <20020926122156.2021e446.Alexander@Leidinger.net>
	<BIEJKCLHCIOIHAGOKOLHCECBDLAA.tim.one@comcast.net>
	<20020926194504.1617cc6d.Alexander@Leidinger.net>
	<15763.22486.386646.291054@anthem.wooz.org>
Message-ID: <20020926211911.7f2e54ad.Alexander@Leidinger.net>

On Thu, 26 Sep 2002 14:54:14 -0400 barry@zope.com (Barry A. Warsaw)
wrote:

> 
>     From owner-freebsd-stable  Sun Nov 12  0: 6:26 2000
> -------------------------------------------^^^
> 
> This From_ line doesn't match mailbox.UnixMailbox._fromlinepattern so
> it looks like a non-delimiting From_ line, thus it gets escaped.

Yes. And there are a lot of such lines in the entire archive. IMHO
mailbox.UnixMailbox._fromlinepattern should know about it (be forgiving
what you accept, but strict what you generate).

Bye,
Alexander.

-- 
              To boldly go where I surely don't belong.

http://www.Leidinger.net                       Alexander @ Leidinger.net
  GPG fingerprint = C518 BC70 E67F 143F BE91  3365 79E2 9C60 B006 3FE7

From guido@python.org  Thu Sep 26 20:28:44 2002
From: guido@python.org (Guido van Rossum)
Date: Thu, 26 Sep 2002 15:28:44 -0400
Subject: [Spambayes] Re: Mailbox class in the spambayes project &
	python2.2.1
In-Reply-To: Your message of "Thu, 26 Sep 2002 14:54:14 EDT."
             <15763.22486.386646.291054@anthem.wooz.org> 
References: <20020926122156.2021e446.Alexander@Leidinger.net>
	<BIEJKCLHCIOIHAGOKOLHCECBDLAA.tim.one@comcast.net>
	<20020926194504.1617cc6d.Alexander@Leidinger.net>  
	<15763.22486.386646.291054@anthem.wooz.org> 
Message-ID: <200209261928.g8QJSie01738@odiug.zope.com>

>     From owner-freebsd-stable  Sun Nov 12  0: 6:26 2000
> -------------------------------------------^^^
> 
> This From_ line doesn't match mailbox.UnixMailbox._fromlinepattern so
> it looks like a non-delimiting From_ line, thus it gets escaped.

But all the spambayes code uses the PortableUnixMailbox class, which
doesn't care about this -- any line starting with "From " is
considered a Unix From line.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From barry@zope.com  Thu Sep 26 20:28:22 2002
From: barry@zope.com (Barry A. Warsaw)
Date: Thu, 26 Sep 2002 15:28:22 -0400
Subject: [Spambayes] Re: Mailbox class in the spambayes project &
	python2.2.1
References: <20020926122156.2021e446.Alexander@Leidinger.net>
	<BIEJKCLHCIOIHAGOKOLHCECBDLAA.tim.one@comcast.net>
	<20020926194504.1617cc6d.Alexander@Leidinger.net>
	<15763.22486.386646.291054@anthem.wooz.org>
	<20020926211911.7f2e54ad.Alexander@Leidinger.net>
Message-ID: <15763.24534.459280.398765@anthem.wooz.org>


>>>>> "AL" == Alexander Leidinger <Alexander@leidinger.net> writes:

    AL> Yes. And there are a lot of such lines in the entire
    AL> archive. IMHO mailbox.UnixMailbox._fromlinepattern should know
    AL> about it (be forgiving what you accept, but strict what you
    AL> generate).

That's why I recommend /in general/ people use PortableUnixMailbox to
slog through mbox files.  See this article for an indepth discussion:

http://home.netscape.com/eng/mozilla/2.0/relnotes/demo/content-length.html

-Barry

From skip@pobox.com  Thu Sep 26 20:46:39 2002
From: skip@pobox.com (Skip Montanaro)
Date: Thu, 26 Sep 2002 14:46:39 -0500
Subject: [Spambayes] Re: Mailbox class in the spambayes project &
        python2.2.1
In-Reply-To: <20020926211911.7f2e54ad.Alexander@Leidinger.net>
References: <20020926122156.2021e446.Alexander@Leidinger.net>
        <BIEJKCLHCIOIHAGOKOLHCECBDLAA.tim.one@comcast.net>
        <20020926194504.1617cc6d.Alexander@Leidinger.net>
        <15763.22486.386646.291054@anthem.wooz.org>
        <20020926211911.7f2e54ad.Alexander@Leidinger.net>
Message-ID: <15763.25631.935382.476983@12-248-11-90.client.attbi.com>


    >> This From_ line doesn't match mailbox.UnixMailbox._fromlinepattern so
    >> it looks like a non-delimiting From_ line, thus it gets escaped.

    Alexander> Yes. And there are a lot of such lines in the entire
    Alexander> archive. IMHO mailbox.UnixMailbox._fromlinepattern should
    Alexander> know about it (be forgiving what you accept, but strict what
    Alexander> you generate).

Just switch to mailbox.PortableUnixMailbox.  It only matches on '^From ':

    http://www.python.org/doc/current/lib/module-mailbox.html

Skip


From tim.one@comcast.net  Thu Sep 26 20:51:10 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 26 Sep 2002 15:51:10 -0400
Subject: [Spambayes] Re: There Can Be Only One
In-Reply-To: <20020926183517.GB17272@glacier.arctrix.com>
Message-ID: <BIEJKCLHCIOIHAGOKOLHMECKDLAA.tim.one@comcast.net>

>>> SPAM: *  2.2 -- From: has a malformed address

>> We have no code to catch that.

[Neil]
> Perhaps we could generate a token based on the form of the To and From
> addresses (e.g. if they include a real name and are valid based on
> RFCs).

Sure.  For example,

    yield 'From: has a malformed address'

The "words" we generate can be any strings whatsover.  But you knew that.

>>> SPAM: *  1.7 -- Message-Id has no @ sign

>> Ditto.

> Again, maybe we could generate tokens based on the form of the
> Message-Ids.  Common MTAs and MTUs have specific ways of generating
> message ids.

Ditto.

>>> SPAM: *  1.6 -- Invalid Date: header (not RFC 2822)

>> Ditto.

>>> SPAM: *  1.2 -- Message-Id is not valid, according to RFC 2822

>> Ditto.

> Same as above.

Ditto.  If you want to tackle this, that would be great, and I expect these
are pieces of evidence we could enable regardless of corpus source.  Very
brief msgs are darned hard to score, and I expect there's a world of info in
the headers I'm not getting at now.


From gward@python.net  Thu Sep 26 21:50:19 2002
From: gward@python.net (Greg Ward)
Date: Thu, 26 Sep 2002 16:50:19 -0400
Subject: [Spambayes] Re: Mailbox class in the spambayes project &
	python2.2.1
In-Reply-To: <20020926202823.11f1755e.Alexander@Leidinger.net>
References: <20020926122156.2021e446.Alexander@Leidinger.net>
	<BIEJKCLHCIOIHAGOKOLHCECBDLAA.tim.one@comcast.net>
	<20020926194504.1617cc6d.Alexander@Leidinger.net>
	<20020926175422.GB26120@cthulhu.gerg.ca>
	<20020926202823.11f1755e.Alexander@Leidinger.net>
Message-ID: <20020926205019.GA26433@cthulhu.gerg.ca>

On 26 September 2002, Alexander Leidinger said:
> Shouldn't I first fix the From lines (those which need to have a '>' in
> front of them)?

It depends how smart your mbox parser is.  Neither of the classes
supplied by mailbox.py is smart enough -- UnixMailbox is too strict (it
doesn't like the goofy "hh: m:ss" time format), and PortableUnixMailbox
is too loose (it assumes every occurence of "\nFrom " is a message
delimiter).

The good thing about mailbox.py is that it's really easy to define your
own message delimiter; from a script I was messing on the other day to
deal with concatenation of messages that can hardly be called an "mbox
file":

class MyUnixMailbox (UnixMailbox):
    _fromlinepattern = (r'From .*?(\S+)\s+'
                        r'\w\w\w\s+\w\w\w\s+\d?\d\s+'
                        r'\d?\d:\d\d(:\d\d)?(\s+[^\s]+)?\s+\d\d\d\d\s*$')

If you can come up with a regex to match the real "From " lines in
your mail archive -- in particular the wonky "hh: m:ss" time format --
then drop it in there and away-hey-hey you go.

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
And I wonder ... will Elvis take the place of Jesus in a thousand years?
    -- Dead Kennedys

From gward@python.net  Thu Sep 26 21:51:40 2002
From: gward@python.net (Greg Ward)
Date: Thu, 26 Sep 2002 16:51:40 -0400
Subject: [Spambayes] Re: Mailbox class in the spambayes project &
	python2.2.1
In-Reply-To: <15763.25631.935382.476983@12-248-11-90.client.attbi.com>
References: <20020926122156.2021e446.Alexander@Leidinger.net>
	<BIEJKCLHCIOIHAGOKOLHCECBDLAA.tim.one@comcast.net>
	<20020926194504.1617cc6d.Alexander@Leidinger.net>
	<15763.22486.386646.291054@anthem.wooz.org>
	<20020926211911.7f2e54ad.Alexander@Leidinger.net>
	<15763.25631.935382.476983@12-248-11-90.client.attbi.com>
Message-ID: <20020926205140.GB26433@cthulhu.gerg.ca>

On 26 September 2002, Skip Montanaro said:
> Just switch to mailbox.PortableUnixMailbox.  It only matches on '^From ':

Except PortableUnixMailbox is extremely demanding in that every
non-message-delimiting "\nFrom " must be mangled to "\n>From ",
otherwise it will be mistaken for a delimiter.  Since Alexander's
whole problem is that he has un-mangled "\nFrom " lines in his archive,
PortableUnixMailbox won't work for him.

Have I mentioned that the mbox format sucks?  ;-)

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
What the hell, go ahead and put all your eggs in one basket.

From tim.one@comcast.net  Thu Sep 26 22:34:49 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 26 Sep 2002 17:34:49 -0400
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <20020926171116.GA26049@cthulhu.gerg.ca>
Message-ID: <BIEJKCLHCIOIHAGOKOLHOECPDLAA.tim.one@comcast.net>

[Greg Ward]
> ...
> Here are the tokenization options used for both runs:
>
> [Tokenizer]
> basic_header_tokenize: True
> basic_header_skip: received envelope-to delivered-to delivery-date
> mine_received_headers: True
>
> And here are the additional options for my Robinson run:
>
> [Classifier]
> use_robinson_combining: True
> use_robinson_probability: True
> robinson_probability_x: 0.5
> robinson_probability_a: 0.5

Note that everyone(?) has reported better results with "a" small than this;
e.g., I used 0.225 (or something like that -- read the archives <wink>) for
the last run I reported.  The value of this parameter is important:  it
makes a difference.  This group grope is the first systematic attempt at
tuning it.

> max_discriminators: 150
> robinson_minimum_prob_strength: 0.1
>
> [TestDriver]
> spam_cutoff: 0.525
>
> ...ie. nothing special there.  Turns out that I don't benefit from
> spam_cutoff < 0.5 after all, as I initially thought.  Oh well.
>
> Comparison stats, Graham vs. Robinson:
>
> """
> run1.logs.txt -> run2.logs.txt
> [...]
> false positive percentages
>     0.000  0.500  lost  +(was 0)
>     0.000  0.000  tied
>     0.000  0.000  tied
>     0.500  1.500  lost  +200.00%
>     0.000  0.000  tied
>     1.000  1.000  tied
>     0.000  0.000  tied
>     0.000  0.000  tied
>     0.500  0.500  tied
>     1.000  2.500  lost  +150.00%
>
> won   0 times
> tied  7 times
> lost  3 times
>
> total unique fp went from 6 to 12 lost  +100.00%
> mean fp % went from 0.3 to 0.6 lost  +100.00%
>
> false negative percentages
>     2.000  1.000  won    -50.00%
>     2.000  1.500  won    -25.00%
>     1.000  0.500  won    -50.00%
>     1.500  1.000  won    -33.33%
>     1.000  0.500  won    -50.00%
>     0.000  0.500  lost  +(was 0)
>     1.000  1.000  tied
>     1.500  1.000  won    -33.33%
>     1.000  1.000  tied
>     1.000  1.000  tied
>
> won   6 times
> tied  3 times
> lost  1 times
>
> total unique fn went from 24 to 18 won    -25.00%
> mean fn % went from 1.2 to 0.9 won    -25.00%
>
> ham mean                     ham sdev

Note that means and sdevs aren't interesting when comparing Graham to
Robinson -- they're wildly different in their basic combining approach.

> ...
> Bottom line: Graham is better at avoiding FPs, Robinson is better at
> avoiding FNs.

This has everything to do with your spam_cutoff choice in the Robinson run.
Changing that necessarily improves one error rate at the expense of harming
the other.  So if you *want* fewer f-p and more f-n, raise spam_cutoff.  The
histograms can guide you without needing to rerun the test; increasing
nbuckets helps when there's significant overlap (changing nbuckets doesn't
change any scores, it just allows finer-grained analysis of what would
happen if you were to change spam_cutoff).  See below.

> Great.  At least there's an obvious middle ground with Robinson.

Indeed there is, and it's a good one.  Unfortunately, exactly where it lies
seems quite corpus-dependent.

>
> and now Robinson (not nearly as pretty as Tim's, but still quite
> tolerable):

The last histos I posted covered 20,000 ham and 14,000 spam, and look much
more like normal distributions because of that.  READ MY NOTES inside this
quote of your histograms:

> -> <stat> Ham scores for all runs: 2000 items; mean 20.62; sdev 9.21
> * = 5 items
>   0.00   0
>   2.50  24 *****
>   5.00  79 ****************
>   7.50 125 *************************
>  10.00 175 ***********************************
>  12.50 162 *********************************
>  15.00 199 ****************************************
>  17.50 245 *************************************************
>  20.00 239 ************************************************
>  22.50 219 ********************************************
>  25.00 147 ******************************
>  27.50 114 ***********************
>  30.00  85 *****************
>  32.50  52 ***********
>  35.00  42 *********
>  37.50  27 ******
>  40.00  15 ***
>  42.50  11 ***
>  45.00  12 ***
>  47.50   9 **
>  50.00   7 **
------------------------- 0.525 is your spam_cutoff
>  52.50   5 *
------------------------- if you raised it to 0.55, you'd lose 5 f-p
>  55.00   2 *
------------------------- and if you raised it to 0.575, 5+2 = 7 f-p
>  57.50   4 *
------------------------- and if you raised it to 0.6, 5+2+4 = 11 f-p
                          at which point you'd have only 1 f-p left
>  60.00   0
>  62.50   1 *
>  65.00   0
>  67.50   0
>  70.00   0
>  72.50   0
>  75.00   0
>  77.50   0
>  80.00   0
>  82.50   0
>  85.00   0
>  87.50   0
>  90.00   0
>  92.50   0
>  95.00   0
>  97.50   0
>
> -> <stat> Spam scores for all runs: 2000 items; mean 81.60; sdev 11.11
> * = 4 items
>   0.00   0
>   2.50   0
>   5.00   0
>   7.50   0
>  10.00   0
>  12.50   0
>  15.00   0
>  17.50   0
>  20.00   0
>  22.50   0
>  25.00   0
>  27.50   0
>  30.00   0
>  32.50   0
>  35.00   0
>  37.50   0
>  40.00   2 *
>  42.50   4 *
>  45.00   3 *
>  47.50   5 **
>  50.00   4 *
------------------------------------- your spam_cutoff
>  52.50   8 **
------------------------------------- raised to 0.55, you'd gain 8 f-n
>  55.00  15 ****
>  57.50  23 ******
>  60.00  31 ********
>  62.50  49 *************
>  65.00  78 ********************
>  67.50 114 *****************************
>  70.00 137 ***********************************
>  72.50 139 ***********************************
>  75.00 114 *****************************
>  77.50 112 ****************************
>  80.00 117 ******************************
>  82.50 139 ***********************************
>  85.00 162 *****************************************
>  87.50 190 ************************************************
>  90.00 213 ******************************************************
>  92.50 140 ***********************************
>  95.00  88 **********************
>  97.50 113 *****************************
> -> best cutoff for all runs: 0.525
> ->     with 12 fp + 18 fn = 30 mistakes
>
> Chew on that, stat-boy!

I did <wink>.  If you raised your spam_cutoff to 0.55, your error rates
would be almost identical across the two methods, except that the Robinson
scheme's f-p and f-n almost all live in a narrow band.  You're almost
certainly not using the best value for robinson_probability_a.  Note that
the default histogram analysis minimizes fp+fn.  If you hate fp more than
you hate fn (and you've said elsewhere that you do), figure out exactly how
much more you hate it <wink>.  Call that h.  Then the histogram analysis
will minimize

    h * fp + fn

instead if you set option best_cutoff_fp_weight to h.  Increasing nbuckets
will help it make a finer-grained decision.

It would be good to try again with a better value for robinson_probability_a
(although part of this exercise is for you to get your data to tell you
which value works best for it, not to have someone else guess).


From tim.one@comcast.net  Fri Sep 27 01:02:26 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 26 Sep 2002 20:02:26 -0400
Subject: [Spambayes] More HTML strippage.
In-Reply-To: <200209260900.g8Q90e613628@localhost.localdomain>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEPABGAB.tim.one@comcast.net>

[Anthony Baxter]
> I'm seeing email with stuff like:
>
>     <TD
>     style="PADDING-RIGHT: 0.75pt; PADDING-LEFT: 0.75pt;
>     PADDING-BOTTOM: 0.75pt; WIDTH: 15.75pt; PADDING-TOP: 0.75pt"
>     width=21 rowSpan=4>
>
> in it - the current HTML stripping RE doesn't pick this up,
> as it's limited to 128 characters,

That's roughly true.

> and doesn't span lines.

That's not true.  The character class

    [^>]{0,128}

matches everything except a right-angle, including newlines.  I've also
gotten burned by long HTML comments, stuffed with style-sheet like
directives specific to the HTML editor the poster was using.

> I propose we change it from
> ...
> to
>
>     html_re = re.compile(r"""
>         <
>         [^\s<>]      # e.g., don't match 'a < b' or '<<<' or 'i
> << 5' or 'a<>b'
>         [^>]{0,256}? # search for the next '>', but with non-greedy RE
>         >
>     """, re.VERBOSE|re.DOTALL)

The DOTALL doesn't accomplish anything there (simple proof:  there isn't a
dot in the regexp <wink>).

> Will this cause people pain? Is there a better way?

I've got nothing against 256.  There are three things to watch out for:

1. The re engine is prone to blowing the stack if you do minimal
   matches instead, such as you did in the thing you actually
   checked in.

2. Since this gimmick is applied to *all* text, there's a danger of
   consuming legitimate msg text by accident.  That's the primary
   reason I always put bounds on how much text one of these will
   chew up.

3. Multiple passes are expensive; it's better to fold all these
   HTML-ish thingies into a single regexp with alternatives.
   I'll do that.

An alternative is to use a real HTML parser; I'm not going to make time for
that, though.


From tim.one@comcast.net  Fri Sep 27 02:28:49 2002
From: tim.one@comcast.net (Tim Peters)
Date: Thu, 26 Sep 2002 21:28:49 -0400
Subject: [Spambayes] More HTML strippage.
In-Reply-To: <LNBBLJKPBEHFEDALKOLCIEPABGAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCAEPEBGAB.tim.one@comcast.net>

I beefed up the HTML stripping.  This actually redeemed one of my marginal
false positives under the f(w) scheme, leaving it with 2 fp (out of 20,000)
and 18 fn (out of 14,000).  It reduced both the ham and the spam mean scores
a little, increased the ham score variance, and decreased the spam score
variance.

Laugh of the day.  Even an idiot could have identified one of the f-n just
by looking at who it was addressed to:

To: <bait@blast.net>
Cc: <bait@accsoft.com.au>,
        <bait@cac.net>,
        <bait@agrc.com>,
        <bait@acenet.com.au>,
        <bait@europe.com>,
        <bait@access.net.au>,
        <bait@fishhunt.com>,
        <bait@eclipse.net>,
        <bait@australis.net.au>,
        <bait@enterprise.net>,
        <bait@ix.netcom.com>,
        <bait@lakemichiganangler.com>,
        <bait@em.ca>,
        <bait@goldrush.net>,
        <bait@addr.com>,
        <bait@lycosmail.com>,
        <bait99@yahoo.com>,
        <bait@charleston.net>,
        <bait@adnet.aust.com>,
        <bait@dragon.acadiau.ca>

I leave you to guess the nature of this spam by quoting a misleading part of
it <wink>:

"""
Q) One of his favourite expressions was, "Fuck him! I hope he
dies!" Who was he?

A) I'll give you another clue: The former part of the expression
proved all too apt when we finally found out, at his death in 1985,
that he was gay. Yup! You got it! It was Rock Hudson.
"""

prob('favourite') = 0.00556242, btw.  The British aren't known for spamming.


From barry@python.org  Fri Sep 27 04:43:44 2002
From: barry@python.org (Barry A. Warsaw)
Date: Thu, 26 Sep 2002 23:43:44 -0400
Subject: [Spambayes] Re: Mailbox class in the spambayes project &
	python2.2.1
References: <20020926122156.2021e446.Alexander@Leidinger.net>
	<BIEJKCLHCIOIHAGOKOLHCECBDLAA.tim.one@comcast.net>
	<20020926194504.1617cc6d.Alexander@Leidinger.net>
	<15763.22486.386646.291054@anthem.wooz.org>
	<20020926211911.7f2e54ad.Alexander@Leidinger.net>
	<15763.25631.935382.476983@12-248-11-90.client.attbi.com>
	<20020926205140.GB26433@cthulhu.gerg.ca>
Message-ID: <15763.54256.204247.611306@anthem.wooz.org>


>>>>> "GW" == Greg Ward <gward@python.net> writes:

    GW> Except PortableUnixMailbox is extremely demanding in that
    GW> every non-message-delimiting "\nFrom " must be mangled to
    GW> "\n>From ", otherwise it will be mistaken for a delimiter.

That's part of the definition <wink> of mbox format!

-Barry

From tim.one@comcast.net  Fri Sep 27 05:43:17 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 27 Sep 2002 00:43:17 -0400
Subject: [Spambayes] RE: Central Limit Theorem??!!     :)
In-Reply-To: <B9B5C548.166B0%grobinson@transpose.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCOEPJBGAB.tim.one@comcast.net>

[Gary Robinson]
> ...
> I wonder if the errors for lcm are mostly in the region where there
> are a small number of data points, such that the central mean theorem
> isn't really kicking in.
>
> That is, it may be that there is some number n for which f(w)
> gives the best results if the number of non-middling words is < n and
> and lcm gives the best results if the number of non-middling words
> is > n.
>
> That WOULD make a lot of theoretical sense, because for small
> enough n, the central mean theorem is meaningless and can only make
> trouble for us.
>
> Something else for you to test in your copious free time. ;)

This may be very cool!  I speculated about another possibility for errors in
this approach with Guido, and dumping in some instrumentation appears to
confirm it.

First, yes, some errors are due to very low n -- like it only finds 8 words
in an entire msg.  Those are hard to score for any scheme.  But these cases
usually *also* suffer the same problem I'll eventually get around to
revealing <wink>.

Second, here are some *typical* internal z-scores while predicting ham, all
with at least 30 non-middling words (this is just a slice I took from the
output, while it was predicting against 6 known ham):

 zham           zspam
 2.29985206263  -76.3424101961
 0.187039535126 -60.6685540929
 0.16058734364  -43.5223790527
 0.303545599809 -64.5043366748
 2.32811619768  -80.3108808262
 2.08243355217  -56.6967511599

Now if something is 60 spam sdevs away from the spam mean, and 1 ham sdev
away from the ham mean, extreme confidence is surely justified.  While
predicting a spam, it's never so extreme, because the population ham
variance is much larger than the population spam variance, so the z-scores
away from the ham mean simply can't get as large (btw, I believe this is why
it has such a pronounced tendency to err on the false negative side); even
so, extreme confidence is still justified with numbers like these (a typical
slice when predicting against 6 known spam):

 zham           zspam
 -26.1507326771 -0.680077213248
 -28.3253589669  1.10297422272
 -28.3253589669  1.10297422272
 -28.9332374355  1.31350047503
 -26.5203302612 -0.236968008101
 -37.2333822428 -0.722498689497

That brings us to the mistakes, and this seems a *typical* mistake when a
false negative pops up (there are plenty of words here; that's not the
problem):

 zham            zspam
 -17.8741370033  -20.4279646914

The best guess it can make is that it's closer to ham, but let's get real
about this <wink>:  these are honest-to-God probabilities (well, directly
related to honest-to-God probabilities), and at 18 sdevs away from the ham
mean, the system is screaming there's not a chance in hell the msg fits what
it knows about ham.  It's *also* screaming there's not a chance in hell the
msg fits what it knows about spam.

The only rational conclusion to draw is that the system is utterly baffled,
*and knows it*, so should kick such a msg out for manual review.  Not only
"a middle ground", but a principled middle ground where the system itself
knows it has no confidence in its decision, because both outcomes are
astronomically unlikely based on all it knows.

The messages that fall into this class *are* unusual, too!  I'm still
staring at two from the last run trying to decide whether they're really ham
or spam!  A third is one we debated on this list, and it took a google
search for related msgs to decide it was really spam.  That's extremely
cool, if the pattern holds:  nothing else has been so certain about its
uncertainty, and nothing else has pinpointed msgs I'm also uncertain about.

I can't make more time to pursue this now, but it's very exciting.

Another thing that *may* be cool:  the central limit approaches are a bitch
to train over time, because new messages change probabilities, probabilities
change extremes, and that means whenever you add a message then "in theory"
you should go back over all the ham and spam you've ever trained on and grab
what may be new extremes from them (in order to compute new population means
and vars).

But here are the ham and spam population means and vars from two runs on
disjoint random subsets of 2000 ham and 1400 spam (this is the logarithmic
version):

Run 1
hammean  -0.324091120632 hamvar  0.555156484853
spammean -0.104654537681 spamvar 0.121601025545

Run 2
hammean  -0.321945924099 hamvar  0.546761402392
spammean -0.105809267575 spamvar 0.124020283754

They're much the same across runs.  This, combined with the exteme imbalance
in z-scores during "typical" predictions, suggests that it may be possible
to do this *part* of training only once -- if the population means and
variances here have any sort of objective meaning <wink>, they're simply not
going to change much provided they were trained on lots of data to begin
with.


From nas@python.ca  Fri Sep 27 05:59:28 2002
From: nas@python.ca (Neil Schemenauer)
Date: Thu, 26 Sep 2002 21:59:28 -0700
Subject: [Spambayes] RE: Central Limit Theorem??!!     :)
In-Reply-To: <LNBBLJKPBEHFEDALKOLCOEPJBGAB.tim.one@comcast.net>
References: <B9B5C548.166B0%grobinson@transpose.com>
	<LNBBLJKPBEHFEDALKOLCOEPJBGAB.tim.one@comcast.net>
Message-ID: <20020927045928.GA18658@glacier.arctrix.com>

Tim Peters wrote:
>  zham            zspam
>  -17.8741370033  -20.4279646914
> 
> The best guess it can make is that it's closer to ham, but let's get real
> about this <wink>:  these are honest-to-God probabilities (well, directly
> related to honest-to-God probabilities), and at 18 sdevs away from the ham
> mean, the system is screaming there's not a chance in hell the msg fits what
> it knows about ham.  It's *also* screaming there's not a chance in hell the
> msg fits what it knows about spam.

I don't have anything useful to add except to say that this looks very
promising.  I'm an idiot when it comes to statistics but even I know
what 18 standard deviations means.  Also, having a system that generates
a "confidence" value in addition to a "rating" is a huge bonus.

No time to play with it now, unfortunately.  Maybe tomorrow.

  Neil

From anthony@interlink.com.au  Fri Sep 27 06:09:21 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Fri, 27 Sep 2002 15:09:21 +1000
Subject: [Spambayes] RE: Central Limit Theorem??!! :) 
In-Reply-To: <20020927045928.GA18658@glacier.arctrix.com> 
Message-ID: <200209270509.g8R59Mu27508@localhost.localdomain>


>>> Neil Schemenauer wrote
> I don't have anything useful to add except to say that this looks very
> promising.  I'm an idiot when it comes to statistics but even I know
> what 18 standard deviations means. 

I think it means "what the heck???"

> Also, having a system that generates
> a "confidence" value in addition to a "rating" is a huge bonus.

Yes. From an end-user point of view, saying "I trust the system X amount"
(tiny,moderate,huge), and letting it go for it is a pretty neat idea.

Assuming I (and Neil) are understanding you correctly :)

Anthony

From grobinson@transpose.com  Fri Sep 27 11:57:45 2002
From: grobinson@transpose.com (Gary Robinson)
Date: Fri, 27 Sep 2002 06:57:45 -0400
Subject: [Spambayes] RE: Central Limit Theorem??!!     :)
In-Reply-To: <LNBBLJKPBEHFEDALKOLCOEPJBGAB.tim.one@comcast.net>
Message-ID: <B9B9B1E9.169A5%grobinson@transpose.com>

> That brings us to the mistakes, and this seems a *typical* mistake when a
> false negative pops up (there are plenty of words here; that's not the
> problem):
> 
> zham            zspam
> -17.8741370033  -20.4279646914
> 
> The best guess it can make is that it's closer to ham, but let's get real
> about this <wink>:  these are honest-to-God probabilities (well, directly
> related to honest-to-God probabilities), and at 18 sdevs away from the ham
> mean, the system is screaming there's not a chance in hell the msg fits what
> it knows about ham.  It's *also* screaming there's not a chance in hell the
> msg fits what it knows about spam.


I think what you say above makes a lot of sense. Very interesting!

> The only rational conclusion to draw is that the system is utterly baffled,
> *and knows it*, so should kick such a msg out for manual review.  Not only
> "a middle ground", but a principled middle ground where the system itself
> knows it has no confidence in its decision, because both outcomes are
> astronomically unlikely based on all it knows.


Yes. That sounds right!


--Gary


-- 
Gary Robinson
CEO
Transpose, LLC
grobinson@transpose.com
207-942-3463
http://www.emergentmusic.com
http://radio.weblogs.com/0101454


> From: Tim Peters <tim.one@comcast.net>
> Date: Fri, 27 Sep 2002 00:43:17 -0400
> To: Gary Robinson <grobinson@transpose.com>
> Cc: SpamBayes <spambayes@python.org>, Greg Louis <glouis@dynamicro.on.ca>
> Subject: RE: [Spambayes] RE: Central Limit Theorem??!!     :)
> 
> [Gary Robinson]
>> ...
>> I wonder if the errors for lcm are mostly in the region where there
>> are a small number of data points, such that the central mean theorem
>> isn't really kicking in.
>> 
>> That is, it may be that there is some number n for which f(w)
>> gives the best results if the number of non-middling words is < n and
>> and lcm gives the best results if the number of non-middling words
>> is > n.
>> 
>> That WOULD make a lot of theoretical sense, because for small
>> enough n, the central mean theorem is meaningless and can only make
>> trouble for us.
>> 
>> Something else for you to test in your copious free time. ;)
> 
> This may be very cool!  I speculated about another possibility for errors in
> this approach with Guido, and dumping in some instrumentation appears to
> confirm it.
> 
> First, yes, some errors are due to very low n -- like it only finds 8 words
> in an entire msg.  Those are hard to score for any scheme.  But these cases
> usually *also* suffer the same problem I'll eventually get around to
> revealing <wink>.
> 
> Second, here are some *typical* internal z-scores while predicting ham, all
> with at least 30 non-middling words (this is just a slice I took from the
> output, while it was predicting against 6 known ham):
> 
> zham           zspam
> 2.29985206263  -76.3424101961
> 0.187039535126 -60.6685540929
> 0.16058734364  -43.5223790527
> 0.303545599809 -64.5043366748
> 2.32811619768  -80.3108808262
> 2.08243355217  -56.6967511599
> 
> Now if something is 60 spam sdevs away from the spam mean, and 1 ham sdev
> away from the ham mean, extreme confidence is surely justified.  While
> predicting a spam, it's never so extreme, because the population ham
> variance is much larger than the population spam variance, so the z-scores
> away from the ham mean simply can't get as large (btw, I believe this is why
> it has such a pronounced tendency to err on the false negative side); even
> so, extreme confidence is still justified with numbers like these (a typical
> slice when predicting against 6 known spam):
> 
> zham           zspam
> -26.1507326771 -0.680077213248
> -28.3253589669  1.10297422272
> -28.3253589669  1.10297422272
> -28.9332374355  1.31350047503
> -26.5203302612 -0.236968008101
> -37.2333822428 -0.722498689497
> 
> That brings us to the mistakes, and this seems a *typical* mistake when a
> false negative pops up (there are plenty of words here; that's not the
> problem):
> 
> zham            zspam
> -17.8741370033  -20.4279646914
> 
> The best guess it can make is that it's closer to ham, but let's get real
> about this <wink>:  these are honest-to-God probabilities (well, directly
> related to honest-to-God probabilities), and at 18 sdevs away from the ham
> mean, the system is screaming there's not a chance in hell the msg fits what
> it knows about ham.  It's *also* screaming there's not a chance in hell the
> msg fits what it knows about spam.
> 
> The only rational conclusion to draw is that the system is utterly baffled,
> *and knows it*, so should kick such a msg out for manual review.  Not only
> "a middle ground", but a principled middle ground where the system itself
> knows it has no confidence in its decision, because both outcomes are
> astronomically unlikely based on all it knows.
> 
> The messages that fall into this class *are* unusual, too!  I'm still
> staring at two from the last run trying to decide whether they're really ham
> or spam!  A third is one we debated on this list, and it took a google
> search for related msgs to decide it was really spam.  That's extremely
> cool, if the pattern holds:  nothing else has been so certain about its
> uncertainty, and nothing else has pinpointed msgs I'm also uncertain about.
> 
> I can't make more time to pursue this now, but it's very exciting.
> 
> Another thing that *may* be cool:  the central limit approaches are a bitch
> to train over time, because new messages change probabilities, probabilities
> change extremes, and that means whenever you add a message then "in theory"
> you should go back over all the ham and spam you've ever trained on and grab
> what may be new extremes from them (in order to compute new population means
> and vars).
> 
> But here are the ham and spam population means and vars from two runs on
> disjoint random subsets of 2000 ham and 1400 spam (this is the logarithmic
> version):
> 
> Run 1
> hammean  -0.324091120632 hamvar  0.555156484853
> spammean -0.104654537681 spamvar 0.121601025545
> 
> Run 2
> hammean  -0.321945924099 hamvar  0.546761402392
> spammean -0.105809267575 spamvar 0.124020283754
> 
> They're much the same across runs.  This, combined with the exteme imbalance
> in z-scores during "typical" predictions, suggests that it may be possible
> to do this *part* of training only once -- if the population means and
> variances here have any sort of objective meaning <wink>, they're simply not
> going to change much provided they were trained on lots of data to begin
> with.
> 


From tim@zope.com  Fri Sep 27 15:52:47 2002
From: tim@zope.com (Tim Peters)
Date: Fri, 27 Sep 2002 10:52:47 -0400
Subject: Ham or Spam? (was RE: [Spambayes] RE: Central Limit Theorem??!!
	:))
In-Reply-To: <LNBBLJKPBEHFEDALKOLCOEPJBGAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEAOBHAB.tim@zope.com>

This is a multi-part message in MIME format.

---------------------- multipart/mixed attachment
[Tim]
> ...
> That brings us to the mistakes, and this seems a *typical* mistake
> when a false negative pops up (there are plenty of words here; that's
> not the problem):
>
>  zham            zspam
>  -17.8741370033  -20.4279646914
>
> The best guess it can make is that it's closer to ham, but let's get
> real about this <wink>:  these are honest-to-God probabilities (well,
> directly related to honest-to-God probabilities), and at 18 sdevs away
> from the ham mean, the system is screaming there's not a chance in
> hell the msg fits what it knows about ham.  It's *also* screaming
> there's not a chance in hell the msg fits what it knows about spam.

Attached is the msg that produced those scores.  To my eyes, it doesn't look
anything like c.l.py traffic (my ham) *or* like spam -- it looks like a
fishing newletter someone signed up for.  My best guess is that it's really
ham, mistakenly filed as spam in bruceg's collection.  Whaddya think?

---------------------- multipart/mixed attachment
Return-Path: <warden@hotspotsporting.com>=0A=
Delivered-To: bfsmedia-bluegoose.kennels@bfsmedia.com=0A=
Received: (qmail 28626 invoked from network); 29 May 2002 03:23:55 -0000=0A=
Received: from enchanter.real-time.com (208.20.202.11)=0A=
  by churchill.factcomp.com with SMTP; 29 May 2002 03:23:55 -0000=0A=
Received: from www.hotspothunting.com (www.hotspothunting.com =
[65.193.17.228])=0A=
	by enchanter.real-time.com (8.11.6/8.11.6) with ESMTP id g4T3LjU30815=0A=
	for <bluegoose.kennels@bfsmedia.com>; Tue, 28 May 2002 22:21:45 -0500=0A=
Received: from mail pickup service by www.hotspothunting.com with =
Microsoft SMTPSVC;=0A=
	 Tue, 28 May 2002 22:16:39 -0500=0A=
From: "Game Warden" <warden@hotspotsporting.com>=0A=
To: <bluegoose.kennels@bfsmedia.com>=0A=
Subject: Hot Spot Fishing News=0A=
Date: Tue, 28 May 2002 22:16:39 -0500=0A=
MIME-Version: 1.0=0A=
Content-Type: multipart/alternative;=0A=
	boundary=3D"----=3D_NextPart_000_030C_01C20695.5695B220"=0A=
X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3=0A=
Message-ID: <005933916031d52HSHUNTING@www.hotspothunting.com>=0A=
Content-Length: 15420=0A=
Lines: 356=0A=
=0A=
This is a multi-part message in MIME format.=0A=
=0A=
------=3D_NextPart_000_030C_01C20695.5695B220=0A=
Content-Type: text/plain;=0A=
	charset=3D"iso-8859-1"=0A=
Content-Transfer-Encoding: 7bit=0A=
=0A=
Minnesota Fishing=0A=
Pehrson Lodge=0A=
www.pehrsonlodge.com <http://www.pehrsonlodge.com>  =0A=
Hot Spot Fishing Newsletter=0A=
May, 2002=0A=
www.HotSpotFishing.com <http://www.hotspotfishing.com/> =0A=
Newfoundland Fishing=0A=
Gander River Outfitters=0A=
www.ganderriver.com <http://www.ganderriver.com>  =0A=
=0A=
=0A=
* Lunker Tournament Winner! =0A=
Hot Spot Fishing member NMIU from Michigan won our recent online Lunker=0A=
Tournament sponsored by Time Off Outdoors - an online fly shop=0A=
<http://www.timeoffoutdoors.com> . He won a wonderful boxed set of 26=0A=
flies. Mike Chadwell of Time Off Outdoors is offering a 10% discount to=0A=
Hot Spot members on any order you make before June 15th. I've seen their=0A=
flies and they look great. =0A=
 =0A=
We're just about finished with a 'short' Lunker Tournament that will end=0A=
on May 31st. It is sponsored by Pehrson Lodge=0A=
<http://www.pehrsonlodge.com>  in Cook, Minnesota. =0A=
 =0A=
To win, 'catch' the trophy fish (a crappie) that appears on pages of Hot=0A=
Spot Fishing. Click the correct fish to tally another entry in the=0A=
tournament. A random winner is selected, but more entries means more=0A=
chances to win. Just a couple days left to enter. =0A=
* Fishing Quiz =0A=
Only a couple days left before we pick a winner in this month's Fishing=0A=
Quiz. Give it a shot and maybe win a fishing lure. Here's a hint: A=0A=
walleye/sauger hybrid is NOT called BOB. =0A=
* Did You Know? =0A=
The Minnesota record Walleye is 17 lb. 8 oz. and was caught in the=0A=
Seagull River near Cook, Minnesota on May 13, 1979. But, Arkansas,=0A=
Kentucky, Missouri, and Tennessee all have state records over 20 pounds!=0A=
Check out Other Walleye Records=0A=
<http://www.hotspotfishing.com/records/fish-records-walleye.asp> . If=0A=
you'd like to show off fish records on your own web site, read the Free=0A=
Web Data <http://www.hotspotsporting.com/webdata.htm>  page. =0A=
* Catch Anything? =0A=
Now that our fishing reports are ready for you to log your trips, we're=0A=
all anxious to hear what you caught. Take some pictures and keep track=0A=
of how successful you were depending on the weather, time, and tackle.=0A=
Next year, you'll be glad you did when you try to remember what worked. =0A=
 =0A=
Plus, your photos are displayed in the Trophy Gallery and on your Hot=0A=
Spot web page. Not only that, the fishing reports you enter can also be=0A=
displayed on your personal or company web site - wherever it is.  Read=0A=
the Free Web Data <http://www.hotspotsporting.com/webdata.htm>  page. =0A=
* Fishing Articles and Stories =0A=
The Hot Spot Library is open! It's a GREAT way for you to share your=0A=
fishing experience with other Hot Spot members. Log on and click the [MY=0A=
CONTENT] link to get started. Easily create your own Biography and add=0A=
any stories or articles you have to share. We can help you - just ask! =0A=
 =0A=
Hot Spot member Jason Akl recently submitted an article on Building an=0A=
Epoxy Drier for Flies=0A=
<http://www.hotspotfishing.com/public/articles/fish-99.asp>  which is an=0A=
awesome 'How To' article. Tom Lester of Fishing Pro Staff submitted Fall=0A=
Feeding Opportunities=0A=
<http://www.hotspotfishing.com/public/articles/fish-85.asp>  among his=0A=
collection of articles to share. =0A=
 =0A=
Guess what - fishing articles can also be displayed on your personal or=0A=
company web site. =0A=
* Caddis Fly Swap =0A=
We are finishing our Caddis Pattern Fly Swap this week. We have 19 tiers=0A=
for this swap and the flies are looking great!   The photos of these=0A=
flies and their tying instructions are online in the Fly Gallery so you=0A=
can try them yourself. =0A=
 =0A=
You can sign up to receive announcements of new Swaps as they are=0A=
created and we'd love to hear your request for a fly swap theme. =0A=
* Links Please =0A=
A link from your web site to Hot Spot Fishing would greatly help us=0A=
grow. Click Here <http://www.hotspotsporting.com/linktous.htm>  for more=0A=
info. =0A=
 =0A=
Links give people another path to us and helps our ranking on many=0A=
search engines so new anglers can find us. Every Hot Spot member that=0A=
has their own web site already has at least one link from us to you so a=0A=
reciprocal link from you to us would sure be nice. =0A=
 =0A=
To give you yet another link and way for people to find you, we've=0A=
started a new program.   Link Theme Lists are an easy, valuable way for=0A=
you to make your site easier to find and boost your search engine=0A=
rankings. We've just started a Top Fishing Hot Spots list to provide you=0A=
with a fishing-only place to list your site. Check it out at=0A=
www.HotSpotFishing.com/rank/=0A=
<http://www.hotspotfishing.com/rank/default.asp>  =0A=
 =0A=
We published the list to major search engines on Memorial Day and we'll=0A=
do it again on the 4th of July so make sure you are listed before then.=0A=
Any fishing related sites are eligible and the small button you put on=0A=
your web page shows your current rank on the list. =0A=
* Free Content for Your Site =0A=
As I mentioned, you can use Hot Spot Fishing data on your own web site.=0A=
Here's a couple sites that are taking advantage of the free content to=0A=
make their sites more valuable: =0A=
*	Troutlet.com <http://www.troutlet.com/staterecords.htm>  - click=0A=
a state for state records=0A=
*	Fishing with Fitzy <http://www.ausfish.com.au/fitzy/Weather.htm>=0A=
=0A=
It's simple to soup up your site as well. But, of course, if you have=0A=
any trouble or questions, we're happy to help. =0A=
Thanks again for joining the 'Hottest' online fishing community!   Tell=0A=
a Friend and take a kid fishing! =0A=
See you online! Warden@HotSpotSporting.com=0A=
<mailto:warden@hotspotsporting.com>  =0A=
=0A=
=0A=
To receive no more Fishing email from Hot Spot Sporting, log on to your=0A=
favorite Hot Spot community and set your preferences in your Profile=0A=
page. Or, go to: Click Here=0A=
<http://www.hotspotsporting.com/registration/nomail.asp?id=3DVEND-08826> =
 =0A=
Your Hot Spot username is: VEND-08826=0A=
Go to http://www.hotspotsporting.com/securedir/lostpass.asp  to retrieve=0A=
your password. =0A=
=0A=
=0A=
=0A=
------=3D_NextPart_000_030C_01C20695.5695B220=0A=
Content-Type: text/html;=0A=
	charset=3D"iso-8859-1"=0A=
Content-Transfer-Encoding: quoted-printable=0A=
=0A=
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">=0A=
<HTML><HEAD><TITLE>Hot Spot Fishing Newsletter</TITLE>=0A=
=0A=
<style type=3D3D"text/css">=0A=
<!--=0A=
.HE { font-family: Tahoma, Arial, Helvetica; FONT-SIZE: 18px; =3D=0A=
COLOR:"#004669"; FONT-WEIGHT: bold }=0A=
.HE1 { font-family: Tahoma, Arial, Helvetica; FONT-SIZE: 16px; =3D=0A=
COLOR:"#0024ad"; FONT-WEIGHT: bold }=0A=
.nav { font-family: Tahoma, Arial, Helvetica; FONT-SIZE: 12px; =3D=0A=
COLOR:"#004669"; FONT-WEIGHT: bold }=0A=
A { font-family: Tahoma, Arial, Helvetica; text-decoration: underline; =3D=0A=
FONT-SIZE: 12px; }=0A=
P { font-family: Tahoma, Arial, Helvetica; text-decoration: none; =3D=0A=
FONT-SIZE: 12px; }=0A=
-->=0A=
</STYLE>=0A=
=0A=
</HEAD>=0A=
<BODY text=3D3D#000000 vLink=3D3D#400040 aLink=3D3D#0024ad =
link=3D3D#400040 =3D=0A=
bgColor=3D3D#FFFFFF background=3D3D"">=0A=
<A name=3D3D#TOP><FONT face=3D3D"tahoma, arial, helvetica">=0A=
<TABLE width=3D3D"100%">=0A=
  <TBODY>=0A=
  <TR>=0A=
    <TD align=3D3Dcenter >=0A=
<FONT face=3D3D"tahoma, arial, helvetica" color=3D3D#004669>Minnesota =3D=0A=
Fishing<BR>=0A=
<A target=3D3D_blank href=3D3D"http://www.pehrsonlodge.com"><FONT =3D=0A=
face=3D3D"tahoma, arial, helvetica" color=3D3D#004669>Pehrson =3D=0A=
Lodge<BR>www.pehrsonlodge.com</FONT></A></font>=0A=
    </TD>=0A=
    <TD align=3D3Dcenter class=3D3DHE >=0A=
    <P class=3D3DHE >=0A=
    Hot Spot Fishing Newsletter<BR>May, 2002<BR>=0A=
     <A class=3D3DHE =3D=0A=
href=3D3D"http://www.hotspotfishing.com/">www.HotSpotFishing.com</A></P>=0A=
    </TD>=0A=
    <TD align=3D3Dcenter >=0A=
<FONT face=3D3D"tahoma, arial, helvetica" color=3D3D#004669>Newfoundland =
=3D=0A=
Fishing<BR>=0A=
<A target=3D3D_blank href=3D3D"http://www.ganderriver.com"><FONT =3D=0A=
face=3D3D"tahoma, arial, helvetica" color=3D3D#004669>Gander River =3D=0A=
Outfitters<BR>www.ganderriver.com</FONT></A></font>=0A=
    </TD></TR></TBODY></TABLE>=0A=
      <BR>=0A=
<CENTER>=0A=
<TABLE cellPadding=3D3D12 width=3D3D"500" bgColor=3D3D#ffffff =
border=3D3D0>=0A=
  <TBODY>=0A=
  <TR>=0A=
    <TD vAlign=3D3Dtop>=0A=
      <P align=3D3Djustify>=0A=
<SPAN class=3D3DHE1 >* Lunker Tournament Winner! </SPAN>=0A=
<BR><FONT face=3D3D"tahoma, arial, helvetica" size=3D3D2 >=0A=
Hot Spot Fishing member <B>NMIU</B> from Michigan won our recent online =3D=0A=
Lunker Tournament sponsored=0A=
by <A HREF=3D3D"http://www.timeoffoutdoors.com" target=3D3D"_other">Time =
Off =3D=0A=
Outdoors - an online fly shop</A>.  He won a wonderful boxed set of 26=0A=
flies.  Mike Chadwell of Time Off Outdoors is offering a <B>10% =3D=0A=
discount</B> to Hot Spot members on any order you make before June 15th. =
=3D=0A=
 I've seen their flies=0A=
and they look great.=0A=
<BR>&nbsp;<BR>=0A=
We're just about finished with a 'short' Lunker Tournament that will end =
=3D=0A=
on May 31st.  It is sponsored by <A =
HREF=3D3D"http://www.pehrsonlodge.com" =3D=0A=
target=3D3D"_other">Pehrson Lodge</A> in Cook, Minnesota. =3D20=0A=
 <BR>&nbsp;<BR>=0A=
 To win, 'catch' the trophy fish (a crappie) that appears on pages of =3D=0A=
Hot Spot Fishing.  Click the correct fish to=3D20=0A=
 tally another entry in the tournament.  A random winner is selected, =3D=0A=
but more entries means more chances to win.  Just a couple days left to =3D=0A=
enter.=0A=
</FONT></P>=0A=
=0A=
=0A=
<P align=3D3Djustify><SPAN class=3D3DHE1 >* Fishing Quiz</SPAN> =3D20=0A=
<BR><FONT face=3D3D"tahoma, arial, helvetica"  size=3D3D2>=0A=
Only a couple days left before we pick a winner in this month's Fishing =3D=0A=
Quiz.  Give it a shot and maybe win a fishing lure.  Here's a hint:  A =3D=0A=
walleye/sauger hybrid is NOT called BOB.=0A=
</FONT></P>=0A=
               =3D20=0A=
=0A=
<P align=3D3Djustify><SPAN class=3D3DHE1 >* Did You Know?</SPAN> =3D20=0A=
<BR><FONT face=3D3D"tahoma, arial, helvetica"  size=3D3D2>=0A=
The Minnesota record Walleye is 17 lb. 8 oz. and was caught in the =3D=0A=
Seagull River near Cook, Minnesota on May 13, 1979.  But, Arkansas, =3D=0A=
Kentucky, Missouri, and Tennessee all have state records over 20 pounds! =
=3D=0A=
Check out=3D20=0A=
<A =3D=0A=
HREF=3D3D"http://www.hotspotfishing.com/records/fish-records-walleye.asp"=
 =3D=0A=
target=3D3D"_other">Other Walleye Records</A>.  If you'd like to show =
off =3D=0A=
fish records on=0A=
your own web site, read the <a =3D=0A=
href=3D3D"http://www.hotspotsporting.com/webdata.htm" =3D=0A=
target=3D3D"_other">Free Web Data</a> page.=0A=
</FONT></P>=0A=
               =3D20=0A=
=0A=
<P align=3D3Djustify><SPAN class=3D3DHE1 >* Catch Anything?</SPAN> =3D20=0A=
<BR><FONT size=3D3D2 face=3D3D"tahoma, arial, helvetica">=0A=
Now that our fishing reports are ready for you to log your trips, we're =3D=0A=
all anxious to hear what you caught.  Take some pictures and keep track =3D=0A=
of how successful you were depending on the weather, time, and tackle.  =3D=0A=
Next year, you'll be glad you did when you try to remember what worked.  =
=3D=0A=
=0A=
<BR>&nbsp;<BR>Plus, your photos are displayed in the Trophy Gallery and =3D=0A=
on your Hot Spot web page.  Not only that, the fishing reports you enter =
=3D=0A=
can also be displayed on your personal or company web site=3D20=0A=
      - wherever it is.&nbsp; Read the=0A=
      <a href=3D3D"http://www.hotspotsporting.com/webdata.htm" =3D=0A=
target=3D3D"_other">Free Web Data</a> page.=0A=
</FONT></P>=0A=
=0A=
<P align=3D3Djustify><SPAN class=3D3DHE1 >* Fishing Articles and =3D=0A=
Stories</SPAN> =3D20=0A=
<BR><FONT size=3D3D2 face=3D3D"tahoma, arial, helvetica">=0A=
The Hot Spot Library is open! It's a GREAT way for you to share your =3D=0A=
fishing experience with other Hot Spot members. Log on and click the [MY =
=3D=0A=
CONTENT] link to get started. Easily create your own Biography and add =3D=0A=
any stories or articles you have to share. We can help you - just =
ask!=3D20=0A=
<BR>&nbsp;<BR>=0A=
Hot Spot member Jason Akl recently submitted an article on <a =3D=0A=
href=3D3D"http://www.hotspotfishing.com/public/articles/fish-99.asp" =3D=0A=
target=3D3D"_other">Building an Epoxy Drier for Flies</A> which is an =3D=0A=
awesome 'How To' article.  Tom Lester of Fishing Pro Staff submitted <a =3D=0A=
href=3D3D"http://www.hotspotfishing.com/public/articles/fish-85.asp" =3D=0A=
target=3D3D"_other">Fall Feeding Opportunities</A> among his collection =
of =3D=0A=
articles to share.=0A=
<BR>&nbsp;<BR>=0A=
Guess what - fishing articles can also be displayed on your personal or =3D=0A=
company web site.=0A=
</FONT></P>=0A=
=0A=
=0A=
<P align=3D3Djustify> <SPAN class=3D3DHE1 >* Caddis Fly Swap</SPAN>=0A=
<BR><FONT face=3D3D"tahoma, arial, helvetica" size=3D3D2>We are =
finishing =3D=0A=
our Caddis Pattern Fly Swap this week.=0A=
We have 19 tiers for this swap and the flies are looking great!  &nbsp; =3D=0A=
The photos of these flies and their tying instructions are online in the =
=3D=0A=
=0A=
Fly Gallery so you can try them yourself.=0A=
<BR>&nbsp;<BR>=0A=
You can sign up to receive announcements of new Swaps as they are =3D=0A=
created and we'd love to hear your request for a fly swap theme. =3D20=0A=
</FONT>=0A=
 </P>=0A=
=3D20=0A=
=0A=
<P align=3D3Djustify><SPAN class=3D3DHE1 >* Links Please</SPAN>=3D20=0A=
<BR><FONT face=3D3D"tahoma, arial, helvetica" size=3D3D2>A link from =
your =3D=0A=
web site to Hot Spot Fishing would greatly help us grow. =3D20=0A=
<A HREF=3D3D"http://www.hotspotsporting.com/linktous.htm" =3D=0A=
target=3D3D"_other">Click Here</A> for more info.=0A=
<BR>&nbsp;<BR>=0A=
Links give people another path to us and helps our ranking on many =3D=0A=
search engines so new anglers can find us. =3D20=0A=
Every Hot Spot member that has their own web site already has at least =3D=0A=
one link from us to you so a reciprocal link from you to us would sure =3D=0A=
be nice.=0A=
<BR>&nbsp;<BR>=0A=
To give you yet another link and way for people to find you, we've =3D=0A=
started a new program. &nbsp;=0A=
<B>Link Theme Lists</B> are an easy, valuable way for you to make your =3D=0A=
site easier to find and boost your search engine rankings.  We've just =3D=0A=
started=0A=
a Top Fishing Hot Spots list to provide you with a fishing-only place to =
=3D=0A=
list your site.  Check it out at=0A=
<A HREf=3D3D"http://www.hotspotfishing.com/rank/default.asp" =3D=0A=
target=3D3D"_other">www.HotSpotFishing.com/rank/</A>=0A=
<BR>&nbsp;<BR>=0A=
We published the list to major search engines on Memorial Day and we'll =3D=0A=
do it again on the 4th of July so make sure you are listed before then.  =
=3D=0A=
Any fishing related sites are eligible and the=3D20=0A=
small button you put on your web page shows your current rank on the =3D=0A=
list.=0A=
</FONT>   =3D20=0A=
</P>=0A=
      =3D20=0A=
=3D20=0A=
=0A=
<P align=3D3Djustify> <SPAN class=3D3DHE1 >* Free Content for Your =3D=0A=
Site</SPAN>=0A=
<BR><FONT face=3D3D"tahoma, arial, helvetica" size=3D3D2>As I mentioned, =
you =3D=0A=
can use Hot Spot Fishing data on your own web site.  Here's a couple =3D=0A=
sites=0A=
that are taking advantage of the free content to make their sites more =3D=0A=
valuable:=0A=
<ul>=0A=
<li><a href=3D3D"http://www.troutlet.com/staterecords.htm" =3D=0A=
target=3D3D"_other">Troutlet.com</a> - click a state for state =3D=0A=
records</li>=0A=
<li><a href=3D3D"http://www.ausfish.com.au/fitzy/Weather.htm" =3D=0A=
target=3D3D"_other">Fishing with Fitzy</A></li>=0A=
</ul>=0A=
It's simple to soup up your site as well.  But, of course, if you have =3D=0A=
any trouble or questions, we're happy to help.=3D20=0A=
</FONT>=0A=
 </P>=0A=
=3D20=0A=
                  =3D20=0A=
<P><FONT size=3D3D2>Thanks again for joining the 'Hottest' online  =
fishing =3D=0A=
community! &nbsp; Tell a Friend and take a kid fishing! </FONT>          =
=3D=0A=
=0A=
 =3D20=0A=
              =3D20=0A=
<P><FONT size=3D3D2>See you online! </FONT>=0A=
<A href=3D3D"mailto:warden@hotspotsporting.com"><FONT =3D=0A=
size=3D3D2>Warden@HotSpotSporting.com</FONT></A>=0A=
     =3D20=0A=
</TD></TR></TBODY></TABLE><BR>=0A=
=0A=
<table width=3D3D500><tbody><tr><td><P><FONT face=3D3D"tahoma, arial, =3D=0A=
helvetica" size=3D3D-1>To receive no more Fishing email from Hot Spot =3D=0A=
Sporting, log on to your favorite Hot Spot community  and set your =3D=0A=
preferences in your Profile page.  Or, go to:  <A =3D=0A=
HREF=3D3D"http://www.hotspotsporting.com/registration/nomail.asp?id=3D3DV=
END-=3D=0A=
08826" target=3D3D_blank>Click Here</A>=0A=
</P><P>Your Hot Spot username is: VEND-08826<BR>Go to <A =3D=0A=
HREF=3D3D"http://www.hotspotsporting.com/securedir/lostpass.asp">http://w=
ww=3D=0A=
.hotspotsporting.com/securedir/lostpass.asp</A>  to retrieve your =3D=0A=
password. </FONT></TD></TR></TBODY></TABLE><BR></BODY></HTML>=0A=
------=3D_NextPart_000_030C_01C20695.5695B220--=0A=

---------------------- multipart/mixed attachment--


From python-spambayes@discworld.dyndns.org  Fri Sep 27 15:58:44 2002
From: python-spambayes@discworld.dyndns.org (Charles Cazabon)
Date: Fri, 27 Sep 2002 08:58:44 -0600
Subject: Ham or Spam? (was RE: [Spambayes] RE: Central Limit Theorem??!!
	:))
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEAOBHAB.tim@zope.com>;
	from tim@zope.com on Fri, Sep 27, 2002 at 10:52:47AM -0400
References: <LNBBLJKPBEHFEDALKOLCOEPJBGAB.tim.one@comcast.net>
	<LNBBLJKPBEHFEDALKOLCKEAOBHAB.tim@zope.com>
Message-ID: <20020927085844.A26502@discworld.dyndns.org>

Tim Peters <tim@zope.com> wrote:
> 
> Attached is the msg that produced those scores.  To my eyes, it doesn't look
> anything like c.l.py traffic (my ham) *or* like spam -- it looks like a
> fishing newletter someone signed up for.  My best guess is that it's really
> ham, mistakenly filed as spam in bruceg's collection.  Whaddya think?

> Return-Path: <warden@hotspotsporting.com>
> Delivered-To: bfsmedia-bluegoose.kennels@bfsmedia.com

Notice that this is the same envelope recipient that produced the other
bewildering false positive (the dog show/TV one).  I'm starting to think
that perhaps Bruce filed all mail to this address as spam for some reason
(expired account, perhaps).

Charles
-- 
-----------------------------------------------------------------------
Charles Cazabon                 <python-spambayes@discworld.dyndns.org>
GPL'ed software available at:     http://www.qcc.ca/~charlesc/software/
-----------------------------------------------------------------------

From guido@python.org  Fri Sep 27 16:17:52 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 27 Sep 2002 11:17:52 -0400
Subject: Ham or Spam? (was RE: [Spambayes] RE: Central Limit Theorem??!!
	:))
In-Reply-To: Your message of "Fri, 27 Sep 2002 08:58:44 MDT."
             <20020927085844.A26502@discworld.dyndns.org> 
References: <LNBBLJKPBEHFEDALKOLCOEPJBGAB.tim.one@comcast.net>
	<LNBBLJKPBEHFEDALKOLCKEAOBHAB.tim@zope.com>  
	<20020927085844.A26502@discworld.dyndns.org> 
Message-ID: <200209271517.g8RFHrM05745@pcp02138704pcs.reston01.va.comcast.net>

> > Attached is the msg that produced those scores.  To my eyes, it
> > doesn't look anything like c.l.py traffic (my ham) *or* like spam
> > -- it looks like a fishing newletter someone signed up for.  My
> > best guess is that it's really ham, mistakenly filed as spam in
> > bruceg's collection.  Whaddya think?
> 
> > Return-Path: <warden@hotspotsporting.com>
> > Delivered-To: bfsmedia-bluegoose.kennels@bfsmedia.com
> 
> Notice that this is the same envelope recipient that produced the other
> bewildering false positive (the dog show/TV one).  I'm starting to think
> that perhaps Bruce filed all mail to this address as spam for some reason
> (expired account, perhaps).

I grepped the spam collection for bluegoose.kennels.  It occurs a
*lot*, so I guess it was one of Bruce's bait addresses.

I've received lots of spams that try to look like legitimate
newsletters for some specialty topic.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From skip@pobox.com  Fri Sep 27 16:10:50 2002
From: skip@pobox.com (Skip Montanaro)
Date: Fri, 27 Sep 2002 10:10:50 -0500
Subject: Ham or Spam? (was RE: [Spambayes] RE: Central Limit Theorem??!!
	:))
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEAOBHAB.tim@zope.com>
References: <LNBBLJKPBEHFEDALKOLCOEPJBGAB.tim.one@comcast.net>
        <LNBBLJKPBEHFEDALKOLCKEAOBHAB.tim@zope.com>
Message-ID: <15764.29946.157827.108795@12-248-11-90.client.attbi.com>

    Tim> it looks like a fishing newletter someone signed up for.  My best
    Tim> guess is that it's really ham, mistakenly filed as spam in bruceg's
    Tim> collection.  Whaddya think?

Looks like ham (or perhaps walleye) to me...

S

From tim@zope.com  Fri Sep 27 16:41:17 2002
From: tim@zope.com (Tim Peters)
Date: Fri, 27 Sep 2002 11:41:17 -0400
Subject: Ham or Spam? (was RE: [Spambayes] RE: Central Limit Theorem??!!
	:))
In-Reply-To: <200209271517.g8RFHrM05745@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCEEBDBHAB.tim@zope.com>

[Guido]
> I grepped the spam collection for bluegoose.kennels.  It occurs a
> *lot*,

Same here, of course.

> so I guess it was one of Bruce's bait addresses.

Or it's a legitimate address Bruce happened to capture by mistake:  it's not
surprising that a legit address also gets lots of spam.

> I've received lots of spams that try to look like legitimate
> newsletters for some specialty topic.

Have you ever gotten a spam mentioning an ESPN dog show?  A chatty spam
about fishing?  I have not.  "Newsletters" about stocks and health issues,
yes, endless quantities of both, and they all try to sell me something on
the spot; but not about "outdoor life manly man" topics, and not that never
get around to trying to part me from a slice of my wallet.  The URLs in both
of those putative spam are still in business too.

This nails it for me:

    http://www.bfsmedia.com/experience.php3

list Bluegoose Kennels as one of their clients.  The central limit scheme
knows more about spam than we do <wink>.


From tim@zope.com  Fri Sep 27 16:41:43 2002
From: tim@zope.com (Tim Peters)
Date: Fri, 27 Sep 2002 11:41:43 -0400
Subject: Ham or Spam? (was RE: [Spambayes] RE: Central Limit Theorem??!!
	:))
In-Reply-To: <20020927085844.A26502@discworld.dyndns.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEBDBHAB.tim@zope.com>

[Charles Cazabon]
> Notice that this is the same envelope recipient that produced the other
> bewildering false positive (the dog show/TV one).

I did, but as Guido pointed out there are a lot of msgs to bluegoose.kennels
in the spam collection, and most are obviously spam.

> I'm starting to think that perhaps Bruce filed all mail to this
> address as spam for some reason (expired account, perhaps).

An expired accout would make sense.  While bfsmedia still lists Bluegoose
Kennels as a client (see earlier email),

    http://bfsmedia.com/bluegoose/

is a 404 now (Google finds that URL on three pages with links to dog
breeders).


From tim.one@comcast.net  Fri Sep 27 17:23:39 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 27 Sep 2002 12:23:39 -0400
Subject: Ham or Spam? (was RE: [Spambayes] RE: Central Limit Theorem??!!
	:))
In-Reply-To: <LNBBLJKPBEHFEDALKOLCEEBDBHAB.tim@zope.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEBIBHAB.tim.one@comcast.net>

FYI, another preliminary observation is that the logarithmic central-limit
scheme seems (in my data) to be very unsure (high sdevs away from both
means) about:

+ Msgs in German.
+ Msgs in French.
+ Msgs in Spanish.
+ Some msgs in Asian languages.
+ Msgs composed almost entirely of Javascript.

I'm inclined to call that a good sign, and except for the msgs in
Javascript, I rarely have a clue about whether these are ham or spam either.
WRT the Javascript, a possible weakness of our HTML stripping is that we
lose easy clues about script sections.


From python-spambayes@discworld.dyndns.org  Fri Sep 27 17:46:20 2002
From: python-spambayes@discworld.dyndns.org (Charles Cazabon)
Date: Fri, 27 Sep 2002 10:46:20 -0600
Subject: Ham or Spam? (was RE: [Spambayes] RE: Central Limit Theorem??!!
	:))
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEBIBHAB.tim.one@comcast.net>;
	from tim.one@comcast.net on Fri, Sep 27, 2002 at 12:23:39PM -0400
References: <LNBBLJKPBEHFEDALKOLCEEBDBHAB.tim@zope.com>
	<LNBBLJKPBEHFEDALKOLCGEBIBHAB.tim.one@comcast.net>
Message-ID: <20020927104620.A27466@discworld.dyndns.org>

Tim Peters <tim.one@comcast.net> wrote:
> FYI, another preliminary observation is that the logarithmic central-limit
> scheme seems (in my data) to be very unsure (high sdevs away from both
> means) about:

I'd be most curious as to how ham HTML messages vs. spam HTML messages compare
with the above scheme if you longer strip HTML tags.  I realize you don't have
unlimited time for testing, but it might be useful if HTML spam message rate
as "high likelihood, high confidence" while HTML ham is "high likelihood,
lower confidence" ...

Charles
-- 
-----------------------------------------------------------------------
Charles Cazabon                 <python-spambayes@discworld.dyndns.org>
GPL'ed software available at:     http://www.qcc.ca/~charlesc/software/
-----------------------------------------------------------------------

From tim.one@comcast.net  Fri Sep 27 18:13:29 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 27 Sep 2002 13:13:29 -0400
Subject: [Spambayes] RE: Central Limit Theorem??!!     :)
In-Reply-To: <20020927045928.GA18658@glacier.arctrix.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEBOBHAB.tim.one@comcast.net>

[Neil Schemenauer]
> I don't have anything useful to add except to say that this looks very
> promising.

So far, yes!  The logarithmic variation of Gary's scheme (which I was using)
computes both a ham and a spam statistic from the word probs, and this seems
akin to the gimmick we speculated about earlier for using two "no false
positives" and "no false negatives" scoring gimmicks to grope at some notion
of confidence.

> I'm an idiot when it comes to statistics but even I know
> what 18 standard deviations means.

It means 1 chance in about 10**72.  I've forgotten how many electrons there
are in the universe, although I've long suspected there's really only one
<wink>.

> Also, having a system that generates a "confidence" value in addition
> to a "rating" is a huge bonus.
>
> No time to play with it now, unfortunately.  Maybe tomorrow.

If you can make some time, it's

"""
[Classifier]
use_robinson_probability: True
robinson_probability_x: 0.5
robinson_probability_a: 0.225
max_discriminators: 30
robinson_minimum_prob_strength: 0.1
use_central_limit2: True

[TestDriver]
spam_cutoff: 0.5
"""

The relevant routines in classifier.py are

    central_limit_compute_population_stats2()
    central_limit_spamprob2()

The latter replaces the default spamprob() (enabling use_central_limit2 does
that by magic).  The statistic it computes at the end (the "score" it
returns):

        stat = abs(zham) - abs(zspam)  # > 0 for spam, < 0 for ham

        if stat < -20.0:
            stat = -20.0
        elif stat > 20.0:
            stat = 20.0
        stat = 0.5 + stat / 40.0

is simply looking at the sign bit of the difference between the z-scores,
scaling it in an absurd way to fit in [0., 1.].  These scores don't make any
sense, though, *beyond* mapping - and + to < 0.5 and > 0.5 respectively.

This scheme can't be used with timcv (yet), but can be used with timtest.

Things that need investigating:

+ How to compute a sensible score?
  An honest-to-God probability can be computed from a z-score, and
  I can supply code to do that (although I'd look on google first
  for Python code that can already do that; a simple search over a
  canned sorted list of (abs(zscore), probability) tuples is the
  quickest way to get a good idea).

+ How to represent, compute, and return a measure of confidence?

+ Once that's done, how well does it work?  Does it ever err when
  it's confident?  Since the answer is "yes" <wink>, what are the
  error rates in the region of confidence?  What rates of ham and
  spam is it uncertain about?  If that's high, it may not be usable.
  In my quick tests, it's possible to deduce from the absurdly
  scaled result that it was extremely confident about 3932 of 4000 ham
  and made no errors on those.  Also possible to deduce it was
  extremely confident about 2401 of 2800 spam, and made no errors
  on those.  Overall, it got 0 fp and 9 fn from looking at just
  the sign bit.  The z-scores are both "large" in the cases it
  erred, although it gets to be a delicate decision when the number
  of words is small.  For example, one fn here had only 8 tokens,
  and had z-scores of -3.8 and -5.2.  Neither is astronomically
  unlikely, but both are "quite unlikely".  OTOH, plenty of ham
  and spam have one "quite unlikely" and one "astronomically
  unlikely" z-score each, and it's probably best to be quite
  confident about those.  I simply don't know a good way to combine
  this evidence yet.

+ After a sensible way of computing scores is worked out:
  - What's a better value for robinson_minimum_prob_strength?
    Gary would be happiest if it turned out to be 0 <wink>.
  - What's a better value for max_discriminators?  30 is just an
    educated guess at the minimum that's almost certain to be
    robust.  It may be robust at smaller values.  It may also get
    real benefit from sucking in more clues (bet then shorter
    msgs may become more of a puzzle too).


From tim.one@comcast.net  Fri Sep 27 18:21:36 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 27 Sep 2002 13:21:36 -0400
Subject: Ham or Spam? (was RE: [Spambayes] RE: Central Limit Theorem??!!
	:))
In-Reply-To: <20020927104620.A27466@discworld.dyndns.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEBPBHAB.tim.one@comcast.net>

[Charles Cazabon]
> I'd be most curious as to how ham HTML messages vs. spam HTML
> messages compare with the above scheme if you longer strip HTML tags.

"No longer", right?

> I realize you don't have unlimited time for testing, but it might be
> useful if HTML spam  message rate as "high likelihood, high
> confidence" while HTML ham is "high likelihood, lower confidence" ...

I can run tests in the background easily enough, but this is something I
can't test at all:  there is almost no HTML ham in c.l.py traffic.  About
the only instances of it are inappropriate (but not spam) postings from
first-time posters, with bodies usually of the form "confirm" or
"unsubscribe" (i.e., newbies who can't follow directions and post their
mailing list administrivia to the newsgroup).

If anyone else can test this, be my guest.  In the absence of volunteers,
I'll appoint Charles <wink>.


From python-spambayes@discworld.dyndns.org  Fri Sep 27 18:34:03 2002
From: python-spambayes@discworld.dyndns.org (Charles Cazabon)
Date: Fri, 27 Sep 2002 11:34:03 -0600
Subject: Ham or Spam? (was RE: [Spambayes] RE: Central Limit Theorem??!!
	:))
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEBPBHAB.tim.one@comcast.net>;
	from tim.one@comcast.net on Fri, Sep 27, 2002 at 01:21:36PM -0400
References: <20020927104620.A27466@discworld.dyndns.org>
	<LNBBLJKPBEHFEDALKOLCKEBPBHAB.tim.one@comcast.net>
Message-ID: <20020927113403.A27827@discworld.dyndns.org>

Tim Peters <tim.one@comcast.net> wrote:
> [Charles Cazabon]
> > I'd be most curious as to how ham HTML messages vs. spam HTML
> > messages compare with the above scheme if you longer strip HTML tags.
> 
> "No longer", right?

Yup -- typing too fast for accuracy :).  The more typos I make, the more
interested I am in the subject.

> > I realize you don't have unlimited time for testing, but it might be
> > useful if HTML spam  message rate as "high likelihood, high
> > confidence" while HTML ham is "high likelihood, lower confidence" ...
> 
> I can run tests in the background easily enough, but this is something I
> can't test at all:  there is almost no HTML ham in c.l.py traffic.
[...]
> If anyone else can test this, be my guest.  In the absence of volunteers,
> I'll appoint Charles <wink>.

I can accumulate ham containing HTML easily enough (many of the mailing lists
I'm on get multipart/alternative messages from newbies all the time, and
html-only the odd time), but I'm afraid the machine I'd have to run this stuff
on (the mailserver) isn't up to the task -- it's an older machine, 64MB RAM
and no room for expansion.  As soon as it's driven into swap, I may as well
not be running the tests at all.

Alternately, I suppose I could try to put together a corpus for someone with
greater hardware resources to test.  To have reasonable confidence in the
results, do you need a full 2000 messages each of ham and spam?

Charles
-- 
-----------------------------------------------------------------------
Charles Cazabon                 <python-spambayes@discworld.dyndns.org>
GPL'ed software available at:     http://www.qcc.ca/~charlesc/software/
-----------------------------------------------------------------------

From Alexander@Leidinger.net  Fri Sep 27 18:54:53 2002
From: Alexander@Leidinger.net (Alexander Leidinger)
Date: Fri, 27 Sep 2002 19:54:53 +0200
Subject: [Spambayes] Re: Ham or Spam?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEBIBHAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCEEBDBHAB.tim@zope.com>
	<LNBBLJKPBEHFEDALKOLCGEBIBHAB.tim.one@comcast.net>
Message-ID: <20020927195453.6d581bca.Alexander@Leidinger.net>

This is a multi-part message in MIME format.

---------------------- multipart/mixed attachment
On Fri, 27 Sep 2002 12:23:39 -0400 Tim Peters <tim.one@comcast.net>
wrote:

> either. WRT the Javascript, a possible weakness of our HTML stripping
> is that we lose easy clues about script sections.

I'm seperating the spam from the ham at the moment (attached is a patch
for hammie.py to allow more than one -s, -g or -u option) and every
message with Javascript was spam so far.

Bye,
Alexander.

-- 
                  Weird enough for government work.

http://www.Leidinger.net                       Alexander @ Leidinger.net
  GPG fingerprint = C518 BC70 E67F 143F BE91  3365 79E2 9C60 B006 3FE7

---------------------- multipart/mixed attachment
A non-text attachment was scrubbed...
Name: hammie.py.diff
Type: application/octet-stream
Size: 2922 bytes
Desc: not available
Url : http://mail.python.org/pipermail-21/spambayes/attachments/20020927/e7996e64/hammie.py.exe

---------------------- multipart/mixed attachment--

From gward@python.net  Fri Sep 27 19:07:35 2002
From: gward@python.net (Greg Ward)
Date: Fri, 27 Sep 2002 14:07:35 -0400
Subject: Ham or Spam? (was RE: [Spambayes] RE: Central Limit Theorem??!!
	:))
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEBIBHAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCEEBDBHAB.tim@zope.com>
	<LNBBLJKPBEHFEDALKOLCGEBIBHAB.tim.one@comcast.net>
Message-ID: <20020927180734.GC29009@cthulhu.gerg.ca>

On 27 September 2002, Tim Peters said:
> FYI, another preliminary observation is that the logarithmic central-limit
> scheme seems (in my data) to be very unsure (high sdevs away from both
> means) about:
> 
> + Msgs in German.
> + Msgs in French.
> + Msgs in Spanish.

Good, I think!  Two of my persistent FPs are:

  * a message from CIRA (the Canadian Internet Registration Authority),
    which is half in French
  * a one-line "subscribe" sent to <mumble>-request@python.org,
    with a loooong disclaimer in Spanish

Both of these are killed by perfectly ordinary, vanilla French and
Spanish words, because I get very little real email in French and none
in Spanish.  (Oddly enough, I get very little spam in French too.  In
fact, most of the spam that gets past SpamAssassin to me is in German,
which highlights another of SA's weaknesses and a *potential* strength
of the statistical approach.  I used to get a lot of spam in Portuguese,
but I added some SA tests that score messages from Brazil and Portugal a
bit higher.  Seems to have put a stop to that.)

I bet Italian would suffer the same fate except that I have a week's
worth of traffic from zope-it@zope.org in my ham collection.

Incidentally, here are the probs for the CIRA message -- interestingly,
this is one of the few messages in the gward-ham corpus that was *not*
sent to gward@python.net, so it still has many traces of my ISP in the
Received headers.  None of my spam went through my ISP, so all those
clues are 0.01 clues -- but they still didn't help.

Data/Ham/Set9/1014826921.10087.cthulhu.gerg.ca:2,S
prob = 1.0
prob('membres') = 0.01
prob('to:videotron.ca') = 0.01
prob('received:greg') = 0.01
prob('cher') = 0.01
prob('positions.') = 0.01
prob('received:videotron.ca') = 0.01
prob('header:Return-path:1') = 0.01
prob('received:bellnexxia.net') = 0.01
prob('nominations') = 0.01
prob('re:') = 0.01
prob('received:24.201') = 0.01
prob('received:pop.videotron.ca') = 0.01
prob('received:24.201.245') = 0.01
prob('avis') = 0.01
prob('received:24.201.245.36') = 0.01
prob('received:sc1.videotron.ca') = 0.01
prob('return-path:members') = 0.01
prob('received:vl-ms-mr002.sc1.videotron.ca') = 0.01
prob('des') = 0.9875
prob('sera') = 0.99
prob('comit\xe9') = 0.99
prob('qui') = 0.99
prob('enabling') = 0.99
prob('voter') = 0.99
prob('mai') = 0.99
prob('besoin') = 0.99
prob('mises') = 0.99
prob('apr\xe8s') = 0.99
prob('ligne') = 0.99
prob('mars') = 0.99
prob('candidat') = 0.99
prob('election') = 0.99
prob('vous') = 0.99
prob('aussi') = 0.99
prob('une') = 0.99
prob('trois') = 0.99
prob('entre') = 0.99
prob('ces') = 0.99
prob('avril') = 0.99
prob('leur') = 0.99
prob('directors.') = 0.99
prob('objet') = 0.99
prob('faire') = 0.99
prob('nominated') = 0.99
prob('personnes') = 0.99
prob('veuillez') = 0.99
prob('elected') = 0.99
prob('toute') = 0.99
prob('received:sympatico.ca') = 0.99
prob('possession,') = 0.99
prob('poser') = 0.99
prob('peut') = 0.99

Would be interesting to score just the English half of that message.

Oh, here are the probs for the "subscribe" request with Spanish
disclaimer:

Data/Ham/Set10/17rK7U-0007e7-00:2,S
prob = 1.0
prob('subject:subscribe') = 0.01
prob('confidencial') = 0.01
prob('autorizado') = 0.01
prob('contenida') = 0.01
prob('fon') = 0.01
prob('mensaje.') = 0.01
prob('message-id:skip:A 40') = 0.01
prob('est\xe1') = 0.99
prob('cual') = 0.99
prob('mensajes') = 0.99
prob('pueda') = 0.99
prob('hacer') = 0.99
prob('diego') = 0.99
prob('llegar') = 0.99
prob('informaci\xf3n') = 0.99
prob('alguna') = 0.99
prob('exclusivo') = 0.99
prob('empresa') = 0.99
prob('motivo') = 0.99
prob('telef\xf3nica') = 0.99
prob('gesti\xf3n') = 0.99
prob('haber') = 0.99
prob('received:167]') = 0.99
prob('propias') = 0.99
prob('ud.') = 0.99
prob('todas') = 0.99
prob('mismo,') = 0.99
prob('son') = 0.99
prob('personales') = 0.99
prob('pueden') = 0.99
prob('responsable') = 0.99

This message has got to win some sort of prize for the wheat-to-chaff
ratio: it requires 4.6 kB of signature, MIME junk, and disclaimer to say
"subscribe".  Here are the headers:

"""
Return-Path: <DQuevedo@uniFON.com.ar>
Envelope-To: python-list-request@python.org
Received: from [200.16.211.167] (helo=estcp.tcp.com.ar)
        by mail.python.org with esmtp (Exim 4.05)
        id 17rK7U-0007e7-00
        for python-list-request@python.org; Tue, 17 Sep 2002 11:18:32 -0400
Received: by noticias.unifon.com.ar with Internet Mail Service (5.5.2653.19)
        id <S6X3XHKB>; Tue, 17 Sep 2002 12:15:11 -0300
Message-ID: <A128D751272CD411BC9200508BC2194D019B6215@escpl.tcp.com.ar>
From: "Quevedo, Diego" <DQuevedo@uniFON.com.ar>
To: "'python-list-request@python.org'" <python-list-request@python.org>
Subject: subscribe
Date: Tue, 17 Sep 2002 12:15:33 -0300
Return-Receipt-To: "Quevedo, Diego" <DQuevedo@uniFON.com.ar>
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2653.19)
Content-Type: multipart/alternative;
        boundary="----_=_NextPart_001_01C25E5D.1189FAE0"
"""

and here is the text/plain part:

"""
subscribe


Diego Quevedo
Gesti?n de Red
Tel: (54-11) 4324-9103
Cel: 15-5132-0135
dquevedo@unifon.com.ar
unifon








. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
ADVERTENCIA

La informaci?n contenida en este mensaje y cualquier archivo anexo al mismo,
son para uso exclusivo del destinatario y pueden contener informaci?n
confidencial o propietaria, cuya divulgaci?n es sancionada por la ley.

Si Ud. No es uno de los destinatarios consignados o la persona responsable
de hacer llegar este mensaje a los destinatarios consignados, no est?
autorizado a divulgar, copiar, distribuir o retener informaci?n (o parte de
ella) contenida en este mensaje. Por favor notif?quenos respondiendo al
remitente, borre el mensaje original y borre las copias (impresas o grabadas
en cualquier medio magn?tico) que pueda haber realizado del mismo.

Todas las opiniones contenidas en este mail son propias del autor del
mensaje y no necesariamente coinciden con las de Telef?nica Comunicaciones
Personales S.A. o alguna empresa asociada.

Los mensajes electr?nicos pueden ser alterados, motivo por el cual
Telef?nica Comunicaciones Personales S.A. no aceptar? ninguna obligaci?n
cualquiera sea el resultante de este mensaje.

Muchas Gracias.
"""

And I'm not even showing the text/html part -- good *grief*!

        Greg
-- 
Greg Ward <gward@python.net>                         http://www.gerg.ca/
Yield to temptation; it may not pass your way again.

From tim.one@comcast.net  Fri Sep 27 19:24:45 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 27 Sep 2002 14:24:45 -0400
Subject: [Spambayes] RE: Ham or Spam?
In-Reply-To: <20020927195453.6d581bca.Alexander@Leidinger.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCOECFBHAB.tim.one@comcast.net>

[Alexander Leidinger]
> I'm seperating the spam from the ham at the moment (attached is a patch
> for hammie.py to allow more than one -s, -g or -u option) and every
> message with Javascript was spam so far.

Would someone who uses hammie.py please look at Alex's patch?

We may need to require that patches be uploaded to our SF project's patch
manager.  On any mailing list, the usual fate of patches is that they get
lost forever.


From tim.one@comcast.net  Fri Sep 27 19:31:21 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 27 Sep 2002 14:31:21 -0400
Subject: Ham or Spam? (was RE: [Spambayes] RE: Central Limit Theorem??!!
	:))
In-Reply-To: <20020927113403.A27827@discworld.dyndns.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCGECGBHAB.tim.one@comcast.net>

[Charles Cazabon]
> ...
> Alternately, I suppose I could try to put together a corpus for
> someone with greater hardware resources to test.  To have reasonable
> confidence in the results, do you need a full 2000 messages each of
> ham and spam?

Nobody knows -- no controlled experiments have been run yet to quantify
corpus size, or corpus ratios, versus mean error rates and cross-corpus
variance.  2000 of each was just an educated guess for the last round of
testing, and it seemed to work pretty well.

Note that Paul Graham is begging for spam; I hope he broadens his quest to
include ham too, because clean ham collections are harder to come by:

    http://www.paulgraham.com/wantspam.html


From neale@woozle.org  Fri Sep 27 19:42:58 2002
From: neale@woozle.org (Neale Pickett)
Date: 27 Sep 2002 11:42:58 -0700
Subject: [Spambayes] RE: Ham or Spam?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCOECFBHAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCOECFBHAB.tim.one@comcast.net>
Message-ID: <w53k7l72s0d.fsf@woozle.org>

So then, Tim Peters <tim.one@comcast.net> is all like:

> [Alexander Leidinger]
> > I'm seperating the spam from the ham at the moment (attached is a patch
> > for hammie.py to allow more than one -s, -g or -u option) and every
> > message with Javascript was spam so far.
> 
> Would someone who uses hammie.py please look at Alex's patch?

Yeah, it looks good to me (although I would have made ham and spam start
out as empty lists, and skipped the None check).

Tim, can you add npickett to the list of developers on the project?  I
don't expect SF management to get around to replying to my queries about
my old account any time soon :)

Otherwise, someone please feel free to commit Alexander's patch.

Neale

From guido@python.org  Fri Sep 27 19:56:32 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 27 Sep 2002 14:56:32 -0400
Subject: [Spambayes] RE: Ham or Spam?
In-Reply-To: Your message of "Fri, 27 Sep 2002 14:24:45 EDT."
             <LNBBLJKPBEHFEDALKOLCOECFBHAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCOECFBHAB.tim.one@comcast.net> 
Message-ID: <200209271856.g8RIuWI10520@pcp02138704pcs.reston01.va.comcast.net>

> [Alexander Leidinger]
> > I'm seperating the spam from the ham at the moment (attached is a patch
> > for hammie.py to allow more than one -s, -g or -u option) and every
> > message with Javascript was spam so far.
> 
> Would someone who uses hammie.py please look at Alex's patch?

Done.  I've checked something in that does the same thing.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.one@comcast.net  Fri Sep 27 19:57:21 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 27 Sep 2002 14:57:21 -0400
Subject: [Spambayes] RE: Ham or Spam?
In-Reply-To: <w53k7l72s0d.fsf@woozle.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCGECJBHAB.tim.one@comcast.net>

[Neale Pickett, on Alex's hammie patch]
> Yeah, it looks good to me (although I would have made ham and spam
> start out as empty lists, and skipped the None check).
>
> Tim, can you add npickett to the list of developers on the project?  I
> don't expect SF management to get around to replying to my queries about
> my old account any time soon :)

Done.  Welcome back!

> Otherwise, someone please feel free to commit Alexander's patch.

Note that Guido already did (and thanks!).

From tim.one@comcast.net  Fri Sep 27 20:06:15 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 27 Sep 2002 15:06:15 -0400
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <20020926140052.C01D6F5A7@cashew.wolfskeep.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKECKBHAB.tim.one@comcast.net>

...
[Tim]
>> If not, I'll tally results tomorrow.

[T. Alexander Popiel]
> Feel free... as I said, while I want to participate, I may not
> be able to until Saturday (or later, if closing on the house
> hits any more snags).

I think closing on a house is the very definition of snags <wink>.

I've seen enough on this particular experiment to Pronounce, but there are
many more things that need testing, so don't sweat missing this one.


From neale@woozle.org  Fri Sep 27 20:45:03 2002
From: neale@woozle.org (Neale Pickett)
Date: 27 Sep 2002 12:45:03 -0700
Subject: [Spambayes] new Hammie class
Message-ID: <w53elbf2p4w.fsf@woozle.org>

Heads up to anyone using hammie.py: it now has a Hammie class that you
ought to be using.  Richie Hindle inspired me to do this with his
pop3proxy.  It should make for a cleaner interface.

Neale

From neale@woozle.org  Fri Sep 27 20:47:04 2002
From: neale@woozle.org (Neale Pickett)
Date: 27 Sep 2002 12:47:04 -0700
Subject: [Spambayes] new Hammie class
In-Reply-To: <w53elbf2p4w.fsf@woozle.org>
References: <w53elbf2p4w.fsf@woozle.org>
Message-ID: <w5365wr2p1j.fsf@woozle.org>

So then, Neale Pickett <neale@woozle.org> is all like:

> Heads up to anyone using hammie.py: it now has a Hammie class that you
> ought to be using.

What I meant was, "Heads up to anyone *importing* hammie".  If you're
just using it to filter your mail, it will keep working the same way.

Neale

From tim.one@comcast.net  Fri Sep 27 20:52:49 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 27 Sep 2002 15:52:49 -0400
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <20020926135437.GA24403@cthulhu.gerg.ca>
Message-ID: <LNBBLJKPBEHFEDALKOLCAECPBHAB.tim.one@comcast.net>

The result:  Gary's f(w) scheme wins.

After looking at all the reports once, I was *going* to say that Graham's
scheme had only one clear advantage:  a single spam_cutoff value that works
well for everyone.

But that wasn't so:  although it was outside the test protocol (which is why
I ignored it the first time around), on Anthony's full and very difficult
11,000 spam and 19,000 ham corpus, Graham got 135 fp to his tuned-Robinson
10 fp.

The value I initially suggested for spam_cutoff was generally too small, and
the value for robinson_probability_a too large.  The people who worked
hardest at helping to tune these (Neil and Anthony) converged on an "a"
close to 0.2, which also worked well for me.  The other parameters (i.e.,
apart from "a" and spam_cutoff) were close to optimal from the start, and
didn't show signs of corpus dependence.  Everyone who worked at tuning got
results at least as good as with Graham, several slightly better than with
Graham, and some much better.

Apart from all that, as the guy who has written all of the core scoring code
so far, I'm *much* more comfortable with Gary's formulation:  it has fewer
tuning parameters than Graham's, and they *make sense*, both from
theoretical and practical POVs.  I said in the first msg of this thread that
Gary's scheme would win for that reason alone if it merely did no worse than
Graham's.  That it's often a small win and sometimes a large win nails it.

spam_cutoff remains "a problem" or "an opportunity", depending on how you
look at it.  I think it's some of each:

+ Against it, the best value does vary by corpus, there's no
  way evident to guess the best value a priori, and it's not
  trivial to figure out the best value a posteriori either.

+ For it, there's a real and valuable "middle ground" here:  moving
  spam_cutoff a few points allows people to favor f-n over f-p (or
  vice versa) easily, effectively, and gradually.

The "cosmetics" are better under f(w) too.  By that I mean things that
aren't really technical advantages either way, but that people *like*
better.  For example, I expect everyone hated seeing f-p under Graham's
scheme that came out with a score of 1.0 ("that's certainly spam!"), and
seeing f-n come out with a score of 1e-29 ("that's certainly ham!").  f(w)
appears never to do that:  the bulk of f-n and f-p scores are a short
distance from the corpus's best spam_cutoff, and even in Skip's untuned f(w)
run on his very hard corpus, no f-p scored above 0.60, and no f-n below
0.20.

Thanks for playing!  Unless someone has a killer objection, I'm going to
purge the stuff unique to the Graham scheme from the codebase, and switch
the defaults to Gary's f(w) scheme.

If you're dying to test something else, here are two open questions of real
practical importance:

1. How does the ham/spam training ratio versus real-life ham/spam
   ratio affect error rates?

2. How does the absolute ham+spam training size affect error rates?
   (i.e., how many messages do we need to train on to get a desired
   accuracy level?)

A third:

3. Is it possible to "seed" a database with somebody else's data
   and get decent results out of the box?

If anyone would care to champion one of these, grab it, dream up a test
strategy, announce it here and solicit results.  I'm least interested in #3
right now because I've got the slimmest of evidence that a key part of the
central-limit approaches may indeed be sharable for both ham and spam, and
it's premature to test anything with those approaches yet (we don't yet have
a handle on how to produce a reasonable score from these -- the scores they
produce right now are absurd).


From neale@woozle.org  Fri Sep 27 21:16:42 2002
From: neale@woozle.org (Neale Pickett)
Date: 27 Sep 2002 13:16:42 -0700
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <LNBBLJKPBEHFEDALKOLCAECPBHAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCAECPBHAB.tim.one@comcast.net>
Message-ID: <w531y7f2no5.fsf@woozle.org>

So then, Tim Peters <tim.one@comcast.net> is all like:

> If you're dying to test something else, here are two open questions of
> real practical importance:
>
> 3. Is it possible to "seed" a database with somebody else's data and
>    get decent results out of the box?

$FIRM has a tangible interest in the answer to this question.  And by
tangible, I mean that I can hack on it during work hours and not feel
guilty about it.  It also means I can recruit corpora from our
employees.

What sort of results would be the most useful to ya, Tim, the output of
timcv.py, or something else?  Any configuration options you'd like to
see twiddled for this?

Neale


From guido@python.org  Fri Sep 27 21:32:10 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 27 Sep 2002 16:32:10 -0400
Subject: [Spambayes] There Can Be Only One
In-Reply-To: Your message of "Fri, 27 Sep 2002 15:52:49 EDT."
             <LNBBLJKPBEHFEDALKOLCAECPBHAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCAECPBHAB.tim.one@comcast.net> 
Message-ID: <200209272032.g8RKWAA23028@pcp02138704pcs.reston01.va.comcast.net>

> Thanks for playing!  Unless someone has a killer objection, I'm
> going to purge the stuff unique to the Graham scheme from the
> codebase, and switch the defaults to Gary's f(w) scheme.

Can't you just switch to Gary by default without nuking the Graham
code?  A negative result is also a result!

Are there considerations of database size, classification speed, or
required corpus size that could cause someone to prefer Graham?  (IOW
are there any circumstances where Gary might be considered unfeasible
but where Graham would work?)

Also, the tuning you've done to Graham may be useful for people who
reimplement Graham in another language.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.one@comcast.net  Fri Sep 27 21:46:31 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 27 Sep 2002 16:46:31 -0400
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <200209272032.g8RKWAA23028@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEDDBHAB.tim.one@comcast.net>

[Guido van Rossum]
> Can't you just switch to Gary by default without nuking the Graham
> code?  A negative result is also a result!

I thought I covered this in the first msg.  The sheer number of options and
alternatives is overwhelming to most people, and even I'm having a hard time
keeping them all straight and working.  The technical core of this project
has been about killing non-winners from the start, else there are an
exponentially growing number of choices to deal with, almost all of which
have been proven not to help.  TESTING.txt has warned about that from the
start too.

> Are there considerations of database size,

No, the database is identical either way.

> classification speed,

Dominated by tokenization and I/O time.  If you set max_discriminators to a
very large value (which hasn't done anyone any good in testing, although
it's known to hurt under the Graham scheme), it's significantly slower
because the priority queue gimmick was designed for a small number.

> or required corpus size that could cause someone to prefer Graham?

Untested, and there are much more important things than that to test.

> (IOW are there any circumstances where Gary might be considered
> unfeasible but where Graham would work?)

I doubt it, but don't know.  This project can't make progress unless it
gives up on dead ends; there's never perfect evidence that a thing *is* a
dead end, but the evidence for this one is enough (by my lights).

> Also, the tuning you've done to Graham may be useful for people who
> reimplement Graham in another language.

I can tag the CVS repository before nuking the code.  I'm not going to
resurrect Python's samplesort hybrid either <wink>.


From tim.one@comcast.net  Fri Sep 27 22:07:47 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 27 Sep 2002 17:07:47 -0400
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEDDBHAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCOEDEBHAB.tim.one@comcast.net>

FYI, the current state of the project has been tagged, via

    cvs tag Last-Graham

Since Guido didn't volunteer to maintain the Graham code <0.9 wink>, it's
going away, but you can get it back via this tag.


From guido@python.org  Fri Sep 27 22:30:13 2002
From: guido@python.org (Guido van Rossum)
Date: Fri, 27 Sep 2002 17:30:13 -0400
Subject: [Spambayes] There Can Be Only One
In-Reply-To: Your message of "Fri, 27 Sep 2002 16:46:31 EDT."
             <LNBBLJKPBEHFEDALKOLCGEDDBHAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCGEDDBHAB.tim.one@comcast.net> 
Message-ID: <200209272130.g8RLUEC26077@pcp02138704pcs.reston01.va.comcast.net>

OK, sounds like Graham has nothing over Gary.  Nuke it.

From skip@pobox.com  Fri Sep 27 22:52:56 2002
From: skip@pobox.com (Skip Montanaro)
Date: Fri, 27 Sep 2002 16:52:56 -0500
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <200209272032.g8RKWAA23028@pcp02138704pcs.reston01.va.comcast.net>
References: <LNBBLJKPBEHFEDALKOLCAECPBHAB.tim.one@comcast.net>
        <200209272032.g8RKWAA23028@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <15764.54072.790093.931806@12-248-11-90.client.attbi.com>


    Guido> Also, the tuning you've done to Graham may be useful for people
    Guido> who reimplement Graham in another language.

How about creating a baseline named "graham-rip" before ripping it out so
software archaeologists can find it later?

Skip


From tim.one@comcast.net  Fri Sep 27 23:37:31 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 27 Sep 2002 18:37:31 -0400
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <15764.54072.790093.931806@12-248-11-90.client.attbi.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEDLBHAB.tim.one@comcast.net>

[Skip Montanaro]
> How about creating a baseline named "graham-rip" before ripping it
> out so software archaeologists can find it later?

I'm not sure what "creating a baseline" means.  I tagged the project with
Last-Graham before changing the world, though, so if that's what you mean,
it's done.

The world has also changed:

These options have vanished:

    hambias
    spambias
    min_spamprob
    max_spamprob
    unknown_word_spamprob
    use_robinson_combining
    use_robinson_probability
    use_robinson_ranking

[note that the other Graham "mincount" mystery knob vanished a long
 time ago, and max_discriminators pretty much has the same meaning
 it had before]

These options have changed default value:

    robinson_probability_a: 0.225 (was 1.0)
    robinson_minimum_prob_strength: 0.1 (was 0.0)
    max_discriminators: 150 (was 16)
    spam_cutoff: 0.570 (was 0.90) # THIS IS CORPUS-DEPENDENT!

The GrahamBayes class was renamed to Bayes.


From tim.one@comcast.net  Sat Sep 28 00:46:43 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 27 Sep 2002 19:46:43 -0400
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <w531y7f2no5.fsf@woozle.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEDOBHAB.tim.one@comcast.net>

[TIm]
>> 3. Is it possible to "seed" a database with somebody else's data and
>>    get decent results out of the box?

[Neale Pickett]
> $FIRM has a tangible interest in the answer to this question.  And by
> tangible, I mean that I can hack on it during work hours and not feel
> guilty about it.  It also means I can recruit corpora from our
> employees.
>
> What sort of results would be the most useful to ya, Tim, the output of
> timcv.py, or something else?  Any configuration options you'd like to
> see twiddled for this?

You should first tune the Robinson parameters for this "general corpus",
whatever that may mean.

I don't know how you would intend to deploy this, but a test design should
simulate the key features of intended deployment as closely as feasible.

Cross-validation runs are great for testing two schemes *against* each other
(whether they be different schemes or minor tweaks of a single scheme), but
I don't see that you've got two schemes in mind here.  Instead you've got
one seed database and N users, and the only interesting thing appears to be
how well the seed does on each user's whole collection of email.

So I'd build a custom test driver on top of TestDriver, like so:

d = TestDriver.Driver()
d.train(ham, spam)  # create the seed database
for user in users:
    d.test(user.ham, user.spam)
    d.finishtest()
d.alldone()

That's literally about all you need; user.ham ahd user.spam have to be
iterable objects producing streams of messages that "act like" msgs.Msg
objects (a .tag attr that gives each msg a string name for identification in
oupout, and an iterator that produces tokens; note that mboxtest subclasses
msgs.Msg, while timcv and timtest use it indirectly as-is via
msgs.MsgStream).

The output will display results for each user individually, and an aggregate
across all users.  Then you'll want to stare at the output to see how well
it does.  Come back when you get that far <wink>.


From skip@pobox.com  Sat Sep 28 03:16:23 2002
From: skip@pobox.com (Skip Montanaro)
Date: Fri, 27 Sep 2002 21:16:23 -0500
Subject: [Spambayes] There Can Be Only One
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEDLBHAB.tim.one@comcast.net>
References: <15764.54072.790093.931806@12-248-11-90.client.attbi.com>
        <LNBBLJKPBEHFEDALKOLCKEDLBHAB.tim.one@comcast.net>
Message-ID: <15765.4343.548119.188531@12-248-11-90.client.attbi.com>


    Tim> I'm not sure what "creating a baseline" means.  I tagged the
    Tim> project with Last-Graham before changing the world, though, so if
    Tim> that's what you mean, it's done.

Precisely.  I should have refreshed my Python mailbox before replying to
that message.  I was a few minutes behind...

S

From crunch@shopip.com  Sat Sep 28 03:31:46 2002
From: crunch@shopip.com (John Draper)
Date: Fri, 27 Sep 2002 19:31:46 -0700
Subject: [Spambayes] Have you ever....?
Message-ID: <v03110701b9bac4a92623@[192.168.0.2]>

Ever notice the random set of characters at the end of most spam messages these days?    Does anyone know what they are for?   Perhaps these characters might be of use.   Are they some sort of checksum or signature?

John



From skip@pobox.com  Sat Sep 28 04:07:44 2002
From: skip@pobox.com (Skip Montanaro)
Date: Fri, 27 Sep 2002 22:07:44 -0500
Subject: [Spambayes] Have you ever....?
In-Reply-To: <v03110701b9bac4a92623@[192.168.0.2]>
References: <v03110701b9bac4a92623@[192.168.0.2]>
Message-ID: <15765.7424.819070.647176@12-248-11-90.client.attbi.com>


    John> Ever notice the random set of characters at the end of most spam
    John> messages these days?  Does anyone know what they are for?  Perhaps
    John> these characters might be of use.  Are they some sort of checksum
    John> or signature?

They are probably there to try and foil systems which use checksums to try
and detect spam.  Large email providers like AOL or MSN can probably very
quickly conclude an email is spam if they see the same checksum a handful of
times (100? 1000? 10000?).  The more messages which arrive with the same
checksum, the more likely they are to be spam.  The random drivel is an
attempt to work around these sorts of systems.

There are distributed systems, like Razor, which rely on checksums as well.
For an example of a script which tries to compute a "loose" checksum (and
may fail in the case you mentioned), look at the loosecksum.py script in the
spambayes CVS repository.

Skip


From tim.one@comcast.net  Sat Sep 28 04:33:22 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 27 Sep 2002 23:33:22 -0400
Subject: [Spambayes] Have you ever....?
In-Reply-To: <v03110701b9bac4a92623@[192.168.0.2]>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEEFBHAB.tim.one@comcast.net>

[John Draper]
> Ever notice the random set of characters at the end of most spam
> messages these days?    Does anyone know what they are for?
> Perhaps these characters might be of use.   Are they some sort of
> checksum or signature?

Skip's right:  they're indeed random gibberish, to frustrate stupid "exact
match" filters.  Random characters often appear at the end of Subject lines
too, for the same reason.

All they do for us is bloat our database with random strings; it's one of
several reasons that a learning database needs a policy for "expiring" words
over time (else it would grow without bound).


From popiel@wolfskeep.com  Sat Sep 28 04:46:09 2002
From: popiel@wolfskeep.com (T. Alexander Popiel)
Date: Fri, 27 Sep 2002 20:46:09 -0700
Subject: [Spambayes] Have you ever....? 
In-Reply-To: Message from John Draper <crunch@shopip.com> 
   of "Fri, 27 Sep 2002 19:31:46 PDT." <v03110701b9bac4a92623@[192.168.0.2]> 
References: <v03110701b9bac4a92623@[192.168.0.2]> 
Message-ID: <20020928034609.2C791F5B5@cashew.wolfskeep.com>

In message:  <v03110701b9bac4a92623@[192.168.0.2]>
             John Draper <crunch@shopip.com> writes:

>Ever notice the random set of characters at the end of most spam
>messages these days?    Does anyone know what they are for?
>Perhaps these characters might be of use.   Are they some sort of
>checksum or signature?

As far as I can tell, these are tags to indicate which email
address is the hit when people respond to spam.  In my single
source collection, there are a handful of these 'words' which
have appeared many times... so my best guess is that there
are a handful of spam-blast software packages that compute
these codes reproducably based on the email address.

So... just another tracking method, to help the spammers
identify live addresses, even if you've taken steps to
disguise yourself when replying.

- Alex

From tim.one@comcast.net  Sat Sep 28 04:47:27 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 27 Sep 2002 23:47:27 -0400
Subject: [Spambayes] Incompatible change! ([Spambayes-checkins] spambayes
 Options.py,1.35,1.36 classifier.py,1.22,1.23)
Message-ID: <LNBBLJKPBEHFEDALKOLCOEEGBHAB.tim.one@comcast.net>

Heads up:

...

Modified Files:
	Options.py classifier.py 
Log Message:
Gary Robinson changed the forumla he uses to adjust the Graham
probabilities since we first implemented it.  The new formula is
identical to the old in what it computes, but it looks a little different
and is easier to understand.  As a result,

     robinson_probability_a

no longer exists, and

     robinson_probability_s

takes its place (the "s" is for "strength").  If you used non-default
values of a and/or x before, x doesn't change, but you should set

     robinson_probability_s

to robinson_probability_a / robinson_probability_x.

For example, before this checkin, the defaults were a=0.225 and x= 0.5.
Now 'a' is gone, and s defaults to 0.225/0.5 = 0.45.  Computed results
are identical.

Sorry for the hassle, but Gary's webpage does a very nice job of
explaining this formula, and I really don't want to reword it all for
this project -- keeping an obvious connection between our implementation
and Gary's explanation is worth the disruption.

From tim.one@comcast.net  Sat Sep 28 04:50:21 2002
From: tim.one@comcast.net (Tim Peters)
Date: Fri, 27 Sep 2002 23:50:21 -0400
Subject: [Spambayes] Have you ever....?
In-Reply-To: <20020928034609.2C791F5B5@cashew.wolfskeep.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEEHBHAB.tim.one@comcast.net>

[T. Alexander Popiel]
> As far as I can tell, these are tags to indicate which email
> address is the hit when people respond to spam.  In my single
> source collection, there are a handful of these 'words' which
> have appeared many times... so my best guess is that there
> are a handful of spam-blast software packages that compute
> these codes reproducably based on the email address.

Interesting!  It's sure plausible.  At least one of us should become a
professional spammer so we get a better handle on these tricks.  Anyone want
to get rich quick?


From anthony@interlink.com.au  Sat Sep 28 06:00:49 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Sat, 28 Sep 2002 15:00:49 +1000
Subject: [Spambayes] Have you ever....? 
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEEHBHAB.tim.one@comcast.net> 
Message-ID: <200209280500.g8S50ns07270@localhost.localdomain>


>>> Tim Peters wrote
> Interesting!  It's sure plausible.  At least one of us should become a
> professional spammer so we get a better handle on these tricks.  Anyone want
> to get rich quick?

One of the things I do at work is to make sure that our *ahem* valuable
marketing messages (these are to existing clients, who've checked the
newsletter box, so don't give me grief about it, ok?) pass spamassassin
before they get sent out. SA always seems a twitchy little bastard, and
all sorts of stuff set it off.

Anthony

-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.


From anthony@interlink.com.au  Sat Sep 28 06:04:46 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Sat, 28 Sep 2002 15:04:46 +1000
Subject: [Spambayes] RE: Central Limit Theorem??!! :) 
In-Reply-To: <LNBBLJKPBEHFEDALKOLCIEBOBHAB.tim.one@comcast.net> 
Message-ID: <200209280504.g8S54k007303@localhost.localdomain>


>>> Tim Peters wrote
> + How to compute a sensible score?
>   An honest-to-God probability can be computed from a z-score, and
>   I can supply code to do that (although I'd look on google first
>   for Python code that can already do that; a simple search over a
>   canned sorted list of (abs(zscore), probability) tuples is the
>   quickest way to get a good idea).

Isn't there actually _two_ z scores? "how much does it look like ham"
and "how much does it look like spam"? (I might be mis-understanding the
code here). In this case, isn't the scoring going to need to handle
"hey, both are greater than X sdevs, set it to 50%"

It seems like you'd actually end up with two cutoff numbers, or one
as a difference from 50%... 

Anthony
-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.


From tim.one@comcast.net  Sat Sep 28 06:16:40 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 28 Sep 2002 01:16:40 -0400
Subject: [Spambayes] RE: [Spambayes-checkins] spambayes tokenizer.py,1.40,1.41
In-Reply-To: <E17umON-0001Fi-00@usw-pr-cvs1.sourceforge.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEEKBHAB.tim.one@comcast.net>

[Neil Schemenauer]
> ...
> Modified Files:
> 	tokenizer.py
> Log Message:
> Add basic message-id tokenization.  Right now it just checks that it
> exists and conforms to the usual syntax.  If it does, the host part is
> also returned.  I tried doing more but the extra stuff was never
> considered a good discriminator.  Stupid wins again. :-)

Neil, is there a reason to make this an option?  That is, as opposed to just
doing it all the time?  Like, could this screw up a mixed-source corpus
somehow?  An invalid message id is a strong spam indicator in my
mixed-source corpus for what appear to be legit reasons, and was strong
enough to get rid of 3(!) of my remaining 18 false negatives.

[Classifier] # all current defaults
robinson_probability_x = 0.5
robinson_minimum_prob_strength = 0.1
robinson_probability_s = 0.45
max_discriminators = 150

[Tokenizer]
mine_message_ids: False on the left and True on the right:

-> <stat> tested 2000 hams & 1400 spams against 18000 hams & 12600 spams
   ...

false positive percentages
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.050  0.050  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.050  0.050  tied

won   0 times
tied 10 times
lost  0 times

total unique fp went from 2 to 2 tied
mean fp % went from 0.01 to 0.01 tied

false negative percentages
    0.071  0.071  tied
    0.071  0.071  tied
    0.000  0.000  tied
    0.143  0.071  won    -50.35%
    0.143  0.143  tied
    0.214  0.214  tied
    0.143  0.143  tied
    0.214  0.143  won    -33.18%
    0.286  0.214  won    -25.17%
    0.000  0.000  tied

won   3 times
tied  7 times
lost  0 times

total unique fn went from 18 to 15 won    -16.67%
mean fn % went from 0.128571428571 to 0.107142857143 won    -16.67%

ham mean                     ham sdev
  28.24   28.00   -0.85%        5.81    5.80   -0.17%
  28.17   27.93   -0.85%        5.63    5.62   -0.18%
  28.16   27.91   -0.89%        5.75    5.76   +0.17%
  28.27   28.02   -0.88%        5.68    5.67   -0.18%
  28.07   27.82   -0.89%        5.85    5.85   +0.00%
  28.14   27.88   -0.92%        5.54    5.53   -0.18%
  28.30   28.05   -0.88%        5.69    5.69   +0.00%
  28.25   28.00   -0.88%        5.54    5.54   +0.00%
  28.39   28.14   -0.88%        5.61    5.61   +0.00%
  28.41   28.16   -0.88%        5.94    5.93   -0.17%

ham mean and sdev for all runs
  28.24   27.99   -0.89%        5.71    5.70   -0.18%

spam mean                    spam sdev
  84.88   85.00   +0.14%        6.96    6.92   -0.57%
  84.66   84.80   +0.17%        6.73    6.66   -1.04%
  84.35   84.48   +0.15%        6.62    6.57   -0.76%
  84.88   85.01   +0.15%        6.71    6.65   -0.89%
  84.89   85.01   +0.14%        6.54    6.49   -0.76%
  84.77   84.89   +0.14%        6.87    6.82   -0.73%
  84.46   84.61   +0.18%        6.76    6.68   -1.18%
  84.87   85.00   +0.15%        6.61    6.52   -1.36%
  84.88   85.02   +0.16%        6.85    6.78   -1.02%
  84.81   84.96   +0.18%        6.55    6.47   -1.22%

spam mean and sdev for all runs
  84.75   84.88   +0.15%        6.72    6.66   -0.89%

ham/spam mean difference: 56.51 56.89 +0.38

So it was a small but pure win, reduced the ham mean a little, increased the
spam mean a little and decreased its variance, and increased the mean spread
a little.  At these rates <wink>, can't get much purer than that (the effect
on ham variance was tiny and random)!

Unless someone sees a problem I'm missing, I recommend dropping the option
and always doing this.


From tim.one@comcast.net  Sat Sep 28 06:57:25 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 28 Sep 2002 01:57:25 -0400
Subject: [Spambayes] RE: Central Limit Theorem??!! :)
In-Reply-To: <200209280504.g8S54k007303@localhost.localdomain>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEELBHAB.tim.one@comcast.net>

[Anthony Baxter]
> Isn't there actually _two_ z scores? "how much does it look like ham"
> and "how much does it look like spam"?

Yes.

> (I might be mis-understanding the code here).

Nope, you got it.

> In this case, isn't the scoring going to need to handle
> "hey, both are greater than X sdevs,

Yes.  z-scores are highly non-linear, though, and converting both to
probabilities may (or may not) simplify reasoning about how best to combine
them.

> set it to 50%"
>
> It seems like you'd actually end up with two cutoff numbers, or one
> as a difference from 50%...

The only three outcomes I've seen can only be called "there's no chance this
is spam but some real chance it's ham", "there's no chance this is spam but
some real chance it's spam", and "oops!  best I can tell, there's no earthly
chance it's either".  The "there's some real chance it's ham and some real
chance it's spam" outcome has never happened clearly, except perhaps in very
short msgs (where it's common enough to see both z-scores at about 3 to 4).


From tim.one@comcast.net  Sat Sep 28 07:27:33 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 28 Sep 2002 02:27:33 -0400
Subject: Ham or Spam? (was RE: [Spambayes] RE: Central Limit Theorem??!!
	:))
In-Reply-To: <20020927180734.GC29009@cthulhu.gerg.ca>
Message-ID: <LNBBLJKPBEHFEDALKOLCMEEMBHAB.tim.one@comcast.net>

[Tim]
> FYI, another preliminary observation is that the logarithmic
> central-limit scheme seems (in my data) to be very unsure (high
> sdevs away from both means) about:
>
> + Msgs in German.
> + Msgs in French.
> + Msgs in Spanish.

On second look, this *may* be limited to short msgs in "funny languages".

[Greg Ward]
> Good, I think!  Two of my persistent FPs are:
>
>   * a message from CIRA (the Canadian Internet Registration Authority),
>     which is half in French
>   * a one-line "subscribe" sent to <mumble>-request@python.org,
>     with a loooong disclaimer in Spanish
>
> Both of these are killed by perfectly ordinary, vanilla French and
> Spanish words, because I get very little real email in French and none
> in Spanish.  (Oddly enough, I get very little spam in French too.  In
> fact, most of the spam that gets past SpamAssassin to me is in German,
> which highlights another of SA's weaknesses and a *potential* strength
> of the statistical approach.  I used to get a lot of spam in Portuguese,
> but I added some SA tests that score messages from Brazil and Portugal a
> bit higher.  Seems to have put a stop to that.)

Virtually all words from inferior languages <wink> have high spamprob in my
corpus too.  For a long time, an announcement of a new Spanish Python group
was an f-p for me; I'm not sure what eventually redeemed it.

> I bet Italian would suffer the same fate except that I have a week's
> worth of traffic from zope-it@zope.org in my ham collection.
>
> Incidentally, here are the probs for the CIRA message -- interestingly,
> this is one of the few messages in the gward-ham corpus that was *not*
> sent to gward@python.net, so it still has many traces of my ISP in the
> Received headers.  None of my spam went through my ISP, so all those
> clues are 0.01 clues -- but they still didn't help.

You have to stop using the Graham scheme now (because the code no longer
exists <wink>).  Random cancellation of many 0.01 and 0.99 clues was unique
to it, and was an artifact of its arbitrary limitations.  This kind of thing
doesn't happen under the new scheme.  That doesn't mean it will stop being
an f-p, but it *will* stop getting a score of 1.0.  Let me know how it does
now.

> Data/Ham/Set9/1014826921.10087.cthulhu.gerg.ca:2,S
> prob = 1.0
> prob('membres') = 0.01
> prob('to:videotron.ca') = 0.01
> prob('received:greg') = 0.01
> prob('cher') = 0.01
> prob('positions.') = 0.01
> prob('received:videotron.ca') = 0.01
> prob('header:Return-path:1') = 0.01
> prob('received:bellnexxia.net') = 0.01
> prob('nominations') = 0.01
> prob('re:') = 0.01
> prob('received:24.201') = 0.01
> prob('received:pop.videotron.ca') = 0.01
> prob('received:24.201.245') = 0.01
> prob('avis') = 0.01
> prob('received:24.201.245.36') = 0.01
> prob('received:sc1.videotron.ca') = 0.01
> prob('return-path:members') = 0.01
> prob('received:vl-ms-mr002.sc1.videotron.ca') = 0.01
> prob('des') = 0.9875
> prob('sera') = 0.99
> prob('comit\xe9') = 0.99
> prob('qui') = 0.99
> prob('enabling') = 0.99
> prob('voter') = 0.99
> prob('mai') = 0.99
> prob('besoin') = 0.99
> prob('mises') = 0.99
> prob('apr\xe8s') = 0.99
> prob('ligne') = 0.99
> prob('mars') = 0.99
> prob('candidat') = 0.99
> prob('election') = 0.99
> prob('vous') = 0.99
> prob('aussi') = 0.99
> prob('une') = 0.99
> prob('trois') = 0.99
> prob('entre') = 0.99
> prob('ces') = 0.99
> prob('avril') = 0.99
> prob('leur') = 0.99
> prob('directors.') = 0.99
> prob('objet') = 0.99
> prob('faire') = 0.99
> prob('nominated') = 0.99
> prob('personnes') = 0.99
> prob('veuillez') = 0.99
> prob('elected') = 0.99
> prob('toute') = 0.99
> prob('received:sympatico.ca') = 0.99
> prob('possession,') = 0.99
> prob('poser') = 0.99
> prob('peut') = 0.99
>
> Would be interesting to score just the English half of that message.

Expensive, too <0.5 wink>.

> Oh, here are the probs for the "subscribe" request with Spanish
> disclaimer:

This is more Grahamian "cancellation disease".

> Data/Ham/Set10/17rK7U-0007e7-00:2,S
> prob = 1.0
> prob('subject:subscribe') = 0.01
> prob('confidencial') = 0.01
> prob('autorizado') = 0.01
> prob('contenida') = 0.01
> prob('fon') = 0.01
> prob('mensaje.') = 0.01
> prob('message-id:skip:A 40') = 0.01
> prob('est\xe1') = 0.99
> prob('cual') = 0.99
> prob('mensajes') = 0.99
> prob('pueda') = 0.99
> prob('hacer') = 0.99
> prob('diego') = 0.99
> prob('llegar') = 0.99
> prob('informaci\xf3n') = 0.99
> prob('alguna') = 0.99
> prob('exclusivo') = 0.99
> prob('empresa') = 0.99
> prob('motivo') = 0.99
> prob('telef\xf3nica') = 0.99
> prob('gesti\xf3n') = 0.99
> prob('haber') = 0.99
> prob('received:167]') = 0.99
> prob('propias') = 0.99
> prob('ud.') = 0.99
> prob('todas') = 0.99
> prob('mismo,') = 0.99
> prob('son') = 0.99
> prob('personales') = 0.99
> prob('pueden') = 0.99
> prob('responsable') = 0.99
>
> This message has got to win some sort of prize for the wheat-to-chaff
> ratio: it requires 4.6 kB of signature, MIME junk, and disclaimer to say
> "subscribe".  Here are the headers:
>
> """
> Return-Path: <DQuevedo@uniFON.com.ar>
> Envelope-To: python-list-request@python.org
> Received: from [200.16.211.167] (helo=estcp.tcp.com.ar)
>         by mail.python.org with esmtp (Exim 4.05)
>         id 17rK7U-0007e7-00
>         for python-list-request@python.org; Tue, 17 Sep 2002
> 11:18:32 -0400
> Received: by noticias.unifon.com.ar with Internet Mail Service
> (5.5.2653.19)
>         id <S6X3XHKB>; Tue, 17 Sep 2002 12:15:11 -0300
> Message-ID: <A128D751272CD411BC9200508BC2194D019B6215@escpl.tcp.com.ar>
> From: "Quevedo, Diego" <DQuevedo@uniFON.com.ar>
> To: "'python-list-request@python.org'" <python-list-request@python.org>
> Subject: subscribe
> Date: Tue, 17 Sep 2002 12:15:33 -0300
> Return-Receipt-To: "Quevedo, Diego" <DQuevedo@uniFON.com.ar>
> MIME-Version: 1.0
> X-Mailer: Internet Mail Service (5.5.2653.19)
> Content-Type: multipart/alternative;
>         boundary="----_=_NextPart_001_01C25E5D.1189FAE0"
> """
>
> and here is the text/plain part:
>
> """
> subscribe
>
>
> Diego Quevedo
> Gesti?n de Red
> Tel: (54-11) 4324-9103
> Cel: 15-5132-0135
> dquevedo@unifon.com.ar
> unifon

I'm going to spare everyone a repetition of the rest <wink>.  "subscribe" /
"unsubscribe" msgs are a real bitch, and I expect that under the current
scheme you'll find that both words have high spamprob (well, they did under
the Graham scheme too, but buried under so many bogus probabilities of 0.01
and 0.99 that sometimes you never got to looking at them).  BTW, note that
"subscribe" in the Subject line had a spamprob of 0.01 there:  context
tagging is an extension of the Graham scheme, and was a big win for us.


From tim.one@comcast.net  Sat Sep 28 08:29:57 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 28 Sep 2002 03:29:57 -0400
Subject: [Spambayes] On counting words more than once
In-Reply-To: <B9B9B1E9.169A5%grobinson@transpose.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCOEENBHAB.tim.one@comcast.net>

FYI.  Under the Graham scheme, experiments showed that scoring a word more
than once in a msg hurt during prediction, but that counting a word only
once during training also hurt.

The reasons were clear as mud, though, since (a) boosting hamcounts by a
factor of 2, and (b) ignoring word counts entirely until 2 * hamcount +
spamcount reached 5, both acted in the direction of wanting *some* gimmick
to artificially boost spamcounts too, in order to counteract those biases.
Counting a word more than once during training did act like that for rare
words in spam (e.g., else "Viagra" would have to appear in 5 distinct
training spams before it counted at all).

Under your scheme, though, all that cruft is gone, so the experiment needs a
fresh look.  Here's the results on my large test; counting a word multiple
times in a msg during training is "before", and counting a word only once is
"after":

cvs -> tcaps
-> <stat> tested 2000 hams & 1400 spams against 18000 hams & 12600 spams
   [ditto 19 times]

false positive percentages
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.050  0.050  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.050  0.050  tied

won   0 times
tied 10 times
lost  0 times

total unique fp went from 2 to 2 tied
mean fp % went from 0.01 to 0.01 tied

false negative percentages
    0.071  0.071  tied
    0.071  0.071  tied
    0.000  0.000  tied
    0.071  0.143  lost  +101.41%
    0.143  0.143  tied
    0.214  0.286  lost   +33.64%
    0.143  0.143  tied
    0.143  0.214  lost   +49.65%
    0.214  0.286  lost   +33.64%
    0.000  0.000  tied

won   0 times
tied  6 times
lost  4 times

total unique fn went from 15 to 19 lost   +26.67%
mean fn % went from 0.107142857143 to 0.135714285714 lost   +26.67%

ham mean                     ham sdev
  28.00   25.76   -8.00%        5.80    5.76   -0.69%
  27.93   25.69   -8.02%        5.62    5.56   -1.07%
  27.91   25.64   -8.13%        5.76    5.74   -0.35%
  28.02   25.74   -8.14%        5.67    5.61   -1.06%
  27.82   25.51   -8.30%        5.85    5.78   -1.20%
  27.88   25.58   -8.25%        5.53    5.45   -1.45%
  28.05   25.73   -8.27%        5.69    5.63   -1.05%
  28.00   25.69   -8.25%        5.54    5.47   -1.26%
  28.14   25.91   -7.92%        5.61    5.54   -1.25%
  28.16   25.90   -8.03%        5.93    5.88   -0.84%

ham mean and sdev for all runs
  27.99   25.72   -8.11%        5.70    5.65   -0.88%

spam mean                    spam sdev
  85.00   84.07   -1.09%        6.92    7.10   +2.60%
  84.80   83.83   -1.14%        6.66    6.83   +2.55%
  84.48   83.46   -1.21%        6.57    6.80   +3.50%
  85.01   84.02   -1.16%        6.65    6.87   +3.31%
  85.01   84.07   -1.11%        6.49    6.68   +2.93%
  84.89   83.95   -1.11%        6.82    6.99   +2.49%
  84.61   83.62   -1.17%        6.68    6.84   +2.40%
  85.00   84.02   -1.15%        6.52    6.71   +2.91%
  85.02   84.08   -1.11%        6.78    7.02   +3.54%
  84.96   83.97   -1.17%        6.47    6.65   +2.78%

spam mean and sdev for all runs
  84.88   83.91   -1.14%        6.66    6.85   +2.85%

ham/spam mean difference: 56.89 58.19 +1.30

The reduction in the ham-score mean is consistent and dramatic!  It also
consistently reduced the variance of the ham scores.  I count those as both
good.

The decrease in the spam mean is unattractive, but overall the difference
between the means increased, and that's good.  The increase in spam-score
variance is as surprising to me as the decrease in ham scores, and I'm not
sure what to make of it.

The seemingly dramatic increase in the f-n rate is an artifact of how touchy
spam_cutoff is in this scheme.  Since both mean scores decreased,
spam_cutoff should also be decreased.  I had it set to 0.57.  There's a
single ham scoring above 0.565; if that didn't exist, I could have dropped
spam_cutoff to 0.545 without creating any other f-p, and in the process
gotten rid of 7 false negatives.  In that case the total f-n would have
fallen from 15 to 12, instead of the 4-msg increase reported above.

In short, I suspect it works better under your scheme *not* to count words
multiple times in a msg during training, but there's a blip (1 oddball ham
out of 20000) in my data that obscures this conclusion.

Testers?  You can try this by enabling the new option

[Classifier]
count_duplicates_only_once_in_training: True

It's false by default, but given the above I really want to see how it does
on other test data.  (BTW, I had mine_message_ids enabled too.)


From jcarlson@uci.edu  Sat Sep 28 08:40:09 2002
From: jcarlson@uci.edu (Josiah Carlson)
Date: Sat, 28 Sep 2002 00:40:09 -0700
Subject: [Spambayes] Good evening/morning/afternoon everyone
Message-ID: <20020928002231.CD68.JCARLSON@uci.edu>

Hello spambayes mailing list,

After emailing for a bit with Anthony, he suggested that I join the list. 
So I have.

I am the current developer for PASP (Python Anti-Spam Proxy), a pop3
proxy similar in function to popfile, only written in Python, and alot
lighter in terms of overhead.  http://sourceforge.net/projects/pasp

Similar to spambayes, I started with the ideas covered by Paul Graham 
(in august), then recently added another portion to cover the S filter
described by Gary Robinson.

After browsing the mailing list archives and looking through CVS, I
notice quite a bit of work has been going on to make this project just
about the de-facto standard for bayesian and bayesian-like statistical
email categorization.  Great work everyone.  Hell, even the BDFL himself
is part of this project, which makes me feel good about at least
listening in on what's going on, even better about wanting to contribute.

I also notice that spambayes is using the PSF license.  I've been
considering swapping licenses from GPL to something else to remove the
virus-like quality, and to allow the swapping of code if desirable.  I
know I am interested in taking a look at the spambayes source code, in a
similar fashion that I hope at least one of you is interested in taking
a look at the code for pasp.

Anyways, enough with the introduction, I look forward to collaborating
with everyone here,
 - Josiah Carlson
http://sourceforge.net/projects/pasp
http://come.to/josiah


From richie@entrian.com  Sat Sep 28 10:10:27 2002
From: richie@entrian.com (richie@entrian.com)
Date: Sat, 28 Sep 2002 10:10:27 +0100
Subject: [Spambayes] Good evening/morning/afternoon everyone
In-Reply-To: <20020928002231.CD68.JCARLSON@uci.edu>
Message-ID: <E17vDcJ-000NRH-0Y@anchor-post-34.mail.demon.net>

Hi Josiah, and welcome!

I'm the principle author of pop3proxy.py, the piece of the spambayes
project that does exactly what your PASP project does - we can probably
learn a lot from each other's work.

> I know I am interested in taking a look at the spambayes source code, in a
> similar fashion that I hope at least one of you is interested in taking
> a look at the code for pasp.

Definitely!  I'm away from home right now, but when I get back tomorrow I'll
have a good look at it.

Looking forward to working with you,

-- 
Richie Hindle
richie@entrian.com




From jcarlson@uci.edu  Sat Sep 28 11:35:42 2002
From: jcarlson@uci.edu (Josiah Carlson)
Date: Sat, 28 Sep 2002 03:35:42 -0700
Subject: [Spambayes] Finally getting around to regression testing
Message-ID: <20020928024525.CD6D.JCARLSON@uci.edu>

Hey,

It's kind of funny.  When I was using pasp for just my own stuff, I
didn't really do any testing because I didn't really feel like it.  Now
that I expect that more people are going to be using it and checking it
out...I decided to try doing actual testing on the little guy.

After receiving what was obviously a spam email (the subject I will not
divulge), and having it pass through two different filters (PaulGraham at
=2E02,GaryRobinson p/q/s at .49), I was curious as to what was going on
and why it could pass through so well.

It turned out that those keywords that were making the most difference
in terms of spamminess also didn't show up all that often.  After
spending a little time, I decided to try a simple weighted average.  It
turned out to work pretty well.

Then I thought that maybe the bias in doubling the effectiveness of
ham words was a bit overdone, so I tried dropping to no bias.

So you want some numbers to see what's going on?

Spam: 747
Ham: 5145

testtext -> Paul Graham combination
testtext2 -> Gary Robinson p/q/s combination
testtext3 -> weighted average
(What are the used names for each of these algorithms for future posts?)

As you can see, my output is quite different from what is normally
posted...that's because I'm using a completely different piece of
software, though the numbers should be comparable.

For my testing corpus, I only have 16 spams (the latest 16 spams
received) and 62 hams (all random email in my inbox that I've been too
lazy to categorize...includes two domain expiration notices that are
legit, they show up later)

with the bias of 2*good:
results for testtext:
ctf     fn      fp      fnp             fpp
0.5     2       0       0.125000        0.000000
0.6     2       0       0.125000        0.000000
0.7     2       0       0.125000        0.000000
0.8     2       0       0.125000        0.000000
0.9     2       0       0.125000        0.000000
0.95    2       0       0.125000        0.000000

results for testtext2:
ctf     fn      fp      fnp             fpp
0.5     2       0       0.125000        0.000000
0.6     4       0       0.250000        0.000000
0.7     6       0       0.375000        0.000000
0.8     7       0       0.437500        0.000000
0.9     8       0       0.500000        0.000000
0.95    8       0       0.500000        0.000000

results for testtext3:
ctf     fn      fp      fnp             fpp
0.5     3       0       0.187500        0.000000
0.6     4       0       0.250000        0.000000
0.7     5       0       0.312500        0.000000
0.8     5       0       0.312500        0.000000
0.9     6       0       0.375000        0.000000
0.95    6       0       0.375000        0.000000


without the bias:
results for testtext:
ctf     fn      fp      fnp             fpp
0.5     0       2       0.000000        0.032258
0.6     0       2       0.000000        0.032258
0.7     0       2       0.000000        0.032258
0.8     0       2       0.000000        0.032258
0.9     0       2       0.000000        0.032258
0.95    0       2       0.000000        0.032258

results for testtext2:
ctf     fn      fp      fnp             fpp
0.5     0       2       0.000000        0.032258
0.6     3       0       0.187500        0.000000
0.7     3       0       0.187500        0.000000
0.8     4       0       0.250000        0.000000
0.9     5       0       0.312500        0.000000
0.95    5       0       0.312500        0.000000

results for testtext3:
ctf     fn      fp      fnp             fpp
0.5     0       6       0.000000        0.096774
0.6     0       6       0.000000        0.096774
0.7     0       5       0.000000        0.080645
0.8     1       1       0.062500        0.016129
0.9     3       0       0.187500        0.000000
0.95    4       0       0.250000        0.000000


With the bias, there are no false-positives in my current inbox, and
testtext performs the best with any cutoff.  The two false-negatives are
the aforementioned passed through spam and a dvd for $.49 spam.  I
apparently haven't been receiving that many spams of either type lately.

In any case, upon removal of the bias, I notice that all three
algorithms produce fewer false negatives, but have an increase in false
positives.  It may make sense to check different biases between 1 and 2.=20
I would, but it's 3:30AM here, and I really should get to bed.

Goodnight,
 - Josiah


From skip@pobox.com  Sat Sep 28 14:03:59 2002
From: skip@pobox.com (Skip Montanaro)
Date: Sat, 28 Sep 2002 08:03:59 -0500
Subject: [Spambayes] to From_ or not to From_?
Message-ID: <15765.43199.308115.812787@12-248-11-90.client.attbi.com>


I just thought of something.  If I'm using the MH-style setup (one message
per file in Data/{Ham,Spam}/Set* as understood by timcv.py) are my messages
supposed to have From_ lines or not?

Skip

From skip@pobox.com  Sat Sep 28 14:24:59 2002
From: skip@pobox.com (Skip Montanaro)
Date: Sat, 28 Sep 2002 08:24:59 -0500
Subject: [Spambayes] dumb question - why not simply the subject?
Message-ID: <15765.44459.455695.618431@12-248-11-90.client.attbi.com>

If humans can differentiate ham and spam in the blink of an eye with ne=
ar
100% accuracy looking at just the subject of a message, why should we n=
eed
to feed any other content to a decent spam detector?

na=EFve-ly, y'rs,

Skip

From guido@python.org  Sat Sep 28 15:29:53 2002
From: guido@python.org (Guido van Rossum)
Date: Sat, 28 Sep 2002 10:29:53 -0400
Subject: [Spambayes] to From_ or not to From_?
In-Reply-To: Your message of "Sat, 28 Sep 2002 08:03:59 CDT."
             <15765.43199.308115.812787@12-248-11-90.client.attbi.com> 
References: <15765.43199.308115.812787@12-248-11-90.client.attbi.com> 
Message-ID: <200209281429.g8SETri20138@pcp02138704pcs.reston01.va.comcast.net>

> I just thought of something.  If I'm using the MH-style setup (one message
> per file in Data/{Ham,Spam}/Set* as understood by timcv.py) are my messages
> supposed to have From_ lines or not?

That's not the MH style setup.  There's been quite a bit of confusion
about MH style setup.

In an MH mailbox, only files whose name is a sequence of digits not
starting with 0 are used; everything else is ignored.  MH messages do
*not* have Unix From lines.

In Tim's setup, *every file* in the directory is used; and the
messages *do* have Unix From lines.  To convert an MH folder to Tim
form, splitndirs.py can be used, but you must have a genuine MH setup,
or at least the MH folder must have a path of the form
~/Mail/folder/.

Then you can use +folder to reference it.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From chk@pobox.com  Sat Sep 28 15:55:13 2002
From: chk@pobox.com (Harald Koch)
Date: Sat, 28 Sep 2002 10:55:13 -0400
Subject: [Spambayes] Re: to From_ or not to From_? 
In-Reply-To: guido's message of "Sat, 28 Sep 2002 10:29:53 -0400".
	 <200209281429.g8SETri20138@pcp02138704pcs.reston01.va.comcast.net> 
References: <15765.43199.308115.812787@12-248-11-90.client.attbi.com>
            <200209281429.g8SETri20138@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <2863.1033224913@elisabeth.cfrq.net>

Of all the gin joints in all the towns in all the world, Guido van Rossum
had to walk into mine and say:
>
> That's not the MH style setup.  There's been quite a bit of confusion
> about MH style setup.
> 
> In an MH mailbox, only files whose name is a sequence of digits not
> starting with 0 are used; everything else is ignored.  MH messages do
> *not* have Unix From lines.

Sure they do, at least here.

It appears to depend on whether you're using 'inc' or 'rcvstore' to get
the messages into your folders. The former strips (or converts) From_
lines when splitting a folder; the latter saves the message intact, as
received from the mailer.

I call rcvstore from procmail. To get rid of From_ lines in my MH
folders, I *also* have to turn off sendmail's automatic generation of
From_ lines. Even that doesn't work anymore, since *fetchmail*
autogenerates From_ lines, and I can't seem to turn that off...

Anyway, the point is that the From_ line does sometimes appear in MH
messages. This is definitely a bug, since it breaks -annotate support.

(I'm running nmh-1.0.4).

-- 
Harald Koch     <chk@pobox.com>

From python-spambayes@discworld.dyndns.org  Sat Sep 28 17:44:05 2002
From: python-spambayes@discworld.dyndns.org (Charles Cazabon)
Date: Sat, 28 Sep 2002 10:44:05 -0600
Subject: [Spambayes] Have you ever....?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEEHBHAB.tim.one@comcast.net>;
	from tim.one@comcast.net on Fri, Sep 27, 2002 at 11:50:21PM -0400
References: <20020928034609.2C791F5B5@cashew.wolfskeep.com>
	<LNBBLJKPBEHFEDALKOLCGEEHBHAB.tim.one@comcast.net>
Message-ID: <20020928104405.B5685@discworld.dyndns.org>

Tim Peters <tim.one@comcast.net> wrote:
> [T. Alexander Popiel]
> > As far as I can tell, these are tags to indicate which email
> > address is the hit when people respond to spam.  In my single
> > source collection, there are a handful of these 'words' which
> > have appeared many times... so my best guess is that there
> > are a handful of spam-blast software packages that compute
> > these codes reproducably based on the email address.
> 
> Interesting!  It's sure plausible.  At least one of us should become a
> professional spammer so we get a better handle on these tricks.  Anyone want
> to get rich quick?

You're at least getting the lingo down, Tim, but you give spammers too much
credit.  The strings at the end of the message, normally enclosed in square
brackets, are only there to foil the DCC (Distributed Checksum something or
other) project which blocks spam based on message checksums.  There appear to
be only two pieces of spamware that do this; one puts multiple pieces of
alphanumeric garbage in a single set of square brackets, separated by
hyphens/dashes like this:

  [SdfA983DF-DFSDKJSDF-D7sadf7jd]

while the other uses a single long string inside square brackets.  A recent
addition is using two lines like the above instead of one.

Some spammers do include tags in the message to identify the recipient of the
message, but they're much easier to pick out, and are normally either the
email address of the recipient, or a single decimal integer.  I've been
waiting for the spammers to get smarter about hiding these, but the best
effort I've seen yet is putting them inside an HTML comment.

As a side note, the author of qmail has long said that someday, someone
competent would join the spammers, and that the anti-spam fight would get a
whole lot tougher then.  I don't think it's happened yet, but until Graham,
Robinson, and Peters <wink>, anti-spam measures have mostly been laughable.

Charles
-- 
-----------------------------------------------------------------------
Charles Cazabon                 <python-spambayes@discworld.dyndns.org>
GPL'ed software available at:     http://www.qcc.ca/~charlesc/software/
-----------------------------------------------------------------------

From tim.one@comcast.net  Sat Sep 28 18:43:30 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 28 Sep 2002 13:43:30 -0400
Subject: [Spambayes] to From_ or not to From_?
In-Reply-To: <200209281429.g8SETri20138@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEFKBHAB.tim.one@comcast.net>

[Guido]
> In Tim's setup, *every file* in the directory is used;

This is so.

> and the messages *do* have Unix From lines.

Actually, none of mine do, because BruceG's spam didn't.  I removed all the
"From " lines from the c.l.py archive to match that (easier than inventing
such lines for Bruce's msgs).  I don't know that it makes any difference for
the way I run the tests, but it certainly could make a difference if "From "
lines were getting mined for clues.  I forced all my msgs alike in this
respect just to cut off that possibility.


From popiel@wolfskeep.com  Sat Sep 28 18:46:32 2002
From: popiel@wolfskeep.com (T. Alexander Popiel)
Date: Sat, 28 Sep 2002 10:46:32 -0700
Subject: [Spambayes] Have you ever....? 
In-Reply-To: Message from Charles Cazabon
	<python-spambayes@discworld.dyndns.org> 
	of "Sat, 28 Sep 2002 10:44:05 MDT."
	<20020928104405.B5685@discworld.dyndns.org> 
References: <20020928034609.2C791F5B5@cashew.wolfskeep.com>
	<LNBBLJKPBEHFEDALKOLCGEEHBHAB.tim.one@comcast.net>
	<20020928104405.B5685@discworld.dyndns.org> 
Message-ID: <20020928174632.D3920F4D4@cashew.wolfskeep.com>

In message:  <20020928104405.B5685@discworld.dyndns.org>
             Charles Cazabon <python-spambayes@discworld.dyndns.org> writes:
>> [T. Alexander Popiel]
>> > As far as I can tell, these are tags to indicate which email
>> > address is the hit when people respond to spam.  [...] so my
>> > best guess is that there are a handful of spam-blast software
>> > packages that compute these codes reproducably based on the
>> > email address.

>[...] you give spammers too much credit.  The strings at the end of
>the message [...] are only there to foil the DCC [...]

>Some spammers do include tags in the message to identify the recipient of the
>message, but they're much easier to pick out, and are normally either the
>email address of the recipient, or a single decimal integer.  I've been
>waiting for the spammers to get smarter about hiding these, but the best
>effort I've seen yet is putting them inside an HTML comment.

I'm obviously getting spam from different sources from you; one I just
received last night contained the line:

cbcvry^fahtuneobe(pbz

... which has appeared in 111 (of 2523) spam messages with radically
different text ranging from credit card offers to home based employment
to bestiality videos.

After looking at it a little more, I see that it's a simple cipher on
my email address:

  abcdefghijklmnopqrstuvwxyz @.
  nop r t v  yzabc ef h      ^(

A bit of interpolation shows that they're using an augmented rot13.
This is confirmed by doing a search on a slightly different form of
the address "cbcvry^jbysfxrrc(pbz" and getting another 12 hits in my
spam corpus.  (Those reading closely will discover that I've got
two domain names.)

Apparently the spammers are getting marginally better about hiding
the tracking info... at least they're enciphering it, now. ;-)

Uh, oh... I just cracked an encryption scheme used to protect
tracking data.  I better worry about the spammers suing me for
DMCA violations.

- Alex

From tim.one@comcast.net  Sat Sep 28 18:55:09 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 28 Sep 2002 13:55:09 -0400
Subject: [Spambayes] dumb question - why not simply the subject?
In-Reply-To: <15765.44459.455695.618431@12-248-11-90.client.attbi.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEFLBHAB.tim.one@comcast.net>

[Skip Montanaro]
> If humans can differentiate ham and spam in the blink of an eye with
> near 100% accuracy looking at just the subject of a message, why
> should we need to feed any other content to a decent spam detector?

If it's true that humans can do this, we shouldn't need to.  What evidence
do you have for believing the antecedent, though?  I'll bet a dollar that
our classifier today would blow humans out of the water on both error rates
if such an experiment were to be conducted.

First we need to decide how much time we're going to give people for each
msg.

    http://www.improb.com/news/2000/aug2000/impe-twinkle-2000-08.html
    Hakkanen, Summala, et al. found a typical 8.23 millisecond blink
    duration in sleepy bus drivers, and a typical 5.19 millisecond
    blink duration in non-sleepy non-bus drivers.

Even at the slothful sleepy bus driver rate, we'll have to hold our human
subjects to classifying at least 120 messages per second <wink>.


From nas@python.ca  Sat Sep 28 19:15:37 2002
From: nas@python.ca (Neil Schemenauer)
Date: Sat, 28 Sep 2002 11:15:37 -0700
Subject: [Spambayes] RE: [Spambayes-checkins] spambayes
	tokenizer.py,1.40,1.41
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEEKBHAB.tim.one@comcast.net>
References: <E17umON-0001Fi-00@usw-pr-cvs1.sourceforge.net>
	<LNBBLJKPBEHFEDALKOLCKEEKBHAB.tim.one@comcast.net>
Message-ID: <20020928181536.GA21963@glacier.arctrix.com>

Tim Peters wrote:
> Neil, is there a reason to make this an option?  That is, as opposed to just
> doing it all the time?  Like, could this screw up a mixed-source corpus
> somehow?

I don't think so.  In your ham collection there are probably certain
hosts that appear in the message IDs often.  The hosts for the spam
message IDs should be pretty random.  Either they are generated by the
spammer or by the open relay MTA.

  Neil

From jcarlson@uci.edu  Sat Sep 28 19:42:13 2002
From: jcarlson@uci.edu (Josiah Carlson)
Date: Sat, 28 Sep 2002 11:42:13 -0700
Subject: [Spambayes] realization of mistakes...
Message-ID: <20020928102746.D67A.JCARLSON@uci.edu>

Good day,

I've finally had a chance to go through the current spambayes code in
cvs.  I notice that the project looks to have abandoned Paul Graham's
ideas entirely, as I could not find anything that looked remotely like
what was offered in his paper.

I suppose this is not necessarily a good thing or a bad thing, but
through recent developments in pasp (in the last 24 hours), I've noticed
that it is good to have multiple classifiers to compare to one another. 
Considering the experimental status of statistical email categorization,
I believe that the more information one can have about the performance
of varying algorithms, the better.  But that could be just me.


On that note, I've looked through the Gary Robinson implementation that
is in CVS.  It looks like you have done a complete implementation of
those algorithms.  I should note that those numbers I gave earlier for
my Gary Robinson algorithm, was basically a drop-in replacement from
Paul Graham's (converted for comparison sake):
P = ((1-p1)*(1-p2)*...*(1-pn))
Q = (p1*p2*...*pn)
S = Q / ( P + Q )

to Gary Robinson's:
P = 1 - ((1-p1)*(1-p2)*...*(1-pn))^(1/n)
Q = 1 - (p1*p2*...*pn)^(1/n)
S = .5 +  ((P - Q) / (P + Q)) / 2

I apologize for any confusion as to why the Gary Robinson version I use
doesn't do so well, it's because I haven't added any of the "Further
Improvements", which apparently does do alot.  I've been contemplating
borrowing some classification code from classifier.py, but am shying
away because I like to keep things simple, mostly because I like to
understand what is going on with the math I use.  Of course that makes
comparison between algorithms used and fn/fp performance somewhat
difficult, but I believe that at least some useful information can be
gained.


If anyone is curious, I decided to continue on the thought path that led
me to alter the bias in Paul Graham's probability function last evening.
          bias
            |
(let ((g (* 2 (or (gethash word good) 0)))
      (b (or (gethash word bad) 0)))
   (unless (< (+ g b) 5)
     (max .01
          (min .99 (float (/ (min 1 (/ b nbad))
                             (+ (min 1 (/ g ngood))   
                                (min 1 (/ b nbad)))))))))

Hams: 5145
Spams: 747
Test hams: 62
Test spams: 122

Using the cutoffs [.5, .6, .7, .8, .9, .95] to find ranges of fn/fp.

If you don't feel like reading the ranges, considering the desireable 
as low as possible fn, and zero fp, on my corpus of
spam/ham/testspam/testham, it seems that adjusting the per-word probability
function bias to some number in the range of 1-1.5 produces both a
minimum number of false negatives, and zero false positives with basic
Paul Graham.

What is also quite interesting is that just a plain weighted average
performs (on the most part), somewhere between PaulGraham and
GaryRobinson.

Anyways, I'll stop typing and get around to doing another release with
what I currently know.

 - Josiah

Bias 2:
alg   fn range in %   fp range in %
PG -> [5 - 5]         [0 - 0]
GR -> [5 - 43]        [0 - 0]
WA -> [9 - 23]        [0 - 0]

Bias 1.75:
alg   fn range in %   fp range in %
PG -> [3 - 3]         [0 - 0]
GR -> [3 - 43]        [0 - 0]
WA -> [7 - 20]        [0 - 0]

Bias 1.5:
alg   fn range in %   fp range in %
PG -> [2 - 2]         [0 - 0]
GR -> [2 - 41]        [0 - 0]
WA -> [4 - 19]        [0 - 0]

Bias 1.25:
alg   fn range in %   fp range in %
PG -> [1 - 1]         [0 - 0]
GR -> [1 - 35]        [0 - 0]
WA -> [2 - 16]        [0 - 0]

Bias 1:
alg   fn range in %   fp range in %
PG -> [1 - 1]         [3 - 3]
GR -> [1 - 23]        [3 - 0]
WA -> [2 - 12]        [10 - 0]



From nas@python.ca  Sat Sep 28 19:49:16 2002
From: nas@python.ca (Neil Schemenauer)
Date: Sat, 28 Sep 2002 11:49:16 -0700
Subject: [Spambayes] On counting words more than once
In-Reply-To: <LNBBLJKPBEHFEDALKOLCOEENBHAB.tim.one@comcast.net>
References: <B9B9B1E9.169A5%grobinson@transpose.com>
	<LNBBLJKPBEHFEDALKOLCOEENBHAB.tim.one@comcast.net>
Message-ID: <20020928184916.GA22062@glacier.arctrix.com>

Tim Peters wrote:
> Testers?  You can try this by enabling the new option
> 
> [Classifier]
> count_duplicates_only_once_in_training: True

It's a win for me:

    false positive percentages
        0.000  0.000  tied          
        1.000  1.000  tied          
        0.000  0.000  tied          
        0.000  0.000  tied          
        0.500  0.500  tied          
        0.500  1.000  lost  +100.00%
        0.500  0.500  tied          
        0.000  0.000  tied          
        0.500  0.500  tied          
        0.000  0.500  lost  +(was 0)

    won   0 times
    tied  8 times
    lost  2 times

    total unique fp went from 6 to 8 lost   +33.33%
    mean fp % went from 0.3 to 0.4 lost   +33.33%

    false negative percentages
        0.000  0.000  tied          
        1.000  0.500  won    -50.00%
        1.000  0.500  won    -50.00%
        0.500  0.500  tied          
        2.000  1.500  won    -25.00%
        1.500  1.000  won    -33.33%
        0.000  0.000  tied          
        0.500  0.000  won   -100.00%
        0.500  0.000  won   -100.00%
        0.000  0.000  tied          

    won   6 times
    tied  4 times
    lost  0 times

    total unique fn went from 14 to 8 won    -42.86%
    mean fn % went from 0.7 to 0.4 won    -42.86%

    ham mean                     ham sdev
      30.01   27.92   -6.96%        8.43    8.42   -0.12%
      28.50   26.74   -6.18%        8.83    8.69   -1.59%
      27.93   26.04   -6.77%        8.20    7.94   -3.17%
      29.55   27.33   -7.51%        8.24    8.23   -0.12%
      29.05   27.19   -6.40%        8.28    8.15   -1.57%
      31.40   29.48   -6.11%        9.41    9.25   -1.70%
      29.31   27.49   -6.21%        8.13    8.10   -0.37%
      29.33   27.16   -7.40%        7.86    7.89   +0.38%
      28.72   27.22   -5.22%        9.05    8.97   -0.88%
      29.04   26.87   -7.47%        7.28    7.22   -0.82%

    ham mean and sdev for all runs
      29.28   27.34   -6.63%        8.44    8.35   -1.07%

    spam mean                    spam sdev
      82.98   81.91   -1.29%        9.83   10.16   +3.36%
      82.02   81.04   -1.19%        9.92   10.09   +1.71%
      81.19   80.28   -1.12%        9.69    9.86   +1.75%
      82.51   81.66   -1.03%        9.92   10.23   +3.13%
      82.60   81.60   -1.21%       10.12   10.33   +2.08%
      82.24   81.36   -1.07%        9.25    9.71   +4.97%
      81.74   80.85   -1.09%        9.30    9.49   +2.04%
      81.70   80.64   -1.30%        9.51    9.81   +3.15%
      82.39   81.45   -1.14%        9.87   10.18   +3.14%
      82.44   81.45   -1.20%        9.49    9.73   +2.53%

    spam mean and sdev for all runs
      82.18   81.22   -1.17%        9.71    9.97   +2.68%

    ham/spam mean difference: 52.90 53.88 +0.98

From guido@python.org  Sat Sep 28 19:49:34 2002
From: guido@python.org (Guido van Rossum)
Date: Sat, 28 Sep 2002 14:49:34 -0400
Subject: [Spambayes] to From_ or not to From_?
In-Reply-To: Your message of "Sat, 28 Sep 2002 13:43:30 EDT."
             <LNBBLJKPBEHFEDALKOLCCEFKBHAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCCEFKBHAB.tim.one@comcast.net> 
Message-ID: <200209281849.g8SInZp20629@pcp02138704pcs.reston01.va.comcast.net>

> > and the messages *do* have Unix From lines.
> 
> Actually, none of mine do, because BruceG's spam didn't.  I removed
> all the "From " lines from the c.l.py archive to match that (easier
> than inventing such lines for Bruce's msgs).  I don't know that it
> makes any difference for the way I run the tests, but it certainly
> could make a difference if "From " lines were getting mined for
> clues.  I forced all my msgs alike in this respect just to cut off
> that possibility.

Weird.  I used splitndirs.py to create my normalized test data setup
and it wrote Unix From lines.  In fact, looking at the code, it uses
str(msg), which forces unixfrom=1, which always writes a Unix From
line.

But it's possible that you created your data setup using a different
version of splitndirs.py.

Anyway, the email package always recognizes a Unix From line (it's
hard to mistake for an rfc822 header line) and stores it in a special
attribute of the Message object.  Unless you wrote code in your
tokenizer to look at that, I'm pretty sure you're ignoring it. :-)

So Skip can stop worrying: presence or absence of Unix From lines
doesn't matter.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From bkc@murkworks.com  Sat Sep 28 20:04:45 2002
From: bkc@murkworks.com (Brad Clements)
Date: Sat, 28 Sep 2002 15:04:45 -0400
Subject: [Spambayes] Only one .. some #'s
Message-ID: <3D95C4FF.18765.3A1BCB6@localhost>

Sorry, I missed the "only one shoot out".

I've updated my src tree and re-ran timcv using mostly defaults. I then compared those 
results with my last (earlier) results.

Note that I still have some spam in my ham.. I haven't changed my corpus since 
starting the tests.

Executive summary:  The algorithm is getting better every day.


So comparing (Old)

[TestDriver]
save_trained_pickles = True
show_histograms = True
show_ham_lo = 1.0
show_best_discriminators = 30
show_spam_lo = 0.0
show_ham_hi = 0.0
show_false_positives = True
pickle_basename = class
show_false_negatives = True
spam_cutoff = 0.575
nbuckets = 40
show_spam_hi = 0.45
show_charlimit = 3000

[Classifier]
min_spamprob = 0.01
use_robinson_combining = True
hambias = 2.0
use_robinson_ranking = False
spambias = 1.0
robinson_probability_x = 0.5
use_robinson_probability = True
robinson_minimum_prob_strength = 0.1
unknown_spamprob = 0.5
max_discriminators = 1500
robinson_probability_a = 1.0
max_spamprob = 0.99

[Tokenizer]
ignore_redundant_html = False
mine_received_headers = True
count_all_header_lines = False
retain_pure_html_tags = False
basic_header_tokenize = False
safe_headers = abuse-reports-to

with (New)

[TestDriver]
pickle_basename = class
save_trained_pickles = True
show_histograms = True
show_ham_lo = 1.0
show_best_discriminators = 30
nbuckets = 40
show_ham_hi = 0.0
spam_cutoff = 0.575
spam_directories = Data/Spam/Set%d
show_spam_lo = 0.0
show_false_negatives = True
ham_directories = Data/Ham/Set%d
compute_best_cutoffs_from_histograms = True
show_false_positives = True
best_cutoff_fp_weight = 1
show_spam_hi = 0.45
save_histogram_pickles = True
show_charlimit = 3000

[Classifier]
count_duplicates_only_once_in_training = True
robinson_probability_x = 0.5
robinson_minimum_prob_strength = 0.1
robinson_probability_s = 0.45
max_discriminators = 150
use_central_limit2 = False
use_central_limit = False

[Tokenizer]
mine_received_headers = True
octet_prefix_size = 5
count_all_header_lines = False
check_octets = False
ignore_redundant_html = False
mine_message_ids = False
basic_header_tokenize = False
safe_headers = abuse-reports-to

New Hist

-> <stat> Ham scores for all runs: 13000 items; mean 26.25; sdev 7.31
* = 41 items
  0.0   22 *
  2.5   17 *
  5.0   26 *
  7.5   82 **
 10.0   87 ***
 12.5  260 *******
 15.0  476 ************
 17.5  913 ***********************
 20.0 1734 *******************************************
 22.5 2479 *************************************************************
 25.0 2288 ********************************************************
 27.5 1680 *****************************************
 30.0  948 ************************
 32.5  557 **************
 35.0  473 ************
 37.5  428 ***********
 40.0  158 ****
 42.5  101 ***
 45.0   81 **
 47.5   59 **
 50.0   50 **
 52.5   29 *
 55.0   19 *
 57.5   13 *
 60.0    5 *
 62.5    5 *
 65.0    3 *
 67.5    2 *
 70.0    2 *
 72.5    1 *
 75.0    0
 77.5    2 *
 80.0    0

-> <stat> Spam scores for all runs: 13000 items; mean 80.77; sdev 8.46
* = 38 items
  0.0    0
  2.5    0
  5.0    0
  7.5    0
 10.0    0
 12.5    0
 15.0    0
 17.5    0
 20.0    1 *
 22.5    3 *
 25.0    2 *
 27.5    3 *
 30.0    0
 32.5    6 *
 35.0   37 *
 37.5   37 *
 40.0   12 *
 42.5    8 *
 45.0   11 *
 47.5   13 *
 50.0   15 *
 52.5   36 *
 55.0   59 **
 57.5   65 **
 60.0  131 ****
 62.5  178 *****
 65.0  244 *******
 67.5  322 *********
 70.0  532 **************
 72.5  646 *****************
 75.0  900 ************************
 77.5 1386 *************************************
 80.0 1891 **************************************************
 82.5 2300 *************************************************************
 85.0 2031 ******************************************************
 87.5 1304 ***********************************
 90.0  471 *************
 92.5  232 *******
 95.0   72 **
 97.5   52 **
-> best cutoff for all runs: 0.525
->     with weighted total 1*81 fp + 148 fn = 229
->     fp rate 0.623%  fn rate 1.14%

Comparing old (left) to new (right)


false positive percentages
    0.538  0.308  won    -42.75%
    0.308  0.308  tied          
    0.308  0.231  won    -25.00%
    0.538  0.308  won    -42.75%
    0.385  0.231  won    -40.00%
    0.154  0.077  won    -50.00%
    0.462  0.231  won    -50.00%
    0.231  0.154  won    -33.33%
    0.538  0.385  won    -28.44%
    0.385  0.308  won    -20.00%

won   9 times
tied  1 times
lost  0 times

total unique fp went from 50 to 33 won    -34.00%
mean fp % went from 0.384615384615 to 0.253846153846 won    -34.00%

false negative percentages
    1.615  2.000  lost   +23.84%
    2.385  2.462  lost    +3.23%
    2.000  1.769  won    -11.55%
    1.615  1.692  lost    +4.77%
    1.769  1.769  tied          
    1.615  1.615  tied          
    1.385  1.692  lost   +22.17%
    1.846  1.615  won    -12.51%
    2.000  2.000  tied          
    2.077  2.077  tied          

won   2 times
tied  4 times
lost  4 times

total unique fn went from 238 to 243 lost    +2.10%
mean fn % went from 1.83076923077 to 1.86923076923 lost    +2.10%

ham mean                     ham sdev
  29.46   26.30  -10.73%        7.32    7.57   +3.42%
  29.31   26.18  -10.68%        7.24    7.29   +0.69%
  29.90   26.61  -11.00%        7.11    7.37   +3.66%
  29.83   26.46  -11.30%        7.13    7.38   +3.51%
  29.47   26.27  -10.86%        7.23    7.37   +1.94%
  29.62   26.41  -10.84%        7.10    7.08   -0.28%
  29.36   26.21  -10.73%        7.10    7.23   +1.83%
  29.23   26.03  -10.95%        6.91    7.00   +1.30%
  29.13   25.97  -10.85%        7.25    7.46   +2.90%
  29.32   26.09  -11.02%        6.97    7.31   +4.88%

ham mean and sdev for all runs
  29.46   26.25  -10.90%        7.14    7.31   +2.38%

spam mean                    spam sdev
  78.63   80.57   +2.47%        7.79    8.42   +8.09%
  78.67   80.68   +2.55%        8.25    8.95   +8.48%
  78.85   80.83   +2.51%        8.15    8.70   +6.75%
  79.00   81.11   +2.67%        7.81    8.22   +5.25%
  78.79   80.64   +2.35%        7.77    8.20   +5.53%
  79.05   81.05   +2.53%        7.37    7.82   +6.11%
  78.52   80.65   +2.71%        7.60    8.26   +8.68%
  78.71   80.70   +2.53%        7.99    8.41   +5.26%
  79.02   80.99   +2.49%        8.05    8.64   +7.33%
  78.46   80.49   +2.59%        8.37    8.87   +5.97%

spam mean and sdev for all runs
  78.77   80.77   +2.54%        7.92    8.46   +6.82%

ham/spam mean difference: 49.31 54.52 +5.21





Brad Clements,                bkc@murkworks.com   (315)268-1000
http://www.murkworks.com                          (315)268-9812 Fax
AOL-IM: BKClements


From B-Morgan@concentric.net  Sat Sep 28 20:50:52 2002
From: B-Morgan@concentric.net (Brad Morgan)
Date: Sat, 28 Sep 2002 13:50:52 -0600
Subject: [Spambayes] Help for Windows / Outlook Users
In-Reply-To: <3D95C4FF.18765.3A1BCB6@localhost>
Message-ID: <NABBJOOEOFODEALNMJAJEEBEGNAA.B-Morgan@concentric.net>

Hello,

I am following this discussion with great interest and would like to
participate.

My primary mail usage is on Windows using Outlook.  I have a Linux machine
acting as my firewall and has minimal mail capabilities.

I have installed Python 2.2.1 on my main machine and the next major obstacle
is figuring out how to create my spam and ham corpus (or should that be
corpora?).  Can someone on this list provide some hints?

Thanks,

Brad Morgan



From tim.one@comcast.net  Sat Sep 28 21:01:23 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 28 Sep 2002 16:01:23 -0400
Subject: [Spambayes] RE: Central Limit Theorem??!! :)
In-Reply-To: <LNBBLJKPBEHFEDALKOLCIEELBHAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEGABHAB.tim.one@comcast.net>

This is a log-central-limit experiment with a 10x10 grid.  500 ham and 500
spam selected at random from each of my sets, lumped into 10 pairs.  Then

train on pair 1, predict on pairs 2 thru 10
train on pair 2, predict on pairs 1, and 3 thru 10
...
train on pair 10, predict on pairs 1 thru 9

In all, that's 90 prediction runs on 1000 msgs per run, for 90,000 total
predictions.  Each of the 10*1000 = 10,000 unique msgs is predicted 9 times.

This is a hard test, as 500ham+500spam isn't a lot of training data, and
each of the 10 classifiers has to predict against 9x more msgs than it's
been taught about.

There are only 4 distinct scores in this experiment:

0.00    I'm sure it's ham.
0.49    I guess it's ham, but I may be baffled.
0.51    I guess it's spam, but I may be baffled.
1.00    I'm sure it's spam.

This code replaced central_limit_spamprob2()'s score computation (the first
3 lines are already there; I include them for context):

        zham = (hmean - self.hammean) / sqrt(self.hamvar / n)
        zspam = (smean - self.spammean) / sqrt(self.spamvar / n)
        stat = abs(zham) - abs(zspam)  # > 0 for spam, < 0 for ham

        if min(abs(zham), abs(zspam)) < 10.0 and abs(stat) > 5.0:
            stat = stat > 0.0 and 1.0 or 0.0
        else:
            print '*', n, zham, zspam, stat,
            if stat > 0.0:
                stat = 0.51
            else:
                stat = 0.49
            print stat

IOW, it's certain iff at least one z-score is "not insanely large", and the
other z-score is "substantially larger".  This is purely a hack just to see
what would happen.  No direct accounting is made of the number of words in a
message, although because n is in the denominators of the sample mean
variances, a small n leads to larger variances than a large n, and that
makes it much harder for the "and abs(stat) > 5.0" to succeed.

Relevant non-default options:

"""
[Tokenizer]
mine_message_ids: True

[Classifier]
count_duplicates_only_once_in_training: True
max_discriminators: 50
use_central_limit2: True

[TestDriver]
spam_cutoff: 0.50
nbuckets: 4
"""

Note that I boosted max_discriminators from 30.  I'm not sure why I did.
Why not <wink>?

The bottom lines are in the tiny histograms:

-> <stat> Ham scores for all runs: 45000 items; mean 1.18; sdev 7.51
* = 720 items
  0.0 43917 *************************************************************
 25.0  1069 **    unsure and guessed right
 50.0    14 *     unsure and guessed wrong
 75.0     0       sure and guessed wrong

-> <stat> Spam scores for all runs: 45000 items; mean 99.04; sdev 6.91
* = 724 items
  0.0    12 *     sure and guessed wrong
 25.0    56 *     unsure and guessed wrong
 50.0   802 **    unsure and guessed right
 75.0 44130 *************************************************************

So,

When predicting ham:
    97.59% of the time it was sure of itself.
        In those cases, it was never wrong.
     2.41% of the time it had strong reason to doubt itself.
        In those cases, it guessed wrong 1.31% of the time.

When predicting spam:
    98.09% of the time it was sure of itself.
        In those cases, it was wrong 0.027% of the time.
     1.91% of the time it had strong reason to doubt itself.
        In those cases, it guessed wrong 6.53% of the time.

Recall that each message is predicted against by 9 different classifiers
here.  The 12 cases in which spam was misidentified are due to fewer than 12
distinct messages.  The first has few words (so the central-limit gimmick
has an excuse <wink>):

"""
Data/Spam/Set6/11926.txt
prob = 0.0
prob('steve') = 0.0847751
prob('mike,') = 0.0918367
prob('subject:Thanks') = 0.155172
prob('regards,') = 0.171593
prob('thanks') = 0.285091
prob('content-type:text/plain') = 0.308556
prob('again') = 0.640598
prob('header:Reply-To:1') = 0.701388
prob('best') = 0.736387
prob('header:Message-Id:1') = 0.823243
prob('header:Return-Path:1') = 0.906194
prob('subject:For') = 0.908163
prob('great!') = 0.908163

Return-Path: <joelackey@ns1.ehost2102.com>
Delivered-To: em-ca-bait@em.ca
Received: (qmail 15585 invoked from network); 28 Feb 2002 18:30:34 -0000
Received: from unknown (HELO smtp4.westnet24.com) (64.215.52.101)
  by agamemnon.bfsmedia.com with SMTP; 28 Feb 2002 18:30:34 -0000
Date: Thu, 28 Feb 2002 13:37:57 -0500
Message-Id: <200202281837.g1SIbvi09018@smtp4.westnet24.com>
X-Mailer: Mozilla 4.6 [en] (WinNT; U)
Reply-To: <joelackey@ns1.ehost2102.com>
From: <joelackey@ns1.ehost2102.com>
To: <bait_7@earthlink.net>
Subject: Thanks For Lunch Mike                    beksng
Content-Length: 71
Lines: 10

Mike,


  Thanks again for lunch it was great!


Best regards,

Steve
"""

The second also had few words, and was a mass of Javascript without any
whitespace except for arbitrary line breaks (each line is about the same
length).  The "words" show that the tokenizer really had no idea what to do
with this:

prob('skip:" 30') = 0.0412844
prob('function') = 0.072284
prob('skip:( 50') = 0.0918367
prob('skip:" 70') = 0.0918367
prob('skip:" 60') = 0.155172
prob('skip:f 20') = 0.292388
prob('subject:new') = 0.298658
prob('skip:" 10') = 0.341427
prob('header:Message-ID:1') = 0.368361
prob('else') = 0.373061
prob('header:MIME-Version:1') = 0.634608
prob('charset:iso-8859-1') = 0.749673
prob('subject:information') = 0.844828
prob('header:Return-Path:1') = 0.906194
prob('skip:f 50') = 0.908163
prob('skip:= 10') = 0.958716
prob('content-type:text/html') = 0.999284

I got too bored to continue then.  Unfortunately, I didn't have enough
output to determine what the z-scores for these guys were.

I'm not a statistician, and I'm sure there are better ways to extract "a
decision" and "a confidence" out of the z-scores; this was just a quick hack
to see whether it had promise.

One thing I noted here is that, with this little training data, the
population ham variances across classifiers didn't hold steady (these are
the mean of the ln(1-spamprob) "extreme word" ham population statistics):

hammean -0.275301462139 hamvar 0.241400997965
hammean -0.270075033304 hamvar 0.237971084164
hammean -0.27729534552  hamvar 0.242090771903
hammean -0.28018840187  hamvar 0.242071754063
hammean -0.262743024167 hamvar 0.226818888396
hammean -0.293745611772 hamvar 0.272036821231
hammean -0.271746041979 hamvar 0.232452901988
hammean -0.24251831616  hamvar 0.186177919836
hammean -0.256875930876 hamvar 0.208263894923
hammean -0.264274555704 hamvar 0.221385382894

Ditto for population spam variance:

spammean -0.147931424503 spamvar 0.0733613298447
spammean -0.157632947165 spamvar 0.0966048083774
spammean -0.153901431979 spamvar 0.0750210065953
spammean -0.143638282908 spamvar 0.0653981265422
spammean -0.152840521441 spamvar 0.0825987634543
spammean -0.146861931682 spamvar 0.0760974846927
spammean -0.150647341728 spamvar 0.0733247594002
spammean -0.167612616966 spamvar 0.10231334993
spammean -0.163965666611 spamvar 0.0985899485577
spammean -0.152115124735 spamvar 0.0776889891763


Finally, exactly the same test but with the current default scheme, fiddled
via

"""
[Tokenizer]
mine_message_ids: True

[Classifier]
count_duplicates_only_once_in_training: True

[TestDriver]
spam_cutoff: 0.56
"""

(spam_cutoff is reduced from the default because
count_duplicates_only_once_in_training reduces both means)

total unique false pos 7
total unique false neg 27
average fp % 0.0555555555556
average fn % 0.142222222222

That's not too shabby itself, although note that "total unique" only counts
distinct msgs misclassified, and, e.g., if 9 classifiers all call a
particular ham a spam, it only counts once against the "total unique false
pos" count.  It's hard to compare these runs directly.

I noticed that the two specific squashed-Javascript and "Thanks again for
lunch" spams mentioned above were popular false negatives under this run
too.


From jcarlson@uci.edu  Sat Sep 28 21:14:24 2002
From: jcarlson@uci.edu (Josiah Carlson)
Date: Sat, 28 Sep 2002 13:14:24 -0700
Subject: [Spambayes] Help for Windows / Outlook Users
In-Reply-To: <NABBJOOEOFODEALNMJAJEEBEGNAA.B-Morgan@concentric.net>
References: <3D95C4FF.18765.3A1BCB6@localhost>
	<NABBJOOEOFODEALNMJAJEEBEGNAA.B-Morgan@concentric.net>
Message-ID: <20020928130829.D67D.JCARLSON@uci.edu>

Brad,

I do not know if there is a currently accepted way to create the
corpus...corpusai...corpai...corpus (I really don't know either), but if
you have access to most any version of netscape mail that comes with
netscape navigator, or mozilla mail and news that comes with mozilla, or
even eudora, you can do it.

Import your Outlook email, then export it into unix mbox format, which I
believe works with Spambayes.

I hope this helps,
 - Josiah

> Hello,
> 
> I am following this discussion with great interest and would like to
> participate.
> 
> My primary mail usage is on Windows using Outlook.  I have a Linux machine
> acting as my firewall and has minimal mail capabilities.
> 
> I have installed Python 2.2.1 on my main machine and the next major obstacle
> is figuring out how to create my spam and ham corpus (or should that be
> corpora?).  Can someone on this list provide some hints?
> 
> Thanks,
> 
> Brad Morgan


From guido@python.org  Sat Sep 28 21:55:39 2002
From: guido@python.org (Guido van Rossum)
Date: Sat, 28 Sep 2002 16:55:39 -0400
Subject: [Spambayes] The plural of corpus
In-Reply-To: Your message of "Sat, 28 Sep 2002 13:14:24 PDT."
             <20020928130829.D67D.JCARLSON@uci.edu> 
References: <3D95C4FF.18765.3A1BCB6@localhost>
	<NABBJOOEOFODEALNMJAJEEBEGNAA.B-Morgan@concentric.net>  
	<20020928130829.D67D.JCARLSON@uci.edu> 
Message-ID: <200209282055.g8SKtdC20874@pcp02138704pcs.reston01.va.comcast.net>

> corpus...corpusai...corpai...corpus (I really don't know either),

For future reference, the plural of corpus is corpora, according to my
dictionary. :-)

--Guido van Rossum (home page: http://www.python.org/~guido/)

From crunch@shopip.com  Sat Sep 28 22:37:13 2002
From: crunch@shopip.com (John Draper)
Date: Sat, 28 Sep 2002 14:37:13 -0700
Subject: [Spambayes] Some ideas I have....
Message-ID: <v03110703b9bbc5d85869@[192.168.0.102]>

HI,

I want to start up another discussion about what the direction of the group=
 is heading,  as far as addressing the issues of where spam filter should=
 take place.   IE:  Client side,  Vs Server side.   We've all been through=
 this before,  but I want a little more clarity the group is heading.  Just=
 a quick summary of the issues I'm percieving...

Client side filtering allows for more individual customization.  =
 Individualized Corpuses are needed,  but this then puts the bandwidth=
 burdon on the indivisual.   They STILL have to pull down the spam from the=
 POP server.   But this POP server can also have filtering taking place as=
 well.   Is it my understanding that there is to be a more generalized type=
 of filtering,  applicable to the general group of Email users sharing the=
 same POP server?   Then having a more customizable filtering take place=
 (either through processing scripts on the POP serer,   or on Client level=
 code,  running as a POP Proxy,  or as some other localized stript needed=
 for filtering.

Most of all this theorical statistical discussions are far above my head, =
 and the lack of my old College Staatistical textbooks where are probably=
 outdated anyway,   I decided to take a less focus on these discussions, =
 and want to contribute by helping make our efforts more usable to the spam=
 fighting community.

My experience in programming,  is more to throw something together to get a=
 "proof of permance" system up and running.   My best talants are to=
 "organize" different "Object" modules so they can be used for the largest=
 range of applications as possible.

Let me throw in some really cool concepts I learned,  which i feel sort of=
 "attached" to.

I had an opportunity to use Symantic's Think Class Liberty,  a collection or=
 wuite of powerful C++ object that define computer functionality analygous=
 to the way "People" work.   Let me give you an example...   Lets say we=
 have a dialog box,  with numerous emdbedded and hierarchical "views".   One=
 can think of the overall dialog as an "organization",   not onlike an=
 "Accounting" department,  or "Development" department.  So,  in effect, =
 the Dialog would be a "supervisor" to raign over each of the elements=
 inside.  But each of these elements can also be "supervisors" over things=
 inside them.   NOT to be confused with "Inheritance" hierarchy,  there is a=
 "Command" hierarchy.

The rule is simple...  each of the "Subordinate Bureaucrats" as I like to=
 call them (name comes from Think Class Library),  never has to know=
 anything about it's supervisor.   It is the supervisor's responsibility to=
 manage and "inform" the collection of objects under it's "rule" or "control=
".

A "bureaucrat",  which is derived from "Object" has just two instance=
 variables.   One is a pointer to it's supervisor,  and the other is a=
 pointer to a List or Collection object,   also pointing to other=
 bureaucrats in the organization.

If I subordinate does something,  that other subordinate objects have to=
 "know about",   it sends a "DoCommand" message with an enumerated command=
 number (Known ONLY to the supervisor,   would then delegate control up to=
 the Supervisor,  which would then know what to do,  like notify all the=
 other subordinate objects if appropriate.

I propose that I come up with a list of "Bureaucrats" as I like to call=
 them,  not unlike what might be desired in a "mailroom".   Then define Spam=
 fighting Bureaucrats that can work together on ANY machine,  and allow=
 people to easily put together a host or Object Class Library of not just=
 spam fighting "agents",  but also "mail handling" functions.

I'm making up a list now,  of bureaucrats I would want to see this have.  =
 When I'm done with this list,  I would like to publish it to the group for =
discussion.

In meantime,   start sending me your nominations for "bureaucrats",  and I=
 can start a list of them.   Start with what you've done alreay (in the way=
 of Object classes that can be turned into bureacrats),  define their=
 relationships,  then group them into "agencies" or collections of=
 bureaucrats needed for whatever function appropriate.

I'll throw out some "teasers".   We need a "Configurator" bureaucrat,  that=
 all it does is manage the server or client's "comfiguration".   Some people=
 find configuration files convenient to manage their configurations,  others=
 profer GUI's of local or web based types.   The Configurator would store=
 it's objects in some persistant manner,  plus be able to read in a=
 "configuration" file and setup that way.

I'm betting this is already being done,  but with different names.   So I'm=
 proposing we collect these objects not unlike what a "human" organization=
 would have to be,  to manage someone's mail.   Python already has these... =
 "Message",  and a few others.   We just make "Message" have an additional=
 parent of a Bureaucrat.

A "Corpus manager" would manage a "corpus" and have all the smarts necessary=
 to do that.

A "Tokenizer" would manage and control all the "Token" objects.

Another one would be Josiah Carlson's "PASP" which would derive from a=
 "POP3" object (of course also deriving from Bureaucrat).

Anyway,  I ask for your comments on this approach to defining the "objects"=
 needed to make all this useful.

John






From tim.one@comcast.net  Sat Sep 28 22:51:51 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 28 Sep 2002 17:51:51 -0400
Subject: [Spambayes] realization of mistakes...
In-Reply-To: <20020928102746.D67A.JCARLSON@uci.edu>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEGEBHAB.tim.one@comcast.net>

[Josiah Carlson]
> I've finally had a chance to go through the current spambayes code in
> cvs.  I notice that the project looks to have abandoned Paul Graham's
> ideas entirely, as I could not find anything that looked remotely like
> what was offered in his paper.

Read the first 20 lines of classifier.py; there was a "death match" between
the schemes just this week, and the current scheme won.

> I suppose this is not necessarily a good thing or a bad thing,

For this project, it's necessarily a good thing.  Read TESTING.txt:  the
things we do here are driven by the results of solid statistical testing.
Things that don't win are ruthlessly purged, and we have high confidence
that the things that survive are genuine improvements (which doesn't mean
they always are -- prior decisions are revisted endlessly too, for that
reason).

> but through recent developments in pasp (in the last 24 hours), I've
> noticed that it is good to have multiple classifiers to compare to one
> another.   Considering the experimental status of statistical email
> categorization, I believe that the more information one can have aout
> the performance of varying algorithms, the better.  But that could be
> just me.

Nope, that's a solid idea.  A difficulty is that solid testing takes time,
there are hundreds of things that need testing, and at any given time I push
the core algorithms toward what I *guess* has the highest likelihood of
paying off next.  At the moment, that's Gary's "central limit" algorithms
(which aren't mentioned on his web pages, but are discussed in this list's
archives).

> On that note, I've looked through the Gary Robinson implementation that
> is in CVS.  It looks like you have done a complete implementation of
> those algorithms.  I should note that those numbers I gave earlier for
> my Gary Robinson algorithm, was basically a drop-in replacement from
> Paul Graham's (converted for comparison sake):
> P = ((1-p1)*(1-p2)*...*(1-pn))
> Q = (p1*p2*...*pn)
> S = Q / ( P + Q )
>
> to Gary Robinson's:
> P = 1 - ((1-p1)*(1-p2)*...*(1-pn))^(1/n)
> Q = 1 - (p1*p2*...*pn)^(1/n)
> S = .5 +  ((P - Q) / (P + Q)) / 2
>
> I apologize for any confusion as to why the Gary Robinson version I use
> doesn't do so well, it's because I haven't added any of the "Further
> Improvements", which apparently does do alot.

To use Gary's scheme it's best to purge all the artificial biases from
Paul's scheme.  For example, min and max probs of 0.01 and 0.99 must go,
artificially multiplying ham counts by 2 must go, ignoring words until
2*hamcount + spamcount >= 5 must go, and just over the last day we're
getting evidence that it's *also* good in Gary's scheme to remove the
asymmetry in Paul's scheme wrt how many times to count a word that appears
multiple times in a msg (Paul counts such a beast only once when scoring,
but multiple times when training; in effect, it's Yet Another Bias that, in
his full scheme, appears to partly cancel out the "let's ignore words that
don't appear often" bias).

> I've been contemplating borrowing some classification code from
> classifier.py, but am shying away because I like to keep things simple,

??? Gary's scheme is simpler than Paul's -- it doesn't have a pile of
"mystery knobs" to trip over.  The classifier is quite simple; tokenizer.py
is getting hairy, though.

> mostly because I like to understand what is going on with the math I
> use.

This gets stranger and stranger <wink>.

> Of course that makes comparison between algorithms used and fn/fp
> performance somewhat difficult, but I believe that at least some
> useful information can be gained.

This would be really interesting with a best-effort implementation of both
schemes.  If you follow the suggestion at the top to read the comments,
they'll tell you how to get our last implementation of Paul's scheme; its
performance was very good (and much better than Paul's scheme before
fiddling it).

> If anyone is curious, I decided to continue on the thought path that led
> me to alter the bias in Paul Graham's probability function last evening.
>
>           bias
>             |
> (let ((g (* 2 (or (gethash word good) 0)))
>       (b (or (gethash word bad) 0)))
>    (unless (< (+ g b) 5)
>      (max .01
>           (min .99 (float (/ (min 1 (/ b nbad))
>                              (+ (min 1 (/ g ngood))
>                                 (min 1 (/ b nbad)))))))))
>
> Hams: 5145
> Spams: 747
> Test hams: 62
> Test spams: 122
>
> Using the cutoffs [.5, .6, .7, .8, .9, .95] to find ranges of fn/fp.
>
> If you don't feel like reading the ranges, considering the desireable
> as low as possible fn, and zero fp, on my corpus of
> spam/ham/testspam/testham,

As your corpus grows, you're going to discover that Paul's scheme sometimes
gives a score of exactly 1.0 to ham; at that point, the only way to avoid
false positives is never to call anything spam.  Gary's scheme hasn't been
caught doing that yet, and the people on this list have several corpora with
tens of thousands of msgs.

> it seems that adjusting the per-word probability function bias to some
> number in the range of 1-1.5 produces both a minimum number of false
> negatives, and zero false positives with basic Paul Graham.
>
> What is also quite interesting is that just a plain weighted average
> performs (on the most part), somewhere between PaulGraham and
> GaryRobinson.

Note that there's a large literature on how best to combine multiple
schemes.  AdaBoost has been mentioned before, but nobody here has persued it
yet (it would be interesting in at least two ways:  combining multiple
"whole cloth" schemes, and viewing each spamprob value as an independent
mini-classifier in its own right).

> Anyways, I'll stop typing and get around to doing another release with
> what I currently know.
>
>  - Josiah
>
> Bias 2:
> alg   fn range in %   fp range in %
> PG -> [5 - 5]         [0 - 0]
> GR -> [5 - 43]        [0 - 0]
> WA -> [9 - 23]        [0 - 0]
>
> Bias 1.75:
> alg   fn range in %   fp range in %
> PG -> [3 - 3]         [0 - 0]
> GR -> [3 - 43]        [0 - 0]
> WA -> [7 - 20]        [0 - 0]
>
> Bias 1.5:
> alg   fn range in %   fp range in %
> PG -> [2 - 2]         [0 - 0]
> GR -> [2 - 41]        [0 - 0]
> WA -> [4 - 19]        [0 - 0]
>
> Bias 1.25:
> alg   fn range in %   fp range in %
> PG -> [1 - 1]         [0 - 0]
> GR -> [1 - 35]        [0 - 0]
> WA -> [2 - 16]        [0 - 0]
>
> Bias 1:
> alg   fn range in %   fp range in %
> PG -> [1 - 1]         [3 - 3]
> GR -> [1 - 23]        [3 - 0]
> WA -> [2 - 12]        [10 - 0]

For reasons explained in TESTING.txt, results obtained from a single test
set don't carry any weight here, beyond suggesting that it may be worthwhile
to run a rigorous multi-corpora test.  In the universe of all emails, your
test data is a very small sample, and so you can never be sure whether
you're observing improvements in your algorithm, or just observing random
quirks in your data.  You could get much more confidence in your results,
even with this small amount of data, by adapting the cross-validation
testing framework used in mboxtest.py and timcv.py.  That slices-and-dices
your data N ways, training on N/(N-1) of it and predicting the remaining 1/N
of it, N times.  Sometimes after making a change, you'll find that half the
runs win and the other half lose -- in such a case, it's hard to sell that
the change made a real difference in either direction.  But also in such a
case, had you run only one test, you would have had a 1 in 2 chance of
erroneously concluding that the change helped -- or that the change hurt.
We're taking great pains to avoid such traps here.


From tim.one@comcast.net  Sat Sep 28 22:54:47 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 28 Sep 2002 17:54:47 -0400
Subject: [Spambayes] The plural of corpus
In-Reply-To: <200209282055.g8SKtdC20874@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCEEGFBHAB.tim.one@comcast.net>

>> corpus...corpusai...corpai...corpus (I really don't know either),

[Guido]
> For future reference, the plural of corpus is corpora, according to my
> dictionary. :-)

Your dictionary is correct.  Note too that the plural of walrus is walrora,
and the plural of circus is circora, and the singular of flora is flus.
English is wonderfully consistent, unlike some Dutch languages we could name
<wink>.


From mhoffman@lightlink.com  Sat Sep 28 23:02:24 2002
From: mhoffman@lightlink.com (Mark M. Hoffman)
Date: Sat, 28 Sep 2002 15:02:24 -0700
Subject: [Spambayes] More HTML strippage.
In-Reply-To: <LNBBLJKPBEHFEDALKOLCAEPEBGAB.tim.one@comcast.net>;
	from tim.one@comcast.net on Thu, Sep 26, 2002 at 09:28:49PM -0400
References: <LNBBLJKPBEHFEDALKOLCIEPABGAB.tim.one@comcast.net>
	<LNBBLJKPBEHFEDALKOLCAEPEBGAB.tim.one@comcast.net>
Message-ID: <20020928150224.A10925@mercury.solarsys.net>

Hi Tim:

* Tim Peters <tim.one@comcast.net> [2002-09-26 21:28:49 -0400]:
<snip>

> Laugh of the day.  Even an idiot could have identified one of the f-n just
> by looking at who it was addressed to:
> 
> To: <bait@blast.net>
> Cc: <bait@accsoft.com.au>,
>         <bait@cac.net>,
>         <bait@agrc.com>,
>         <bait@acenet.com.au>,
>         <bait@europe.com>,
>         <bait@access.net.au>,
>         <bait@fishhunt.com>,
>         <bait@eclipse.net>,
>         <bait@australis.net.au>,
>         <bait@enterprise.net>,
>         <bait@ix.netcom.com>,
>         <bait@lakemichiganangler.com>,
>         <bait@em.ca>,
>         <bait@goldrush.net>,
>         <bait@addr.com>,
>         <bait@lycosmail.com>,
>         <bait99@yahoo.com>,
>         <bait@charleston.net>,
>         <bait@adnet.aust.com>,
>         <bait@dragon.acadiau.ca>
<snip>

A modest proposal:  tokenize the number of message recipients e.g.
header:num_recip_21 for the above.

Regards,

(from a list lurker who doesn't python)

-- 
Mark M. Hoffman
mhoffman@lightlink.com


From whisper@oz.net  Sat Sep 28 23:12:55 2002
From: whisper@oz.net (David LeBlanc)
Date: Sat, 28 Sep 2002 15:12:55 -0700
Subject: [Spambayes] The plural of corpus
In-Reply-To: <LNBBLJKPBEHFEDALKOLCEEGFBHAB.tim.one@comcast.net>
Message-ID: <GCEDKONBLEFPPADDJCOEGEPPFDAA.whisper@oz.net>

And naturally, the plural of mongoose is mongeese ;)

David LeBlanc
Seattle, WA USA 

> -----Original Message-----
> From: spambayes-bounces@python.org
> [mailto:spambayes-bounces@python.org]On Behalf Of Tim Peters
> Sent: Saturday, September 28, 2002 14:55
> To: Guido van Rossum; spambayes@python.org
> Subject: RE: [Spambayes] The plural of corpus
> 
> 
> >> corpus...corpusai...corpai...corpus (I really don't know either),
> 
> [Guido]
> > For future reference, the plural of corpus is corpora, according to my
> > dictionary. :-)
> 
> Your dictionary is correct.  Note too that the plural of walrus 
> is walrora,
> and the plural of circus is circora, and the singular of flora is flus.
> English is wonderfully consistent, unlike some Dutch languages we 
> could name
> <wink>.
> 
> 
> _______________________________________________
> Spambayes mailing list
> Spambayes@python.org
> http://mail.python.org/mailman-21/listinfo/spambayes

From jcarlson@uci.edu  Sun Sep 29 00:29:15 2002
From: jcarlson@uci.edu (Josiah Carlson)
Date: Sat, 28 Sep 2002 16:29:15 -0700
Subject: [Spambayes] Good evening/morning/afternoon everyone
In-Reply-To: <ot6cpucv3lujj2dd6er9mf08j4ckgp5jes@4ax.com>
References: <20020928002231.CD68.JCARLSON@uci.edu>
	<ot6cpucv3lujj2dd6er9mf08j4ckgp5jes@4ax.com>
Message-ID: <20020928153427.D68E.JCARLSON@uci.edu>

Richie,

> Rather than modifying the Subject line, I add an extra header.  I believe
> all email clients that have any filtering ability will allow you to filter
> on an arbitrary header.  This is also how Hammie, an alternative driver
> for the spambayes code written by Neale Pickett, works (Hammie is designed
> to fit into a Unix mail delivery system).  The header that pop3proxy.py
> adds is called X-Hammie-Disposition, so that it matches Hammie.

The popfile.sourceforge.net project does a similar thing, adding an
X-Text-Classification header (he is also submitting it as an RFC). I
originally had that be the default behavior of PASP, but remembered that
I have (in the past) had email software that doesn't allow arbitrary
header matching. By inserting the Subject, I guarantee that ANY email
software can filter it.

> pop3proxy.py adjusts the message sizes reported by STAT and LIST to
> include the added X-Hammie-Disposition header.  I'm not sure this makes
> any real difference, but there may be an email client out there that
> malloc()s just enough space and then reads the message into it...

I never thought of that.  Looks like I'm going to have to do some
adjustment.  Cursed C.  For those emails that turn out to not be spam,
I am thinking that it wouldn't be a big deal to just put some blank
spaces at the end of the email.  I hope that no one would really notice.

> I like your system of adjusting the USER to allow proxying to multiple
> accounts - pop3proxy.py doesn't do this.

That is actually another cue I got from popfile.  To be truthful, I
started working on PASP because I wanted something that worked like
popfile, but was free.  At the time, popfile was closed source, but has
recently gone open source.

> My email client (Forte Agent) times out after a couple of minutes of
> inactivity.  I guess others do too.  This caused a problem with large
> messages (eg. with attachments) - the proxy would take so long downloading
> the message without sending anything to the client that the client would
> time out.  For this reason, I added a timeout to pop3proxy.py for long
> messages - it downloads as much as it can in 30 seconds, then classifies
> according to the data it's received and starts sending data back to the
> client.

I also hadn't thought about timeouts.  I'm on 1mbit DSL and never
receive emails larger than 200k, and this only occurs twice a week when
I have to update my GF's webcomic (www.blue-comic.com if you are curious). 
The timeout idea is good, especially for anyone with a dialup, slow dsl
connection, or even shared connections with a packetshaper or heavy
traffic.

I am going to look into adding that.

> I'm told that MSN uses a proprietary POP3 authentication protocol - I
> couldn't find much information about it on the web, but I think it's based
> on POP3-AUTH.  I think pop3proxy.py should support it, because it accepts
> any command and assumes that unknown commands are not multiline.  I think
> your code does the same - is that right? - your 'supported' dictionary
> seems to be unused.  Do you know anything about this MSN system, or do you
> have the means to test against it ('cos I don't...)

I have no idea.  I know that Hotmail requires either pay for use pop3
email, or you have to use Outlook [Express].  It wouldn't be too much
work to do some network sniffing and reverse-engineer the protocol, as
I've been looking to add in a hotmail/yahoomail portion in the future.

I don't suppose anyone out there is willing to take YahooPOPS! and do a
code translation into python...

Currently the code doesn't check for supported/unknown commands, but the
below modification adds it...

elif self.lcmd in supported or self.lcmd == 'usent':
    self.client.send(data)
    self.client.send('\r\n')
else:
    self.client.send('-ERR command not supported, goodbye\r\n')
    self.client.close_when_done()
    #and all the other stuff to kill the connection to the server.

> A question: could you explain 'usent'?  I see self.lcmd being set to
> 'usent', but that seems to be the only place it gets used.  Is it trying
> to cope with POP3 servers that print an extra line after the response to
> the USER command?

Ahhh.  It is mostly just a placeholder.  Because I wanted the mail
client to see the pop3 proxy greeting, the pop3 server greeting, and the
pop3 server USER response I had to do a couple things.  I also didn't
want to send extra +OK messages on different lines, in case the client
had problems with such things.

The following is the type of thing you would capture in your client logs.

+OK pop3 proxy ready
USER randomuser:pop3.rs.com
+OK pop3 server at pop3.rs.com +OK password required for randomuser
PASS ******
+OK Contgratulations.
STAT

The second +OK line is what the 'usent' is for.  It doesn't need it
until I add in the lines earlier in this email.

You don't have to deal with this because your pop3proxy KNOWS which
server to connect to before the client connects.  PASP doesn't know
until it gets the USER user:server command.

> Another question: "pop_proxy.callback(self.parent, ...)" seems like an
> unusual way of writing "self.parent.callback(...)" - is this just a
> stylistic difference, or is there a special reason you do it this way?

I started with self.parent.callback(...), but I kept getting errors with
the parent not being able to access self.server and self.client at all. 
pop_proxy.callback(self.parent,...) works, and I haven't bothered to see
if I can change it back and still have it work.  I'm lazy like that.

 - Josiah

From skip@pobox.com  Sun Sep 29 00:30:04 2002
From: skip@pobox.com (Skip Montanaro)
Date: Sat, 28 Sep 2002 18:30:04 -0500
Subject: [Spambayes] dumb question - why not simply the subject?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCIEFLBHAB.tim.one@comcast.net>
References: <15765.44459.455695.618431@12-248-11-90.client.attbi.com>
        <LNBBLJKPBEHFEDALKOLCIEFLBHAB.tim.one@comcast.net>
Message-ID: <15766.15228.469762.255496@12-248-11-90.client.attbi.com>


    Tim> [Skip Montanaro]
    >> If humans can differentiate ham and spam in the blink of an eye with
    >> near 100% accuracy looking at just the subject of a message, why
    >> should we need to feed any other content to a decent spam detector?

    Tim> If it's true that humans can do this, we shouldn't need to.  What
    Tim> evidence do you have for believing the antecedent, though?  I'll
    Tim> bet a dollar that our classifier today would blow humans out of the
    Tim> water on both error rates if such an experiment were to be
    Tim> conducted.

It was (I thought obviously) more of a rhetorical question than anything.  I
have no problem identifying spam vs non-spam in my own corpus.  I don't
doubt that the current timcv would do better than your average human,
especially if you require response times in the millisecond range, but I
suspect with a human and timcv restricted to examining just the subject the
human would win.

Skip

From jcarlson@uci.edu  Sun Sep 29 01:33:12 2002
From: jcarlson@uci.edu (Josiah Carlson)
Date: Sat, 28 Sep 2002 17:33:12 -0700
Subject: [Spambayes] Help for Windows / Outlook Users
In-Reply-To: <NABBJOOEOFODEALNMJAJOEBIGNAA.B-Morgan@concentric.net>
References: <20020928130829.D67D.JCARLSON@uci.edu>
	<NABBJOOEOFODEALNMJAJOEBIGNAA.B-Morgan@concentric.net>
Message-ID: <20020928172855.D694.JCARLSON@uci.edu>

Brad,

I just installed mozilla and imported some mail from outlook express to
check it out.

If you can find your mozilla mail folder (it'll have some .msf files and
others without extensions), the files without extensions are, I believe,
in unix mbox format.  They seem to be in plain-text, using the From:
header as a separator like in mbox files.

Try those,
 - Josiah

> I looked at both Mozilla and Eudora and I couldn't find "export it to unix
> mbox" format (or any other export format).  Can you elaborate at little on
> how to do that?
> 
> Seems that most email programs are real good at getting stuff into their
> proprietary format but none of them are good at getting it out.
> 
> Regards,
> 
> Brad


From jcarlson@uci.edu  Sun Sep 29 01:36:14 2002
From: jcarlson@uci.edu (Josiah Carlson)
Date: Sat, 28 Sep 2002 17:36:14 -0700
Subject: [Spambayes] realization of mistakes...
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEGEBHAB.tim.one@comcast.net>
References: <20020928102746.D67A.JCARLSON@uci.edu>
	<LNBBLJKPBEHFEDALKOLCKEGEBHAB.tim.one@comcast.net>
Message-ID: <20020928153432.D691.JCARLSON@uci.edu>

[Tim Peters]
> Read the first 20 lines of classifier.py; there was a "death match" between
> the schemes just this week, and the current scheme won.

Ahh, a death match.  I'm looking through the different versions of
classifier.py in the archives to look for all the tweaks.

> > I suppose this is not necessarily a good thing or a bad thing,
> 
> For this project, it's necessarily a good thing.  Read TESTING.txt:  the
> things we do here are driven by the results of solid statistical testing.
> Things that don't win are ruthlessly purged, and we have high confidence
> that the things that survive are genuine improvements (which doesn't mean
> they always are -- prior decisions are revisted endlessly too, for that
> reason).

I just read TESTING.txt.  And I agree that one should move on, but
ruthlessly purging old code, which is good for keeping bloat low or
nonexistant, only makes sense if the code is bloated already.  Never be
afraid to comment out the code.

> > but through recent developments in pasp (in the last 24 hours), I've
> > noticed that it is good to have multiple classifiers to compare to one
> > another.   Considering the experimental status of statistical email
> > categorization, I believe that the more information one can have aout
> > the performance of varying algorithms, the better.  But that could be
> > just me.
> 
> Nope, that's a solid idea.  A difficulty is that solid testing takes time,
> there are hundreds of things that need testing, and at any given time I push
> the core algorithms toward what I *guess* has the highest likelihood of
> paying off next.  At the moment, that's Gary's "central limit" algorithms
> (which aren't mentioned on his web pages, but are discussed in this list's
> archives).

That is a legitimate case for chucking old stuff.  It's better to not
have to test all the old stuff that may work well when you have since
moved on...especially when you have too much to test already.

> > On that note, I've looked through the Gary Robinson implementation that
> > is in CVS.  It looks like you have done a complete implementation of
> > those algorithms.  I should note that those numbers I gave earlier for
> > my Gary Robinson algorithm, was basically a drop-in replacement from
> > Paul Graham's (converted for comparison sake):
> > P = ((1-p1)*(1-p2)*...*(1-pn))
> > Q = (p1*p2*...*pn)
> > S = Q / ( P + Q )
> >
> > to Gary Robinson's:
> > P = 1 - ((1-p1)*(1-p2)*...*(1-pn))^(1/n)
> > Q = 1 - (p1*p2*...*pn)^(1/n)
> > S = .5 +  ((P - Q) / (P + Q)) / 2
> >
> > I apologize for any confusion as to why the Gary Robinson version I use
> > doesn't do so well, it's because I haven't added any of the "Further
> > Improvements", which apparently does do alot.
> 
> To use Gary's scheme it's best to purge all the artificial biases from
> Paul's scheme.  For example, min and max probs of 0.01 and 0.99 must go,
> artificially multiplying ham counts by 2 must go, ignoring words until
> 2*hamcount + spamcount >= 5 must go, and just over the last day we're
> getting evidence that it's *also* good in Gary's scheme to remove the
> asymmetry in Paul's scheme wrt how many times to count a word that appears
> multiple times in a msg (Paul counts such a beast only once when scoring,
> but multiple times when training; in effect, it's Yet Another Bias that, in
> his full scheme, appears to partly cancel out the "let's ignore words that
> don't appear often" bias).

I'll have a go at the removal of the biases.

> > I've been contemplating borrowing some classification code from
> > classifier.py, but am shying away because I like to keep things simple,
> 
> ??? Gary's scheme is simpler than Paul's -- it doesn't have a pile of
> "mystery knobs" to trip over.  The classifier is quite simple; tokenizer.py
> is getting hairy, though.

I've been having very good results with just translating all non
alpha-numerics into whitespace, uppercase into lowercase, then using
split().  It has the benefit of (I think) being all C for the translate
and split calls.

I'm also currently only tokenizing words that are 3 <= wlen <= 10, that
way I don't need to strip the subject header of 'SUSPECTEDSPAM', which I
use as a filter-able keyword.

> > mostly because I like to understand what is going on with the math I
> > use.
> 
> This gets stranger and stranger <wink>.

Yes, the desire for being able to understand the software one is working
on is quite strange.  I rather like to understand ALL of it, and not
just pieces of it...but only my code.  If I can understand someone
else's at all, I'm happy.

> > Of course that makes comparison between algorithms used and fn/fp
> > performance somewhat difficult, but I believe that at least some
> > useful information can be gained.
> 
> This would be really interesting with a best-effort implementation of both
> schemes.  If you follow the suggestion at the top to read the comments,
> they'll tell you how to get our last implementation of Paul's scheme; its
> performance was very good (and much better than Paul's scheme before
> fiddling it).

I'm checking it out.


> Note that there's a large literature on how best to combine multiple
> schemes.  AdaBoost has been mentioned before, but nobody here has persued it
> yet (it would be interesting in at least two ways:  combining multiple
> "whole cloth" schemes, and viewing each spamprob value as an independent
> mini-classifier in its own right).

If I have the time, I'll look into it.

> For reasons explained in TESTING.txt, results obtained from a single test
> set don't carry any weight here, beyond suggesting that it may be worthwhile
> to run a rigorous multi-corpora test.  In the universe of all emails, your
> test data is a very small sample, and so you can never be sure whether
> you're observing improvements in your algorithm, or just observing random
> quirks in your data.  You could get much more confidence in your results,
> even with this small amount of data, by adapting the cross-validation
> testing framework used in mboxtest.py and timcv.py.  That slices-and-dices
> your data N ways, training on N/(N-1) of it and predicting the remaining 1/N
> of it, N times.  Sometimes after making a change, you'll find that half the
> runs win and the other half lose -- in such a case, it's hard to sell that
> the change made a real difference in either direction.  But also in such a
> case, had you run only one test, you would have had a 1 in 2 chance of
> erroneously concluding that the change helped -- or that the change hurt.
> We're taking great pains to avoid such traps here.

You make a good point (which is also in the TESTING.txt file *grin*). 
I'll remember that when I report my 'results'.

I really should be getting on with other errands...I've spent at lest 5
hours on this today *gasp*.

 - Josiah

From eppstein@ics.uci.edu  Sun Sep 29 02:18:17 2002
From: eppstein@ics.uci.edu (David Eppstein)
Date: Sat, 28 Sep 2002 18:18:17 -0700
Subject: [Spambayes] Re: Some ideas I have....
References: <v03110703b9bbc5d85869@[192.168.0.102]>
Message-ID: <eppstein-1894D1.18181728092002@main.gmane.org>

In article <v03110703b9bbc5d85869@[192.168.0.102]>,
 John Draper <crunch@shopip.com> wrote:

> I want to start up another discussion about what the direction of the group 
> is heading,  as far as addressing the issues of where spam filter should take 
> place.   IE:  Client side,  Vs Server side.

The place that seems most convenient to me is to .forward through a 
program similar to procmail.  That way I don't have to persuade the mail 
client or smtp server or IMAP server developers or my sysadmins to 
include it...

-- 
David Eppstein       UC Irvine Dept. of Information & Computer Science
eppstein@ics.uci.edu http://www.ics.uci.edu/~eppstein/



From tim.one@comcast.net  Sun Sep 29 02:32:11 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 28 Sep 2002 21:32:11 -0400
Subject: [Spambayes] More HTML strippage.
In-Reply-To: <20020928150224.A10925@mercury.solarsys.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEGLBHAB.tim.one@comcast.net>

[Mark M. Hoffman]
> A modest proposal:  tokenize the number of message recipients e.g.
> header:num_recip_21 for the above.

Yes, we don't do anything with To, Cc, etc, now, except count the number of
times (>= 1) each header line appears.  There are lots of "little things" of
that nature that could help; Greg Ward suggested others based on stuff
SpamAssassin gets value from; we're short on people who test things, though,
and even shorter on people who will code things and qualify them by running
their own tests first (e.g., Neil Schemenauer ran with one of Greg's
suggested and showed that its core idea was a winner, so he's in the Spam
Busters Hall of Extreme Fame).

> Regards,
>
> (from a list lurker who doesn't python)

    http://www.python.org

It's free, and even idiots are born knowing how to program fluently in
Python <wink>.


From mhoffman@lightlink.com  Sun Sep 29 03:25:41 2002
From: mhoffman@lightlink.com (Mark M. Hoffman)
Date: Sat, 28 Sep 2002 19:25:41 -0700
Subject: [Spambayes] To think like a spammer...
Message-ID: <20020928192541.A14014@mercury.solarsys.net>

... that is, if spammers could think.

The spambayes scheme (and others like it that I've seen) can be defeated
easily, with something like this...

<spam>

THIS  IS  A   F A N T A S T I C   O P P O R T U N I T Y ! !

W A N T  TO  MAKE  A  TON  OF  M O N E Y   ? ? ?

THIS   W E B S I T E   HAS  THE  A N S W E R   TO  ALL  YOUR
F I N A N C I N G   P R O B L E M S ! ! !

IT  COULD  NOT  BE   E A S I E R ! ! !   C L I C K  HERE NOW ! ! !

http://blah.blah.com/blah/blah.html

</spam>

Even if you back up and allow single character tokens, you're only going
to recognize a handful of those.  A spammer could stuff the end of the
message with ham words to overcome the effect of the single char. tokens.
Do these space-words have to be collapsed to defeat the effect?

Actually, protection against stuffing the end of a spam w/ ham words is
an angle we have to be careful about anyway.

Regards,

-- 
Mark M. Hoffman
mhoffman@lightlink.com


From anthony@interlink.com.au  Sun Sep 29 03:25:37 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Sun, 29 Sep 2002 12:25:37 +1000
Subject: [Spambayes] realization of mistakes... 
In-Reply-To: <20020928102746.D67A.JCARLSON@uci.edu> 
Message-ID: <200209290225.g8T2Pbj21032@localhost.localdomain>


>>> Josiah Carlson wrote
> I suppose this is not necessarily a good thing or a bad thing, but
> through recent developments in pasp (in the last 24 hours), I've noticed
> that it is good to have multiple classifiers to compare to one another. 
> Considering the experimental status of statistical email categorization,
> I believe that the more information one can have about the performance
> of varying algorithms, the better.  But that could be just me.

The problem is that there is _so_ much that needs testing. You just 
have to cull the weaklings from the herd of algorithms. FWIW, as part
of this, I fed part or all of my 30,000 message corpus into the cross
verification system something like 80 times all up - it was about 2 
work days worth of processing time. We don't have the resources to be
able to test everything every time (nor do we have the testers with the
time and the large and useful corpus)

Anthony
-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.


From anthony@interlink.com.au  Sun Sep 29 03:29:21 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Sun, 29 Sep 2002 12:29:21 +1000
Subject: [Spambayes] To think like a spammer... 
In-Reply-To: <20020928192541.A14014@mercury.solarsys.net> 
Message-ID: <200209290229.g8T2TL721093@localhost.localdomain>


>>> "Mark M. Hoffman" wrote
> ... that is, if spammers could think.
> 
> The spambayes scheme (and others like it that I've seen) can be defeated
> easily, with something like this...
> 
> <spam>
> 
> THIS  IS  A   F A N T A S T I C   O P P O R T U N I T Y ! !

In practice, they only put the "obvious" keywords in the gappy text.
The nice thing about SB is that it does really quite well even without
the really obvious clues. Unless they start putting the whole page in
this sort of text, in which case it's going to be near-unreadable,
anyway.

A more ugly one I'm seeing (which is a persistent source of a few fn)
is HTML email which is a huge slab of javascript, and the message 
text encoded inside the message. 


-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.


From guido@python.org  Sun Sep 29 03:33:13 2002
From: guido@python.org (Guido van Rossum)
Date: Sat, 28 Sep 2002 22:33:13 -0400
Subject: [Spambayes] To think like a spammer...
In-Reply-To: Your message of "Sat, 28 Sep 2002 19:25:41 PDT."
             <20020928192541.A14014@mercury.solarsys.net> 
References: <20020928192541.A14014@mercury.solarsys.net> 
Message-ID: <200209290233.g8T2XDZ24279@pcp02138704pcs.reston01.va.comcast.net>

> The spambayes scheme (and others like it that I've seen) can be defeated
> easily, with something like this...
> 
> THIS  IS  A   F A N T A S T I C   O P P O R T U N I T Y ! !

It's an arms race.  I expect that the classification scheme can be
kept relatively constant (the math doesn't change) but the tokenizing
(better called feature extraction) scheme can and should be adapted
occasionally, to deal with new ways of hiding spam.  This particular
style can easily be recognized[*] *if* it becomes popular among
spammers; for anything you can come up with there's a tokenizer that
recognizes it.

But spammers will only start worrying if their return rates go down,
and that will only happen once almost everybody is using anti-spam
technology.  We've got a long way to go before that's the case.  So
let's not be stymied by worries about what the spammers can do.

[*] I wouldn't even bother trying to recover the words FANTASTIC
OPPORTUNITY.  This style is so completely unseen in ham that simply
looking for many consecutive one-letter words and inserting a token
representing such a presence would most likely be enough.

--Guido van Rossum (home page: http://www.python.org/~guido/)

From jcarlson@uci.edu  Sun Sep 29 03:54:18 2002
From: jcarlson@uci.edu (Josiah Carlson)
Date: Sat, 28 Sep 2002 19:54:18 -0700
Subject: [Spambayes] To think like a spammer...
In-Reply-To: <20020928192541.A14014@mercury.solarsys.net>
References: <20020928192541.A14014@mercury.solarsys.net>
Message-ID: <20020928193324.D69B.JCARLSON@uci.edu>

> Even if you back up and allow single character tokens, you're only going
> to recognize a handful of those.  A spammer could stuff the end of the
> message with ham words to overcome the effect of the single char. tokens.
> Do these space-words have to be collapsed to defeat the effect?
> 
> Actually, protection against stuffing the end of a spam w/ ham words is
> an angle we have to be careful about anyway.

If there were enough spammers that used such things, they would
definitely find their messages in the spam corpus.  If one then started
allowing 1-character tokens, through spambayes, because most words are
not of a single letter, those letters that happen to not be used as
initials in sigs or otherwise, would likely be flagged as being more
spammy than hammy.

The problem with removing spaces is that the software would need to have
a listing of valid words, and one would need to check all the possible
concatenation of tokens that could _potentially_ create a word.  You
could look at it in terms of a bitmap.

Given a list of bits, each representing a different token, you can use 1
to represent a continuation of the previous word (concatenated) and a 0
to represent the beginning of a new word. Current non-concatenation
gives us all 0's.  You can create a secondary bitmap to show which of
the tokens/bits can be a continuation.

If you take a random approach to it, you're looking at 2**(number of
single-character entries) for the number of ways the message can be
concatenated.  That's REALLY bad.

However, you could do a DFS on a tree of valid words, which would
produce unbelievably fast results in C, and even all right results using
python dictionaries (something that I feel like coding up this evening,
it'll be neat), but you would still have problems with purposeful
misspellings.

Good point though,
 - Josiah

From tim.one@comcast.net  Sun Sep 29 04:13:16 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sat, 28 Sep 2002 23:13:16 -0400
Subject: [Spambayes] dumb question - why not simply the subject?
In-Reply-To: <15766.15228.469762.255496@12-248-11-90.client.attbi.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEHEBHAB.tim.one@comcast.net>

[Skip Montanaro]
> It was (I thought obviously) more of a rhetorical question than
> anything.  I have no problem identifying spam vs non-spam in my own
> corpus.

I believe you overestimate your own ability to do this based on subje=
ct line
alone, due to selective memory of outrageously spammish subject lines=
.
That's not a jab at you, it's how humans work.  For example, *just* g=
rep for
the Subject lines in the compilation of false negatives you put on th=
e web.
If you can nail 100% of those not knowing that you already believe th=
ey're
spam, I won't believe you <wink>.  What are you going to do with a su=
bject
line that says just "HELP", or "hi"?  I get plenty of those in both m=
y
real-life ham and spam.

You'll also find that you left at least one

Subject: New subscription request to list CEDU from edshipp@ameritech=
.net

message in your false negatives, which pretty much kills the notion t=
hat
you're infallible <wink>.

> I don't doubt that the current timcv would do better than
> your average human, especially if you require response times in the
> millisecond range,

I was kidding about that part.  I'm certain that the error rates it d=
isplays
on my corpus are better than I could do by hand, though.

> but I suspect with a human and timcv restricted
> to examining just the subject the human would win.

That's a different scenario entirely.  To mine subject lines, and onl=
y
subject lines, likely requires vast semantic knowledge to do really w=
ell.
Then you're in the realm of AI, and that's got a track record that sp=
eaks
for itself <wink>.

As an experiment, I changed tokenize() to ignore the body completely,=
 and
changed tokenize_headers() to do only this:

        x =3D msg.get('subject', '')
        if x =3D=3D '':
            yield 'No subject'
            return
        lastt =3D ''
        for w in subject_word_re.findall(x):
            for t in tokenize_word(w):
                yield 'subject:' + t
                yield 'subject:' + t.lower()
                if lastt:
                    yield 'subject:' + lastt + '<->' + t
                lastt =3D t
        for w in x.split():
            yield 'subject:' + w
            yield 'subject:' + w.lower()
        for w in junk_re.findall(x):
            yield 'subject:' + w
        return

where

    junk_re =3D re.compile(r"\W+")

IOW, it goes hog wild on the Subject line alone, doing unigrams and b=
igrams,
case-folding and not case-folding, splitting on whitespace and search=
ing for
alphanumeric runs, and even tokenizing runs of pure punctuation.

Of course this doesn't do as well as our regular tokenization, but it=
 does
suggest there's a world of clues in subject lines that we could be ge=
tting
more value from.  Cool:  the first false positive it turned up is act=
ually
spam that nothing else has caught!

*********************************************************************=
****
Data/Ham/Set1/156846.txt
prob =3D 0.637311663424
prob('subject:post') =3D 0.061641
prob('subject:Link') =3D 0.0652174
prob('subject:.') =3D 0.398331
prob('subject:inside!') =3D 0.844828
prob('subject:fresh') =3D 0.909196
prob('subject:Fresh') =3D 0.909218
prob('subject:Inside') =3D 0.929104
prob('subject:! ') =3D 0.950089
prob('subject:free') =3D 0.983887
prob('subject:Free') =3D 0.984495

Path:
news.baymountain.net!uunet!ash.uu.net!priapus.visi.com!news-out.visi.=
com!her
mes.visi.com!newsfeeds-atl2!news.webusenet.com!pc01.webusenet.com!e35=
00-atl1
=2Eusenetserver.com.POSTED!ijehs.ac.th!YWAVNU
=46rom: Wayne <YWAVNU@ijehs.ac.th>
Message-ID: <4FD1F099.2544C6BA@BASIC.BS.WEBUSENET.COM>
Newsgroups: comp.lang.prograph,comp.lang.prolog,comp.lang.python
Subject: Free P.0.R.N.0! Link Inside! Fresh post!        5vIiiTQU RtP=
5cQY0
Lines: 96
X-Complaints-To: abuse@usenetserver.com
X-Abuse-Info: Please be sure to forward a copy of ALL headers
X-Abuse-Info: Otherwise we will be unable to process your complaint
properly.
NNTP-Posting-Date: Mon, 05 Aug 2002 23:34:02 EDT
Organization: WEBUSENET.com
Date: Mon, 6 Aug 2002 03:30:28 GMT
Xref: news.baymountain.net comp.lang.prograph:12849 comp.lang.prolog:=
26979
        comp.lang.python:176063
To: python-list@python.org
Sender: python-list-admin@python.org
Errors-To: python-list-admin@python.org
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.0.13 (101270)
Precedence: bulk
List-Help: <mailto:python-list-request@python.org?subject=3Dhelp>
List-Post: <mailto:python-list@python.org>
List-Subscribe: <http://mail.python.org/mailman/listinfo/python-list>=
,
        <mailto:python-list-request@python.org?subject=3Dsubscribe>
List-Id: General discussion list for the Python programming language
        <python-list.python.org>
List-Unsubscribe: <http://mail.python.org/mailman/listinfo/python-lis=
t>,
        <mailto:python-list-request@python.org?subject=3Dunsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>

http://www.virtualwebmedia.com

F-R-E-E p o r  n! NO S+P+A+M!

obusj

D2sMJ0EOdQMT

Try arriving the shore's upper coffee and Eve will explain you!

yzehpu@virtualwebmedia.com

nomjyd@virtualwebmedia.com

ogeddusu@virtualwebmedia.com

uN9xyVlj

Are you shallow, I mean, pouring in front of humble bushs?

Sometimes, tags expect near upper houses, unless they're blank.

Hardly any empty ointments look Fred, and they weekly talk Brahimi to=
o.
Hey, it solves a disk too shallow beside her wet office.  I am
surprisingly strong, so I learn you.  It will grasp locally if
Jimmy's potter isn't polite.  Otherwise the kettle in Cristof's
card might recollect some old desks.  Will you scold among the
lane, if Julieta furiously laughs the pen?

He'll be liking within urban Norm until his onion climbs lazily.  As
globally as Ramzi attacks, you can hate the puddle much more
gently.  Daoud!  You'll pour stickers.  Sometimes, I'll nibble the
yogi.  I was dining to sow you some of my pathetic films.

Until Ayub cleans the doses stupidly, Jimmy won't dye any proud
stations.  If the strange pickles can wander undoubtably, the
worthwhile poultice may promise more showers.  Some weak smart
dusts tamely fear as the filthy sauces pull.  They are irrigating
within the spring now, won't seek powders later.  Try not to
excuse a envelope!  Many angry tapes inside the dull monolith were
behaving between the younger bedroom.

Hardly any rural bizarre jars will totally attempt the bowls.
When will we jump after Susan joins the fresh shore's spoon?  She'd r=
ather
taste finitely than receive with Ziad's new lemon.

Other abysmal sticky clouds will tease quickly with elbows.  Better
measure porters now or George will deeply love them on you.

We cook them, then we dully waste Saad and Charlene's lazy book.

For Usha the cat's sad, alongside me it's open, whereas in back of yo=
u it's
departing short.  The coffees, buckets, and twigs are all raw and
sharp.  Do not improve nearly while you're recommending before a
cold teacher.  Try covering the navel's tired draper and Oscar will
move you!  Saeed, have a dry shoe.  You won't open it.

Lately Beryl will comb the lentil, and if Jbilou incredibly burns it =
too,
the
hat will converse behind the dirty night.  If you will live Rachel's
structure within figs, it will biweekly call the floor.  Gawd,
Wally never judges until Zakariya creeps the think ache usably.
What Penny's kind can walks, Harvey moulds beside poor, pretty
halls.

Are you dark, I mean, kicking beside weird hens?  I was dreaming
pumpkins to deep Amber, who's playing through the dryer's stable.
Mahammed explains, then Hamza loudly fills a ugly cap near Karim's
light.  You won't answer me helping before your wide river.  Let's
lift outside the handsome rivers, but don't reject the cosmetic
pitchers.  Lots of unique painters are hot and other brave jugs are
lean, but will Yosri care that?  Her dog was humble, blunt, and
changes under the cellar.  We smell once, shout grudgingly, then
irritate inside the gardner behind the desert.  She will annually
order outer and arrives our durable, cheap bushs with a ceiling.

A lot of frogs rigidly believe the difficult doorway.  She wants to
mould thin smogs above Shah's fog.  Who talks weakly, when Gregory
joins the full pin without the kiosk?  Every rich farmer or plain, an=
d
she'll
inadvertently smell everybody.  It's very clean today, I'll waste
unbelievably or Allahdad will attack the units.  Where did Agha
laugh within all the barbers?  We can't fear buttons unless Timothy w=
ill
virtually expect afterwards.  No sauces will be active sweet
eggs.  David, on games young and healthy, teases within it, behaving
smartly.  When did Ali kill the printer around the clever weaver?
I like quiet plates, do you learn them?  It might neatly dream
against Stephanie when the heavy boats open to the lost camp.
He will kick stale wrinkles beside the good glad ladder, whilst
Guido stupidly creeps them too.  Just receiving without a goldsmith
in back of the highway is too inner for Elisabeth to converse it.  Th=
e
distant walnut rarely grasps Byron, it sows Lisette instead.
She may excuse the easy cup and reject it within its square.  While
oranges familiarly judge codes, the cases often burn at the lower
tailors.
*********************************************************************=
****

The next false positive was

    Subject: Viewing JPG's

and certainly was not a spam.  Ditto

    Subject: If that interests you...

following, and then

    Subject: Oil on troubled waters ...

and then

    Subject: Find where you left off...

The first false negatives were

    Subject: howdy ukqip
    Subject: =B0=A2=C4=B7=C8=F0=CC=D8(=D1=C7=D6=DE)=CD=F8=C2=E7=D3=
=D0=CF=DE=B9=AB=CB=BE
    Subject: Re: hi!
    Subject: a link with minimal effort.
        [must be about Solaris shared libraries <wink>]
    Subject: Re: info
    Subject: Interview Charles Payne
    Subject: re: order
    Subject: Possible Cause of Brain Cancer
    Subject: re
    Subject: cool
    Subject: It is beneficial to your library & its patrons to have
             the book (Please suggest)
    Subject: Re: ...something to think about?
    Subject: Python Video Promotion!
        [LOL!]
    Subject: RE: hey!
    Subject: you too?
    Subject: http://untroubled.org/relay-ctrl/
    Subject: Surprise ?
        [variations of "surprise" have high spamprob, but this was
         saved by prob('subject: ?') =3D 0.0291312
         for whatever reason, newbies on c.l.py often leave a space
         in front of their question marks, and the tokenize-runs-
         of-punctuation gimmick found that]
    Subject: Help Needed
    Subject: Waste Review
    Subject: Re: Email Confirmation
    Subject: Re: about this email...
    Subject: Re: new information
    Subject: Networking Position
    Subject: RE:
    Subject: What do you get when you cross a sorority girl with an a=
pe?
    Subject: re: entry conf.

Enough already -- it did have one triumph, and plenty of those false
negatives would have been false negatives for me "by hand" too.  Over=
all:

-> <stat> tested 2000 hams & 1400 spams against 18000 hams & 12600 sp=
ams
   [ditto 19 times]

false positive percentages
    0.000  0.300  lost  +(was 0)
    0.000  0.200  lost  +(was 0)
    0.000  0.350  lost  +(was 0)
    0.000  0.400  lost  +(was 0)
    0.050  0.500  lost  +900.00%
    0.000  0.400  lost  +(was 0)
    0.000  0.550  lost  +(was 0)
    0.000  0.400  lost  +(was 0)
    0.000  0.300  lost  +(was 0)
    0.050  0.450  lost  +800.00%

won   0 times
tied  0 times
lost 10 times

total unique fp went from 2 to 77 lost  +3750.00%
mean fp % went from 0.01 to 0.385 lost  +3750.00%

false negative percentages
    0.071  4.000  lost  +5533.80%
    0.071  2.429  lost  +3321.13%
    0.000  2.643  lost  +(was 0)
    0.071  3.571  lost  +4929.58%
    0.143  2.786  lost  +1848.25%
    0.214  3.357  lost  +1468.69%
    0.143  3.071  lost  +2047.55%
    0.143  3.071  lost  +2047.55%
    0.214  2.929  lost  +1268.69%
    0.000  2.429  lost  +(was 0)

won   0 times
tied  0 times
lost 10 times

total unique fn went from 15 to 424 lost  +2726.67%
mean fn % went from 0.107142857143 to 3.02857142857 lost  +2726.67%

ham mean                     ham sdev
  28.00   14.11  -49.61%        5.80    9.86  +70.00%
  27.93   13.79  -50.63%        5.62    9.60  +70.82%
  27.91   14.02  -49.77%        5.76   10.08  +75.00%
  28.02   13.79  -50.79%        5.67   10.28  +81.31%
  27.82   14.02  -49.60%        5.85   10.38  +77.44%
  27.88   13.84  -50.36%        5.53   10.68  +93.13%
  28.05   14.16  -49.52%        5.69   10.35  +81.90%
  28.00   14.07  -49.75%        5.54   10.33  +86.46%
  28.14   14.27  -49.29%        5.61   10.17  +81.28%
  28.16   13.95  -50.46%        5.93   10.61  +78.92%

ham mean and sdev for all runs
  27.99   14.00  -49.98%        5.70   10.24  +79.65%

spam mean                    spam sdev
  85.00   84.96   -0.05%        6.92   12.31  +77.89%
  84.80   85.33   +0.63%        6.66   11.52  +72.97%
  84.48   85.83   +1.60%        6.57   11.41  +73.67%
  85.01   85.25   +0.28%        6.65   12.08  +81.65%
  85.01   85.83   +0.96%        6.49   11.06  +70.42%
  84.89   85.60   +0.84%        6.82   11.71  +71.70%
  84.61   85.21   +0.71%        6.68   11.83  +77.10%
  85.00   85.85   +1.00%        6.52   11.46  +75.77%
  85.02   85.61   +0.69%        6.78   11.09  +63.57%
  84.96   85.80   +0.99%        6.47   11.78  +82.07%

spam mean and sdev for all runs
  84.88   85.53   +0.77%        6.66   11.64  +74.77%

ham/spam mean difference: 56.89 71.53 +14.64

Histogram analysis suggested:

-> best cutoff for all runs: 0.575
->     with weighted total 10*65 fp + 486 fn =3D 1136
->     fp rate 0.325%  fn rate 3.47%

If someone wants to run with this, the list of best discriminators ma=
y be
helpful (although under Gary's scheme this is more like a list of
most-frequent discriminators):

    best discriminators:
        'subject:are' 493 0.846732
        'subject:mortgage' 500 0.999499
        'subject:can' 507 0.606534
        'subject:now' 508 0.927579
        'subject:PEP' 512 0.000488652
        'subject:(was' 515 0.000488652
        'subject:newbie' 522 0.000480307
        'subject: $' 523 0.992175
        'subject:no' 528 0.751766
        'subject:You' 532 0.981916
        'subject:] ' 535 0.196305
        'subject:,' 541 0.894722
        'subject:pep' 546 0.000457829
        'subject:help' 556 0.288172
        'subject:problem' 560 0.0836141
        'subject: & ' 570 0.75976
        'subject:! ' 597 0.947754
        'subject:FREE' 598 0.999584
        'subject:this' 606 0.695102
        'subject:&' 637 0.731907
        'subject:How' 671 0.252991
        'subject:get' 684 0.876916
        'subject:new' 770 0.672979
        'subject:question' 810 0.00423811
        'subject:Your' 883 0.991676
        'subject:from' 899 0.399686
        'subject:how' 962 0.236597
        'subject:was' 969 0.0148958
        'subject:.' 1036 0.40169
        'subject: - ' 1126 0.615247
        'subject:/' 1180 0.357549
        'subject:free' 1204 0.982183
        'subject:is' 1216 0.349481
        'subject:RE:' 1326 0.133728
        'subject:)' 1364 0.258211
        'subject: (' 1480 0.139366
        'subject:with' 1496 0.331923
        'subject:you' 1737 0.941033
        'subject:, ' 1955 0.640872
        'subject:the' 1984 0.635883
        'subject:!' 2072 0.87807
        'subject:your' 2260 0.984736
        'subject:in' 2362 0.310965
        'subject:and' 2414 0.390133
        'subject:Python' 4321 0.00115551
        'subject:?' 4796 0.19175
        'subject:python' 5482 0.000912976
        'subject:Re:' 15254 0.0194725
        'subject:re:' 16662 0.0399341
        'subject:: ' 17253 0.0965151

Note that variations of "Re:" really gave some spam a boost.



From whisper@oz.net  Sun Sep 29 04:22:42 2002
From: whisper@oz.net (David LeBlanc)
Date: Sat, 28 Sep 2002 20:22:42 -0700
Subject: [Spambayes] dumb question - why not simply the subject?
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEHEBHAB.tim.one@comcast.net>
Message-ID: <GCEDKONBLEFPPADDJCOEEEAPFEAA.whisper@oz.net>

I've gotten spams with subjects like "About your recent order" - not easy to
discern that's spam.

Also, sender's email address isn't reliable either. Those people change
their fake sender's address faster then a politician during election season
changes his campaign platform ;)

David LeBlanc
Seattle, WA USA

> -----Original Message-----
> From: spambayes-bounces+whisper=oz.net@python.org
> [mailto:spambayes-bounces+whisper=oz.net@python.org]On Behalf Of Tim
> Peters
> Sent: Saturday, September 28, 2002 20:13
> To: skip@pobox.com
> Cc: spambayes@python.org
> Subject: RE: [Spambayes] dumb question - why not simply the subject?
>
>
> [Skip Montanaro]
> > It was (I thought obviously) more of a rhetorical question than
> > anything.  I have no problem identifying spam vs non-spam in my own
> > corpus.
>
> I believe you overestimate your own ability to do this based on
> subject line
> alone, due to selective memory of outrageously spammish subject lines.
> That's not a jab at you, it's how humans work.  For example,
> *just* grep for
> the Subject lines in the compilation of false negatives you put
> on the web.
> If you can nail 100% of those not knowing that you already believe they're
> spam, I won't believe you <wink>.  What are you going to do with a subject
> line that says just "HELP", or "hi"?  I get plenty of those in both my
> real-life ham and spam.
>
> You'll also find that you left at least one
>
> Subject: New subscription request to list CEDU from edshipp@ameritech.net
>
> message in your false negatives, which pretty much kills the notion that
> you're infallible <wink>.
>
> > I don't doubt that the current timcv would do better than
> > your average human, especially if you require response times in the
> > millisecond range,
>
> I was kidding about that part.  I'm certain that the error rates
> it displays
> on my corpus are better than I could do by hand, though.
>
> > but I suspect with a human and timcv restricted
> > to examining just the subject the human would win.
>
> That's a different scenario entirely.  To mine subject lines, and only
> subject lines, likely requires vast semantic knowledge to do really well.
> Then you're in the realm of AI, and that's got a track record that speaks
> for itself <wink>.
>
> As an experiment, I changed tokenize() to ignore the body completely, and
> changed tokenize_headers() to do only this:
>
>         x = msg.get('subject', '')
>         if x == '':
>             yield 'No subject'
>             return
>         lastt = ''
>         for w in subject_word_re.findall(x):
>             for t in tokenize_word(w):
>                 yield 'subject:' + t
>                 yield 'subject:' + t.lower()
>                 if lastt:
>                     yield 'subject:' + lastt + '<->' + t
>                 lastt = t
>         for w in x.split():
>             yield 'subject:' + w
>             yield 'subject:' + w.lower()
>         for w in junk_re.findall(x):
>             yield 'subject:' + w
>         return
>
> where
>
>     junk_re = re.compile(r"\W+")
>
> IOW, it goes hog wild on the Subject line alone, doing unigrams
> and bigrams,
> case-folding and not case-folding, splitting on whitespace and
> searching for
> alphanumeric runs, and even tokenizing runs of pure punctuation.
>
> Of course this doesn't do as well as our regular tokenization, but it does
> suggest there's a world of clues in subject lines that we could be getting
> more value from.  Cool:  the first false positive it turned up is actually
> spam that nothing else has caught!
>
> *************************************************************************
> Data/Ham/Set1/156846.txt
> prob = 0.637311663424
> prob('subject:post') = 0.061641
> prob('subject:Link') = 0.0652174
> prob('subject:.') = 0.398331
> prob('subject:inside!') = 0.844828
> prob('subject:fresh') = 0.909196
> prob('subject:Fresh') = 0.909218
> prob('subject:Inside') = 0.929104
> prob('subject:! ') = 0.950089
> prob('subject:free') = 0.983887
> prob('subject:Free') = 0.984495
>
> Path:
> news.baymountain.net!uunet!ash.uu.net!priapus.visi.com!news-out.vi
> si.com!her
> mes.visi.com!newsfeeds-atl2!news.webusenet.com!pc01.webusenet.com!
> e3500-atl1
> .usenetserver.com.POSTED!ijehs.ac.th!YWAVNU
> From: Wayne <YWAVNU@ijehs.ac.th>
> Message-ID: <4FD1F099.2544C6BA@BASIC.BS.WEBUSENET.COM>
> Newsgroups: comp.lang.prograph,comp.lang.prolog,comp.lang.python
> Subject: Free P.0.R.N.0! Link Inside! Fresh post!        5vIiiTQU RtP5cQY0
> Lines: 96
> X-Complaints-To: abuse@usenetserver.com
> X-Abuse-Info: Please be sure to forward a copy of ALL headers
> X-Abuse-Info: Otherwise we will be unable to process your complaint
> properly.
> NNTP-Posting-Date: Mon, 05 Aug 2002 23:34:02 EDT
> Organization: WEBUSENET.com
> Date: Mon, 6 Aug 2002 03:30:28 GMT
> Xref: news.baymountain.net comp.lang.prograph:12849 comp.lang.prolog:26979
>         comp.lang.python:176063
> To: python-list@python.org
> Sender: python-list-admin@python.org
> Errors-To: python-list-admin@python.org
> X-BeenThere: python-list@python.org
> X-Mailman-Version: 2.0.13 (101270)
> Precedence: bulk
> List-Help: <mailto:python-list-request@python.org?subject=help>
> List-Post: <mailto:python-list@python.org>
> List-Subscribe: <http://mail.python.org/mailman/listinfo/python-list>,
>         <mailto:python-list-request@python.org?subject=subscribe>
> List-Id: General discussion list for the Python programming language
>         <python-list.python.org>
> List-Unsubscribe: <http://mail.python.org/mailman/listinfo/python-list>,
>         <mailto:python-list-request@python.org?subject=unsubscribe>
> List-Archive: <http://mail.python.org/pipermail/python-list/>
>
> http://www.virtualwebmedia.com
>
> F-R-E-E p o r  n! NO S+P+A+M!
>
> obusj
>
> D2sMJ0EOdQMT
>
> Try arriving the shore's upper coffee and Eve will explain you!
>
> yzehpu@virtualwebmedia.com
>
> nomjyd@virtualwebmedia.com
>
> ogeddusu@virtualwebmedia.com
>
> uN9xyVlj
>
> Are you shallow, I mean, pouring in front of humble bushs?
>
> Sometimes, tags expect near upper houses, unless they're blank.
>
> Hardly any empty ointments look Fred, and they weekly talk Brahimi too.
> Hey, it solves a disk too shallow beside her wet office.  I am
> surprisingly strong, so I learn you.  It will grasp locally if
> Jimmy's potter isn't polite.  Otherwise the kettle in Cristof's
> card might recollect some old desks.  Will you scold among the
> lane, if Julieta furiously laughs the pen?
>
> He'll be liking within urban Norm until his onion climbs lazily.  As
> globally as Ramzi attacks, you can hate the puddle much more
> gently.  Daoud!  You'll pour stickers.  Sometimes, I'll nibble the
> yogi.  I was dining to sow you some of my pathetic films.
>
> Until Ayub cleans the doses stupidly, Jimmy won't dye any proud
> stations.  If the strange pickles can wander undoubtably, the
> worthwhile poultice may promise more showers.  Some weak smart
> dusts tamely fear as the filthy sauces pull.  They are irrigating
> within the spring now, won't seek powders later.  Try not to
> excuse a envelope!  Many angry tapes inside the dull monolith were
> behaving between the younger bedroom.
>
> Hardly any rural bizarre jars will totally attempt the bowls.
> When will we jump after Susan joins the fresh shore's spoon?  She'd rather
> taste finitely than receive with Ziad's new lemon.
>
> Other abysmal sticky clouds will tease quickly with elbows.  Better
> measure porters now or George will deeply love them on you.
>
> We cook them, then we dully waste Saad and Charlene's lazy book.
>
> For Usha the cat's sad, alongside me it's open, whereas in back
> of you it's
> departing short.  The coffees, buckets, and twigs are all raw and
> sharp.  Do not improve nearly while you're recommending before a
> cold teacher.  Try covering the navel's tired draper and Oscar will
> move you!  Saeed, have a dry shoe.  You won't open it.
>
> Lately Beryl will comb the lentil, and if Jbilou incredibly burns it too,
> the
> hat will converse behind the dirty night.  If you will live Rachel's
> structure within figs, it will biweekly call the floor.  Gawd,
> Wally never judges until Zakariya creeps the think ache usably.
> What Penny's kind can walks, Harvey moulds beside poor, pretty
> halls.
>
> Are you dark, I mean, kicking beside weird hens?  I was dreaming
> pumpkins to deep Amber, who's playing through the dryer's stable.
> Mahammed explains, then Hamza loudly fills a ugly cap near Karim's
> light.  You won't answer me helping before your wide river.  Let's
> lift outside the handsome rivers, but don't reject the cosmetic
> pitchers.  Lots of unique painters are hot and other brave jugs are
> lean, but will Yosri care that?  Her dog was humble, blunt, and
> changes under the cellar.  We smell once, shout grudgingly, then
> irritate inside the gardner behind the desert.  She will annually
> order outer and arrives our durable, cheap bushs with a ceiling.
>
> A lot of frogs rigidly believe the difficult doorway.  She wants to
> mould thin smogs above Shah's fog.  Who talks weakly, when Gregory
> joins the full pin without the kiosk?  Every rich farmer or plain, and
> she'll
> inadvertently smell everybody.  It's very clean today, I'll waste
> unbelievably or Allahdad will attack the units.  Where did Agha
> laugh within all the barbers?  We can't fear buttons unless Timothy will
> virtually expect afterwards.  No sauces will be active sweet
> eggs.  David, on games young and healthy, teases within it, behaving
> smartly.  When did Ali kill the printer around the clever weaver?
> I like quiet plates, do you learn them?  It might neatly dream
> against Stephanie when the heavy boats open to the lost camp.
> He will kick stale wrinkles beside the good glad ladder, whilst
> Guido stupidly creeps them too.  Just receiving without a goldsmith
> in back of the highway is too inner for Elisabeth to converse it.  The
> distant walnut rarely grasps Byron, it sows Lisette instead.
> She may excuse the easy cup and reject it within its square.  While
> oranges familiarly judge codes, the cases often burn at the lower
> tailors.
> *************************************************************************
>
> The next false positive was
>
>     Subject: Viewing JPG's
>
> and certainly was not a spam.  Ditto
>
>     Subject: If that interests you...
>
> following, and then
>
>     Subject: Oil on troubled waters ...
>
> and then
>
>     Subject: Find where you left off...
>
> The first false negatives were
>
>     Subject: howdy ukqip
>     Subject: °¢Ä·ÈðÌØ(ÑÇÖÞ)ÍøÂçÓÐÏÞ¹«Ë¾
>     Subject: Re: hi!
>     Subject: a link with minimal effort.
>         [must be about Solaris shared libraries <wink>]
>     Subject: Re: info
>     Subject: Interview Charles Payne
>     Subject: re: order
>     Subject: Possible Cause of Brain Cancer
>     Subject: re
>     Subject: cool
>     Subject: It is beneficial to your library & its patrons to have
>              the book (Please suggest)
>     Subject: Re: ...something to think about?
>     Subject: Python Video Promotion!
>         [LOL!]
>     Subject: RE: hey!
>     Subject: you too?
>     Subject: http://untroubled.org/relay-ctrl/
>     Subject: Surprise ?
>         [variations of "surprise" have high spamprob, but this was
>          saved by prob('subject: ?') = 0.0291312
>          for whatever reason, newbies on c.l.py often leave a space
>          in front of their question marks, and the tokenize-runs-
>          of-punctuation gimmick found that]
>     Subject: Help Needed
>     Subject: Waste Review
>     Subject: Re: Email Confirmation
>     Subject: Re: about this email...
>     Subject: Re: new information
>     Subject: Networking Position
>     Subject: RE:
>     Subject: What do you get when you cross a sorority girl with an ape?
>     Subject: re: entry conf.
>
> Enough already -- it did have one triumph, and plenty of those false
> negatives would have been false negatives for me "by hand" too.  Overall:
>
> -> <stat> tested 2000 hams & 1400 spams against 18000 hams & 12600 spams
>    [ditto 19 times]
>
> false positive percentages
>     0.000  0.300  lost  +(was 0)
>     0.000  0.200  lost  +(was 0)
>     0.000  0.350  lost  +(was 0)
>     0.000  0.400  lost  +(was 0)
>     0.050  0.500  lost  +900.00%
>     0.000  0.400  lost  +(was 0)
>     0.000  0.550  lost  +(was 0)
>     0.000  0.400  lost  +(was 0)
>     0.000  0.300  lost  +(was 0)
>     0.050  0.450  lost  +800.00%
>
> won   0 times
> tied  0 times
> lost 10 times
>
> total unique fp went from 2 to 77 lost  +3750.00%
> mean fp % went from 0.01 to 0.385 lost  +3750.00%
>
> false negative percentages
>     0.071  4.000  lost  +5533.80%
>     0.071  2.429  lost  +3321.13%
>     0.000  2.643  lost  +(was 0)
>     0.071  3.571  lost  +4929.58%
>     0.143  2.786  lost  +1848.25%
>     0.214  3.357  lost  +1468.69%
>     0.143  3.071  lost  +2047.55%
>     0.143  3.071  lost  +2047.55%
>     0.214  2.929  lost  +1268.69%
>     0.000  2.429  lost  +(was 0)
>
> won   0 times
> tied  0 times
> lost 10 times
>
> total unique fn went from 15 to 424 lost  +2726.67%
> mean fn % went from 0.107142857143 to 3.02857142857 lost  +2726.67%
>
> ham mean                     ham sdev
>   28.00   14.11  -49.61%        5.80    9.86  +70.00%
>   27.93   13.79  -50.63%        5.62    9.60  +70.82%
>   27.91   14.02  -49.77%        5.76   10.08  +75.00%
>   28.02   13.79  -50.79%        5.67   10.28  +81.31%
>   27.82   14.02  -49.60%        5.85   10.38  +77.44%
>   27.88   13.84  -50.36%        5.53   10.68  +93.13%
>   28.05   14.16  -49.52%        5.69   10.35  +81.90%
>   28.00   14.07  -49.75%        5.54   10.33  +86.46%
>   28.14   14.27  -49.29%        5.61   10.17  +81.28%
>   28.16   13.95  -50.46%        5.93   10.61  +78.92%
>
> ham mean and sdev for all runs
>   27.99   14.00  -49.98%        5.70   10.24  +79.65%
>
> spam mean                    spam sdev
>   85.00   84.96   -0.05%        6.92   12.31  +77.89%
>   84.80   85.33   +0.63%        6.66   11.52  +72.97%
>   84.48   85.83   +1.60%        6.57   11.41  +73.67%
>   85.01   85.25   +0.28%        6.65   12.08  +81.65%
>   85.01   85.83   +0.96%        6.49   11.06  +70.42%
>   84.89   85.60   +0.84%        6.82   11.71  +71.70%
>   84.61   85.21   +0.71%        6.68   11.83  +77.10%
>   85.00   85.85   +1.00%        6.52   11.46  +75.77%
>   85.02   85.61   +0.69%        6.78   11.09  +63.57%
>   84.96   85.80   +0.99%        6.47   11.78  +82.07%
>
> spam mean and sdev for all runs
>   84.88   85.53   +0.77%        6.66   11.64  +74.77%
>
> ham/spam mean difference: 56.89 71.53 +14.64
>
> Histogram analysis suggested:
>
> -> best cutoff for all runs: 0.575
> ->     with weighted total 10*65 fp + 486 fn = 1136
> ->     fp rate 0.325%  fn rate 3.47%
>
> If someone wants to run with this, the list of best discriminators may be
> helpful (although under Gary's scheme this is more like a list of
> most-frequent discriminators):
>
>     best discriminators:
>         'subject:are' 493 0.846732
>         'subject:mortgage' 500 0.999499
>         'subject:can' 507 0.606534
>         'subject:now' 508 0.927579
>         'subject:PEP' 512 0.000488652
>         'subject:(was' 515 0.000488652
>         'subject:newbie' 522 0.000480307
>         'subject: $' 523 0.992175
>         'subject:no' 528 0.751766
>         'subject:You' 532 0.981916
>         'subject:] ' 535 0.196305
>         'subject:,' 541 0.894722
>         'subject:pep' 546 0.000457829
>         'subject:help' 556 0.288172
>         'subject:problem' 560 0.0836141
>         'subject: & ' 570 0.75976
>         'subject:! ' 597 0.947754
>         'subject:FREE' 598 0.999584
>         'subject:this' 606 0.695102
>         'subject:&' 637 0.731907
>         'subject:How' 671 0.252991
>         'subject:get' 684 0.876916
>         'subject:new' 770 0.672979
>         'subject:question' 810 0.00423811
>         'subject:Your' 883 0.991676
>         'subject:from' 899 0.399686
>         'subject:how' 962 0.236597
>         'subject:was' 969 0.0148958
>         'subject:.' 1036 0.40169
>         'subject: - ' 1126 0.615247
>         'subject:/' 1180 0.357549
>         'subject:free' 1204 0.982183
>         'subject:is' 1216 0.349481
>         'subject:RE:' 1326 0.133728
>         'subject:)' 1364 0.258211
>         'subject: (' 1480 0.139366
>         'subject:with' 1496 0.331923
>         'subject:you' 1737 0.941033
>         'subject:, ' 1955 0.640872
>         'subject:the' 1984 0.635883
>         'subject:!' 2072 0.87807
>         'subject:your' 2260 0.984736
>         'subject:in' 2362 0.310965
>         'subject:and' 2414 0.390133
>         'subject:Python' 4321 0.00115551
>         'subject:?' 4796 0.19175
>         'subject:python' 5482 0.000912976
>         'subject:Re:' 15254 0.0194725
>         'subject:re:' 16662 0.0399341
>         'subject:: ' 17253 0.0965151
>
> Note that variations of "Re:" really gave some spam a boost.
>
>
>
> _______________________________________________
> Spambayes mailing list
> Spambayes@python.org
> http://mail.python.org/mailman-21/listinfo/spambayes


From mhoffman@lightlink.com  Sun Sep 29 04:34:21 2002
From: mhoffman@lightlink.com (Mark M. Hoffman)
Date: Sat, 28 Sep 2002 20:34:21 -0700
Subject: [Spambayes] To think like a spammer...
In-Reply-To: 
	<200209290233.g8T2XDZ24279@pcp02138704pcs.reston01.va.comcast.net>; from
	guido@python.org on Sat, Sep 28, 2002 at 10:33:13PM -0400
References: <20020928192541.A14014@mercury.solarsys.net>
	<200209290233.g8T2XDZ24279@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <20020928203421.A16022@mercury.solarsys.net>

* Guido van Rossum <guido@python.org> [2002-09-28 22:33:13 -0400]:
> > The spambayes scheme (and others like it that I've seen) can be defeated
> > easily, with something like this...
> > 
> > THIS  IS  A   F A N T A S T I C   O P P O R T U N I T Y ! !
> 
> It's an arms race.  I expect that the classification scheme can be
> kept relatively constant (the math doesn't change) but the tokenizing
> (better called feature extraction) scheme can and should be adapted
> occasionally, to deal with new ways of hiding spam.  This particular
> style can easily be recognized[*] *if* it becomes popular among
> spammers; for anything you can come up with there's a tokenizer that
> recognizes it.

I agree that this will eventually become a problem of lexical analysis.

> But spammers will only start worrying if their return rates go down,
> and that will only happen once almost everybody is using anti-spam
> technology.  We've got a long way to go before that's the case.  So
> let's not be stymied by worries about what the spammers can do.
>
> [*] I wouldn't even bother trying to recover the words FANTASTIC
> OPPORTUNITY.  This style is so completely unseen in ham that simply
> looking for many consecutive one-letter words and inserting a token
> representing such a presence would most likely be enough.

True enough that this isn't seen in ham at all.  However, a single
token which recognizes it can be overpowered by all the rest.

I had to laugh when Tim posted a spam that had slipped through all
previous tests in <LNBBLJKPBEHFEDALKOLCGEHEBHAB.tim.one@comcast.net>.
That is exactly the kind of message I'm talking about.

Regards,

-- 
Mark M. Hoffman
mhoffman@lightlink.com


From richard@reich.com  Sun Sep 29 04:35:02 2002
From: richard@reich.com (Richard Reich)
Date: Sat, 28 Sep 2002 20:35:02 -0700
Subject: [Spambayes] To think like a spammer...
References: <20020928192541.A14014@mercury.solarsys.net>
	<20020928193324.D69B.JCARLSON@uci.edu>
Message-ID: <002901c26769$32299c50$0300a8c0@bizou>

Or you can drop the reliance on words entirely.  See
http://www-106.ibm.com/developerworks/library/l-spamf.html for an
interesting treatment of a statistical filtering technique based on
trigrams.

-r

----- Original Message -----
From: "Josiah Carlson" <jcarlson@uci.edu>
To: <spambayes@python.org>
Sent: Saturday, September 28, 2002 7:54 PM
Subject: Re: [Spambayes] To think like a spammer...


> > Even if you back up and allow single character tokens, you're only going
> > to recognize a handful of those.  A spammer could stuff the end of the
> > message with ham words to overcome the effect of the single char.
tokens.
> > Do these space-words have to be collapsed to defeat the effect?
> >
> > Actually, protection against stuffing the end of a spam w/ ham words is
> > an angle we have to be careful about anyway.
>
> If there were enough spammers that used such things, they would
> definitely find their messages in the spam corpus.  If one then started
> allowing 1-character tokens, through spambayes, because most words are
> not of a single letter, those letters that happen to not be used as
> initials in sigs or otherwise, would likely be flagged as being more
> spammy than hammy.
>
> The problem with removing spaces is that the software would need to have
> a listing of valid words, and one would need to check all the possible
> concatenation of tokens that could _potentially_ create a word.  You
> could look at it in terms of a bitmap.
>
> Given a list of bits, each representing a different token, you can use 1
> to represent a continuation of the previous word (concatenated) and a 0
> to represent the beginning of a new word. Current non-concatenation
> gives us all 0's.  You can create a secondary bitmap to show which of
> the tokens/bits can be a continuation.
>
> If you take a random approach to it, you're looking at 2**(number of
> single-character entries) for the number of ways the message can be
> concatenated.  That's REALLY bad.
>
> However, you could do a DFS on a tree of valid words, which would
> produce unbelievably fast results in C, and even all right results using
> python dictionaries (something that I feel like coding up this evening,
> it'll be neat), but you would still have problems with purposeful
> misspellings.
>
> Good point though,
>  - Josiah
>
>



From nas@python.ca  Sun Sep 29 04:45:28 2002
From: nas@python.ca (Neil Schemenauer)
Date: Sat, 28 Sep 2002 20:45:28 -0700
Subject: [Spambayes] More HTML strippage.
In-Reply-To: <LNBBLJKPBEHFEDALKOLCIEGLBHAB.tim.one@comcast.net>
References: <20020928150224.A10925@mercury.solarsys.net>
	<LNBBLJKPBEHFEDALKOLCIEGLBHAB.tim.one@comcast.net>
Message-ID: <20020929034528.GA22839@glacier.arctrix.com>

Tim Peters wrote:
> [Mark M. Hoffman]
> > A modest proposal:  tokenize the number of message recipients e.g.
> > header:num_recip_21 for the above.
> 
> Yes, we don't do anything with To, Cc, etc, now, except count the number of
> times (>= 1) each header line appears.

I think it would be better to generate:

    header:num_recip_1
    header:num_recip_2
    header:num_recip_3
    header:num_recip_4
    ...
    header:num_recip_21

We should probably do the same thing when counting headers.  I'll give
it a try.

  Neil

From tim.one@comcast.net  Sun Sep 29 05:08:41 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 29 Sep 2002 00:08:41 -0400
Subject: [Spambayes] To think like a spammer...
In-Reply-To: <20020928192541.A14014@mercury.solarsys.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEHIBHAB.tim.one@comcast.net>

[Mark M. Hoffman]
> ...
> The spambayes scheme (and others like it that I've seen) can be defeated
> easily, with something like this...
>
> <spam>
>
> THIS  IS  A   F A N T A S T I C   O P P O R T U N I T Y ! !
>
> W A N T  TO  MAKE  A  TON  OF  M O N E Y   ? ? ?
>
> THIS   W E B S I T E   HAS  THE  A N S W E R   TO  ALL  YOUR
> F I N A N C I N G   P R O B L E M S ! ! !
>
> IT  COULD  NOT  BE   E A S I E R ! ! !   C L I C K  HERE NOW ! ! !
>
> http://blah.blah.com/blah/blah.html
>
> </spam>

Spam is cheap to send, but it's not free to send.  Crap like that will
certainly lower the response rate, and at some point it won't repay the
cost.

> Even if you back up and allow single character tokens, you're only
> going to recognize a handful of those.

It's easy to teach the tokenizer about patterns like these if they become
popular.

> A spammer could stuff the end of the message with ham words to overcome
> the effect of the single char. tokens.

*That* would be very expensive for spammers to do:  "ham words" aren't
universal, they depend on the kinds of things the end user is interested in.
If spammers had the resource to determine this, they wouldn't be using a
bottom-feeding advertising medium like spam; they'd be using traditional
targeted marketing techniques, which are darned near a science but also cost
Real Money to pursue.

> Do these space-words have to be collapsed to defeat the effect?

Our current tokenizer won't even "see" them (it ignores "words" with just 1
or 2 chars).  If there's anything spammish in the URL or the headers, the
trick won't do them any good.  A simple and cheap tokenization trick could
merely count the number of 1-character "words", and generate meta-tokens
based on that.  For example, while our current tokenizer doesn't know beans
about Asian languages, all the Asian spam in my corpus is caught because the
tokenizer generates meta-tokens indicating what percentage of characters in
a word have the sign bit set.  Asian spam generates lots of meta-tokens of
that nature, and they all end up with high spamprobs as part of normal
training.

> Actually, protection against stuffing the end of a spam w/ ham words is
> an angle we have to be careful about anyway.

There's plenty of spam that does that already.  The error rates I'm seeing
say that spammers can't guess ham words that are strong enough to overcome
their spam words, and that stands to reason.  I *suspect* this is one reason
I find that our robinson_minimum_prob_strength setting lowers the f-n rate
across scheme after scheme:  the best spammers can hope to achieve without
targeted marketing is to load up on mountains of bland words, but we
systematically ignore those with robinson_minimum_prob_strength set to 0.1.
And they can't even know what our non-bland words are without access to the
database internals.


From tim.one@comcast.net  Sun Sep 29 05:15:16 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 29 Sep 2002 00:15:16 -0400
Subject: [Spambayes] To think like a spammer...
In-Reply-To: <200209290229.g8T2TL721093@localhost.localdomain>
Message-ID: <LNBBLJKPBEHFEDALKOLCEEHJBHAB.tim.one@comcast.net>

[Anthony Baxter]
> ...
> A more ugly one I'm seeing (which is a persistent source of a few fn)
> is HTML email which is a huge slab of javascript, and the message
> text encoded inside the message.

Me too, and especially when whitespace has been squashed out of the
Javascript.  The split-on-whitespace strategy only generates 'skip:x' tokens
then, and only one per line of code.

We could certainly do better on this.  Part of the *problem* is the HTML
tag-stripping, which removes all direct knowledge about the presence of
script before real tokenization even begins.  I expect there are lots of
clues we're missing by stripping HTML, btw; the only ones we catch are
embedded http/https/ftp thingies (which are mined before HTML stripping
takes place).

A cute one:  I had one f-n that had about 50 spam phrases hiding in a META
KEYWORDS thingie.  Of course we threw all of them away unlooked at.


From nas@python.ca  Sun Sep 29 05:21:44 2002
From: nas@python.ca (Neil Schemenauer)
Date: Sat, 28 Sep 2002 21:21:44 -0700
Subject: [Spambayes] More HTML strippage.
In-Reply-To: <20020929034528.GA22839@glacier.arctrix.com>
References: <20020928150224.A10925@mercury.solarsys.net>
	<LNBBLJKPBEHFEDALKOLCIEGLBHAB.tim.one@comcast.net>
	<20020929034528.GA22839@glacier.arctrix.com>
Message-ID: <20020929042144.GA22920@glacier.arctrix.com>

Neil Schemenauer wrote:
> I think it would be better to generate:
> 
>     header:num_recip_1
>     header:num_recip_2
>     header:num_recip_3
>     header:num_recip_4
>     ...
>     header:num_recip_21
> 
> We should probably do the same thing when counting headers.  I'll give
> it a try.

Not so good.  This seems to work good though:

    to:2**4

IOW, log2(n).  The idea is that going from 8 to 16 is about that same as
going from 1 to 2.  I wonder if we should to this for 'skip:' as well.

  Neil

From tim.one@comcast.net  Sun Sep 29 05:23:39 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 29 Sep 2002 00:23:39 -0400
Subject: [Spambayes] To think like a spammer...
In-Reply-To: <20020928203421.A16022@mercury.solarsys.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEHKBHAB.tim.one@comcast.net>

[Mark M. Hoffman]
> ...
> I had to laugh when Tim posted a spam that had slipped through all
> previous tests in <LNBBLJKPBEHFEDALKOLCGEHEBHAB.tim.one@comcast.net>.

Not as hard as I did <wink>.

> That is exactly the kind of message I'm talking about.

It's 1 in the 20,000 hams I tested on, and if you look harder at that
message, the real reason it got thru is that it did lots of things "right".
For example, it had an Organization header, and X-Complaints-To, and
X-Abuse-Info.  Those are rare in spam.  OTOH, if we were doing more header
analysis, it would have lost big for cross-posting.  No single trick can
fool this kind of scheme, and if 20 spam in 20,000 can, BFD.


From tim.one@comcast.net  Sun Sep 29 05:28:09 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 29 Sep 2002 00:28:09 -0400
Subject: [Spambayes] To think like a spammer...
In-Reply-To: <002901c26769$32299c50$0300a8c0@bizou>
Message-ID: <LNBBLJKPBEHFEDALKOLCMEHKBHAB.tim.one@comcast.net>

[Richard Reich]
> Or you can drop the reliance on words entirely.  See
> http://www-106.ibm.com/developerworks/library/l-spamf.html for an
> interesting treatment of a statistical filtering technique based on
> trigrams.

Read the comments in tokenizer.py for our test results on this.  Trigrams
did worse than what we're doing now.  Surprisingly, character 5-grams did
even worse than that.  Those were on the max_discriminators=16 Graham
scheme, though, and it's one of the things that should be revisited since we
switched to Gary's scheme (which is looking at 150 by default now).  I note
that Mertz there says he used 100 trigrams, although it's not clear to me
exactly what he did.


From tim.one@comcast.net  Sun Sep 29 05:45:28 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 29 Sep 2002 00:45:28 -0400
Subject: [Spambayes] More HTML strippage.
In-Reply-To: <20020929042144.GA22920@glacier.arctrix.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEHLBHAB.tim.one@comcast.net>

[Neil Schemenauer]
>> I think it would be better to generate:
>>
>>     header:num_recip_1
>>     header:num_recip_2
>>     header:num_recip_3
>>     header:num_recip_4
>>     ...
>>     header:num_recip_21
>>
>> We should probably do the same thing when counting headers.  I'll give
>> it a try.


> Not so good.  This seems to work good though:
>
>     to:2**4
>
> IOW, log2(n).

Cool!  Thank you.  I'm running tests now (it will take a while to finish).

> The idea is that going from 8 to 16 is about that same as
> going from 1 to 2.

Makes good sense to me!  I had wondered in the past, but not pursued,
whether a log gimmick would have been better in Graham's "count multiple
words in a msg multiple times during training" scheme too.

> I wonder if we should to this for 'skip:' as well.

As the comments say <wink>, I have no idea what skip accomplishes, just that
it improves error rates, and that every variation I've ever tried did worse.
I haven't tried a logarithmic version, though.  I suspect that whenever skip
helps, it's exposing a systematic weakness of the tokenizer, but it's very
time-consuming to analyze things at that level.  Feel encouraged to
experiment!  I don't like skip, but the data says "it works" (or, more
accurately, that it helped the last time I ran a controlled experiment
changing just it).


From mhoffman@lightlink.com  Sun Sep 29 05:52:38 2002
From: mhoffman@lightlink.com (Mark M. Hoffman)
Date: Sat, 28 Sep 2002 21:52:38 -0700
Subject: [Spambayes] To think like a spammer...
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCEHKBHAB.tim.one@comcast.net>;
	from tim.one@comcast.net on Sun, Sep 29, 2002 at 12:23:39AM -0400
References: <20020928203421.A16022@mercury.solarsys.net>
	<LNBBLJKPBEHFEDALKOLCCEHKBHAB.tim.one@comcast.net>
Message-ID: <20020928215238.A16776@mercury.solarsys.net>

* Tim Peters <tim.one@comcast.net> [2002-09-29 00:23:39 -0400]:
> [Mark M. Hoffman]
> > ...
> > I had to laugh when Tim posted a spam that had slipped through all
> > previous tests in <LNBBLJKPBEHFEDALKOLCGEHEBHAB.tim.one@comcast.net>.
> 
> Not as hard as I did <wink>.
> 
> > That is exactly the kind of message I'm talking about.
> 
> It's 1 in the 20,000 hams I tested on, and if you look harder at that
> message, the real reason it got thru is that it did lots of things "right".
> For example, it had an Organization header, and X-Complaints-To, and
> X-Abuse-Info.  Those are rare in spam.  OTOH, if we were doing more header
> analysis, it would have lost big for cross-posting.  No single trick can
> fool this kind of scheme, and if 20 spam in 20,000 can, BFD.

Don't get me wrong: I'm not trying to poo-poo your results, which are
fantastic.  I'm just trying to keep this in mind from the Graham paper:

>> Still, anyone who proposes a plan for spam filtering has to be able to
>> answer the question: if the spammers knew exactly what you were doing,
>> how well could they get past you?

Regards,

-- 
Mark M. Hoffman
mhoffman@lightlink.com


From tim.one@comcast.net  Sun Sep 29 06:09:42 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 29 Sep 2002 01:09:42 -0400
Subject: [Spambayes] On counting words more than once
In-Reply-To: <20020928184916.GA22062@glacier.arctrix.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCMEHMBHAB.tim.one@comcast.net>

[Neil Schemenauer, tries
   [Classifier]
   count_duplicates_only_once_in_training: True
]
> It's a win for me:
>
>     false positive percentages
>         0.000  0.000  tied
>         1.000  1.000  tied
>         0.000  0.000  tied
>         0.000  0.000  tied
>         0.500  0.500  tied
>         0.500  1.000  lost  +100.00%
>         0.500  0.500  tied
>         0.000  0.000  tied
>         0.500  0.500  tied
>         0.000  0.500  lost  +(was 0)
>
>     won   0 times
>     tied  8 times
>     lost  2 times
>
>     total unique fp went from 6 to 8 lost   +33.33%
>     mean fp % went from 0.3 to 0.4 lost   +33.33%
>
>     false negative percentages
>         0.000  0.000  tied
>         1.000  0.500  won    -50.00%
>         1.000  0.500  won    -50.00%
>         0.500  0.500  tied
>         2.000  1.500  won    -25.00%
>         1.500  1.000  won    -33.33%
>         0.000  0.000  tied
>         0.500  0.000  won   -100.00%
>         0.500  0.000  won   -100.00%
>         0.000  0.000  tied
>
>     won   6 times
>     tied  4 times
>     lost  0 times
>
>     total unique fn went from 14 to 8 won    -42.86%
>     mean fn % went from 0.7 to 0.4 won    -42.86%
>
>     ham mean                     ham sdev
>       30.01   27.92   -6.96%        8.43    8.42   -0.12%
>       28.50   26.74   -6.18%        8.83    8.69   -1.59%
>       27.93   26.04   -6.77%        8.20    7.94   -3.17%
>       29.55   27.33   -7.51%        8.24    8.23   -0.12%
>       29.05   27.19   -6.40%        8.28    8.15   -1.57%
>       31.40   29.48   -6.11%        9.41    9.25   -1.70%
>       29.31   27.49   -6.21%        8.13    8.10   -0.37%
>       29.33   27.16   -7.40%        7.86    7.89   +0.38%
>       28.72   27.22   -5.22%        9.05    8.97   -0.88%
>       29.04   26.87   -7.47%        7.28    7.22   -0.82%
>
>     ham mean and sdev for all runs
>       29.28   27.34   -6.63%        8.44    8.35   -1.07%
>
>     spam mean                    spam sdev
>       82.98   81.91   -1.29%        9.83   10.16   +3.36%
>       82.02   81.04   -1.19%        9.92   10.09   +1.71%
>       81.19   80.28   -1.12%        9.69    9.86   +1.75%
>       82.51   81.66   -1.03%        9.92   10.23   +3.13%
>       82.60   81.60   -1.21%       10.12   10.33   +2.08%
>       82.24   81.36   -1.07%        9.25    9.71   +4.97%
>       81.74   80.85   -1.09%        9.30    9.49   +2.04%
>       81.70   80.64   -1.30%        9.51    9.81   +3.15%
>       82.39   81.45   -1.14%        9.87   10.18   +3.14%
>       82.44   81.45   -1.20%        9.49    9.73   +2.53%
>
>     spam mean and sdev for all runs
>       82.18   81.22   -1.17%        9.71    9.97   +2.68%
>
>     ham/spam mean difference: 52.90 53.88 +0.98

So all the same things I saw:  ham and spam means decrease, overall mean
spread increases, ham variance is a mixed bag, and spam variance increases.

Anyone else?  Based on Neil and my results (which are all we have), we
should make this change and get rid of the option.

Ponder.  I conjecture there's a set of words that are common in both ham and
spam, but are more likely to appear more often in spam than in ham.  The
multiple-count gimmick would then give them larger spamprobs during
training.  Taking that bias away thus stops penalizing ham for using them
too (mean decreases), but makes spam fuzzier (variance increases).

It also suggests there's *some* useful info to be had about how often a word
appears in a msg, but that adjusting spamprob isn't the right way to exploit
it.  If Gary is dying for something to think about <wink>, is there a simple
way to model a multinomial distribution in this framework?


From tim.one@comcast.net  Sun Sep 29 06:14:40 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 29 Sep 2002 01:14:40 -0400
Subject: [Spambayes] to From_ or not to From_?
In-Reply-To: <200209281849.g8SInZp20629@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEHNBHAB.tim.one@comcast.net>

[Guido, on Unix "From " lines, and that Tim doesn't have any]
> Weird.  I used splitndirs.py to create my normalized test data setup
> and it wrote Unix From lines.  In fact, looking at the code, it uses
> str(msg), which forces unixfrom=1, which always writes a Unix From
> line.
>
> But it's possible that you created your data setup using a different
> version of splitndirs.py.

No, it's not this deep a mystery:  I wrote a little script whose only
purpose was to remove these lines, and to normalize line endings.  At the
start of this, I had no idea how the email pkg worked, and didn't know
whether having one corpus with these lines and one without was going to
screw up my results.  So I forced uniformity.  At the time, I figured I was
the only person who would "suffer" from mixed-source corpora too, so didn't
bother polishing that script and checking it in.

> Anyway, the email package always recognizes a Unix From line (it's
> hard to mistake for an rfc822 header line) and stores it in a special
> attribute of the Message object.  Unless you wrote code in your
> tokenizer to look at that, I'm pretty sure you're ignoring it. :-)

Yup!  I realize that now.

> So Skip can stop worrying: presence or absence of Unix From lines
> doesn't matter.

Agreed.


From tim.one@comcast.net  Sun Sep 29 06:32:56 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 29 Sep 2002 01:32:56 -0400
Subject: [Spambayes] To think like a spammer...
In-Reply-To: <20020928215238.A16776@mercury.solarsys.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEHOBHAB.tim.one@comcast.net>

[Mark M. Hoffman]
> Don't get me wrong:

I'm not <wink>.

> I'm not trying to poo-poo your results, which are fantastic.
> I'm just trying to keep this in mind from the Graham paper:

> >> Still, anyone who proposes a plan for spam filtering has to be able
> >> to> answer the question: if the spammers knew exactly what you were
> >> doing, how well could they get past you?

That's very much worth testing too, and John Draper specifically suggested
it last week.  Like most projects, we have no lack of ideas to pursue, what
we need are people to work on them.  You're elected <wink>:  dummy up some
spam, and we'll see how it does.  Don't forget that the spam has to be
interesting enough to *someone*, though, to more than pay back your costs.

BTW, I expect I could easily write a spam that would get thru to
comp.lang.python; e.g.,

    Is that a snake in your pants, or are you just glad to see me?
    MOBYMONSTERDICK is the object-oriented Viagra favored by
    Guido van Rossum and Mark Hammond, creators of the Python
    programming language and the Win32 extension modules.  Yes,
    now even a Windows-loving Aussie can have a MOBYMONSTERDICK!
    Don't be intimidated by metaclasses another minute.
    Details at http://www.blandname.com/mobymonsterdick

    tell-'em-uncle-timmy-sent-you-ly y'rs  - larry wall

Heck, that may even get thru to comp.lang.perl.misc too, although it's not
going to fly on rec.sport.pro-wrestling <wink>.


From guido@python.org  Sun Sep 29 15:37:41 2002
From: guido@python.org (Guido van Rossum)
Date: Sun, 29 Sep 2002 10:37:41 -0400
Subject: [Spambayes] On counting words more than once
In-Reply-To: Your message of "Sun, 29 Sep 2002 01:09:42 EDT."
             <LNBBLJKPBEHFEDALKOLCMEHMBHAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCMEHMBHAB.tim.one@comcast.net> 
Message-ID: <200209291437.g8TEbf809551@pcp02138704pcs.reston01.va.comcast.net>

It's a win for me too (total 2000 ham, 2000 spam):

false positive percentages
    0.000  0.000  tied          
    0.500  0.500  tied          
    0.000  0.000  tied          
    0.500  0.000  won   -100.00%
    1.000  0.500  won    -50.00%
    0.000  0.000  tied          
    1.000  0.500  won    -50.00%
    0.500  0.500  tied          
    0.000  0.000  tied          
    0.500  0.500  tied          

won   3 times
tied  7 times
lost  0 times

total unique fp went from 8 to 5 won    -37.50%
mean fp % went from 0.4 to 0.25 won    -37.50%

false negative percentages
    0.500  0.500  tied          
    0.000  0.000  tied          
    0.000  0.000  tied          
    0.000  0.000  tied          
    0.500  0.500  tied          
    0.500  0.500  tied          
    0.000  0.000  tied          
    0.000  0.000  tied          
    0.000  0.000  tied          
    0.000  0.000  tied          

won   0 times
tied 10 times
lost  0 times

total unique fn went from 3 to 3 tied          
mean fn % went from 0.15 to 0.15 tied          

--Guido van Rossum (home page: http://www.python.org/~guido/)

From nas@python.ca  Sun Sep 29 16:26:28 2002
From: nas@python.ca (Neil Schemenauer)
Date: Sun, 29 Sep 2002 08:26:28 -0700
Subject: [Spambayes] More HTML strippage.
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEHLBHAB.tim.one@comcast.net>
References: <20020929042144.GA22920@glacier.arctrix.com>
	<LNBBLJKPBEHFEDALKOLCKEHLBHAB.tim.one@comcast.net>
Message-ID: <20020929152628.GA23576@glacier.arctrix.com>

Tim Peters wrote:
> As the comments say <wink>, I have no idea what skip accomplishes, just that
> it improves error rates, and that every variation I've ever tried did worse.
> I haven't tried a logarithmic version, though.

It seems to do a little worse than the current code.

  Neil

From tim.one@comcast.net  Sun Sep 29 17:07:18 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 29 Sep 2002 12:07:18 -0400
Subject: [Spambayes] RE: Central Limit Theorem??!! :)
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEGABHAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCAEIHBHAB.tim.one@comcast.net>

[Tim]
> This is a log-central-limit experiment with a 10x10 grid.  500 ham
> and 500 spam selected at random from each of my sets, lumped into 10
> pairs.  Then
>
> train on pair 1, predict on pairs 2 thru 10
> train on pair 2, predict on pairs 1, and 3 thru 10
> ...
> train on pair 10, predict on pairs 1 thru 9
>
> In all, that's 90 prediction runs on 1000 msgs per run, for 90,000 total
> predictions.  Each of the 10*1000 = 10,000 unique msgs is predicted 9
> times.
>
> This is a hard test ...

Then I made this up as a measure of "certainty":

> ...
>         if min(abs(zham), abs(zspam)) < 10.0 and abs(stat) > 5.0:
>             stat = stat > 0.0 and 1.0 or 0.0
>         else:
>             if stat > 0.0:
>                 stat = 0.51
>             else:
>                 stat = 0.49
>
> IOW, it's certain iff at least one z-score is "not insanely
> large", and the other z-score is "substantially larger".  This is
> purely a hack just to see what would happen.

On a different run than I reported on, here's the bottom line:

for all ham
    45000 total
    certain    44020 97.822%
        wrong      0  0.000%
    unsure       980  2.178%
        wrong     37  3.776%

for all spam
    45000 total
    certain    44151 98.113%
        wrong      8  0.018%
    unsure       849  1.887%
        wrong     76  8.952%

Staring at the errors suggested a different measure of certainty:

        zham = abs(zham)
        zspam = abs(zspam)
        ratio = max(zham, zspam) / min(zham, zspam)
        certain = ratio > 3.0 or (n > 30 and ratio > 2.0)

n is the number of extreme words found in the msg.  If it doesn't exceed 30,
appeal to the central limit theorem is dubious.  So, in that case, I require
a larger ratio.  On the same data, the bottom line changes to:

for all ham
    45000 total
    certain    44698 99.329%
        wrong      0  0.000%
    unsure       302  0.671%
        wrong     37 12.252%
for all spam
    45000 total
    certain    44166 98.147%
        wrong      0  0.000%
    unsure       834  1.853%
        wrong     84 10.072%

IOW, it's certain more often (much more often for ham!), never made a
mistake when it was certain, and made mistakes at higher rates when it
wasn't certain.

The lesson I take is that somebody who actually knows what they're doing
could get something mondo useful out of this.

I noted that adding another clause

     ... or (n > 40 and ratio > 1.75)

changed the bottom line to

for all ham
    45000 total
    certain    44731 99.402%
        wrong      0  0.000%
    unsure       269  0.598%
        wrong     37 13.755%
for all spam
    45000 total
    certain    44258 98.351%
        wrong      0  0.000%
    unsure       742  1.649%
        wrong     84 11.321%

I've no idea how much useful info I'm leaving untouched.


From tim.one@comcast.net  Sun Sep 29 18:00:05 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 29 Sep 2002 13:00:05 -0400
Subject: [Spambayes] Checkin notification is hosed
Message-ID: <LNBBLJKPBEHFEDALKOLCKEIIBHAB.tim.one@comcast.net>

SourceForge apparently can't connect to python.org:

Checking in rebal.py;
/cvsroot/spambayes/spambayes/rebal.py,v  <--  rebal.py
new revision: 1.8; previous revision: 1.7
done
Mailing spambayes-checkins@python.org...
Generating notification message...
Generating notification message... done.
Mailing spambayes-checkins@python.org...
Generating notification message...
Traceback (innermost last):
  File "/cvsroot/spambayes/CVSROOT/syncmail", line 336, in ?
    main()
  File "/cvsroot/spambayes/CVSROOT/syncmail", line 329, in main
    blast_mail(subject, people, specs[1:], contextlines, fromhost)
  File "/cvsroot/spambayes/CVSROOT/syncmail", line 227, in blast_mail
    conn.connect(MAILHOST, MAILPORT)
  File "/usr/lib/python1.5/smtplib.py", line 216, in connect
    self.sock.connect(host, port)
socket.error: (111, 'Connection refused')


rebal.py now has a -d (dry run) option:  If you specify -d, rebal will
display how many files it's going to move, from where and to where, but
won't actually move anything.


From tim.one@comcast.net  Sun Sep 29 19:08:05 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 29 Sep 2002 14:08:05 -0400
Subject: [Spambayes] On counting words more than once
In-Reply-To: <200209291437.g8TEbf809551@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEINBHAB.tim.one@comcast.net>

SF still isn't able to mail checkin notifications.

Because Neil, Guido and I all reported improvement via counting duplicate
words (within a message) only once during training, I removed the recent
option for trying this, and we do this all the time now.  The checkin
comment is below.  Note that you may need to change spam_cutoff!

"""
Removed option count_duplicates_only_once_in_training:  this is always
done now.  Counting duplicate words in a msg more than once during
training appears to have been helpful under the Graham scheme only because
it acted to counteract other biases.

Under Robinson's unbiased scheme, results improve by counting duplicates
only once during training (just as duplicates are counted only once during
scoring), the ham score mean decreases significantly and consistently,
likewise ham score variance, the spam score mean decreases consistently
(but less than the ham mean decreased, so the spread increases), and spam
score variance increaeses.  That implies there's *some* value to be gotten
out of knowing how often a word appears in a msg, but that distorting
spamprob isn't the right way to exploit it.

WordInfo.hamcount now has a different meaning:  it's the number of hams in
which the word appears, instead of the number of times the word appears
across all ham.  Likewise for WordInfo.spamcount.

Note that because both mean scores decreased, you'll probably want a
smaller spam_cutoff value now.  The default spam_cutoff has been changed
from 0.57 to 0.56.  But this is corpus-dependent, so be sure to tune your
value for your corpus.
"""


From Alexander@Leidinger.net  Sun Sep 29 19:08:58 2002
From: Alexander@Leidinger.net (Alexander Leidinger)
Date: Sun, 29 Sep 2002 20:08:58 +0200
Subject: [Spambayes] Checkin notification is hosed
In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEIIBHAB.tim.one@comcast.net>
References: <LNBBLJKPBEHFEDALKOLCKEIIBHAB.tim.one@comcast.net>
Message-ID: <20020929200858.1a85fa9a.Alexander@Leidinger.net>

On Sun, 29 Sep 2002 13:00:05 -0400 Tim Peters <tim.one@comcast.net>
wrote:

> SourceForge apparently can't connect to python.org:

It looks more it can't connect to it's own MTA...

>     conn.connect(MAILHOST, MAILPORT)
>   File "/usr/lib/python1.5/smtplib.py", line 216, in connect
>     self.sock.connect(host, port)
> socket.error: (111, 'Connection refused')

# Which SMTP server to do we connect to?  Empty string means localhost.
MAILHOST = ''
MAILPORT = 25

Bye,
Alexander.

-- 
                  Weird enough for government work.

http://www.Leidinger.net                       Alexander @ Leidinger.net
  GPG fingerprint = C518 BC70 E67F 143F BE91  3365 79E2 9C60 B006 3FE7

From tim.one@comcast.net  Sun Sep 29 20:55:49 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 29 Sep 2002 15:55:49 -0400
Subject: [Spambayes] RE: Central Limit Theorem??!!     :)
In-Reply-To: <B9B9B1E9.169A5%grobinson@transpose.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEJCBHAB.tim.one@comcast.net>

Something odd I noticed in my log central-limit code:  the z-scores looked
"too big", but maybe I'm confusing myself.

I was running with max_discriminators=50.  Of the 45,000 each of ham and
spam predicted against, 27,094 ham had at least 50 extreme words, and 36,491
spam had at least 50 extreme words.  In what follows I'm looking solely at
those, and 50 should be plenty big enough for the theorem to kick in.

What I *expected* is, that in

        zham  = (hmean - self.hammean) / sqrt(self.hamvar / n)
        zspam = (smean - self.spammean) / sqrt(self.spamvar / n)

where
    n is the number of extreme words (always 50 here)
    hmean is the mean of the n message ln(1-p) values
    smean is the mean of the n message ln(p) values
    hammean is the ham training population mean ln(1-p)
    hamvar  is the ham training population ln(1-p) variance
    spammean is the spam training population mean ln(p)
    spamvar  is the spam training population ln(p) variance

that zham would be approximately normally distributed (mean 0, sdev 1) when
predicting a ham, and zspam similarly when predicting a spam.

But it's nowhere near that:

When predicting a ham,

This % of hams   had abs(zham) <= this
--------------   ---------------------
18.377%           1.0
36.525%           2.0
53.650%           3.0
67.919%           4.0
78.301%           5.0
85.831%           6.0
90.788%           7.0
93.762%           8.0
95.696%           9.0
97.044%          10.0

and when predicting a spam,

This % of spams  had abs(zspam) <= this
---------------  ----------------------
20.337%           1.0
44.120%           2.0
69.732%           3.0
89.559%           4.0
92.702%           5.0
94.656%           6.0
96.191%           7.0
97.205%           8.0
97.964%           9.0
98.539%          10.0

I *expected* that about 68% of messages would have abs(z) <= 1.0, and about
95% <= 2.0, etc.  What I'm seeing is more like a linear relationship!

So I'm confused, or something's very fishy here.  I've stared at the code
until my eyes bled, but it still looks like it's doing everything right, and
I'm still using scaled unbounded integers to compute the population means
and variances so there's no possibility of roundoff error polluting the
results.

Got a clue?  It's very curious that the 68% and 95% I was looking for show
up at 4 and 8-9 in the ham results, and at 3 and 6-7 in the spam results.


From tim.one@comcast.net  Sun Sep 29 21:34:58 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 29 Sep 2002 16:34:58 -0400
Subject: [Spambayes] Another change
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEINBHAB.tim.one@comcast.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCOEJDBHAB.tim.one@comcast.net>

Change checked in to tokenizer.py:

tokenize_headers():  Based on a silly experiment that *only* tokenized
Subject lines, added a gimmick here to generate tokens for runs of
punctuation characters (\W+) in subject lines.

-> <stat> tested 2000 hams & 1400 spams against 18000 hams & 12600 spams
   [ditto 19 times]

false positive percentages
    0.050  0.000  won   -100.00%
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.050  0.050  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.000  0.000  tied
    0.050  0.050  tied

won   1 times
tied  9 times
lost  0 times

total unique fp went from 3 to 2 won    -33.33%
mean fp % went from 0.015 to 0.01 won    -33.33%

false negative percentages
    0.071  0.071  tied
    0.071  0.071  tied
    0.000  0.000  tied
    0.143  0.143  tied
    0.143  0.143  tied
    0.214  0.214  tied
    0.143  0.143  tied
    0.143  0.143  tied
    0.214  0.214  tied
    0.000  0.000  tied

won   0 times
tied 10 times
lost  0 times

total unique fn went from 16 to 16 tied
mean fn % went from 0.114285714286 to 0.114285714286 tied

ham mean                     ham sdev
  25.74   25.65   -0.35%        5.74    5.67   -1.22%
  25.69   25.61   -0.31%        5.56    5.50   -1.08%
  25.64   25.57   -0.27%        5.74    5.67   -1.22%
  25.74   25.66   -0.31%        5.61    5.54   -1.25%
  25.50   25.42   -0.31%        5.78    5.72   -1.04%
  25.58   25.51   -0.27%        5.44    5.39   -0.92%
  25.73   25.65   -0.31%        5.63    5.59   -0.71%
  25.69   25.61   -0.31%        5.47    5.41   -1.10%
  25.92   25.84   -0.31%        5.54    5.48   -1.08%
  25.90   25.81   -0.35%        5.88    5.81   -1.19%

ham mean and sdev for all runs
  25.71   25.63   -0.31%        5.64    5.58   -1.06%

spam mean                    spam sdev
  84.07   83.86   -0.25%        7.10    7.09   -0.14%
  83.83   83.64   -0.23%        6.84    6.83   -0.15%
  83.46   83.27   -0.23%        6.80    6.81   +0.15%
  84.03   83.82   -0.25%        6.88    6.88   +0.00%
  84.08   83.89   -0.23%        6.68    6.65   -0.45%
  83.96   83.78   -0.21%        6.99    6.96   -0.43%
  83.62   83.42   -0.24%        6.84    6.82   -0.29%
  84.04   83.86   -0.21%        6.71    6.71   +0.00%
  84.08   83.88   -0.24%        7.01    6.98   -0.43%
  83.97   83.75   -0.26%        6.65    6.65   +0.00%

spam mean and sdev for all runs
  83.91   83.72   -0.23%        6.85    6.84   -0.15%

ham/spam mean difference: 58.20 58.09 -0.11

This is consistent but weak.  Staring at the false negatives shows
that it's moving them "in the right direction", though, and histogram
analysis says something stronger:

-> best cutoff for all runs: 0.55
->     with weighted total 10*2 fp + 11 fn = 31
->     fp rate 0.01%  fn rate 0.0786%

That is, if I had run at spam_cutoff 0.55 instead of 0.56, it would
have been a pure win, leaving f-p alone but dropping 5(!) of the f-n.

From agmsmith@rogers.com  Sun Sep 29 22:26:02 2002
From: agmsmith@rogers.com (Alexander G. M. Smith)
Date: Sun, 29 Sep 2002 17:26:02 EDT (-0400)
Subject: [Spambayes] Purging Old Words - Date vs Sequence
Message-ID: <19432086158-BeMail@CR593174-A>

While implementing a BeOS version of Paul Graham's spam detection
algorithm (available at http://www.bebits.com/app/3055 - I'll be
switching to Gary Robinson's algorithm soon), I had a need to purge
old words from the database.  More of a need than usual since I'm
simplisticly considering the whole message and breaking it into
simple words, even binary attachments.  Then deleting the unused
binary garbage after a while.  I suppose that technique could even
find spam encoded as pictures.

I thought of using a date stamp, like the spambayes project does,
but that could unevenly remove messages since they aren't added
at uniform times.  Instead, I assign a sequentially increasing
serial number to each example of spam or ham, and store that
along with the words and frequency counts.  If the word is in
a later example message, the serial number of the new example
replaces the old one for that word.  Then I can purge words which
appeared N messages before the latest one (usually in combination
with having a low frequency count).  I suppose you could even
factor in the kill count or have a last used to kill spam
date/serial number too.  Anyway, that's the only novel idea I've
had on the topic, everything else I've thought of has been
covered here on the mailing list (well, except for the pretty
graphics display of the word list).

Thanks to all for moving spam detection forward so much, and doing
all that tedious experimental testing to find the best settings.

- Alex



From grobinson@transpose.com  Sun Sep 29 22:35:40 2002
From: grobinson@transpose.com (Gary Robinson)
Date: Sun, 29 Sep 2002 14:35:40 -0700
Subject: [Spambayes] RE: Central Limit Theorem??!!     :)
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEJCBHAB.tim.one@comcast.net>
Message-ID: <B9BCC03C.16B76%grobinson@transpose.com>

We know that the choice of extreme words in any given spam is not
independent. An email that has some spammy words is likely to have more of
them, and an email that has some hammy words is likely to have more of them.

Either way, it will pull the email to one side or another of the mean.

SO, I *think* it is very arguable that that is enough to explain the effect
you are observing, and that it is not a problem for our purposes.


--Gary


-- 
Gary Robinson
CEO
Transpose, LLC
grobinson@transpose.com
207-942-3463
http://www.emergentmusic.com
http://radio.weblogs.com/0101454


> From: Tim Peters <tim.one@comcast.net>
> Date: Sun, 29 Sep 2002 15:55:49 -0400
> To: Gary Robinson <grobinson@transpose.com>
> Cc: SpamBayes <spambayes@python.org>, Greg Louis <glouis@dynamicro.on.ca>
> Subject: RE: [Spambayes] RE: Central Limit Theorem??!!     :)
> 
> Something odd I noticed in my log central-limit code:  the z-scores looked
> "too big", but maybe I'm confusing myself.
> 
> I was running with max_discriminators=50.  Of the 45,000 each of ham and
> spam predicted against, 27,094 ham had at least 50 extreme words, and 36,491
> spam had at least 50 extreme words.  In what follows I'm looking solely at
> those, and 50 should be plenty big enough for the theorem to kick in.
> 
> What I *expected* is, that in
> 
>       zham  = (hmean - self.hammean) / sqrt(self.hamvar / n)
>       zspam = (smean - self.spammean) / sqrt(self.spamvar / n)
> 
> where
>   n is the number of extreme words (always 50 here)
>   hmean is the mean of the n message ln(1-p) values
>   smean is the mean of the n message ln(p) values
>   hammean is the ham training population mean ln(1-p)
>   hamvar  is the ham training population ln(1-p) variance
>   spammean is the spam training population mean ln(p)
>   spamvar  is the spam training population ln(p) variance
> 
> that zham would be approximately normally distributed (mean 0, sdev 1) when
> predicting a ham, and zspam similarly when predicting a spam.
> 
> But it's nowhere near that:
> 
> When predicting a ham,
> 
> This % of hams   had abs(zham) <= this
> --------------   ---------------------
> 18.377%           1.0
> 36.525%           2.0
> 53.650%           3.0
> 67.919%           4.0
> 78.301%           5.0
> 85.831%           6.0
> 90.788%           7.0
> 93.762%           8.0
> 95.696%           9.0
> 97.044%          10.0
> 
> and when predicting a spam,
> 
> This % of spams  had abs(zspam) <= this
> ---------------  ----------------------
> 20.337%           1.0
> 44.120%           2.0
> 69.732%           3.0
> 89.559%           4.0
> 92.702%           5.0
> 94.656%           6.0
> 96.191%           7.0
> 97.205%           8.0
> 97.964%           9.0
> 98.539%          10.0
> 
> I *expected* that about 68% of messages would have abs(z) <= 1.0, and about
> 95% <= 2.0, etc.  What I'm seeing is more like a linear relationship!
> 
> So I'm confused, or something's very fishy here.  I've stared at the code
> until my eyes bled, but it still looks like it's doing everything right, and
> I'm still using scaled unbounded integers to compute the population means
> and variances so there's no possibility of roundoff error polluting the
> results.
> 
> Got a clue?  It's very curious that the 68% and 95% I was looking for show
> up at 4 and 8-9 in the ham results, and at 3 and 6-7 in the spam results.
> 


From guido@python.org  Mon Sep 30 00:22:15 2002
From: guido@python.org (Guido van Rossum)
Date: Sun, 29 Sep 2002 19:22:15 -0400
Subject: [Spambayes] Re: On counting words more than once
In-Reply-To: Your message of "Sun, 29 Sep 2002 14:08:05 EDT."
             <LNBBLJKPBEHFEDALKOLCGEINBHAB.tim.one@comcast.net> 
References: <LNBBLJKPBEHFEDALKOLCGEINBHAB.tim.one@comcast.net> 
Message-ID: <200209292322.g8TNMTp12249@pcp02138704pcs.reston01.va.comcast.net>

> Because Neil, Guido and I all reported improvement via counting
> duplicate words (within a message) only once during training, I
> removed the recent option for trying this, and we do this all the
> time now.  The checkin comment is below.  Note that you may need to
> change spam_cutoff!

I ran this comparison on my full corpus, and it made a significant
improvement for the f-p rate, while increasing the f-n by a tiny
fraction.  So I agree it's a win!

false positive percentages
    0.109  0.085  won    -22.02%
    0.036  0.024  won    -33.33%
    0.084  0.060  won    -28.57%
    0.060  0.060  tied          
    0.120  0.084  won    -30.00%
    0.121  0.109  won     -9.92%
    0.037  0.037  tied          
    0.061  0.049  won    -19.67%
    0.048  0.048  tied          
    0.095  0.095  tied          

won   6 times
tied  4 times
lost  0 times

total unique fp went from 64 to 54 won    -15.62%
mean fp % went from 0.0770443274685 to 0.0649864716148 won    -15.65%

false negative percentages
    0.251  0.376  lost   +49.80%
    0.311  0.249  won    -19.94%
    0.128  0.128  tied          
    0.315  0.252  won    -20.00%
    0.278  0.340  lost   +22.30%
    0.314  0.283  won     -9.87%
    0.398  0.459  lost   +15.33%
    0.154  0.154  tied          
    0.190  0.190  tied          
    0.365  0.334  won     -8.49%

won   4 times
tied  3 times
lost  3 times

total unique fn went from 87 to 89 lost    +2.30%
mean fn % went from 0.270497065577 to 0.276634600712 lost    +2.27%

--Guido van Rossum (home page: http://www.python.org/~guido/)

From grobinson@transpose.com  Mon Sep 30 00:38:33 2002
From: grobinson@transpose.com (Gary Robinson)
Date: Sun, 29 Sep 2002 16:38:33 -0700
Subject: [Spambayes] RE: Central Limit Theorem??!!     :)
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEJCBHAB.tim.one@comcast.net>
Message-ID: <B9BCDD09.16B87%grobinson@transpose.com>

I thought it might be worth adding that the reason it doesn't bother me that
the words are not independent is that under the null hypothesis that the
email contains a random assemblage of words rather than a collection of
words that is distinctly hammy or spammy, they ARE independent. Now, it is
the case that there are very few emails that are consistent with the null
hypothesis, because most emails are clearly hammy or spammy, and therefore
have non-independent words. But that's exactly the kind of thing we are
trying to determine by means of the z-scores.

IOW, it's arguable that most emails have extreme z-scores because most are
either distinctly hammy or distinctly spammy. I think that's fine.


--Gary


-- 
Gary Robinson
CEO
Transpose, LLC
grobinson@transpose.com
207-942-3463
http://www.emergentmusic.com
http://radio.weblogs.com/0101454


> From: Tim Peters <tim.one@comcast.net>
> Date: Sun, 29 Sep 2002 15:55:49 -0400
> To: Gary Robinson <grobinson@transpose.com>
> Cc: SpamBayes <spambayes@python.org>, Greg Louis <glouis@dynamicro.on.ca>
> Subject: RE: [Spambayes] RE: Central Limit Theorem??!!     :)
> 
> Something odd I noticed in my log central-limit code:  the z-scores looked
> "too big", but maybe I'm confusing myself.
> 
> I was running with max_discriminators=50.  Of the 45,000 each of ham and
> spam predicted against, 27,094 ham had at least 50 extreme words, and 36,491
> spam had at least 50 extreme words.  In what follows I'm looking solely at
> those, and 50 should be plenty big enough for the theorem to kick in.
> 
> What I *expected* is, that in
> 
>       zham  = (hmean - self.hammean) / sqrt(self.hamvar / n)
>       zspam = (smean - self.spammean) / sqrt(self.spamvar / n)
> 
> where
>   n is the number of extreme words (always 50 here)
>   hmean is the mean of the n message ln(1-p) values
>   smean is the mean of the n message ln(p) values
>   hammean is the ham training population mean ln(1-p)
>   hamvar  is the ham training population ln(1-p) variance
>   spammean is the spam training population mean ln(p)
>   spamvar  is the spam training population ln(p) variance
> 
> that zham would be approximately normally distributed (mean 0, sdev 1) when
> predicting a ham, and zspam similarly when predicting a spam.
> 
> But it's nowhere near that:
> 
> When predicting a ham,
> 
> This % of hams   had abs(zham) <= this
> --------------   ---------------------
> 18.377%           1.0
> 36.525%           2.0
> 53.650%           3.0
> 67.919%           4.0
> 78.301%           5.0
> 85.831%           6.0
> 90.788%           7.0
> 93.762%           8.0
> 95.696%           9.0
> 97.044%          10.0
> 
> and when predicting a spam,
> 
> This % of spams  had abs(zspam) <= this
> ---------------  ----------------------
> 20.337%           1.0
> 44.120%           2.0
> 69.732%           3.0
> 89.559%           4.0
> 92.702%           5.0
> 94.656%           6.0
> 96.191%           7.0
> 97.205%           8.0
> 97.964%           9.0
> 98.539%          10.0
> 
> I *expected* that about 68% of messages would have abs(z) <= 1.0, and about
> 95% <= 2.0, etc.  What I'm seeing is more like a linear relationship!
> 
> So I'm confused, or something's very fishy here.  I've stared at the code
> until my eyes bled, but it still looks like it's doing everything right, and
> I'm still using scaled unbounded integers to compute the population means
> and variances so there's no possibility of roundoff error polluting the
> results.
> 
> Got a clue?  It's very curious that the 68% and 95% I was looking for show
> up at 4 and 8-9 in the ham results, and at 3 and 6-7 in the spam results.
> 


From tim.one@comcast.net  Mon Sep 30 01:15:20 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 29 Sep 2002 20:15:20 -0400
Subject: [Spambayes] RE: Central Limit Theorem??!!     :)
In-Reply-To: <B9BCC03C.16B76%grobinson@transpose.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEKBBHAB.tim.one@comcast.net>

[Tim, agonizes over that his logarithmic z-scores aren't normal]

[Gary Robinson]
> We know that the choice of extreme words in any given spam is not
> independent. An email that has some spammy words is likely to have
> more of them, and an email that has some hammy words is likely to
> have more of them.

IOW, the sample I'm drawing isn't random, so the clt doesn't really apply.
I can buy that.  Indeed, after the next point, I have to <wink>:

> Either way, it will pull the email to one side or another of the mean.

Turns out this isn't symmetric either.  Still restricted to the msgs with at
least 50 extreme words, only 10.5% of predicted hams had a positive zham,
but 61.3% of predicted spams had a positive zspam.  I guess that says it's
very much easier to be "very spammy" than it is to be "very hammy", and
that's consistent with other clues that we've seen.

If I look at all msgs (regardless of how many words they contain) from this
run, 9.1% of predicted hams had positive zham, and 51.5% of predicted spam
had positive zspam.

More-- and this may be useful <wink> --whenever the z-score with the smaller
magnitude was positive, the prediction was always correct.

> SO, I *think* it is very arguable that that is enough to explain
> the effect you are observing, and that it is not a problem for our
> purposes.

Well, it blows all hell out of the notion that z-scores can be converted to
probabilities in an obvious way.  It leaves us with "two numbers".  We've
been in worse spots than that <wink>.

I changed my best heuristic cheap stab at guessing certainity to:

        certain = False
        if abs(zham) < abs(zspam):
            if zham > 0:
                certain = True
            else:
                ratio = zspam / zham
        else:
            if zspam > 0:
                certain = True
            else:
                ratio = zham / zspam

        if not certain:
            ratio = abs(ratio)
            certain = (ratio > 3.0 or
                       (n > 30 and ratio > 2.0) or
                       (n > 40 and ratio > 1.75))

It turns out that gave exactly the same results as before I noticed the
"whenever the z-score with the smaller magnitude was positive, the
prediction was always correct" bit:

for all ham
    45000 total
    certain    44731 99.402%
        wrong      0  0.000%
    unsure       269  0.598%
        wrong     37 13.755%

for all spam
    45000 total
    certain    44258 98.351%
        wrong      0  0.000%
    unsure       742  1.649%
        wrong     84 11.321%

The first cutoff can be reduced from 3 to 2.6 without making an error on "a
certain" one:

for all ham
    45000 total
    certain    44764 99.476%
        wrong      0  0.000%
    unsure       236  0.524%
        wrong     37 15.678%

for all spam
    45000 total
    certain    44371 98.602%
        wrong      0  0.000%
    unsure       629  1.398%
        wrong     84 13.355%

but if reduced to 2.5 it starts to screw up:

for all ham
    45000 total
    certain    44774 99.498%
        wrong      0  0.000%
    unsure       226  0.502%
        wrong     37 16.372%

for all spam
    45000 total
    certain    44395 98.656%
        wrong      1  0.002%
    unsure       605  1.344%
        wrong     83 13.719%

In the "certain but wrong" case, the heuristic was certain a spam was ham at
cutoff 2.5:

    n = 21
    zham  =  -6.73
    zspam = -17.12

It's hard to generalize from one example, though <wink>.


From grobinson@transpose.com  Mon Sep 30 01:27:02 2002
From: grobinson@transpose.com (Gary Robinson)
Date: Sun, 29 Sep 2002 17:27:02 -0700
Subject: [Spambayes] RE: Central Limit Theorem??!!     :)
In-Reply-To: <LNBBLJKPBEHFEDALKOLCCEKBBHAB.tim.one@comcast.net>
Message-ID: <B9BCE866.16B95%grobinson@transpose.com>

> Well, it blows all hell out of the notion that z-scores can be converted to
> probabilities in an obvious way.  It leaves us with "two numbers".  We've
> been in worse spots than that <wink>.


I *think* that depends on what probability you're talking about. That is, if
the null hypothesis is that a given email is a random collection of words,
then I *think* it DOES correspond to the probability that such an extreme
random collection of words (extreme in the spammy or hammy direction), or a
more extreme one, would have happened by chance alone.

So in that sense, it seems to me that it probably can be useful as a
rigorous probability, if we remain conscious of what null hypothesis we are
operating against. 


--Gary


-- 
Gary Robinson
CEO
Transpose, LLC
grobinson@transpose.com
207-942-3463
http://www.emergentmusic.com
http://radio.weblogs.com/0101454


> From: Tim Peters <tim.one@comcast.net>
> Date: Sun, 29 Sep 2002 20:15:20 -0400
> To: Gary Robinson <grobinson@transpose.com>
> Cc: SpamBayes <spambayes@python.org>, Greg Louis <glouis@dynamicro.on.ca>
> Subject: RE: [Spambayes] RE: Central Limit Theorem??!!     :)
> 
> [Tim, agonizes over that his logarithmic z-scores aren't normal]
> 
> [Gary Robinson]
>> We know that the choice of extreme words in any given spam is not
>> independent. An email that has some spammy words is likely to have
>> more of them, and an email that has some hammy words is likely to
>> have more of them.
> 
> IOW, the sample I'm drawing isn't random, so the clt doesn't really apply.
> I can buy that.  Indeed, after the next point, I have to <wink>:
> 
>> Either way, it will pull the email to one side or another of the mean.
> 
> Turns out this isn't symmetric either.  Still restricted to the msgs with at
> least 50 extreme words, only 10.5% of predicted hams had a positive zham,
> but 61.3% of predicted spams had a positive zspam.  I guess that says it's
> very much easier to be "very spammy" than it is to be "very hammy", and
> that's consistent with other clues that we've seen.
> 
> If I look at all msgs (regardless of how many words they contain) from this
> run, 9.1% of predicted hams had positive zham, and 51.5% of predicted spam
> had positive zspam.
> 
> More-- and this may be useful <wink> --whenever the z-score with the smaller
> magnitude was positive, the prediction was always correct.
> 
>> SO, I *think* it is very arguable that that is enough to explain
>> the effect you are observing, and that it is not a problem for our
>> purposes.
> 
> Well, it blows all hell out of the notion that z-scores can be converted to
> probabilities in an obvious way.  It leaves us with "two numbers".  We've
> been in worse spots than that <wink>.
> 
> I changed my best heuristic cheap stab at guessing certainity to:
> 
>       certain = False
>       if abs(zham) < abs(zspam):
>           if zham > 0:
>               certain = True
>           else:
>               ratio = zspam / zham
>       else:
>           if zspam > 0:
>               certain = True
>           else:
>               ratio = zham / zspam
> 
>       if not certain:
>           ratio = abs(ratio)
>           certain = (ratio > 3.0 or
>                      (n > 30 and ratio > 2.0) or
>                      (n > 40 and ratio > 1.75))
> 
> It turns out that gave exactly the same results as before I noticed the
> "whenever the z-score with the smaller magnitude was positive, the
> prediction was always correct" bit:
> 
> for all ham
>   45000 total
>   certain    44731 99.402%
>       wrong      0  0.000%
>   unsure       269  0.598%
>       wrong     37 13.755%
> 
> for all spam
>   45000 total
>   certain    44258 98.351%
>       wrong      0  0.000%
>   unsure       742  1.649%
>       wrong     84 11.321%
> 
> The first cutoff can be reduced from 3 to 2.6 without making an error on "a
> certain" one:
> 
> for all ham
>   45000 total
>   certain    44764 99.476%
>       wrong      0  0.000%
>   unsure       236  0.524%
>       wrong     37 15.678%
> 
> for all spam
>   45000 total
>   certain    44371 98.602%
>       wrong      0  0.000%
>   unsure       629  1.398%
>       wrong     84 13.355%
> 
> but if reduced to 2.5 it starts to screw up:
> 
> for all ham
>   45000 total
>   certain    44774 99.498%
>       wrong      0  0.000%
>   unsure       226  0.502%
>       wrong     37 16.372%
> 
> for all spam
>   45000 total
>   certain    44395 98.656%
>       wrong      1  0.002%
>   unsure       605  1.344%
>       wrong     83 13.719%
> 
> In the "certain but wrong" case, the heuristic was certain a spam was ham at
> cutoff 2.5:
> 
>   n = 21
>   zham  =  -6.73
>   zspam = -17.12
> 
> It's hard to generalize from one example, though <wink>.
> 


From mhammond@skippinet.com.au  Mon Sep 30 01:52:25 2002
From: mhammond@skippinet.com.au (Mark Hammond)
Date: Mon, 30 Sep 2002 10:52:25 +1000
Subject: [Spambayes] Practical applications 
In-Reply-To: <LCEPIIGDJPKCOIHOBJEPCEDPGKAA.mhammond@skippinet.com.au>
Message-ID: <LCEPIIGDJPKCOIHOBJEPOENNGKAA.mhammond@skippinet.com.au>

> > Something similar came up on Python-list, so I followed up with Mark.
> > Here's a summary:
>
> I followed up too :)
>
> I have a working Outlook Addin in place.  This used the generic "Office
> Plugin" architecture, so should work with all of Office 2000 (but
> outlook is
> the target).  I assume it will work with Outlook 2002/XP, I'm not sure.  I
> doubt it will work with Office 98, but again I am not sure.  It will not
> work with Outlook Express.
>
> There are some bugs/incompleteness in the new universal gateway code I am
> fixing, so this will rely on yet-to-be-released win32all versions.
>
> I'll keep you updated.

I have a version of the Outlook Plugin working.  At the moment, all it does
is add a "Python" button to the main toolbar, which displays a message box.

Unfortunately, I wont be getting a new win32all out for a week or so - so if
anyone builds win32all from the CVS tree, then please pull and build the
latest version, and ask me for the current outlook plugin code.

Mark.


From tim.one@comcast.net  Mon Sep 30 01:55:09 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 29 Sep 2002 20:55:09 -0400
Subject: [Spambayes] RE: Central Limit Theorem??!!     :)
In-Reply-To: <B9BCE866.16B95%grobinson@transpose.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCIEKDBHAB.tim.one@comcast.net>

[Tim]
> Well, it blows all hell out of the notion that z-scores can be
> converted to probabilities in an obvious way.  It leaves us with "two
> numbers".  We've been in worse spots than that <wink>.

[Gary Robinson]
> I *think* that depends on what probability you're talking about.

When I believed that zham followed a unit normal distribution, then, e.g.,
if I saw a msg with zham=1, I could say

    P(msg is ham | abs(zham) >= 1) ~= 0.32

That's an example of "convert[ing] to probabilities in an obvious way".  I
meant it in a quantitative sense, not a qualitative one.

> That is, if the null hypothesis is that a given email is a random
> collection of words, then I *think* it DOES correspond to the
> probability that such an extreme random collection of words
> (extreme in the spammy or hammy direction), or a more extreme
> one, would have happened by chance alone.

The test results I posted do show that it knows an awful lot about whether a
thing is ham or spam, and also knows an awful lot about how certain it is.
Huge steaming mounds <wink> of useful information is clearly there, but I'm
not sure how to go about extracting and quantifying it -- computing
"pseudo-z-score ratios" and picking cutoffs out of thin air was just
heuristic hackery, but a better approach than that hasn't revealed itself.


From tim.one@comcast.net  Mon Sep 30 02:57:18 2002
From: tim.one@comcast.net (Tim Peters)
Date: Sun, 29 Sep 2002 21:57:18 -0400
Subject: [Spambayes] Purging Old Words - Date vs Sequence
In-Reply-To: <19432086158-BeMail@CR593174-A>
Message-ID: <LNBBLJKPBEHFEDALKOLCGEKHBHAB.tim.one@comcast.net>

[Alexander G. M. Smith]
> While implementing a BeOS version of Paul Graham's spam detection
> algorithm (available at http://www.bebits.com/app/3055 - I'll be
> switching to Gary Robinson's algorithm soon),

I hear that Python runs on BeOS too -- feel free to use our engine.  Feel
even freer to contribute great ideas to it <wink>.

> I had a need to purge old words from the database.  More of a need
> than usual since I'm simplisticly considering the whole message and
> breaking it into simple words, even binary attachments.  Then
> deleting the unused binary garbage after a while.  I suppose that
> technique could even find spam encoded as pictures.

Probably not.  Compression is part of the gif and jpeg schemes, and the
better the compression the more the bytes act random (this is why your
favorite compression program saves very little when compressing these
guys -- it can't find structure to exploit).  There may be some value in
mining the first few bytes to capture "magic numbers", although I expect we
get much the same thing by generating tokens for the MIME decorations saying
which kind of binary gibberish a thing is, and the intended file name if
it's an attachment.

> I thought of using a date stamp, like the spambayes project does,
> but that could unevenly remove messages since they aren't added
> at uniform times.

Heh.  AFAIK, nobody has even *tried* calling our clearjunk() method yet.
That's all up in the air, and I'm hoping somebody else will be its savior.
Note, though, that the timestamps are updated by our engine during
*prediction*:  it records the last time a word was accessed.  That's equal
to the creation time if and only if a word has never been seen again in a
message scored.  Provided the prediction part is run routinely, then, the
words actually being used by prediction will have current timestamps.
Consuming an 8-byte float per word for this purpose is absurd, though.

> Instead, I assign a sequentially increasing serial number to each
> example of spam or ham, and store that along with the words and
> frequency counts.  If the word is in a later example message, the
> serial number of the new example replaces the old one for that word.
> Then I can purge words which appeared N messages before the latest
> one (usually in combination with having a low frequency count).

All expiration schemes need testing, because they're cheating:  word counts
are added on a per-msg basis.  Expiration that doesn't also work on a
per-msg basis is thus a distortion of real life.  Whether that's harmful in
this scheme isn't known.  For example, msg 1 contains words A and B.  They
each get a count of 1.  Msg 2 contains words B and C.  Now we have

       count  serial
    A    1       1
    B    2       2
    C    1       2

Now we decide message #1 is stale, but the only word still having serial 1
is A.  This leaves us with B=2 and C=1, which doesn't match any previous
real state.  It isn't obvious to me that this won't destroy performance over
time; but it isn't obvious to me that it will hurt it, either.  I find it
easier to test such things than think about them, though <wink>.

> I suppose you could even factor in the kill count or have a last used
> to kill spam date/serial number too.

Our killcount was put in to help research more than anything else.  It
turned out to be *great* for identifying changes that gave great results for
bad reasons (e.g., your ham goes back 5 years, but all your spam was
collected in the last month -- then clues about dates, or even the ISP you
were using at various times, become extremely strong and frequent
discriminators, but for bad reasons -- and their high killcounts make this
obvious).

> Anyway, that's the only novel idea I've had on the topic,

Expiration will be very important over time.  Keep thinking!  Even if you
don't tokenize huge masses of random binary gibberish, it's common in all
computer indexing applications that databases grow without bounds (typos,
message ids, random bits of fly-by-night URLs, ..., they pile up endlessly).

> everything else I've thought of has been covered here on the mailing
> list (well, except for the pretty graphics display of the word list).
>
> Thanks to all for moving spam detection forward so much, and doing
> all that tedious experimental testing to find the best settings.

The best way to thank the folks here is to contribute your own tedious
experimental testing back to us -- we don't have enough of that, and there's
no end in sight for things that should be tested.  Pull up a comfy chair and
get obsessed <wink>.


From jcarlson@uci.edu  Mon Sep 30 07:40:35 2002
From: jcarlson@uci.edu (Josiah Carlson)
Date: Sun, 29 Sep 2002 23:40:35 -0700
Subject: [Spambayes] Another use for the spam corpus...
Message-ID: <20020929232955.CF66.JCARLSON@uci.edu>

Hey,

I thought you would think this humorous.  I've been trying to get a
web-mail application running to remotely access my email without having
to use vnc.

I'm also using ssl...for that added measure of security (and the
software requires it).  As I was generating a key...it said 'warning,
not much extra random data, consider using the -rand option'

Hrm...Why don't I put those 700 pieces of spam email to use?  Use it as
semi-random input to the ssl key generator...  *grin*

It's not really semi-random, but it's better than nothing.

 - Josiah

From anthony@interlink.com.au  Mon Sep 30 09:33:07 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Mon, 30 Sep 2002 18:33:07 +1000
Subject: [Spambayes] varying proportion of spam:ham testing.
Message-ID: <200209300833.g8U8X8l05310@localhost.localdomain>

Ok, as promised last week, here's the results for me of
varying the proportion of spam:ham in my personal spam/ham
corpus (note that this is not the monster corpus, but a new
one from my personal email).

It's currently set up as a 4-way split, there's 210 spam 
and 2500 ham in each Set. 

I ran timcv with --spam=200 in each, and 
--ham=200
--ham=400
--ham=1000
--ham=2000

Standard settings, aside from
spam_cutoff: 0.58
mine_received_headers: True

count_all_header_lines, as reported last week, makes the results
considerably worse, so is left off.


==> ant_1:1s.txt <==
-> <stat> Ham scores for all runs: 800 items; mean 24.58; sdev 10.16
-> <stat> Spam scores for all runs: 800 items; mean 80.11; sdev 7.99
-> best cutoff for all runs: 0.575
->     with weighted total 1*3 fp + 7 fn = 10
->     fp rate 0.375%  fn rate 0.875%
total unique false pos 3
total unique false neg 8
average fp % 0.375
average fn % 1.0

==> ant_2:1s.txt <==
-> <stat> Ham scores for all runs: 1600 items; mean 23.18; sdev 9.17
-> <stat> Spam scores for all runs: 800 items; mean 79.29; sdev 8.58
-> best cutoff for all runs: 0.55
->     with weighted total 1*2 fp + 8 fn = 10
->     fp rate 0.125%  fn rate 1%
total unique false pos 1
total unique false neg 15
average fp % 0.0625
average fn % 1.875

==> ant_5:1s.txt <==
-> <stat> Ham scores for all runs: 4000 items; mean 21.02; sdev 8.44
-> <stat> Spam scores for all runs: 800 items; mean 77.80; sdev 9.45
-> best cutoff for all runs: 0.525
->     with weighted total 1*8 fp + 10 fn = 18
->     fp rate 0.2%  fn rate 1.25%
total unique false pos 2
total unique false neg 23
average fp % 0.05
average fn % 2.875

==> ant_10:1s.txt <==
-> <stat> Ham scores for all runs: 8000 items; mean 19.46; sdev 7.89
-> <stat> Spam scores for all runs: 800 items; mean 76.09; sdev 10.45
-> best cutoff for all runs: 0.5
->     with weighted total 1*8 fp + 11 fn = 19
->     fp rate 0.1%  fn rate 1.38%
total unique false pos 2
total unique false neg 53
average fp % 0.025
average fn % 6.625

So increasing the proportion of spam:ham drags down the ham mean and
std dev, but also drags down the spam mean and stddev.

Anthony

From rob@hooft.net  Mon Sep 30 09:49:42 2002
From: rob@hooft.net (Rob Hooft)
Date: Mon, 30 Sep 2002 10:49:42 +0200
Subject: [Spambayes] Central limit
Message-ID: <3D981023.5070900@hooft.net>

While my computer is eating away cycles running timcv on my 24000 ham/ 
5750 spam messages, a few remarks on the Central Limit Theorem:

  - The standard deviations seem "underestimated". Gary already said this
    can be caused by correlations between scores. Alternatively this can
    indicate that the data is not 1D: in more than one dimension, a
    higher percentage of normally distributed data lies outside of the
    "core regions". Anyway, something can be done about this: just
    calculate the RMS Z-score, and scale it to 1.0. After applying that
    scale the normal 68%/95% rule should apply. One would hope.

  - The "certainty" rule of Tim should be formalized. This is a very
    powerful concept not only for those people that believe in a middle
    ground, but also for those that want to use a pre-trained spambayes
    system for other corpora. Once this has crystallized a bit more we
    should exchange pickles and see how well we can do with each-others
    training data!

  - It should somehow be possible to classify messages into any sumber of
    distinct groups using this trick. A new message can get scored a
    Z-score to describe the likelyhood that it is part of any of the
    groups, if all of these numbers are large, the test message does not
    belong to any class. I guess, e.g. that it should not be too
    difficult for the bayesian algorithms used here to judge whether
    E-mail I receive is for "work", "private" or "spam". What would take
    this really to the next generation would be an algorithm that can
    make the classification "ab initio" as a sort of clustering
    algorithm: e.g. something that would start with two of the most
    different messages in a single corpus, and add single messages to
    either of the two groups until it finds a message that has 2 large
    Z-scores. Then it starts a third group.

Just dreaming.

Rob

-- 
Rob W.W. Hooft  ||  rob@hooft.net  ||  http://www.hooft.net/people/rob/


From msergeant@startechgroup.co.uk  Mon Sep 30 13:49:46 2002
From: msergeant@startechgroup.co.uk (Matt Sergeant)
Date: Mon, 30 Sep 2002 13:49:46 +0100
Subject: [Spambayes] Matt Sergeant: Introduction
Message-ID: <3D98486A.1050208@startechgroup.co.uk>

Just wanted to stick my hand up and say "Hi". I've been following this 
list on gmane.org for a while now (it's a mail to nntp gateway for those 
interested in following multiple technical mailing lists in a read-only 
fashion), but decided it was time to bite the bullet and subscribe, 
mostly because I keep itching to reply to certain posts :-)

First of all, I'm a perl guy (boo, hiss). Some of you may or may have 
heard of me from the perl community (I won the ActiveState awards for 
Perl and XSLT this year, though I didn't really deserve the XSLT one 
<grin>). I'm also actively involved in the SpamAssassin project. By day 
I work as an anti-spam technologist (one of the lucky few to actually 
work day-in day-out to fight this nuisance) for MessageLabs. I was doing 
Bayesian probability techniques for spam detection before Paul Graham 
published his article and set the world on fire. Like you all, I 
discovered very quickly that it's the tokenisation techniques that are 
the biggest "win" when it comes down to it.

To answer the curious, yes we're going to add some sort of bayesian 
technique to spamassassin - one of our developers has written code 
independantly from my stuff at work (because I'm contract bound not to 
give that away) that uses SpamAssassin in a "training" mode before 
switching on to full scoring mode. It'll basically work much like the 
other SA rules, where the probability gives a score. If we go with 
Graham it'll likely be boolean, if we go with Robinson we can give a 
gradual score range. But that's mostly out of my hands (due to day-job 
conflicts).

Anyway, just wanted to say "Hi", and to let you know that I have 
converted the PG (final) code to Perl, and it worked well, and I've done 
the Robinson stuff without the central limit theorem and it didn't work 
quite as well, so I'm hopefully going to get CLT done this week and see 
how it fares. Unfortunately I find python incredibly difficult to read, 
so it takes me a while!

Oh, and I'll also be talking at Paul Graham's spam conference about 
doing spam detection at the internet level (at messagelabs you point 
your MX to us, and we "clean" your email then forward it on), and the 
issues that gives rise to, such as how the probability stuff works so 
much better on individuals' corpora (or on a particular mailing list's 
corpus) than it does for hundreds of thousands of users.

Have fun,
Matt.


From anthony@interlink.com.au  Mon Sep 30 14:58:54 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Mon, 30 Sep 2002 23:58:54 +1000
Subject: [Spambayes] Central limit 
In-Reply-To: <3D981023.5070900@hooft.net> 
Message-ID: <200209301358.g8UDwsx03055@localhost.localdomain>


>>> Rob Hooft wrote
>   - It should somehow be possible to classify messages into any sumber of
>     distinct groups using this trick. A new message can get scored a
>     Z-score to describe the likelyhood that it is part of any of the
>     groups, if all of these numbers are large, the test message does not
>     belong to any class. I guess, e.g. that it should not be too
>     difficult for the bayesian algorithms used here to judge whether
>     E-mail I receive is for "work", "private" or "spam". What would take
>     this really to the next generation would be an algorithm that can
>     make the classification "ab initio" as a sort of clustering
>     algorithm: e.g. something that would start with two of the most
>     different messages in a single corpus, and add single messages to
>     either of the two groups until it finds a message that has 2 large
>     Z-scores. Then it starts a third group.

Interestingly, I was thinking about doing something like this for 
attempting to guess the language of text - I was planning on trying 
to use letter trigrams. I have a need for something like this in the 
future - text to speech engines, even if they can do non-english,
require that you tell it the language it's reading (you should hear
what the american english dictionary does to german - it's not at 
all pretty).

Anthony
-- 
Anthony Baxter     <anthony@interlink.com.au>   
It's never too late to have a happy childhood.


From msergeant@startechgroup.co.uk  Mon Sep 30 15:08:04 2002
From: msergeant@startechgroup.co.uk (Matt Sergeant)
Date: Mon, 30 Sep 2002 15:08:04 +0100
Subject: [Spambayes] Central limit
References: <200209301358.g8UDwsx03055@localhost.localdomain>
Message-ID: <3D985AC4.1050901@startechgroup.co.uk>

Anthony Baxter wrote:
>>>>Rob Hooft wrote
>>>
>>  - It should somehow be possible to classify messages into any sumber of
>>    distinct groups using this trick. A new message can get scored a
>>    Z-score to describe the likelyhood that it is part of any of the
>>    groups, if all of these numbers are large, the test message does not
>>    belong to any class. I guess, e.g. that it should not be too
>>    difficult for the bayesian algorithms used here to judge whether
>>    E-mail I receive is for "work", "private" or "spam". What would take
>>    this really to the next generation would be an algorithm that can
>>    make the classification "ab initio" as a sort of clustering
>>    algorithm: e.g. something that would start with two of the most
>>    different messages in a single corpus, and add single messages to
>>    either of the two groups until it finds a message that has 2 large
>>    Z-scores. Then it starts a third group.
> 
> 
> Interestingly, I was thinking about doing something like this for 
> attempting to guess the language of text - I was planning on trying 
> to use letter trigrams. I have a need for something like this in the 
> future - text to speech engines, even if they can do non-english,
> require that you tell it the language it's reading (you should hear
> what the american english dictionary does to german - it's not at 
> all pretty).

SpamAssassin uses TextCat. It's *very* effective at guessing languages.

   http://odur.let.rug.nl/~vannoord/TextCat/

Matt.


From guido@python.org  Mon Sep 30 15:13:37 2002
From: guido@python.org (Guido van Rossum)
Date: Mon, 30 Sep 2002 10:13:37 -0400
Subject: [Spambayes] Matt Sergeant: Introduction
In-Reply-To: Your message of "Mon, 30 Sep 2002 13:49:46 BST."
             <3D98486A.1050208@startechgroup.co.uk> 
References: <3D98486A.1050208@startechgroup.co.uk> 
Message-ID: <200209301413.g8UEDbE14786@pcp02138704pcs.reston01.va.comcast.net>

Hi Matt.  Nice to hear you here.

> (at messagelabs you point your MX to us, and we "clean" your email
> then forward it on)

Do you actually delete spam or just mark it?  If you delete it, how do
you deal with false positives?

--Guido van Rossum (home page: http://www.python.org/~guido/)

From msergeant@startechgroup.co.uk  Mon Sep 30 15:28:37 2002
From: msergeant@startechgroup.co.uk (Matt Sergeant)
Date: Mon, 30 Sep 2002 15:28:37 +0100
Subject: [Spambayes] Matt Sergeant: Introduction
References: <3D98486A.1050208@startechgroup.co.uk>
	<200209301413.g8UEDbE14786@pcp02138704pcs.reston01.va.comcast.net>
Message-ID: <3D985F95.8010706@startechgroup.co.uk>

Guido van Rossum wrote:
> Hi Matt.  Nice to hear you here.
> 
> 
>>(at messagelabs you point your MX to us, and we "clean" your email
>>then forward it on)
> 
> 
> Do you actually delete spam or just mark it?  If you delete it, how do
> you deal with false positives?

Customer configurable with two thresholds. i.e. at one threshold forward 
to an admin (or tag with a header) and at another threshold delete.

We deal with FP's as seriously as possible - we'll globally whitelist 
anyone who is consistently sending out spammy subscriber newsletters, 
and we consistently make things better as we go along (like by feeding 
FP's back into the classifier multiple times <grin>). Other than that 
there's not a lot we can do - it gets lost in the bit bucket (we've 
thought about quarantining, but the disk space required to do that even 
for 24 hours would be just insane - approximately 25 Gig of storage per 
day). It's a lot easier for viruses, where we just quarantine everything 
for 30 days ;-)

Matt.



From agmsmith@rogers.com  Mon Sep 30 17:20:58 2002
From: agmsmith@rogers.com (Alexander G. M. Smith)
Date: Mon, 30 Sep 2002 12:20:58 EDT (-0400)
Subject: [Spambayes] Purging Old Words - Date vs Sequence
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEKHBHAB.tim.one@comcast.net>
Message-ID: <3600366498-BeMail@CR593174-A>

Tim Peters wrote:
> I hear that Python runs on BeOS too -- feel free to use our engine.  Feel
> even freer to contribute great ideas to it <wink>.

Yes, I know some people who do amazing things with Python, such as
very quickly adding audible reading to a chat program, and then adding
a thesaurus to convert words (from or to swear words).  However, after
spending a month on it (after thinking it would only be a few days), 95%
of the code is GUI, scripting support and other infrastructure, so it's
not worth doing in Python, unless very complex algorithms start being
developed.  Plus it was a good excuse to finally learn about the C++
Standard Template Library.

> Probably not.  Compression is part of the gif and jpeg schemes, and the
> better the compression the more the bytes act random (this is why your
> favorite compression program saves very little when compressing these
> guys -- it can't find structure to exploit).  There may be some value in
> mining the first few bytes to capture "magic numbers", although I expect we
> get much the same thing by generating tokens for the MIME decorations saying
> which kind of binary gibberish a thing is, and the intended file name if
> it's an attachment.

The file size would also be a big clue to identifying a particular image,
if it appears in multiple spam.  Guess I don't really need to tokenize
everything (I plan to break the message into MIME components (there's
already code library somewhere for that) and only tokenize the text/*
ones, possibly also rendering HTML to text).

> Heh.  AFAIK, nobody has even *tried* calling our clearjunk() method yet.
> That's all up in the air, and I'm hoping somebody else will be its savior.
> Note, though, that the timestamps are updated by our engine during
> *prediction*:  it records the last time a word was accessed.  That's equal
> to the creation time if and only if a word has never been seen again in a
> message scored.  Provided the prediction part is run routinely, then, the
> words actually being used by prediction will have current timestamps.
> Consuming an 8-byte float per word for this purpose is absurd, though.

I'll leave a kill count and date out for now, otherwise every time the
system checks for mail, it will write out a new database file.  Though
for experimental use, it's nice to have.

> All expiration schemes need testing, because they're cheating:  word counts
> are added on a per-msg basis.  Expiration that doesn't also work on a
> per-msg basis is thus a distortion of real life.  Whether that's harmful in
> this scheme isn't known.  For example, msg 1 contains words A and B.  They
> each get a count of 1.  Msg 2 contains words B and C.  Now we have
> 
>        count  serial
>     A    1       1
>     B    2       2
>     C    1       2
> 
> Now we decide message #1 is stale, but the only word still having serial 1
> is A.  This leaves us with B=3D2 and C=3D1, which doesn't match any previous
> real state.  It isn't obvious to me that this won't destroy performance over
> time; but it isn't obvious to me that it will hurt it, either.  I find it
> easier to test such things than think about them, though <wink>.

Good point.  To do it correctly would require keeping around all the
original example messages (maybe not that big a deal=3F).  But for
removing binary garbage words, it's great!

> The best way to thank the folks here is to contribute your own tedious
> experimental testing back to us -- we don't have enough of that, and there's
> no end in sight for things that should be tested.  Pull up a comfy chair and
> get obsessed <wink>.

Ugh.  I've already done a bit of that, though without as many spam
messages as people seem to have.  My word list is saved in a tab
delimited format (header info on total messages plus for each word:
word, age, genuine count, spam count) which is easily loaded up into a
spreadsheet.  It's interesting to chart the number of words at each
frequency count, and hunt around for other things.  But then you find out
that your spreadsheet is limited to 16K rows :-).

- Alex



From tim.one@comcast.net  Mon Sep 30 17:38:30 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 30 Sep 2002 12:38:30 -0400
Subject: [Spambayes] Central limit
In-Reply-To: <3D981023.5070900@hooft.net>
Message-ID: <LNBBLJKPBEHFEDALKOLCCEMPBHAB.tim.one@comcast.net>

[Rob Hooft]
>   - The standard deviations seem "underestimated". Gary already said
>     this can be caused by correlations between scores. Alternatively
>     this can indicate that the data is not 1D: in more than one
>     dimension, a higher percentage of normally distributed data lies
>     outside of the "core regions". Anyway, something can be done about
>     this: just calculate the RMS Z-score, and scale it to 1.0.

Sorry, I don't know what that means or how to compute it; neither does
google <wink>.  Let's say this is my population:  {2, 5, 10, 64}.  Then what
are the "RMS Z-score scaled to 1.0" thingies of 1, 2, 32, 64, and 1000?

> ...
>   - The "certainty" rule of Tim should be formalized.

Sure, but how?   I made up a combination of "look at ratios" and "different
cutoffs for different n" by iteratively staring at the errors and making
stuff up.  Even then all I get is a binary "certain or uncertain?" decision
out of it, and without a clear connection to quantifiable probabilities I
don't have strong reason to believe it's a sensible approach in general.

An alternative I haven't tried:  Consider the populations to be the set of
all ham-scores and spam-scores of msgs as a whole (rather than as funky
collections of individual "extreme word" probabilities).  I expect that has
a much better shot at being normally distributed; I don't know whether the
ham and spam populations so obtained would be separated well enough to be
useful, though.

> ...
>     Once this has crystallized a bit more we should exchange pickles
>     and see how well we can do with each-others training data!

One thing I noted before:  when training on more data than what I reported
on most recently, the log-central-limit population ham and spam means and
variances appeared remarkably insensitive to which random subset of msgs I
took to be "the population".  That was a good sign.  If we can figure out
what to *do* with these things <wink>, it's easy for me to whip up a little
program that will just compute these stats and display them; then we can
directly find out how sensitive they are cross-corpus for the people playing
along here.

>   - It should somehow be possible to classify messages into any
>     sumber of distinct groups using this trick. A new message can
>     get scored a Z-score to describe the likelyhood that it is part
>     of any of the groups, if all of these numbers are large, the test
>     message does not  belong to any class. I guess, e.g. that it
>     should not be too  difficult for the bayesian algorithms used
>     here to judge whether E-mail I receive is for "work", "private"
>     or "spam".

I don't expect this to work.  N-way classsification is a natural task for a
Bayesian classifier, but the only thing Bayesian about the approach here is
Gary's adjustment to the spamprobs as computed by counting.  This is
discussed in detail on Gary's web page, including a link to the tortured
reasoning it takes to find anything Bayesian in Paul's formulation.

WRT Gary's clt ideas, they seem to get huge bang for the buck out of that
we're making a binary decision, so that the two statistics of interest for
each word are ln(spamprob) and ln(1-spamprob).  This makes a very sharp
distinction for words with low or high spamprob; it's unclear how to split
this into an N-way distinction, or that doing so would be useful.  I suppose
that for each category C, you could compute ln(Cprob) and ln(1-Cprob).  Then
I wave my hands, and it all works great <wink>.


From jcarlson@uci.edu  Mon Sep 30 18:12:18 2002
From: jcarlson@uci.edu (Josiah Carlson)
Date: Mon, 30 Sep 2002 10:12:18 -0700
Subject: [Spambayes] Central limit
In-Reply-To: <3D981023.5070900@hooft.net>
References: <3D981023.5070900@hooft.net>
Message-ID: <20020930093458.DCA3.JCARLSON@uci.edu>

Hey people...I've been thinking.

It could be that the desire to classify email into only two distinct
categories is a less than tractable problem.  The methods currently in
use do pretty well for the majority of cases, but are not perfect as are
shown by fn and fp results.

Then Rob Hooft sent the email with this:

>   - It should somehow be possible to classify messages into any sumber of
>     distinct groups using this trick. A new message can get scored a
>     Z-score to describe the likelyhood that it is part of any of the
>     groups, if all of these numbers are large, the test message does not
>     belong to any class. I guess, e.g. that it should not be too
>     difficult for the bayesian algorithms used here to judge whether
>     E-mail I receive is for "work", "private" or "spam". What would take
>     this really to the next generation would be an algorithm that can
>     make the classification "ab initio" as a sort of clustering
>     algorithm: e.g. something that would start with two of the most
>     different messages in a single corpus, and add single messages to
>     either of the two groups until it finds a message that has 2 large
>     Z-scores. Then it starts a third group.

I've heard that popfile does something like that, where you can have as
many categories as you want.  I do not know how they do it
however...maybe Matt Sergeant (the new perl guy) can check it out and
tell us what's going on.

It is entirely possible that by categorizing our email into only two
distinct categories, spam and ham, that we are asking the statistics to
generalize too much.  Imagine if instead of spam and ham, you had spam
and spambayes.  Wouldn't you think it would do much better in
classifying incoming email into one of those two?  Or even a mother and
girlfriend (assuming all email is either from your mother or girlfriend,
or spam and spambayes for the earlier comparison). Considering the
variety of email I receive (and everyone else no doubt), I classify it
into multiple folders (each person I email with gets their own folder,
each mailing list gets another folder, etc.).  In fact, I have over 50
folders for email.

I think that asking a piece of software, regardless of how much
potential it has mathematically, is foolish.  "Hey software, tell me if
this email should go into ANY ONE of my 50 folders, or if it should be
thrown away as spam."  That's a big generalization.

But what if you had 50 corpora of good email, and one bad.  At 50 it
gets to be alot of processing (wait for it), but at 10, it's not so bad. 
In order to properly classify an email into one of those 10+1 categories
under the current 1 dimensional scheme, I think we would have to compare
all possible categorization pairs.  That is, we would need to have every
category compete against every other category for categorization rights.
Something like this (which gives us n choose 2 categorization operations
for each email):

def big_classifier(classifier_list, email):
	results = {}
	cl = classifier_list[:]
	while cl:
		left = cl.pop()
		for right in cl:
			if simple_classifier(left, right)<.5:
				try:
					results[left.__name__]+=1
				except:
					results[left.__name__] =1
			else:
				try:
					results[right.__name__]+=1
				except:
					results[right.__name__] =1
	lst = [(j,i) for i,j in results.iteritems()]
	lst.sort()
	return lst[-1][1]

That should return the single category that the email is most likely to
fit in, because it has 'won' the most trials.  It is not complete
because it doesn't take into account multiple categories having the same
numbers of 'wins', but I'll leave that as an exercise to the reader.

The problem is all the processing that is required to categorize a
single email.  With 9+1 categories, you're looking at 45 categorizations
required, and with 49+1 categories, it's up to 1225 categorizations.
Hell, under teh current scheme you can categorize 45 or 1225 emails in
the time it would take this system to do the same thing.

Or you could just pit 'left' against a null set, <number of categories>
times.  The lower the number, the more likely it is in a particular
category.  Though I am unsure as to the statastical validity to this, it
would be a hell of alot faster than the binomial categorization method
described above.

Just some ideas.  When I have some time, I'm going to put it together as
an experiment.
 - Josiah


From msergeant@startechgroup.co.uk  Mon Sep 30 18:29:26 2002
From: msergeant@startechgroup.co.uk (Matt Sergeant)
Date: Mon, 30 Sep 2002 18:29:26 +0100
Subject: [Spambayes] Central limit
References: <3D981023.5070900@hooft.net>
	<20020930093458.DCA3.JCARLSON@uci.edu>
Message-ID: <3D9889F6.7000208@startechgroup.co.uk>

Josiah Carlson wrote:
> Then Rob Hooft sent the email with this:
> 
> 
>>  - It should somehow be possible to classify messages into any sumber of
>>    distinct groups using this trick. A new message can get scored a
>>    Z-score to describe the likelyhood that it is part of any of the
>>    groups, if all of these numbers are large, the test message does not
>>    belong to any class. I guess, e.g. that it should not be too
>>    difficult for the bayesian algorithms used here to judge whether
>>    E-mail I receive is for "work", "private" or "spam". What would take
>>    this really to the next generation would be an algorithm that can
>>    make the classification "ab initio" as a sort of clustering
>>    algorithm: e.g. something that would start with two of the most
>>    different messages in a single corpus, and add single messages to
>>    either of the two groups until it finds a message that has 2 large
>>    Z-scores. Then it starts a third group.
> 
> 
> I've heard that popfile does something like that, where you can have as
> many categories as you want.  I do not know how they do it
> however...maybe Matt Sergeant (the new perl guy) can check it out and
> tell us what's going on.

It's just an extension of what you're already doing. Imagine the 
equations extended to multiple categories. At the moment you are asking 
it two questions: Is this email spam, and is this email ham [and then 
some fancy equations kick in to decide that given the two probabilities 
which is most likely the truth]. Just extend that to more questions. My 
original implementation of bayesian probability just returned multiple 
categories that the email happened to be in, although it only trained on 
two categories.

> It is entirely possible that by categorizing our email into only two
> distinct categories, spam and ham, that we are asking the statistics to
> generalize too much.  Imagine if instead of spam and ham, you had spam
> and spambayes.  Wouldn't you think it would do much better in
> classifying incoming email into one of those two?  Or even a mother and
> girlfriend (assuming all email is either from your mother or girlfriend,
> or spam and spambayes for the earlier comparison). Considering the
> variety of email I receive (and everyone else no doubt), I classify it
> into multiple folders (each person I email with gets their own folder,
> each mailing list gets another folder, etc.).  In fact, I have over 50
> folders for email.
> 
> I think that asking a piece of software, regardless of how much
> potential it has mathematically, is foolish.  "Hey software, tell me if
> this email should go into ANY ONE of my 50 folders, or if it should be
> thrown away as spam."  That's a big generalization.

Yes and no. Asking it to get more fine-grained than spam vs ham simply 
takes away information from the system - you get fewer tokens in each 
category and it makes it harder for the system to decide. You possibly 
get less FN's, but you'll get more FP's I think.

Matt.


From tim.one@comcast.net  Mon Sep 30 18:43:20 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 30 Sep 2002 13:43:20 -0400
Subject: [Spambayes] Central limit
In-Reply-To: <20020930093458.DCA3.JCARLSON@uci.edu>
Message-ID: <LNBBLJKPBEHFEDALKOLCGENDBHAB.tim.one@comcast.net>

[Josiah Carlson, ponders N-way classification]

There's a large literature on this.  A good start is Jason Rennie's paper on
ifile, a classic Bayesian N-way classifier.  The paper is available from
ifile's homepage:

    http://www.ai.mit.edu/~jrennie/ifile/

Error rates reported there range from 10-20%.  Rennie (ifile's author)
reports an error rate of about 15% on his on own email, doing 50-way
classification (this info is in the ifile FAQ).  Before assuming that
popfile does better than that, measure it <wink>.  The ham-vs-spam error
rates I'm seeing on my corpus are at least 100x better; keep in mind that
people have non-zero error rates too, and several times on this list we've
had vigorous debates about whether a specific piece of email *was* spam.
It's not always clear, and in my set of 20,000 hams I'm still keeping a
message that added a one-line comment to a quote of an entire Nigerian-scam
spam -- that's one of the 2 false positives remaining in my corpus.


From jcarlson@uci.edu  Mon Sep 30 19:07:37 2002
From: jcarlson@uci.edu (Josiah Carlson)
Date: Mon, 30 Sep 2002 11:07:37 -0700
Subject: [Spambayes] Central limit
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGENDBHAB.tim.one@comcast.net>
References: <20020930093458.DCA3.JCARLSON@uci.edu>
	<LNBBLJKPBEHFEDALKOLCGENDBHAB.tim.one@comcast.net>
Message-ID: <20020930110208.DCA6.JCARLSON@uci.edu>


> There's a large literature on this.  A good start is Jason Rennie's paper on
> ifile, a classic Bayesian N-way classifier.  The paper is available from
> ifile's homepage:

I'm going to check it out later this week.  Thank you.

> Error rates reported there range from 10-20%.  Rennie (ifile's author)
> reports an error rate of about 15% on his on own email, doing 50-way
> classification (this info is in the ifile FAQ).  Before assuming that
> popfile does better than that, measure it <wink>.  The ham-vs-spam error
> rates I'm seeing on my corpus are at least 100x better; keep in mind that
> people have non-zero error rates too, and several times on this list we've
> had vigorous debates about whether a specific piece of email *was* spam.
> It's not always clear, and in my set of 20,000 hams I'm still keeping a
> message that added a one-line comment to a quote of an entire Nigerian-scam
> spam -- that's one of the 2 false positives remaining in my corpus.

My thoughts may be addressed in the faq, so yeah.  Even if we do n-way
classification, what if we really only care if it is in one of those 50
good folders, or in that one bad folder.  What I was trying to express
is that generalizing those 50 good folders into 1 good folder could be
generalizing too much.  Though it could also be that our spam
categorization is equally as over-generalized.  What if we were to split
it up into financial spam, porn spam, etc.  I would think that would
even the playing field a bit.  Then one could have two lists, the spam
list (categories of spam) and the ham list (categories of ham).  We
really only are concerned with ham or spam, but by doing a bit more work
on our side, it could make the computer's job easier.

Again, thank you for the information,
 - Josiah

From tim.one@comcast.net  Mon Sep 30 19:23:10 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 30 Sep 2002 14:23:10 -0400
Subject: [Spambayes] Central limit
In-Reply-To: <20020930110208.DCA6.JCARLSON@uci.edu>
Message-ID: <LNBBLJKPBEHFEDALKOLCOENGBHAB.tim.one@comcast.net>

[Josiah Carlson]
> ...
> What if we were to split it up into financial spam, porn spam, etc.
> I would think that would even the playing field a bit.

ifile users in fact report that ifile does better at finding spam if they
set up multiple categories for various kinds of spam.  But despite what some
people may tell you here, the approach this project is taking, and Paul
Graham's approach, have very little in common with Bayesian classifiers.
The clearest connection is that Paul put the word "Bayesian" on his web
page, and then linked to an article doing a non-Bayesian probability
calculation (although the article didn't claim it was doing a Bayesian
calculation; it didn't mention Bayes at all).

> Then one could have two lists, the spam list (categories of spam) and
> the ham list (categories of ham).  We really only are concerned with
> ham or spam, but by doing a bit more work on our side, it could make
> the computer's job easier.

You can test this.  The error rates on my corpus are too low now to measure
an improvement reliably if one were to be made.  Other people here aren't
faring *that* well with this system, and I expect it's because their ham is
more varied than mine (which is composed of comp.lang.python traffic; that
newsgroup has historically had a very generous definition of "on topic", but
the majority of messages still seem to mention Python at least once, or at
least profane Guido's name <wink>).


From tim.one@comcast.net  Mon Sep 30 19:43:52 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 30 Sep 2002 14:43:52 -0400
Subject: [Spambayes] Matt Sergeant: Introduction
In-Reply-To: <3D98486A.1050208@startechgroup.co.uk>
Message-ID: <LNBBLJKPBEHFEDALKOLCMENIBHAB.tim.one@comcast.net>

[Matt Sergeant]

Thanks for the introduction, Matt!  Welcome.

> ...
> Like you all, I discovered very quickly that it's the tokenisation
> techniques that are the biggest "win" when it comes down to it.

The first thing I tried after implementing Graham's scheme was special
tokenization and tagging of embedded http/https/ftp thingies.  That
instantly cut the false negative rate in half.  It remains the single
biggest win we ever got.  The rest has been an aggregation of many smaller
wins, and the benefit gotten over time from finding and removing the biases
in Paul's formulation has been highly significant.  That eventually hit a
wall,where this set of 3 artificialities was stubborn:

    artificially clamping spamprobs into [0.01, 0.99]
    artificially boosting ham counts
    looking at only the 16 most-extreme words

Changing any one, or any two, of those, gave at best mixed results.  It took
wholesale adoption of all of Gary Robinson's ideas at once (some of which
aren't really explained (yet?) on his webpage) to nuke them all.  The fewer
the number of "mystery knobs", the better results have gotten, but the
original biases sometimes acted to cancel each other out in the areas they
hurt most, so you can't get here from there removing just one at a time.

> ...
> and I've done the Robinson stuff without the central limit theorem and
> it didn't work quite as well,

It won't unless you also remove the biases from Paul's scheme.  One of the
biggest wins we got was removing the gimmick that said "well, unless
hamcount*2 + spamcount >= 5, let's pretend we've never seen the word".  That
in particular doesn't seem to play well with Gary's combining formula.

> so I'm hopefully going to get CLT done this week and see how it fares.
> Unfortunately I find python incredibly difficult to read, so it takes
> me a while!

Hmm.  I could tell you to mentally translate

    a.b

to

    $a->{b}

but I doubt your problem is at that level <wink>.  Post a snippet of Python
you find "incredibly difficult to read", and someone will be happy to walk
you thru it.  I really can't guess, as this particular criticism of Python
is one I've never heard before!

> ...
> such as how the probability stuff works so much better on individuals'
> corpora (or on a particular mailing list's corpus) than it does for
> hundreds of thousands of users.

That's been my suspicion, but we haven't tested it here yet.  So save us the
effort and tell us the bottom line from your tests <wink>.


From python-spambayes@discworld.dyndns.org  Mon Sep 30 19:49:46 2002
From: python-spambayes@discworld.dyndns.org (Charles Cazabon)
Date: Mon, 30 Sep 2002 12:49:46 -0600
Subject: [Spambayes] Central limit
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGENDBHAB.tim.one@comcast.net>;
	from tim.one@comcast.net on Mon, Sep 30, 2002 at 01:43:20PM -0400
References: <20020930093458.DCA3.JCARLSON@uci.edu>
	<LNBBLJKPBEHFEDALKOLCGENDBHAB.tim.one@comcast.net>
Message-ID: <20020930124946.A26512@discworld.dyndns.org>

Tim Peters <tim.one@comcast.net> wrote:

> It's not always clear, and in my set of 20,000 hams I'm still keeping a
> message that added a one-line comment to a quote of an entire Nigerian-scam
> spam -- that's one of the 2 false positives remaining in my corpus.

What did the quoting look like?  If it wasn't a top-posting quote, then the
presence of lines beginning with "> " could be noticed by the tokenizer.  Such
standard quote markers are present in roughly ham my ham, and basically none
of my spam.

Charles
-- 
-----------------------------------------------------------------------
Charles Cazabon                 <python-spambayes@discworld.dyndns.org>
GPL'ed software available at:     http://www.qcc.ca/~charlesc/software/
-----------------------------------------------------------------------

From nas@python.ca  Mon Sep 30 20:26:03 2002
From: nas@python.ca (Neil Schemenauer)
Date: Mon, 30 Sep 2002 12:26:03 -0700
Subject: [Spambayes] Central limit
In-Reply-To: <20020930124946.A26512@discworld.dyndns.org>
References: <20020930093458.DCA3.JCARLSON@uci.edu>
	<LNBBLJKPBEHFEDALKOLCGENDBHAB.tim.one@comcast.net>
	<20020930124946.A26512@discworld.dyndns.org>
Message-ID: <20020930192602.GA26508@glacier.arctrix.com>

Charles Cazabon wrote:
> Tim Peters <tim.one@comcast.net> wrote:
> > It's not always clear, and in my set of 20,000 hams I'm still keeping a
> > message that added a one-line comment to a quote of an entire Nigerian-scam
> > spam -- that's one of the 2 false positives remaining in my corpus.
> 
> What did the quoting look like?  If it wasn't a top-posting quote, then the
> presence of lines beginning with "> " could be noticed by the tokenizer.  Such
> standard quote markers are present in roughly ham my ham, and basically none
> of my spam.

Excellent observation.  The spambayes tokenizer currently discards this
valuable information.  I'll test this when I get home if someone else
doesn't beat me too it.

   Neil

From python-spambayes@discworld.dyndns.org  Mon Sep 30 20:31:41 2002
From: python-spambayes@discworld.dyndns.org (Charles Cazabon)
Date: Mon, 30 Sep 2002 13:31:41 -0600
Subject: [Spambayes] Central limit
In-Reply-To: <20020930124946.A26512@discworld.dyndns.org>;
	from python-spambayes@discworld.dyndns.org on Mon, Sep 30, 2002 at
	12:49:46PM -0600
References: <20020930093458.DCA3.JCARLSON@uci.edu>
	<LNBBLJKPBEHFEDALKOLCGENDBHAB.tim.one@comcast.net>
	<20020930124946.A26512@discworld.dyndns.org>
Message-ID: <20020930133141.A26917@discworld.dyndns.org>

Charles Cazabon <python-spambayes@discworld.dyndns.org> wrote:
> What did the quoting look like?  If it wasn't a top-posting quote, then the
> presence of lines beginning with "> " could be noticed by the tokenizer.  Such
> standard quote markers are present in roughly ham my ham [...]

I meant "half my ham", of course.  errno = ENOCOFFEE.

Charles
-- 
-----------------------------------------------------------------------
Charles Cazabon                 <python-spambayes@discworld.dyndns.org>
GPL'ed software available at:     http://www.qcc.ca/~charlesc/software/
-----------------------------------------------------------------------

From papaDoc@videotron.ca  Mon Sep 30 21:30:27 2002
From: papaDoc@videotron.ca (papaDoc)
Date: Mon, 30 Sep 2002 16:30:27 -0400
Subject: [Spambayes] Patch and info on how to run a test
Message-ID: <3D98B463.3040006@videotron.ca>

This is a multi-part message in MIME format.

---------------------- multipart/mixed attachment
Hi,

    This is a small patch to help people which don't have python in 
their path.
They will need to change runtest.sh at only one place.

runtest.sh is talking about
    # This test requires you have an appropriately-modified
    # Tester.py.new and classifier.py.new as detailed in
    # <LNBBLJKPBEHFEDALKOLCKENMBEAB.tim.one@comcast.net>
Where can I find those two files ?
There is no search on the mailing list archive and I subscribed to the 
list after this message.

Also I can run some test but I need to a good .ini file can someone 
provide me with one.
I don't have 2000 spams yet only 1546  now but going up every days.


papaDoc


---------------------- multipart/mixed attachment
Index: runtest.sh
===================================================================
RCS file: /cvsroot/spambayes/spambayes/runtest.sh,v
retrieving revision 1.4
diff -r1.4 runtest.sh
31a32,33
> PYTHON=python
> 
34,35c36,37
<     python rebal.py -r Data/Ham/reservoir -s Data/Ham/Set -n 0 -Q
<     python rebal.py -r Data/Spam/reservoir -s Data/Spam/Set -n 0 -Q
---
>     $PYTHON rebal.py -r Data/Ham/reservoir -s Data/Ham/Set -n 0 -Q
>     $PYTHON rebal.py -r Data/Spam/reservoir -s Data/Spam/Set -n 0 -Q
37,38c39,40
<     python rebal.py -r Data/Ham/reservoir -s Data/Ham/Set -n $RNUM -Q
<     python rebal.py -r Data/Spam/reservoir -s Data/Spam/Set -n $RNUM -Q
---
>     $PYTHON rebal.py -r Data/Ham/reservoir -s Data/Ham/Set -n $RNUM -Q
>     $PYTHON rebal.py -r Data/Spam/reservoir -s Data/Spam/Set -n $RNUM -Q
43c45
< 	python timcv.py -n $SETS > run1.txt
---
> 	$PYTHON timcv.py -n $SETS > run1.txt
46c48
< 	python timcv.py -n $SETS > run2.txt
---
> 	$PYTHON timcv.py -n $SETS > run2.txt
48c50
<         python rates.py run1 run2 > runrates.txt
---
>         $PYTHON rates.py run1 run2 > runrates.txt
50c52
<         python cmp.py run1s run2s | tee results.txt
---
>         $PYTHON cmp.py run1s run2s | tee results.txt
57c59
< 	python timcv.py -n $SETS > run1.txt
---
> 	$PYTHON timcv.py -n $SETS > run1.txt
63c65
< 	python timcv.py -n $SETS > run2.txt
---
> 	$PYTHON timcv.py -n $SETS > run2.txt
65c67
< 	python rates.py run1 run2 > runrates.txt
---
> 	$PYTHON rates.py run1 run2 > runrates.txt
67c69
<         python cmp.py run1s run2s | tee results.txt
---
>         $PYTHON cmp.py run1s run2s | tee results.txt
80c82
< 	python timcv.py -n $SETS > run1.txt
---
> 	$PYTHON timcv.py -n $SETS > run1.txt
90c92
< 	python timcv.py -n $SETS > run2.txt
---
> 	$PYTHON timcv.py -n $SETS > run2.txt
92c94
< 	python rates.py run1 run2 > runrates.txt
---
> 	$PYTHON rates.py run1 run2 > runrates.txt
94c96
< 	python cmp.py run1s run2s | tee results.txt
---
> 	$PYTHON cmp.py run1s run2s | tee results.txt

---------------------- multipart/mixed attachment--

From tim.one@comcast.net  Mon Sep 30 22:00:40 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 30 Sep 2002 17:00:40 -0400
Subject: [Spambayes] Central limit
In-Reply-To: <20020930124946.A26512@discworld.dyndns.org>
Message-ID: <LNBBLJKPBEHFEDALKOLCEEOCBHAB.tim.one@comcast.net>

[Tim]
> ... in my set of 20,000 hams I'm still keeping a message that added
> a one-line comment to a quote of an entire Nigerian-scam spam --
> that's one of the 2 false positives remaining in my corpus.

[Charles Cazabon]
> What did the quoting look like?

It looked like

> Here's a line of Nigerian scam spam and oops! I didn't bother
to reset my width
> so that the tail end of each original line ends up on a separate
line without a '> ' thingie.


> If it wasn't a top-posting quote, then the presence of lines
> beginning with "> " could be noticed by the tokenizer.

Barry Warsaw suggested "doing something about this" near the start, but
nobody followed up on it yet.

Producing a single "looks like this contains a quoted msg" token isn't going
to redeem the msg in question; the evidence against it is overwhelming:

prob = 0.654333643274
prob('header:X-Complaints-To:1') = 0.000210544

That's the strongest thing in its favor, and its very low spamprob is what
keeps the overall score as low as it is.  Spammers will soon catch on to
that, though.

prob('wrote') = 0.0100494

This one is subtle:  it's *already* a "this is a reply to another msg" clue,
due to

"Abass Kabiru" <akabiru@fmwh-ng.com> wrote in message news:etc

kinds of intros.  "wrote:" is also a strong ham clue.

prob('header:Organization:1') = 0.012483
prob('header:Errors-To:1') = 0.021814
prob('ended.') = 0.0412844
prob('tel/fax:') = 0.0412844
prob('besides,') = 0.0445081
prob('complex,') = 0.0488866
prob('forbidding') = 0.0505618
prob('flynn') = 0.0652174

That was the surname of the poster, BTW.

prob('rulers') = 0.0918367
prob('subject:: ') = 0.0960138

That's the ': ' from 'Re: ' in the subject line, another "this is a reply"
clue.

prob('practical') = 0.106896
prob('exported') = 0.142351
prob('subject: (') = 0.143428
prob('floating') = 0.14671
prob('email addr:engineer.com') = 0.155172
prob('jeez,') = 0.155172

This was Flynn's comment, in its entirety:

    Aw jeez, another one of these Nigerian wire scams. This one has
    been around for 20 years.

prob('regards') = 0.164914
prob('existence') = 0.165053
prob('might') = 0.175766
prob('maintaining') = 0.194761
prob('skip:n 50') = 0.225142
prob('propose') = 0.238668
prob('regards,') = 0.247521
prob('which') = 0.248843
prob('executed') = 0.267601
prob('subject:)') = 0.272844
prob('hope') = 0.288649
prob('content-type:text/plain') = 0.30566
prob('use') = 0.310592
prob('hence') = 0.313828
prob('therefore') = 0.328694
prob('had') = 0.348322
prob('works') = 0.351337

All the rest damn it, and there's just an overwhelming number of strong
ones:

prob('their') = 0.647744
prob('completely') = 0.651573
prob('own') = 0.664906
prob('seventeen') = 0.666806
prob('will') = 0.683015
prob('ourselves') = 0.69539
prob('three') = 0.701154
prob('share') = 0.705119
prob('recently') = 0.708577
prob('web') = 0.711128
prob('stories') = 0.717668
prob('include') = 0.725245
prob('present') = 0.725666
prob('amount') = 0.72762
prob('strict') = 0.730353
prob('your') = 0.753282
prob('safe') = 0.755277
prob('chairman') = 0.763306
prob('state.') = 0.76889
prob('colleagues') = 0.771379
prob('panel') = 0.775393
prob('best') = 0.777128
prob('subject:Relationship') = 0.779381
prob('hundred') = 0.784061
prob('review') = 0.785455
prob('details') = 0.792105
prob('thousand') = 0.806379
prob('delegated') = 0.825203
prob('forward') = 0.828775
prob('sending') = 0.833239
prob('years.') = 0.841341
prob('discredit') = 0.844828
prob('ikoyi,') = 0.844828
prob('secretariat') = 0.844828
prob('fax') = 0.844952
prob('transfer') = 0.845108
prob('trust') = 0.849266
prob('agreed') = 0.850212
prob('transfer.') = 0.851067
prob('request') = 0.855474
prob('personal') = 0.860963
prob('subject:Urgent') = 0.86356
prob('report') = 0.865439
prob('five') = 0.868865
prob('expertise') = 0.871287
prob('relationship') = 0.871766
prob('buy') = 0.87972
prob('balance') = 0.88566
prob('confidential') = 0.893285
prob('confidence') = 0.893354
prob('please') = 0.898501
prob('account') = 0.899067
prob('contract') = 0.904767
prob('owner') = 0.905379
prob('(fmwh)') = 0.908163
prob('(twenty') = 0.908163
prob('agricultural') = 0.908163
prob('from:"ryan"') = 0.908163
prob('scams.') = 0.908163
prob('25%') = 0.910276
prob('laws') = 0.914263
prob('investigate') = 0.91655
prob('company') = 0.921738
prob('reply') = 0.923642
prob('accounts') = 0.923935
prob('solicit') = 0.930563
prob('letter.') = 0.932677
prob('urgent') = 0.933103
prob('foreign') = 0.934779
prob('(you)') = 0.934783
prob('identified') = 0.936407
prob('states') = 0.93739
prob('transferred') = 0.943072
prob('accounts,') = 0.943946
prob('our') = 0.944073
prob('blueprint') = 0.949438
prob('paid') = 0.951581
prob('military') = 0.954317
prob('ministry') = 0.955993
prob('100%') = 0.958727
prob('business') = 0.960023
prob('duties,') = 0.965116
prob('million') = 0.965902
prob('partner') = 0.966006
prob('money') = 0.96939
prob('country') = 0.969802
prob('contracts') = 0.970674
prob('civil') = 0.971325
prob('legal.') = 0.972617
prob('engr.') = 0.973373
prob('telephone') = 0.977239
prob('transaction') = 0.978387
prob('dollars)') = 0.978598
prob('bank') = 0.982743
prob('servants') = 0.984429
prob('housing') = 0.985369
prob('awarded,') = 0.985437
prob('lagos.') = 0.986322
prob('funds') = 0.986686
prob('united') = 0.986689
prob('country.') = 0.98727
prob('government') = 0.987615
prob('subject:Business') = 0.988133
prob('partner.') = 0.98951
prob('70%') = 0.99126
prob('regime') = 0.991803
prob('purchase') = 0.992473
prob('overseas') = 0.993653
prob('nigerian') = 0.994148
prob('payment') = 0.994381
prob('nigeria.') = 0.99545
prob('nigeria') = 0.996851
prob('domestic') = 0.998494
prob('federal') = 0.998609
prob('estate') = 0.998921

Another possibility is to strip quoted portions, which is more what Barry
suggested.  It probably still wouldn't be enough to redeem this msg, since
the quoting was broken due to linewrap.  It's also unknown how many more
false positives that would create:  many newsgroup msgs consist of nothing
except a long quote and a one-line-- or even one-word --comment.  The long
quote usually contains the most-hammy stuff then.


From skip@pobox.com  Mon Sep 30 22:37:31 2002
From: skip@pobox.com (Skip Montanaro)
Date: Mon, 30 Sep 2002 16:37:31 -0500
Subject: [Spambayes] mining dates?
Message-ID: <15768.50203.838893.944644@12-248-11-90.client.attbi.com>


---------------------- multipart/mixed attachment

I notice when I arrive in the morning that a fairly large number of spam
messages are waiting for me, already isolated, courtesy SpamAssassin.  I
thought, "hmmm...  I wonder if more spam is sent overnight than during the
day?", so I performed a simple experiment.  I "mined" the Date: headers from
my ham and spam collections and plopped the times into 6-minute buckets
(local time) like so:

    find Data/Ham -type f \
    | xargs cat \
    | sed -n -e '/^From /,/^$/p' \
    | egrep '^Date: ' \
    | sed -e 's/.*\([0-9][0-9]:[0-9][0-9]:[0-9][0-9]\).*/\1/' \
    | awk -F: '{print int(($1*60+$2)/10)}' \
    | sort -n \
    | uniq -c \
    | awk '{print $2, $1}'  > hamhour.plt

and then plotted the output of both the ham and spam collections using
gnuplot.  The result is attached.

It didn't prove my hypothesis, but may have exposed something as useful.
Spam seems to be sent at a fairly constant rate throughout the day, which
stands to reason, since it's probably all sent automatically.  However, ham
definitely seems to be sent predominantly during waking hours (doh!).  I'm
going to give a little date mining a try.

Skip


---------------------- multipart/mixed attachment
A non-text attachment was scrubbed...
Name: hour.png
Type: image/png
Size: 7616 bytes
Desc: not available
Url : http://mail.python.org/pipermail-21/spambayes/attachments/20020930/b6c12eae/hour.png

---------------------- multipart/mixed attachment--

From skip@pobox.com  Mon Sep 30 23:07:49 2002
From: skip@pobox.com (Skip Montanaro)
Date: Mon, 30 Sep 2002 17:07:49 -0500
Subject: [Spambayes] new option: generate_long_skips
Message-ID: <15768.52021.408205.624157@12-248-11-90.client.attbi.com>


I just checked in a new option for the tokenizer: generate_long_skips.  The
default is True.  I noticed when reviewing my false positives that one was
overwhelmingly dominated by these tokens (which scored very high) because it
contained an Excel spreadsheet attachment.  "cutoff.ini" is

    [TestDriver]
    spam_cutoff: 0.4

while "noskips.ini" is

    [Tokenizer]
    generate_long_skips: False

    [TestDriver]
    spam_cutoff: 0.4

I am currently running a test with 10 sets of 200 messages per set.  With 5
sets I got two more fn's and two less fp's:

    cutoffs -> noskipss
    -> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
    -> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
    -> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
    -> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
    -> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
    -> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
    -> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
    -> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
    -> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams
    -> <stat> tested 200 hams & 200 spams against 800 hams & 800 spams

    false positive percentages
        1.500  1.000  won    -33.33%
        4.000  4.000  tied          
        3.000  2.500  won    -16.67%
        1.500  1.500  tied          
        1.000  1.000  tied          

    won   2 times
    tied  3 times
    lost  0 times

    total unique fp went from 22 to 20 won     -9.09%
    mean fp % went from 2.2 to 2.0 won     -9.09%

    false negative percentages
        2.000  2.500  lost   +25.00%
        1.000  1.000  tied          
        0.500  0.500  tied          
        1.500  1.500  tied          
        2.000  2.500  lost   +25.00%

    won   0 times
    tied  3 times
    lost  2 times

    total unique fn went from 14 to 16 lost   +14.29%
    mean fn % went from 1.4 to 1.6 lost   +14.29%

    ham mean                     ham sdev
      22.12   21.85   -1.22%        6.01    5.74   -4.49%
      23.46   23.25   -0.90%        7.31    6.93   -5.20%
      23.50   23.38   -0.51%        6.64    6.51   -1.96%
      23.54   23.32   -0.93%        6.88    6.87   -0.15%
      23.08   22.79   -1.26%        6.77    6.62   -2.22%

    ham mean and sdev for all runs
      23.14   22.92   -0.95%        6.76    6.57   -2.81%

    spam mean                    spam sdev
      72.49   71.95   -0.74%       13.82   14.02   +1.45%
      71.34   70.61   -1.02%       13.70   13.45   -1.82%
      73.12   72.58   -0.74%       12.88   12.80   -0.62%
      72.40   72.01   -0.54%       12.71   12.65   -0.47%
      70.71   70.10   -0.86%       13.91   13.74   -1.22%

    spam mean and sdev for all runs
      72.01   71.45   -0.78%       13.44   13.37   -0.52%

    ham/spam mean difference: 48.87 48.53 -0.34

I think it might be helpful for people whose ham tends to get the occasional
legitimate binary attachment.  In any case, it's easier for people to test
it if I check it in.  We can always remove it if it turns out not to be
generally useful.  Also, it makes it easier for me to add the date mining
without interference with this change.

Note the very low spam_cutoff.  That was as suggested by an earlier run.
After I switched I got a dramatic drop in fn's.  

Skip

From bkc@murkworks.com  Mon Sep 30 23:09:34 2002
From: bkc@murkworks.com (Brad Clements)
Date: Mon, 30 Sep 2002 18:09:34 -0400
Subject: [Spambayes] Central limit
In-Reply-To: <20020930192602.GA26508@glacier.arctrix.com>
References: <20020930124946.A26512@discworld.dyndns.org>
Message-ID: <3D989345.22448.E97AFB9@localhost>

On 30 Sep 2002 at 12:26, Neil Schemenauer wrote:

> Excellent observation.  The spambayes tokenizer currently discards this
> valuable information.  I'll test this when I get home if someone else
> doesn't beat me too it.

So I send you a message:

>  Ha ha , this spam is ignored..


Brad Clements,                bkc@murkworks.com   (315)268-1000
http://www.murkworks.com                          (315)268-9812 Fax
AOL-IM: BKClements


From nas@python.ca  Mon Sep 30 23:20:28 2002
From: nas@python.ca (Neil Schemenauer)
Date: Mon, 30 Sep 2002 15:20:28 -0700
Subject: [Spambayes] Central limit
In-Reply-To: <3D989345.22448.E97AFB9@localhost>
References: <20020930124946.A26512@discworld.dyndns.org>
	<3D989345.22448.E97AFB9@localhost>
Message-ID: <20020930222028.GA27037@glacier.arctrix.com>

Brad Clements wrote:
> On 30 Sep 2002 at 12:26, Neil Schemenauer wrote:
> 
> > Excellent observation.  The spambayes tokenizer currently discards this
> > valuable information.  I'll test this when I get home if someone else
> > doesn't beat me too it.
> 
> So I send you a message:
> 
> >  Ha ha , this spam is ignored..

No but your message would probably get a bonus for using the '>'
character.

  Neil

From tim.one@comcast.net  Mon Sep 30 23:20:25 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 30 Sep 2002 18:20:25 -0400
Subject: [Spambayes] new option: generate_long_skips
In-Reply-To: <15768.52021.408205.624157@12-248-11-90.client.attbi.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCEEOLBHAB.tim.one@comcast.net>

[Skip Montanaro]
> I just checked in a new option for the tokenizer:
> generate_long_skips.  The default is True.  I noticed when reviewing
> my false positives that one was overwhelmingly dominated by these
> tokens (which scored very high) because it contained an Excel
> spreadsheet attachment.

We ignore MIME sections that don't have text/* type.  Was this a spreadsheet
that identified itself as having text content-type?

>     false positive percentages
>         1.500  1.000  won    -33.33%
>         4.000  4.000  tied
>         3.000  2.500  won    -16.67%
>         1.500  1.500  tied
>         1.000  1.000  tied
>
>     won   2 times
>     tied  3 times
>     lost  0 times

Pure but small win.

>     total unique fp went from 22 to 20 won     -9.09%
>     mean fp % went from 2.2 to 2.0 won     -9.09%
>
>     false negative percentages
>         2.000  2.500  lost   +25.00%
>         1.000  1.000  tied
>         0.500  0.500  tied
>         1.500  1.500  tied
>         2.000  2.500  lost   +25.00%
>
>     won   0 times
>     tied  3 times
>     lost  2 times

Pure but small loss.

>     total unique fn went from 14 to 16 lost   +14.29%
>     mean fn % went from 1.4 to 1.6 lost   +14.29%
>
>     ham mean                     ham sdev
>       22.12   21.85   -1.22%        6.01    5.74   -4.49%
>       23.46   23.25   -0.90%        7.31    6.93   -5.20%
>       23.50   23.38   -0.51%        6.64    6.51   -1.96%
>       23.54   23.32   -0.93%        6.88    6.87   -0.15%
>       23.08   22.79   -1.26%        6.77    6.62   -2.22%
>
>     ham mean and sdev for all runs
>       23.14   22.92   -0.95%        6.76    6.57   -2.81%
>
>     spam mean                    spam sdev
>       72.49   71.95   -0.74%       13.82   14.02   +1.45%
>       71.34   70.61   -1.02%       13.70   13.45   -1.82%
>       73.12   72.58   -0.74%       12.88   12.80   -0.62%
>       72.40   72.01   -0.54%       12.71   12.65   -0.47%
>       70.71   70.10   -0.86%       13.91   13.74   -1.22%
>
>     spam mean and sdev for all runs
>       72.01   71.45   -0.78%       13.44   13.37   -0.52%
>
>     ham/spam mean difference: 48.87 48.53 -0.34

Mixed bag, but overall brought your ham and spam a little closer together.

> I think it might be helpful for people whose ham tends to get the
> occasional legitimate binary attachment.

The code intends to ignore those already; perhaps the MIME in the example
you looked at was incorrect, or perhaps tokenizer.textparts() is buggy?

> ...
> Note the very low spam_cutoff. [.40]

I did <wink>.  Your corpus remains unique this way!


From tim.one@comcast.net  Mon Sep 30 23:24:08 2002
From: tim.one@comcast.net (Tim Peters)
Date: Mon, 30 Sep 2002 18:24:08 -0400
Subject: [Spambayes] Central limit
In-Reply-To: <20020930222028.GA27037@glacier.arctrix.com>
Message-ID: <LNBBLJKPBEHFEDALKOLCKEOMBHAB.tim.one@comcast.net>

[Neil Schemenauer]
> No but your message would probably get a bonus for using the '>'
> character.

I too suspect that would work better than stripping quoted parts, and that
testing will show it's a small win, but haven't tested it either way.
You'll go mad by the time you figure out how to recognize Barry's unreadably
fancy-pants Emacs quoting, though <wink>.

For my own use, I'd add a logarithmic "# of quoted lines" token -- most
posts consisting of massive quotes are spam in my eyes <0.7 wink>.


From anthony@interlink.com.au  Mon Sep 30 23:41:42 2002
From: anthony@interlink.com.au (Anthony Baxter)
Date: Tue, 01 Oct 2002 08:41:42 +1000
Subject: [Spambayes] mining dates? 
In-Reply-To: <15768.50203.838893.944644@12-248-11-90.client.attbi.com> 
Message-ID: <200209302241.g8UMfgJ08118@localhost.localdomain>


>>> Skip Montanaro wrote
> I notice when I arrive in the morning that a fairly large number of spam
> messages are waiting for me, already isolated, courtesy SpamAssassin.  I
> thought, "hmmm...  I wonder if more spam is sent overnight than during the
> day?", so I performed a simple experiment.  I "mined" the Date: headers from
> my ham and spam collections and plopped the times into 6-minute buckets
> (local time) like so:

Wow. I think this is the first time the Date header's been used in this
project constructively. Normally it is one of the ones people swear at
because it breaks their mixed-corpus data too easily.

> It didn't prove my hypothesis, but may have exposed something as useful.
> Spam seems to be sent at a fairly constant rate throughout the day, which
> stands to reason, since it's probably all sent automatically.  However, ham
> definitely seems to be sent predominantly during waking hours (doh!).  I'm
> going to give a little date mining a try.

Interesting. I'm not sure it actually buys that much, timezones being
what they are. Unless you have evidence that, say, all spam is actually
sent by a small team of Belgians, in which case we can just knock out
stuff sent during business hours in belgian standard time.

On the other hand, there _might_ be value from tokenizing the timezone
part of the field... hm.

Interesting nonetheless.

Anthony

From skip@pobox.com  Mon Sep 30 23:43:38 2002
From: skip@pobox.com (Skip Montanaro)
Date: Mon, 30 Sep 2002 17:43:38 -0500
Subject: [Spambayes] new option: generate_long_skips
Message-ID: <15768.54170.102815.684984@12-248-11-90.client.attbi.com>


    > I just checked in a new option for the tokenizer: generate_long_skips.
    ...
    > I am currently running a test with 10 sets of 200 messages per set.

Almost exactly the same:

    cutoffs -> noskipss
    -> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
    -> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
    -> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
    -> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
    -> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
    -> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
    -> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
    -> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
    -> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
    -> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
    -> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
    -> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
    -> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
    -> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
    -> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
    -> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
    -> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
    -> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
    -> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams
    -> <stat> tested 200 hams & 200 spams against 1800 hams & 1800 spams

    false positive percentages
        1.000  1.000  tied          
        1.500  1.500  tied          
        1.000  1.000  tied          
        1.000  1.000  tied          
        1.000  1.000  tied          
        1.500  1.000  won    -33.33%
        3.500  3.500  tied          
        1.500  1.500  tied          
        1.500  1.500  tied          
        1.500  1.500  tied          

    won   1 times
    tied  9 times
    lost  0 times

    total unique fp went from 30 to 29 won     -3.33%
    mean fp % went from 1.5 to 1.45 won     -3.33%

    false negative percentages
        0.500  0.500  tied          
        1.500  1.500  tied          
        0.500  0.500  tied          
        0.500  1.000  lost  +100.00%
        2.000  2.000  tied          
        0.000  0.000  tied          
        1.000  1.000  tied          
        1.000  1.000  tied          
        0.000  0.000  tied          
        1.500  1.500  tied          

    won   0 times
    tied  9 times
    lost  1 times

    total unique fn went from 17 to 18 lost    +5.88%
    mean fn % went from 0.85 to 0.9 lost    +5.88%

    ham mean                     ham sdev
      20.82   20.59   -1.10%        6.43    6.18   -3.89%
      21.86   21.66   -0.91%        6.63    6.26   -5.58%
      21.38   21.26   -0.56%        6.49    6.30   -2.93%
      21.96   21.79   -0.77%        6.26    6.24   -0.32%
      21.51   21.29   -1.02%        6.72    6.62   -1.49%
      21.66   21.43   -1.06%        6.98    6.84   -2.01%
      21.45   21.32   -0.61%        7.66    7.65   -0.13%
      21.74   21.51   -1.06%        6.69    6.64   -0.75%
      21.71   21.49   -1.01%        7.44    7.21   -3.09%
      21.87   21.73   -0.64%        5.93    5.92   -0.17%

    ham mean and sdev for all runs
      21.60   21.41   -0.88%        6.75    6.61   -2.07%

    spam mean                    spam sdev
      74.10   73.56   -0.73%       12.99   13.13   +1.08%
      72.47   71.74   -1.01%       13.92   13.78   -1.01%
      74.05   73.52   -0.72%       13.00   13.07   +0.54%
      74.00   73.54   -0.62%       12.27   12.16   -0.90%
      72.43   71.91   -0.72%       13.73   13.52   -1.53%
      72.68   72.24   -0.61%       13.27   13.26   -0.08%
      72.57   71.84   -1.01%       13.03   12.99   -0.31%
      71.50   71.30   -0.28%       12.12   12.14   +0.17%
      73.25   72.68   -0.78%       12.67   12.53   -1.10%
      73.02   72.70   -0.44%       12.44   12.43   -0.08%

    spam mean and sdev for all runs
      73.01   72.50   -0.70%       12.98   12.94   -0.31%

    ham/spam mean difference: 51.41 51.09 -0.32

I notice it's suggesting an even lower cutoff now (0.375).

Before:

    -> best cutoff for all runs: 0.4
    ->     with weighted total 1*30 fp + 17 fn = 47
    ->     fp rate 1.5%  fn rate 0.85%

After:

    -> best cutoff for all runs: 0.375
    ->     with weighted total 1*35 fp + 7 fn = 42
    ->     fp rate 1.75%  fn rate 0.35%

Skip

From skip@pobox.com  Mon Sep 30 23:55:43 2002
From: skip@pobox.com (Skip Montanaro)
Date: Mon, 30 Sep 2002 17:55:43 -0500
Subject: [Spambayes] RE: [Spambayes-checkins] spambayes
 Options.py,1.39,1.40tokenizer.py,1.45,1.46
In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEOJBHAB.tim.one@comcast.net>
References: <E17w8Wj-0002W2-00@usw-pr-cvs1.sourceforge.net>
        <LNBBLJKPBEHFEDALKOLCGEOJBHAB.tim.one@comcast.net>
Message-ID: <15768.54895.618529.785669@12-248-11-90.client.attbi.com>


    Tim> [Skip Montanaro]
    >> allow users to disable the long word skip tokens (e.g "skip:c 70")
    >> under the assumption that people who do receive mail which contains
    >> attachements will be penalized.

    Tim> Skip, what is your reasoning here?  We ignore attachments entirely
    Tim> unless they have text/* type.  I don't see what skip tokens have to
    Tim> do with this.

Well, now that you mention it, I'm just a touch confused.  Maybe the message
which got hammered by those tokens couldn't be parsed by the email package
so the entire body got interpreted as text?

I added these options a week or so ago:

    # If true, the first few characters of application/octet-stream sections
    # are used, undecoded.  What 'few' means is decided by octet_prefix_size.
    check_octets: False
    octet_prefix_size: 5

though I currently have check_octets set to False.  I'm rerunning my tests
with it enabled to see how it works.

    Tim> Besides, I named those tokens after you <wink>.

I didn't mention it before, but I've already updated my resume to reflect
this latest award... <wink back>

Skip