[Spambayes] Comparing chi to zcombine
Brad Clements
bkc@murkworks.com
Mon, 14 Oct 2002 14:08:15 -0400
First, cmp.py
results/chitrues.txt -> results/zcombines.txt
-> <stat> tested 1300 hams & 1300 spams against 11700 hams & 11700 spams
false positive percentages
0.154 0.154 tied
0.231 0.077 won -66.67%
0.077 0.154 lost +100.00%
0.154 0.462 lost +200.00%
0.154 0.154 tied
0.077 0.154 lost +100.00%
0.077 0.077 tied
0.000 0.000 tied
0.231 0.538 lost +132.90%
0.000 0.154 lost +(was 0)
won 1 times
tied 4 times
lost 5 times
total unique fp went from 15 to 25 lost +66.67%
mean fp % went from 0.115384615385 to 0.192307692308 lost +66.67%
false negative percentages
0.846 1.308 lost +54.61%
1.231 1.538 lost +24.94%
1.154 1.308 lost +13.34%
0.615 0.846 lost +37.56%
0.923 1.000 lost +8.34%
1.308 1.154 won -11.77%
0.692 1.077 lost +55.64%
1.077 1.231 lost +14.30%
1.231 1.154 won -6.26%
1.231 1.077 won -12.51%
won 3 times
tied 0 times
lost 7 times
total unique fn went from 134 to 152 lost +13.43%
mean fn % went from 1.03076923077 to 1.16923076923 lost +13.43%
ham mean ham sdev
1.40 1.37 -2.14% 8.67 9.24 +6.57%
1.12 1.03 -8.04% 8.09 8.37 +3.46%
1.12 1.02 -8.93% 8.02 7.86 -2.00%
1.26 1.13 -10.32% 8.62 9.07 +5.22%
1.06 1.04 -1.89% 8.03 8.27 +2.99%
1.01 0.86 -14.85% 6.87 7.08 +3.06%
0.85 0.71 -16.47% 6.57 6.55 -0.30%
0.96 0.90 -6.25% 7.06 7.56 +7.08%
1.15 1.00 -13.04% 8.38 8.67 +3.46%
1.01 0.77 -23.76% 7.62 7.11 -6.69%
ham mean and sdev for all runs
1.09 0.98 -10.09% 7.83 8.03 +2.55%
spam mean spam sdev
99.74 99.75 +0.01% 3.59 3.85 +7.24%
99.67 99.71 +0.04% 4.17 3.95 -5.28%
99.68 99.70 +0.02% 4.12 4.43 +7.52%
99.83 99.81 -0.02% 2.68 3.16 +17.91%
99.84 99.91 +0.07% 2.20 0.96 -56.36%
99.66 99.73 +0.07% 4.29 3.92 -8.62%
99.67 99.74 +0.07% 4.68 4.28 -8.55%
99.79 99.81 +0.02% 2.98 2.52 -15.44%
99.75 99.78 +0.03% 3.24 2.85 -12.04%
99.54 99.68 +0.14% 5.07 4.96 -2.17%
spam mean and sdev for all runs
99.72 99.76 +0.04% 3.80 3.66 -3.68%
ham/spam mean difference: 98.63 98.78 +0.15
And now, the zcombine histogram
-> <stat> Ham scores for all runs: 13000 items; mean 0.98; sdev 8.03
-> <stat> min -6.66134e-14; median 0; max 100
* = 205 items
0.0 12487 *************************************************************
0.5 84 *
1.0 44 *
1.5 29 *
2.0 28 *
2.5 17 *
3.0 7 *
3.5 7 *
4.0 11 *
4.5 14 *
5.0 1 *
5.5 5 *
6.0 5 *
6.5 6 *
7.0 3 *
7.5 6 *
8.0 8 *
8.5 2 *
9.0 1 *
9.5 3 *
10.0 1 *
10.5 3 *
11.0 6 *
11.5 3 *
12.0 6 *
12.5 6 *
13.0 8 *
13.5 1 *
14.0 1 *
14.5 0
15.0 1 *
15.5 2 *
16.0 3 *
16.5 1 *
17.0 2 *
17.5 3 *
18.0 2 *
18.5 1 *
19.0 2 *
19.5 2 *
20.0 0
20.5 3 *
21.0 1 *
21.5 0
22.0 2 *
22.5 0
23.0 3 *
23.5 1 *
24.0 0
24.5 2 *
25.0 1 *
25.5 3 *
26.0 0
26.5 2 *
27.0 0
27.5 1 *
28.0 0
28.5 2 *
29.0 2 *
29.5 1 *
30.0 1 *
30.5 2 *
31.0 0
31.5 3 *
32.0 2 *
32.5 0
33.0 0
33.5 1 *
34.0 2 *
34.5 0
35.0 2 *
35.5 0
36.0 2 *
36.5 2 *
37.0 1 *
37.5 2 *
38.0 1 *
38.5 0
39.0 2 *
39.5 1 *
40.0 1 *
40.5 4 *
41.0 1 *
41.5 1 *
42.0 0
42.5 0
43.0 1 *
43.5 2 *
44.0 0
44.5 0
45.0 1 *
45.5 1 *
46.0 1 *
46.5 1 *
47.0 1 *
47.5 1 *
48.0 1 *
48.5 2 *
49.0 0
49.5 1 *
50.0 0
50.5 0
51.0 1 *
51.5 0
52.0 1 *
52.5 1 *
53.0 2 *
53.5 0
54.0 0
54.5 1 *
55.0 1 *
55.5 1 *
56.0 1 *
56.5 1 *
57.0 1 *
57.5 0
58.0 2 *
58.5 2 *
59.0 2 *
59.5 3 *
60.0 1 *
60.5 2 *
61.0 1 *
61.5 1 *
62.0 2 *
62.5 1 *
63.0 2 *
63.5 0
64.0 0
64.5 1 *
65.0 3 *
65.5 0
66.0 0
66.5 2 *
67.0 0
67.5 1 *
68.0 0
68.5 1 *
69.0 0
69.5 0
70.0 0
70.5 0
71.0 3 *
71.5 1 *
72.0 0
72.5 1 *
73.0 0
73.5 0
74.0 0
74.5 0
75.0 0
75.5 0
76.0 0
76.5 0
77.0 0
77.5 1 *
78.0 0
78.5 0
79.0 2 *
79.5 0
80.0 3 *
80.5 0
81.0 0
81.5 0
82.0 0
82.5 1 *
83.0 1 *
83.5 1 *
84.0 0
84.5 1 *
85.0 0
85.5 1 *
86.0 1 *
86.5 0
87.0 1 *
87.5 2 *
88.0 3 *
88.5 4 *
89.0 1 *
89.5 1 *
90.0 2 *
90.5 2 *
91.0 0
91.5 0
92.0 0
92.5 0
93.0 0
93.5 1 *
94.0 0
94.5 0
95.0 3 *
95.5 0
96.0 2 *
96.5 2 *
97.0 1 *
97.5 1 *
98.0 4 *
98.5 0
99.0 2 *
99.5 23 *
-> <stat> Spam scores for all runs: 13000 items; mean 99.76; sdev 3.66
-> <stat> min 0; median 100; max 100
* = 210 items
0.0 5 *
0.5 1 *
1.0 1 *
1.5 2 *
2.0 1 *
2.5 0
3.0 1 *
3.5 0
4.0 0
4.5 0
5.0 0
5.5 0
6.0 0
6.5 0
7.0 0
7.5 0
8.0 0
8.5 0
9.0 0
9.5 0
10.0 0
10.5 0
11.0 0
11.5 0
12.0 0
12.5 0
13.0 0
13.5 0
14.0 0
14.5 0
15.0 0
15.5 0
16.0 0
16.5 0
17.0 0
17.5 1 *
18.0 0
18.5 0
19.0 0
19.5 0
20.0 0
20.5 1 *
21.0 0
21.5 0
22.0 0
22.5 0
23.0 0
23.5 0
24.0 0
24.5 0
25.0 0
25.5 0
26.0 0
26.5 0
27.0 0
27.5 0
28.0 0
28.5 0
29.0 0
29.5 0
30.0 0
30.5 0
31.0 0
31.5 0
32.0 0
32.5 1 *
33.0 0
33.5 0
34.0 0
34.5 2 *
35.0 0
35.5 0
36.0 1 *
36.5 0
37.0 0
37.5 0
38.0 0
38.5 0
39.0 0
39.5 0
40.0 0
40.5 0
41.0 0
41.5 0
42.0 1 *
42.5 0
43.0 0
43.5 0
44.0 0
44.5 0
45.0 0
45.5 0
46.0 0
46.5 0
47.0 1 *
47.5 0
48.0 0
48.5 0
49.0 1 *
49.5 0
50.0 0
50.5 0
51.0 0
51.5 0
52.0 0
52.5 0
53.0 0
53.5 0
54.0 0
54.5 0
55.0 0
55.5 1 *
56.0 0
56.5 0
57.0 0
57.5 0
58.0 0
58.5 0
59.0 0
59.5 0
60.0 2 *
60.5 0
61.0 2 *
61.5 0
62.0 0
62.5 0
63.0 0
63.5 1 *
64.0 1 *
64.5 0
65.0 0
65.5 1 *
66.0 1 *
66.5 1 *
67.0 0
67.5 0
68.0 0
68.5 1 *
69.0 0
69.5 0
70.0 0
70.5 1 *
71.0 0
71.5 0
72.0 2 *
72.5 0
73.0 0
73.5 0
74.0 1 *
74.5 0
75.0 0
75.5 0
76.0 0
76.5 0
77.0 0
77.5 3 *
78.0 2 *
78.5 1 *
79.0 3 *
79.5 0
80.0 0
80.5 1 *
81.0 0
81.5 0
82.0 0
82.5 0
83.0 1 *
83.5 0
84.0 3 *
84.5 1 *
85.0 1 *
85.5 3 *
86.0 1 *
86.5 1 *
87.0 1 *
87.5 2 *
88.0 0
88.5 1 *
89.0 1 *
89.5 2 *
90.0 2 *
90.5 1 *
91.0 2 *
91.5 2 *
92.0 2 *
92.5 3 *
93.0 6 *
93.5 1 *
94.0 2 *
94.5 2 *
95.0 5 *
95.5 6 *
96.0 10 *
96.5 4 *
97.0 6 *
97.5 14 *
98.0 21 *
98.5 17 *
99.0 37 *
99.5 12794 *************************************************************
-> best cutoff for all runs: 0.985
-> with weighted total 10*25 fp + 152 fn = 402
-> fp rate 0.192% fn rate 1.17%
saving ham histogram pickle to class_hamhist.pik
saving spam histogram pickle to class_spamhist.pik
.ini for zcombine run
[Tokenizer]
mine_received_headers: True
[Classifier]
use_central_limit = False
use_central_limit2 = False
use_central_limit3 = False
use_tim_combining: False
use_chi_squared_combining: False
use_z_combining: True
robinson_minimum_prob_strength: 0.0
[TestDriver]
spam_cutoff: 0.985
show_false_negatives: True
show_false_positives: True
nbuckets: 200
best_cutoff_fp_weight: 10
show_spam_lo: 0.4
show_spam_hi: 0.80
show_ham_lo = 0.40
show_ham_hi = 0.80
show_charlimit: 10000
save_trained_pickles: True
save_histogram_pickles: True
Brad Clements, bkc@murkworks.com (315)268-1000
http://www.murkworks.com (315)268-9812 Fax
AOL-IM: BKClements