[spambayes-dev] How much tokenizer improvement is enough to justify a change?

[Sean True]

I ran the current tokenizer on timcv.py on 10000 hams and 10000 spams split
into 10 buckets.
Here's the base set ... Comments?

-> best cost for all runs: $169.60
-> per-fp cost $10.00; per-fn cost $1.00; per-unsure cost $0.20
-> achieved at ham & spam cutoffs 0.025 & 0.275
->     fp 6; fn 55; unsure ham 47; unsure spam 226
->     fp rate 0.06%; fn rate 0.55%; unsure rate 1.36%
-> <stat> all runs false positives: 0
-> <stat> all runs false negatives: 201
-> <stat> all runs unsure: 1460
-> <stat> all runs false positive %: 0.0
-> <stat> all runs false negative %: 2.01
-> <stat> all runs unsure %: 7.3
-> <stat> all runs cost: $493.00
Total messages: 20000; 10000 (50.0%) ham + 10000 (50.0%) spam
Ham: 9989 (99.89%) ok, 11 (0.11%) unsure, 0 (0.00%) fp
Spam: 8350 (83.50%) ok, 1449 (14.49%) unsure, 201 (2.01%) fn
Score False: 1.00% Unsure 7.30%
Standard Cost: $493.0000
Flex Cost: $761.9844
Flex**2 Cost: $529.8252
Delayed-Total messages: 20000; 10000 (50.0%) ham + 10000 (50.0%) spam
Delayed-Ham: 9947 (99.47%) ok, 47 (0.47%) unsure, 6 (0.06%) fp
Delayed-Spam: 9719 (97.19%) ok, 226 (2.26%) unsure, 55 (0.55%) fn
Delayed-Score False: 0.30% Unsure 1.36%
Delayed-Standard Cost: $169.6000
Delayed-Flex Cost: $318.2508
Delayed-Flex**2 Cost: $232.0352

After change (breaking up compound words > maxwordlen into smaller words)
-> best cost for all runs: $157.60
-> per-fp cost $10.00; per-fn cost $1.00; per-unsure cost $0.20
-> achieved at ham & spam cutoffs 0.015 & 0.295
->     fp 5; fn 44; unsure ham 88; unsure spam 230
->     fp rate 0.05%; fn rate 0.44%; unsure rate 1.59%
-> <stat> all runs false positives: 0
-> <stat> all runs false negatives: 182
-> <stat> all runs unsure: 1430
-> <stat> all runs false positive %: 0.0
-> <stat> all runs false negative %: 1.82
-> <stat> all runs unsure %: 7.15
-> <stat> all runs cost: $468.00
Total messages: 20000; 10000 (50.0%) ham + 10000 (50.0%) spam
Ham: 9990 (99.90%) ok, 10 (0.10%) unsure, 0 (0.00%) fp
Spam: 8398 (83.98%) ok, 1420 (14.20%) unsure, 182 (1.82%) fn
Score False: 0.91% Unsure 7.15%
Standard Cost: $468.0000
Flex Cost: $724.4078
Flex**2 Cost: $498.5645
Delayed-Total messages: 20000; 10000 (50.0%) ham + 10000 (50.0%) spam
Delayed-Ham: 9907 (99.07%) ok, 88 (0.88%) unsure, 5 (0.05%) fp
Delayed-Spam: 9726 (97.26%) ok, 230 (2.30%) unsure, 44 (0.44%) fn
Delayed-Score False: 0.24% Unsure 1.59%
Delayed-Standard Cost: $157.6000
Delayed-Flex Cost: $365.5402
Delayed-Flex**2 Cost: $237.8641