From montanaro at users.sourceforge.net  Tue Sep 21 06:00:57 2010
From: montanaro at users.sourceforge.net (montanaro at users.sourceforge.net)
Date: Tue, 21 Sep 2010 04:00:57 +0000
Subject: [Spambayes-checkins] SF.net SVN: spambayes:[3268]
	trunk/spambayes/contrib/tte.py
Message-ID: <E1Oxu2L-0000YT-20@sfp-svn-4.v30.ch3.sourceforge.com>

Revision: 3268
          http://spambayes.svn.sourceforge.net/spambayes/?rev=3268&view=rev
Author:   montanaro
Date:     2010-09-21 04:00:56 +0000 (Tue, 21 Sep 2010)

Log Message:
-----------
Try to catch problems with message processing.

Modified Paths:
--------------
    trunk/spambayes/contrib/tte.py

Modified: trunk/spambayes/contrib/tte.py
===================================================================
--- trunk/spambayes/contrib/tte.py	2010-04-02 02:22:46 UTC (rev 3267)
+++ trunk/spambayes/contrib/tte.py	2010-09-21 04:00:56 UTC (rev 3268)
@@ -152,8 +152,12 @@
             sys.stdout.flush()
 
             tokens = list(tokenize(train_msg))
-            score = store.spamprob(tokens)
             selector = train_msg["message-id"] or train_msg["subject"]
+            try:
+                score = store.spamprob(tokens)
+            except UnicodeDecodeError:
+                print >> sys.stderr, "Unicode error while processing", selector
+                continue
 
             if misclassified(train_spam, score) and selector is not None:
                 if verbose:


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.