[Spambayes-checkins] spambayes/spambayes tokenizer.py,1.36,1.37

[Tony Meyer]

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv5584/spambayes

Modified Files:
	tokenizer.py 
Log Message:
Make it easier to subclass and use options in testing.

Index: tokenizer.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/tokenizer.py,v
retrieving revision 1.36
retrieving revision 1.37
diff -C2 -d -r1.36 -r1.37
*** tokenizer.py	26 May 2005 01:38:42 -0000	1.36
--- tokenizer.py	15 Nov 2005 00:16:20 -0000	1.37
***************
*** 1224,1227 ****
--- 1224,1235 ----
  
      def __init__(self):
+         self.setup()
+ 
+     def setup(self):
+         """Get the tokenizer ready to use; this should be called after
+         all options have been set."""
+         # We put this here, rather than in __init__, so that this can be
+         # done after we set options at runtime (since the tokenizer
+         # instance is generally created when this module is imported).
          if options["Tokenizer", "basic_header_tokenize"]:
              self.basic_skip = [re.compile(s)
***************
*** 1609,1611 ****
                          yield t
  
! tokenize = Tokenizer().tokenize
--- 1617,1620 ----
                          yield t
  
! global_tokenizer = Tokenizer()
! tokenize = global_tokenizer.tokenize