From mhammond at users.sourceforge.net  Tue Dec  2 01:05:28 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Tue Dec  2 01:05:31 2003
Subject: [Spambayes-checkins] spambayes/scripts sb_server.py,1.11,1.12
Message-ID: <E1AR3f6-0006Vm-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/scripts
In directory sc8-pr-cvs1:/tmp/cvs-serv25000

Modified Files:
	sb_server.py 
Log Message:
As reported by Richie in Email, sb_server ignored command-line args


Index: sb_server.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/scripts/sb_server.py,v
retrieving revision 1.11
retrieving revision 1.12
diff -C2 -d -r1.11 -r1.12
*** sb_server.py	7 Oct 2003 00:36:30 -0000	1.11
--- sb_server.py	2 Dec 2003 06:05:25 -0000	1.12
***************
*** 847,851 ****
  
  def prepare(state):
-     state.init()
      state.prepare()
      # Launch any SMTP proxies.  Note that if the user hasn't specified any
--- 847,850 ----



From mhammond at users.sourceforge.net  Tue Dec  2 19:34:19 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Tue Dec  2 19:34:22 2003
Subject: [Spambayes-checkins] spambayes/spambayes/test sb_test_support.py,
	NONE, 1.1 test_programs.py, NONE, 1.1
Message-ID: <E1ARKyB-0001Pv-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes/test
In directory sc8-pr-cvs1:/tmp/cvs-serv5358

Added Files:
	sb_test_support.py test_programs.py 
Log Message:
New sb_test_support file for utility functions used by test cases.
test_programs attempts to start and stop a number of programs - currently
sb_server.py, sb_server.py with a custom UI port, and the windows service.


--- NEW FILE: sb_test_support.py ---
# General utilities for the SpamBayes test suite
#
#
import sys, os
import unittest

def fix_sys_path():
    # XXX - MarkH had the bright idea *after* writing this that we should
    # ensure the CVS version of SpamBayes is *not* used to resolve SB imports.
    # This would allow us to effectively test the distutils setup script, so
    # any modules or files missing from the installed version raise errors.
    """Fix sys.path so that the core SpamBayes package,
    *and* the SpamBayes scripts can be imported.
    """
    this_dir = os.path.dirname(__file__)
    try:
        import spambayes.Version
    except ImportError:
        # Apparently SpamBayes is yet to be "setup.py install"
        # We are in 'spambayes\spambayes\test' - 2 parents up should
        # do it.
        sb_dir = os.path.abspath(
                     os.path.join(this_dir, "..", ".."))
        sys.path.insert(0, sb_dir)
        import SpamBayes.Version

    # Now do the same for the sb_* scripts
    try:
        import sb_server
    except ImportError:
        # Scripts are usually in "spambayes/scripts" (for an 
        # installed SpamBayes, they appear to be in
        # os.path.join(sys.prefix(), "scripts"), which we may like to 
        # leverage - however, these test scripts are not currently
        # installed).
        script_dir = os.path.abspath(
                     os.path.join(this_dir, "..", "..", "scripts"))
        sys.path.insert(0, script_dir)
        import sb_server

# Entry point for all our 'simple' based test programs
def unittest_main(*args, **kwargs):
    # I bet one day this will be more than this <wink>
    unittest.main(*args, **kwargs)

--- NEW FILE: test_programs.py ---
import os
import sys
import errno
import unittest
import time

import sb_test_support
sb_test_support.fix_sys_path()

try:
    True;False
except NameError: # 2.2 compat
    True=(None is None);
    False=not True

import sb_server

from spambayes.Options import options
default_shutdown_port = options["html_ui", "port"]

verbose = 0

class Spawner:
    def __init__(self, test_case, spawn_args):
        self.test_case = test_case
        self.spawn_args = spawn_args
        # If the command is a .py file, insert an executable.
        if os.path.splitext(self.spawn_args[0])[1]=='.py':
            self.spawn_args.insert(0, sys.executable)
        self.pid = None
    def _spawn(self, args):
        return os.spawnv(os.P_NOWAIT, self.spawn_args[0], self.spawn_args)
    def start(self):
        raise NotImplementedError
    def stop(self):
        raise NotImplementedError
    def is_running(self):
        if self.pid is None:
            return False
        # damn - we could implement os.waitpid correctly
        # using the win32api - but as at 2.3, you can't check
        # if a pid is still running with os.waitpid
        if sys.platform.startswith("win32"):
            import win32process # sorry, ya gotta have win32all to run tests
            import win32con
            try:
                rc = win32process.GetExitCodeProcess(self.pid)
                result = rc==win32con.STILL_ACTIVE
            except win32process.error:
                result = False
        else:
            try:
                os.waitpid(self.pid, os.WNOHANG)
                result = True
            except os.error, details:
                if details.errno == errno.ECHILD:
                    result = False
                # other exceptions invalid?
                raise
        # Wait a few seconds for the global mutex to catch up
        for i in range(20):
            time.sleep(0.25)
            if result==is_any_sb_server_running():
                break
        # Check the platform agrees (could do xor, but even I wont be able
        # to read it in a few weeks <wink>
        if result:
            self.test_case.failUnless(is_any_sb_server_running(),
                    "My server stopped, but global server mutex held")
        else:
            self.test_case.failUnless(not is_any_sb_server_running(),
                    "My server running, but no global server mutex held")
        return result

class Spawner_sb_server(Spawner):
    def __init__(self, test_case, args, shutdown_port = default_shutdown_port):
        self.shutdown_port = shutdown_port
        f = sb_server.__file__
        if f.endswith(".pyc") or f.endswith(".pyo"):
            f = f[:-1]
        Spawner.__init__(self, test_case, [f]+args)
    def start(self):
        self.test_case.failUnless(not is_any_sb_server_running(),
                                  "Should be no server running")
        if verbose > 1:
            print "Spawning", self.spawn_args
        self.pid = self._spawn(self.spawn_args)
        # wait for it to start - 5 secs, 0.25 per check
        for i in range(20):
            time.sleep(0.25)
            if verbose > 1:
                print "Waiting for start flags: running=%s, global_mutex=%s" \
                       % (self.is_running(), is_any_sb_server_running())
            if self.is_running() and is_any_sb_server_running():
                return
        # gave up waiting.
        self.test_case.fail("sb_server appeared to not start")

    def stop(self):
        # Copied from sb_server.stop()
        # Shutdown as though through the web UI.  This will save the DB, allow
        # any open proxy connections to complete, etc.
        from urllib import urlopen, urlencode
        urlopen('http://localhost:%d/save' % self.shutdown_port,
                urlencode({'how': 'Save & shutdown'})).read()
        # wait for it to stop - 5 secs, 0.25 per check
        for i in range(20):
            time.sleep(0.25)
            if not self.is_running() and not is_any_sb_server_running():
                return
        # gave up waiting.
        self.test_case.fail("sb_server appeared to not stop")

def is_any_sb_server_running():
    # reach into sb_server internals, as it is authoritative (sometimes <wink>)
    try:
        mutex = sb_server.open_platform_mutex()
        sb_server.close_platform_mutex(mutex)
        return False
    except sb_server.AlreadyRunningException:
        return True
        
class TestServer(unittest.TestCase):
    def setUp(self):
        self.failUnless(not is_any_sb_server_running(),
                        "Can't do sb_server tests while a server is running "\
                        "(platform mutex held)")
    def tearDown(self):
        # If we cause failure here, we mask the underlying error which left
        # the server running - so just print the warning.
        if is_any_sb_server_running():
            print "WARNING:", self, "completed with the platform mutex held"
    def _start_spawner(self, spawner):
        self.failUnless(not spawner.is_running(),
                        "this spawneer can't be running")
        spawner.start()
        self.failUnless(spawner.is_running(),
                        "this spawner must be running after successful start")
        self.failUnless(is_any_sb_server_running(),
                "Platform mutex not held after starting")
    def _stop_spawner(self, spawner):
        self.failUnless(spawner.is_running(), "must be running to stop")
        self.failUnless(is_any_sb_server_running(),
                "Platform mutex must be held to stop")
        spawner.stop()
        self.failUnless(not spawner.is_running(), "didn't stop after stop")
        self.failUnless(not is_any_sb_server_running(),
                "Platform mutex still help after stop")
    def test_sb_server_default(self):
        # Should be using the default port from the options file.
        from spambayes.Options import options
        port = options["html_ui", "port"]
        s = Spawner_sb_server(self, [])
        self._start_spawner(s)
        self._stop_spawner(s)

    def test_sb_server_ui_port(self):
        # Should be using the default port from the options file.
        s = Spawner_sb_server(self, ["-u8899"], 8899)
        self._start_spawner(s)
        self._stop_spawner(s)

if sys.platform.startswith("win"):
    import win32service # You need win32all to run the tests!
    import win32serviceutil
    import winerror
    service_name = "pop3proxy"

    class TestService(unittest.TestCase):
        def setUp(self):
            try:
                win32serviceutil.QueryServiceStatus(service_name)
            except win32service.error, details:
                if details[0]==winerror.ERROR_SERVICE_DOES_NOT_EXIST:
                    self.was_installed = False
                raise
            else:
                self.was_installed = True
            self.failUnless(not is_any_sb_server_running(),
                            "Can't do service tests while a server is running "\
                            "(platform mutex held)")
        def tearDown(self):
            if is_any_sb_server_running():
                print "WARNING:", self, "completed with the platform mutex held"
        def _start_service(self):
            win32serviceutil.StartService(service_name)
            for i in range(10):
                time.sleep(0.5)
                status = win32serviceutil.QueryServiceStatus(service_name)
                if status[1] == win32service.SERVICE_RUNNING:
                    break
                if verbose > 1:
                    print "Service status is %d - still waiting" % status[1]
            else:
                self.fail("Gave up waiting for service to start")
        def _stop_service(self):
            # StopServiceWithDeps checks the status of each service as it
            # stops it, which is exactly what we want here.
            win32serviceutil.StopServiceWithDeps(service_name)

        def test_simple_startstop(self):
            self._start_service()
            self._stop_service()

        def test_remote_shutdown(self):
            self._start_service()
            # Should be using the default port from the options file.
            from spambayes.Options import options
            from urllib import urlopen, urlencode
            port = options["html_ui", "port"]
            urlopen('http://localhost:%d/save' % port,
                    urlencode({'how': 'Save & shutdown'})).read()
            # wait for it to stop - 5 secs, 0.25 per check
            for i in range(10):
                time.sleep(0.5)
                status = win32serviceutil.QueryServiceStatus(service_name)
                if status[1] == win32service.SERVICE_STOPPED:
                    break
            else:
                self.fail("Gave up waiting for service to stop")
            self.failUnless(not is_any_sb_server_running(),
                            "Should be no platform mutex held after stopping")
    
if __name__=='__main__':
    sb_test_support.unittest_main()



From mhammond at users.sourceforge.net  Tue Dec  2 19:35:18 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Tue Dec  2 19:35:23 2003
Subject: [Spambayes-checkins] spambayes/spambayes/test test_storage.py, 1.3,
	1.4
Message-ID: <E1ARKz8-0001St-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes/test
In directory sc8-pr-cvs1:/tmp/cvs-serv5530

Modified Files:
	test_storage.py 
Log Message:
Use sb_test_support utilities, and prevent a warning message we explicitly
test for from being written to stdout


Index: test_storage.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/test/test_storage.py,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** test_storage.py	11 Sep 2003 07:47:13 -0000	1.3
--- test_storage.py	3 Dec 2003 00:35:16 -0000	1.4
***************
*** 2,14 ****
  
  import unittest, os, sys
- 
- # Hack sys.path - 2 levels up must be on sys.path.
  try:
!     this_file = __file__
! except NameError:
!     this_file = sys.argv[0]
! sb_dir = os.path.abspath(os.path.dirname(os.path.dirname(os.path.dirname(this_file))))
! if sb_dir not in sys.path:
!     sys.path.append(sb_dir)
  
  from spambayes.storage import DBDictClassifier, PickledClassifier
--- 2,12 ----
  
  import unittest, os, sys
  try:
!     import cStringIO as StringIO
! except ImportError:
!     import StringIO
! 
! import sb_test_support
! sb_test_support.fix_sys_path()
  
  from spambayes.storage import DBDictClassifier, PickledClassifier
***************
*** 154,162 ****
          sys_exit = sys.exit
          sys.exit = self.success
          self.succeeded = False
          db_name = tempfile.mktemp("nodbmtest")
!         s = open_storage(db_name, True)
!         DBDictClassifier.load = DBDictClassifier_load
!         sys.exit = sys_exit
          if not self.succeeded:
              self.fail()
--- 152,168 ----
          sys_exit = sys.exit
          sys.exit = self.success
+         # redirect sys.stderr and sys.exit, as storage.py prints
+         # an error to stderr, then attempts to sys.exit
+         # (we probably could just catch SystemExit for sys.exit?)
+         sys_stderr = sys.stderr
+         sys.stderr = StringIO.StringIO()
          self.succeeded = False
          db_name = tempfile.mktemp("nodbmtest")
!         try:
!             s = open_storage(db_name, True)
!         finally:
!             DBDictClassifier.load = DBDictClassifier_load
!             sys.exit = sys_exit
!             sys.stderr = sys_stderr
          if not self.succeeded:
              self.fail()
***************
*** 172,174 ****
  
  if __name__=='__main__':
!     unittest.main(argv=sys.argv + ['suite'])
--- 178,180 ----
  
  if __name__=='__main__':
!     sb_test_support.unittest_main(argv=sys.argv + ['suite'])



From mhammond at users.sourceforge.net  Tue Dec  2 19:36:00 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Tue Dec  2 19:36:02 2003
Subject: [Spambayes-checkins] 
	spambayes/spambayes/test test_sb-server.py, 1.4, 1.5
Message-ID: <E1ARKzo-0001U0-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes/test
In directory sc8-pr-cvs1:/tmp/cvs-serv5670

Modified Files:
	test_sb-server.py 
Log Message:
sys.path munging didn't work for me, so I replaced it with a 
sb_test_support utility function.


Index: test_sb-server.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/test/test_sb-server.py,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** test_sb-server.py	19 Sep 2003 22:47:13 -0000	1.4
--- test_sb-server.py	3 Dec 2003 00:35:58 -0000	1.5
***************
*** 82,93 ****
  import re
  import getopt
  
! # We need to import sb_server, but it may not be on the PYTHONPATH.
! # Hack around this, so that if we are running in a cvs-like setup
! # everything still works.
! import sys
! import os
! sys.path.insert(-1, os.getcwd())
! sys.path.insert(-1, os.path.join(os.getcwd(), "scripts"))
  
  from spambayes import Dibbler
--- 82,89 ----
  import re
  import getopt
+ import sys, os
  
! import sb_test_support
! sb_test_support.fix_sys_path()
  
  from spambayes import Dibbler



From anadelonbrin at users.sourceforge.net  Tue Dec  2 22:29:28 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Tue Dec  2 22:29:33 2003
Subject: [Spambayes-checkins] website faq.txt,1.51,1.52
Message-ID: <E1ARNhg-00009L-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/website
In directory sc8-pr-cvs1:/tmp/cvs-serv564

Modified Files:
	faq.txt 
Log Message:
Two new FAQs:

Missing mail: it's odd that this is very suddenly a FAQ, and nowhere near a release.
I'm confident that it's not the plug-in, though, so that's what I've said.  I've put some
general 'how to find mail' tips there to try and help people along.
(Anyone else wondering if ISPs are to blame because they've put in too stringent
filters to try and block all the worm crud recently?)

Uninstall:  how to uninstall the plug-in, including removing the toolbar (the oldest
still-open bug!).  I'm pretty sure that this is the current situation...


Index: faq.txt
===================================================================
RCS file: /cvsroot/spambayes/website/faq.txt,v
retrieving revision 1.51
retrieving revision 1.52
diff -C2 -d -r1.51 -r1.52
*** faq.txt	5 Nov 2003 12:11:04 -0000	1.51
--- faq.txt	3 Dec 2003 03:29:26 -0000	1.52
***************
*** 601,604 ****
--- 601,672 ----
  
  
+ Some of my mail is going missing!
+ ---------------------------------
+ 
+ The plug-in *can not* delete your mail - if it arrived in Outlook, then
+ either it's still there, or something else removed it.
+ 
+ Check that the mail isn't in your 'Spam' folder *or* your 'Unsure'
+ folder, and that you are checking the correct folders (i.e. go to the
+ 'Filtering' tab of the SpamBayes dialog, and click on 'Browse' next to
+ the two movement options and check that you're moving to the folders you
+ think that you are).
+ 
+ You can try to search for the mail, too, using Outlook's 'Advanced Find'
+ (in the 'Tools' menu).  Even if you don't know the subject of the 'missing'
+ mail, you can search for all messages that have arrived since a certain date
+ or time.
+ 
+ Sometimes mail is in the Inbox (or another folder), but not visible.  This,
+ too, isn't SpamBayes's doing.  You can set Outlook up to only show a subset
+ of the messages in the folder, with a "View Filter".  The Outlook documentation
+ has more information about this (although the 'Advanced Find' will still
+ show the message).
+ 
+ Note that if you are using sb_server to receive your mail, then (1) you're
+ reading the wrong section of the FAQ <wink>, and (2) there is a slim chance
+ that SpamBayes is at fault - please use the normal bug reporting procedures.
+ 
+ 
+ How do I uninstall the plug-in?
+ -------------------------------
+ 
+ Note that if you simply want to disable the plug-in for a while, you can
+ do this by unticking the "Enable SpamBayes" box on the front page of the
+ "Manager" dialog.
+ 
+ If you installed the plug-in from source, then you simply need to re-run the
+ "addin.py" script, with the parameter "--unregister", and then delete the
+ toolbar in Outlook.
+ 
+ Otherwise, you uninstall the plug-in like you uninstall (almost) any other
+ Windows program, using the "Add/Remove Programs" Control Panel.  You'll see
+ the plug-in listed under "S" for "SpamBayes".  This will stop remove all the
+ program files that the plug-in installed, and stop the plug-in from working.
+ It would be a good idea to do this while Outlook is *not* running.
+ 
+ Note that this does *not* remove your personal setup files (deliberately).
+ This includes your databases and configuration files.  This means that if you
+ later reinstall the plug-in (or a later version), those files will still be
+ there, ready to use.  If you wish to remove these as well, you can remove them
+ like you would delete any other file (move them to the Recycle Bin).  The
+ `backup question`_ explains where you can find these files.
+ 
+ Note that `a bug with the plug-in`_ means that the SpamBayes toolbar is not
+ deleted on uninstall - although it *will* stop working.  You can delete this
+ yourself in Outlook: right-click on the toolbar, choose "Customize", then
+ select the SpamBayes toolbar and click "Delete".
+ 
+ Recently, there have been reports that the uninstaller has left some entries
+ in the registry - if this is the case with you, please add to that `bug tracker`_;
+ the next release will hopefully resolve this issue.  These should be harmless,
+ but if they bother you (and you are confident mucking about with the registry,
+ which we do *not* recommend), then you can remove those keys yourself.
+ 
+ .. _`backup question`: #can-i-back-up-the-outlook-database-should-i-do-this
+ .. _`bug tracker`: http://sourceforge.net/tracker/index.php?func=detail&aid=832183&group_id=61702&atid=498103
+ .. _`a bug with the plug-in`: http://sourceforge.net/tracker/index.php?func=detail&aid=675811&group_id=61702&atid=498103
+ 
+ 
  Using SpamBayes
  ===============



From anadelonbrin at users.sourceforge.net  Wed Dec  3 18:45:41 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Wed Dec  3 18:45:43 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000/dialogs/resources
	dialogs.rc, 1.40, 1.41
Message-ID: <E1ARggf-0007O1-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000/dialogs/resources
In directory sc8-pr-cvs1:/tmp/cvs-serv28382/Outlook2000/dialogs/resources

Modified Files:
	dialogs.rc 
Log Message:
Add a period to make the dialog consistent.

Index: dialogs.rc
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/dialogs/resources/dialogs.rc,v
retrieving revision 1.40
retrieving revision 1.41
diff -C2 -d -r1.40 -r1.41
*** dialogs.rc	29 Sep 2003 02:14:26 -0000	1.40
--- dialogs.rc	3 Dec 2003 23:45:38 -0000	1.41
***************
*** 520,524 ****
                      SS_CENTERIMAGE | SS_SUNKEN,11,21,175,12
      PUSHBUTTON      "&Browse...",IDC_BROWSE_HAM,192,20,50,14
!     LTEXT           "Folders with spam or other junk messages",IDC_STATIC,11,
                      36,171,9
      CONTROL         "Static",IDC_STATIC_SPAM,"Static",SS_LEFTNOWORDWRAP | 
--- 520,524 ----
                      SS_CENTERIMAGE | SS_SUNKEN,11,21,175,12
      PUSHBUTTON      "&Browse...",IDC_BROWSE_HAM,192,20,50,14
!     LTEXT           "Folders with spam or other junk messages.",IDC_STATIC,11,
                      36,171,9
      CONTROL         "Static",IDC_STATIC_SPAM,"Static",SS_LEFTNOWORDWRAP | 



From anadelonbrin at users.sourceforge.net  Wed Dec  3 21:57:45 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Wed Dec  3 21:57:49 2003
Subject: [Spambayes-checkins] spambayes/windows pop3proxy_tray.py,1.15,1.16
Message-ID: <E1ARjgX-0000JA-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/windows
In directory sc8-pr-cvs1:/tmp/cvs-serv1181/windows

Modified Files:
	pop3proxy_tray.py 
Log Message:
Remove an old comment.

Change the default (double-click) behaviour of the tray to "review messages" rather
than "display information".  It seems likely that this would be the most commonly
needed action.

Since 15/10/03, SetDefaultItem has been available for menus in win32all, so use that
as we should.  (It appears to set the font of the item correctly, but not have any effect
in terms of action, so still capture the double-click ourselves.  Someone correct me if
I've done this wrongly).

Index: pop3proxy_tray.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/windows/pop3proxy_tray.py,v
retrieving revision 1.15
retrieving revision 1.16
diff -C2 -d -r1.15 -r1.16
*** pop3proxy_tray.py	25 Sep 2003 00:10:32 -0000	1.15
--- pop3proxy_tray.py	4 Dec 2003 02:57:43 -0000	1.16
***************
*** 166,172 ****
          stoppedIconPathName = "%s\\..\\windows\\resources\\sb-stopped.ico" % \
                         (os.path.dirname(sb_server.__file__),)
-         # When 1.0a6 is released, the above line will need to change to:
- ##        iconPathName = "%s\\..\\windows\\resources\\sbicon.ico" % \
- ##                       (os.path.dirname(sb_server.__file__),)
          if hasattr(sys, "frozen"):
              self.hstartedicon = self.hstoppedicon = None
--- 166,169 ----
***************
*** 352,356 ****
              # XXX text) through the win32 calls, but win32all doesn't
              # XXX include SetDefault(), which it needs to...
!             self.OpenInterface()
          elif lparam==win32con.WM_RBUTTONUP:
              # check our state before creating the menu, so it reflects the
--- 349,353 ----
              # XXX text) through the win32 calls, but win32all doesn't
              # XXX include SetDefault(), which it needs to...
!             self.OpenReview()
          elif lparam==win32con.WM_RBUTTONUP:
              # check our state before creating the menu, so it reflects the
***************
*** 368,371 ****
--- 365,378 ----
              pos = GetCursorPos()
              SetForegroundWindow(self.hwnd)
+             # Set the default menu item ("Review messages", currently).
+             # Make sure that the item here matches the behaviour in
+             # DBCLICK above!
+             # This is only available in recent versions of win32all,
+             # so for those that don't have it, they just get a dull
+             # menu.
+             try:
+                 SetMenuDefaultItem(menu, 2, 1)
+             except NameError:
+                 pass
              TrackPopupMenu(menu, win32con.TPM_LEFTALIGN, pos[0], pos[1], 0,
                             self.hwnd, None)



From anadelonbrin at users.sourceforge.net  Wed Dec  3 22:12:44 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Wed Dec  3 22:12:48 2003
Subject: [Spambayes-checkins] spambayes/windows spambayes.iss,1.2,1.3
Message-ID: <E1ARjv2-0000oh-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/windows
In directory sc8-pr-cvs1:/tmp/cvs-serv3138/windows

Modified Files:
	spambayes.iss 
Log Message:
These dlls end up in the lib directory here, and I'm pretty sure that that's where
they're meant to be these days.  An old .iss, maybe?

Index: spambayes.iss
===================================================================
RCS file: /cvsroot/spambayes/spambayes/windows/spambayes.iss,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** spambayes.iss	23 Oct 2003 22:54:09 -0000	1.2
--- spambayes.iss	4 Dec 2003 03:12:41 -0000	1.3
***************
*** 19,24 ****
  Source: "py2exe\dist\lib\*.*"; DestDir: "{app}\lib"; Flags: ignoreversion
  Source: "py2exe\dist\bin\python23.dll"; DestDir: "{app}\bin"; Flags: ignoreversion
! Source: "py2exe\dist\bin\pythoncom23.dll"; DestDir: "{app}\bin"; Flags: ignoreversion
! Source: "py2exe\dist\bin\PyWinTypes23.dll"; DestDir: "{app}\bin"; Flags: ignoreversion
  
  Source: "py2exe\dist\bin\outlook_addin.dll"; DestDir: "{app}\bin"; Check: InstallingOutlook; Flags: ignoreversion regserver
--- 19,24 ----
  Source: "py2exe\dist\lib\*.*"; DestDir: "{app}\lib"; Flags: ignoreversion
  Source: "py2exe\dist\bin\python23.dll"; DestDir: "{app}\bin"; Flags: ignoreversion
! Source: "py2exe\dist\lib\pythoncom23.dll"; DestDir: "{app}\bin"; Flags: ignoreversion
! Source: "py2exe\dist\lib\PyWinTypes23.dll"; DestDir: "{app}\bin"; Flags: ignoreversion
  
  Source: "py2exe\dist\bin\outlook_addin.dll"; DestDir: "{app}\bin"; Check: InstallingOutlook; Flags: ignoreversion regserver



From anadelonbrin at users.sourceforge.net  Wed Dec  3 23:29:49 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Wed Dec  3 23:29:52 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000/sandbox mapi_driver.py,
	1.5, 1.6
Message-ID: <E1ARl7d-0004NR-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000/sandbox
In directory sc8-pr-cvs1:/tmp/cvs-serv16818/Outlook2000/sandbox

Modified Files:
	mapi_driver.py 
Log Message:
If the *first* name returned wasn't the default store, then dump_props
would say that there wasn't one.  Fix that.

Index: mapi_driver.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/sandbox/mapi_driver.py,v
retrieving revision 1.5
retrieving revision 1.6
diff -C2 -d -r1.5 -r1.6
*** mapi_driver.py	1 Sep 2003 05:36:34 -0000	1.5
--- mapi_driver.py	4 Dec 2003 04:29:47 -0000	1.6
***************
*** 81,90 ****
          names = [n.lower() for n in name.split("\\")]
          if names[0]:
              for store, name, is_default in self.GetMessageStores():
                  if is_default:
                      store_name = name.lower()
                      break
!                 else:
!                     raise RuntimeError, "Can't find a default message store"
              folder_names = names
          else:
--- 81,91 ----
          names = [n.lower() for n in name.split("\\")]
          if names[0]:
+             store_name = None
              for store, name, is_default in self.GetMessageStores():
                  if is_default:
                      store_name = name.lower()
                      break
!             if store_name is None:
!                 raise RuntimeError, "Can't find a default message store"
              folder_names = names
          else:



From mhammond at users.sourceforge.net  Thu Dec  4 23:41:28 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Thu Dec  4 23:41:31 2003
Subject: [Spambayes-checkins] spambayes/spambayes ImapUI.py, 1.23,
	1.24 __init__.py, 1.7, 1.8
Message-ID: <E1AS7mS-0008Fr-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv31600

Modified Files:
	ImapUI.py __init__.py 
Log Message:
Merging old 1.0 trunk onto the branch.


Index: ImapUI.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/ImapUI.py,v
retrieving revision 1.23
retrieving revision 1.24
diff -C2 -d -r1.23 -r1.24
*** ImapUI.py	6 Oct 2003 01:24:12 -0000	1.23
--- ImapUI.py	5 Dec 2003 04:41:25 -0000	1.24
***************
*** 168,191 ****
          self._writePreamble("Select Filter Folders")
          self._login_to_imap()
!         available_folders = self.imap.folder_list()
!         content = self.html.configForm.clone()
!         content.configFormContent = ""
!         content.introduction = """This page allows you to change which
!         folders are filtered, and where filtered mail ends up."""
!         content.config_submit.value = "Save Filter Folders"
!         content.optionsPathname = optionsPathname
  
!         for opt in ("unsure_folder", "spam_folder",
!                     "filter_folders"):
!             folderBox = self._buildFolderBox("imap", opt, available_folders)
!             content.configFormContent += folderBox
  
!         self.write(content)
!         self._writePostamble()
  
      def _login_to_imap(self):
          if self.imap_logged_in:
              return
!         if self.imap is None:
              server = options["imap", "server"][0]
              if server.find(':') > -1:
--- 168,192 ----
          self._writePreamble("Select Filter Folders")
          self._login_to_imap()
!         if self.imap_logged_in:
!             available_folders = self.imap.folder_list()
!             content = self.html.configForm.clone()
!             content.configFormContent = ""
!             content.introduction = """This page allows you to change which
!             folders are filtered, and where filtered mail ends up."""
!             content.config_submit.value = "Save Filter Folders"
!             content.optionsPathname = optionsPathname
  
!             for opt in ("unsure_folder", "spam_folder",
!                         "filter_folders"):
!                 folderBox = self._buildFolderBox("imap", opt, available_folders)
!                 content.configFormContent += folderBox
  
!             self.write(content)
!             self._writePostamble()
  
      def _login_to_imap(self):
          if self.imap_logged_in:
              return
!         if self.imap is None and len(options["imap", "server"]) > 0:
              server = options["imap", "server"][0]
              if server.find(':') > -1:
***************
*** 217,235 ****
          self._writePreamble("Select Training Folders")
          self._login_to_imap()
!         available_folders = self.imap.folder_list()
!         content = self.html.configForm.clone()
!         content.configFormContent = ""
!         content.introduction = """This page allows you to change which
!         folders contain mail to train Spambayes."""
!         content.config_submit.value = "Save Training Folders"
!         content.optionsPathname = optionsPathname
  
!         for opt in ("ham_train_folders",
!                     "spam_train_folders"):
!             folderBox = self._buildFolderBox("imap", opt, available_folders)
!             content.configFormContent += folderBox
  
!         self.write(content)
!         self._writePostamble()
  
      def onChangeopts(self, **parms):
--- 218,237 ----
          self._writePreamble("Select Training Folders")
          self._login_to_imap()
!         if self.imap_logged_in:
!             available_folders = self.imap.folder_list()
!             content = self.html.configForm.clone()
!             content.configFormContent = ""
!             content.introduction = """This page allows you to change which
!             folders contain mail to train Spambayes."""
!             content.config_submit.value = "Save Training Folders"
!             content.optionsPathname = optionsPathname
  
!             for opt in ("ham_train_folders",
!                         "spam_train_folders"):
!                 folderBox = self._buildFolderBox("imap", opt, available_folders)
!                 content.configFormContent += folderBox
  
!             self.write(content)
!             self._writePostamble()
  
      def onChangeopts(self, **parms):

Index: __init__.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/__init__.py,v
retrieving revision 1.7
retrieving revision 1.8
diff -C2 -d -r1.7 -r1.8
*** __init__.py	19 Sep 2003 01:04:16 -0000	1.7
--- __init__.py	5 Dec 2003 04:41:25 -0000	1.8
***************
*** 1,3 ****
  # package marker.
  
! __version__ = '1.0a6'
--- 1,3 ----
  # package marker.
  
! __version__ = '1.0a7'



From mhammond at users.sourceforge.net  Thu Dec  4 23:45:48 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Thu Dec  4 23:45:52 2003
Subject: [Spambayes-checkins] spambayes README.txt, 1.60,
	1.61 README-DEVEL.txt, 1.10, 1.11 WHAT_IS_NEW.txt, 1.20, 1.21
Message-ID: <E1AS7qe-0008Oh-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv32249

Modified Files:
	README.txt README-DEVEL.txt WHAT_IS_NEW.txt 
Log Message:
Merging old 1.0 branch


Index: README.txt
===================================================================
RCS file: /cvsroot/spambayes/spambayes/README.txt,v
retrieving revision 1.60
retrieving revision 1.61
diff -C2 -d -r1.60 -r1.61
*** README.txt	17 Nov 2003 21:47:47 -0000	1.60
--- README.txt	5 Dec 2003 04:45:43 -0000	1.61
***************
*** 123,127 ****
  All you need to do to configure SpamBayes is to open a web page to
  <http://localhost:8880>, click on the "Configuration" link at the top
! right, and fill in the relevant details.  Everything should be ok with the
  defaults, except for the POP3 and SMTP server information at the top, which
  is required.  Note that *nix users may not have permission to bind ports
--- 123,127 ----
  All you need to do to configure SpamBayes is to open a web page to
  <http://localhost:8880>, click on the "Configuration" link at the top
! right, and fill in the relevant details.  Everything should be OK with the
  defaults, except for the POP3 and SMTP server information at the top, which
  is required.  Note that *nix users may not have permission to bind ports
***************
*** 146,150 ****
  To configure SpamBayes, run "sb_imapfilter.py -b", which should open a web
  page to <http://localhost:8880>, click on the "Configuration" link at the
! top right, and fill in the relevant details.  Everything should be ok with
  the defaults, except for the server information at the top.
  
--- 146,150 ----
  To configure SpamBayes, run "sb_imapfilter.py -b", which should open a web
  page to <http://localhost:8880>, click on the "Configuration" link at the
! top right, and fill in the relevant details.  Everything should be OK with
  the defaults, except for the server information at the top.
  
***************
*** 186,190 ****
  
  (Replace the latter with the location of the .hammiedb file that
! sb_filter created in the first step).
  
  Once you've trained SpamBayes on your
--- 186,190 ----
  
  (Replace the latter with the location of the .hammiedb file that
! sb_filter.py created in the first step).
  
  Once you've trained SpamBayes on your
***************
*** 203,210 ****
  will still complain if you don't specify a lock file.)
  
! The result of running sb_filter.py in filter mode is that Procmail will use the
! output from the run as the mail message for further processing downstream.
! sb_filter.py inserts an X-SpamBayes-Classification header in the output message
! which looks like:
  
      X-SpamBayes-Classification: ham; 0.00; '*H*': 1.00; '*S*': 0.00; 'python': 0.00;
--- 203,210 ----
  will still complain if you don't specify a lock file.)
  
! The result of running sb_filter.py in filter mode is that Procmail will
! use the output from the run as the mail message for further processing
! downstream. sb_filter.py inserts an X-SpamBayes-Classification header in
! the output message which looks like:
  
      X-SpamBayes-Classification: ham; 0.00; '*H*': 1.00; '*S*': 0.00; 'python': 0.00;
***************
*** 267,274 ****
  same number of spams as hams.
  
! You can train it on lots of messages in one go by either using the Hammie
! script as explained in the "Command-line training" section, or by giving
! messages to the web interface via the "Train" form on the Home page.  You
! can train on individual messages (which is tedious) or using mbox files.
  
  
--- 267,275 ----
  same number of spams as hams.
  
! You can train it on lots of messages in one go by either using the
! sb_filter.py script as explained in the "Command-line training" section,
! or by giving messages to the web interface via the "Train" form on the
! Home page. You can train on individual messages (which is tedious) or
! using mbox files.
  
  
***************
*** 385,389 ****
     contain the word Viagra"?), find particular messages, and most
     importantly, train it on the emails you've received.  When you start
!    using the system, unless you train it using the Hammie script it will
     classify most things as Unsure, and often make mistakes.  But it keeps
     copies of all the email's its seen, and through the web interface you
--- 386,390 ----
     contain the word Viagra"?), find particular messages, and most
     importantly, train it on the emails you've received.  When you start
!    using the system, unless you train it using the sb_filter script it will
     classify most things as Unsure, and often make mistakes.  But it keeps
     copies of all the email's its seen, and through the web interface you
***************
*** 402,411 ****
     easy to file all the suspected spam into its own folder, for instance.
  
!  o The Hammie script.  This does three jobs: command-line training,
     procmail filtering, and XML-RPC.  See below for details of how to use
!    Hammie for training, and how to use it as procmail filter.  Hammie can
!    also run as an XML-RPC server, so that a programmer can write code that
     uses a remote server to classify emails programmatically - see
!    hammiesrv.py.
  
   o The IMAP filter.  This is a cross between the POP3 proxy and the Outlook
--- 403,412 ----
     easy to file all the suspected spam into its own folder, for instance.
  
!  o The sb_filter.py script.  This does three jobs: command-line training,
     procmail filtering, and XML-RPC.  See below for details of how to use
!    sb_filter for training, and how to use it as procmail filter. You  can
!    also run an XML-RPC server, so that a programmer can write code that
     uses a remote server to classify emails programmatically - see
!    sb_xmlrpcserver.py.
  
   o The IMAP filter.  This is a cross between the POP3 proxy and the Outlook

Index: README-DEVEL.txt
===================================================================
RCS file: /cvsroot/spambayes/spambayes/README-DEVEL.txt,v
retrieving revision 1.10
retrieving revision 1.11
diff -C2 -d -r1.10 -r1.11
*** README-DEVEL.txt	5 Nov 2003 12:50:13 -0000	1.10
--- README-DEVEL.txt	5 Dec 2003 04:45:44 -0000	1.11
***************
*** 77,80 ****
--- 77,86 ----
  following section refers to old application names.
  
+ IMPORTANT NOTE
+ ==============
+ 
+ The applications have all been renamed in preparation for 1.0 - the
+ following section refers to old application names.
+ 
  Apps
  ====
***************
*** 246,250 ****
      currently asking for.  The idea is, if you have a standard directory
      structure (below), you can run this thing, go have some tea while it
!     works, then paste the output to the spambayes list for good karma.
  
  
--- 252,256 ----
      currently asking for.  The idea is, if you have a standard directory
      structure (below), you can run this thing, go have some tea while it
!     works, then paste the output to the SpamBayes list for good karma.
  
  
***************
*** 507,518 ****
  Anthony's Alternate Approach to Building the Zipfile
  
! o Unpack the tarball somewhere, making a spambayes-1.0a7 directory
!   (version number will obviously change in future releases)
! o Run the following two commands:
  
!     find spambayes-1.0a7 -type f -name '*.txt' | xargs zip -l sb107.zip 
!     find spambayes-1.0a7 -type f \! -name '*.txt' | xargs zip sb107.zip 
  
! o This makes a tarball where the .txt files are mangled, but everything
!   else is left alone.
  
--- 513,524 ----
  Anthony's Alternate Approach to Building the Zipfile
  
!  o Unpack the tarball somewhere, making a spambayes-1.0a7 directory
!    (version number will obviously change in future releases)
!  o Run the following two commands:
  
!      find spambayes-1.0a7 -type f -name '*.txt' | xargs zip -l sb107.zip 
!      find spambayes-1.0a7 -type f \! -name '*.txt' | xargs zip sb107.zip 
  
!  o This makes a tarball where the .txt files are mangled, but everything
!    else is left alone.
  

Index: WHAT_IS_NEW.txt
===================================================================
RCS file: /cvsroot/spambayes/spambayes/WHAT_IS_NEW.txt,v
retrieving revision 1.20
retrieving revision 1.21
diff -C2 -d -r1.20 -r1.21
*** WHAT_IS_NEW.txt	5 Nov 2003 12:50:13 -0000	1.20
--- WHAT_IS_NEW.txt	5 Dec 2003 04:45:44 -0000	1.21
***************
*** 10,14 ****
  noted in the "Transition" section.
  
! New in Alpha Release 6
  ======================
  
--- 10,14 ----
  noted in the "Transition" section.
  
! New in Alpha Release 7
  ======================
  
***************
*** 17,22 ****
  --------------------------
  
! There are two major changes in this release, which *will* effect you if you
! are upgrading from an older version:
  
   o The scripts have all moved (in the archive), and their names have been
--- 17,33 ----
  --------------------------
  
!  o If you are using a pickle for storage, your 'message info' database
!    would previously still have been a dbm (where available).  This is
!    no longer the case - if you are using a pickle for the statistics
!    database, you have a pickle for everything.  Your old 'message info'
!    database is not converted (and there is no utility provided to do so).
!    If you are using a pickle for storage, you should delete your old
!    'spambayes.messageinfo.db' file before restarting after the upgrade.
!    You should not suffer any ill effects from this, *unless* you are
!    using sb_imapfilter.py.  In that case, you will find that the filter
!    trains and classifies all messages in the folders it examines, even
!    if it has seen them before - this will only occur once, however.
! 
! There should be no other incompatible changes (from 1.0a6) in this release.
  
   o The scripts have all moved (in the archive), and their names have been
***************
*** 74,106 ****
  POP3 Proxy / SMTP Proxy
  -----------------------
!  o If running Windows, and have the win32 extensions installed, and a
!    configuration file cannot be found, then default to placing it in the
!    "Application Data" directory (as defined by Windows).  Also default to
!    storing the caches and databases in this directory.
!  o Correctly save and close the database when changing options.  This
!    fixes a bug for those using gdbm databases.
  
  Web Interface
  -------------
!  o Place a threshold on the number of items displayed per section in the
!    review page.
!  o Correctly display the proxy data when it is specified on the command
!    line.
  
  POP3 Proxy Service / POP3 Proxy Tray Application
  ------------------------------------------------
  
!  o The POP3 proxy tray application (for Windows users), which quietly
!    appeared in the 1.0a5 release, should be ready for use.  If you have
!    the service installed, it will attempt to use that; if you do not,
!    it will start up sb_server for you.
!    ** Note that the tray application currently only works with Windows
!    NT, Windows 2000 and Windows XP, not Windows 95, 98, or ME.  This
!    will be remedied in a future version, and in the binary release of
!    the tray application. **
!  o As the service starts, it reports the username and configuration file
!    that it is using.
!  o Added a version check option to the tray application.
!  o Improved logged for both the service and the tray application.
  
  IMAP Filter
--- 85,103 ----
  POP3 Proxy / SMTP Proxy
  -----------------------
!  o An error where a failure message would be printed by
!    the SMTP proxy, even on success, was fixed.
  
  Web Interface
  -------------
!  o The bug which caused the "TypeError" when trying to access
!    the database after setting a configuration option via the
!    interface has been fixed.
  
  POP3 Proxy Service / POP3 Proxy Tray Application
  ------------------------------------------------
  
!  o Both the pop3proxy_service.py and pop3proxy_tray.py
!    scripts are now installed (with "setup.py install") if
!    the user is using Windows.
  
  IMAP Filter
***************
*** 117,137 ****
  General
  -------
!  o Add a new file: NEWTRICKS.TXT to record ideas that have and haven't
!    been tried.  If you would like to add to this, please submit a patch
!    via the Sourceforge system.
!  o Change the default to tokenizer the "To", "CC", "Reply-To", "Sender",
!    and "From" address headers, rather than just "From".
!  o Fix the database opening code so that hammie* again works with both
!    dbm and pickles.
!  o Minor performance boosts to hammiebulk and mboxtrain.
! 
! Developer
! ---------
!  o Add a close method to the various storage classes.
!  o Removed the gary_combining option and code.
  
  Transition
  ==========
! If you are transitioning from a version older than 1.0a5, please also
  read the notes in the previous release notes (accessible from
  <http://sourceforge.net/project/showfiles.php?group_id=61702>).
--- 114,137 ----
  General
  -------
!  o Various improvements have been made to the management of the
!    'message info' database.  As outlined above, it will now be
!    stored as a pickle, if your statistics database uses a pickle.
!    In addition, we attempt to close the database when we should,
!    and make sure that we explicitly update it.  This should hopefully
!    go some way to solving the "DB_RUN_RECOVERY" errors that have
!    been regularly reported - we would be interested to hear from
!    you if upgrading to 1.0a7 does appear to solve this problem
!    for you (email spambayes@python.org).
!  o We now try to determine the type of dbm storage used from the
!    file, if one already exists.  This should make the transition
!    between formats a little easier.
!  o Fix sb_xmlrpcserver to work with the renamed (since 1.0a5)
!    scripts.
!  o Fix the sense of include_trained in sb_mboxtrain.
!    
  
  Transition
  ==========
! If you are transitioning from a version older than 1.0a6, please also
  read the notes in the previous release notes (accessible from
  <http://sourceforge.net/project/showfiles.php?group_id=61702>).
***************
*** 144,149 ****
  ===================
  The following bugs tracked via the Sourceforge system were fixed:
! 803501, 802545, 802347, 801952, 798362, 800555, 806632, 795145, 806238, 
! 805351
  
  A URL containing the details of these bugs can be made by appending the
--- 144,148 ----
  ===================
  The following bugs tracked via the Sourceforge system were fixed:
! 809769, 814322, 816400, 810342, 818552
  
  A URL containing the details of these bugs can be made by appending the



From mhammond at users.sourceforge.net  Fri Dec  5 00:12:35 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Fri Dec  5 00:12:37 2003
Subject: [Spambayes-checkins] website/apps/outlook bugs.txt,1.2,1.3
Message-ID: <E1AS8GZ-00010o-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/website/apps/outlook
In directory sc8-pr-cvs1:/tmp/cvs-serv3813

Modified Files:
	bugs.txt 
Log Message:
Add new Frequently Reported Bug - "Could not watch the specified folders" 
error


Index: bugs.txt
===================================================================
RCS file: /cvsroot/spambayes/website/apps/outlook/bugs.txt,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** bugs.txt	27 Oct 2003 09:40:38 -0000	1.2
--- bugs.txt	5 Dec 2003 05:12:33 -0000	1.3
***************
*** 75,78 ****
--- 75,96 ----
  .. _FAQ entry on this issue: http://spambayes.sourceforge.net/faq.html#how-can-i-get-rid-of-the-envelope-tray-icon-for-spam
  
+ Addin loads with "Could not watch the specified folders" error
+ --------------------------------------------------------------
+ In this bug, the log file contains:
+ 
+ ::
+ 
+   ...
+   File "out1.pyz/addin", line 1273, in _HookFolderEvents
+   File "out1.pyz/msgstore", line 539, in GetOutlookItem
+   File "...", line 4597, in GetFolderFromID
+   ...
+   com_error: (-2147352567, 'Exception occurred.', (4096, 'Microsoft Office Outlook', 'The operation failed.',None, 0, -2147221241), None)
+ 
+ We are tracking this bug in 
+ `[ 843521 ] E_FAIL exception watching folders disables SpamBayes`_.
+ 
+ .. _[ 843521 ] E_FAIL exception watching folders disables SpamBayes: https://sourceforge.net/tracker/index.php?func=detail&aid=843521&group_id=61702&atid=498103
+ 
  Customize Outlook Today stops working
  -------------------------------------



From montanaro at users.sourceforge.net  Fri Dec  5 16:23:37 2003
From: montanaro at users.sourceforge.net (Skip Montanaro)
Date: Fri Dec  5 16:23:40 2003
Subject: [Spambayes-checkins] spambayes/contrib spamcounts.py,1.2,1.3
Message-ID: <E1ASNQH-0000T9-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/contrib
In directory sc8-pr-cvs1:/tmp/cvs-serv1794

Modified Files:
	spamcounts.py 
Log Message:
allow the user to see the 'saved state' token


Index: spamcounts.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/contrib/spamcounts.py,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** spamcounts.py	20 Nov 2003 02:50:03 -0000	1.2
--- spamcounts.py	5 Dec 2003 21:23:35 -0000	1.3
***************
*** 71,75 ****
          seen.add(t)
  
!         sc, hc = db.get(t, (0, 0))
          if sc == hc == 0:
              continue
--- 71,78 ----
          seen.add(t)
  
!         try:
!             sc, hc = db.get(t, (0, 0))
!         except ValueError:
!             _, sc, hc = db.get(t, (0, 0, 0))
          if sc == hc == 0:
              continue



From mhammond at users.sourceforge.net  Sun Dec  7 19:55:45 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Sun Dec  7 19:55:49 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 addin.py,1.114,1.115
Message-ID: <E1AT9gf-0007jX-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv29168

Modified Files:
	addin.py 
Log Message:
A number of changes to better support us existing in the 'COM Addins'
list when running the binary version:
* By default, no longer register in both HKLM and HKCU - registering
  in both means we don't appear in the addins list.
* Support special registration command-line to register in HKLM instead
  of the default HKCU - see the comments on this
* Support the addin being selected and de-selected multiple times via
  the Outlook GUI.  This was mainly removing old object references when
  'Close' is called.
* When the user manually de-selects us via the 'Addins' list, we remove
  the SpamBayes toolbar.  TODO: somehow exploit this so that at uninstall 
  time we can also remove the toolbar.


Index: addin.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/addin.py,v
retrieving revision 1.114
retrieving revision 1.115
diff -C2 -d -r1.114 -r1.115
*** addin.py	27 Oct 2003 03:36:24 -0000	1.114
--- addin.py	8 Dec 2003 00:55:43 -0000	1.115
***************
*** 3,6 ****
--- 3,7 ----
  import warnings
  import traceback
+ import _winreg
  
  # *sigh* - this is for the binary installer, and for the sake of one line
***************
*** 86,94 ****
  gencache.EnsureModule('{2DF8D04C-5BFA-101B-BDE5-00AA0044DE52}', 0, 2, 1,
                          bForDemand=True, bValidateFile=bValidateGencache) # Office 9
  
- # Register what vtable based interfaces we need to implement.
- # Damn - we should use EnsureModule for the _IDTExtensibility2 typelib, but
- # win32all 155 and earlier don't like us pre-generating :(
- universal.RegisterInterfaces('{AC0714F2-3D04-11D1-AE7D-00A0C90F26F4}', 0, 1, 0, ["_IDTExtensibility2"])
  try:
      from win32com.client import CastTo, WithEvents
--- 87,97 ----
  gencache.EnsureModule('{2DF8D04C-5BFA-101B-BDE5-00AA0044DE52}', 0, 2, 1,
                          bForDemand=True, bValidateFile=bValidateGencache) # Office 9
+ # We the "Addin Designer" typelib for its constants
+ gencache.EnsureModule('{AC0714F2-3D04-11D1-AE7D-00A0C90F26F4}', 0, 1, 0,
+                         bForDemand=True, bValidateFile=bValidateGencache)
+ # ... and also for its _IDTExtensibility2 vtable interface.
+ universal.RegisterInterfaces('{AC0714F2-3D04-11D1-AE7D-00A0C90F26F4}', 0, 1, 0,
+                              ["_IDTExtensibility2"])
  
  try:
      from win32com.client import CastTo, WithEvents
***************
*** 965,980 ****
                          # for not broken - can't find toolbar.  Create a new one.
                          # Create it as a permanent one (which is default)
-                         if self.explorers_collection.have_created_toolbar:
-                             # Eeek - we have already created a toolbar, but
-                             # now we can't find it.  It is likely this is the
-                             # first time we are being run, and outlook is
-                             # being started with multiple Windows open.
-                             # Hopefully things will get back to normal once
-                             # Outlook is restarted (which testing shows it does)
-                             return
- 
                          print "Creating new SpamBayes toolbar to host our buttons"
!                         self.toolbar = bars.Add(toolbar_name, constants.msoBarTop, Temporary=False)
!                         self.explorers_collection.have_created_toolbar = True
                      self.toolbar.Visible = True
                  parent = self.toolbar
--- 968,975 ----
                          # for not broken - can't find toolbar.  Create a new one.
                          # Create it as a permanent one (which is default)
                          print "Creating new SpamBayes toolbar to host our buttons"
!                         self.toolbar = bars.Add(toolbar_name,
!                                                 constants.msoBarTop,
!                                                 Temporary=False)
                      self.toolbar.Visible = True
                  parent = self.toolbar
***************
*** 1122,1129 ****
          self.manager = manager
          self.explorers = []
-         self.have_created_toolbar = False
          self.button_event_map = {}
  
      def Close(self):
          self.explorers = None
  
--- 1117,1125 ----
          self.manager = manager
          self.explorers = []
          self.button_event_map = {}
  
      def Close(self):
+         while self.explorers:
+             self._DoDeadExplorer(self.explorers[0])
          self.explorers = None
  
***************
*** 1228,1231 ****
--- 1224,1232 ----
                  # still screw up.
                  self.manager.LogDebug(0, "*** SpamBayes is NOT enabled, so will not filter incoming mail. ***")
+             if connectMode == constants.ext_cm_AfterStartup:
+                 # We are being enabled after startup, which means we don't get
+                 # the 'OnStartupComplete()' event - call it manually so we
+                 # bootstrap code that can't happen until startup is complete.
+                 self.OnStartupComplete(None)
          except:
              print "Error connecting to Outlook!"
***************
*** 1331,1336 ****
                  hook.Close()
              self.folder_hooks = None
!         self.application = None
!         self.explorers_events = None
          if self.manager is not None:
              # Save database - bsddb databases will generally do nothing here
--- 1332,1338 ----
                  hook.Close()
              self.folder_hooks = None
!         if self.explorers_events is not None:
!             self.explorers_events.Close()
!             self.explorers_events = None
          if self.manager is not None:
              # Save database - bsddb databases will generally do nothing here
***************
*** 1344,1347 ****
--- 1346,1370 ----
              self.manager = None
  
+         if mode==constants.ext_dm_UserClosed:
+             # The user has de-selected us.  Remove the toolbars we created
+             # (Maybe we can exploit this later to remove toolbars as part
+             # of uninstall?)
+             print "SpamBayes is being manually disabled - deleting toolbar"
+             try:
+                 explorers = self.application.Explorers
+                 for i in range(explorers.Count):
+                     explorer = explorers.Item(i+1)
+                     try:
+                         toolbar = explorer.CommandBars.Item(toolbar_name)
+                     except pythoncom.com_error:
+                         print "Could not find our toolbar to delete!"
+                     else:
+                         toolbar.Delete()
+             except:
+                 print "ERROR deleting toolbar"
+                 traceback.print_exc()
+ 
+         self.application = None
+ 
          print "Addin terminating: %d COM client and %d COM servers exist." \
                % (pythoncom._GetInterfaceCount(), pythoncom._GetGatewayCount())
***************
*** 1373,1377 ****
  
  def _DoRegister(klass, root):
-     import _winreg
      key = _winreg.CreateKey(root,
                              "Software\\Microsoft\\Office\\Outlook\\Addins")
--- 1396,1399 ----
***************
*** 1382,1404 ****
      _winreg.SetValueEx(subkey, "FriendlyName", 0, _winreg.REG_SZ, "SpamBayes")
  
! def RegisterAddin(klass):
!     import _winreg
!     # Try and register twice - once in HKLM, and once in HKCU.  This seems
!     # to help roaming profiles, etc.  Once registered, it is both registered
!     # on this machine for the current user (even when they roam, assuming it
!     # has been installed on the remote machine also) and for any user on this
!     # machine.
!     try:
          _DoRegister(klass, _winreg.HKEY_LOCAL_MACHINE)
      except WindowsError:
-         # But they may not have the rights to install there.
          pass
-     # We don't catch exception registering just for this user though
-     # that is fatal!
      _DoRegister(klass, _winreg.HKEY_CURRENT_USER)
      print "Registration complete."
  
! def UnregisterAddin(klass):
!     import _winreg
      try:
          _winreg.DeleteKey(_winreg.HKEY_LOCAL_MACHINE,
--- 1404,1452 ----
      _winreg.SetValueEx(subkey, "FriendlyName", 0, _winreg.REG_SZ, "SpamBayes")
  
! # Note that Addins can be registered either in HKEY_CURRENT_USER or
! # HKEY_LOCAL_MACHINE.  If the former, then:
! # * Only available for the user that installed the addin.
! # * Appears in the 'COM Addins' list, and can be removed by the user.
! # If HKEY_LOCAL_MACHINE:
! # * Available for every user who uses the machine.  This is useful for site
! #   admins, so it works with "roaming profiles" as users move around.
! # * Does not appear in 'COM Addins', and thus can not be disabled by the user.
! 
! # Note that if the addin is registered in both places, it acts as if it is
! # only installed in HKLM - ie, does not appear in the addins list.
! # For this reason, the addin can be registered in HKEY_LOCAL_MACHINE
! # by executing 'regsvr32 /i:hkey_local_machine outlook_addin.dll'
! # (or 'python addin.py hkey_local_machine' for source code users.
! # Note to Binary Builders: You need py2exe dated 8-Dec-03+ for this to work.
! 
! # Called when "regsvr32 /i:whatever" is used.  We support 'hkey_local_machine'
! def DllInstall(bInstall, cmdline):
!     klass = OutlookAddin
!     if bInstall and cmdline.lower().find('hkey_local_machine')>=0:
!         # Unregister the old installation, if one exists.
!         DllUnregisterServer()
!         # Don't catch exceptions here - if it fails, the Dll registration
!         # must fail.
          _DoRegister(klass, _winreg.HKEY_LOCAL_MACHINE)
+         print "Registration (in HKEY_LOCAL_MACHINE) complete."
+     
+ def DllRegisterServer():
+     klass = OutlookAddin
+     # *sigh* - we used to *also* register in HKLM, but as above, this makes
+     # things work like we are *only* installed in HKLM.  Thus, we explicitly
+     # remove the HKLM registration here (but it can be re-added - see the
+     # notes above.)
+     try:
+         _winreg.DeleteKey(_winreg.HKEY_LOCAL_MACHINE,
+                           "Software\\Microsoft\\Office\\Outlook\\Addins\\" \
+                           + klass._reg_progid_)
      except WindowsError:
          pass
      _DoRegister(klass, _winreg.HKEY_CURRENT_USER)
      print "Registration complete."
  
! def DllUnregisterServer():
!     klass = OutlookAddin
!     # Try to remove the HKLM version.
      try:
          _winreg.DeleteKey(_winreg.HKEY_LOCAL_MACHINE,
***************
*** 1415,1429 ****
          pass
  
- def DllRegisterServer():
-     RegisterAddin(OutlookAddin)
- 
- def DllUnregisterServer():
-     UnregisterAddin(OutlookAddin)
- 
  if __name__ == '__main__':
      import win32com.server.register
      win32com.server.register.UseCommandLine(OutlookAddin)
      if "--unregister" in sys.argv:
          DllUnregisterServer()
      else:
          DllRegisterServer()
--- 1463,1483 ----
          pass
  
  if __name__ == '__main__':
      import win32com.server.register
      win32com.server.register.UseCommandLine(OutlookAddin)
+     # todo - later win32all versions of  UseCommandLine support
+     # finalize_register and finalize_unregister keyword args, passing the
+     # functions.
+     # (But DllInstall may get support in UseCommandLine later, so let's
+     # wait and see)
      if "--unregister" in sys.argv:
          DllUnregisterServer()
      else:
          DllRegisterServer()
+         # Support 'hkey_local_machine' on the commandline, to work in
+         # the same way as 'regsvr32 /i:hkey_local_machine' does.
+         # regsvr32 calls it after DllRegisterServer, (and our registration
+         # logic relies on that) so we will too.
+         for a in sys.argv[1:]:
+             if a.lower()=='hkey_local_machine':
+                 DllInstall(True, 'hkey_local_machine')



From mhammond at users.sourceforge.net  Sun Dec  7 19:59:21 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Sun Dec  7 19:59:24 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 manager.py,1.88,1.89
Message-ID: <E1AT9k9-0007qF-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv29828

Modified Files:
	manager.py 
Log Message:
* Add/Fix a number of 'unicode file' related comments.
* Allow multiple manager objects to work in the same process (but not
  at the same time):
  - If the global manager is being closed, drop the global var holding it,
    so the next request for the manager will make a new one.
  - In this case, 'spambayes.Options' has already been imported, so our
    simple 'imported too early' check failed.  If spambayes.Options has
    been imported, we assert that the INI file it is using is our INI.
* Wrap a few long lines.


Index: manager.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/manager.py,v
retrieving revision 1.88
retrieving revision 1.89
diff -C2 -d -r1.88 -r1.89
*** manager.py	29 Sep 2003 02:14:25 -0000	1.88
--- manager.py	8 Dec 2003 00:59:19 -0000	1.89
***************
*** 60,66 ****
                                        win32con.MB_ICONQUESTION) == win32con.IDYES
  
! # Notes on Unicode directory names
! # You will have much more success with extended characters in
! # directory names using Python 2.3.
  try:
      filesystem_encoding = sys.getfilesystemencoding()
--- 60,65 ----
                                        win32con.MB_ICONQUESTION) == win32con.IDYES
  
! # Non-ascii characters in file or directory names only fully work in
! # Python 2.3.3+, but latin-1 "compatible" filenames should work in 2.2
  try:
      filesystem_encoding = sys.getfilesystemencoding()
***************
*** 71,85 ****
  # the directory of our main .py/.dll/.exe file we
  # are running from.
! try:
!     if hasattr(sys, "frozen"):
!         if sys.frozen == "dll":
!             this_filename = win32api.GetModuleFileName(sys.frozendllhandle)
!         else:
!             # Don't think we will ever run as a .EXE, but...
!             this_filename = os.path.abspath(sys.argv[0])
!     else:
          this_filename = os.path.abspath(__file__)
! except NameError: # no __file__
!     this_filename = os.path.abspath(sys.argv[0])
  
  # See if we can use the new bsddb module. (The old one is unreliable
--- 70,81 ----
  # the directory of our main .py/.dll/.exe file we
  # are running from.
! if hasattr(sys, "frozen"):
!     assert sys.frozen == "dll", "outlook only supports inproc servers"
!     this_filename = win32api.GetModuleFileName(sys.frozendllhandle)
! else:
!     try:
          this_filename = os.path.abspath(__file__)
!     except NameError: # no __file__ - means Py2.2 and __name__=='__main__'
!         this_filename = os.path.abspath(sys.argv[0])
  
  # See if we can use the new bsddb module. (The old one is unreliable
***************
*** 96,99 ****
--- 92,97 ----
      except ImportError:
          # No DB library at all!
+         assert not hasattr(sys, "frozen"), \
+                "Don't build binary versions without bsbbd!"
          use_db = False
  
***************
*** 117,122 ****
  
  def import_core_spambayes_stuff(ini_filename):
!     assert "spambayes.Options" not in sys.modules, \
!         "'spambayes.Options' was imported too early"
      global bayes_classifier, bayes_tokenize, bayes_storage
      # ini_filename is Unicode, but environ not unicode aware
--- 115,129 ----
  
  def import_core_spambayes_stuff(ini_filename):
!     if "spambayes.Options" in sys.modules:
!         # Manager probably being re-initialized (via the Outlook 'addin' GUI
!         # Check that nothing has changed underneath us.
!         if __debug__:
!             import spambayes.Options
!             assert spambayes.Options.optionsPathname == \
!                    ini_filename.encode(filesystem_encoding), \
!                    "'spambayes.Options' was imported too early, with the " \
!                    "incorrect directory %r" \
!                    % (spambayes.Options.optionsPathname,)
!         return
      global bayes_classifier, bayes_tokenize, bayes_storage
      # ini_filename is Unicode, but environ not unicode aware
***************
*** 131,138 ****
          "Expected 'spambayes.Options' to be loaded here"
  
- class ManagerError(Exception):
-     pass
- 
- 
  # Function to "safely" save a pickle, only overwriting
  # the existing file after a successful write.
--- 138,141 ----
***************
*** 351,361 ****
                  if not os.path.isdir(value):
                      os.makedirs(value)
!                 if not os.path.isdir(value):
!                     raise os.error
                  value = os.path.abspath(value)
              except os.error:
!                 print "The configuration files have specified a data directory of"
!                 print repr(value)
!                 print "but it is not valid.  Using default"
                  value = None
          if value:
--- 354,363 ----
                  if not os.path.isdir(value):
                      os.makedirs(value)
!                 assert os.path.isdir(value), "just made the *ucker"
                  value = os.path.abspath(value)
              except os.error:
!                 print "The configuration files have specified a data " \
!                       "directory of", repr(value), "but it is not valid. " \
!                       "Using default."
                  value = None
          if value:
***************
*** 373,377 ****
          self.LoadConfig()
  
!         bayes_options_filename = os.path.join(self.data_directory, "default_bayes_customize.ini")
          import_core_spambayes_stuff(bayes_options_filename)
  
--- 375,380 ----
          self.LoadConfig()
  
!         bayes_options_filename = os.path.join(self.data_directory,
!                                               "default_bayes_customize.ini")
          import_core_spambayes_stuff(bayes_options_filename)
  
***************
*** 487,491 ****
          # A bit of a nod to save people doing a full retrain.
          # Try and locate our files in the old location, and move
!         # then to the new one.
          # Also used first time SpamBayes is run - this will cause
          # the ini file to be *copied* to the correct directory
--- 490,494 ----
          # A bit of a nod to save people doing a full retrain.
          # Try and locate our files in the old location, and move
!         # them to the new one.
          # Also used first time SpamBayes is run - this will cause
          # the ini file to be *copied* to the correct directory
***************
*** 753,762 ****
  
      def Close(self):
          self.classifier_data.Close()
!         self.config = None
          if self.message_store is not None:
              self.message_store.Close()
              self.message_store = None
          self.outlook = None
  
      def score(self, msg, evidence=False):
--- 756,770 ----
  
      def Close(self):
+         global _mgr
          self.classifier_data.Close()
!         self.config = self.options = None
          if self.message_store is not None:
              self.message_store.Close()
              self.message_store = None
          self.outlook = None
+         self.addin = None
+         # If we are the global manager, reset that
+         if _mgr is self:
+             _mgr = None
  
      def score(self, msg, evidence=False):
***************
*** 850,860 ****
  
  def main(verbose_level = 1):
!     try:
!         mgr = GetManager()
!         mgr.verbose = max(mgr.verbose, verbose_level)
!     except ManagerError, d:
!         print "Error initializing Bayes manager"
!         print d
!         return 1
      ShowManager(mgr)
      mgr.Save()
--- 858,863 ----
  
  def main(verbose_level = 1):
!     mgr = GetManager()
!     mgr.verbose = max(mgr.verbose, verbose_level)
      ShowManager(mgr)
      mgr.Save()



From mhammond at users.sourceforge.net  Sun Dec  7 20:00:53 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Sun Dec  7 20:00:56 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 about.html,1.25,1.26
Message-ID: <E1AT9ld-0007wg-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv30500

Modified Files:
	about.html 
Log Message:
Add notes about how to register SpamBayes globally for all users on the
machine.


Index: about.html
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/about.html,v
retrieving revision 1.25
retrieving revision 1.26
diff -C2 -d -r1.25 -r1.26
*** about.html	5 Nov 2003 13:05:14 -0000	1.25
--- about.html	8 Dec 2003 01:00:49 -0000	1.26
***************
*** 136,139 ****
--- 136,147 ----
  rules based on user-defined fields. That's why this addin supplies its
  own filtering rules.<br>
+ <h2>Installing for all users on a machine</h2>
+ By default, SpamBayes will install itself only for the user who
+ actually installed it.&nbsp; If other users log onto the same machine,
+ their Outlook installation will not have SpamBayes available.&nbsp;
+ Even though this is the way SpamBayes is designed, our troubleshooting
+ guide has <a href="docs/troubleshooting.html#register-all-users">details
+ on how you can change this</a>.<br>
+ <br>
  <h2><a name="Your_help_is_needed"></a>Your help is needed!</h2>
  <p>This is free software.&nbsp; Please offer any help you are able



From mhammond at users.sourceforge.net  Sun Dec  7 20:00:54 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Sun Dec  7 20:00:58 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000/docs
	troubleshooting.html, 1.16, 1.17
Message-ID: <E1AT9le-0007wl-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000/docs
In directory sc8-pr-cvs1:/tmp/cvs-serv30500/docs

Modified Files:
	troubleshooting.html 
Log Message:
Add notes about how to register SpamBayes globally for all users on the
machine.


Index: troubleshooting.html
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/docs/troubleshooting.html,v
retrieving revision 1.16
retrieving revision 1.17
diff -C2 -d -r1.16 -r1.17
*** troubleshooting.html	5 Nov 2003 13:05:14 -0000	1.16
--- troubleshooting.html	8 Dec 2003 01:00:51 -0000	1.17
***************
*** 18,21 ****
--- 18,24 ----
    <li><a href="#Messages_have_incorrect_or_unexpected">Messages have
  incorrect or unexpected Spam values</a></li>
+   <li><a href="#register-all-users">SpamBayes is not available for all
+ users on the machine</a><br>
+   </li>
    <li><a href="#All_other_problems">All other problems</a></li>
  </ul>
***************
*** 98,102 ****
      <li>If the SpamBayes addin is not listed, then SpamBayes should be
  reinstalled (Note that running <span style="font-style: italic;">regsvr32.exe
! spambayes_addin.dll</span> from the SpamBayes directory may also solve
  this problem)</li>
      <li>If the SpamBayes addin is listed but not checked, then simply
--- 101,105 ----
      <li>If the SpamBayes addin is not listed, then SpamBayes should be
  reinstalled (Note that running <span style="font-style: italic;">regsvr32.exe
! outlook_addin.dll</span> from the SpamBayes directory may also solve
  this problem)</li>
      <li>If the SpamBayes addin is listed but not checked, then simply
***************
*** 192,195 ****
--- 195,223 ----
    </ul>
  </ul>
+ <h2><a name="register-all-users"></a>SpamBayes is not available for all
+ users on the machine.</h2>
+ When SpamBayes is installed, by default it is available only for the
+ user who installed it.&nbsp; This is to allow SpamBayes to appear in
+ Microsoft Outlook's Com-Addin list, and therefore able to be activated
+ and de-activated by the user inside Outlook.<br>
+ It is possible to register the addin so it is available to all users on
+ a particular machine, which can be useful in enterprise arrangements
+ where users have 'roaming profiles'<br>
+ To register SpamBayes in this way, you must execute the command
+ (obviously with the correct patch substituted):<br>
+ <span style="font-style: italic;">&nbsp;&nbsp;&nbsp; regsvr32.exe
+ /i:hkey_local_machine "c:\Program Files\SpamBayes\bin\outlook_addin.dll"</span><br>
+ If you <a href="#CheckLog">check the installation log</a> after
+ performing such an install, you should see the following messages:<br>
+ <span style="font-style: italic;">&nbsp;&nbsp;&nbsp; Registered:
+ SpamBayes.OutlookAddin </span><br style="font-style: italic;">
+ <span style="font-style: italic;">&nbsp;&nbsp;&nbsp; Registration
+ complete.</span><br style="font-style: italic;">
+ <span style="font-style: italic;">&nbsp;&nbsp;&nbsp; Registration (in
+ HKEY_LOCAL_MACHINE) complete.</span><br style="font-style: italic;">
+ Note the last line, which does not exist when registration is performed
+ only for the current user.&nbsp; Once you have performed this
+ registration, the Addin will be available for all users - but as noted
+ above, it will no longer appear in Outlook's Com-Addin list.<br>
  <h2><a name="All_other_problems"></a>All other problems</h2>
  If you are simply unsure about what SpamBayes is doing, please <a



From mhammond at users.sourceforge.net  Mon Dec  8 07:07:39 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Mon Dec  8 07:07:41 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 manager.py,1.89,1.90
Message-ID: <E1ATKAt-0005Ob-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv20527

Modified Files:
	manager.py 
Log Message:
Oops - typo in comment - latin-1, non-ascii filenames work in 2.3, not 2.2


Index: manager.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/manager.py,v
retrieving revision 1.89
retrieving revision 1.90
diff -C2 -d -r1.89 -r1.90
*** manager.py	8 Dec 2003 00:59:19 -0000	1.89
--- manager.py	8 Dec 2003 12:07:37 -0000	1.90
***************
*** 61,65 ****
  
  # Non-ascii characters in file or directory names only fully work in
! # Python 2.3.3+, but latin-1 "compatible" filenames should work in 2.2
  try:
      filesystem_encoding = sys.getfilesystemencoding()
--- 61,65 ----
  
  # Non-ascii characters in file or directory names only fully work in
! # Python 2.3.3+, but latin-1 "compatible" filenames should work in 2.3
  try:
      filesystem_encoding = sys.getfilesystemencoding()



From mhammond at users.sourceforge.net  Mon Dec  8 22:59:04 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Mon Dec  8 22:59:07 2003
Subject: [Spambayes-checkins] website faq.txt,1.52,1.53
Message-ID: <E1ATZ1c-0001oi-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/website
In directory sc8-pr-cvs1:/tmp/cvs-serv6929

Modified Files:
	faq.txt 
Log Message:
Clarification in Outlook training, as suggested by a user.


Index: faq.txt
===================================================================
RCS file: /cvsroot/spambayes/website/faq.txt,v
retrieving revision 1.52
retrieving revision 1.53
diff -C2 -d -r1.52 -r1.53
*** faq.txt	3 Dec 2003 03:29:26 -0000	1.52
--- faq.txt	9 Dec 2003 03:59:00 -0000	1.53
***************
*** 757,770 ****
  -----------------------------------------------
  
! Instructions about training the Outlook plugin can be found in the documentation
! for the plugin.  However, basically what you need to do is move as much spam
! as you have into your spam folder, tell the plugin which folder that is and
! which folders contain examples of ham, and it will do the rest.
  
  The plugin does *not* train on all incoming mail.  However, if you use the
  "Delete as spam" and "Recover from spam" buttons, those messages will be
  (re)trained as necessary.  If you have set it to use incremental training
! then it will also train on messages which are moved into the spam folder
! and those folders that you are 'watching'.
  
  
--- 757,772 ----
  -----------------------------------------------
  
! Instructions about training the Outlook plugin can be found in the
! documentation for the plugin, and the 'Configuration Wizard' will attempt to
! guide you through an initial training process.  Basically what you need to do
! is move as much spam as you have into your spam folder, tell the plugin which
! folder that is and which folders contain examples of ham, and it will do
! the rest.
  
  The plugin does *not* train on all incoming mail.  However, if you use the
  "Delete as spam" and "Recover from spam" buttons, those messages will be
  (re)trained as necessary.  If you have set it to use incremental training
! then it will also train on messages which are manually moved into the spam
! folder and those folders that you are 'watching'.
  
  



From mhammond at users.sourceforge.net  Mon Dec  8 23:02:46 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Mon Dec  8 23:02:48 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000/dialogs/resources
	dialogs.rc, 1.41, 1.42
Message-ID: <E1ATZ5C-0001xt-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000/dialogs/resources
In directory sc8-pr-cvs1:/tmp/cvs-serv7459

Modified Files:
	dialogs.rc 
Log Message:
Make it clear in the config wizard that using the 'Browse' button allows
them to de-select the list of folders.
Incremental training dialog: s/Inbox/watched folders/, as suggested by
a user.


Index: dialogs.rc
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/dialogs/resources/dialogs.rc,v
retrieving revision 1.41
retrieving revision 1.42
diff -C2 -d -r1.41 -r1.42
*** dialogs.rc	3 Dec 2003 23:45:38 -0000	1.41
--- dialogs.rc	9 Dec 2003 04:02:43 -0000	1.42
***************
*** 224,229 ****
      LTEXT           "SpamBayes needs to know what folders are used to receive new messages.  In most cases, this will be your Inbox, but you may also specify additional folders to be watched for spam.",
                      IDC_STATIC,20,21,247,25
!     LTEXT           "The following folders will be watched for new messages",
!                     IDC_STATIC,20,90,247,13
      LTEXT           "If you use the Outlook rule wizard to move messages into folders, you may like to select these folders in addition to your inbox.",
                      IDC_STATIC,20,51,241,20
--- 224,229 ----
      LTEXT           "SpamBayes needs to know what folders are used to receive new messages.  In most cases, this will be your Inbox, but you may also specify additional folders to be watched for spam.",
                      IDC_STATIC,20,21,247,25
!     LTEXT           "The following folders will be watched for new messages.  Use the Browse button to change the list, or Next if the list of folders is correct.",
!                     IDC_STATIC,20,79,247,20
      LTEXT           "If you use the Outlook rule wizard to move messages into folders, you may like to select these folders in addition to your inbox.",
                      IDC_STATIC,20,51,241,20
***************
*** 520,525 ****
                      SS_CENTERIMAGE | SS_SUNKEN,11,21,175,12
      PUSHBUTTON      "&Browse...",IDC_BROWSE_HAM,192,20,50,14
!     LTEXT           "Folders with spam or other junk messages.",IDC_STATIC,11,
!                     36,171,9
      CONTROL         "Static",IDC_STATIC_SPAM,"Static",SS_LEFTNOWORDWRAP | 
                      SS_CENTERIMAGE | SS_SUNKEN | WS_GROUP,11,46,174,12
--- 520,525 ----
                      SS_CENTERIMAGE | SS_SUNKEN,11,21,175,12
      PUSHBUTTON      "&Browse...",IDC_BROWSE_HAM,192,20,50,14
!     LTEXT           "Folders with spam or other junk messages.",IDC_STATIC,
!                     11,36,171,9
      CONTROL         "Static",IDC_STATIC_SPAM,"Static",SS_LEFTNOWORDWRAP | 
                      SS_CENTERIMAGE | SS_SUNKEN | WS_GROUP,11,46,174,12
***************
*** 535,539 ****
                      IDC_PROGRESS_TEXT,75,89,149,17
      GROUPBOX        "Incremental Training",IDC_STATIC,4,117,244,87
!     CONTROL         "Train that a message is good when it is moved from a spam folder back to the Inbox",
                      IDC_BUT_TRAIN_FROM_SPAM_FOLDER,"Button",BS_AUTOCHECKBOX | 
                      BS_MULTILINE | WS_TABSTOP,11,127,204,18
--- 535,539 ----
                      IDC_PROGRESS_TEXT,75,89,149,17
      GROUPBOX        "Incremental Training",IDC_STATIC,4,117,244,87
!     CONTROL         "Train that a message is good when it is moved from a spam folder back to the Inbox.",
                      IDC_BUT_TRAIN_FROM_SPAM_FOLDER,"Button",BS_AUTOCHECKBOX | 
                      BS_MULTILINE | WS_TABSTOP,11,127,204,18
***************
*** 542,546 ****
      COMBOBOX        IDC_RECOVER_RS,127,145,114,54,CBS_DROPDOWNLIST | 
                      WS_VSCROLL | WS_TABSTOP
!     CONTROL         "Train that a message is spam when it is moved to the spam folder",
                      IDC_BUT_TRAIN_TO_SPAM_FOLDER,"Button",BS_AUTOCHECKBOX | 
                      BS_MULTILINE | WS_TABSTOP,11,163,204,16
--- 542,546 ----
      COMBOBOX        IDC_RECOVER_RS,127,145,114,54,CBS_DROPDOWNLIST | 
                      WS_VSCROLL | WS_TABSTOP
!     CONTROL         "Train that a message is spam when it is moved to the spam folder.",
                      IDC_BUT_TRAIN_TO_SPAM_FOLDER,"Button",BS_AUTOCHECKBOX | 
                      BS_MULTILINE | WS_TABSTOP,11,163,204,16



From montanaro at users.sourceforge.net  Tue Dec  9 10:36:32 2003
From: montanaro at users.sourceforge.net (Skip Montanaro)
Date: Tue Dec  9 10:36:36 2003
Subject: [Spambayes-checkins] spambayes/spambayes tokenizer.py,1.15,1.16
Message-ID: <E1ATjua-0004S4-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv17103

Modified Files:
	tokenizer.py 
Log Message:
tweak docstring to match current options practice


Index: tokenizer.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/tokenizer.py,v
retrieving revision 1.15
retrieving revision 1.16
diff -C2 -d -r1.15 -r1.16
*** tokenizer.py	18 Sep 2003 21:00:10 -0000	1.15
--- tokenizer.py	9 Dec 2003 15:36:29 -0000	1.16
***************
*** 1356,1361 ****
          """Generate a stream of tokens from an email Message.
  
!         If options.check_octets is True, the first few undecoded characters
!         of application/octet-stream parts of the message body become tokens.
          """
  
--- 1356,1362 ----
          """Generate a stream of tokens from an email Message.
  
!         If options['Tokenizer', 'check_octets'] is True, the first few
!         undecoded characters of application/octet-stream parts of the
!         message body become tokens.
          """
  



From montanaro at users.sourceforge.net  Tue Dec  9 10:52:15 2003
From: montanaro at users.sourceforge.net (Skip Montanaro)
Date: Tue Dec  9 10:52:19 2003
Subject: [Spambayes-checkins] spambayes/spambayes OptionsClass.py,1.15,1.16
Message-ID: <E1ATk9n-0005Cz-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv20010

Modified Files:
	OptionsClass.py 
Log Message:
Loosen constraints on HEADER_VALUE regular expression.


Add an experimental/deprecation capability.  (This is more difficult to
describe than to use...)  Options which are experimental or deprecated are
prefixed with 'X-'.  The distinction between the two rather artificial.  The
description of deprecated options should start with "(DEPRECATED)".

There are four cases:

    * option is foo, user sets foo -  Set foo silently (current behavior)

    * option is X-foo, user sets X-foo - Set X-foo silently (current
      behavior)

    * option is foo, user sets X-foo - option has moved from experimental to
      accepted, but the user still sets the option using the experimental
      form.  Set foo silently.

    * option is X-foo, user sets foo - option has moved from accepted to
      deprecated, but the user still sets the non-prefixed option.  Set
      X-foo and emit a warning.

Of course, this should all be considered experimental for the time being and
is subject to change... <wink>


Index: OptionsClass.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/OptionsClass.py,v
retrieving revision 1.15
retrieving revision 1.16
diff -C2 -d -r1.15 -r1.16
*** OptionsClass.py	12 Nov 2003 22:01:39 -0000	1.15
--- OptionsClass.py	9 Dec 2003 15:52:12 -0000	1.16
***************
*** 22,25 ****
--- 22,41 ----
  via an instance of this class.
  
+ Experimental or deprecated options are prefixed with 'X-', borrowing the
+ practice from RFC-822 mail.  If the user sets an option like:
+ 
+     [Tokenizer]
+     X-transmogrify: True
+ 
+ and an 'X-transmogrify' or 'transmogrify' option exists, it is set silently
+ to the value given by the user.  If the user sets an option like:
+ 
+     [Tokenizer]
+     transmogrify: True
+ 
+ and no 'transmogrify' option exists, but an 'X-transmogrify' option does,
+ the latter is set to the value given by the users and a deprecation message
+ is printed to standard error.
+ 
  To Do:
   o Stop allowing invalid options in configuration files
***************
*** 495,498 ****
--- 511,520 ----
              self.merge_file(file)
  
+     def convert_and_set(self, section, option, value):
+         if self.multiple_values_allowed(section, option):
+             value = self.convert(section, option, value)
+         value = self.convert(section, option, value)
+         self.set(section, option, value)
+ 
      def merge_file(self, filename):
          import ConfigParser
***************
*** 505,516 ****
                  option = opt
                  if not self._options.has_key((section, option)):
!                     print >> sys.stderr, ("Invalid option %s in"
!                                           " section %s in file %s" %
!                                           (opt, sect, filename))
                  else:
!                     if self.multiple_values_allowed(section, option):
!                         value = self.convert(section, option, value)
!                     value = self.convert(section, option, value)
!                     self.set(section, option, value)
  
      # not strictly necessary, but convenient shortcuts to self._options
--- 527,554 ----
                  option = opt
                  if not self._options.has_key((section, option)):
!                     if option.startswith('x-'):
!                         # try setting option without the x- prefix
!                         option = option[2:]
!                         if self._options.has_key((section, option)):
!                             self.convert_and_set(section, option, value)
!                         # not an error if an x- option is missing
!                     else:
!                         option = 'x-'+option
!                         # going the other way, if the option has been
!                         # deprecated, set its x-prefixed version and
!                         # emit a warning
!                         if self._options.has_key((section, option)):
!                             self.convert_and_set(section, option, value)
!                             print >> sys.stderr, (
!                                 "warning: option %s in"
!                                 " section %s is deprecated" %
!                                 (opt, sect))
!                         else:
!                             print >> sys.stderr, (
!                                 "warning: Invalid option %s in"
!                                 " section %s in file %s" %
!                                 (opt, sect, filename))
                  else:
!                     self.convert_and_set(section, option, value)
  
      # not strictly necessary, but convenient shortcuts to self._options
***************
*** 637,641 ****
  
      def options(self, prepend_section_name=False):
!         '''Return a alphabetical list of all the options, optionally
          prefixed with [section_name]'''
          all = []
--- 675,679 ----
  
      def options(self, prepend_section_name=False):
!         '''Return an alphabetical list of all the options, optionally
          prefixed with [section_name]'''
          all = []
***************
*** 691,695 ****
  # you may use any regex or tuple you wish.
  HEADER_NAME = r"[\w\.\-\*]+"
! HEADER_VALUE = r"[\w\.\-\*]+"
  INTEGER = r"[\d]+"              # actually, a *positive* integer
  REAL = r"[\d]+[\.]?[\d]*"       # likewise, a *positive* real
--- 729,733 ----
  # you may use any regex or tuple you wish.
  HEADER_NAME = r"[\w\.\-\*]+"
! HEADER_VALUE = r".+"
  INTEGER = r"[\d]+"              # actually, a *positive* integer
  REAL = r"[\d]+[\.]?[\d]*"       # likewise, a *positive* real



From montanaro at users.sourceforge.net  Tue Dec  9 11:04:18 2003
From: montanaro at users.sourceforge.net (Skip Montanaro)
Date: Tue Dec  9 11:04:22 2003
Subject: [Spambayes-checkins] spambayes/spambayes tokenizer.py, 1.16,
	1.17 Options.py, 1.83, 1.84
Message-ID: <E1ATkLS-0005m1-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv22154

Modified Files:
	tokenizer.py Options.py 
Log Message:
deprecate generate_time_buckets and extract_dow


Index: tokenizer.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/tokenizer.py,v
retrieving revision 1.16
retrieving revision 1.17
diff -C2 -d -r1.16 -r1.17
*** tokenizer.py	9 Dec 2003 15:36:29 -0000	1.16
--- tokenizer.py	9 Dec 2003 16:04:16 -0000	1.17
***************
*** 1292,1296 ****
  
          # Date:
!         if options["Tokenizer", "generate_time_buckets"]:
              for header in msg.get_all("date", ()):
                  mat = self.date_hms_re.search(header)
--- 1292,1296 ----
  
          # Date:
!         if options["Tokenizer", "x-generate_time_buckets"]:
              for header in msg.get_all("date", ()):
                  mat = self.date_hms_re.search(header)
***************
*** 1302,1306 ****
                      yield 'time:%02d:%d' % (h, bucket)
  
!         if options["Tokenizer", "extract_dow"]:
              for header in msg.get_all("date", ()):
                  # extract the day of the week
--- 1302,1306 ----
                      yield 'time:%02d:%d' % (h, bucket)
  
!         if options["Tokenizer", "x-extract_dow"]:
              for header in msg.get_all("date", ()):
                  # extract the day of the week

Index: Options.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/Options.py,v
retrieving revision 1.83
retrieving revision 1.84
diff -C2 -d -r1.83 -r1.84
*** Options.py	21 Oct 2003 21:42:22 -0000	1.83
--- Options.py	9 Dec 2003 16:04:16 -0000	1.84
***************
*** 137,147 ****
       INTEGER, RESTORE),
  
!     ("generate_time_buckets", "Generate time buckets", False,
!      """Generate tokens which resemble the posting time in 10-minute
!      buckets:  'time:'  hour  ':'  minute//10""",
       BOOLEAN, RESTORE),
  
!     ("extract_dow", "Extract day-of-week", False,
!      """Extract day of the week tokens from the Date: header.""",
       BOOLEAN, RESTORE),
  
--- 137,147 ----
       INTEGER, RESTORE),
  
!     ("X-generate_time_buckets", "Generate time buckets", False,
!      """(DEPRECATED) Generate tokens which resemble the posting time
!      in 10-minute buckets:  'time:'  hour  ':'  minute//10""",
       BOOLEAN, RESTORE),
  
!     ("X-extract_dow", "Extract day-of-week", False,
!      """(DEPRECATED) Extract day of the week tokens from the Date: header.""",
       BOOLEAN, RESTORE),
  



From montanaro at users.sourceforge.net  Tue Dec  9 11:31:41 2003
From: montanaro at users.sourceforge.net (Skip Montanaro)
Date: Tue Dec  9 11:31:55 2003
Subject: [Spambayes-checkins] spambayes/spambayes Options.py, 1.84,
	1.85 OptionsClass.py, 1.16, 1.17
Message-ID: <E1ATklx-0007Iy-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv27984/spambayes

Modified Files:
	Options.py OptionsClass.py 
Log Message:
Friggin' Windows...  Why couldn't they do case sensitivity?


Index: Options.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/Options.py,v
retrieving revision 1.84
retrieving revision 1.85
diff -C2 -d -r1.84 -r1.85
*** Options.py	9 Dec 2003 16:04:16 -0000	1.84
--- Options.py	9 Dec 2003 16:31:39 -0000	1.85
***************
*** 137,146 ****
       INTEGER, RESTORE),
  
!     ("X-generate_time_buckets", "Generate time buckets", False,
       """(DEPRECATED) Generate tokens which resemble the posting time
       in 10-minute buckets:  'time:'  hour  ':'  minute//10""",
       BOOLEAN, RESTORE),
  
!     ("X-extract_dow", "Extract day-of-week", False,
       """(DEPRECATED) Extract day of the week tokens from the Date: header.""",
       BOOLEAN, RESTORE),
--- 137,146 ----
       INTEGER, RESTORE),
  
!     ("x-generate_time_buckets", "Generate time buckets", False,
       """(DEPRECATED) Generate tokens which resemble the posting time
       in 10-minute buckets:  'time:'  hour  ':'  minute//10""",
       BOOLEAN, RESTORE),
  
!     ("x-extract_dow", "Extract day-of-week", False,
       """(DEPRECATED) Extract day of the week tokens from the Date: header.""",
       BOOLEAN, RESTORE),

Index: OptionsClass.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/OptionsClass.py,v
retrieving revision 1.16
retrieving revision 1.17
diff -C2 -d -r1.16 -r1.17
*** OptionsClass.py	9 Dec 2003 15:52:12 -0000	1.16
--- OptionsClass.py	9 Dec 2003 16:31:39 -0000	1.17
***************
*** 528,536 ****
                  if not self._options.has_key((section, option)):
                      if option.startswith('x-'):
!                         # try setting option without the x- prefix
                          option = option[2:]
                          if self._options.has_key((section, option)):
                              self.convert_and_set(section, option, value)
!                         # not an error if an x- option is missing
                      else:
                          option = 'x-'+option
--- 528,536 ----
                  if not self._options.has_key((section, option)):
                      if option.startswith('x-'):
!                         # try setting option without the X- prefix
                          option = option[2:]
                          if self._options.has_key((section, option)):
                              self.convert_and_set(section, option, value)
!                         # not an error if an X- option is missing
                      else:
                          option = 'x-'+option



From mhammond at users.sourceforge.net  Wed Dec 10 00:06:25 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Wed Dec 10 00:06:29 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000/sandbox dump_props.py,
	1.10, 1.11
Message-ID: <E1ATwYL-000491-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000/sandbox
In directory sc8-pr-cvs1:/tmp/cvs-serv15812

Modified Files:
	dump_props.py 
Log Message:
Add new options to dump tables, and dump the magic variable with Outlook's
user defined properties.


Index: dump_props.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/sandbox/dump_props.py,v
retrieving revision 1.10
retrieving revision 1.11
diff -C2 -d -r1.10 -r1.11
*** dump_props.py	1 Sep 2003 06:04:08 -0000	1.10
--- dump_props.py	10 Dec 2003 05:06:23 -0000	1.11
***************
*** 10,13 ****
--- 10,28 ----
  import mapi_driver
  
+ try:
+     TBL_ALL_COLUMNS = mapi.TBL_ALL_COLUMNS
+ except AttributeError: # missing in early versions
+     TBL_ALL_COLUMNS = 1
+ 
+ PR_USERFIELDS = 0x36E30102 # PROP_TAG(PT_BINARY, 0x36e3)
+ 
+ def GetPropTagName(obj, prop_tag):
+     hr, tags, array = obj.GetNamesFromIDs( (prop_tag,) )
+     if type(array[0][1])==type(u''):
+         name = array[0][1]
+     else:
+         name = mapiutil.GetPropTagName(prop_tag)
+     return name
+ 
  # Also in new versions of mapituil
  def GetAllProperties(obj, make_pretty = True):
***************
*** 17,28 ****
      for tag, val in data:
          if make_pretty:
!             hr, tags, array = obj.GetNamesFromIDs( (tag,) )
!             if type(array[0][1])==type(u''):
!                 name = array[0][1]
!             else:
!                 name = mapiutil.GetPropTagName(tag)
!             # pretty value transformations
!             if PROP_TYPE(tag)==PT_ERROR:
!                 val = mapiutil.GetScodeString(val)
          else:
              name = tag
--- 32,36 ----
      for tag, val in data:
          if make_pretty:
!             name = GetPropTagName(obj, tag)
          else:
              name = tag
***************
*** 43,67 ****
      return "".join(chunks)
  
! def DumpItemProps(item, shorten, get_large_props):
!     all_props = GetAllProperties(item)
!     all_props.sort() # sort by first tuple item, which is name :)
!     for prop_name, prop_tag, prop_val in all_props:
!         # Do some magic rtf conversion
!         if PROP_ID(prop_tag) == PROP_ID(PR_RTF_COMPRESSED):
!             rtf_stream = item.OpenProperty(PR_RTF_COMPRESSED, pythoncom.IID_IStream,
!                                                   0, 0)
!             html_stream = mapi.WrapCompressedRTFStream(rtf_stream, 0)
!             prop_val = mapi.RTFStreamToHTML(html_stream)
!             prop_name = "PR_RTF_COMPRESSED (to HTML)"
!             prop_tag = PROP_TAG(PT_STRING8, PR_RTF_COMPRESSED)
          if get_large_props and \
!            PROP_TYPE(prop_tag)==PT_ERROR and \
!            prop_val in [mapi.MAPI_E_NOT_ENOUGH_MEMORY,'MAPI_E_NOT_ENOUGH_MEMORY']:
              # Use magic to get a large property.
              prop_val = GetLargeProperty(item, prop_tag)
! 
          prop_repr = repr(prop_val)
!         if shorten:
!             prop_repr = prop_repr[:50]
          print "%-20s: %s" % (prop_name, prop_repr)
  
--- 51,88 ----
      return "".join(chunks)
  
! def FormatPropertyValue(prop_tag, prop_val, item, shorten, get_large_props):
!     # Do some magic rtf conversion
!     if PROP_ID(prop_tag) == PROP_ID(PR_RTF_COMPRESSED):
!         rtf_stream = item.OpenProperty(PR_RTF_COMPRESSED,
!                                        pythoncom.IID_IStream, 0, 0)
!         html_stream = mapi.WrapCompressedRTFStream(rtf_stream, 0)
!         prop_val = mapi.RTFStreamToHTML(html_stream)
!         prop_tag = PROP_TAG(PT_STRING8, PR_RTF_COMPRESSED)
!     prop_repr = None
!     if PROP_TYPE(prop_tag)==PT_ERROR:
          if get_large_props and \
!            prop_val in [mapi.MAPI_E_NOT_ENOUGH_MEMORY,
!                         'MAPI_E_NOT_ENOUGH_MEMORY']:
              # Use magic to get a large property.
              prop_val = GetLargeProperty(item, prop_tag)
!             prop_repr = repr(prop_val)
!         else:
!             prop_val = prop_repr = mapiutil.GetScodeString(prop_val)
!     if prop_repr is None:
          prop_repr = repr(prop_val)
!     if shorten:
!         prop_repr = prop_repr[:50]
!     return prop_repr
! 
! def DumpItemProps(item, shorten, get_large_props):
!     all_props = GetAllProperties(item)
!     all_props.sort() # sort by first tuple item, which is name :)
!     for prop_name, prop_tag, prop_val in all_props:
!         # If we want 'short' variables, drop 'not found' props.
!         if shorten and PROP_TYPE(prop_tag)==PT_ERROR \
!            and prop_val == mapi.MAPI_E_NOT_FOUND:
!             continue
!         prop_repr = FormatPropertyValue(prop_tag, prop_val, item,
!                                         shorten, get_large_props)
          print "%-20s: %s" % (prop_name, prop_repr)
  
***************
*** 78,98 ****
                  attach_num = row[0][1]
                  print "Dumping attachment (PR_ATTACH_NUM=%d)" % (attach_num,)
!                 attach = item.OpenAttach(attach_num, None, mapi.MAPI_DEFERRED_ERRORS)
                  DumpItemProps(attach, shorten, get_large)
              print
          print
  
! def usage(driver):
      folder_doc = driver.GetFolderNameDoc()
      msg = """\
! Usage: %s [-f foldername] subject of the message
  -f - Search for the message in the specified folder (default = Inbox)
  -s - Shorten long property values.
  -a - Include attachments
! -l - Get the data for very large properties
  -n - Show top-level folder names and exit
! 
! Dumps all properties for all messages that match the subject.  Subject
! matching is substring and ignore-case.
  
  %s
--- 99,166 ----
                  attach_num = row[0][1]
                  print "Dumping attachment (PR_ATTACH_NUM=%d)" % (attach_num,)
!                 attach = item.OpenAttach(attach_num, None,
!                                          mapi.MAPI_DEFERRED_ERRORS)
                  DumpItemProps(attach, shorten, get_large)
              print
          print
  
! # Generic table dumper.
! def DumpTable(driver, table, name_query_ob, shorten, large_props):
!     cols = table.QueryColumns(TBL_ALL_COLUMNS)
!     table.SetColumns(cols, 0)
!     rows = mapi.HrQueryAllRows(table, cols, None, None, 0)
!     print "Table has %d rows, each with %d columns" % (len(rows), len(cols))
!     for row in rows:
!         print "-- new row --"
!         for col in row:
!             prop_tag, prop_val = col
!             # If we want 'short' variables, drop 'not found' props.
!             if shorten and PROP_TYPE(prop_tag)==PT_ERROR \
!                and prop_val == mapi.MAPI_E_NOT_FOUND:
!                 continue
!             prop_name = GetPropTagName(name_query_ob, prop_tag)
!             prop_repr = FormatPropertyValue(prop_tag, prop_val, name_query_ob,
!                                             shorten, large_props)
!             print "%-20s: %s" % (prop_name, prop_repr)
! 
! # This dumps the raw binary data of the property Outlook uses to store
! # user defined fields.
! def FindAndDumpTableUserProps(driver, table, folder, shorten, get_large_props):
!     restriction = (mapi.RES_PROPERTY,
!                   (mapi.RELOP_EQ,
!                    PR_MESSAGE_CLASS_A,
!                    (PR_MESSAGE_CLASS_A, 'IPC.MS.REN.USERFIELDS')))
!     cols = (PR_USERFIELDS,)
!     table.SetColumns(cols, 0)
!     rows = mapi.HrQueryAllRows(table, cols, restriction, None, 0)
!     assert len(rows)<=1, "Only expecting 1 (or 0) rows"
!     tag, val = rows[0][0]
!     prop_name = GetPropTagName(folder, tag)
!     prop_repr = FormatPropertyValue(tag, val, folder,
!                                     shorten, get_large_props)
!     print "%-20s: %s" % (prop_name, prop_repr)
!     
! def usage(driver, extra = None):
      folder_doc = driver.GetFolderNameDoc()
+     if extra:
+         print extra
+         print
      msg = """\
! Usage: %s [options ...] subject of the message
! 
! Dumps all properties for all messages that match the subject.  Subject
! matching is substring and ignore-case.
! 
  -f - Search for the message in the specified folder (default = Inbox)
  -s - Shorten long property values.
  -a - Include attachments
! -l - Get the data for very large properties via a stream
  -n - Show top-level folder names and exit
! --dump-folder
!      Dump the properties of the specified folder.
! --dump-folder-assoc-contents
!      Dump the 'associated contents' table of the specified folder.
! --dump-folder-user-props
!      Find and dump the PR_USERFIELDS field for the specified table.
  
  %s
***************
*** 100,103 ****
--- 168,172 ----
      % (os.path.basename(sys.argv[0]),folder_doc)
      print msg
+     sys.exit(1)
  
  def main():
***************
*** 106,115 ****
      import getopt
      try:
!         opts, args = getopt.getopt(sys.argv[1:], "af:snl")
      except getopt.error, e:
!         print e
!         print
!         usage(driver)
!         sys.exit(1)
      folder_name = ""
  
--- 175,185 ----
      import getopt
      try:
!         opts, args = getopt.getopt(sys.argv[1:], "af:snl",
!                                    ["dump-folder",
!                                     "dump-folder-assoc-contents",
!                                     "dump-folder-user-props",
!                                     ])
      except getopt.error, e:
!         usage(driver, e)
      folder_name = ""
  
***************
*** 117,123 ****
--- 187,200 ----
      get_large_props = False
      include_attach = False
+     dump_folder = dump_folder_assoc_contents = dump_folder_user_props = False
      for opt, opt_val in opts:
          if opt == "-f":
              folder_name = opt_val
+         elif opt == "--dump-folder":
+             dump_folder = True
+         elif opt == "--dump-folder-assoc-contents":
+             dump_folder_assoc_contents = True
+         elif opt == "--dump-folder-user-props":
+             dump_folder_user_props = True
          elif opt == "-s":
              shorten = True
***************
*** 130,135 ****
              sys.exit(1)
          else:
!             print "Invalid arg"
!             return
  
      if not folder_name:
--- 207,211 ----
              sys.exit(1)
          else:
!             usage(driver, "Unknown arg '%s'" % opt)
  
      if not folder_name:
***************
*** 137,146 ****
  
      subject = " ".join(args)
!     if not subject:
!         print "You must specify a subject"
!         print
!         usage(driver)
!         sys.exit(1)
! 
      try:
          folder = driver.FindFolder(folder_name)
--- 213,224 ----
  
      subject = " ".join(args)
!     is_table_dump = dump_folder_assoc_contents or \
!                     dump_folder or dump_folder_user_props
!     if is_table_dump and subject or not is_table_dump and not subject:
!         if is_table_dump:
!             extra = "You must not specify a subject with '-p'"
!         else:
!             extra = "You must specify a subject (unless you use '-p')"
!         usage(driver, extra)
      try:
          folder = driver.FindFolder(folder_name)
***************
*** 149,153 ****
          sys.exit(1)
  
!     DumpProps(driver, folder, subject, include_attach, shorten, get_large_props)
  
  if __name__=='__main__':
--- 227,243 ----
          sys.exit(1)
  
!     if is_table_dump:
!         if dump_folder:
!             DumpItemProps(folder, shorten, get_large_props)
!         if dump_folder_assoc_contents:
!             table = folder.GetContentsTable(mapi.MAPI_ASSOCIATED)
!             DumpTable(driver, table, folder, shorten, get_large_props)
!         if dump_folder_user_props:
!             table = folder.GetContentsTable(mapi.MAPI_ASSOCIATED)
!             FindAndDumpTableUserProps(driver, table, folder,
!                                       shorten, get_large_props)
!     else:
!         DumpProps(driver, folder, subject, include_attach,
!                   shorten, get_large_props)
  
  if __name__=='__main__':



From mhammond at users.sourceforge.net  Wed Dec 10 00:08:39 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Wed Dec 10 00:08:42 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000/docs
	troubleshooting.html, 1.17, 1.18
Message-ID: <E1ATwaV-0004F5-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000/docs
In directory sc8-pr-cvs1:/tmp/cvs-serv16226/docs

Modified Files:
	troubleshooting.html 
Log Message:
Add link to the "Frequently Reported Bug List", and minor tweaks.


Index: troubleshooting.html
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/docs/troubleshooting.html,v
retrieving revision 1.17
retrieving revision 1.18
diff -C2 -d -r1.17 -r1.18
*** troubleshooting.html	8 Dec 2003 01:00:51 -0000	1.17
--- troubleshooting.html	10 Dec 2003 05:08:36 -0000	1.18
***************
*** 23,33 ****
    <li><a href="#All_other_problems">All other problems</a></li>
  </ul>
! There is also an <a href="http://spambayes.sourceforge.net/faq.html">online
! FAQ for SpamBayes</a> which may have additional information of
! use.&nbsp; You may also wish to see the <a
   href="http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/spambayes/spambayes/Outlook2000/docs/troubleshooting.html?rev=HEAD">latest
! online version of this document</a> for problems added since
! release.&nbsp; If you must send someone a mail about SpamBayes, <a
!  href="#mail">please read this first</a>.<br>
  <hr style="width: 100%; height: 2px;">
  <h2><a name="NoToolbar"></a>Toolbar items appear, but fail to work</h2>
--- 23,43 ----
    <li><a href="#All_other_problems">All other problems</a></li>
  </ul>
! Some other resources that may be useful in tracking down any problems:<br>
! <ul>
!   <li>There is an <a href="http://spambayes.sourceforge.net/faq.html">online
! FAQ for SpamBayes</a> with lots of information, including a section <a
!  href="http://spambayes.sourceforge.net/faq.html#outlook-plugin">specific
! to the Outlook addin</a>.</li>
!   <li>The <a
   href="http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/spambayes/spambayes/Outlook2000/docs/troubleshooting.html?rev=HEAD">latest
! online version of this document</a> may have information added since
! release.</li>
!   <li>The online <a
!  href="http://www.sourceforge.net/apps/outlook/bugs.html">Frequently
! Reported Outlook Bugs</a> list has the most common problems people are
! experiencing, and links to more information about the bug.</li>
! </ul>
! If you must send someone a mail about SpamBayes, <a href="#mail">please
! read this first</a>.<br>
  <hr style="width: 100%; height: 2px;">
  <h2><a name="NoToolbar"></a>Toolbar items appear, but fail to work</h2>
***************
*** 204,208 ****
  a particular machine, which can be useful in enterprise arrangements
  where users have 'roaming profiles'<br>
! To register SpamBayes in this way, you must execute the command
  (obviously with the correct patch substituted):<br>
  <span style="font-style: italic;">&nbsp;&nbsp;&nbsp; regsvr32.exe
--- 214,219 ----
  a particular machine, which can be useful in enterprise arrangements
  where users have 'roaming profiles'<br>
! To register SpamBayes in this way, you must log on as a user with
! permissions to modify the system registry, then execute the command
  (obviously with the correct patch substituted):<br>
  <span style="font-style: italic;">&nbsp;&nbsp;&nbsp; regsvr32.exe



From mhammond at users.sourceforge.net  Wed Dec 10 00:28:12 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Wed Dec 10 00:28:16 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 addin.py,1.115,1.116
Message-ID: <E1ATwtQ-0004r7-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv18472

Modified Files:
	addin.py 
Log Message:
Try and add the Spam field to the 'Unsure' folder in the same way we
do for the Spam and watch folders.


Index: addin.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/addin.py,v
retrieving revision 1.115
retrieving revision 1.116
diff -C2 -d -r1.115 -r1.116
*** addin.py	8 Dec 2003 00:55:43 -0000	1.115
--- addin.py	10 Dec 2003 05:28:10 -0000	1.116
***************
*** 1266,1269 ****
--- 1266,1283 ----
              self.manager.ReportFatalStartupError(
                  "Could not watch the specified folders")
+         # UpdateFolderHooks takes care of ensuring the Outlook field exists
+         # for all folders we watch - but we never watch the 'Unsure'
+         # folder, and this one is arguably the most important to have it.
+         unsure_id = self.manager.config.filter.unsure_folder_id
+         if unsure_id:
+             try:
+                 self.manager.EnsureOutlookFieldsForFolder(unsure_id)
+             except:
+                 # If this fails, just log an error - don't bother with
+                 # the traceback
+                 print "Error adding field to 'Unsure' folder %r" % (unsure_id,)
+                 etype, value, tb = sys.exc_info()
+                 tb = None # dont want it, and nuke circular ref
+                 traceback.print_exception(etype, value, tb)
  
      def UpdateFolderHooks(self):



From mhammond at users.sourceforge.net  Wed Dec 10 00:30:53 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Wed Dec 10 00:30:58 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 manager.py, 1.90,
	1.91 msgstore.py, 1.77, 1.78
Message-ID: <E1ATww1-0004wN-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv18693

Modified Files:
	manager.py msgstore.py 
Log Message:
Change the way we check for and create user properties:
* Reach into where Outlook actually stores UserProperties to check
  if the field exists.  This can be done without opening an item in the
  folder.
* If the field does not exist, create a temporary message and add the
  properties via that, then delete the temporary message.  This then
  leaves the field in the folder, even when the folder has no items.
Fixes [ 856141 ] Spam field not added to unsure or empty folders


Index: manager.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/manager.py,v
retrieving revision 1.90
retrieving revision 1.91
diff -C2 -d -r1.90 -r1.91
*** manager.py	8 Dec 2003 12:07:37 -0000	1.90
--- manager.py	10 Dec 2003 05:30:50 -0000	1.91
***************
*** 528,539 ****
  
      def EnsureOutlookFieldsForFolder(self, folder_id, include_sub=False):
          # Ensure that our fields exist on the Outlook *folder*
!         # Setting properties via our msgstore (via Ext Mapi) gets the props
          # on the message OK, but Outlook doesn't see it as a "UserProperty".
          # Using MAPI to set them directly on the folder also has no effect.
!         # So until we know better, use Outlook to hack this in.
!         # Should be called once per folder you are watching/filtering etc
!         #
!         # Oh the tribulations of our property grail
          # We originally wanted to use the "Integer" Outlook field,
          # but it seems this property type alone is not expose via the Object
--- 528,554 ----
  
      def EnsureOutlookFieldsForFolder(self, folder_id, include_sub=False):
+         # Should be called at least once once per folder you are
+         # watching/filtering etc
          # Ensure that our fields exist on the Outlook *folder*
!         # Setting properties via our msgstore (via Ext Mapi) sets the props
          # on the message OK, but Outlook doesn't see it as a "UserProperty".
          # Using MAPI to set them directly on the folder also has no effect.
!         # Later: We have since discovered that Outlook stores user property
!         # information in the 'associated contents' folder - see 
!         # msgstore.MAPIMsgStoreFolder.DoesFolderHaveOutlookField() for more
!         # details.  We can reverse engineer this well enough to determine
!         # if a property exists, but not well enough to actually add a
!         # property.  Thus, we resort to the Outlook object model to actually
!         # add it.
!         # Note that this means we need an object in the folder to modify.
!         # We could go searching for an existing item then modify and save it
!         # (indeed, we did once), but this could be bad-form, as the message
!         # we randomly choose to modify will then have a meaningless 'Spam'
!         # field.  If we are going to go to the effort of creating a temp
!         # item when no item exists, we may as well do it all the time,
!         # especially now we know how to check if the folder has the field
!         # without opening an Outlook item.
! 
!         # Regarding the property type:
          # We originally wanted to use the "Integer" Outlook field,
          # but it seems this property type alone is not expose via the Object
***************
*** 541,600 ****
          # (which really is OK!)
          assert self.outlook is not None, "I need outlook :("
!         try:
!             msgstore_folder = self.message_store.GetFolder(folder_id)
!         except self.message_store.MsgStoreException:
!             print "Checking a folder for our field failed - "\
!                   "there is no such folder."
!             return
!             
!         folder = msgstore_folder.GetOutlookItem()
!         folder_name = msgstore_folder.GetFQName()
!         self.LogDebug(2, "Checking folder '%s' for field '%s'" \
!                   % (folder_name, self.config.general.field_score_name))
!         items = folder.Items
!         item = items.GetFirst()
!         while item is not None:
!             if item.Class != win32com.client.constants.olMail:
!                 item = items.GetNext()
                  continue
!             break
!         # OK - item is either a mail item, or None
!         if item is not None:
!             ups = item.UserProperties
!             # *sigh* - need to search by int index
!             for i in range(ups.Count):
!                 up = ups[i+1]
!                 if up.Name == self.config.general.field_score_name:
!                     break
!             else: # for not broken
!                 try:
!                     # Display format is documented as being the 1-based index in
!                     # the combo box in the outlook UI for the given data type.
!                     # 1 is the first - "Rounded", which seems fine.
!                     format = 1
!                     ups.Add(self.config.general.field_score_name,
!                            win32com.client.constants.olPercent,
!                            True, # Add to folder
!                            format)
!                     item.Save()
!                     self.LogDebug(2, "Created the UserProperty!")
!                 except pythoncom.com_error, details:
!                     if msgstore.IsReadOnlyCOMException(details):
!                         self.LogDebug(1, "The folder '%s' is read-only - user property can't be added" % \
!                                       (folder_name,))
!                     else:
!                         print "Warning: failed to create the Outlook " \
!                               "user-property in folder '%s'" \
!                               % (folder_name,)
!                         print "", details
!             if include_sub:
!                 # Recurse down the folder list.
!                 folders = item.Parent.Folders
!                 folder = folders.GetFirst()
!                 while folder is not None:
!                     this_id = folder.StoreID, folder.EntryID
!                     self.EnsureOutlookFieldsForFolder(this_id, True)
!                     folder = folders.GetNext()
!         # else no items in this folder - not much worth doing!
  
      def PrepareConfig(self):
--- 556,602 ----
          # (which really is OK!)
          assert self.outlook is not None, "I need outlook :("
!         field_name = self.config.general.field_score_name
!         for msgstore_folder in self.message_store.GetFolderGenerator(
!                                                     [folder_id], include_sub):
!             folder_name = msgstore_folder.GetFQName()
!             if msgstore_folder.DoesFolderHaveOutlookField(field_name):
!                 self.LogDebug(1, "Folder '%s' already has field '%s'" \
!                                  % (folder_name, field_name))
                  continue
!             self.LogDebug(0, "Folder '%s' has no field named '%s' - creating" \
!                       % (folder_name, field_name))
!             # Creating the item via the Outlook model does some strange
!             # things (such as moving it to "Drafts" on save), so we create
!             # it using extended MAPI (via our msgstore)
!             message = msgstore_folder.CreateTemporaryMessage(msg_flags=1)
!             outlook_message = message.GetOutlookItem()
!             ups = outlook_message.UserProperties
!             try:
!                 # Display format is documented as being the 1-based index in
!                 # the combo box in the outlook UI for the given data type.
!                 # 1 is the first - "Rounded", which seems fine.
!                 format = 1
!                 ups.Add(field_name,
!                        win32com.client.constants.olPercent,
!                        True, # Add to folder
!                        format)
!                 outlook_message.Save()
!             except pythoncom.com_error, details:
!                 if msgstore.IsReadOnlyCOMException(details):
!                     self.LogDebug(1, "The folder '%s' is read-only - user "
!                                      "property can't be added" % (folder_name,))
!                 else:
!                     print "Warning: failed to create the Outlook " \
!                           "user-property in folder '%s'" \
!                           % (folder_name,)
!                     print "", details
!             msgstore_folder.DeleteMessages((message,))
!             # Check our DoesFolderHaveOutlookField logic holds up.
!             if not msgstore_folder.DoesFolderHaveOutlookField(field_name):
!                 self.LogDebug(0,
!                         "WARNING: We just created the user field in folder "
!                         "%s, but it appears to not exist.  Something is "
!                         "probably wrong with DoesFolderHaveOutlookField()" % \
!                         folder_name)
  
      def PrepareConfig(self):
***************
*** 815,818 ****
--- 817,824 ----
          # And re-save now, just incase Outlook dies on the way down.
          self.SaveConfig()
+         # And tell the addin that our filters may have changed.
+         if self.addin is not None:
+             self.addin.FiltersChanged()
+ 
      def ShowFilterNow(self):
          import dialogs

Index: msgstore.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/msgstore.py,v
retrieving revision 1.77
retrieving revision 1.78
diff -C2 -d -r1.77 -r1.78
*** msgstore.py	10 Oct 2003 01:01:12 -0000	1.77
--- msgstore.py	10 Dec 2003 05:30:50 -0000	1.78
***************
*** 624,627 ****
--- 624,708 ----
          return self._FolderFromMAPIFolder(ret)
  
+     def DoesFolderHaveOutlookField(self, field_name):
+         # Returns True if the specified folder has an *Outlook* field with
+         # the given name, False if the folder does not have it, or None
+         # if we can't tell, or there was an error, etc.
+         # We have discovered that Outlook stores 'Fields' for a folder as a
+         # PR_USERFIELDS field in the hidden, 'associated' message with
+         # message class IPC.MS.REN.USERFIELDS.  This is a binary property
+         # which is undocumented, but probably could be reverse-engineered
+         # with a little effort (see 'dump_props --dump-folder-user-props' for
+         # an example of the raw data.  For now, the simplest thing appears
+         # to be to check for a \0 character, followed by the property name
+         # as an ascii string.
+         try:
+             folder = self.msgstore._OpenEntry(self.id)
+             table = folder.GetContentsTable(mapi.MAPI_ASSOCIATED)
+             restriction = (mapi.RES_PROPERTY,
+                           (mapi.RELOP_EQ,
+                            PR_MESSAGE_CLASS_A,
+                            (PR_MESSAGE_CLASS_A, 'IPC.MS.REN.USERFIELDS')))
+             cols = (PR_USERFIELDS,)
+             table.SetColumns(cols, 0)
+             rows = mapi.HrQueryAllRows(table, cols, restriction, None, 0)
+             if len(rows)>1:
+                 print "Eeek - only expecting one row from IPC.MS.REN.USERFIELDS"
+                 print "got", repr(rows)
+                 return None
+             if len(rows)==0:
+                 # New folders with no userdefined fields do not have such a row,
+                 # but this is a clear indication it does not exist.
+                 return False
+             row = rows[0]
+             val = GetPotentiallyLargeStringProp(folder, cols[0], row[0])
+         except pythoncom.com_error, details:
+             raise MsgStoreExceptionFromCOMException(details)
+         if type(val) != type(''):
+             print "Value type incorrect - expected string, got", repr(val)
+             return None
+         return val.find("\0" + field_name) >= 0
+ 
+     def DeleteMessages(self, message_things):
+         # A *permanent* delete - MAPI has no concept of 'Deleted Items',
+         # only Outlook does.  If you want a "soft" delete, you must locate
+         # deleted item (via a special ID) and move it to there yourself
+         # message_things may be ID tuples, or MAPIMsgStoreMsg instances.
+         real_ids = []
+         for thing in message_things:
+             if isinstance(thing, MAPIMsgStoreMsg):
+                 real_ids.append( thing.id[1] )
+                 thing.mapi_object = thing.id = thing.folder_id = None
+             else:
+                 real_ids.append(self.msgstore.NormalizeID(thing)[1])
+         try:
+             folder = self.msgstore._OpenEntry(self.id)
+             # Nuke my MAPI reference, and set my ID to None
+             rc = folder.DeleteMessages(real_ids, 0, None, 0)
+         except pythoncom.com_error, details:
+             raise MsgStoreExceptionFromCOMException(details)
+     
+     def CreateTemporaryMessage(self, msg_flags = None):
+         # Create a message designed to be used temporarily.  It is your
+         # responsibility to delete when you are done with it.
+         # If msg_flags is not None, it should be an integer for the
+         # PR_MESSAGE_FLAGS property.  Note that Outlook appears to refuse
+         # to set user properties on a message marked as 'unsent', which
+         # is the default.  Setting to, eg, 1 marks it as a "not unsent, read"
+         # message, which works fine with user properties.
+         try:
+             folder = self.msgstore._OpenEntry(self.id)
+             imsg = folder.CreateMessage(None, 0)
+             if msg_flags is not None:
+                 props = (PR_MESSAGE_FLAGS,msg_flags),
+                 imsg.SetProps(props)
+             imsg.SaveChanges(0)
+             hr, data = imsg.GetProps((PR_ENTRYID, PR_STORE_ENTRYID), 0)
+             eid = data[0][1]
+             storeid = data[1][1]
+             msg_id = mapi.HexFromBin(storeid), mapi.HexFromBin(eid)
+         except pythoncom.com_error, details:
+             raise MsgStoreExceptionFromCOMException(details)
+         return self.msgstore.GetMessage(msg_id)
+ 
  class MAPIMsgStoreMsg:
      # All the properties we must initialize a message with.



From mhammond at users.sourceforge.net  Wed Dec 10 02:18:55 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Wed Dec 10 02:18:59 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 msgstore.py,1.78,1.79
Message-ID: <E1ATycZ-0000SF-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv1736

Modified Files:
	msgstore.py 
Log Message:
PR_USERFIELDS only in newer win32alls.


Index: msgstore.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/msgstore.py,v
retrieving revision 1.78
retrieving revision 1.79
diff -C2 -d -r1.78 -r1.79
*** msgstore.py	10 Dec 2003 05:30:50 -0000	1.78
--- msgstore.py	10 Dec 2003 07:18:53 -0000	1.79
***************
*** 17,20 ****
--- 17,25 ----
  import winerror
  
+ try:
+     PR_USERFIELDS # only in new win32all
+ except NameError:
+     PR_USERFIELDS = 0x36E30102 # PROP_TAG(PT_BINARY, 0x36e3)
+ 
  # Additional MAPI constants we dont have in Python
  MESSAGE_MOVE = 0x1 # from MAPIdefs.h



From anadelonbrin at users.sourceforge.net  Wed Dec 10 21:56:09 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Wed Dec 10 21:56:13 2003
Subject: [Spambayes-checkins] website faq.txt,1.53,1.54
Message-ID: <E1AUGzp-0007rf-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/website
In directory sc8-pr-cvs1:/tmp/cvs-serv30214

Modified Files:
	faq.txt 
Log Message:
Everyone seems to be deleting their spam folder these days.
Explain how to fix that.

Index: faq.txt
===================================================================
RCS file: /cvsroot/spambayes/website/faq.txt,v
retrieving revision 1.53
retrieving revision 1.54
diff -C2 -d -r1.53 -r1.54
*** faq.txt	9 Dec 2003 03:59:00 -0000	1.53
--- faq.txt	11 Dec 2003 02:56:07 -0000	1.54
***************
*** 629,632 ****
--- 629,658 ----
  
  
+ Help!  I deleted the Unsure/Spam folder.
+ ----------------------------------------
+ Firstly, if you haven't emptied your Deleted Items folder, you can probably
+ find the unsure/spam folder in it and recover it that way.  The little plus
+ next to the Deleted Items folder will reveal it (if there's no plus, there's
+ no folder to recover), and you can drag it to where it belongs.
+ 
+ Otherwise, you need to open up the SpamBayes Manager dialog, click the
+ "Filtering" tab, and reenter the folder selection (you might need to create
+ a new unsure/spam folder first - just like you'd create a folder normally).
+ The dialog will probably have "<unknown folder>" in the place where the
+ unsure/spam folder used to be.  Just click Browse and select the correct
+ folder.
+ 
+ This should always work.  If you're really stuck, though, you can reset
+ your configuration completely (you'll have to redo any setup, but *not*
+ any training) by deleting the appropriate .ini file from your `Data Directory`_.
+ You want to remove the .ini ("Configuration file") that has the same
+ name as your Outlook profile (for some, this will be something generic like
+ "Outlook.ini", or "Outlook Internet Settings.ini").  Just remove or rename
+ this file and your configuration will be reset.  Then select the folders as you
+ would normally.
+ 
+ .. _`Data Directory`: #can-i-back-up-the-outlook-database-should-i-do-this
+ 
+ 
  How do I uninstall the plug-in?
  -------------------------------



From montanaro at users.sourceforge.net  Thu Dec 11 13:40:23 2003
From: montanaro at users.sourceforge.net (Skip Montanaro)
Date: Thu Dec 11 13:40:41 2003
Subject: [Spambayes-checkins] spambayes/spambayes Corpus.py,1.9,1.10
Message-ID: <E1AUVjb-0000KN-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv1217

Modified Files:
	Corpus.py 
Log Message:
conservative pychecker cleanup


Index: Corpus.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/Corpus.py,v
retrieving revision 1.9
retrieving revision 1.10
diff -C2 -d -r1.9 -r1.10
*** Corpus.py	2 Oct 2003 06:01:12 -0000	1.9
--- Corpus.py	11 Dec 2003 18:40:21 -0000	1.10
***************
*** 92,96 ****
  import sys           # for output of docstring
  import time
- import re
  from spambayes import tokenizer
  from spambayes.Options import options
--- 92,95 ----
***************
*** 190,194 ****
  
      def get(self, key, default=None):
!         if self.msgs.get(key, "") is "":
              return default
          else:
--- 189,193 ----
  
      def get(self, key, default=None):
!         if self.msgs.get(key, "") == "":
              return default
          else:



From montanaro at users.sourceforge.net  Thu Dec 11 13:40:56 2003
From: montanaro at users.sourceforge.net (Skip Montanaro)
Date: Thu Dec 11 13:41:00 2003
Subject: [Spambayes-checkins] spambayes/spambayes Dibbler.py,1.9,1.10
Message-ID: <E1AUVk8-0000Lc-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv1325

Modified Files:
	Dibbler.py 
Log Message:
conservative pychecker cleanup


Index: Dibbler.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/Dibbler.py,v
retrieving revision 1.9
retrieving revision 1.10
diff -C2 -d -r1.9 -r1.10
*** Dibbler.py	6 Nov 2003 23:44:59 -0000	1.9
--- Dibbler.py	11 Dec 2003 18:40:54 -0000	1.10
***************
*** 728,732 ****
      """Run a self-test."""
      # Run the calendar server in a separate thread.
!     import re, threading, urllib
      testServerReady = threading.Event()
      threading.Thread(target=runTestServer, args=(testServerReady,)).start()
--- 728,732 ----
      """Run a self-test."""
      # Run the calendar server in a separate thread.
!     import threading, urllib
      testServerReady = threading.Event()
      threading.Thread(target=runTestServer, args=(testServerReady,)).start()



From montanaro at users.sourceforge.net  Thu Dec 11 13:41:36 2003
From: montanaro at users.sourceforge.net (Skip Montanaro)
Date: Thu Dec 11 13:41:41 2003
Subject: [Spambayes-checkins] spambayes/spambayes FileCorpus.py,1.8,1.9
Message-ID: <E1AUVkm-0000Nx-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv1442

Modified Files:
	FileCorpus.py 
Log Message:
conservative pychecker cleanup


Index: FileCorpus.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/FileCorpus.py,v
retrieving revision 1.8
retrieving revision 1.9
diff -C2 -d -r1.8 -r1.9
*** FileCorpus.py	30 Sep 2003 03:05:13 -0000	1.8
--- FileCorpus.py	11 Dec 2003 18:41:34 -0000	1.9
***************
*** 88,92 ****
  from spambayes import message
  from spambayes import storage
! import sys, os, gzip, fnmatch, getopt, errno, time, stat
  from spambayes.Options import options
  
--- 88,92 ----
  from spambayes import message
  from spambayes import storage
! import sys, os, gzip, fnmatch, getopt, time, stat
  from spambayes.Options import options
  
***************
*** 271,275 ****
          '''Instance as a representative string'''
  
-         elip = ''
          sub = self.as_string()
  
--- 271,274 ----
***************
*** 298,302 ****
          try:
              stats = os.stat(self.pathname())
!         except OSError, e:
              ctime = time.time()
          else:
--- 297,301 ----
          try:
              stats = os.stat(self.pathname())
!         except OSError:
              ctime = time.time()
          else:
***************
*** 506,510 ****
              raise
      else:
!         for filename in os.listdir(dirname):
              fn = os.path.join(dirname, filename)
              os.unlink(fn)
--- 505,509 ----
              raise
      else:
!         for filename in flist:
              fn = os.path.join(dirname, filename)
              os.unlink(fn)



From montanaro at users.sourceforge.net  Thu Dec 11 13:42:43 2003
From: montanaro at users.sourceforge.net (Skip Montanaro)
Date: Thu Dec 11 13:42:46 2003
Subject: [Spambayes-checkins] spambayes/spambayes ImapUI.py,1.24,1.25
Message-ID: <E1AUVlr-0000RP-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv1686

Modified Files:
	ImapUI.py 
Log Message:
conservative pychecker cleanup


Index: ImapUI.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/ImapUI.py,v
retrieving revision 1.24
retrieving revision 1.25
diff -C2 -d -r1.24 -r1.25
*** ImapUI.py	5 Dec 2003 04:41:25 -0000	1.24
--- ImapUI.py	11 Dec 2003 18:42:41 -0000	1.25
***************
*** 116,119 ****
--- 116,121 ----
              parm_list.remove(("imap", "use_ssl"))
              parm_map = tuple(parm_list)
+         else:
+             del IMAP4_SSL
          UserInterface.UserInterface.__init__(self, cls, parm_map, adv_map)
          self.classifier = cls



From montanaro at users.sourceforge.net  Thu Dec 11 13:43:19 2003
From: montanaro at users.sourceforge.net (Skip Montanaro)
Date: Thu Dec 11 13:43:22 2003
Subject: [Spambayes-checkins] spambayes/spambayes ProxyUI.py,1.29,1.30
Message-ID: <E1AUVmR-0000T2-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv1788

Modified Files:
	ProxyUI.py 
Log Message:
conservative pychecker cleanup


Index: ProxyUI.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/ProxyUI.py,v
retrieving revision 1.29
retrieving revision 1.30
diff -C2 -d -r1.29 -r1.30
*** ProxyUI.py	21 Oct 2003 03:58:49 -0000	1.29
--- ProxyUI.py	11 Dec 2003 18:43:17 -0000	1.30
***************
*** 68,72 ****
  import UserInterface
  from spambayes.Options import options
- import spambayes.mboxutils
  from email.Iterators import typed_subpart_iterator
  
--- 68,71 ----
***************
*** 414,418 ****
          # that match those criteria.
          elif params.get('find') is not None:
!             prior = this = next = 0
              keys = Set()        # so we don't end up with duplicates
              push = keys.add
--- 413,417 ----
          # that match those criteria.
          elif params.get('find') is not None:
!             prior = next = 0
              keys = Set()        # so we don't end up with duplicates
              push = keys.add



From montanaro at users.sourceforge.net  Thu Dec 11 13:44:02 2003
From: montanaro at users.sourceforge.net (Skip Montanaro)
Date: Thu Dec 11 13:44:05 2003
Subject: [Spambayes-checkins] spambayes/spambayes Stats.py,1.2,1.3
Message-ID: <E1AUVn8-0000Xq-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv1859

Modified Files:
	Stats.py 
Log Message:
conservative pychecker cleanup


Index: Stats.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/Stats.py,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** Stats.py	6 Oct 2003 02:29:58 -0000	1.2
--- Stats.py	11 Dec 2003 18:44:00 -0000	1.3
***************
*** 88,93 ****
          perc_spam = 100.0 * self.cls_spam / self.total
          perc_unsure = 100.0 * self.cls_unsure / self.total
!         format_dict = dict(perc_spam=perc_spam, perc_ham=perc_ham,
!                            perc_unsure=perc_unsure, num_seen = self.total)
          format_dict.update(self.__dict__)
          push("SpamBayes has processed %(num_seen)d messages - " \
--- 88,97 ----
          perc_spam = 100.0 * self.cls_spam / self.total
          perc_unsure = 100.0 * self.cls_unsure / self.total
!         format_dict = {
!             'perc_spam': perc_spam,
!             'perc_ham': perc_ham,
!             'perc_unsure': perc_unsure,
!             'num_seen': self.total
!             }
          format_dict.update(self.__dict__)
          push("SpamBayes has processed %(num_seen)d messages - " \



From montanaro at users.sourceforge.net  Thu Dec 11 13:44:25 2003
From: montanaro at users.sourceforge.net (Skip Montanaro)
Date: Thu Dec 11 13:44:28 2003
Subject: [Spambayes-checkins] spambayes/spambayes UserInterface.py, 1.34,
	1.35
Message-ID: <E1AUVnV-0000Z5-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv2155

Modified Files:
	UserInterface.py 
Log Message:
conservative pychecker cleanup


Index: UserInterface.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/UserInterface.py,v
retrieving revision 1.34
retrieving revision 1.35
diff -C2 -d -r1.34 -r1.35
*** UserInterface.py	26 Nov 2003 00:03:30 -0000	1.34
--- UserInterface.py	11 Dec 2003 18:44:23 -0000	1.35
***************
*** 1011,1015 ****
  
          from email import Encoders
-         from email.Message import Message
          from email.MIMEBase import MIMEBase
          from email.MIMEAudio import MIMEAudio
--- 1011,1014 ----



From montanaro at users.sourceforge.net  Thu Dec 11 13:44:49 2003
From: montanaro at users.sourceforge.net (Skip Montanaro)
Date: Thu Dec 11 13:44:51 2003
Subject: [Spambayes-checkins] spambayes/spambayes Version.py,1.24,1.25
Message-ID: <E1AUVnt-0000a7-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv2203

Modified Files:
	Version.py 
Log Message:
conservative pychecker cleanup


Index: Version.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/Version.py,v
retrieving revision 1.24
retrieving revision 1.25
diff -C2 -d -r1.24 -r1.25
*** Version.py	18 Sep 2003 22:56:09 -0000	1.24
--- Version.py	11 Dec 2003 18:44:47 -0000	1.25
***************
*** 203,206 ****
--- 203,207 ----
  
  def main(args):
+     import sys
      if '-g' in args:
          make_cfg(sys.stdout)



From montanaro at users.sourceforge.net  Thu Dec 11 13:45:09 2003
From: montanaro at users.sourceforge.net (Skip Montanaro)
Date: Thu Dec 11 13:45:20 2003
Subject: [Spambayes-checkins] spambayes/spambayes cdb.py,1.2,1.3
Message-ID: <E1AUVoD-0000c8-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv2285

Modified Files:
	cdb.py 
Log Message:
conservative pychecker cleanup


Index: cdb.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/cdb.py,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** cdb.py	14 Jan 2003 05:38:20 -0000	1.2
--- cdb.py	11 Dec 2003 18:45:07 -0000	1.3
***************
*** 12,16 ****
  import struct
  import mmap
- import sys
  
  def uint32_unpack(buf):
--- 12,15 ----
***************
*** 52,56 ****
      def __iter__(self, fn=None):
          len = 2048
-         ret = []
          while len < self.eod:
              klen, vlen = struct.unpack("<LL", self.map[len:len+8])
--- 51,54 ----



From montanaro at users.sourceforge.net  Thu Dec 11 13:45:33 2003
From: montanaro at users.sourceforge.net (Skip Montanaro)
Date: Thu Dec 11 13:45:36 2003
Subject: [Spambayes-checkins] spambayes/spambayes message.py,1.43,1.44
Message-ID: <E1AUVob-0000dE-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv2409

Modified Files:
	message.py 
Log Message:
conservative pychecker cleanup


Index: message.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/message.py,v
retrieving revision 1.43
retrieving revision 1.44
diff -C2 -d -r1.43 -r1.44
*** message.py	26 Nov 2003 22:09:51 -0000	1.43
--- message.py	11 Dec 2003 18:45:31 -0000	1.44
***************
*** 92,97 ****
  import math
  import re
- import sys
- import types
  import errno
  import shelve
--- 92,95 ----



From montanaro at users.sourceforge.net  Thu Dec 11 13:45:49 2003
From: montanaro at users.sourceforge.net (Skip Montanaro)
Date: Thu Dec 11 13:45:52 2003
Subject: [Spambayes-checkins] spambayes/spambayes oe_mailbox.py,1.1,1.2
Message-ID: <E1AUVor-0000dt-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv2462

Modified Files:
	oe_mailbox.py 
Log Message:
conservative pychecker cleanup


Index: oe_mailbox.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/oe_mailbox.py,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** oe_mailbox.py	20 Aug 2003 08:33:26 -0000	1.1
--- oe_mailbox.py	11 Dec 2003 18:45:46 -0000	1.2
***************
*** 279,283 ****
    def getIndexDataType(self, dbxIndex):
      """Returns the data type of the given index."""
!     return dt_None
  
    def getValue(self, dbxIndex):
--- 279,283 ----
    def getIndexDataType(self, dbxIndex):
      """Returns the data type of the given index."""
!     return DT_NONE
  
    def getValue(self, dbxIndex):



From mhammond at users.sourceforge.net  Fri Dec 12 01:29:59 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Fri Dec 12 01:30:02 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000/docs
	troubleshooting.html, 1.18, 1.19
Message-ID: <E1AUgoJ-00061G-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000/docs
In directory sc8-pr-cvs1:/tmp/cvs-serv23045

Modified Files:
	troubleshooting.html 
Log Message:
Add information about the data directory and configuration files.


Index: troubleshooting.html
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/docs/troubleshooting.html,v
retrieving revision 1.18
retrieving revision 1.19
diff -C2 -d -r1.18 -r1.19
*** troubleshooting.html	10 Dec 2003 05:08:36 -0000	1.18
--- troubleshooting.html	12 Dec 2003 06:29:56 -0000	1.19
***************
*** 18,21 ****
--- 18,23 ----
    <li><a href="#Messages_have_incorrect_or_unexpected">Messages have
  incorrect or unexpected Spam values</a></li>
+   <li><a href="#ResetConfiguration">Resetting SpamBayes configuration</a><br>
+   </li>
    <li><a href="#register-all-users">SpamBayes is not available for all
  users on the machine</a><br>
***************
*** 205,208 ****
--- 207,241 ----
    </ul>
  </ul>
+ <h2><a name="ResetConfiguration"></a>Resetting SpamBayes configuration</h2>
+ In some cases, it may become necessary to reset your SpamBayes
+ configuration, especially if your configuration becomes invalid.&nbsp;
+ SpamBayes attempts to detect this situation, but doesn't always get it
+ right.&nbsp; This section details where critical configuration files
+ are stored.<br>
+ SpamBayes stores all configuration data in <a
+  href="#LocateDataDirectory">your data directory</a>.&nbsp; The
+ configuration information is stored in a file called <span
+  style="font-style: italic;">[profile name].ini</span>, where <span
+  style="font-style: italic;">profile name</span> is the name of your
+ Microsoft Outlook profile.&nbsp; The default profile name is usually <span
+  style="font-style: italic;">Outlook</span>
+ or <span style="font-style: italic;">Microsoft Outlook&nbsp; Internet
+ Settings</span>, but Outlook can be configured to use any number of
+ profiles, with any name.<br>
+ Note that in this directory, you will also find a file named <span
+  style="font-style: italic;">default_bayes_customize.ini</span> - this
+ file <span style="font-weight: bold;">is not </span>used to configure
+ the Outlook side of SpamBayes - look for any other .ini files in that
+ directory.<br>
+ If you delete the configuration file, SpamBayes will be completely
+ reset.&nbsp; Note you will not lose your training data, only your
+ configuration information.&nbsp; The next time you start Outlook, the
+ SpamBayes configuration wizard should appear, guiding you through the
+ configuration process<br>
+ Your training data is also stored in this directory, but in files with
+ a <span style="font-style: italic;">.db</span> extension.&nbsp; If you
+ ever want to delete your training information, remove the two <span
+  style="font-style: italic;">.db</span> files in this directory.<br>
+ You may like to consider backing up this directory.<br>
  <h2><a name="register-all-users"></a>SpamBayes is not available for all
  users on the machine.</h2>
***************
*** 311,315 ****
  information, or zip it up.&nbsp; If you don't know what that means,
  please send a mail.<br>
! <h3><a name="ReportABug"></a>Report a bug</h3>
  All SpamBayes bugs are maintained in <a
   href="http://sourceforge.net/tracker/?group_id=61702&amp;atid=498103">this
--- 344,370 ----
  information, or zip it up.&nbsp; If you don't know what that means,
  please send a mail.<br>
! <h3><a name="LocateDataDirectory"></a>Locating your Data Directory</h3>
! SpamBayes stores all configuration and database information in a single
! directory.&nbsp; By default, this directory is located under the user's
! <span style="font-style: italic;">Application Data</span>
! directory.&nbsp; You can locate this directory by
! using the <span style="font-style: italic;">Show Data Folder</span>
! button on the <span style="font-style: italic;">Advanced</span> tab of
! the main SpamBayes Manager dialog.<br>
! If you need to locate it by hand, on
! Windows NT/2000/XP, it will probably be <span
!  style="font-style: italic;">C:\Documents
! and Settings\[username]\Application Data\Spambayes</span>, or on other
! versions of Windows it will probably be <span
!  style="font-style: italic;">C:\Windows\Application Data\Spambayes</span>.&nbsp;
! Note that the <span style="font-style: italic;">Application Data </span>folder
! may be hidden, so Windows
! Explorer may not show it by default, but you can enter the path into
! the Address Bar and Explorer will open it.<br>
! Note that by modifying the configuration files, you can tell SpamBayes
! to store this data in any directory, so it is possible your data is
! being stored elsewhere - contact your network administrator if this
! appear to be the case.
! <h3>Report a bug</h3>
  All SpamBayes bugs are maintained in <a
   href="http://sourceforge.net/tracker/?group_id=61702&amp;atid=498103">this



From montanaro at users.sourceforge.net  Fri Dec 12 15:21:17 2003
From: montanaro at users.sourceforge.net (Skip Montanaro)
Date: Fri Dec 12 15:21:22 2003
Subject: [Spambayes-checkins] spambayes/contrib spamcounts.py,1.3,1.4
Message-ID: <E1AUtmn-0000ku-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/contrib
In directory sc8-pr-cvs1:/tmp/cvs-serv2789/contrib

Modified Files:
	spamcounts.py 
Log Message:
initialize usepickle from persisten_use_database option


Index: spamcounts.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/contrib/spamcounts.py,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** spamcounts.py	5 Dec 2003 21:23:35 -0000	1.3
--- spamcounts.py	12 Dec 2003 20:21:15 -0000	1.4
***************
*** 97,101 ****
      usere = False
      dbname = options["Storage", "persistent_storage_file"]
!     ispickle = False
      tokenizestdin = False
      for opt, arg in opts:
--- 97,101 ----
      usere = False
      dbname = options["Storage", "persistent_storage_file"]
!     ispickle = not options["Storage", "persistent_use_database"]
      tokenizestdin = False
      for opt, arg in opts:



From anadelonbrin at users.sourceforge.net  Sat Dec 13 21:08:27 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Sun Dec 14 03:35:57 2003
Subject: [Spambayes-checkins] spambayes/spambayes ImapUI.py,1.26,1.27
Message-ID: <E1AVLgJ-000269-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv7964/spambayes

Modified Files:
	ImapUI.py 
Log Message:
Fix bug found by Skip/PyChecker.

(When logging in was done by the UI (to show available folders) we assigned the imap_session object to the wrong name).

Index: ImapUI.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/ImapUI.py,v
retrieving revision 1.26
retrieving revision 1.27
diff -C2 -d -r1.26 -r1.27
*** ImapUI.py	14 Dec 2003 01:36:49 -0000	1.26
--- ImapUI.py	14 Dec 2003 02:08:25 -0000	1.27
***************
*** 200,204 ****
                  else:
                      port = 143
!             imap = self.imap_session_class(server, port)
          if self.imap is None:
              content = self._buildBox("Error", None,
--- 200,204 ----
                  else:
                      port = 143
!             self.imap = self.imap_session_class(server, port)
          if self.imap is None:
              content = self._buildBox("Error", None,



From mhammond at users.sourceforge.net  Sat Dec 13 20:34:45 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Sun Dec 14 03:36:24 2003
Subject: [Spambayes-checkins] spambayes/spambayes Options.py,1.85,1.86
Message-ID: <E1AVL9g-00048e-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv15771

Modified Files:
	Options.py 
Log Message:
Move the option loading code to a function, then call this function as
the module loads.  This will allows us to re-read the options file
by calling this function, rather than reloading the entire module.


Index: Options.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/Options.py,v
retrieving revision 1.85
retrieving revision 1.86
diff -C2 -d -r1.85 -r1.86
*** Options.py	9 Dec 2003 16:31:39 -0000	1.85
--- Options.py	14 Dec 2003 01:34:42 -0000	1.86
***************
*** 986,1061 ****
  optionsPathname = None
  
! options = OptionsClass()
! options.load_defaults(defaults)
  
! alternate = None
! if hasattr(os, 'getenv'):
!     alternate = os.getenv('BAYESCUSTOMIZE')
! if alternate:
!     filenames = alternate.split(os.pathsep)
!     options.merge_files(filenames)
!     optionsPathname = os.path.abspath(filenames[-1])
! else:
!     alts = []
!     for path in ['bayescustomize.ini', '~/.spambayesrc']:
!         epath = os.path.expanduser(path)
!         if os.path.exists(epath):
!             alts.append(epath)
!     if alts:
!         options.merge_files(alts)
!         optionsPathname = os.path.abspath(alts[-1])
  
! if not optionsPathname:
!     optionsPathname = os.path.abspath('bayescustomize.ini')
!     if sys.platform.startswith("win") and not os.path.isfile(optionsPathname):
!         # If we are on Windows and still don't have an INI, default to the
!         # 'per-user' directory.
!         try:
!             from win32com.shell import shell, shellcon
!             windowsUserDirectory = os.path.join(
!                     shell.SHGetFolderPath(0,shellcon.CSIDL_APPDATA,0,0),
!                     "SpamBayes", "Proxy")
              try:
!                 if not os.path.isdir(windowsUserDirectory):
!                     os.makedirs(windowsUserDirectory)
!                 optionsPathname = os.path.join(windowsUserDirectory,
!                                                'bayescustomize.ini')
!                 # Not everyone is unicode aware - keep it a string.
!                 optionsPathname = optionsPathname.encode("mbcs")
!                 # If the file exists, then load it.
!                 if os.path.exists(optionsPathname):
!                     options.merge_file(optionsPathname)
!                 else:
!                     # If the file doesn't exist, then let's get the user to
!                     # store their databases and caches here as well, by
!                     # default, and save the file.
!                     db_name = os.path.join(windowsUserDirectory,
!                                            "statistics_database.db")
!                     mi_db = os.path.join(windowsUserDirectory,
!                                            "message_info_database.db")
!                     h_cache = os.path.join(windowsUserDirectory,
!                                            "ham_cache").encode("mbcs")
!                     u_cache = os.path.join(windowsUserDirectory,
!                                            "unknown_cache").encode("mbcs")
!                     s_cache = os.path.join(windowsUserDirectory,
!                                            "spam_cache").encode("mbcs")
!                     options["Storage", "spam_cache"] = s_cache
!                     options["Storage", "ham_cache"] = h_cache
!                     options["Storage", "unknown_cache"] = u_cache
!                     options["Storage", "persistent_storage_file"] = \
!                                        db_name.encode("mbcs")
!                     options["Storage", "messageinfo_storage_file"] = \
!                                        mi_db.encode("mbcs")
!                     options.update_file(optionsPathname)
!             except os.error:
!                 # unable to make the directory - stick to default.
!                 pass
!         except ImportError:
!             # We are on Windows, with no BAYESCUSTOMIZE set, no ini file
!             # in the current directory, and no win32 extensions installed
!             # to locate the "user" directory - seeing things are so lamely
!             # setup, it is worth printing a warning
!             print "NOTE: We can not locate an INI file for SpamBayes, and the"
!             print "Python for Windows extensions are not installed, meaning we"
!             print "can't locate your 'user' directory.  An empty configuration"
!             print "file at '%s' will be used." % optionsPathname.encode('mbcs')
--- 986,1074 ----
  optionsPathname = None
  
! # The global options object - created by load_options
! options = None
  
! def load_options():
!     global optionsPathname, options
!     options = OptionsClass()
!     options.load_defaults(defaults)
  
!     alternate = None
!     if hasattr(os, 'getenv'):
!         alternate = os.getenv('BAYESCUSTOMIZE')
!     if alternate:
!         filenames = alternate.split(os.pathsep)
!         options.merge_files(filenames)
!         optionsPathname = os.path.abspath(filenames[-1])
!     else:
!         alts = []
!         for path in ['bayescustomize.ini', '~/.spambayesrc']:
!             epath = os.path.expanduser(path)
!             if os.path.exists(epath):
!                 alts.append(epath)
!         if alts:
!             options.merge_files(alts)
!             optionsPathname = os.path.abspath(alts[-1])
!     
!     if not optionsPathname:
!         optionsPathname = os.path.abspath('bayescustomize.ini')
!         if sys.platform.startswith("win") and \
!            not os.path.isfile(optionsPathname):
!             # If we are on Windows and still don't have an INI, default to the
!             # 'per-user' directory.
              try:
!                 from win32com.shell import shell, shellcon
!                 windowsUserDirectory = os.path.join(
!                         shell.SHGetFolderPath(0,shellcon.CSIDL_APPDATA,0,0),
!                         "SpamBayes", "Proxy")
!                 try:
!                     if not os.path.isdir(windowsUserDirectory):
!                         os.makedirs(windowsUserDirectory)
!                     optionsPathname = os.path.join(windowsUserDirectory,
!                                                    'bayescustomize.ini')
!                     # Not everyone is unicode aware - keep it a string.
!                     optionsPathname = optionsPathname.encode("mbcs")
!                     # If the file exists, then load it.
!                     if os.path.exists(optionsPathname):
!                         options.merge_file(optionsPathname)
!                     else:
!                         # If the file doesn't exist, then let's get the user to
!                         # store their databases and caches here as well, by
!                         # default, and save the file.
!                         db_name = os.path.join(windowsUserDirectory,
!                                                "statistics_database.db")
!                         mi_db = os.path.join(windowsUserDirectory,
!                                                "message_info_database.db")
!                         h_cache = os.path.join(windowsUserDirectory,
!                                                "ham_cache").encode("mbcs")
!                         u_cache = os.path.join(windowsUserDirectory,
!                                                "unknown_cache").encode("mbcs")
!                         s_cache = os.path.join(windowsUserDirectory,
!                                                "spam_cache").encode("mbcs")
!                         options["Storage", "spam_cache"] = s_cache
!                         options["Storage", "ham_cache"] = h_cache
!                         options["Storage", "unknown_cache"] = u_cache
!                         options["Storage", "persistent_storage_file"] = \
!                                            db_name.encode("mbcs")
!                         options["Storage", "messageinfo_storage_file"] = \
!                                            mi_db.encode("mbcs")
!                         options.update_file(optionsPathname)
!                 except os.error:
!                     # unable to make the directory - stick to default.
!                     pass
!             except ImportError:
!                 # We are on Windows, with no BAYESCUSTOMIZE set, no ini file
!                 # in the current directory, and no win32 extensions installed
!                 # to locate the "user" directory - seeing things are so lamely
!                 # setup, it is worth printing a warning
!                 print "NOTE: We can not locate an INI file for SpamBayes, and the"
!                 print "Python for Windows extensions are not installed, meaning we"
!                 print "can't locate your 'user' directory.  An empty configuration"
!                 print "file at '%s' will be used." % optionsPathname.encode('mbcs')
! 
! # Ideally, we should not create the objects at import time - but we have
! # done it this way forever!
! # We avoid having the options loading code at the module level, as then
! # the only way to re-read is to reload this module, and as at 2.3, that
! # doesn't work in a .zip file.
! load_options()



From mhammond at users.sourceforge.net  Sat Dec 13 20:30:51 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Sun Dec 14 03:36:26 2003
Subject: [Spambayes-checkins] spambayes/scripts sb_server.py,1.12,1.13
Message-ID: <E1AVL5v-000423-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/scripts
In directory sc8-pr-cvs1:/tmp/cvs-serv15075/scripts

Modified Files:
	sb_server.py 
Log Message:
Fix [ 859215 ] "Restore Defaults" causes assertion error at exit.
There was some confusion between the global 'state' and local state passed
to the global sb_server functions.  This meant that even when a state was
recreated during run(), the old state was closed as run() finished.
The global sb_server methods now all use the global sb_server state - if
you need finer state control, use your own state but don't call the global
sb_server functions.


Index: sb_server.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/scripts/sb_server.py,v
retrieving revision 1.12
retrieving revision 1.13
diff -C2 -d -r1.12 -r1.13
*** sb_server.py	2 Dec 2003 06:05:25 -0000	1.12
--- sb_server.py	14 Dec 2003 01:30:49 -0000	1.13
***************
*** 674,677 ****
--- 674,678 ----
      def prepare(self):
          # If we can, prevent multiple servers from running at the same time.
+         assert self.platform_mutex is None, "Should not already have the mutex"
          self.platform_mutex = open_platform_mutex()
      
***************
*** 832,838 ****
      state = State()
  
!     prepare(state)
      _createProxies(state.servers, state.proxyPorts)
-     
      return state
  
--- 833,838 ----
      state = State()
  
!     prepare()
      _createProxies(state.servers, state.proxyPorts)
      return state
  
***************
*** 846,850 ****
      Dibbler.run(launchBrowser=launchUI)
  
! def prepare(state):
      state.prepare()
      # Launch any SMTP proxies.  Note that if the user hasn't specified any
--- 846,850 ----
      Dibbler.run(launchBrowser=launchUI)
  
! def prepare():
      state.prepare()
      # Launch any SMTP proxies.  Note that if the user hasn't specified any
***************
*** 859,863 ****
      state.buildServerStrings()
  
! def start(state):
      # kick everything off
      assert state.prepared, "starting before preparing state"
--- 859,863 ----
      state.buildServerStrings()
  
! def start():
      # kick everything off
      assert state.prepared, "starting before preparing state"
***************
*** 867,871 ****
          state.close()
  
! def stop(state):
      # Shutdown as though through the web UI.  This will save the DB, allow
      # any open proxy connections to complete, etc.
--- 867,871 ----
          state.close()
  
! def stop():
      # Shutdown as though through the web UI.  This will save the DB, allow
      # any open proxy connections to complete, etc.
***************
*** 925,929 ****
  
          try:
!             prepare(state=state)
          except AlreadyRunningException:
              print  >>sys.stderr, \
--- 925,929 ----
  
          try:
!             prepare()
          except AlreadyRunningException:
              print  >>sys.stderr, \
***************
*** 931,935 ****
              print  >>sys.stderr, "Please stop the existing proxy and try again"
              return
!         start(state=state)
  
      else:
--- 931,935 ----
              print  >>sys.stderr, "Please stop the existing proxy and try again"
              return
!         start()
  
      else:



From mhammond at users.sourceforge.net  Sat Dec 13 20:12:10 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Sun Dec 14 03:46:52 2003
Subject: [Spambayes-checkins] 
	spambayes/spambayes/test test_programs.py, 1.1, 1.2
Message-ID: <E1AVKnq-0000Ms-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes/test
In directory sc8-pr-cvs1:/tmp/cvs-serv32042

Modified Files:
	test_programs.py 
Log Message:
* When "calling" URLs, check the output for tracebacks
* Check the exit code of processes we spawn.
* Add test for "[ 859215 ] "Restore Defaults" causes assertion error at 
  exit" by explicitly "calling" this reload URL from the test script.


Index: test_programs.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/test/test_programs.py,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** test_programs.py	3 Dec 2003 00:34:17 -0000	1.1
--- test_programs.py	14 Dec 2003 01:12:08 -0000	1.2
***************
*** 4,7 ****
--- 4,8 ----
  import unittest
  import time
+ from urllib import urlopen, urlencode
  
  import sb_test_support
***************
*** 21,24 ****
--- 22,33 ----
  verbose = 0
  
+ def call_web_function(url, **kw):
+     got = urlopen(url, urlencode(kw)).read()
+     # Very simple - just look for tracebacks
+     if got.find("Traceback (most recent call last)")>=0:
+         print "FAILED calling URL", url
+         print got
+         raise AssertionError, "Opening URL %s appeared to fail" % (url,)
+ 
  class Spawner:
      def __init__(self, test_case, spawn_args):
***************
*** 101,111 ****
          # Shutdown as though through the web UI.  This will save the DB, allow
          # any open proxy connections to complete, etc.
!         from urllib import urlopen, urlencode
!         urlopen('http://localhost:%d/save' % self.shutdown_port,
!                 urlencode({'how': 'Save & shutdown'})).read()
          # wait for it to stop - 5 secs, 0.25 per check
          for i in range(20):
              time.sleep(0.25)
              if not self.is_running() and not is_any_sb_server_running():
                  return
          # gave up waiting.
--- 110,123 ----
          # Shutdown as though through the web UI.  This will save the DB, allow
          # any open proxy connections to complete, etc.
!         call_web_function('http://localhost:%d/save' % self.shutdown_port,
!                           how='Save & shutdown')
          # wait for it to stop - 5 secs, 0.25 per check
          for i in range(20):
              time.sleep(0.25)
              if not self.is_running() and not is_any_sb_server_running():
+                 # stopped - check the exit code
+                 temp_pid, rc = os.waitpid(self.pid, 0)
+                 if rc:
+                     self.test_case.fail("sb_server returned exit code %s" % rc)
                  return
          # gave up waiting.
***************
*** 161,164 ****
--- 173,186 ----
          self._stop_spawner(s)
  
+     def test_sb_server_restore(self):
+         # Make sure we can do a restore defaults and shutdown without incident.
+         from spambayes.Options import options
+         port = options["html_ui", "port"]
+         s = Spawner_sb_server(self, [], shutdown_port=port)
+         self._start_spawner(s)
+         # do the reload
+         call_web_function('http://localhost:%d/restoredefaults' % port, how='')
+         self._stop_spawner(s)
+ 
  if sys.platform.startswith("win"):
      import win32service # You need win32all to run the tests!
***************
*** 207,214 ****
              # Should be using the default port from the options file.
              from spambayes.Options import options
-             from urllib import urlopen, urlencode
              port = options["html_ui", "port"]
!             urlopen('http://localhost:%d/save' % port,
!                     urlencode({'how': 'Save & shutdown'})).read()
              # wait for it to stop - 5 secs, 0.25 per check
              for i in range(10):
--- 229,235 ----
              # Should be using the default port from the options file.
              from spambayes.Options import options
              port = options["html_ui", "port"]
!             call_web_function ('http://localhost:%d/save' % port,
!                                how='Save & shutdown')
              # wait for it to stop - 5 secs, 0.25 per check
              for i in range(10):



From mhammond at users.sourceforge.net  Sat Dec 13 20:30:51 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Sun Dec 14 03:47:16 2003
Subject: [Spambayes-checkins] spambayes/windows pop3proxy_service.py, 1.15,
	1.16 pop3proxy_tray.py, 1.16, 1.17
Message-ID: <E1AVL5v-000426-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/windows
In directory sc8-pr-cvs1:/tmp/cvs-serv15075/windows

Modified Files:
	pop3proxy_service.py pop3proxy_tray.py 
Log Message:
Fix [ 859215 ] "Restore Defaults" causes assertion error at exit.
There was some confusion between the global 'state' and local state passed
to the global sb_server functions.  This meant that even when a state was
recreated during run(), the old state was closed as run() finished.
The global sb_server methods now all use the global sb_server state - if
you need finer state control, use your own state but don't call the global
sb_server functions.


Index: pop3proxy_service.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/windows/pop3proxy_service.py,v
retrieving revision 1.15
retrieving revision 1.16
diff -C2 -d -r1.15 -r1.16
*** pop3proxy_service.py	29 Sep 2003 02:18:27 -0000	1.15
--- pop3proxy_service.py	14 Dec 2003 01:30:49 -0000	1.16
***************
*** 102,106 ****
          self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
          self.event_stopping.set()
!         sb_server.stop(sb_server.state)        
  
      def SvcDoRun(self):
--- 102,106 ----
          self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
          self.event_stopping.set()
!         sb_server.stop()        
  
      def SvcDoRun(self):
***************
*** 108,112 ****
          # Setup our state etc
          try:
!             sb_server.prepare(state=sb_server.state)
          except sb_server.AlreadyRunningException:
              msg = "The SpamBayes proxy service could not be started, as "\
--- 108,112 ----
          # Setup our state etc
          try:
!             sb_server.prepare()
          except sb_server.AlreadyRunningException:
              msg = "The SpamBayes proxy service could not be started, as "\
***************
*** 169,173 ****
          try:
              try:
!                 sb_server.start(sb_server.state)
              except SystemExit:
                  # user requested shutdown
--- 169,173 ----
          try:
              try:
!                 sb_server.start()
              except SystemExit:
                  # user requested shutdown

Index: pop3proxy_tray.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/windows/pop3proxy_tray.py,v
retrieving revision 1.16
retrieving revision 1.17
diff -C2 -d -r1.16 -r1.17
*** pop3proxy_tray.py	4 Dec 2003 02:57:43 -0000	1.16
--- pop3proxy_tray.py	14 Dec 2003 01:30:49 -0000	1.17
***************
*** 392,396 ****
          if self.started and not self.use_service:
              try:
!                 sb_server.stop(sb_server.state)
              except:
                  print "Error stopping proxy at shutdown"
--- 392,396 ----
          if self.started and not self.use_service:
              try:
!                 sb_server.stop()
              except:
                  print "Error stopping proxy at shutdown"
***************
*** 405,409 ****
          self.started = True
          try:
!             sb_server.start(sb_server.state)
          finally:
              self.started = False
--- 405,409 ----
          self.started = True
          try:
!             sb_server.start()
          finally:
              self.started = False
***************
*** 429,433 ****
              if not self.have_prepared_state:
                  try:
!                     sb_server.prepare(state=sb_server.state)
                      self.have_prepared_state = True
                  except sb_server.AlreadyRunningException:
--- 429,433 ----
              if not self.have_prepared_state:
                  try:
!                     sb_server.prepare()
                      self.have_prepared_state = True
                  except sb_server.AlreadyRunningException:
***************
*** 461,465 ****
              if not use_service:
                  if verbose: print "Stopping local server"
!                 sb_server.stop(sb_server.state)
          except:
              print "There was an error stopping the server"
--- 461,465 ----
              if not use_service:
                  if verbose: print "Stopping local server"
!                 sb_server.stop()
          except:
              print "There was an error stopping the server"



From mhammond at users.sourceforge.net  Sat Dec 13 20:36:51 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Sun Dec 14 03:47:22 2003
Subject: [Spambayes-checkins] spambayes/spambayes ImapUI.py, 1.25,
	1.26 ProxyUI.py, 1.30, 1.31 ServerUI.py, 1.2, 1.3
Message-ID: <E1AVLBj-0004DT-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv16201/spambayes

Modified Files:
	ImapUI.py ProxyUI.py ServerUI.py 
Log Message:
Fix [ 856628 ] reload(Options) fails in windows binary
Instead of reload(Options), we call the new load_options function.


Index: ImapUI.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/ImapUI.py,v
retrieving revision 1.25
retrieving revision 1.26
diff -C2 -d -r1.25 -r1.26
*** ImapUI.py	11 Dec 2003 18:42:41 -0000	1.25
--- ImapUI.py	14 Dec 2003 01:36:49 -0000	1.26
***************
*** 155,162 ****
          """Called by the config page when the user saves some new options, or
          restores the defaults."""
!         # Reload the options.
          self.classifier.store()
          import Options
!         reload(Options)
          global options
          from Options import options
--- 155,162 ----
          """Called by the config page when the user saves some new options, or
          restores the defaults."""
!         # Re-read the options.
          self.classifier.store()
          import Options
!         Options.load_options()
          global options
          from Options import options

Index: ProxyUI.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/ProxyUI.py,v
retrieving revision 1.30
retrieving revision 1.31
diff -C2 -d -r1.30 -r1.31
*** ProxyUI.py	11 Dec 2003 18:43:17 -0000	1.30
--- ProxyUI.py	14 Dec 2003 01:36:49 -0000	1.31
***************
*** 685,692 ****
          """Called by the config page when the user saves some new options, or
          restores the defaults."""
!         # Reload the options.
          global state
          import Options
!         reload(Options)
          global options
          from Options import options
--- 685,692 ----
          """Called by the config page when the user saves some new options, or
          restores the defaults."""
!         # Re-read the options.
          global state
          import Options
!         Options.load_options()
          global options
          from Options import options

Index: ServerUI.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/ServerUI.py,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** ServerUI.py	24 Sep 2003 05:28:53 -0000	1.2
--- ServerUI.py	14 Dec 2003 01:36:49 -0000	1.3
***************
*** 94,98 ****
          self.state.bayes.store()
          import Options
!         reload(Options)
          global options
          from Options import options
--- 94,98 ----
          self.state.bayes.store()
          import Options
!         Options.load_options()
          global options
          from Options import options



From tim_one at users.sourceforge.net  Sat Dec 13 23:12:34 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Sun Dec 14 03:50:34 2003
Subject: [Spambayes-checkins] spambayes/spambayes Options.py, 1.86,
	1.87 classifier.py, 1.10, 1.11
Message-ID: <E1AVNcQ-00062A-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv20174/spambayes

Modified Files:
	Options.py classifier.py 
Log Message:
Removed support code for the defunct
experimental_ham_spam_imbalance_adjustment option, and fiddled some docs
accordingly.  Options.py still knows about it, and various UI components
building on Options.py, to avoid breaking anything that's using it.
Unsure to get rid of it completely.


Index: Options.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/Options.py,v
retrieving revision 1.86
retrieving revision 1.87
diff -C2 -d -r1.86 -r1.87
*** Options.py	14 Dec 2003 01:34:42 -0000	1.86
--- Options.py	14 Dec 2003 04:12:32 -0000	1.87
***************
*** 409,413 ****
      # than min(# ham trained on, # spam trained on) justifies.  I *expect*
      # this option will go away (and become the default), but people *with*
!     # strong imbalance need to test it first.
  
      ("experimental_ham_spam_imbalance_adjustment", "Compensate for unequal numbers of spam and ham", False,
--- 409,416 ----
      # than min(# ham trained on, # spam trained on) justifies.  I *expect*
      # this option will go away (and become the default), but people *with*
!     # strong imbalance need to test it first.\
!     # LATER:  this option sucked, creating more problems than it solved.
!     # XXX The code in classifier.py is gone now.  How can we get rid of
!     # XXX the option gracefully?
  
      ("experimental_ham_spam_imbalance_adjustment", "Compensate for unequal numbers of spam and ham", False,

Index: classifier.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/classifier.py,v
retrieving revision 1.10
retrieving revision 1.11
diff -C2 -d -r1.10 -r1.11
*** classifier.py	15 Sep 2003 01:08:11 -0000	1.10
--- classifier.py	14 Dec 2003 04:12:32 -0000	1.11
***************
*** 248,257 ****
          prob = spamratio / (hamratio + spamratio)
  
-         if options["Classifier", "experimental_ham_spam_imbalance_adjustment"]:
-             spam2ham = min(nspam / nham, 1.0)
-             ham2spam = min(nham / nspam, 1.0)
-         else:
-             spam2ham = ham2spam = 1.0
- 
          S = options["Classifier", "unknown_word_strength"]
          StimesX = S * options["Classifier", "unknown_word_prob"]
--- 248,251 ----
***************
*** 274,296 ****
          # less so the larger n is, or the smaller s is.
  
!         # Experimental:
!         # Picking a good value for n is interesting:  how much empirical
!         # evidence do we really have?  If nham == nspam,
!         # hamcount + spamcount makes a lot of sense, and the code here
!         # does that by default.
!         # But if, e.g., nham is much larger than nspam, p(w) can get a
!         # lot closer to 0.0 than it can get to 1.0.  That in turn makes
!         # strong ham words (high hamcount) much stronger than strong
!         # spam words (high spamcount), and that makes the accidental
!         # appearance of a strong ham word in spam much more damaging than
!         # the accidental appearance of a strong spam word in ham.
!         # So we don't give hamcount full credit when nham > nspam (or
!         # spamcount when nspam > nham):  instead we knock hamcount down
!         # to what it would have been had nham been equal to nspam.  IOW,
!         # we multiply hamcount by nspam/nham when nspam < nham; or, IOOW,
!         # we don't "believe" any count to an extent more than
!         # min(nspam, nham) justifies.
! 
!         n = hamcount * spam2ham  +  spamcount * ham2spam
          prob = (StimesX + n * prob) / (S + n)
  
--- 268,272 ----
          # less so the larger n is, or the smaller s is.
  
!         n = hamcount + spamcount
          prob = (StimesX + n * prob) / (S + n)
  
***************
*** 380,384 ****
          this point.  Introduced to fix bug #797890."""
          pass
!     
      def _getclues(self, wordstream):
          mindist = options["Classifier", "minimum_prob_strength"]
--- 356,360 ----
          this point.  Introduced to fix bug #797890."""
          pass
! 
      def _getclues(self, wordstream):
          mindist = options["Classifier", "minimum_prob_strength"]



From tim_one at users.sourceforge.net  Sat Dec 13 23:12:34 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Sun Dec 14 03:50:35 2003
Subject: [Spambayes-checkins] website background.ht,1.17,1.18
Message-ID: <E1AVNcQ-00062H-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/website
In directory sc8-pr-cvs1:/tmp/cvs-serv20174/website

Modified Files:
	background.ht 
Log Message:
Removed support code for the defunct
experimental_ham_spam_imbalance_adjustment option, and fiddled some docs
accordingly.  Options.py still knows about it, and various UI components
building on Options.py, to avoid breaking anything that's using it.
Unsure to get rid of it completely.


Index: background.ht
===================================================================
RCS file: /cvsroot/spambayes/website/background.ht,v
retrieving revision 1.17
retrieving revision 1.18
diff -C2 -d -r1.17 -r1.18
*** background.ht	11 Sep 2003 07:04:22 -0000	1.17
--- background.ht	14 Dec 2003 04:12:32 -0000	1.18
***************
*** 190,200 ****
  <li>the system generates remarkably good results on very few trained messages,
  although obviously, the more the better.
! <li>training on an extremely large number of messages actually starts to produce
! a slightly poorer result (once you're training on thousands and thousands of
  messages).
! <li>severe imbalance between the number of ham and spam being trained causes
! some problems - there's a flag 'experimental_ham_spam_imbalance_adjustment'
! that you can set in your bayescustomize.ini file to attempt to compensate for
! this.
  <li>It's <b>really</b> important that you're careful about selecting your
  training sets to be from the same sources, where-ever possible. Several 
--- 190,199 ----
  <li>the system generates remarkably good results on very few trained messages,
  although obviously, the more the better.
! <li>training on an extremely large number of messages may start to produce
! slightly poorer results (once you're training on thousands and thousands of
  messages).
! <li>imbalance between the number of ham and spam trained can cause
! problems, and worse problems the greater the imbalance; the system works
! best when trained on an approximately equal number of ham and spam.
  <li>It's <b>really</b> important that you're careful about selecting your
  training sets to be from the same sources, where-ever possible. Several 



From mhammond at users.sourceforge.net  Sun Dec 14 06:23:05 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Sun Dec 14 06:23:18 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000
	default_bayes_customize.ini, 1.9, 1.10 manager.py, 1.91, 1.92
Message-ID: <E1AVUL3-0001oV-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv6746

Modified Files:
	default_bayes_customize.ini manager.py 
Log Message:
Fix [ 833439 ] default_bayes_customize.ini is confusing.
default_bayes_customize.ini is no longer copied to the data directory.
Options are read from this filename in both the app_dir and data_dir.
Thus, new users will *not* have default_bayes_customize.ini in their
data_dir (it will remain for existing users, but causes no harm)


Index: default_bayes_customize.ini
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/default_bayes_customize.ini,v
retrieving revision 1.9
retrieving revision 1.10
diff -C2 -d -r1.9 -r1.10
*** default_bayes_customize.ini	18 Sep 2003 04:30:59 -0000	1.9
--- default_bayes_customize.ini	14 Dec 2003 11:23:03 -0000	1.10
***************
*** 1,5 ****
! # This is the default INI file for the Outlook addin Bayes database
! # This file must exist, or the addin considers itself confused.
! # As we decide default options, we can add them!
  
  [Tokenizer]
--- 1,22 ----
! # This is the INI file for the *Bayes Engine* as used by the Outlook addin.  
! # It is NOT where configuration information is stored for the addin - see 
! # "[Profile Name].ini" for these settings.
! 
! # Only options which Outlook requires different from the SpamBayes 
! # defaults are listed here.  Thus, there will be very few in the installed 
! # version, but you may list any configuration option valid for the SpamBayes 
! # engine in this file.  Again, these are different from the Outlook addin 
! # configuration options.
! 
! # This file exists in the SpamBayes program directory, and may optionally
! # exist in the SpamBayes data directory.  If you wish to make changes to
! # the default options, setting the option in a file in your data directory
! # will persist even when SpamBayes is upgraded.  If you change the version in
! # the SpamBayes program directory, it will be upgraded along with SpamBayes, 
! # so your changes will be lost.  Options in the data directory file have 
! # precedence over the app directory file.
! # Note that versions 0.81 and earlier of the plugin copied this file
! # to your data directory automatically.  If a file with this name exists in 
! # your data directory and you aren't sure you want it, you may delete it.
  
  [Tokenizer]
***************
*** 15,22 ****
  # It's helpful for Tim <wink>.
  record_header_absence: True
- 
- [Classifier]
- # This will probably go away if testing confirms it's a Good Thing.
- # Testing seems to be indicating that this is a Bad Thing, so we
- # leave it disabled.
- #experimental_ham_spam_imbalance_adjustment: True
--- 32,33 ----

Index: manager.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/manager.py,v
retrieving revision 1.91
retrieving revision 1.92
diff -C2 -d -r1.91 -r1.92
*** manager.py	10 Dec 2003 05:30:50 -0000	1.91
--- manager.py	14 Dec 2003 11:23:03 -0000	1.92
***************
*** 114,132 ****
          sys.path.insert(0, parent)
  
! def import_core_spambayes_stuff(ini_filename):
      if "spambayes.Options" in sys.modules:
!         # Manager probably being re-initialized (via the Outlook 'addin' GUI
!         # Check that nothing has changed underneath us.
!         if __debug__:
!             import spambayes.Options
!             assert spambayes.Options.optionsPathname == \
!                    ini_filename.encode(filesystem_encoding), \
!                    "'spambayes.Options' was imported too early, with the " \
!                    "incorrect directory %r" \
!                    % (spambayes.Options.optionsPathname,)
          return
!     global bayes_classifier, bayes_tokenize, bayes_storage
!     # ini_filename is Unicode, but environ not unicode aware
!     os.environ["BAYESCUSTOMIZE"] = ini_filename.encode(filesystem_encoding)
      from spambayes import classifier
      from spambayes.tokenizer import tokenize
--- 114,141 ----
          sys.path.insert(0, parent)
  
! def import_core_spambayes_stuff(ini_filenames):
!     global bayes_classifier, bayes_tokenize, bayes_storage
      if "spambayes.Options" in sys.modules:
!         # The only thing we are worried about here is spambayes.Options
!         # being imported before we have determined the INI files we need to
!         # use.
!         # The only way this can happen otherwise is when the addin is
!         # de-selected then re-selected via the Outlook GUI - and when
!         # running from source-code, it never appears in this list.
!         # So this should never happen from source-code, and if it does, then
!         # the developer has recently changed something that causes the early
!         # import
!         assert hasattr(sys, "frozen")
!         # And we don't care (we could try and reload the engine options,
!         # but these are very unlikely to have changed)
          return
!     # ini_filenames may contain Unicode, but environ not unicode aware.
!     # Convert if necessary.
!     use_names = []
!     for name in ini_filenames:
!         if isinstance(name, unicode):
!             name = name.encode(filesystem_encoding)
!         use_names.append(name)
!     os.environ["BAYESCUSTOMIZE"] = os.pathsep.join(use_names)
      from spambayes import classifier
      from spambayes.tokenizer import tokenize
***************
*** 375,381 ****
          self.LoadConfig()
  
!         bayes_options_filename = os.path.join(self.data_directory,
!                                               "default_bayes_customize.ini")
!         import_core_spambayes_stuff(bayes_options_filename)
  
          bayes_base = os.path.join(self.data_directory, "default_bayes_database")
--- 384,399 ----
          self.LoadConfig()
  
!         # Load the options for the classifier.  We support
!         # default_bayes_customize.ini in the app directory and user data
!         # directory (version 0.8 and earlier, we copied the app one to the
!         # user dir - that was a mistake - but supporting a version in that
!         # directory wasn't.
!         bayes_option_filenames = []
!         # data dir last so options there win.
!         for look_dir in [self.application_directory, self.data_directory]:
!             look_file = os.path.join(look_dir, "default_bayes_customize.ini")
!             if os.path.isfile(look_file):
!                 bayes_option_filenames.append(look_file)
!         import_core_spambayes_stuff(bayes_option_filenames)
  
          bayes_base = os.path.join(self.data_directory, "default_bayes_database")
***************
*** 491,497 ****
          # Try and locate our files in the old location, and move
          # them to the new one.
!         # Also used first time SpamBayes is run - this will cause
!         # the ini file to be *copied* to the correct directory
!         self._MigrateFile("default_bayes_customize.ini", False)
          self._MigrateFile("default_bayes_database.pck")
          self._MigrateFile("default_bayes_database.db")
--- 509,515 ----
          # Try and locate our files in the old location, and move
          # them to the new one.
!         # Note that this is migrating data for very old versions of the
!         # plugin (before the first decent binary!).  The next time it is
!         # touched it can die :) 
          self._MigrateFile("default_bayes_database.pck")
          self._MigrateFile("default_bayes_database.db")
***************
*** 503,507 ****
      # By default (do_move not specified), the source file is deleted.
      # Pass do_move=False to leave the original file.
!     def _MigrateFile(self, filename, do_move = True):
          src = os.path.join(self.application_directory, filename)
          dest = os.path.join(self.data_directory, filename)
--- 521,525 ----
      # By default (do_move not specified), the source file is deleted.
      # Pass do_move=False to leave the original file.
!     def _MigrateFile(self, filename):
          src = os.path.join(self.application_directory, filename)
          dest = os.path.join(self.data_directory, filename)
***************
*** 510,515 ****
              # Win95 and Win98 don't support MoveFileEx.
              shutil.copyfile(src, dest)
!             if do_move:
!                 os.remove(src)
  
      def FormatFolderNames(self, folder_ids, include_sub):
--- 528,532 ----
              # Win95 and Win98 don't support MoveFileEx.
              shutil.copyfile(src, dest)
!             os.remove(src)
  
      def FormatFolderNames(self, folder_ids, include_sub):



From tim_one at users.sourceforge.net  Sun Dec 14 19:16:22 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Sun Dec 14 19:16:26 2003
Subject: [Spambayes-checkins] spambayes/spambayes ImapUI.py, 1.27,
	1.28 Options.py, 1.87, 1.88 ProxyUI.py, 1.31, 1.32 ServerUI.py,
	1.3, 1.4
Message-ID: <E1AVgPO-0006cb-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv25360/spambayes

Modified Files:
	ImapUI.py Options.py ProxyUI.py ServerUI.py 
Log Message:
Finished deprecating the experimental_ham_spam_imbalance_adjustment option.


Index: ImapUI.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/ImapUI.py,v
retrieving revision 1.27
retrieving revision 1.28
diff -C2 -d -r1.27 -r1.28
*** ImapUI.py	14 Dec 2003 02:08:25 -0000	1.27
--- ImapUI.py	15 Dec 2003 00:16:19 -0000	1.28
***************
*** 80,84 ****
  adv_map = (
      ('Statistics Options',  None),
-     ('Classifier',          'experimental_ham_spam_imbalance_adjustment'),
      ('Classifier',          'max_discriminators'),
      ('Classifier',          'minimum_prob_strength'),
--- 80,83 ----

Index: Options.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/Options.py,v
retrieving revision 1.87
retrieving revision 1.88
diff -C2 -d -r1.87 -r1.88
*** Options.py	14 Dec 2003 04:12:32 -0000	1.87
--- Options.py	15 Dec 2003 00:16:19 -0000	1.88
***************
*** 364,368 ****
       assigning certainty (0 or 1) to a word that has appeared in only ham
       or only spam.  This is a disaster.
!      
       As unknown_word_strength tends toward infintity, all probabilities
       tend toward unknown_word_prob.  All reports were that a value near 0.4
--- 364,368 ----
       assigning certainty (0 or 1) to a word that has appeared in only ham
       or only spam.  This is a disaster.
! 
       As unknown_word_strength tends toward infintity, all probabilities
       tend toward unknown_word_prob.  All reports were that a value near 0.4
***************
*** 411,418 ****
      # strong imbalance need to test it first.\
      # LATER:  this option sucked, creating more problems than it solved.
!     # XXX The code in classifier.py is gone now.  How can we get rid of
!     # XXX the option gracefully?
  
!     ("experimental_ham_spam_imbalance_adjustment", "Compensate for unequal numbers of spam and ham", False,
       """If your training database has significantly more ham than
       spam, or vice versa, you may start seeing an increase in incorrect
--- 411,420 ----
      # strong imbalance need to test it first.\
      # LATER:  this option sucked, creating more problems than it solved.
!     # It's deprecated, and the support code has gone away.
!     # XXX The "x-" prefix can't be "X-" instead, else it's considered
!     # XXX an invalid option instead of a deprecated one.  That behavior
!     # XXX doesn't match the OptionsClass comments.
  
!     ("x-experimental_ham_spam_imbalance_adjustment", "Compensate for unequal numbers of spam and ham", False,
       """If your training database has significantly more ham than
       spam, or vice versa, you may start seeing an increase in incorrect
***************
*** 951,955 ****
       PORT, RESTORE),
    ),
!  
    "globals" : (
      ("verbose", "Verbose", False,
--- 953,957 ----
       PORT, RESTORE),
    ),
! 
    "globals" : (
      ("verbose", "Verbose", False,
***************
*** 1013,1017 ****
              options.merge_files(alts)
              optionsPathname = os.path.abspath(alts[-1])
!     
      if not optionsPathname:
          optionsPathname = os.path.abspath('bayescustomize.ini')
--- 1015,1019 ----
              options.merge_files(alts)
              optionsPathname = os.path.abspath(alts[-1])
! 
      if not optionsPathname:
          optionsPathname = os.path.abspath('bayescustomize.ini')

Index: ProxyUI.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/ProxyUI.py,v
retrieving revision 1.31
retrieving revision 1.32
diff -C2 -d -r1.31 -r1.32
*** ProxyUI.py	14 Dec 2003 01:36:49 -0000	1.31
--- ProxyUI.py	15 Dec 2003 00:16:19 -0000	1.32
***************
*** 105,109 ****
  adv_map = (
      ('Statistics Options',  None),
-     ('Classifier',          'experimental_ham_spam_imbalance_adjustment'),
      ('Classifier',          'max_discriminators'),
      ('Classifier',          'minimum_prob_strength'),
--- 105,108 ----

Index: ServerUI.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/ServerUI.py,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** ServerUI.py	14 Dec 2003 01:36:49 -0000	1.3
--- ServerUI.py	15 Dec 2003 00:16:20 -0000	1.4
***************
*** 57,61 ****
      ('Categorization',      'ham_cutoff'),
      ('Categorization',      'spam_cutoff'),
-     ('Classifier',          'experimental_ham_spam_imbalance_adjustment'),
  )
  
--- 57,60 ----



From anadelonbrin at users.sourceforge.net  Mon Dec 15 02:02:35 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Mon Dec 15 02:02:39 2003
Subject: [Spambayes-checkins] spambayes/windows README.txt,1.1,1.2
Message-ID: <E1AVmkV-0006BE-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/windows
In directory sc8-pr-cvs1:/tmp/cvs-serv23755/windows

Modified Files:
	README.txt 
Log Message:
Move a bit closer to current reality.

Index: README.txt
===================================================================
RCS file: /cvsroot/spambayes/spambayes/windows/README.txt,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** README.txt	12 Mar 2003 13:22:44 -0000	1.1
--- README.txt	15 Dec 2003 07:02:33 -0000	1.2
***************
*** 3,7 ****
  Currently this contains:
  * Windows Service version of the pop3 proxy
  
! Fairly short-term plans include:
! * Inno setup installer for pop3proxy and Outlook.
--- 3,14 ----
  Currently this contains:
  * Windows Service version of the pop3 proxy
+ * The beginnings of a script to automatically configure SpamBayes and
+   the user's mail client.
+ * A GUI 'tray' application to control SpamBayes (service or proxy).
+ * Inno setup installer for sb_server, Outlook and a few tools.
  
! If you have the latest version of py2exe, you can build the binary installer
! as follows:
! 
! 1. Run "python setup_all.py" in the py2exe directory in this directory.
! 2. Run the Inno Setup "spambayes.iss" script in this directory.



From anadelonbrin at users.sourceforge.net  Mon Dec 15 02:06:26 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Mon Dec 15 02:06:29 2003
Subject: [Spambayes-checkins] spambayes/spambayes Corpus.py,1.10,1.11
Message-ID: <E1AVmoE-0006Io-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv24226/spambayes-old/spambayes

Modified Files:
	Corpus.py 
Log Message:
Typo in comment.

Index: Corpus.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/Corpus.py,v
retrieving revision 1.10
retrieving revision 1.11
diff -C2 -d -r1.10 -r1.11
*** Corpus.py	11 Dec 2003 18:40:21 -0000	1.10
--- Corpus.py	15 Dec 2003 07:06:23 -0000	1.11
***************
*** 11,15 ****
      A corpus is defined as a set of messages that share some common
      characteristic relative to spamness.  Examples might be spam, ham,
!     unsure, or untrained, or "bayes rating between .4 and .6.  A
      corpus is a collection of messages.  Corpus is a dictionary that
      is keyed by the keys of the messages within it.  It is iterable,
--- 11,15 ----
      A corpus is defined as a set of messages that share some common
      characteristic relative to spamness.  Examples might be spam, ham,
!     unsure, or untrained, or "bayes rating between .4 and .6".  A
      corpus is a collection of messages.  Corpus is a dictionary that
      is keyed by the keys of the messages within it.  It is iterable,
***************
*** 71,75 ****
      o Suggestions?
  
!     '''
  
  # This module is part of the spambayes project, which is Copyright 2002-3
--- 71,75 ----
      o Suggestions?
  
! '''
  
  # This module is part of the spambayes project, which is Copyright 2002-3
***************
*** 151,155 ****
          # This method should probably not be overridden
          key = message.key()
! 
          if options["globals", "verbose"]:
              print 'placing %s in corpus cache' % (key)
--- 151,155 ----
          # This method should probably not be overridden
          key = message.key()
!         
          if options["globals", "verbose"]:
              print 'placing %s in corpus cache' % (key)



From anadelonbrin at users.sourceforge.net  Mon Dec 15 02:48:42 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Mon Dec 15 02:48:45 2003
Subject: [Spambayes-checkins] spambayes/spambayes OptionsClass.py,1.17,1.18
Message-ID: <E1AVnT8-0007rD-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv30198/spambayes

Modified Files:
	OptionsClass.py 
Log Message:
Don't override the built-in file.

It wasn't really clear if "X-" or "x-" was the experimental option prefix, so accept either.
(Personally I think either is as acceptable as picking one of them).

Index: OptionsClass.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/OptionsClass.py,v
retrieving revision 1.17
retrieving revision 1.18
diff -C2 -d -r1.17 -r1.18
*** OptionsClass.py	9 Dec 2003 16:31:39 -0000	1.17
--- OptionsClass.py	15 Dec 2003 07:48:40 -0000	1.18
***************
*** 508,513 ****
  
      def merge_files(self, file_list):
!         for file in file_list:
!             self.merge_file(file)
  
      def convert_and_set(self, section, option, value):
--- 508,513 ----
  
      def merge_files(self, file_list):
!         for f in file_list:
!             self.merge_file(f)
  
      def convert_and_set(self, section, option, value):
***************
*** 534,547 ****
                          # not an error if an X- option is missing
                      else:
!                         option = 'x-'+option
                          # going the other way, if the option has been
                          # deprecated, set its x-prefixed version and
                          # emit a warning
!                         if self._options.has_key((section, option)):
!                             self.convert_and_set(section, option, value)
!                             print >> sys.stderr, (
!                                 "warning: option %s in"
!                                 " section %s is deprecated" %
!                                 (opt, sect))
                          else:
                              print >> sys.stderr, (
--- 534,548 ----
                          # not an error if an X- option is missing
                      else:
!                         l_option = 'x-' + option
!                         u_option = 'X-' + option
                          # going the other way, if the option has been
                          # deprecated, set its x-prefixed version and
                          # emit a warning
!                         if self._options.has_key((section, l_option)):
!                             self.convert_and_set(section, l_option, value)
!                             self._report_deprecated_error(section, option)
!                         elif self._options.has_key((section, u_option)):
!                             self.convert_and_set(section, u_option, value)
!                             self._report_deprecated_error(section, option)
                          else:
                              print >> sys.stderr, (
***************
*** 629,632 ****
--- 630,638 ----
          else:
              self.set(sect, opt, val)
+ 
+     def _report_deprecated_error(self, sect, opt):
+         print >> sys.stderr, (
+             "Warning: option %s in section %s is deprecated" %
+             (opt, sect))
  
      def _report_option_error(self, sect, opt, val, stream, msg):



From anadelonbrin at users.sourceforge.net  Mon Dec 15 02:52:44 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Mon Dec 15 02:52:46 2003
Subject: [Spambayes-checkins] spambayes/spambayes Options.py, 1.88,
	1.89 tokenizer.py, 1.17, 1.18
Message-ID: <E1AVnX2-000801-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv30744/spambayes-old/spambayes

Modified Files:
	Options.py tokenizer.py 
Log Message:
I (some time ago) committed by accident options to control a local token
generator for the habeas (http://habeas.com) headers, but didn't commit
the tokenizing code. (Opps & sorry!)

X'ify the options (for all I know someone has turned them on already, thinking
they work as advertised) and commit the tokenizing code.  If reports indicate
that they make no difference, or we aren't able to get some sort of easy-to-use
testing setup for release users, then I'll strip both of these out after the next
release.

Index: Options.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/Options.py,v
retrieving revision 1.88
retrieving revision 1.89
diff -C2 -d -r1.88 -r1.89
*** Options.py	15 Dec 2003 00:16:19 -0000	1.88
--- Options.py	15 Dec 2003 07:52:41 -0000	1.89
***************
*** 155,172 ****
       BOOLEAN, RESTORE),
  
!     ("search_for_habeas_headers", "", False,
       """If true, search for the habeas headers (see http://www.habeas.com)
!      If they are present and correct, this is a strong ham sign, if they
!      are present and incorrect, this is a strong spam sign""",
       BOOLEAN, RESTORE),
  
!     ("reduce_habeas_headers", "", False,
!      """If search_for_habeas_headers is set, nine tokens are generated for
!      messages with habeas headers.  This should be fine, since messages
!      with the headers should either be ham, or result in FN so that we can
!      send them to habeas so they can be sued.  However, to reduce the
!      strength of habeas headers, we offer the ability to reduce the nine
!      tokens to one. (This option has no effect if search_for_habeas_headers
!      is False)""",
       BOOLEAN, RESTORE),
    ),
--- 155,172 ----
       BOOLEAN, RESTORE),
  
!     ("X-search_for_habeas_headers", "Search for Habeas Headers", False,
       """If true, search for the habeas headers (see http://www.habeas.com)
!      If they are present and correct, this should be a strong ham sign, if
!      they are present and incorrect, this should be a strong spam sign.""",
       BOOLEAN, RESTORE),
  
!     ("X-reduce_habeas_headers", "Reduce Habeas Header Tokens to Single", False,
!      """If SpamBayes is set to search for the Habeas headers, nine tokens
!      are generated for messages with habeas headers.  This should be fine,
!      since messages with the headers should either be ham, or result in FN
!      so that we can send them to habeas so they can be sued.  However, to 
!      reduce the strength of habeas headers, we offer the ability to reduce 
!      the nine tokens to one. (This option has no effect if 
!      search_for_habeas_headers is False)""",
       BOOLEAN, RESTORE),
    ),
***************
*** 364,368 ****
       assigning certainty (0 or 1) to a word that has appeared in only ham
       or only spam.  This is a disaster.
! 
       As unknown_word_strength tends toward infintity, all probabilities
       tend toward unknown_word_prob.  All reports were that a value near 0.4
--- 364,368 ----
       assigning certainty (0 or 1) to a word that has appeared in only ham
       or only spam.  This is a disaster.
!      
       As unknown_word_strength tends toward infintity, all probabilities
       tend toward unknown_word_prob.  All reports were that a value near 0.4
***************
*** 953,957 ****
       PORT, RESTORE),
    ),
! 
    "globals" : (
      ("verbose", "Verbose", False,
--- 953,957 ----
       PORT, RESTORE),
    ),
!  
    "globals" : (
      ("verbose", "Verbose", False,
***************
*** 1015,1019 ****
              options.merge_files(alts)
              optionsPathname = os.path.abspath(alts[-1])
! 
      if not optionsPathname:
          optionsPathname = os.path.abspath('bayescustomize.ini')
--- 1015,1019 ----
              options.merge_files(alts)
              optionsPathname = os.path.abspath(alts[-1])
!     
      if not optionsPathname:
          optionsPathname = os.path.abspath('bayescustomize.ini')

Index: tokenizer.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/tokenizer.py,v
retrieving revision 1.17
retrieving revision 1.18
diff -C2 -d -r1.17 -r1.18
*** tokenizer.py	9 Dec 2003 16:04:16 -0000	1.17
--- tokenizer.py	15 Dec 2003 07:52:41 -0000	1.18
***************
*** 659,692 ****
  def tokenize_word(word, _len=len, maxword=options["Tokenizer",
                                                    "skip_max_word_size"]):
!     n = _len(word)
!     # Make sure this range matches in tokenize().
!     if 3 <= n <= maxword:
!         yield word
  
!     elif n >= 3:
!         # A long word.
  
!         # Don't want to skip embedded email addresses.
!         # An earlier scheme also split up the y in x@y on '.'.  Not splitting
!         # improved the f-n rate; the f-p rate didn't care either way.
!         if n < 40 and '.' in word and word.count('@') == 1:
!             p1, p2 = word.split('@')
!             yield 'email name:' + p1
!             yield 'email addr:' + p2
  
!         else:
!             # There's value in generating a token indicating roughly how
!             # many chars were skipped.  This has real benefit for the f-n
!             # rate, but is neutral for the f-p rate.  I don't know why!
!             # XXX Figure out why, and/or see if some other way of summarizing
!             # XXX this info has greater benefit.
!             if options["Tokenizer", "generate_long_skips"]:
!                 yield "skip:%c %d" % (word[0], n // 10 * 10)
!             if has_highbit_char(word):
!                 hicount = 0
!                 for i in map(ord, word):
!                     if i >= 128:
!                         hicount += 1
!                 yield "8bit%%:%d" % round(hicount * 100.0 / len(word))
  
  # Generate tokens for:
--- 659,692 ----
  def tokenize_word(word, _len=len, maxword=options["Tokenizer",
                                                    "skip_max_word_size"]):
!    n = _len(word)
!    # Make sure this range matches in tokenize().
!    if 3 <= n <= maxword:
!        yield word
  
!    elif n >= 3:
!        # A long word.
  
!        # Don't want to skip embedded email addresses.
!        # An earlier scheme also split up the y in x@y on '.'.  Not splitting
!        # improved the f-n rate; the f-p rate didn't care either way.
!        if n < 40 and '.' in word and word.count('@') == 1:
!            p1, p2 = word.split('@')
!            yield 'email name:' + p1
!            yield 'email addr:' + p2
  
!        else:
!            # There's value in generating a token indicating roughly how
!            # many chars were skipped.  This has real benefit for the f-n
!            # rate, but is neutral for the f-p rate.  I don't know why!
!            # XXX Figure out why, and/or see if some other way of summarizing
!            # XXX this info has greater benefit.
!            if options["Tokenizer", "generate_long_skips"]:
!                yield "skip:%c %d" % (word[0], n // 10 * 10)
!            if has_highbit_char(word):
!                hicount = 0
!                for i in map(ord, word):
!                    if i >= 128:
!                        hicount += 1
!                yield "8bit%%:%d" % round(hicount * 100.0 / len(word))
  
  # Generate tokens for:
***************
*** 1138,1141 ****
--- 1138,1178 ----
                  return
  
+         # Habeas Headers - see http://www.habeas.com
+         if options["Tokenizer", "search_for_habeas_headers"]:
+             habeas_headers = [
+ ("X-Habeas-SWE-1", "winter into spring"),
+ ("X-Habeas-SWE-2", "brightly anticipated"),
+ ("X-Habeas-SWE-3", "like Habeas SWE (tm)"),
+ ("X-Habeas-SWE-4", "Copyright 2002 Habeas (tm)"),
+ ("X-Habeas-SWE-5", "Sender Warranted Email (SWE) (tm). The sender of this"),
+ ("X-Habeas-SWE-6", "email in exchange for a license for this Habeas"),
+ ("X-Habeas-SWE-7", "warrant mark warrants that this is a Habeas Compliant"),
+ ("X-Habeas-SWE-8", "Message (HCM) and not spam. Please report use of this"),
+ ("X-Habeas-SWE-9", "mark in spam to <http://www.habeas.com/report/>.")
+             ]
+             valid_habeas = 0
+             invalid_habeas = False
+             for opt, val in habeas_headers:
+                 habeas = msg.get(opt)
+                 if habeas is not None:
+                     if options["Tokenizer", "reduce_habeas_headers"]:
+                         if habeas == val:
+                             valid_habeas += 1
+                         else:
+                             invalid_habeas = True
+                     else:
+                         if habeas == val:
+                             yield opt.lower() + ":valid"
+                         else:
+                             yield opt.lower() + ":invalid"
+             if options["Tokenizer", "reduce_habeas_headers"]:
+                 # If there was any invalid line, we record as invalid.
+                 # If all nine lines were correct, we record as valid.
+                 # Otherwise we ignore.
+                 if invalid_habeas == True:
+                     yield "x-habeas-swe:invalid"
+                 elif valid_habeas == 9:
+                     yield "x-habeas-swe:valid"
+                     
          # Subject:
          # Don't ignore case in Subject lines; e.g., 'free' versus 'FREE' is



From anadelonbrin at users.sourceforge.net  Mon Dec 15 03:10:07 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Mon Dec 15 03:10:12 2003
Subject: [Spambayes-checkins] spambayes/windows/py2exe .cvsignore,NONE,1.1
Message-ID: <E1AVnnr-0000Iq-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/windows/py2exe
In directory sc8-pr-cvs1:/tmp/cvs-serv1117/windows/py2exe

Added Files:
	.cvsignore 
Log Message:
Ignore autogenerated.

--- NEW FILE: .cvsignore ---
build
dist


From anadelonbrin at users.sourceforge.net  Mon Dec 15 03:10:45 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Mon Dec 15 03:10:48 2003
Subject: [Spambayes-checkins] spambayes/windows .cvsignore,NONE,1.1
Message-ID: <E1AVnoT-0000Jw-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/windows
In directory sc8-pr-cvs1:/tmp/cvs-serv1229/windows

Added Files:
	.cvsignore 
Log Message:
Ignore autogenerated.

--- NEW FILE: .cvsignore ---
SpamBayes-Setup.exe


From anadelonbrin at users.sourceforge.net  Mon Dec 15 03:16:28 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Mon Dec 15 03:16:32 2003
Subject: [Spambayes-checkins] spambayes/spambayes tokenizer.py,1.18,1.19
Message-ID: <E1AVnu0-0000Z7-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv2167/spambayes

Modified Files:
	tokenizer.py 
Log Message:
I'm not sure how, but somehow I ended up dedenting a function in the last
checkin.  Undo that.

Index: tokenizer.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/tokenizer.py,v
retrieving revision 1.18
retrieving revision 1.19
diff -C2 -d -r1.18 -r1.19
*** tokenizer.py	15 Dec 2003 07:52:41 -0000	1.18
--- tokenizer.py	15 Dec 2003 08:16:26 -0000	1.19
***************
*** 659,692 ****
  def tokenize_word(word, _len=len, maxword=options["Tokenizer",
                                                    "skip_max_word_size"]):
!    n = _len(word)
!    # Make sure this range matches in tokenize().
!    if 3 <= n <= maxword:
!        yield word
! 
!    elif n >= 3:
!        # A long word.
! 
!        # Don't want to skip embedded email addresses.
!        # An earlier scheme also split up the y in x@y on '.'.  Not splitting
!        # improved the f-n rate; the f-p rate didn't care either way.
!        if n < 40 and '.' in word and word.count('@') == 1:
!            p1, p2 = word.split('@')
!            yield 'email name:' + p1
!            yield 'email addr:' + p2
  
!        else:
!            # There's value in generating a token indicating roughly how
!            # many chars were skipped.  This has real benefit for the f-n
!            # rate, but is neutral for the f-p rate.  I don't know why!
!            # XXX Figure out why, and/or see if some other way of summarizing
!            # XXX this info has greater benefit.
!            if options["Tokenizer", "generate_long_skips"]:
!                yield "skip:%c %d" % (word[0], n // 10 * 10)
!            if has_highbit_char(word):
!                hicount = 0
!                for i in map(ord, word):
!                    if i >= 128:
!                        hicount += 1
!                yield "8bit%%:%d" % round(hicount * 100.0 / len(word))
  
  # Generate tokens for:
--- 659,692 ----
  def tokenize_word(word, _len=len, maxword=options["Tokenizer",
                                                    "skip_max_word_size"]):
!     n = _len(word)
!     # Make sure this range matches in tokenize().
!     if 3 <= n <= maxword:
!         yield word
!         
!     elif n >= 3:
!         # A long word.
!         
!         # Don't want to skip embedded email addresses.
!         # An earlier scheme also split up the y in x@y on '.'.  Not splitting
!         # improved the f-n rate; the f-p rate didn't care either way.
!         if n < 40 and '.' in word and word.count('@') == 1:
!             p1, p2 = word.split('@')
!             yield 'email name:' + p1
!             yield 'email addr:' + p2
  
!         else:
!             # There's value in generating a token indicating roughly how
!             # many chars were skipped.  This has real benefit for the f-n
!             # rate, but is neutral for the f-p rate.  I don't know why!
!             # XXX Figure out why, and/or see if some other way of summarizing
!             # XXX this info has greater benefit.
!             if options["Tokenizer", "generate_long_skips"]:
!                 yield "skip:%c %d" % (word[0], n // 10 * 10)
!             if has_highbit_char(word):
!                 hicount = 0
!                 for i in map(ord, word):
!                     if i >= 128:
!                         hicount += 1
!                 yield "8bit%%:%d" % round(hicount * 100.0 / len(word))
  
  # Generate tokens for:



From anadelonbrin at users.sourceforge.net  Mon Dec 15 03:33:36 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Mon Dec 15 03:33:40 2003
Subject: [Spambayes-checkins] spambayes/spambayes tokenizer.py,1.19,1.20
Message-ID: <E1AVoAa-0001U4-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv5699/spambayes

Modified Files:
	tokenizer.py 
Log Message:
An x'ified option can only be referred to without the 'x-' when reading
from the file, not otherwise!  Fix this wrt the habeas code.

Index: tokenizer.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/tokenizer.py,v
retrieving revision 1.19
retrieving revision 1.20
diff -C2 -d -r1.19 -r1.20
*** tokenizer.py	15 Dec 2003 08:16:26 -0000	1.19
--- tokenizer.py	15 Dec 2003 08:33:34 -0000	1.20
***************
*** 1139,1143 ****
  
          # Habeas Headers - see http://www.habeas.com
!         if options["Tokenizer", "search_for_habeas_headers"]:
              habeas_headers = [
  ("X-Habeas-SWE-1", "winter into spring"),
--- 1139,1143 ----
  
          # Habeas Headers - see http://www.habeas.com
!         if options["Tokenizer", "X-search_for_habeas_headers"]:
              habeas_headers = [
  ("X-Habeas-SWE-1", "winter into spring"),
***************
*** 1156,1160 ****
                  habeas = msg.get(opt)
                  if habeas is not None:
!                     if options["Tokenizer", "reduce_habeas_headers"]:
                          if habeas == val:
                              valid_habeas += 1
--- 1156,1160 ----
                  habeas = msg.get(opt)
                  if habeas is not None:
!                     if options["Tokenizer", "X-reduce_habeas_headers"]:
                          if habeas == val:
                              valid_habeas += 1
***************
*** 1166,1170 ****
                          else:
                              yield opt.lower() + ":invalid"
!             if options["Tokenizer", "reduce_habeas_headers"]:
                  # If there was any invalid line, we record as invalid.
                  # If all nine lines were correct, we record as valid.
--- 1166,1170 ----
                          else:
                              yield opt.lower() + ":invalid"
!             if options["Tokenizer", "X-reduce_habeas_headers"]:
                  # If there was any invalid line, we record as invalid.
                  # If all nine lines were correct, we record as valid.



From anadelonbrin at users.sourceforge.net  Mon Dec 15 04:13:42 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Mon Dec 15 04:13:50 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 manager.py,1.92,1.93
Message-ID: <E1AVonO-0003Ba-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv12206/Outlook2000

Modified Files:
	manager.py 
Log Message:
Fix a couple of comment typos.

Index: manager.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/manager.py,v
retrieving revision 1.92
retrieving revision 1.93
diff -C2 -d -r1.92 -r1.93
*** manager.py	14 Dec 2003 11:23:03 -0000	1.92
--- manager.py	15 Dec 2003 09:13:40 -0000	1.93
***************
*** 93,97 ****
          # No DB library at all!
          assert not hasattr(sys, "frozen"), \
!                "Don't build binary versions without bsbbd!"
          use_db = False
  
--- 93,97 ----
          # No DB library at all!
          assert not hasattr(sys, "frozen"), \
!                "Don't build binary versions without bsddb!"
          use_db = False
  
***************
*** 388,392 ****
          # directory (version 0.8 and earlier, we copied the app one to the
          # user dir - that was a mistake - but supporting a version in that
!         # directory wasn't.
          bayes_option_filenames = []
          # data dir last so options there win.
--- 388,392 ----
          # directory (version 0.8 and earlier, we copied the app one to the
          # user dir - that was a mistake - but supporting a version in that
!         # directory wasn't).
          bayes_option_filenames = []
          # data dir last so options there win.



From anadelonbrin at users.sourceforge.net  Mon Dec 15 04:15:21 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Mon Dec 15 04:15:23 2003
Subject: [Spambayes-checkins] spambayes/scripts sb_imapfilter.py,1.15,1.16
Message-ID: <E1AVooz-0003Jv-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/scripts
In directory sc8-pr-cvs1:/tmp/cvs-serv12754/scripts

Modified Files:
	sb_imapfilter.py 
Log Message:
Add a warning as a temporary solution for Python bug #845560.
Fix grammer in comment.

Index: sb_imapfilter.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/scripts/sb_imapfilter.py,v
retrieving revision 1.15
retrieving revision 1.16
diff -C2 -d -r1.15 -r1.16
*** sb_imapfilter.py	21 Oct 2003 21:46:31 -0000	1.15
--- sb_imapfilter.py	15 Dec 2003 09:15:19 -0000	1.16
***************
*** 238,246 ****
                      # It is faster to do close() than a single
                      # expunge when we log out (because expunge returns
!                     # a list of all the deleted messages, that we don't do
                      # anything with)
                      imap.close()
              # We *always* use SELECT and not EXAMINE, because this
              # speeds things up considerably.
              response = self.select(folder, None)
              if response[0] != "OK":
--- 238,252 ----
                      # It is faster to do close() than a single
                      # expunge when we log out (because expunge returns
!                     # a list of all the deleted messages which we don't do
                      # anything with)
                      imap.close()
              # We *always* use SELECT and not EXAMINE, because this
              # speeds things up considerably.
+             if folder == "":
+                 # This is Python bug #845560 - if the empty string is
+                 # passed, we get a traceback, not just an 'invalid folder'
+                 # error, so print out a warning and exit.
+                 print "Tried to select an invalid folder"
+                 sys.exit(-1)
              response = self.select(folder, None)
              if response[0] != "OK":



From anadelonbrin at users.sourceforge.net  Mon Dec 15 04:20:35 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Mon Dec 15 04:20:38 2003
Subject: [Spambayes-checkins] spambayes/spambayes Options.py, 1.89,
	1.90 OptionsClass.py, 1.18, 1.19 tokenizer.py, 1.20, 1.21
Message-ID: <E1AVou3-0003fL-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv14085/spambayes

Modified Files:
	Options.py OptionsClass.py tokenizer.py 
Log Message:
Bah.  ConfigParser lower()s all option names, which is why X- didn't work,
and no doubt the explanation behind Skip's cryptic windows & case-sensitivity
checkin comment (should have figured that out earlier!).  We could subclass
ConfigParser to remove this, but I have no idea what other effects that might
have at this point.

So, back out the X-/x- changes, and instead correct the comments.  "x-" is
the correct prefix (although since config files are case-insensitive in option names,
using "X-" there will work, but should be frowned on).

Index: Options.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/Options.py,v
retrieving revision 1.89
retrieving revision 1.90
diff -C2 -d -r1.89 -r1.90
*** Options.py	15 Dec 2003 07:52:41 -0000	1.89
--- Options.py	15 Dec 2003 09:20:33 -0000	1.90
***************
*** 155,159 ****
       BOOLEAN, RESTORE),
  
!     ("X-search_for_habeas_headers", "Search for Habeas Headers", False,
       """If true, search for the habeas headers (see http://www.habeas.com)
       If they are present and correct, this should be a strong ham sign, if
--- 155,159 ----
       BOOLEAN, RESTORE),
  
!     ("x-search_for_habeas_headers", "Search for Habeas Headers", False,
       """If true, search for the habeas headers (see http://www.habeas.com)
       If they are present and correct, this should be a strong ham sign, if
***************
*** 161,165 ****
       BOOLEAN, RESTORE),
  
!     ("X-reduce_habeas_headers", "Reduce Habeas Header Tokens to Single", False,
       """If SpamBayes is set to search for the Habeas headers, nine tokens
       are generated for messages with habeas headers.  This should be fine,
--- 161,165 ----
       BOOLEAN, RESTORE),
  
!     ("x-reduce_habeas_headers", "Reduce Habeas Header Tokens to Single", False,
       """If SpamBayes is set to search for the Habeas headers, nine tokens
       are generated for messages with habeas headers.  This should be fine,

Index: OptionsClass.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/OptionsClass.py,v
retrieving revision 1.18
retrieving revision 1.19
diff -C2 -d -r1.18 -r1.19
*** OptionsClass.py	15 Dec 2003 07:48:40 -0000	1.18
--- OptionsClass.py	15 Dec 2003 09:20:33 -0000	1.19
***************
*** 22,32 ****
  via an instance of this class.
  
! Experimental or deprecated options are prefixed with 'X-', borrowing the
  practice from RFC-822 mail.  If the user sets an option like:
  
      [Tokenizer]
!     X-transmogrify: True
  
! and an 'X-transmogrify' or 'transmogrify' option exists, it is set silently
  to the value given by the user.  If the user sets an option like:
  
--- 22,32 ----
  via an instance of this class.
  
! Experimental or deprecated options are prefixed with 'x-', borrowing the
  practice from RFC-822 mail.  If the user sets an option like:
  
      [Tokenizer]
!     x-transmogrify: True
  
! and an 'x-transmogrify' or 'transmogrify' option exists, it is set silently
  to the value given by the user.  If the user sets an option like:
  
***************
*** 34,38 ****
      transmogrify: True
  
! and no 'transmogrify' option exists, but an 'X-transmogrify' option does,
  the latter is set to the value given by the users and a deprecation message
  is printed to standard error.
--- 34,38 ----
      transmogrify: True
  
! and no 'transmogrify' option exists, but an 'x-transmogrify' option does,
  the latter is set to the value given by the users and a deprecation message
  is printed to standard error.
***************
*** 528,532 ****
                  if not self._options.has_key((section, option)):
                      if option.startswith('x-'):
!                         # try setting option without the X- prefix
                          option = option[2:]
                          if self._options.has_key((section, option)):
--- 528,532 ----
                  if not self._options.has_key((section, option)):
                      if option.startswith('x-'):
!                         # try setting option without the x- prefix
                          option = option[2:]
                          if self._options.has_key((section, option)):
***************
*** 534,548 ****
                          # not an error if an X- option is missing
                      else:
!                         l_option = 'x-' + option
!                         u_option = 'X-' + option
                          # going the other way, if the option has been
                          # deprecated, set its x-prefixed version and
                          # emit a warning
!                         if self._options.has_key((section, l_option)):
!                             self.convert_and_set(section, l_option, value)
!                             self._report_deprecated_error(section, option)
!                         elif self._options.has_key((section, u_option)):
!                             self.convert_and_set(section, u_option, value)
!                             self._report_deprecated_error(section, option)
                          else:
                              print >> sys.stderr, (
--- 534,544 ----
                          # not an error if an X- option is missing
                      else:
!                         option = 'x-' + option
                          # going the other way, if the option has been
                          # deprecated, set its x-prefixed version and
                          # emit a warning
!                         if self._options.has_key((section, option)):
!                             self.convert_and_set(section, option, value)
!                             self._report_deprecated_error(section, opt)
                          else:
                              print >> sys.stderr, (

Index: tokenizer.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/tokenizer.py,v
retrieving revision 1.20
retrieving revision 1.21
diff -C2 -d -r1.20 -r1.21
*** tokenizer.py	15 Dec 2003 08:33:34 -0000	1.20
--- tokenizer.py	15 Dec 2003 09:20:33 -0000	1.21
***************
*** 1139,1143 ****
  
          # Habeas Headers - see http://www.habeas.com
!         if options["Tokenizer", "X-search_for_habeas_headers"]:
              habeas_headers = [
  ("X-Habeas-SWE-1", "winter into spring"),
--- 1139,1143 ----
  
          # Habeas Headers - see http://www.habeas.com
!         if options["Tokenizer", "x-search_for_habeas_headers"]:
              habeas_headers = [
  ("X-Habeas-SWE-1", "winter into spring"),
***************
*** 1156,1160 ****
                  habeas = msg.get(opt)
                  if habeas is not None:
!                     if options["Tokenizer", "X-reduce_habeas_headers"]:
                          if habeas == val:
                              valid_habeas += 1
--- 1156,1160 ----
                  habeas = msg.get(opt)
                  if habeas is not None:
!                     if options["Tokenizer", "x-reduce_habeas_headers"]:
                          if habeas == val:
                              valid_habeas += 1
***************
*** 1166,1170 ****
                          else:
                              yield opt.lower() + ":invalid"
!             if options["Tokenizer", "X-reduce_habeas_headers"]:
                  # If there was any invalid line, we record as invalid.
                  # If all nine lines were correct, we record as valid.
--- 1166,1170 ----
                          else:
                              yield opt.lower() + ":invalid"
!             if options["Tokenizer", "x-reduce_habeas_headers"]:
                  # If there was any invalid line, we record as invalid.
                  # If all nine lines were correct, we record as valid.



From montanaro at users.sourceforge.net  Mon Dec 15 08:58:51 2003
From: montanaro at users.sourceforge.net (Skip Montanaro)
Date: Mon Dec 15 08:58:55 2003
Subject: [Spambayes-checkins] spambayes/spambayes mboxutils.py,1.3,1.4
Message-ID: <E1AVtFL-00083L-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv30955

Modified Files:
	mboxutils.py 
Log Message:
(nobody complained)

This change generalizes the DirOfTxtFileMailbox class to assume all
non-directory files contain a single message and to recursively descend into
subdirectories of the argument directory.



Index: mboxutils.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/mboxutils.py,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** mboxutils.py	19 Sep 2003 23:38:10 -0000	1.3
--- mboxutils.py	15 Dec 2003 13:58:49 -0000	1.4
***************
*** 31,39 ****
  class DirOfTxtFileMailbox:
  
!     """Mailbox directory consisting of .txt and .lorien files."""
  
      def __init__(self, dirname, factory):
!         self.names = (glob.glob(os.path.join(dirname, "*.txt")) +
!                       glob.glob(os.path.join(dirname, "*.lorien")))
          self.names.sort()
          self.factory = factory
--- 31,41 ----
  class DirOfTxtFileMailbox:
  
!     """Directory of files each assumed to contain an RFC-822 message.
! 
!     Subdirectories are traversed recursively.
!     """
  
      def __init__(self, dirname, factory):
!         self.names = glob.glob(os.path.join(dirname, "*"))
          self.names.sort()
          self.factory = factory
***************
*** 41,50 ****
      def __iter__(self):
          for name in self.names:
!             try:
!                 f = open(name)
!             except IOError:
!                 continue
!             yield self.factory(f)
!             f.close()
  
  def _cat(seqs):
--- 43,56 ----
      def __iter__(self):
          for name in self.names:
!             if os.path.isdir(name):
!                 for mbox in DirOfTxtFileMailbox(name, self.factory):
!                     yield mbox
!             else:
!                 try:
!                     f = open(name)
!                 except IOError:
!                     continue
!                 yield self.factory(f)
!                 f.close()
  
  def _cat(seqs):



From anadelonbrin at users.sourceforge.net  Mon Dec 15 19:30:51 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Mon Dec 15 19:30:56 2003
Subject: [Spambayes-checkins] spambayes/spambayes ImapUI.py, 1.28,
	1.29 ProxyUI.py, 1.32, 1.33
Message-ID: <E1AW36x-0000yc-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv3741/spambayes

Modified Files:
	ImapUI.py ProxyUI.py 
Log Message:
Don't offer deprecated options on the config page of the web interface.

Index: ImapUI.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/ImapUI.py,v
retrieving revision 1.28
retrieving revision 1.29
diff -C2 -d -r1.28 -r1.29
*** ImapUI.py	15 Dec 2003 00:16:19 -0000	1.28
--- ImapUI.py	16 Dec 2003 00:30:49 -0000	1.29
***************
*** 66,71 ****
      ('html_ui',               'allow_remote_connections'),
      ('Header Options',        None),
!     ('Headers',             'notate_to'),
!     ('Headers',             'notate_subject'),
      ('Storage Options',       None),
      ('Storage',               'persistent_storage_file'),
--- 66,71 ----
      ('html_ui',               'allow_remote_connections'),
      ('Header Options',        None),
!     ('Headers',               'notate_to'),
!     ('Headers',               'notate_subject'),
      ('Storage Options',       None),
      ('Storage',               'persistent_storage_file'),
***************
*** 94,99 ****
      ('Storage',             'persistent_use_database'),
      ('Tokenising Options',  None),
-     ('Tokenizer',           'extract_dow'),
-     ('Tokenizer',           'generate_time_buckets'),
      ('Tokenizer',           'mine_received_headers'),
      ('Tokenizer',           'replace_nonascii_chars'),
--- 94,97 ----

Index: ProxyUI.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/ProxyUI.py,v
retrieving revision 1.32
retrieving revision 1.33
diff -C2 -d -r1.32 -r1.33
*** ProxyUI.py	15 Dec 2003 00:16:19 -0000	1.32
--- ProxyUI.py	16 Dec 2003 00:30:49 -0000	1.33
***************
*** 88,94 ****
      ('smtpproxy',           'use_cached_message'),
      ('Header Options',      None),
!     ('Headers',           'notate_to'),
!     ('Headers',           'notate_subject'),
!     ('Storage Options',  None),
      ('Storage',             'persistent_storage_file'),
      ('Storage',             'messageinfo_storage_file'),
--- 88,94 ----
      ('smtpproxy',           'use_cached_message'),
      ('Header Options',      None),
!     ('Headers',             'notate_to'),
!     ('Headers',             'notate_subject'),
!     ('Storage Options',     None),
      ('Storage',             'persistent_storage_file'),
      ('Storage',             'messageinfo_storage_file'),
***************
*** 124,134 ****
      ('Storage',             'unknown_cache'),
      ('Tokenising Options',  None),
-     ('Tokenizer',           'extract_dow'),
-     ('Tokenizer',           'generate_time_buckets'),
      ('Tokenizer',           'mine_received_headers'),
      ('Tokenizer',           'replace_nonascii_chars'),
      ('Tokenizer',           'summarize_email_prefixes'),
      ('Tokenizer',           'summarize_email_suffixes'),
!     ('Training Options',   None),
      ('Hammie',              'train_on_filter'),
      ('Interface Options',   None),
--- 124,132 ----
      ('Storage',             'unknown_cache'),
      ('Tokenising Options',  None),
      ('Tokenizer',           'mine_received_headers'),
      ('Tokenizer',           'replace_nonascii_chars'),
      ('Tokenizer',           'summarize_email_prefixes'),
      ('Tokenizer',           'summarize_email_suffixes'),
!     ('Training Options',    None),
      ('Hammie',              'train_on_filter'),
      ('Interface Options',   None),



From anadelonbrin at users.sourceforge.net  Mon Dec 15 21:03:33 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Mon Dec 15 21:03:36 2003
Subject: [Spambayes-checkins] spambayes/spambayes Options.py, 1.90,
	1.91 UserInterface.py, 1.35, 1.36 classifier.py, 1.11, 1.12
Message-ID: <E1AW4Yf-0004kH-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv18229/spambayes

Modified Files:
	Options.py UserInterface.py classifier.py 
Log Message:
New experimental option: x-use_bigrams.

This is the mixed uni/bi-grams scheme outlined by Tim/Graham.  Basically both uni and
bi grams (of *tokens* not words) are generated and the strongest of the unigrams and
bigram is used, rather than all three.

To get more information:
>>> from spambayes.Options import options
>>> print options.display_full("Classifier", "x-use_bigrams")

Also fix a bug in the web interface where the probability would be incorrectly
calculated on 'show clues'.

Index: Options.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/Options.py,v
retrieving revision 1.90
retrieving revision 1.91
diff -C2 -d -r1.90 -r1.91
*** Options.py	15 Dec 2003 09:20:33 -0000	1.90
--- Options.py	16 Dec 2003 02:03:30 -0000	1.91
***************
*** 427,430 ****
--- 427,452 ----
       to retrain your database if you change this option.""",
       BOOLEAN, RESTORE),
+ 
+     ("x-use_bigrams", "(EXPERIMENTAL) Use mixed uni/bi-grams scheme", False,
+      """Enabling this option means that SpamBayes will generate both
+      unigrams (words) and bigrams (pairs of words). However, Graham's
+      scheme is also used, where, for each clue triplet of two unigrams and
+      the bigram they make, the clue that is strongest (i.e. has a
+      probability furtherest from 0.5) is used, and the other two are not.
+ 
+      Note that to really test this option you need to retrain with it on,
+      so that your database includes the bigrams - if you subsequently turn
+      it off, these tokens will have no effect.
+ 
+      Note also that you should probably also increase the max_discriminators
+      (Maximum number of extreme words) option if you enable this option;
+      this may need to be doubled or quadrupled to see the benefit from the
+      bigrams.
+ 
+      This option is experimental, and may be removed in a future release.
+      We would appreciate feedback about it if you use it - email
+      spambayes@python.org with your comments and results.
+      """,
+      BOOLEAN, RESTORE),
    ),
  

Index: UserInterface.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/UserInterface.py,v
retrieving revision 1.35
retrieving revision 1.36
diff -C2 -d -r1.35 -r1.36
*** UserInterface.py	11 Dec 2003 18:44:23 -0000	1.35
--- UserInterface.py	16 Dec 2003 02:03:31 -0000	1.36
***************
*** 306,309 ****
--- 306,313 ----
              for tok in tokens:
                  clues.append((tok, None))
+             # Need to regenerate the tokens (is there a way to
+             # 'rewind' or copy a generator?  Would that be
+             # more effecient?
+             tokens = tokenizer.tokenize(message)
              probability = self.classifier.spamprob(tokens)
              cluesTable = self._fillCluesTable(clues)

Index: classifier.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/classifier.py,v
retrieving revision 1.11
retrieving revision 1.12
diff -C2 -d -r1.11 -r1.12
*** classifier.py	14 Dec 2003 04:12:32 -0000	1.11
--- classifier.py	16 Dec 2003 02:03:31 -0000	1.12
***************
*** 1,3 ****
--- 1,6 ----
  #! /usr/bin/env python
+ 
+ from __future__ import generators
+ 
  # An implementation of a Bayes-like spam classifier.
  #
***************
*** 36,39 ****
--- 39,43 ----
  
  import math
+ import types
  try:
      from sets import Set
***************
*** 206,212 ****
          True, you're telling the classifier this message is definitely spam,
          else that it's definitely not spam.
- 
          """
! 
          self._add_msg(wordstream, is_spam)
  
--- 210,217 ----
          True, you're telling the classifier this message is definitely spam,
          else that it's definitely not spam.
          """
!         if options["Classifier", "x-use_bigrams"] and \
!            isinstance(wordstream, types.GeneratorType):
!             wordstream = self._enhance_wordstream(wordstream)
          self._add_msg(wordstream, is_spam)
  
***************
*** 216,219 ****
--- 221,227 ----
          Pass the same arguments you passed to learn().
          """
+         if options["Classifier", "x-use_bigrams"] and \
+            isinstance(wordstream, types.GeneratorType):
+             wordstream = self._enhance_wordstream(wordstream)
          self._remove_msg(wordstream, is_spam)
  
***************
*** 300,311 ****
      # to exploit it.
      def _add_msg(self, wordstream, is_spam):
-         # I think the string stuff is hiding the cause of the db ham/spam
-         # count problem, so remove it for now.
          self.probcache = {}    # nuke the prob cache
          if is_spam:
-             #self.nspam = int(self.nspam) + 1  # account for string nspam
              self.nspam += 1
          else:
-             #self.nham = int(self.nham) + 1   # account for string nham
              self.nham += 1
  
--- 308,315 ----
***************
*** 359,377 ****
      def _getclues(self, wordstream):
          mindist = options["Classifier", "minimum_prob_strength"]
-         unknown = options["Classifier", "unknown_word_prob"]
  
!         clues = []  # (distance, prob, word, record) tuples
!         pushclue = clues.append
  
!         for word in Set(wordstream):
!             record = self._wordinfoget(word)
!             if record is None:
!                 prob = unknown
!             else:
!                 prob = self.probability(record)
!             distance = abs(prob - 0.5)
!             if distance >= mindist:
!                 pushclue((distance, prob, word, record))
  
          clues.sort()
          if len(clues) > options["Classifier", "max_discriminators"]:
--- 363,410 ----
      def _getclues(self, wordstream):
          mindist = options["Classifier", "minimum_prob_strength"]
  
!         # clues is now a set so that that duplicates are removed.
!         # Otherwise if a token is a stronger clue than the bigrams
!         # it forms with the previous *and* next tokens, then it
!         # would be used twice.  As is, it is possible for a single
!         # token to be present in two bigram tokens (but in different
!         # positions).
!         clues = Set()  # (distance, prob, word, record) tuples
!         pushclue = clues.add
  
!         if options["Classifier", "x-use_bigrams"]:
!             # The tokens list contains 3-tuples of a token,
!             # the bigram it forms with the next token, and the
!             # next token.  This makes it easier to select which
!             # one to use later on.
!             tokens = []
!             p = None
!             while True:
!                 try:
!                     q = wordstream.next()
!                     if p:
!                         tokens.append((p, ' '.join([p, q]), q))
!                     p = q
!                 except StopIteration:
!                     break
!             
!             # Run through all the triplets and add the strongest
!             # clue from each (note also the comment above explaining
!             # how duplicates are treated).
!             for grams in tokens:
!                 winner = (0, None, None, None) # distance, prob, word, record
!                 for gram in grams:
!                     contestant = self._worddistanceget(gram)
!                     if contestant[0] > winner[0]:
!                         winner = contestant
!                 if winner[0] >= mindist:
!                     pushclue(winner)
!         else:
!             for word in wordstream: # Set() not necessary; see above.
!                 contestant = self._worddistanceget(word)
!                 if contestant[0] >= mindist:
!                     pushclue(contestant)
  
+         clues = list(clues)
          clues.sort()
          if len(clues) > options["Classifier", "max_discriminators"]:
***************
*** 380,383 ****
--- 413,426 ----
          return [t[1:] for t in clues]
  
+     def _worddistanceget(self, word):
+         unknown = options["Classifier", "unknown_word_prob"]
+         record = self._wordinfoget(word)
+         if record is None:
+             prob = unknown
+         else:
+             prob = self.probability(record)
+         distance = abs(prob - 0.5)
+         return distance, prob, word, record
+ 
      def _wordinfoget(self, word):
          return self.wordinfo.get(word)
***************
*** 389,392 ****
--- 432,460 ----
          del self.wordinfo[word]
  
+     def _enhance_wordstream(self, wordstream):
+         """Add bigrams to the wordstream.  This wraps the last token
+         to the first one, so a small number of odd tokens might get
+         generated from that, but it shouldn't be significant.  Note
+         that these are *token* bigrams, and not *word* bigrams - i.e.
+         'synthetic' tokens get bigram'ed, too.
+ 
+         The bigram token is simply "unigram1 unigram2" - a space should
+         be sufficient as a separator, since spaces aren't in any other
+         tokens, apart from 'synthetic' ones.
+ 
+         If the experimental "Classifier":"x-use_bigrams" option is
+         removed, this function can be removed, too."""
+         p = None
+         while True:
+             try:
+                 if p:
+                     yield p
+                 q = wordstream.next()
+                 if p:
+                     yield "%s %s" % (p, q)
+                 p = q
+             except StopIteration:
+                 break
+             
      def _wordinfokeys(self):
          return self.wordinfo.keys()



From anadelonbrin at users.sourceforge.net  Mon Dec 15 21:46:44 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Mon Dec 15 21:46:48 2003
Subject: [Spambayes-checkins] spambayes/spambayes UserInterface.py, 1.36,
	1.37 classifier.py, 1.12, 1.13
Message-ID: <E1AW5ES-0006Rh-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv24768/spambayes

Modified Files:
	UserInterface.py classifier.py 
Log Message:
Efficiency improvements, thanks to Tim.

(Use list comprehensions where appropriate).

Index: UserInterface.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/UserInterface.py,v
retrieving revision 1.36
retrieving revision 1.37
diff -C2 -d -r1.36 -r1.37
*** UserInterface.py	16 Dec 2003 02:03:31 -0000	1.36
--- UserInterface.py	16 Dec 2003 02:46:42 -0000	1.37
***************
*** 301,313 ****
      
      def _buildCluesTable(self, message, subject=None, show_tokens=False):
!         tokens = tokenizer.tokenize(message)
          if show_tokens:
!             clues = []
!             for tok in tokens:
!                 clues.append((tok, None))
!             # Need to regenerate the tokens (is there a way to
!             # 'rewind' or copy a generator?  Would that be
!             # more effecient?
!             tokens = tokenizer.tokenize(message)
              probability = self.classifier.spamprob(tokens)
              cluesTable = self._fillCluesTable(clues)
--- 301,307 ----
      
      def _buildCluesTable(self, message, subject=None, show_tokens=False):
!         tokens = list(tokenizer.tokenize(message))
          if show_tokens:
!             clues = [(tok, None) for tok in tokens]
              probability = self.classifier.spamprob(tokens)
              cluesTable = self._fillCluesTable(clues)

Index: classifier.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/classifier.py,v
retrieving revision 1.12
retrieving revision 1.13
diff -C2 -d -r1.12 -r1.13
*** classifier.py	16 Dec 2003 02:03:31 -0000	1.12
--- classifier.py	16 Dec 2003 02:46:42 -0000	1.13
***************
*** 363,366 ****
--- 363,368 ----
      def _getclues(self, wordstream):
          mindist = options["Classifier", "minimum_prob_strength"]
+         if not isinstance(wordstream, types.ListType):
+             wordstream = list(wordstream)
  
          # clues is now a set so that that duplicates are removed.
***************
*** 378,391 ****
              # next token.  This makes it easier to select which
              # one to use later on.
!             tokens = []
!             p = None
!             while True:
!                 try:
!                     q = wordstream.next()
!                     if p:
!                         tokens.append((p, ' '.join([p, q]), q))
!                     p = q
!                 except StopIteration:
!                     break
              
              # Run through all the triplets and add the strongest
--- 380,386 ----
              # next token.  This makes it easier to select which
              # one to use later on.
!             l = wordstream
!             tokens = [(l[i],"%s %s" % (l[i],l[(i+1)%len(l)]),
!                        l[(i+1)%len(l)]) for i in xrange(len(l))]
              
              # Run through all the triplets and add the strongest



From anadelonbrin at users.sourceforge.net  Mon Dec 15 23:48:31 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Mon Dec 15 23:48:35 2003
Subject: [Spambayes-checkins] spambayes/spambayes OptionsClass.py,1.19,1.20
Message-ID: <E1AW78J-0002Se-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv9453/spambayes

Modified Files:
	OptionsClass.py 
Log Message:
Option names are always case insensitive, no matter what.

Index: OptionsClass.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/OptionsClass.py,v
retrieving revision 1.19
retrieving revision 1.20
diff -C2 -d -r1.19 -r1.20
*** OptionsClass.py	15 Dec 2003 09:20:33 -0000	1.19
--- OptionsClass.py	16 Dec 2003 04:48:28 -0000	1.20
***************
*** 552,586 ****
      def display_name(self, sect, opt):
          '''A name for the option suitable for display to a user.'''
!         return self._options[sect, opt].display_name()
      def default(self, sect, opt):
          '''The default value for the option.'''
!         return self._options[sect, opt].default()
      def doc(self, sect, opt):
          '''Documentation for the option.'''
!         return self._options[sect, opt].doc()
      def valid_input(self, sect, opt):
          '''Valid values for the option.'''
!         return self._options[sect, opt].valid_input()
      def no_restore(self, sect, opt):
          '''Do not restore this option when restoring to defaults.'''
!         return self._options[sect, opt].no_restore()
      def is_valid(self, sect, opt, value):
          '''Check if this is a valid value for this option.'''
!         return self._options[sect, opt].is_valid(value)
      def multiple_values_allowed(self, sect, opt):
          '''Multiple values are allowed for this option.'''
!         return self._options[sect, opt].multiple_values_allowed()
  
      def is_boolean(self, sect, opt):
          '''The option is a boolean value. (Support for Python 2.2).'''
!         return self._options[sect, opt].is_boolean()
  
      def convert(self, sect, opt, value):
          '''Convert value from a string to the appropriate type.'''
!         return self._options[sect, opt].convert(value)
  
      def unconvert(self, sect, opt):
          '''Convert value from the appropriate type to a string.'''
!         return self._options[sect, opt].unconvert()
  
      def get_option(self, sect, opt):
--- 552,586 ----
      def display_name(self, sect, opt):
          '''A name for the option suitable for display to a user.'''
!         return self._options[sect, opt.lower()].display_name()
      def default(self, sect, opt):
          '''The default value for the option.'''
!         return self._options[sect, opt.lower()].default()
      def doc(self, sect, opt):
          '''Documentation for the option.'''
!         return self._options[sect, opt.lower()].doc()
      def valid_input(self, sect, opt):
          '''Valid values for the option.'''
!         return self._options[sect, opt.lower()].valid_input()
      def no_restore(self, sect, opt):
          '''Do not restore this option when restoring to defaults.'''
!         return self._options[sect, opt.lower()].no_restore()
      def is_valid(self, sect, opt, value):
          '''Check if this is a valid value for this option.'''
!         return self._options[sect, opt.lower()].is_valid(value)
      def multiple_values_allowed(self, sect, opt):
          '''Multiple values are allowed for this option.'''
!         return self._options[sect, opt.lower()].multiple_values_allowed()
  
      def is_boolean(self, sect, opt):
          '''The option is a boolean value. (Support for Python 2.2).'''
!         return self._options[sect, opt.lower()].is_boolean()
  
      def convert(self, sect, opt, value):
          '''Convert value from a string to the appropriate type.'''
!         return self._options[sect, opt.lower()].convert(value)
  
      def unconvert(self, sect, opt):
          '''Convert value from the appropriate type to a string.'''
!         return self._options[sect, opt.lower()].unconvert()
  
      def get_option(self, sect, opt):
***************
*** 588,598 ****
          if self.conversion_table.has_key((sect, opt)):
              sect, opt = self.conversion_table[sect, opt]
!         return self._options[sect, opt]
  
      def get(self, sect, opt):
          '''Get an option value.'''
!         if self.conversion_table.has_key((sect, opt)):
!             sect, opt = self.conversion_table[sect, opt]
!         return self.get_option(sect, opt).get()
  
      def __getitem__(self, key):
--- 588,598 ----
          if self.conversion_table.has_key((sect, opt)):
              sect, opt = self.conversion_table[sect, opt]
!         return self._options[sect, opt.lower()]
  
      def get(self, sect, opt):
          '''Get an option value.'''
!         if self.conversion_table.has_key((sect, opt.lower())):
!             sect, opt = self.conversion_table[sect, opt.lower()]
!         return self.get_option(sect, opt.lower()).get()
  
      def __getitem__(self, key):
***************
*** 601,612 ****
      def set(self, sect, opt, val=None):
          '''Set an option.'''
!         if self.conversion_table.has_key((sect, opt)):
!             sect, opt = self.conversion_table[sect, opt]
          if self.is_valid(sect, opt, val):
!             self._options[sect, opt].set(val)
          else:
              print >> sys.stderr, ("Attempted to set [%s] %s with invalid"
                                    " value %s (%s)" % 
!                                   (sect, opt, val, type(val)))
  
      def set_from_cmdline(self, arg, stream=None):
--- 601,612 ----
      def set(self, sect, opt, val=None):
          '''Set an option.'''
!         if self.conversion_table.has_key((sect, opt.lower())):
!             sect, opt = self.conversion_table[sect, opt.lower()]
          if self.is_valid(sect, opt, val):
!             self._options[sect, opt.lower()].set(val)
          else:
              print >> sys.stderr, ("Attempted to set [%s] %s with invalid"
                                    " value %s (%s)" % 
!                                   (sect, opt.lower(), val, type(val)))
  
      def set_from_cmdline(self, arg, stream=None):
***************
*** 617,620 ****
--- 617,621 ----
          """
          sect, opt, val = arg.split(':', 2)
+         opt = opt.lower()
          try:
              val = self.convert(sect, opt, val)
***************
*** 716,720 ****
         if section is not None and option is not None:
             output.write(self._options[section,
!                                       option].as_nice_string(section))
             return output.getvalue()
         
--- 717,721 ----
         if section is not None and option is not None:
             output.write(self._options[section,
!                                       option.lower()].as_nice_string(section))
             return output.getvalue()
         
***************
*** 724,728 ****
             if section is not None and sect != section:
                 continue
!            output.write(self._options[sect, opt].as_nice_string(sect))
         return output.getvalue()
  
--- 725,729 ----
             if section is not None and sect != section:
                 continue
!            output.write(self._options[sect, opt.lower()].as_nice_string(sect))
         return output.getvalue()
  



From tim at fourstonesExpressions.com  Mon Dec 15 23:54:14 2003
From: tim at fourstonesExpressions.com (Tim Stone)
Date: Mon Dec 15 23:54:22 2003
Subject: [Spambayes-checkins] spambayes/spambayes OptionsClass.py, 1.19,
	1.20
In-Reply-To: <E1AW78J-0002Se-00@sc8-pr-cvs1.sourceforge.net>
References: <E1AW78J-0002Se-00@sc8-pr-cvs1.sourceforge.net>
Message-ID: <oprz9bwopiit6vze@mail.fourstonesExpressions.com>

Watch the world reel now <wink>

On Mon, 15 Dec 2003 20:48:31 -0800, Tony Meyer 
<anadelonbrin@users.sourceforge.net> wrote:

> Update of /cvsroot/spambayes/spambayes/spambayes
> In directory sc8-pr-cvs1:/tmp/cvs-serv9453/spambayes
>
> Modified Files:
> 	OptionsClass.py
> Log Message:
> Option names are always case insensitive, no matter what.
>
> Index: OptionsClass.py
> ===================================================================
> RCS file: /cvsroot/spambayes/spambayes/spambayes/OptionsClass.py,v
> retrieving revision 1.19
> retrieving revision 1.20
> diff -C2 -d -r1.19 -r1.20
> *** OptionsClass.py	15 Dec 2003 09:20:33 -0000	1.19
> --- OptionsClass.py	16 Dec 2003 04:48:28 -0000	1.20
> ***************
> *** 552,586 ****
>       def display_name(self, sect, opt):
>           '''A name for the option suitable for display to a user.'''
> !         return self._options[sect, opt].display_name()
>       def default(self, sect, opt):
>           '''The default value for the option.'''
> !         return self._options[sect, opt].default()
>       def doc(self, sect, opt):
>           '''Documentation for the option.'''
> !         return self._options[sect, opt].doc()
>       def valid_input(self, sect, opt):
>           '''Valid values for the option.'''
> !         return self._options[sect, opt].valid_input()
>       def no_restore(self, sect, opt):
>           '''Do not restore this option when restoring to defaults.'''
> !         return self._options[sect, opt].no_restore()
>       def is_valid(self, sect, opt, value):
>           '''Check if this is a valid value for this option.'''
> !         return self._options[sect, opt].is_valid(value)
>       def multiple_values_allowed(self, sect, opt):
>           '''Multiple values are allowed for this option.'''
> !         return self._options[sect, opt].multiple_values_allowed()
>
>       def is_boolean(self, sect, opt):
>           '''The option is a boolean value. (Support for Python 2.2).'''
> !         return self._options[sect, opt].is_boolean()
>
>       def convert(self, sect, opt, value):
>           '''Convert value from a string to the appropriate type.'''
> !         return self._options[sect, opt].convert(value)
>
>       def unconvert(self, sect, opt):
>           '''Convert value from the appropriate type to a string.'''
> !         return self._options[sect, opt].unconvert()
>
>       def get_option(self, sect, opt):
> --- 552,586 ----
>       def display_name(self, sect, opt):
>           '''A name for the option suitable for display to a user.'''
> !         return self._options[sect, opt.lower()].display_name()
>       def default(self, sect, opt):
>           '''The default value for the option.'''
> !         return self._options[sect, opt.lower()].default()
>       def doc(self, sect, opt):
>           '''Documentation for the option.'''
> !         return self._options[sect, opt.lower()].doc()
>       def valid_input(self, sect, opt):
>           '''Valid values for the option.'''
> !         return self._options[sect, opt.lower()].valid_input()
>       def no_restore(self, sect, opt):
>           '''Do not restore this option when restoring to defaults.'''
> !         return self._options[sect, opt.lower()].no_restore()
>       def is_valid(self, sect, opt, value):
>           '''Check if this is a valid value for this option.'''
> !         return self._options[sect, opt.lower()].is_valid(value)
>       def multiple_values_allowed(self, sect, opt):
>           '''Multiple values are allowed for this option.'''
> !         return self._options[sect, 
> opt.lower()].multiple_values_allowed()
>
>       def is_boolean(self, sect, opt):
>           '''The option is a boolean value. (Support for Python 2.2).'''
> !         return self._options[sect, opt.lower()].is_boolean()
>
>       def convert(self, sect, opt, value):
>           '''Convert value from a string to the appropriate type.'''
> !         return self._options[sect, opt.lower()].convert(value)
>
>       def unconvert(self, sect, opt):
>           '''Convert value from the appropriate type to a string.'''
> !         return self._options[sect, opt.lower()].unconvert()
>
>       def get_option(self, sect, opt):
> ***************
> *** 588,598 ****
>           if self.conversion_table.has_key((sect, opt)):
>               sect, opt = self.conversion_table[sect, opt]
> !         return self._options[sect, opt]
>
>       def get(self, sect, opt):
>           '''Get an option value.'''
> !         if self.conversion_table.has_key((sect, opt)):
> !             sect, opt = self.conversion_table[sect, opt]
> !         return self.get_option(sect, opt).get()
>
>       def __getitem__(self, key):
> --- 588,598 ----
>           if self.conversion_table.has_key((sect, opt)):
>               sect, opt = self.conversion_table[sect, opt]
> !         return self._options[sect, opt.lower()]
>
>       def get(self, sect, opt):
>           '''Get an option value.'''
> !         if self.conversion_table.has_key((sect, opt.lower())):
> !             sect, opt = self.conversion_table[sect, opt.lower()]
> !         return self.get_option(sect, opt.lower()).get()
>
>       def __getitem__(self, key):
> ***************
> *** 601,612 ****
>       def set(self, sect, opt, val=None):
>           '''Set an option.'''
> !         if self.conversion_table.has_key((sect, opt)):
> !             sect, opt = self.conversion_table[sect, opt]
>           if self.is_valid(sect, opt, val):
> !             self._options[sect, opt].set(val)
>           else:
>               print >> sys.stderr, ("Attempted to set [%s] %s with 
> invalid"
>                                     " value %s (%s)" %
> !                                   (sect, opt, val, type(val)))
>
>       def set_from_cmdline(self, arg, stream=None):
> --- 601,612 ----
>       def set(self, sect, opt, val=None):
>           '''Set an option.'''
> !         if self.conversion_table.has_key((sect, opt.lower())):
> !             sect, opt = self.conversion_table[sect, opt.lower()]
>           if self.is_valid(sect, opt, val):
> !             self._options[sect, opt.lower()].set(val)
>           else:
>               print >> sys.stderr, ("Attempted to set [%s] %s with 
> invalid"
>                                     " value %s (%s)" %
> !                                   (sect, opt.lower(), val, type(val)))
>
>       def set_from_cmdline(self, arg, stream=None):
> ***************
> *** 617,620 ****
> --- 617,621 ----
>           """
>           sect, opt, val = arg.split(':', 2)
> +         opt = opt.lower()
>           try:
>               val = self.convert(sect, opt, val)
> ***************
> *** 716,720 ****
>          if section is not None and option is not None:
>              output.write(self._options[section,
> !                                       option].as_nice_string(section))
>              return output.getvalue()
>
> --- 717,721 ----
>          if section is not None and option is not None:
>              output.write(self._options[section,
> !                                       
> option.lower()].as_nice_string(section))
>              return output.getvalue()
>
> ***************
> *** 724,728 ****
>              if section is not None and sect != section:
>                  continue
> !            output.write(self._options[sect, opt].as_nice_string(sect))
>          return output.getvalue()
>
> --- 725,729 ----
>              if section is not None and sect != section:
>                  continue
> !            output.write(self._options[sect, 
> opt.lower()].as_nice_string(sect))
>          return output.getvalue()
>
>
>
>
> _______________________________________________
> Spambayes-checkins mailing list
> Spambayes-checkins@python.org
> http://mail.python.org/mailman/listinfo/spambayes-checkins
>



-- 

Vous exprimer; Expr?sese; Te stesso esprimere; Express yourself!
Tim Stone
See my photography at www.fourstonesExpressions.com
See my writing at www.xanga.com/obj3kshun

From tim_one at users.sourceforge.net  Tue Dec 16 00:00:00 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Tue Dec 16 00:00:04 2003
Subject: [Spambayes-checkins] spambayes/spambayes classifier.py,1.13,1.14
Message-ID: <E1AW7JQ-0002sq-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv10943/spambayes

Modified Files:
	classifier.py 
Log Message:
_worddistanceget():  Don't do the expensive lookup of unknown_word_prob
unless it's actually needed.


Index: classifier.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/classifier.py,v
retrieving revision 1.13
retrieving revision 1.14
diff -C2 -d -r1.13 -r1.14
*** classifier.py	16 Dec 2003 02:46:42 -0000	1.13
--- classifier.py	16 Dec 2003 04:59:58 -0000	1.14
***************
*** 383,387 ****
              tokens = [(l[i],"%s %s" % (l[i],l[(i+1)%len(l)]),
                         l[(i+1)%len(l)]) for i in xrange(len(l))]
!             
              # Run through all the triplets and add the strongest
              # clue from each (note also the comment above explaining
--- 383,387 ----
              tokens = [(l[i],"%s %s" % (l[i],l[(i+1)%len(l)]),
                         l[(i+1)%len(l)]) for i in xrange(len(l))]
! 
              # Run through all the triplets and add the strongest
              # clue from each (note also the comment above explaining
***************
*** 409,416 ****
  
      def _worddistanceget(self, word):
-         unknown = options["Classifier", "unknown_word_prob"]
          record = self._wordinfoget(word)
          if record is None:
!             prob = unknown
          else:
              prob = self.probability(record)
--- 409,415 ----
  
      def _worddistanceget(self, word):
          record = self._wordinfoget(word)
          if record is None:
!             prob = options["Classifier", "unknown_word_prob"]
          else:
              prob = self.probability(record)
***************
*** 451,455 ****
              except StopIteration:
                  break
!             
      def _wordinfokeys(self):
          return self.wordinfo.keys()
--- 450,454 ----
              except StopIteration:
                  break
! 
      def _wordinfokeys(self):
          return self.wordinfo.keys()



From tim_one at users.sourceforge.net  Tue Dec 16 00:06:35 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Tue Dec 16 00:06:39 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000/dialogs/resources
	rclabels2text.py, 1.1, 1.2 rcparser.py, 1.10, 1.11
Message-ID: <E1AW7Pn-0003Fo-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000/dialogs/resources
In directory sc8-pr-cvs1:/tmp/cvs-serv12349/Outlook2000/dialogs/resources

Modified Files:
	rclabels2text.py rcparser.py 
Log Message:
Whitespace normalization.


Index: rclabels2text.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/dialogs/resources/rclabels2text.py,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** rclabels2text.py	20 Aug 2003 02:12:48 -0000	1.1
--- rclabels2text.py	16 Dec 2003 05:06:33 -0000	1.2
***************
*** 37,41 ****
                  out.write(s)
                  out.write("\n\n")
!             
      out.close()
      os.startfile(outputFilename);
--- 37,41 ----
                  out.write(s)
                  out.write("\n\n")
! 
      out.close()
      os.startfile(outputFilename);

Index: rcparser.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/dialogs/resources/rcparser.py,v
retrieving revision 1.10
retrieving revision 1.11
diff -C2 -d -r1.10 -r1.11
*** rcparser.py	26 Aug 2003 07:23:04 -0000	1.10
--- rcparser.py	16 Dec 2003 05:06:33 -0000	1.11
***************
*** 90,94 ****
          #print t
          return t
!         
  
  class RCParser:
--- 90,94 ----
          #print t
          return t
! 
  
  class RCParser:
***************
*** 99,103 ****
      token = ""
  
!     def __init__(self):    
          self.ids = {"IDOK":1, "IDCANCEL":2, "IDC_STATIC": -1}
          self.names = {1:"IDOK", 2:"IDCANCEL", -1:"IDC_STATIC"}
--- 99,103 ----
      token = ""
  
!     def __init__(self):
          self.ids = {"IDOK":1, "IDCANCEL":2, "IDC_STATIC": -1}
          self.names = {1:"IDOK", 2:"IDCANCEL", -1:"IDC_STATIC"}
***************
*** 114,122 ****
              self.token = None
          return self.token
!     
      def getCommaToken(self):
          tok = self.getToken()
          assert tok == ",", "Token '%s' should be a comma!" % tok
!     
      def loadDialogs(self, rcFileName):
          """
--- 114,122 ----
              self.token = None
          return self.token
! 
      def getCommaToken(self):
          tok = self.getToken()
          assert tok == ",", "Token '%s' should be a comma!" % tok
! 
      def loadDialogs(self, rcFileName):
          """
***************
*** 206,214 ****
              self.names[id] = id_name
          return id
!                 
      def lang(self):
          while self.token[0:4]=="LANG" or self.token[0:7]=="SUBLANG" or self.token==',':
              self.getToken();
!             
      def dialog(self, name):
          dlg = DialogDef(name,self.addId(name))
--- 206,214 ----
              self.names[id] = id_name
          return id
! 
      def lang(self):
          while self.token[0:4]=="LANG" or self.token[0:7]=="SUBLANG" or self.token==',':
              self.getToken();
! 
      def dialog(self, name):
          dlg = DialogDef(name,self.addId(name))
***************
*** 246,250 ****
                  break
          self.dialogs[name] = dlg.createDialogTemplate()
!     
      def dialogStyle(self, dlg):
          dlg.style, dlg.styles = self.styles( [], win32con.WS_VISIBLE | win32con.DS_SETFONT)
--- 246,250 ----
                  break
          self.dialogs[name] = dlg.createDialogTemplate()
! 
      def dialogStyle(self, dlg):
          dlg.style, dlg.styles = self.styles( [], win32con.WS_VISIBLE | win32con.DS_SETFONT)
***************
*** 252,260 ****
          self.getToken()
          dlg.styleEx, dlg.stylesEx = self.styles( [], 0)
!     
      def styles(self, defaults, defaultStyle):
          list = defaults
          style = defaultStyle
!         
          if "STYLE"==self.token:
              self.getToken()
--- 252,260 ----
          self.getToken()
          dlg.styleEx, dlg.stylesEx = self.styles( [], 0)
! 
      def styles(self, defaults, defaultStyle):
          list = defaults
          style = defaultStyle
! 
          if "STYLE"==self.token:
              self.getToken()
***************
*** 285,291 ****
              self.getToken()
          self.debug("style is ",style)
!             
          return style, list
!     
      def dialogCaption(self, dlg):
          if "CAPTION"==self.token:
--- 285,291 ----
              self.getToken()
          self.debug("style is ",style)
! 
          return style, list
! 
      def dialogCaption(self, dlg):
          if "CAPTION"==self.token:



From tim_one at users.sourceforge.net  Tue Dec 16 00:06:35 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Tue Dec 16 00:06:42 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000/dialogs
	FolderSelector.py, 1.31, 1.32 __init__.py, 1.11,
	1.12 async_processor.py, 1.8, 1.9 dialog_map.py, 1.35,
	1.36 dlgcore.py, 1.12, 1.13 opt_processors.py, 1.14,
	1.15 processors.py, 1.12, 1.13 test_dialogs.py, 1.9,
	1.10 wizard_processors.py, 1.9, 1.10
Message-ID: <E1AW7Pn-0003Fg-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000/dialogs
In directory sc8-pr-cvs1:/tmp/cvs-serv12349/Outlook2000/dialogs

Modified Files:
	FolderSelector.py __init__.py async_processor.py dialog_map.py 
	dlgcore.py opt_processors.py processors.py test_dialogs.py 
	wizard_processors.py 
Log Message:
Whitespace normalization.


Index: FolderSelector.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/dialogs/FolderSelector.py,v
retrieving revision 1.31
retrieving revision 1.32
diff -C2 -d -r1.31 -r1.32
*** FolderSelector.py	15 Oct 2003 04:12:25 -0000	1.31
--- FolderSelector.py	16 Dec 2003 05:06:33 -0000	1.32
***************
*** 218,222 ****
                        0, 0)
      return array.array("c", buf), extra
!     
  def UnpackTVItem(buffer):
      item_mask, item_hItem, item_state, item_stateMask, \
--- 218,222 ----
                        0, 0)
      return array.array("c", buf), extra
! 
  def UnpackTVItem(buffer):
      item_mask, item_hItem, item_state, item_stateMask, \
***************
*** 232,236 ****
      if not (item_mask & commctrl.TVIF_SELECTEDIMAGE): item_selimage = None
      if not (item_mask & commctrl.TVIF_STATE): item_state = item_stateMask = None
!     
      if item_textptr:
          text = win32gui.PyGetString(item_textptr)
--- 232,236 ----
      if not (item_mask & commctrl.TVIF_SELECTEDIMAGE): item_selimage = None
      if not (item_mask & commctrl.TVIF_STATE): item_state = item_stateMask = None
! 
      if item_textptr:
          text = win32gui.PyGetString(item_textptr)
***************
*** 448,452 ****
      # Message processing
  #    def GetMessageMap(self):
!     
      def OnInitDialog (self, hwnd, msg, wparam, lparam):
          FolderSelector_Parent.OnInitDialog(self, hwnd, msg, wparam, lparam)
--- 448,452 ----
      # Message processing
  #    def GetMessageMap(self):
! 
      def OnInitDialog (self, hwnd, msg, wparam, lparam):
          FolderSelector_Parent.OnInitDialog(self, hwnd, msg, wparam, lparam)
***************
*** 512,516 ****
          id_name = self._GetIDName(id)
          code = win32api.HIWORD(wparam)
!         
          if code == win32con.BN_CLICKED:
              if id in (win32con.IDOK, win32con.IDCANCEL) and self.in_label_edit:
--- 512,516 ----
          id_name = self._GetIDName(id)
          code = win32api.HIWORD(wparam)
! 
          if code == win32con.BN_CLICKED:
              if id in (win32con.IDOK, win32con.IDCANCEL) and self.in_label_edit:

Index: __init__.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/dialogs/__init__.py,v
retrieving revision 1.11
retrieving revision 1.12
diff -C2 -d -r1.11 -r1.12
*** __init__.py	5 Sep 2003 02:09:42 -0000	1.11
--- __init__.py	16 Dec 2003 05:06:33 -0000	1.12
***************
*** 55,59 ****
          except win32gui.error:
              pass
!         
      import dlgcore
      dlg = dlgcore.ProcessorDialog(parent, manager, config, idd, commands)
--- 55,59 ----
          except win32gui.error:
              pass
! 
      import dlgcore
      dlg = dlgcore.ProcessorDialog(parent, manager, config, idd, commands)
***************
*** 69,73 ****
          print "Cancelling wizard"
          config_wizard.CancelWizardConfig(manager, config)
!     
  def MakePropertyPage(parent, manager, config, idd, yoffset=24):
      """Creates a child dialog box to use as property page in a tab control"""
--- 69,73 ----
          print "Cancelling wizard"
          config_wizard.CancelWizardConfig(manager, config)
! 
  def MakePropertyPage(parent, manager, config, idd, yoffset=24):
      """Creates a child dialog box to use as property page in a tab control"""
***************
*** 78,82 ****
      if not parent:
          raise "Parent must be the tab control"
!         
      import dlgcore
      dlg = dlgcore.ProcessorPage(parent, manager, config, idd, commands, yoffset)
--- 78,82 ----
      if not parent:
          raise "Parent must be the tab control"
! 
      import dlgcore
      dlg = dlgcore.ProcessorPage(parent, manager, config, idd, commands, yoffset)

Index: async_processor.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/dialogs/async_processor.py,v
retrieving revision 1.8
retrieving revision 1.9
diff -C2 -d -r1.8 -r1.9
*** async_processor.py	7 Sep 2003 03:53:13 -0000	1.8
--- async_processor.py	16 Dec 2003 05:06:33 -0000	1.9
***************
*** 160,164 ****
          else:
              raise RuntimeError, "Not one of my messages??"
!     
      def OnFinished(self, wparam, lparam):
          self.seen_finished = True
--- 160,164 ----
          else:
              raise RuntimeError, "Not one of my messages??"
! 
      def OnFinished(self, wparam, lparam):
          self.seen_finished = True
***************
*** 177,181 ****
                                  win32con.WM_SETTEXT,
                                  0, text)
!         
      def OnProgressStatus(self, wparam, lparam):
          self.SetStatusText(self.progress_status)
--- 177,181 ----
                                  win32con.WM_SETTEXT,
                                  0, text)
! 
      def OnProgressStatus(self, wparam, lparam):
          self.SetStatusText(self.progress_status)
***************
*** 191,195 ****
      def OnProgressWarning(self, wparam, lparam):
          pass
!     
      def OnClicked(self, id):
          self.StartProcess()
--- 191,195 ----
      def OnProgressWarning(self, wparam, lparam):
          pass
! 
      def OnClicked(self, id):
          self.StartProcess()
***************
*** 289,292 ****
      for i in range(2):
          p.tick()
!     
      print "Done!"
--- 289,292 ----
      for i in range(2):
          p.tick()
! 
      print "Done!"

Index: dialog_map.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/dialogs/dialog_map.py,v
retrieving revision 1.35
retrieving revision 1.36
diff -C2 -d -r1.35 -r1.36
*** dialog_map.py	29 Sep 2003 02:14:26 -0000	1.35
--- dialog_map.py	16 Dec 2003 05:06:33 -0000	1.36
***************
*** 53,57 ****
                  big = "spam"
                  small = "ham"
!             if big is not None:                
                  db_status = "%s\nWarning: you have much more %s than %s - " \
                              "SpamBayes works best with approximately even " \
--- 53,57 ----
                  big = "spam"
                  small = "ham"
!             if big is not None:
                  db_status = "%s\nWarning: you have much more %s than %s - " \
                              "SpamBayes works best with approximately even " \
***************
*** 151,155 ****
          else:
              unsure_text = "unsure messages untouched"
!             
          watch_names = manager.FormatFolderNames(
                          config.watch_folder_ids, config.watch_include_sub)
--- 151,155 ----
          else:
              unsure_text = "unsure messages untouched"
! 
          watch_names = manager.FormatFolderNames(
                          config.watch_folder_ids, config.watch_include_sub)
***************
*** 165,174 ****
          ControlProcessor.__init__(self, window, control_ids)
          self.page_ids = page_ids.split()
!         
      def Init(self):
          self.pages = {}
          self.currentPage = None
          self.currentPageIndex = -1
!         self.currentPageHwnd = None        
          for index, page_id in enumerate(self.page_ids):
              template = self.window.manager.dialog_parser.dialogs[page_id]
--- 165,174 ----
          ControlProcessor.__init__(self, window, control_ids)
          self.page_ids = page_ids.split()
! 
      def Init(self):
          self.pages = {}
          self.currentPage = None
          self.currentPageIndex = -1
!         self.currentPageHwnd = None
          for index, page_id in enumerate(self.page_ids):
              template = self.window.manager.dialog_parser.dialogs[page_id]
***************
*** 183,187 ****
                  return False
          return True
!     
      def OnNotify(self, nmhdr, wparam, lparam):
          # this does not appear to be in commctrl module
--- 183,187 ----
                  return False
          return True
! 
      def OnNotify(self, nmhdr, wparam, lparam):
          # this does not appear to be in commctrl module
***************
*** 203,207 ****
          self.currentPageIndex = index
          return 0
!         
      def addPage(self, item, idName, label):
          format = "iiiiiii"
--- 203,207 ----
          self.currentPageIndex = index
          return 0
! 
      def addPage(self, item, idName, label):
          format = "iiiiiii"
***************
*** 209,213 ****
          address,l = win32gui.PyGetBufferAddressAndLen(lbuf)
          win32gui.PySetString(address, label)
!         
          buf = struct.pack(format,
              commctrl.TCIF_TEXT, # mask
--- 209,213 ----
          address,l = win32gui.PyGetBufferAddressAndLen(lbuf)
          win32gui.PySetString(address, label)
! 
          buf = struct.pack(format,
              commctrl.TCIF_TEXT, # mask
***************
*** 301,305 ****
          ShowDialog(parent, self.window.manager, self.window.config, self.idd)
          self.window.LoadAllControls()
!         
      def GetPopupHelpText(self, id):
          dd = self.window.manager.dialog_parser.dialogs[self.idd]
--- 301,305 ----
          ShowDialog(parent, self.window.manager, self.window.config, self.idd)
          self.window.LoadAllControls()
! 
      def GetPopupHelpText(self, id):
          dd = self.window.manager.dialog_parser.dialogs[self.idd]
***************
*** 342,346 ****
          # And show the wizard.
          ShowWizard(parent, manager, self.idd, use_existing_config = True)
!     
  def WizardFinish(mgr, window):
      print "Wizard Done!"
--- 342,346 ----
          # And show the wizard.
          ShowWizard(parent, manager, self.idd, use_existing_config = True)
! 
  def WizardFinish(mgr, window):
      print "Wizard Done!"
***************
*** 377,381 ****
      try:
          progress.tick()
!     
          if rescore:
              # Setup the "filter now" config to what we want.
--- 377,381 ----
      try:
          progress.tick()
! 
          if rescore:
              # Setup the "filter now" config to what we want.
***************
*** 390,394 ****
              import filter
              filter.filterer(mgr, config, progress)
!     
          bayes = classifier_data.bayes
          progress.set_status("Completed training with %d spam and %d good messages" \
--- 390,394 ----
              import filter
              filter.filterer(mgr, config, progress)
! 
          bayes = classifier_data.bayes
          progress.set_status("Completed training with %d spam and %d good messages" \
***************
*** 405,409 ****
          (CloseButtonProcessor,    "IDOK IDCANCEL"),
          (TabProcessor,            "IDC_TAB",
!                                   """IDD_GENERAL IDD_FILTER IDD_TRAINING 
                                    IDD_ADVANCED"""),
          (CommandButtonProcessor,  "IDC_ABOUT_BTN", ShowAbout, ()),
--- 405,409 ----
          (CloseButtonProcessor,    "IDOK IDCANCEL"),
          (TabProcessor,            "IDC_TAB",
!                                   """IDD_GENERAL IDD_FILTER IDD_TRAINING
                                    IDD_ADVANCED"""),
          (CommandButtonProcessor,  "IDC_ABOUT_BTN", ShowAbout, ()),
***************
*** 448,452 ****
          (EditNumberProcessor,     "IDC_EDIT_UNSURE IDC_SLIDER_UNSURE",
                                    "Filter.unsure_threshold"),
!         
          (ComboProcessor,          "IDC_ACTION_UNSURE", "Filter.unsure_action"),
          (BoolButtonProcessor,     "IDC_MARK_UNSURE_AS_READ",    "Filter.unsure_mark_as_read"),
--- 448,452 ----
          (EditNumberProcessor,     "IDC_EDIT_UNSURE IDC_SLIDER_UNSURE",
                                    "Filter.unsure_threshold"),
! 
          (ComboProcessor,          "IDC_ACTION_UNSURE", "Filter.unsure_action"),
          (BoolButtonProcessor,     "IDC_MARK_UNSURE_AS_READ",    "Filter.unsure_mark_as_read"),
***************
*** 473,477 ****
          (ComboProcessor,          "IDC_RECOVER_RS", "General.recover_from_spam_message_state",
           "not change the message,mark the message as read,mark the message as unread"),
!         
      ),
      "IDD_ADVANCED" : (
--- 473,477 ----
          (ComboProcessor,          "IDC_RECOVER_RS", "General.recover_from_spam_message_state",
           "not change the message,mark the message as read,mark the message as unread"),
! 
      ),
      "IDD_ADVANCED" : (
***************
*** 531,535 ****
                                      "Training.spam_folder_ids"),
          (BoolButtonProcessor,     "IDC_BUT_RESCORE",    "Training.rescore"),
!         
      ),
      "IDD_WIZARD_TRAIN" : (
--- 531,535 ----
                                      "Training.spam_folder_ids"),
          (BoolButtonProcessor,     "IDC_BUT_RESCORE",    "Training.rescore"),
! 
      ),
      "IDD_WIZARD_TRAIN" : (

Index: dlgcore.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/dialogs/dlgcore.py,v
retrieving revision 1.12
retrieving revision 1.13
diff -C2 -d -r1.12 -r1.13
*** dlgcore.py	27 Aug 2003 02:11:38 -0000	1.12
--- dlgcore.py	16 Dec 2003 05:06:33 -0000	1.13
***************
*** 22,26 ****
          if self.hwnd_tooltip is not None:
              win32gui.SendMessage(self.hwnd_tooltip, commctrl.TTM_TRACKACTIVATE, 0, 0)
!             
      def ShowTooltipForControl(self, control_id, text):
          # Note sure this tooltip stuff is quite right!
--- 22,26 ----
          if self.hwnd_tooltip is not None:
              win32gui.SendMessage(self.hwnd_tooltip, commctrl.TTM_TRACKACTIVATE, 0, 0)
! 
      def ShowTooltipForControl(self, control_id, text):
          # Note sure this tooltip stuff is quite right!
***************
*** 42,46 ****
                                   commctrl.TTM_SETMAXTIPWIDTH,
                                   0, 300)
!         
          format = "iiiiiiiiiii"
          tt_size = struct.calcsize(format)
--- 42,46 ----
                                   commctrl.TTM_SETMAXTIPWIDTH,
                                   0, 300)
! 
          format = "iiiiiiiiiii"
          tt_size = struct.calcsize(format)
***************
*** 50,54 ****
          flags = commctrl.TTF_TRACK | commctrl.TTF_ABSOLUTE
          data = struct.pack(format, tt_size, flags, hwnd_dialog, uID, 0,0,0,0, 0, text_address, 0)
!         
          # Add a tool for this control only if we haven't already
          if control_id not in self.tooltip_tools:
--- 50,54 ----
          flags = commctrl.TTF_TRACK | commctrl.TTF_ABSOLUTE
          data = struct.pack(format, tt_size, flags, hwnd_dialog, uID, 0,0,0,0, 0, text_address, 0)
! 
          # Add a tool for this control only if we haven't already
          if control_id not in self.tooltip_tools:
***************
*** 109,113 ****
          centre_x, centre_y = win32gui.ClientToScreen( desktop, ( (dt_r-dt_l)/2, (dt_b-dt_t)/2) )
          win32gui.MoveWindow(self.hwnd, centre_x-(w/2), centre_y-(h/2), w, h, 0)
!         
      def OnInitDialog(self, hwnd, msg, wparam, lparam):
          self.hwnd = hwnd
--- 109,113 ----
          centre_x, centre_y = win32gui.ClientToScreen( desktop, ( (dt_r-dt_l)/2, (dt_b-dt_t)/2) )
          win32gui.MoveWindow(self.hwnd, centre_x-(w/2), centre_y-(h/2), w, h, 0)
! 
      def OnInitDialog(self, hwnd, msg, wparam, lparam):
          self.hwnd = hwnd
***************
*** 160,164 ****
      def OnDestroy(self, hwnd, msg, wparam, lparam):
          self.tt.HideTooltip()
!         
      def OnHelp(self, hwnd, msg, wparam, lparam):
          format = "iiiiiii"
--- 160,164 ----
      def OnDestroy(self, hwnd, msg, wparam, lparam):
          self.tt.HideTooltip()
! 
      def OnHelp(self, hwnd, msg, wparam, lparam):
          format = "iiiiiii"
***************
*** 226,230 ****
          if cp is not None:
              return cp.GetPopupHelpText(iCtrlId)
!         
          print "Can not get command processor for", self._GetIDName(iCtrlId)
          return None
--- 226,230 ----
          if cp is not None:
              return cp.GetPopupHelpText(iCtrlId)
! 
          print "Can not get command processor for", self._GetIDName(iCtrlId)
          return None
***************
*** 232,236 ****
          for cp in self.command_processors.values():
              cp.OnRButtonUp(wparam,lparam)
!             
      def OnCommandProcessorMessage(self, hwnd, msg, wparam, lparam):
          for p in self.processor_message_map[msg]:
--- 232,236 ----
          for cp in self.command_processors.values():
              cp.OnRButtonUp(wparam,lparam)
! 
      def OnCommandProcessorMessage(self, hwnd, msg, wparam, lparam):
          for p in self.processor_message_map[msg]:
***************
*** 265,269 ****
              win32gui.MessageBox(self.hwnd, str(why), "SpamBayes", mb_flags)
              return False
!         
      def SaveAllControls(self):
          for p in self.all_processors:
--- 265,269 ----
              win32gui.MessageBox(self.hwnd, str(why), "SpamBayes", mb_flags)
              return False
! 
      def SaveAllControls(self):
          for p in self.all_processors:
***************
*** 324,326 ****
          self.template[0][2] = self.template[0][2] | win32con.WS_CHILD
          return win32gui.CreateDialogIndirect(self.hinst, self.template, self.parent, message_map)
- 
--- 324,325 ----

Index: opt_processors.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/dialogs/opt_processors.py,v
retrieving revision 1.14
retrieving revision 1.15
diff -C2 -d -r1.14 -r1.15
*** opt_processors.py	9 Sep 2003 01:13:07 -0000	1.14
--- opt_processors.py	16 Dec 2003 05:06:33 -0000	1.15
***************
*** 1,4 ****
  # Option Control Processors for our dialog.
! # These are extensions to basic Control Processors that are linked with 
  # SpamBayes options.
  
--- 1,4 ----
  # Option Control Processors for our dialog.
! # These are extensions to basic Control Processors that are linked with
  # SpamBayes options.
  
***************
*** 128,132 ****
          else:
              assert 0, "Couldn't find a checked button"
!     
  # A "Combo" processor, that loads valid strings from the option.
  class ComboProcessor(OptionControlProcessor):
--- 128,132 ----
          else:
              assert 0, "Couldn't find a checked button"
! 
  # A "Combo" processor, that loads valid strings from the option.
  class ComboProcessor(OptionControlProcessor):
***************
*** 140,144 ****
              self.option_to_text = zip(self.option.valid_input(),self.option.valid_input())
              self.text_to_option = dict(self.option_to_text)
!         
      def OnCommand(self, wparam, lparam):
          code = win32api.HIWORD(wparam)
--- 140,144 ----
              self.option_to_text = zip(self.option.valid_input(),self.option.valid_input())
              self.text_to_option = dict(self.option_to_text)
! 
      def OnCommand(self, wparam, lparam):
          code = win32api.HIWORD(wparam)
***************
*** 180,184 ****
                     "automatically adjust"
          return OptionControlProcessor.GetPopupHelpText(self, id)
!                       
      def GetMessages(self):
          return [win32con.WM_HSCROLL]
--- 180,184 ----
                     "automatically adjust"
          return OptionControlProcessor.GetPopupHelpText(self, id)
! 
      def GetMessages(self):
          return [win32con.WM_HSCROLL]
***************
*** 202,206 ****
                  # They are typing - value may be currently invalid
                  pass
!         
      def Init(self):
          OptionControlProcessor.Init(self)
--- 202,206 ----
                  # They are typing - value may be currently invalid
                  pass
! 
      def Init(self):
          OptionControlProcessor.Init(self)
***************
*** 249,253 ****
              raise ValueError, "Value must be between %d and %d" % (self.min_val, self.max_val)
          self.SetOptionValue(val)
!     
  # Folder IDs, and the "include_sub" option, if applicable.
  class FolderIDProcessor(OptionControlProcessor):
--- 249,253 ----
              raise ValueError, "Value must be between %d and %d" % (self.min_val, self.max_val)
          self.SetOptionValue(val)
! 
  # Folder IDs, and the "include_sub" option, if applicable.
  class FolderIDProcessor(OptionControlProcessor):

Index: processors.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/dialogs/processors.py,v
retrieving revision 1.12
retrieving revision 1.13
diff -C2 -d -r1.12 -r1.13
*** processors.py	5 Sep 2003 06:54:23 -0000	1.12
--- processors.py	16 Dec 2003 05:06:33 -0000	1.13
***************
*** 68,72 ****
      def GetPopupHelpText(self, cid):
          return None
!     
  class ButtonProcessor(ControlProcessor):
      def OnCommand(self, wparam, lparam):
--- 68,72 ----
      def GetPopupHelpText(self, cid):
          return None
! 
  class ButtonProcessor(ControlProcessor):
      def OnCommand(self, wparam, lparam):
***************
*** 93,97 ****
          args = (self.window,) + self.args
          self.func(*args)
!     
      def GetPopupHelpText(self, ctrlid):
          assert ctrlid == self.control_id
--- 93,97 ----
          args = (self.window,) + self.args
          self.func(*args)
! 
      def GetPopupHelpText(self, ctrlid):
          assert ctrlid == self.control_id

Index: test_dialogs.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/dialogs/test_dialogs.py,v
retrieving revision 1.9
retrieving revision 1.10
diff -C2 -d -r1.9 -r1.10
*** test_dialogs.py	26 Aug 2003 02:27:26 -0000	1.9
--- test_dialogs.py	16 Dec 2003 05:06:33 -0000	1.10
***************
*** 11,15 ****
      except ImportError:
          sys.path.append(os.path.abspath(os.path.join(os.path.dirname(sys.argv[0]), "..")))
!     
      import manager, win32con
      mgr = manager.GetManager()
--- 11,15 ----
      except ImportError:
          sys.path.append(os.path.abspath(os.path.join(os.path.dirname(sys.argv[0]), "..")))
! 
      import manager, win32con
      mgr = manager.GetManager()

Index: wizard_processors.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/dialogs/wizard_processors.py,v
retrieving revision 1.9
retrieving revision 1.10
diff -C2 -d -r1.9 -r1.10
*** wizard_processors.py	26 Aug 2003 04:52:06 -0000	1.9
--- wizard_processors.py	16 Dec 2003 05:06:33 -0000	1.10
***************
*** 88,92 ****
                  except:
                      pass
!                         
                  self.finish_fn(self.window.manager, self.window)
                  win32gui.EndDialog(self.window.hwnd, win32con.IDOK)
--- 88,92 ----
                  except:
                      pass
! 
                  self.finish_fn(self.window.manager, self.window)
                  win32gui.EndDialog(self.window.hwnd, win32con.IDOK)
***************
*** 103,107 ****
              print "Back button switching to page", pageNo
              self.switchToPage(pageNo)
!     
      def switchToPage(self, index):
          if self.currentPageHwnd is not None:
--- 103,107 ----
              print "Back button switching to page", pageNo
              self.switchToPage(pageNo)
! 
      def switchToPage(self, index):
          if self.currentPageHwnd is not None:
***************
*** 129,133 ****
                  return index
          assert 0, "No page '%s'" % next
!         
      # methods to be overridden.  default implementation is simple sequential
      def getNextPage(self):
--- 129,133 ----
                  return index
          assert 0, "No page '%s'" % next
! 
      # methods to be overridden.  default implementation is simple sequential
      def getNextPage(self):
***************
*** 168,172 ****
              ok = not self.window.config.wizard.need_train
          return ok
!         
      def getNextPage(self):
          index = self.currentPageIndex
--- 168,172 ----
              ok = not self.window.config.wizard.need_train
          return ok
! 
      def getNextPage(self):
          index = self.currentPageIndex
***************
*** 221,227 ****
          self.name_joiner = name_joiner
          self.in_setting_name = False
!        
          name_sect_name, name_sub_option_name = option_folder_name.split(".")
!         
          self.option_folder_name = window.config.get_option(name_sect_name,
                                                              name_sub_option_name)
--- 221,227 ----
          self.name_joiner = name_joiner
          self.in_setting_name = False
! 
          name_sect_name, name_sub_option_name = option_folder_name.split(".")
! 
          self.option_folder_name = window.config.get_option(name_sect_name,
                                                              name_sub_option_name)



From tim_one at users.sourceforge.net  Tue Dec 16 00:06:35 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Tue Dec 16 00:06:43 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 addin.py, 1.116,
	1.117 config.py, 1.27, 1.28 config_wizard.py, 1.8,
	1.9 manager.py, 1.93, 1.94 msgstore.py, 1.79, 1.80 oastats.py,
	1.2, 1.3 tester.py, 1.20, 1.21 train.py, 1.34, 1.35
Message-ID: <E1AW7Pn-0003Fa-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv12349/Outlook2000

Modified Files:
	addin.py config.py config_wizard.py manager.py msgstore.py 
	oastats.py tester.py train.py 
Log Message:
Whitespace normalization.


Index: addin.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/addin.py,v
retrieving revision 1.116
retrieving revision 1.117
diff -C2 -d -r1.116 -r1.117
*** addin.py	10 Dec 2003 05:28:10 -0000	1.116
--- addin.py	16 Dec 2003 05:06:32 -0000	1.117
***************
*** 1444,1448 ****
          _DoRegister(klass, _winreg.HKEY_LOCAL_MACHINE)
          print "Registration (in HKEY_LOCAL_MACHINE) complete."
!     
  def DllRegisterServer():
      klass = OutlookAddin
--- 1444,1448 ----
          _DoRegister(klass, _winreg.HKEY_LOCAL_MACHINE)
          print "Registration (in HKEY_LOCAL_MACHINE) complete."
! 
  def DllRegisterServer():
      klass = OutlookAddin

Index: config.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/config.py,v
retrieving revision 1.27
retrieving revision 1.28
diff -C2 -d -r1.27 -r1.28
*** config.py	26 Aug 2003 02:35:45 -0000	1.27
--- config.py	16 Dec 2003 05:06:33 -0000	1.28
***************
*** 77,81 ****
              return ""
          return str(self.value)
!     
      def multiple_values_allowed(self):
          return type(self.value)==types.ListType
--- 77,81 ----
              return ""
          return str(self.value)
! 
      def multiple_values_allowed(self):
          return type(self.value)==types.ListType
***************
*** 240,244 ****
          "Should the timer only be used for 'Inbox' type folders?", True,
          """The point of using a timer is to prevent the SpamBayes filter
!         getting in the way the builtin Outlook rules.  Therefore, is it 
          generally only necessary to use a timer for folders that have new
          items being delivered directly to them.  Folders that are not inbox
--- 240,244 ----
          "Should the timer only be used for 'Inbox' type folders?", True,
          """The point of using a timer is to prevent the SpamBayes filter
!         getting in the way the builtin Outlook rules.  Therefore, is it
          generally only necessary to use a timer for folders that have new
          items being delivered directly to them.  Folders that are not inbox
***************
*** 386,390 ****
      c.filter.unsure_folder_id = None
      if c.filter.unsure_folder_id is not None: raise ValueError, "unsure_id wrong (%r)" % (c.filter.unsure_folder_id,)
!     
      options.set("Filter", "filter_now", True)
      print "Filter_now from container is", c.filter.filter_now
--- 386,390 ----
      c.filter.unsure_folder_id = None
      if c.filter.unsure_folder_id is not None: raise ValueError, "unsure_id wrong (%r)" % (c.filter.unsure_folder_id,)
! 
      options.set("Filter", "filter_now", True)
      print "Filter_now from container is", c.filter.filter_now
***************
*** 405,407 ****
      options.update_file("delme.cfg")
      print "Created 'delme.cfg'"
- 
--- 405,406 ----

Index: config_wizard.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/config_wizard.py,v
retrieving revision 1.8
retrieving revision 1.9
diff -C2 -d -r1.8 -r1.9
*** config_wizard.py	15 Oct 2003 04:11:12 -0000	1.8
--- config_wizard.py	16 Dec 2003 05:06:33 -0000	1.9
***************
*** 38,42 ****
      new_config.filter.watch_folder_ids = []
      new_config.filter.watch_include_sub = False
!     
      wc = new_config.wizard
      if from_existing:
--- 38,42 ----
      new_config.filter.watch_folder_ids = []
      new_config.filter.watch_include_sub = False
! 
      wc = new_config.wizard
      if from_existing:
***************
*** 87,91 ****
          manager.ReportError(msg)
          return None
!     
  def CommitWizardConfig(manager, wc):
      # If the user want to manually configure, then don't do anything
--- 87,91 ----
          manager.ReportError(msg)
          return None
! 
  def CommitWizardConfig(manager, wc):
      # If the user want to manually configure, then don't do anything
***************
*** 150,152 ****
      InitWizardConfig(manager, cfg, from_existing)
      return cfg
-  
\ No newline at end of file
--- 150,151 ----

Index: manager.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/manager.py,v
retrieving revision 1.93
retrieving revision 1.94
diff -C2 -d -r1.93 -r1.94
*** manager.py	15 Dec 2003 09:13:40 -0000	1.93
--- manager.py	16 Dec 2003 05:06:33 -0000	1.94
***************
*** 504,508 ****
              # Can't make the directory.
              return self.application_directory
!             
      def MigrateDataDirectory(self):
          # A bit of a nod to save people doing a full retrain.
--- 504,508 ----
              # Can't make the directory.
              return self.application_directory
! 
      def MigrateDataDirectory(self):
          # A bit of a nod to save people doing a full retrain.
***************
*** 511,515 ****
          # Note that this is migrating data for very old versions of the
          # plugin (before the first decent binary!).  The next time it is
!         # touched it can die :) 
          self._MigrateFile("default_bayes_database.pck")
          self._MigrateFile("default_bayes_database.db")
--- 511,515 ----
          # Note that this is migrating data for very old versions of the
          # plugin (before the first decent binary!).  The next time it is
!         # touched it can die :)
          self._MigrateFile("default_bayes_database.pck")
          self._MigrateFile("default_bayes_database.db")
***************
*** 552,556 ****
          # Using MAPI to set them directly on the folder also has no effect.
          # Later: We have since discovered that Outlook stores user property
!         # information in the 'associated contents' folder - see 
          # msgstore.MAPIMsgStoreFolder.DoesFolderHaveOutlookField() for more
          # details.  We can reverse engineer this well enough to determine
--- 552,556 ----
          # Using MAPI to set them directly on the folder also has no effect.
          # Later: We have since discovered that Outlook stores user property
!         # information in the 'associated contents' folder - see
          # msgstore.MAPIMsgStoreFolder.DoesFolderHaveOutlookField() for more
          # details.  We can reverse engineer this well enough to determine

Index: msgstore.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/msgstore.py,v
retrieving revision 1.79
retrieving revision 1.80
diff -C2 -d -r1.79 -r1.80
*** msgstore.py	10 Dec 2003 07:18:53 -0000	1.79
--- msgstore.py	16 Dec 2003 05:06:33 -0000	1.80
***************
*** 136,140 ****
      exc_val = NormalizeCOMException(exc_val)
      return exc_val[0] in known_failure_codes
!     
  def ReportMAPIError(manager, what, exc_val):
      hr, exc_msg, exc, arg_err = exc_val
--- 136,140 ----
      exc_val = NormalizeCOMException(exc_val)
      return exc_val[0] in known_failure_codes
! 
  def ReportMAPIError(manager, what, exc_val):
      hr, exc_msg, exc, arg_err = exc_val
***************
*** 160,164 ****
      ReadOnlyException = ReadOnlyException
      ObjectChangedException = ObjectChangedException
!     
      def __init__(self, outlook = None):
          self.outlook = outlook
--- 160,164 ----
      ReadOnlyException = ReadOnlyException
      ObjectChangedException = ObjectChangedException
! 
      def __init__(self, outlook = None):
          self.outlook = outlook
***************
*** 686,690 ****
          except pythoncom.com_error, details:
              raise MsgStoreExceptionFromCOMException(details)
!     
      def CreateTemporaryMessage(self, msg_flags = None):
          # Create a message designed to be used temporarily.  It is your
--- 686,690 ----
          except pythoncom.com_error, details:
              raise MsgStoreExceptionFromCOMException(details)
! 
      def CreateTemporaryMessage(self, msg_flags = None):
          # Create a message designed to be used temporarily.  It is your
***************
*** 720,724 ****
                            PR_SUBJECT_A,
                            PR_TRANSPORT_MESSAGE_HEADERS_A,
!                           ) 
  
      def __init__(self, msgstore, prop_row):
--- 720,724 ----
                            PR_SUBJECT_A,
                            PR_TRANSPORT_MESSAGE_HEADERS_A,
!                           )
  
      def __init__(self, msgstore, prop_row):
***************
*** 1195,1199 ****
              prop_ids = PROP_TAG( PT_BINARY, PROP_ID(resolve_ids[0])), \
                         PROP_TAG( PT_BINARY, PROP_ID(resolve_ids[1]))
!     
              prop_tuples = (prop_ids[0],folder.id[0]), (prop_ids[1],folder.id[1])
              self.mapi_object.SetProps(prop_tuples)
--- 1195,1199 ----
              prop_ids = PROP_TAG( PT_BINARY, PROP_ID(resolve_ids[0])), \
                         PROP_TAG( PT_BINARY, PROP_ID(resolve_ids[1]))
! 
              prop_tuples = (prop_ids[0],folder.id[0]), (prop_ids[1],folder.id[1])
              self.mapi_object.SetProps(prop_tuples)

Index: oastats.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/oastats.py,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** oastats.py	30 Sep 2003 14:37:44 -0000	1.2
--- oastats.py	16 Dec 2003 05:06:33 -0000	1.3
***************
*** 74,78 ****
      s.RecordClassification(.2)
      print "\n".join(s.GetStats())
!     
      s = Stats(Config())
      s.RecordClassification(.2)
--- 74,78 ----
      s.RecordClassification(.2)
      print "\n".join(s.GetStats())
! 
      s = Stats(Config())
      s.RecordClassification(.2)

Index: tester.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/tester.py,v
retrieving revision 1.20
retrieving revision 1.21
diff -C2 -d -r1.20 -r1.21
*** tester.py	8 Sep 2003 07:51:51 -0000	1.20
--- tester.py	16 Dec 2003 05:06:33 -0000	1.21
***************
*** 167,171 ****
              ms_folder = self.manager.message_store.GetFolder(folder)
              TestFailed("The test message remained in folder '%s'" % ms_folder.GetFQName())
!         
      def _CleanTestMessageFromFolder(self, folder):
          subject = TEST_SUBJECT
--- 167,171 ----
              ms_folder = self.manager.message_store.GetFolder(folder)
              TestFailed("The test message remained in folder '%s'" % ms_folder.GetFQName())
! 
      def _CleanTestMessageFromFolder(self, folder):
          subject = TEST_SUBJECT
***************
*** 253,257 ****
              TestFailed("Something caused a new ham message to appear")
          check_words(words, bayes, 0, 0)
!     
          # Now move the message back to the inbox - it should get trained.
          store_msg = driver.manager.message_store.GetMessage(spam_msg)
--- 253,257 ----
              TestFailed("Something caused a new ham message to appear")
          check_words(words, bayes, 0, 0)
! 
          # Now move the message back to the inbox - it should get trained.
          store_msg = driver.manager.message_store.GetMessage(spam_msg)
***************
*** 308,315 ****
              if not train.been_trained_as_spam(store_msg, driver.manager.classifier_data):
                  TestFailed("Message was not identified as Spam after moving")
!     
              # word infos still be 'spam'
              check_words(words, bayes, 1, 0)
!     
              # Now undo the damage we did.
              was_spam = train.untrain_message(store_msg, driver.manager.classifier_data)
--- 308,315 ----
              if not train.been_trained_as_spam(store_msg, driver.manager.classifier_data):
                  TestFailed("Message was not identified as Spam after moving")
! 
              # word infos still be 'spam'
              check_words(words, bayes, 1, 0)
! 
              # Now undo the damage we did.
              was_spam = train.untrain_message(store_msg, driver.manager.classifier_data)
***************
*** 323,327 ****
              if need_untrain:
                  train.untrain_message(store_msg, driver.manager.classifier_data)
!     
          # Check all the counts are back where we started.
          if nspam != bayes.nspam:
--- 323,327 ----
              if need_untrain:
                  train.untrain_message(store_msg, driver.manager.classifier_data)
! 
          # Check all the counts are back where we started.
          if nspam != bayes.nspam:
***************
*** 330,339 ****
              TestFailed("Ham count didn't get back to the same")
          check_words(words, bayes, 0, 0)
!     
          if bayes.wordinfo != original_bayes.wordinfo:
              TestFailed("The bayes object's 'wordinfo' did not compare the same at the end of all this!")
          if bayes.probcache != original_bayes.probcache:
              TestFailed("The bayes object's 'probcache' did not compare the same at the end of all this!")
!     
          spam_msg.Delete()
      print "Created a Spam message, and saw it get filtered and trained."
--- 330,339 ----
              TestFailed("Ham count didn't get back to the same")
          check_words(words, bayes, 0, 0)
! 
          if bayes.wordinfo != original_bayes.wordinfo:
              TestFailed("The bayes object's 'wordinfo' did not compare the same at the end of all this!")
          if bayes.probcache != original_bayes.probcache:
              TestFailed("The bayes object's 'probcache' did not compare the same at the end of all this!")
! 
          spam_msg.Delete()
      print "Created a Spam message, and saw it get filtered and trained."
***************
*** 441,445 ****
                            },
                            run_tests, manager)
!     
      apply_with_new_config(manager,
                            {"Filter.timer_enabled": True,
--- 441,445 ----
                            },
                            run_tests, manager)
! 
      apply_with_new_config(manager,
                            {"Filter.timer_enabled": True,
***************
*** 452,456 ****
                            },
                            run_tests, manager)
!     
      apply_with_new_config(manager,
                            {"Filter.timer_enabled": True,
--- 452,456 ----
                            },
                            run_tests, manager)
! 
      apply_with_new_config(manager,
                            {"Filter.timer_enabled": True,
***************
*** 476,480 ****
          for (sect_name, opt_name), val in old_config.items():
              manager.options.set(sect_name, opt_name, val)
!     
  ###############################################################################
  # "Non-filter" tests are those that don't require us to create messages and
--- 476,480 ----
          for (sect_name, opt_name), val in old_config.items():
              manager.options.set(sect_name, opt_name, val)
! 
  ###############################################################################
  # "Non-filter" tests are those that don't require us to create messages and
***************
*** 547,551 ****
              raise TestFailed("Couldn't find unknown folder in names '%s'" % names)
      print "Finished 'invalid ID' tests"
!     
  ###############################################################################
  # "Failure" tests - execute some tests while provoking the msgstore to simulate
--- 547,551 ----
              raise TestFailed("Couldn't find unknown folder in names '%s'" % names)
      print "Finished 'invalid ID' tests"
! 
  ###############################################################################
  # "Failure" tests - execute some tests while provoking the msgstore to simulate
***************
*** 555,559 ****
      msgstore.test_suite_failure = None
      msgstore.test_suite_failure_request = None
!     
  def _setup_for_mapi_failure(checkpoint, hr):
      assert msgstore.test_suite_running, "msgstore should already know its running"
--- 555,559 ----
      msgstore.test_suite_failure = None
      msgstore.test_suite_failure_request = None
! 
  def _setup_for_mapi_failure(checkpoint, hr):
      assert msgstore.test_suite_running, "msgstore should already know its running"
***************
*** 612,616 ****
          # SetReadState???
          _do_single_failure_spam_test(driver, "MAPIMsgStoreMsg._DoCopyMove", mapi.MAPI_E_TABLE_TOO_BIG)
!        
      finally:
          manager.verbose = old_verbose
--- 612,616 ----
          # SetReadState???
          _do_single_failure_spam_test(driver, "MAPIMsgStoreMsg._DoCopyMove", mapi.MAPI_E_TABLE_TOO_BIG)
! 
      finally:
          manager.verbose = old_verbose
***************
*** 626,630 ****
                            },
                            do_failure_tests, manager)
!     
  def filter_message_with_event(msg, mgr, all_actions=True):
      import filter
--- 626,630 ----
                            },
                            do_failure_tests, manager)
! 
  def filter_message_with_event(msg, mgr, all_actions=True):
      import filter
***************
*** 642,646 ****
          filter._original_filter_message = filter.filter_message
          filter.filter_message = filter_message_with_event
!     
      try: # restore the plugin config at exit.
          assert not msgstore.test_suite_running, "already running??"
--- 642,646 ----
          filter._original_filter_message = filter.filter_message
          filter.filter_message = filter_message_with_event
! 
      try: # restore the plugin config at exit.
          assert not msgstore.test_suite_running, "already running??"

Index: train.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/train.py,v
retrieving revision 1.34
retrieving revision 1.35
diff -C2 -d -r1.34 -r1.35
*** train.py	19 Sep 2003 04:03:38 -0000	1.34
--- train.py	16 Dec 2003 05:06:33 -0000	1.35
***************
*** 161,168 ****
  
      real_trainer(classifier_data, config, mgr.message_store, progress)
!     
      if progress.stop_requested():
          return
!     
      if rebuild:
          assert mgr.classifier_data is not classifier_data
--- 161,168 ----
  
      real_trainer(classifier_data, config, mgr.message_store, progress)
! 
      if progress.stop_requested():
          return
! 
      if rebuild:
          assert mgr.classifier_data is not classifier_data



From tim_one at users.sourceforge.net  Tue Dec 16 00:06:36 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Tue Dec 16 00:06:45 2003
Subject: [Spambayes-checkins] spambayes/utilities which_database.py,1.6,1.7
Message-ID: <E1AW7Po-0003H5-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/utilities
In directory sc8-pr-cvs1:/tmp/cvs-serv12349/utilities

Modified Files:
	which_database.py 
Log Message:
Whitespace normalization.


Index: which_database.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/utilities/which_database.py,v
retrieving revision 1.6
retrieving revision 1.7
diff -C2 -d -r1.6 -r1.7
*** which_database.py	14 Oct 2003 19:24:06 -0000	1.6
--- which_database.py	16 Dec 2003 05:06:34 -0000	1.7
***************
*** 76,81 ****
      use_dbm = options["Storage", "persistent_use_database"]
      if not use_dbm:
!             print "Your storage %s is a: pickle" % (hammie,)
!             return
  
      if not os.path.exists(hammie):
--- 76,81 ----
      use_dbm = options["Storage", "persistent_use_database"]
      if not use_dbm:
!         print "Your storage %s is a: pickle" % (hammie,)
!         return
  
      if not os.path.exists(hammie):



From tim_one at users.sourceforge.net  Tue Dec 16 00:06:36 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Tue Dec 16 00:06:46 2003
Subject: [Spambayes-checkins] spambayes/testtools incremental.py, 1.5,
	1.6 mkgraph.py, 1.2, 1.3 regimes.py, 1.2, 1.3 urlslurper.py,
	1.5, 1.6
Message-ID: <E1AW7Po-0003Gv-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/testtools
In directory sc8-pr-cvs1:/tmp/cvs-serv12349/testtools

Modified Files:
	incremental.py mkgraph.py regimes.py urlslurper.py 
Log Message:
Whitespace normalization.


Index: incremental.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/testtools/incremental.py,v
retrieving revision 1.5
retrieving revision 1.6
diff -C2 -d -r1.5 -r1.6
*** incremental.py	5 Sep 2003 01:15:29 -0000	1.5
--- incremental.py	16 Dec 2003 05:06:34 -0000	1.6
***************
*** 333,339 ****
          set = int(set[3:]) - 1
          isspam = (dir.find('Spam') >= 0)
!  
          msg = msgs.Msg(dir, base)
!         
          for j in range(0, nsets):
              if which is not None and j != which:
--- 333,339 ----
          set = int(set[3:]) - 1
          isspam = (dir.find('Spam') >= 0)
! 
          msg = msgs.Msg(dir, base)
! 
          for j in range(0, nsets):
              if which is not None and j != which:
***************
*** 342,346 ****
                  sys.stderr.write("%-78s\r" % ("%s  : %d" % (base, set)))
                  sys.stderr.flush()
!  
                  nham_tested[j].append(tests[j].nham_tested)
                  nham_trained[j].append(tests[j].nham_trained)
--- 342,346 ----
                  sys.stderr.write("%-78s\r" % ("%s  : %d" % (base, set)))
                  sys.stderr.flush()
! 
                  nham_tested[j].append(tests[j].nham_tested)
                  nham_trained[j].append(tests[j].nham_trained)
***************
*** 355,359 ****
                  # tests[j].reset_test_results()
                  rules[j].group_action(j, tests[j])
!  
              if j != set:
                  guess = tests[j].predict([msg], isspam)
--- 355,359 ----
                  # tests[j].reset_test_results()
                  rules[j].group_action(j, tests[j])
! 
              if j != set:
                  guess = tests[j].predict([msg], isspam)
***************
*** 367,371 ****
                  elif todo == 1:
                      tests[j].train([msg], None)
!  
          oldgroup = group
  
--- 367,371 ----
                  elif todo == 1:
                      tests[j].train([msg], None)
! 
          oldgroup = group
  

Index: mkgraph.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/testtools/mkgraph.py,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** mkgraph.py	1 Mar 2003 05:13:08 -0000	1.2
--- mkgraph.py	16 Dec 2003 05:06:34 -0000	1.3
***************
*** 72,77 ****
          print '% linetype=1 linelabel="spam_unsure" markertype=0 linecolor=9'
          line(nspam_unsure)
!    
!     if report == "error": 
          print '$ Data=Curve2d'
          print '% toplabel="%s Error Rates"' % (title)
--- 72,77 ----
          print '% linetype=1 linelabel="spam_unsure" markertype=0 linecolor=9'
          line(nspam_unsure)
! 
!     if report == "error":
          print '$ Data=Curve2d'
          print '% toplabel="%s Error Rates"' % (title)
***************
*** 171,173 ****
  if __name__ == "__main__":
      main()
- 
--- 171,172 ----

Index: regimes.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/testtools/regimes.py,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** regimes.py	8 Mar 2003 00:33:22 -0000	1.2
--- regimes.py	16 Dec 2003 05:06:34 -0000	1.3
***************
*** 109,111 ****
              self.ham[0].append(msg)
          return actual
- 
--- 109,110 ----

Index: urlslurper.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/testtools/urlslurper.py,v
retrieving revision 1.5
retrieving revision 1.6
diff -C2 -d -r1.5 -r1.6
*** urlslurper.py	15 May 2003 00:35:38 -0000	1.5
--- urlslurper.py	16 Dec 2003 05:06:34 -0000	1.6
***************
*** 225,233 ****
          # Build a new opener without any proxy information.
          opener = urllib2.build_opener(urllib2.HTTPHandler)
!      
      # install it
      urllib2.install_opener(opener)
  
!     # read any url cache    
      if os.path.exists(filename):
          f = file(filename, "r")
--- 225,233 ----
          # Build a new opener without any proxy information.
          opener = urllib2.build_opener(urllib2.HTTPHandler)
! 
      # install it
      urllib2.install_opener(opener)
  
!     # read any url cache
      if os.path.exists(filename):
          f = file(filename, "r")



From tim_one at users.sourceforge.net  Tue Dec 16 00:06:36 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Tue Dec 16 00:06:54 2003
Subject: [Spambayes-checkins] website/scripts/ht2html LinkFixer.py, 1.3,
	1.4 PDOGenerator.py, 1.1.1.1, 1.2 Sidebar.py, 1.3,
	1.4 SpamBayesGenerator.py, 1.7, 1.8
Message-ID: <E1AW7Po-0003HH-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/website/scripts/ht2html
In directory sc8-pr-cvs1:/tmp/cvs-serv12349/website/scripts/ht2html

Modified Files:
	LinkFixer.py PDOGenerator.py Sidebar.py SpamBayesGenerator.py 
Log Message:
Whitespace normalization.


Index: LinkFixer.py
===================================================================
RCS file: /cvsroot/spambayes/website/scripts/ht2html/LinkFixer.py,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** LinkFixer.py	11 Nov 2003 22:17:51 -0000	1.3
--- LinkFixer.py	16 Dec 2003 05:06:34 -0000	1.4
***************
*** 40,44 ****
          elif url[-1] == '/':
              url = url + 'index.html'
!         
          # normalize the path, kind of the way os.path.normpath() does.
          # urlparse ought to have something like this built in...
--- 40,44 ----
          elif url[-1] == '/':
              url = url + 'index.html'
! 
          # normalize the path, kind of the way os.path.normpath() does.
          # urlparse ought to have something like this built in...

Index: PDOGenerator.py
===================================================================
RCS file: /cvsroot/spambayes/website/scripts/ht2html/PDOGenerator.py,v
retrieving revision 1.1.1.1
retrieving revision 1.2
diff -C2 -d -r1.1.1.1 -r1.2
*** PDOGenerator.py	19 Sep 2002 08:40:55 -0000	1.1.1.1
--- PDOGenerator.py	16 Dec 2003 05:06:34 -0000	1.2
***************
*** 108,112 ****
      <img alt="" border="0"
           src="%(rootdir)s/pics/%(banner)s"></a></center>''' % \
!     self.__d 
  
      def get_corner_bgcolor(self):
--- 108,112 ----
      <img alt="" border="0"
           src="%(rootdir)s/pics/%(banner)s"></a></center>''' % \
!     self.__d
  
      def get_corner_bgcolor(self):
***************
*** 115,130 ****
          # value.  Some images may be `bizarre'.  See .../pics/backgrounds.py
          return [
!              '#3399ff',  '#6699cc',  '#3399ff',  '#0066cc',  '#3399ff', 
!              '#0066cc',  '#0066cc',  '#3399ff',  '#3399ff',  '#3399ff', 
!              '#3399ff',  '#6699cc',  '#3399ff',  '#3399ff',  '#ffffff', 
!              '#6699cc',  '#0066cc',  '#3399ff',  '#0066cc',  '#3399ff', 
!              '#6699cc',  '#0066cc',  '#6699cc',  '#3399ff',  '#3399ff', 
!              '#6699cc',  '#3399ff',  '#3399ff',  '#6699cc',  '#6699cc', 
!              '#0066cc',  '#6699cc',  '#0066cc',  '#6699cc',  '#0066cc', 
!              '#0066cc',  '#6699cc',  '#3399ff',  '#0066cc',  '#bbd6f1', 
!              '#0066cc',  '#6699cc',  '#3399ff',  '#3399ff',  '#0066cc', 
!              '#0066cc',  '#0066cc',  '#6699cc',  '#6699cc',  '#3399ff', 
!              '#3399ff',  '#6699cc',  '#0066cc',  '#0066cc',  '#6699cc', 
!              '#0066cc',  '#6699cc',  '#3399ff',  '#6699cc',  '#3399ff', 
               '#d6ebff',  '#6699cc',  '#3399ff',  '#0066cc',
               ][self.__whichbanner]
--- 115,130 ----
          # value.  Some images may be `bizarre'.  See .../pics/backgrounds.py
          return [
!              '#3399ff',  '#6699cc',  '#3399ff',  '#0066cc',  '#3399ff',
!              '#0066cc',  '#0066cc',  '#3399ff',  '#3399ff',  '#3399ff',
!              '#3399ff',  '#6699cc',  '#3399ff',  '#3399ff',  '#ffffff',
!              '#6699cc',  '#0066cc',  '#3399ff',  '#0066cc',  '#3399ff',
!              '#6699cc',  '#0066cc',  '#6699cc',  '#3399ff',  '#3399ff',
!              '#6699cc',  '#3399ff',  '#3399ff',  '#6699cc',  '#6699cc',
!              '#0066cc',  '#6699cc',  '#0066cc',  '#6699cc',  '#0066cc',
!              '#0066cc',  '#6699cc',  '#3399ff',  '#0066cc',  '#bbd6f1',
!              '#0066cc',  '#6699cc',  '#3399ff',  '#3399ff',  '#0066cc',
!              '#0066cc',  '#0066cc',  '#6699cc',  '#6699cc',  '#3399ff',
!              '#3399ff',  '#6699cc',  '#0066cc',  '#0066cc',  '#6699cc',
!              '#0066cc',  '#6699cc',  '#3399ff',  '#6699cc',  '#3399ff',
               '#d6ebff',  '#6699cc',  '#3399ff',  '#0066cc',
               ][self.__whichbanner]

Index: Sidebar.py
===================================================================
RCS file: /cvsroot/spambayes/website/scripts/ht2html/Sidebar.py,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** Sidebar.py	5 Jun 2003 15:13:54 -0000	1.3
--- Sidebar.py	16 Dec 2003 05:06:34 -0000	1.4
***************
*** 70,74 ****
                  else:
                      done_one = 1
!                 print '<tr><td %s>'% ( self.getSidebarHeaderAttrs()) 
                  print item
                  print '</font></b></td></tr>'
--- 70,74 ----
                  else:
                      done_one = 1
!                 print '<tr><td %s>'% ( self.getSidebarHeaderAttrs())
                  print item
                  print '</font></b></td></tr>'

Index: SpamBayesGenerator.py
===================================================================
RCS file: /cvsroot/spambayes/website/scripts/ht2html/SpamBayesGenerator.py,v
retrieving revision 1.7
retrieving revision 1.8
diff -C2 -d -r1.7 -r1.8
*** SpamBayesGenerator.py	26 Aug 2003 07:37:35 -0000	1.7
--- SpamBayesGenerator.py	16 Dec 2003 05:06:34 -0000	1.8
***************
*** 96,100 ****
      <a href="http://www.student.virginia.edu/~improv/games/findthespam.html">
      <img alt="" border="0" src="%(rootdir)s/images/logo.png"></a></center>''' % \
!     self.__d 
  
      def get_corner_bgcolor(self):
--- 96,100 ----
      <a href="http://www.student.virginia.edu/~improv/games/findthespam.html">
      <img alt="" border="0" src="%(rootdir)s/images/logo.png"></a></center>''' % \
!     self.__d
  
      def get_corner_bgcolor(self):



From tim_one at users.sourceforge.net  Tue Dec 16 00:06:37 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Tue Dec 16 00:06:56 2003
Subject: [Spambayes-checkins] spambayes/windows autoconfigure.py, 1.7,
	1.8 pop3proxy_service.py, 1.16, 1.17 pop3proxy_tray.py, 1.17, 1.18
Message-ID: <E1AW7Pp-0003HQ-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/windows
In directory sc8-pr-cvs1:/tmp/cvs-serv12349/windows

Modified Files:
	autoconfigure.py pop3proxy_service.py pop3proxy_tray.py 
Log Message:
Whitespace normalization.


Index: autoconfigure.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/windows/autoconfigure.py,v
retrieving revision 1.7
retrieving revision 1.8
diff -C2 -d -r1.7 -r1.8
*** autoconfigure.py	6 Oct 2003 03:01:01 -0000	1.7
--- autoconfigure.py	16 Dec 2003 05:06:34 -0000	1.8
***************
*** 90,99 ****
              s.close()
          except socket.error:
!              portStr = str(port)
!              if portStr in options["pop3proxy", "listen_ports"] or \
!                 portStr in options["smtpproxy", "listen_ports"]:
!                  continue
!              else:
!                  return port
  
  # Let's be safe and use high ports, starting at 1110 and 1025, and going up
--- 90,99 ----
              s.close()
          except socket.error:
!             portStr = str(port)
!             if portStr in options["pop3proxy", "listen_ports"] or \
!                portStr in options["smtpproxy", "listen_ports"]:
!                 continue
!             else:
!                 return port
  
  # Let's be safe and use high ports, starting at 1110 and 1025, and going up
***************
*** 131,135 ****
                      except ConfigParser.NoOptionError:
                          port = None
!                         
                      if us_name.lower()[:4] == "pop3":
                          if port is None:
--- 131,135 ----
                      except ConfigParser.NoOptionError:
                          port = None
! 
                      if us_name.lower()[:4] == "pop3":
                          if port is None:
***************
*** 207,211 ****
      pop_accounts = {}
      smtp_accounts = {}
!     
      r = re.compile(r"user_pref\(\"mail.server.server(\d+).(real)?hostname\", \"([^\"]*)\"\);")
      current_pos = 0
--- 207,211 ----
      pop_accounts = {}
      smtp_accounts = {}
! 
      r = re.compile(r"user_pref\(\"mail.server.server(\d+).(real)?hostname\", \"([^\"]*)\"\);")
      current_pos = 0
***************
*** 244,248 ****
          account_type = prefs[type_loc : \
                               type_loc + prefs[type_loc:].index('"')]
!         
          if account_type == "pop3":
              new_pref = 'user_pref("mail.server.server%s.%shostname", ' \
--- 244,248 ----
          account_type = prefs[type_loc : \
                               type_loc + prefs[type_loc:].index('"')]
! 
          if account_type == "pop3":
              new_pref = 'user_pref("mail.server.server%s.%shostname", ' \
***************
*** 291,295 ****
                     '"127.0.0.1");' % (server_num,)
  
!         # Find the port        
          port_string = 'user_pref("mail.smtpserver.smtp1.port", '
          port_loc = prefs.find(port_string)
--- 291,295 ----
                     '"127.0.0.1");' % (server_num,)
  
!         # Find the port
          port_string = 'user_pref("mail.smtpserver.smtp1.port", '
          port_loc = prefs.find(port_string)
***************
*** 387,391 ****
                      except ConfigParser.NoOptionError:
                          port = None
!                         
                      if us_name.lower()[:4] == "pop3":
                          if port is None:
--- 387,391 ----
                      except ConfigParser.NoOptionError:
                          port = None
! 
                      if us_name.lower()[:4] == "pop3":
                          if port is None:
***************
*** 425,429 ****
      # OE stores its configuration in the registry, not a file.
      key = "Software\\Microsoft\\Internet Account Manager\\Accounts"
!     
      import win32api
      import win32con
--- 425,429 ----
      # OE stores its configuration in the registry, not a file.
      key = "Software\\Microsoft\\Internet Account Manager\\Accounts"
! 
      import win32api
      import win32con
***************
*** 554,665 ****
  
  def configure_pocomail(unused):
!      import win32api
!      import win32con
!  
!      key = "Software\\Poco Systems Inc"
!  
!      pop_proxy  = pop_proxy_port
!      smtp_proxy = smtp_proxy_port
!  
!      reg           = win32api.RegOpenKeyEx(win32con.HKEY_CURRENT_USER, key)
!      subkey_name   = "%s\\%s" % (key, win32api.RegEnumKey(reg, 0))
!      reg           = win32api.RegOpenKeyEx(win32con.HKEY_CURRENT_USER,
!                                            subkey_name)
!      pocomail_path = win32api.RegQueryValueEx(reg, "Path")[0]
!  
!      pocomail_accounts_file = os.path.join(pocomail_path, "accounts.ini")
!  
!      if os.path.exists(pocomail_accounts_file):
!          f = open(pocomail_accounts_file, "r")
!  
!          accountName       = ""
!          pocomail_accounts = { }
!  
!          # Builds the dictionary with all the existing accounts.
!          for line in f.readlines():
!              line = line.rstrip('\n\r')
!              if line == '':
!                  continue
!  
!              if line[0] == '[' and line[-1] == ']':
!                  accountName = line[1:-1]
!                  pocomail_accounts[accountName] = { }
!              else:
!                  separator   = line.find('=')
!                  optionName  = line[:separator]
!                  optionValue = line[separator + 1:]
!  
!                  if optionName == "POPServer":
!                      pop3 = optionValue.split(':')
!                      if len(pop3) == 1:
!                          pop3.append(110)
!                      server = "%s:%s" % tuple(pop3)
!  
!                      proxy     = pop_proxy
!                      pop_proxy = move_to_next_free_port(pop_proxy)
!  
!                      if not server in options["pop3proxy", "remote_servers"]:
!                          options["pop3proxy", "remote_servers"] += (server,)
!                          options["pop3proxy", "listen_ports"]   += (proxy,)
!                      else:
!                          serverIndex = 0
!                          for remoteServer in options["pop3proxy",
!                                                      "remote_servers"]:
!                              if remoteServer == server:
!                                  break
!                              serverIndex += 1
!                          proxy = options["pop3proxy", "listen_ports"][serverIndex]
!  
!                      optionValue = "%s:%s" % ('localhost', proxy)
!  
!                  pocomail_accounts[accountName][optionName] = optionValue
!  
!          f.close()
!          f = open(pocomail_accounts_file, "w")
!          for accountName in pocomail_accounts.keys():
!              f.write('[' + accountName + ']\n')
!              for optionName, optionValue in pocomail_accounts[accountName].items():
!                  f.write("%s=%s\n" % (optionName, optionValue))
!              f.write('\n')
!          f.close()
!  
!          options.update_file(optionsPathname)
!  
!          # Add a filter to pocomail
!          pocomail_filters_file = os.path.join(pocomail_path, "filters.ini")
!  
!          if os.path.exists(pocomail_filters_file):
!              f = open(pocomail_filters_file, "r")
!      
!              pocomail_filters = { }
!              filterName       = ""
!  
!              for line in f.readlines():
!                  line = line.rstrip('\n\r')
!                  if line == '': continue
!      
!                  if line[0] == '[' and line[-1] == ']':
!                      filterName = line[1:-1]
!                      pocomail_filters[filterName] = []
!                  elif line[0] != '{':
!                      pocomail_filters[filterName].append(line)
!              f.close()
!  
!              spamBayesFilter = 'spam,X-Spambayes-Classification,move,' \
!                                '"Junk Mail",0,0,,,0,,,move,In,0,0,,0,,,' \
!                                'move,In,0,0,,0,,,move,In,0,0,,0,,,move,' \
!                                'In,0,0,,0,,,move,In,0,0,1,0'
!              if pocomail_filters.has_key("Incoming") and \
!                 spamBayesFilter not in pocomail_filters["Incoming"]:
!                     pocomail_filters["Incoming"].append(spamBayesFilter)
!  
!              f = open(pocomail_filters_file, "w")
!              f.write('{ Filter list generated by PocoMail 3.01 (1661)' \
!                      '- Licensed Version}\n')
!              for filterName in pocomail_filters.keys():
!                  f.write('\n[' + filterName + ']\n')
!                  for filter in pocomail_filters[filterName]:
!                      f.write(filter + '\n')
!              f.close()
  
  
--- 554,665 ----
  
  def configure_pocomail(unused):
!     import win32api
!     import win32con
! 
!     key = "Software\\Poco Systems Inc"
! 
!     pop_proxy  = pop_proxy_port
!     smtp_proxy = smtp_proxy_port
! 
!     reg           = win32api.RegOpenKeyEx(win32con.HKEY_CURRENT_USER, key)
!     subkey_name   = "%s\\%s" % (key, win32api.RegEnumKey(reg, 0))
!     reg           = win32api.RegOpenKeyEx(win32con.HKEY_CURRENT_USER,
!                                           subkey_name)
!     pocomail_path = win32api.RegQueryValueEx(reg, "Path")[0]
! 
!     pocomail_accounts_file = os.path.join(pocomail_path, "accounts.ini")
! 
!     if os.path.exists(pocomail_accounts_file):
!         f = open(pocomail_accounts_file, "r")
! 
!         accountName       = ""
!         pocomail_accounts = { }
! 
!         # Builds the dictionary with all the existing accounts.
!         for line in f.readlines():
!             line = line.rstrip('\n\r')
!             if line == '':
!                 continue
! 
!             if line[0] == '[' and line[-1] == ']':
!                 accountName = line[1:-1]
!                 pocomail_accounts[accountName] = { }
!             else:
!                 separator   = line.find('=')
!                 optionName  = line[:separator]
!                 optionValue = line[separator + 1:]
! 
!                 if optionName == "POPServer":
!                     pop3 = optionValue.split(':')
!                     if len(pop3) == 1:
!                         pop3.append(110)
!                     server = "%s:%s" % tuple(pop3)
! 
!                     proxy     = pop_proxy
!                     pop_proxy = move_to_next_free_port(pop_proxy)
! 
!                     if not server in options["pop3proxy", "remote_servers"]:
!                         options["pop3proxy", "remote_servers"] += (server,)
!                         options["pop3proxy", "listen_ports"]   += (proxy,)
!                     else:
!                         serverIndex = 0
!                         for remoteServer in options["pop3proxy",
!                                                     "remote_servers"]:
!                             if remoteServer == server:
!                                 break
!                             serverIndex += 1
!                         proxy = options["pop3proxy", "listen_ports"][serverIndex]
! 
!                     optionValue = "%s:%s" % ('localhost', proxy)
! 
!                 pocomail_accounts[accountName][optionName] = optionValue
! 
!         f.close()
!         f = open(pocomail_accounts_file, "w")
!         for accountName in pocomail_accounts.keys():
!             f.write('[' + accountName + ']\n')
!             for optionName, optionValue in pocomail_accounts[accountName].items():
!                 f.write("%s=%s\n" % (optionName, optionValue))
!             f.write('\n')
!         f.close()
! 
!         options.update_file(optionsPathname)
! 
!         # Add a filter to pocomail
!         pocomail_filters_file = os.path.join(pocomail_path, "filters.ini")
! 
!         if os.path.exists(pocomail_filters_file):
!             f = open(pocomail_filters_file, "r")
! 
!             pocomail_filters = { }
!             filterName       = ""
! 
!             for line in f.readlines():
!                 line = line.rstrip('\n\r')
!                 if line == '': continue
! 
!                 if line[0] == '[' and line[-1] == ']':
!                     filterName = line[1:-1]
!                     pocomail_filters[filterName] = []
!                 elif line[0] != '{':
!                     pocomail_filters[filterName].append(line)
!             f.close()
! 
!             spamBayesFilter = 'spam,X-Spambayes-Classification,move,' \
!                               '"Junk Mail",0,0,,,0,,,move,In,0,0,,0,,,' \
!                               'move,In,0,0,,0,,,move,In,0,0,,0,,,move,' \
!                               'In,0,0,,0,,,move,In,0,0,1,0'
!             if pocomail_filters.has_key("Incoming") and \
!                spamBayesFilter not in pocomail_filters["Incoming"]:
!                 pocomail_filters["Incoming"].append(spamBayesFilter)
! 
!             f = open(pocomail_filters_file, "w")
!             f.write('{ Filter list generated by PocoMail 3.01 (1661)' \
!                     '- Licensed Version}\n')
!             for filterName in pocomail_filters.keys():
!                 f.write('\n[' + filterName + ']\n')
!                 for filter in pocomail_filters[filterName]:
!                     f.write(filter + '\n')
!             f.close()
  
  

Index: pop3proxy_service.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/windows/pop3proxy_service.py,v
retrieving revision 1.16
retrieving revision 1.17
diff -C2 -d -r1.16 -r1.17
*** pop3proxy_service.py	14 Dec 2003 01:30:49 -0000	1.16
--- pop3proxy_service.py	16 Dec 2003 05:06:34 -0000	1.17
***************
*** 67,71 ****
      sb_dir = os.path.dirname(os.path.dirname(this_filename))
      sb_scripts_dir = os.path.join(sb_dir,"scripts")
!     
      sys.path.insert(0, sb_dir)
      sys.path.insert(-1, sb_scripts_dir)
--- 67,71 ----
      sb_dir = os.path.dirname(os.path.dirname(this_filename))
      sb_scripts_dir = os.path.join(sb_dir,"scripts")
! 
      sys.path.insert(0, sb_dir)
      sys.path.insert(-1, sb_scripts_dir)
***************
*** 102,106 ****
          self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
          self.event_stopping.set()
!         sb_server.stop()        
  
      def SvcDoRun(self):
--- 102,106 ----
          self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
          self.event_stopping.set()
!         sb_server.stop()
  
      def SvcDoRun(self):
***************
*** 113,118 ****
                    "another SpamBayes server is already running on this machine"
              servicemanager.LogErrorMsg(msg)
!             errCode = winerror.ERROR_SERVICE_SPECIFIC_ERROR 
!             self.ReportServiceStatus(win32service.SERVICE_STOPPED, 
                                       win32ExitCode=errCode, svcExitCode = 1)
              return
--- 113,118 ----
                    "another SpamBayes server is already running on this machine"
              servicemanager.LogErrorMsg(msg)
!             errCode = winerror.ERROR_SERVICE_SPECIFIC_ERROR
!             self.ReportServiceStatus(win32service.SERVICE_STOPPED,
                                       win32ExitCode=errCode, svcExitCode = 1)
              return
***************
*** 123,127 ****
          thread.start()
  
!         # Write an event log record - in debug mode we will also 
          # see this message printed.
          from spambayes.Options import optionsPathname
--- 123,127 ----
          thread.start()
  
!         # Write an event log record - in debug mode we will also
          # see this message printed.
          from spambayes.Options import optionsPathname
***************
*** 154,158 ****
          except KeyboardInterrupt:
              pass
!         
          # Write another event log record.
          s = sb_server.state
--- 154,158 ----
          except KeyboardInterrupt:
              pass
! 
          # Write another event log record.
          s = sb_server.state

Index: pop3proxy_tray.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/windows/pop3proxy_tray.py,v
retrieving revision 1.17
retrieving revision 1.18
diff -C2 -d -r1.17 -r1.18
*** pop3proxy_tray.py	14 Dec 2003 01:30:49 -0000	1.17
--- pop3proxy_tray.py	16 Dec 2003 05:06:34 -0000	1.18
***************
*** 184,188 ****
  
          flags = NIF_ICON | NIF_MESSAGE | NIF_TIP
!         nid = (self.hwnd, 0, flags, WM_TASKBAR_NOTIFY, self.hstartedicon, 
              "SpamBayes")
          Shell_NotifyIcon(NIM_ADD, nid)
--- 184,188 ----
  
          flags = NIF_ICON | NIF_MESSAGE | NIF_TIP
!         nid = (self.hwnd, 0, flags, WM_TASKBAR_NOTIFY, self.hstartedicon,
              "SpamBayes")
          Shell_NotifyIcon(NIM_ADD, nid)
***************
*** 192,196 ****
              print "Service not availible. Using thread."
              self.use_service = False
!       
          # Start up sb_server
          if not self.started:
--- 192,196 ----
              print "Service not availible. Using thread."
              self.use_service = False
! 
          # Start up sb_server
          if not self.started:
***************
*** 213,222 ****
              tip = "SpamBayes is not running"
          return tip
!             
  
      def UpdateIcon(self):
!         flags = NIF_TIP | NIF_ICON        
          if self.started:
!             hicon = self.hstartedicon 
          else:
              hicon = self.hstoppedicon
--- 213,222 ----
              tip = "SpamBayes is not running"
          return tip
! 
  
      def UpdateIcon(self):
!         flags = NIF_TIP | NIF_ICON
          if self.started:
!             hicon = self.hstartedicon
          else:
              hicon = self.hstoppedicon
***************
*** 277,284 ****
              elif dwWaitTime > 10000:
                  dwWaitTime = 10000
!     
              Sleep(dwWaitTime);
              ssStatus = QueryServiceStatus(schService)
!      
              if ssStatus[5] > dwOldCheckPoint:
                  dwStartTickCount = GetTickCount()
--- 277,284 ----
              elif dwWaitTime > 10000:
                  dwWaitTime = 10000
! 
              Sleep(dwWaitTime);
              ssStatus = QueryServiceStatus(schService)
! 
              if ssStatus[5] > dwOldCheckPoint:
                  dwStartTickCount = GetTickCount()
***************
*** 302,306 ****
              CloseServiceHandle(schService)
              return
!         
          ControlService(schService, SERVICE_CONTROL_STOP)
  
--- 302,306 ----
              CloseServiceHandle(schService)
              return
! 
          ControlService(schService, SERVICE_CONTROL_STOP)
  
***************
*** 316,323 ****
              elif dwWaitTime > 10000:
                  dwWaitTime = 10000
!     
              Sleep(dwWaitTime);
              ssStatus = QueryServiceStatus(schService)
!      
              if ssStatus[5] > dwOldCheckPoint:
                  dwStartTickCount = GetTickCount()
--- 316,323 ----
              elif dwWaitTime > 10000:
                  dwWaitTime = 10000
! 
              Sleep(dwWaitTime);
              ssStatus = QueryServiceStatus(schService)
! 
              if ssStatus[5] > dwOldCheckPoint:
                  dwStartTickCount = GetTickCount()
***************
*** 329,333 ****
          CloseServiceHandle(schService)
          self.started = False
!         
      def OnDestroy(self, hwnd, msg, wparam, lparam):
          nid = (self.hwnd, 0)
--- 329,333 ----
          CloseServiceHandle(schService)
          self.started = False
! 
      def OnDestroy(self, hwnd, msg, wparam, lparam):
          nid = (self.hwnd, 0)
***************
*** 401,405 ****
          DestroyWindow(self.hwnd)
          PostQuitMessage(0)
!         
      def _ProxyThread(self):
          self.started = True
--- 401,405 ----
          DestroyWindow(self.hwnd)
          PostQuitMessage(0)
! 
      def _ProxyThread(self):
          self.started = True
***************
*** 440,444 ****
          self.started = True
          self.UpdateUIState()
!         
      def Stop(self):
          self.CheckCurrentState()
--- 440,444 ----
          self.started = True
          self.UpdateUIState()
! 
      def Stop(self):
          self.CheckCurrentState()
***************
*** 472,476 ****
          self.started = IsServerRunningAnywhere()
          self.UpdateUIState()
!         
      def UpdateUIState(self):
          if self.started != self.last_started_state:
--- 472,476 ----
          self.started = IsServerRunningAnywhere()
          self.UpdateUIState()
! 
      def UpdateUIState(self):
          if self.started != self.last_started_state:
***************
*** 491,495 ****
              self.ShowMessage("SpamBayes is not running.")
  
!     def OpenConfig(self):		
          if self.started:
              webbrowser.open_new("http://localhost:%d/config" % \
--- 491,495 ----
              self.ShowMessage("SpamBayes is not running.")
  
!     def OpenConfig(self):
          if self.started:
              webbrowser.open_new("http://localhost:%d/config" % \
***************
*** 497,502 ****
          else:
              self.ShowMessage("SpamBayes is not running.")
!             
!     def OpenReview(self):		
          if self.started:
              webbrowser.open_new("http://localhost:%d/review" % \
--- 497,502 ----
          else:
              self.ShowMessage("SpamBayes is not running.")
! 
!     def OpenReview(self):
          if self.started:
              webbrowser.open_new("http://localhost:%d/review" % \
***************
*** 504,508 ****
          else:
              self.ShowMessage("SpamBayes is not running.")
!             
      def CheckVersion(self):
          # Stolen, with few modifications, from addin.py
--- 504,508 ----
          else:
              self.ShowMessage("SpamBayes is not running.")
! 
      def CheckVersion(self):
          # Stolen, with few modifications, from addin.py
***************
*** 538,542 ****
              # Offer to open up the url
  ##                os.startfile(url)
!       
      def ShowMessage(self, msg):
          MessageBox(self.hwnd, msg, "SpamBayes", win32con.MB_OK)
--- 538,542 ----
              # Offer to open up the url
  ##                os.startfile(url)
! 
      def ShowMessage(self, msg):
          MessageBox(self.hwnd, msg, "SpamBayes", win32con.MB_OK)
***************
*** 544,550 ****
  
  def main():
! 	w = MainWindow()
! 	PumpMessages()
  
  if __name__=='__main__':
! 	main()
--- 544,550 ----
  
  def main():
!     w = MainWindow()
!     PumpMessages()
  
  if __name__=='__main__':
!     main()



From tim_one at users.sourceforge.net  Tue Dec 16 00:06:36 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Tue Dec 16 00:06:58 2003
Subject: [Spambayes-checkins] spambayes/spambayes Corpus.py, 1.11,
	1.12 Dibbler.py, 1.10, 1.11 ImapUI.py, 1.29, 1.30 Options.py,
	1.91, 1.92 OptionsClass.py, 1.20, 1.21 ProxyUI.py, 1.33,
	1.34 ServerUI.py, 1.4, 1.5 UserInterface.py, 1.37,
	1.38 dbmstorage.py, 1.10, 1.11 message.py, 1.44,
	1.45 oe_mailbox.py, 1.2, 1.3 smtpproxy.py, 1.4, 1.5 storage.py,
	1.36, 1.37 tokenizer.py, 1.21, 1.22
Message-ID: <E1AW7Po-0003GZ-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv12349/spambayes

Modified Files:
	Corpus.py Dibbler.py ImapUI.py Options.py OptionsClass.py 
	ProxyUI.py ServerUI.py UserInterface.py dbmstorage.py 
	message.py oe_mailbox.py smtpproxy.py storage.py tokenizer.py 
Log Message:
Whitespace normalization.


Index: Corpus.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/Corpus.py,v
retrieving revision 1.11
retrieving revision 1.12
diff -C2 -d -r1.11 -r1.12
*** Corpus.py	15 Dec 2003 07:06:23 -0000	1.11
--- Corpus.py	16 Dec 2003 05:06:34 -0000	1.12
***************
*** 151,155 ****
          # This method should probably not be overridden
          key = message.key()
!         
          if options["globals", "verbose"]:
              print 'placing %s in corpus cache' % (key)
--- 151,155 ----
          # This method should probably not be overridden
          key = message.key()
! 
          if options["globals", "verbose"]:
              print 'placing %s in corpus cache' % (key)

Index: Dibbler.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/Dibbler.py,v
retrieving revision 1.10
retrieving revision 1.11
diff -C2 -d -r1.10 -r1.11
*** Dibbler.py	11 Dec 2003 18:40:54 -0000	1.10
--- Dibbler.py	16 Dec 2003 05:06:34 -0000	1.11
***************
*** 422,426 ****
                             for line in headers.split('\r\n')
                             if headersRegex.match(line)])
!         
          # HTTP Basic/Digest Authentication support.
          serverAuthMode = self._server.requestAuthenticationMode()
--- 422,426 ----
                             for line in headers.split('\r\n')
                             if headersRegex.match(line)])
! 
          # HTTP Basic/Digest Authentication support.
          serverAuthMode = self._server.requestAuthenticationMode()
***************
*** 474,478 ****
                              if isinstance(dispatcher, Listener):
                                  dispatcher.close()
!                         
                          # Let any existing connections close down first.  This
                          # has happened when all we have left are _HTTPHandlers
--- 474,478 ----
                              if isinstance(dispatcher, Listener):
                                  dispatcher.close()
! 
                          # Let any existing connections close down first.  This
                          # has happened when all we have left are _HTTPHandlers
***************
*** 482,491 ****
                          def isProtected(dispatcher):
                              return not isinstance(dispatcher, _HTTPHandler)
!                         
                          while len(filter(isProtected, contextMap.values())) > 0:
                              asyncore.poll(timeout=1, map=contextMap)
!                         
                          raise SystemExit
!                         
                      message = """<h3>500 Server error</h3><pre>%s</pre>"""
                      details = traceback.format_exception(eType, eValue, eTrace)
--- 482,491 ----
                          def isProtected(dispatcher):
                              return not isinstance(dispatcher, _HTTPHandler)
! 
                          while len(filter(isProtected, contextMap.values())) > 0:
                              asyncore.poll(timeout=1, map=contextMap)
! 
                          raise SystemExit
! 
                      message = """<h3>500 Server error</h3><pre>%s</pre>"""
                      details = traceback.format_exception(eType, eValue, eTrace)
***************
*** 560,564 ****
              self.writeError(500, "Inconsistent authentication mode.")
              return
!         
          headers = []
          headers.append('HTTP/1.0 401 Unauthorized')
--- 560,564 ----
              self.writeError(500, "Inconsistent authentication mode.")
              return
! 
          headers = []
          headers.append('HTTP/1.0 401 Unauthorized')
***************
*** 586,590 ****
  
      def _getCurrentNonce(self):
!         """Returns the current nonce value. This value is a Base64 encoding 
          of current time plus 20 minutes. This means the nonce will expire 20
          minutes from now."""
--- 586,590 ----
  
      def _getCurrentNonce(self):
!         """Returns the current nonce value. This value is a Base64 encoding
          of current time plus 20 minutes. This means the nonce will expire 20
          minutes from now."""
***************
*** 613,617 ****
          def stripQuotes(s):
              return (s[0] == '"' and s[-1] == '"') and s[1:-1] or s
!         
          options  = dict([s.split('=') for s in login.split(", ")])
          userName = stripQuotes(options["username"])
--- 613,617 ----
          def stripQuotes(s):
              return (s[0] == '"' and s[-1] == '"') and s[1:-1] or s
! 
          options  = dict([s.split('=') for s in login.split(", ")])
          userName = stripQuotes(options["username"])

Index: ImapUI.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/ImapUI.py,v
retrieving revision 1.29
retrieving revision 1.30
diff -C2 -d -r1.29 -r1.30
*** ImapUI.py	16 Dec 2003 00:30:49 -0000	1.29
--- ImapUI.py	16 Dec 2003 05:06:34 -0000	1.30
***************
*** 255,259 ****
          UserInterface.UserInterface.onChangeopts(self, **parms)
          self.parm_ini_map = backup
!         
      def _buildFolderBox(self, section, option, available_folders):
          folderTable = self.html.configTable.clone()
--- 255,259 ----
          UserInterface.UserInterface.onChangeopts(self, **parms)
          self.parm_ini_map = backup
! 
      def _buildFolderBox(self, section, option, available_folders):
          folderTable = self.html.configTable.clone()

Index: Options.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/Options.py,v
retrieving revision 1.91
retrieving revision 1.92
diff -C2 -d -r1.91 -r1.92
*** Options.py	16 Dec 2003 02:03:30 -0000	1.91
--- Options.py	16 Dec 2003 05:06:34 -0000	1.92
***************
*** 165,171 ****
       are generated for messages with habeas headers.  This should be fine,
       since messages with the headers should either be ham, or result in FN
!      so that we can send them to habeas so they can be sued.  However, to 
!      reduce the strength of habeas headers, we offer the ability to reduce 
!      the nine tokens to one. (This option has no effect if 
       search_for_habeas_headers is False)""",
       BOOLEAN, RESTORE),
--- 165,171 ----
       are generated for messages with habeas headers.  This should be fine,
       since messages with the headers should either be ham, or result in FN
!      so that we can send them to habeas so they can be sued.  However, to
!      reduce the strength of habeas headers, we offer the ability to reduce
!      the nine tokens to one. (This option has no effect if
       search_for_habeas_headers is False)""",
       BOOLEAN, RESTORE),
***************
*** 364,368 ****
       assigning certainty (0 or 1) to a word that has appeared in only ham
       or only spam.  This is a disaster.
!      
       As unknown_word_strength tends toward infintity, all probabilities
       tend toward unknown_word_prob.  All reports were that a value near 0.4
--- 364,368 ----
       assigning certainty (0 or 1) to a word that has appeared in only ham
       or only spam.  This is a disaster.
! 
       As unknown_word_strength tends toward infintity, all probabilities
       tend toward unknown_word_prob.  All reports were that a value near 0.4
***************
*** 975,979 ****
       PORT, RESTORE),
    ),
!  
    "globals" : (
      ("verbose", "Verbose", False,
--- 975,979 ----
       PORT, RESTORE),
    ),
! 
    "globals" : (
      ("verbose", "Verbose", False,
***************
*** 1037,1041 ****
              options.merge_files(alts)
              optionsPathname = os.path.abspath(alts[-1])
!     
      if not optionsPathname:
          optionsPathname = os.path.abspath('bayescustomize.ini')
--- 1037,1041 ----
              options.merge_files(alts)
              optionsPathname = os.path.abspath(alts[-1])
! 
      if not optionsPathname:
          optionsPathname = os.path.abspath('bayescustomize.ini')

Index: OptionsClass.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/OptionsClass.py,v
retrieving revision 1.20
retrieving revision 1.21
diff -C2 -d -r1.20 -r1.21
*** OptionsClass.py	16 Dec 2003 04:48:28 -0000	1.20
--- OptionsClass.py	16 Dec 2003 05:06:34 -0000	1.21
***************
*** 146,150 ****
          '''Multiple values are allowed for this option.'''
          return type(self.value) in MultiContainerTypes
!     
      def is_valid(self, value):
          '''Check if this is a valid value for this option.'''
--- 146,150 ----
          '''Multiple values are allowed for this option.'''
          return type(self.value) in MultiContainerTypes
! 
      def is_valid(self, value):
          '''Check if this is a valid value for this option.'''
***************
*** 193,197 ****
                  # either no match or too many matches
                  return False
!             
      def _split_values(self, value):
          # do the regex mojo here
--- 193,197 ----
                  # either no match or too many matches
                  return False
! 
      def _split_values(self, value):
          # do the regex mojo here
***************
*** 470,474 ****
          out.close()
          f.close()
!     
      def _add_missing(self, out, written, sect, vi, label=True):
          # add any missing ones, where the value does not equal the default
--- 470,474 ----
          out.close()
          f.close()
! 
      def _add_missing(self, out, written, sect, vi, label=True):
          # add any missing ones, where the value does not equal the default
***************
*** 503,507 ****
                  except TypeError: # opt[0] not a class
                      pass
!                         
                  o = klass(*args)
                  self._options[section, o.name] = o
--- 503,507 ----
                  except TypeError: # opt[0] not a class
                      pass
! 
                  o = klass(*args)
                  self._options[section, o.name] = o
***************
*** 607,611 ****
          else:
              print >> sys.stderr, ("Attempted to set [%s] %s with invalid"
!                                   " value %s (%s)" % 
                                    (sect, opt.lower(), val, type(val)))
  
--- 607,611 ----
          else:
              print >> sys.stderr, ("Attempted to set [%s] %s with invalid"
!                                   " value %s (%s)" %
                                    (sect, opt.lower(), val, type(val)))
  
***************
*** 667,671 ****
          all.sort()
          return all
!     
      def options_in_section(self, section):
          '''Return an alphabetical list of all the options in this section.'''
--- 667,671 ----
          all.sort()
          return all
! 
      def options_in_section(self, section):
          '''Return an alphabetical list of all the options in this section.'''
***************
*** 707,730 ****
  
      def display_full(self, section=None, option=None):
!        '''Display options including all information.'''
!        # Given that the Options class is no longer as nice looking
!        # as it once was, this returns all the information, i.e.
!        # the doc, default values, and so on
!        output = StringIO.StringIO()
  
!        # when section and option are both specified, this
!        # is nothing more than a call to as_nice_string
!        if section is not None and option is not None:
!            output.write(self._options[section,
!                                       option.lower()].as_nice_string(section))
!            return output.getvalue()
!        
!        all = self._options.keys()
!        all.sort()
!        for sect, opt in all:
!            if section is not None and sect != section:
!                continue
!            output.write(self._options[sect, opt.lower()].as_nice_string(sect))
!        return output.getvalue()
  
  # These are handy references to commonly used regex/tuples defining
--- 707,730 ----
  
      def display_full(self, section=None, option=None):
!         '''Display options including all information.'''
!         # Given that the Options class is no longer as nice looking
!         # as it once was, this returns all the information, i.e.
!         # the doc, default values, and so on
!         output = StringIO.StringIO()
  
!         # when section and option are both specified, this
!         # is nothing more than a call to as_nice_string
!         if section is not None and option is not None:
!             output.write(self._options[section,
!                                        option.lower()].as_nice_string(section))
!             return output.getvalue()
! 
!         all = self._options.keys()
!         all.sort()
!         for sect, opt in all:
!             if section is not None and sect != section:
!                 continue
!             output.write(self._options[sect, opt.lower()].as_nice_string(sect))
!         return output.getvalue()
  
  # These are handy references to commonly used regex/tuples defining

Index: ProxyUI.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/ProxyUI.py,v
retrieving revision 1.33
retrieving revision 1.34
diff -C2 -d -r1.33 -r1.34
*** ProxyUI.py	16 Dec 2003 00:30:49 -0000	1.33
--- ProxyUI.py	16 Dec 2003 05:06:34 -0000	1.34
***************
*** 700,704 ****
  
          if pmap == adv_map:
!           return errmsg
  
          # check for equal number of pop3servers and ports
--- 700,704 ----
  
          if pmap == adv_map:
!             return errmsg
  
          # check for equal number of pop3servers and ports

Index: ServerUI.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/ServerUI.py,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** ServerUI.py	15 Dec 2003 00:16:20 -0000	1.4
--- ServerUI.py	16 Dec 2003 05:06:34 -0000	1.5
***************
*** 105,109 ****
          # we have a few extra checks
          errmsg = UserInterface.UserInterface.verifyInput(self, parms)
!         
          # check for equal number of pop3servers and ports
          slist = list(parms['pop3proxy_remote_servers'])
--- 105,109 ----
          # we have a few extra checks
          errmsg = UserInterface.UserInterface.verifyInput(self, parms)
! 
          # check for equal number of pop3servers and ports
          slist = list(parms['pop3proxy_remote_servers'])

Index: UserInterface.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/UserInterface.py,v
retrieving revision 1.37
retrieving revision 1.38
diff -C2 -d -r1.37 -r1.38
*** UserInterface.py	16 Dec 2003 02:46:42 -0000	1.37
--- UserInterface.py	16 Dec 2003 05:06:34 -0000	1.38
***************
*** 130,139 ****
  
          if trustedIPs == "*" or remoteIP == clientSocket.getsockname()[0]:
!            return True
  
          trustedIPs = trustedIPs.replace('.', '\.').replace('*', '([01]?\d\d?|2[04]\d|25[0-5])')
          for trusted in trustedIPs.split(','):
!           if re.search("^" + trusted + "$", remoteIP):
!              return True
  
          return False
--- 130,139 ----
  
          if trustedIPs == "*" or remoteIP == clientSocket.getsockname()[0]:
!             return True
  
          trustedIPs = trustedIPs.replace('.', '\.').replace('*', '([01]?\d\d?|2[04]\d|25[0-5])')
          for trusted in trustedIPs.split(','):
!             if re.search("^" + trusted + "$", remoteIP):
!                 return True
  
          return False
***************
*** 299,303 ****
                                        nham, nspam)
          return cluesTable
!     
      def _buildCluesTable(self, message, subject=None, show_tokens=False):
          tokens = list(tokenizer.tokenize(message))
--- 299,303 ----
                                        nham, nspam)
          return cluesTable
! 
      def _buildCluesTable(self, message, subject=None, show_tokens=False):
          tokens = list(tokenizer.tokenize(message))
***************
*** 477,481 ****
          # Attempt to convert the content from a DBX file to a standard mbox
          if file:
!           content = self._convertToMbox(content)
  
          # Convert platform-specific line endings into unix-style.
--- 477,481 ----
          # Attempt to convert the content from a DBX file to a standard mbox
          if file:
!             content = self._convertToMbox(content)
  
          # Convert platform-specific line endings into unix-style.
***************
*** 513,517 ****
          """Check if the given buffer is in a non-mbox format, and convert it
          into mbox format if so.  If it's already an mbox, return it unchanged.
!         
          Currently, the only supported non-mbox format is Outlook Express DBX.
          In such a case we use the module oe_mailbox to convert the DBX
--- 513,517 ----
          """Check if the given buffer is in a non-mbox format, and convert it
          into mbox format if so.  If it's already an mbox, return it unchanged.
! 
          Currently, the only supported non-mbox format is Outlook Express DBX.
          In such a case we use the module oe_mailbox to convert the DBX
***************
*** 527,536 ****
              fh_entries = oe_mailbox.dbxFileHeader.FH_ENTRIES
              fh_ptr = oe_mailbox.dbxFileHeader.FH_TREE_ROOT_NODE_PTR
!             
              info = oe_mailbox.dbxFileInfo(dbxStream,
                                            header.getEntry(file_info_len))
              entries = header.getEntry(fh_entries)
              address = header.getEntry(fh_ptr)
!             
              if address and entries:
                  tree = oe_mailbox.dbxTree(dbxStream, address, entries)
--- 527,536 ----
              fh_entries = oe_mailbox.dbxFileHeader.FH_ENTRIES
              fh_ptr = oe_mailbox.dbxFileHeader.FH_TREE_ROOT_NODE_PTR
! 
              info = oe_mailbox.dbxFileInfo(dbxStream,
                                            header.getEntry(file_info_len))
              entries = header.getEntry(fh_entries)
              address = header.getEntry(fh_ptr)
! 
              if address and entries:
                  tree = oe_mailbox.dbxTree(dbxStream, address, entries)
***************
*** 863,867 ****
                  if options.multiple_values_allowed(sect, opt) and \
                     value == "":
!                         value = ()
                  value = options.convert(sect, opt, value)
              if not options.is_valid(sect, opt, value):
--- 863,867 ----
                  if options.multiple_values_allowed(sect, opt) and \
                     value == "":
!                     value = ()
                  value = options.convert(sect, opt, value)
              if not options.is_valid(sect, opt, value):

Index: dbmstorage.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/dbmstorage.py,v
retrieving revision 1.10
retrieving revision 1.11
diff -C2 -d -r1.10 -r1.11
*** dbmstorage.py	26 Nov 2003 01:13:02 -0000	1.10
--- dbmstorage.py	16 Dec 2003 05:06:34 -0000	1.11
***************
*** 15,19 ****
  
  def open_dbhash(*args):
!     """Open a bsddb hash.  Don't use this on Windows, unless Python 2.3 or 
      greater is used, in which case bsddb3 is actually named bsddb."""
      import bsddb
--- 15,19 ----
  
  def open_dbhash(*args):
!     """Open a bsddb hash.  Don't use this on Windows, unless Python 2.3 or
      greater is used, in which case bsddb3 is actually named bsddb."""
      import bsddb

Index: message.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/message.py,v
retrieving revision 1.44
retrieving revision 1.45
diff -C2 -d -r1.44 -r1.45
*** message.py	11 Dec 2003 18:45:31 -0000	1.44
--- message.py	16 Dec 2003 05:06:34 -0000	1.45
***************
*** 164,168 ****
          self.load()
  
!     def load(self):        
          try:
              self.dbm = dbmstorage.open(self.db_name, self.mode)
--- 164,168 ----
          self.load()
  
!     def load(self):
          try:
              self.dbm = dbmstorage.open(self.db_name, self.mode)
***************
*** 179,183 ****
          self.close()
  
!     def close(self):        
          # Close our underlying database.  Better not assume all databases
          # have close functions!
--- 179,183 ----
          self.close()
  
!     def close(self):
          # Close our underlying database.  Better not assume all databases
          # have close functions!

Index: oe_mailbox.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/oe_mailbox.py,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** oe_mailbox.py	11 Dec 2003 18:45:46 -0000	1.2
--- oe_mailbox.py	16 Dec 2003 05:06:34 -0000	1.3
***************
*** 16,118 ****
  ###########################################################################
  class dbxFileHeader:
!   """
!      Each Outlook Express DBX file has a file header.
!      This header defines many properties, only a few of which interest us.
  
!      The only properties which are required are defined by indexes. The
!      indexes are static attributes of the class and their names begin with
!      "fh". You can access their values through the method getEntry().
!   """
  
!   HEADER_SIZE    = 0x24bc              # total header size
!   HEADER_ENTRIES = HEADER_SIZE >> 2    # total of entries in the header
  
!   MAGIC_NUMBER   = 0xfe12adcfL         # specific to DBX files
!   OFFLINE        = 0x26fe9d30L         # specific to offline.dbx
!   FOLDERS        = 0x6f74fdc6L         # specific to folders.dbx
!   POP3UIDL       = 0x6f74fdc7L         # specific to pop3uidl.dbx
  
!   # various entries indexes
!   FH_FILE_INFO_LENGTH       = 0x07     # file info length
!   FH_FIRST_FOLDER_LIST_NODE = 0x1b     # pointer to the first folder list node
!   FH_LAST_FOLDER_LIST_NODE  = 0x1c     # pointer to the last folder list node
!   FH_MESSAGE_CONDITIONS_PTR = 0x22     # pointer to the message conditions object
!   FH_FOLDER_CONDITIONS_PTR  = 0x23     # pointer to the folder conditions object
!   FH_ENTRIES                = 0x31     # entries in tree
!   FH_TREE_ROOT_NODE_PTR     = 0x39     # pointer to the root node of a tree
  
!   FILE_HEADER_ENTRIES = \
!   [ ( 0x07,  "file info length"                                ),
!     ( 0x09,  "pointer to the last variable segment"            ),
!     ( 0x0a,  "length of a variable segment"                    ),
!     ( 0x0b,  "used space of the last variable segment"         ),
!     ( 0x0c,  "pointer to the last tree segment"                ),
!     ( 0x0d,  "length of a tree segment"                        ),
!     ( 0x0e,  "used space of the last tree segment"             ),
!     ( 0x0f,  "pointer to the last message segment"             ),
!     ( 0x10,  "length of a message segment"                     ),
!     ( 0x11,  "used space of the last message segment"          ),
!     ( 0x12,  "root pointer to the deleted message list"        ),
!     ( 0x13,  "root pointer to the deleted tree list"           ),
!     ( 0x15,  "used space in the middle sector of the file"     ),
!     ( 0x16,  "reusable space in the middle sector of the file" ),
!     ( 0x17,  "index of the last entry in the tree"             ),
!     ( 0x1b,  "pointer to the first folder list node"           ),
!     ( 0x1c,  "pointer to the last folder list node"            ),
!     ( 0x1f,  "used space of the file"                          ),
!     ( 0x22,  "pointer to the message conditions object"        ),
!     ( 0x23,  "pointer to the folder conditions object"         ),
!     ( 0x31,  "entries in the tree"                             ),
!     ( 0x32,  "entries in the 2.nd tree"                        ),
!     ( 0x33,  "entries in the 3.rd tree"                        ),
!     ( 0x39,  "pointer to the root node of the tree"            ),
!     ( 0x3a,  "pointer to the root node of the 2.nd tree"       ),
!     ( 0x3b,  "pointer to the root node of the 3.rd tree"       ),
!     ( 0x9f,  "used space for indexed info objects"             ),
!     ( 0xa0,  "used space for conditions objects"               ),
!     ( 0xa2,  "used space for folder list objects"              ),
!     ( 0xa3,  "used space for tree objects"                     ),
!     ( 0xa4,  "used space for message objects"                  )]
  
!   def __init__(self, dbxStream):
!     """Initialize the DBX header by reading it directly from the passed
!     stream."""
!     dbxStream.seek(0)
!     self.dbxBuffer = dbxStream.read(dbxFileHeader.HEADER_SIZE)
  
!   def isMessages(self):
!     """Return true iff the DBX is a messages DBX."""
!     return not (self.isFolders() or self.isPOP3UIDL() or self.isOffline())
  
!   def isFolders(self):
!     """Return true if the DBX is the folders DBX."""
!     return self.getEntry(1) == dbxFileHeader.FOLDERS
  
!   def isPOP3UIDL(self):
!     """Return true if the DBX is the POP3UIDL DBX."""
!     return self.getEntry(1) == dbxFileHeader.POP3UIDL
  
!   def isOffline(self):
!     """Return true if the DBX is the offline DBX."""
!     return self.getEntry(1) == dbxFileHeader.OFFLINE
  
!   def isValid(self):
!     """Return true if the DBX is a valid DBX file."""
!     return self.getEntry(0) == dbxFileHeader.MAGIC_NUMBER
  
!   def getHeaderBuffer(self):
!     """Return the bytes buffer containing the whole header."""
!     return self.dbxBuffer
  
!   def getEntry(self, dbxEntry):
!     """Return the n-th entry as a long integer."""
!     return struct.unpack("L",
!                          self.dbxBuffer[dbxEntry * 4:(dbxEntry * 4) + 4])[0]
  
!   def getEntryAsHexStr(self, dbxEntry):
!     """Return the n-th entry as an hexadecimal string.
!     (Little endian encoding!)"""
!     return '0x' + \
!            binascii.hexlify(self.dbxBuffer[dbxEntry * 4:(dbxEntry * 4) + 4])
  
  ###########################################################################
--- 16,118 ----
  ###########################################################################
  class dbxFileHeader:
!     """
!        Each Outlook Express DBX file has a file header.
!        This header defines many properties, only a few of which interest us.
  
!        The only properties which are required are defined by indexes. The
!        indexes are static attributes of the class and their names begin with
!        "fh". You can access their values through the method getEntry().
!     """
  
!     HEADER_SIZE    = 0x24bc              # total header size
!     HEADER_ENTRIES = HEADER_SIZE >> 2    # total of entries in the header
  
!     MAGIC_NUMBER   = 0xfe12adcfL         # specific to DBX files
!     OFFLINE        = 0x26fe9d30L         # specific to offline.dbx
!     FOLDERS        = 0x6f74fdc6L         # specific to folders.dbx
!     POP3UIDL       = 0x6f74fdc7L         # specific to pop3uidl.dbx
  
!     # various entries indexes
!     FH_FILE_INFO_LENGTH       = 0x07     # file info length
!     FH_FIRST_FOLDER_LIST_NODE = 0x1b     # pointer to the first folder list node
!     FH_LAST_FOLDER_LIST_NODE  = 0x1c     # pointer to the last folder list node
!     FH_MESSAGE_CONDITIONS_PTR = 0x22     # pointer to the message conditions object
!     FH_FOLDER_CONDITIONS_PTR  = 0x23     # pointer to the folder conditions object
!     FH_ENTRIES                = 0x31     # entries in tree
!     FH_TREE_ROOT_NODE_PTR     = 0x39     # pointer to the root node of a tree
  
!     FILE_HEADER_ENTRIES = \
!     [ ( 0x07,  "file info length"                                ),
!       ( 0x09,  "pointer to the last variable segment"            ),
!       ( 0x0a,  "length of a variable segment"                    ),
!       ( 0x0b,  "used space of the last variable segment"         ),
!       ( 0x0c,  "pointer to the last tree segment"                ),
!       ( 0x0d,  "length of a tree segment"                        ),
!       ( 0x0e,  "used space of the last tree segment"             ),
!       ( 0x0f,  "pointer to the last message segment"             ),
!       ( 0x10,  "length of a message segment"                     ),
!       ( 0x11,  "used space of the last message segment"          ),
!       ( 0x12,  "root pointer to the deleted message list"        ),
!       ( 0x13,  "root pointer to the deleted tree list"           ),
!       ( 0x15,  "used space in the middle sector of the file"     ),
!       ( 0x16,  "reusable space in the middle sector of the file" ),
!       ( 0x17,  "index of the last entry in the tree"             ),
!       ( 0x1b,  "pointer to the first folder list node"           ),
!       ( 0x1c,  "pointer to the last folder list node"            ),
!       ( 0x1f,  "used space of the file"                          ),
!       ( 0x22,  "pointer to the message conditions object"        ),
!       ( 0x23,  "pointer to the folder conditions object"         ),
!       ( 0x31,  "entries in the tree"                             ),
!       ( 0x32,  "entries in the 2.nd tree"                        ),
!       ( 0x33,  "entries in the 3.rd tree"                        ),
!       ( 0x39,  "pointer to the root node of the tree"            ),
!       ( 0x3a,  "pointer to the root node of the 2.nd tree"       ),
!       ( 0x3b,  "pointer to the root node of the 3.rd tree"       ),
!       ( 0x9f,  "used space for indexed info objects"             ),
!       ( 0xa0,  "used space for conditions objects"               ),
!       ( 0xa2,  "used space for folder list objects"              ),
!       ( 0xa3,  "used space for tree objects"                     ),
!       ( 0xa4,  "used space for message objects"                  )]
  
!     def __init__(self, dbxStream):
!         """Initialize the DBX header by reading it directly from the passed
!         stream."""
!         dbxStream.seek(0)
!         self.dbxBuffer = dbxStream.read(dbxFileHeader.HEADER_SIZE)
  
!     def isMessages(self):
!         """Return true iff the DBX is a messages DBX."""
!         return not (self.isFolders() or self.isPOP3UIDL() or self.isOffline())
  
!     def isFolders(self):
!         """Return true if the DBX is the folders DBX."""
!         return self.getEntry(1) == dbxFileHeader.FOLDERS
  
!     def isPOP3UIDL(self):
!         """Return true if the DBX is the POP3UIDL DBX."""
!         return self.getEntry(1) == dbxFileHeader.POP3UIDL
  
!     def isOffline(self):
!         """Return true if the DBX is the offline DBX."""
!         return self.getEntry(1) == dbxFileHeader.OFFLINE
  
!     def isValid(self):
!         """Return true if the DBX is a valid DBX file."""
!         return self.getEntry(0) == dbxFileHeader.MAGIC_NUMBER
  
!     def getHeaderBuffer(self):
!         """Return the bytes buffer containing the whole header."""
!         return self.dbxBuffer
  
!     def getEntry(self, dbxEntry):
!         """Return the n-th entry as a long integer."""
!         return struct.unpack("L",
!                              self.dbxBuffer[dbxEntry * 4:(dbxEntry * 4) + 4])[0]
  
!     def getEntryAsHexStr(self, dbxEntry):
!         """Return the n-th entry as an hexadecimal string.
!         (Little endian encoding!)"""
!         return '0x' + \
!                binascii.hexlify(self.dbxBuffer[dbxEntry * 4:(dbxEntry * 4) + 4])
  
  ###########################################################################
***************
*** 121,155 ****
  
  class dbxFileInfo:
!   """
!     Following the DBX header there is DBX info. This part gives the name of
!     the folder described by the current DBX.
!   """
  
!   MESSAGE_FILE_INFO = 0x618
  
!   def __init__(self, dbxStream, dbxLength):
!     """Reads the DBX info part from a DBX stream."""
!     dbxStream.seek(dbxFileHeader.HEADER_SIZE)
!     self.dbxLength = dbxLength
!     self.dbxBuffer = dbxStream.read(dbxLength)
  
!   def isFoldersInfo(self):
!     """Return true if the info belongs to folders.dbx."""
!     return self.dbxLength != dbxFileInfo.MESSAGE_FILE_INFO
  
!   def getFolderName(self):
!     """Returns the folder name."""
!     if not self.isFoldersInfo():
!       name = [c for c in self.dbxBuffer[0x105:0x210] if ord(c) != 0]
!       return "".join(name)
!     else:
!       return None
  
!   def getCreationTime(self):
!     """Not implemented yet."""
!     if self.isFoldersInfo():
!       return "Not implemented yet"
!     else:
!       return None
  
  ###########################################################################
--- 121,155 ----
  
  class dbxFileInfo:
!     """
!       Following the DBX header there is DBX info. This part gives the name of
!       the folder described by the current DBX.
!     """
  
!     MESSAGE_FILE_INFO = 0x618
  
!     def __init__(self, dbxStream, dbxLength):
!         """Reads the DBX info part from a DBX stream."""
!         dbxStream.seek(dbxFileHeader.HEADER_SIZE)
!         self.dbxLength = dbxLength
!         self.dbxBuffer = dbxStream.read(dbxLength)
  
!     def isFoldersInfo(self):
!         """Return true if the info belongs to folders.dbx."""
!         return self.dbxLength != dbxFileInfo.MESSAGE_FILE_INFO
  
!     def getFolderName(self):
!         """Returns the folder name."""
!         if not self.isFoldersInfo():
!             name = [c for c in self.dbxBuffer[0x105:0x210] if ord(c) != 0]
!             return "".join(name)
!         else:
!             return None
  
!     def getCreationTime(self):
!         """Not implemented yet."""
!         if self.isFoldersInfo():
!             return "Not implemented yet"
!         else:
!             return None
  
  ###########################################################################
***************
*** 158,204 ****
  
  class dbxTree:
!   """Stands for the tree which stores the messages in a given folder."""
  
!   TREE_NODE_SIZE = 0x27c          # size of a tree node
  
!   def __init__(self, dbxStream, dbxAddress, dbxValues):
!     """Reads the addresses of the stored messages."""
!     self.dbxValues = [i for i in range(dbxValues)]
!     # XXX : silly fix !
!     if dbxAddress > 0:
!       self.__readValues(dbxStream, 0, dbxAddress, 0, dbxValues)
  
!   def __readValues(self, dbxStream, dbxParent, dbxAddress, dbxPosition, dbxValues):
!     dbxStream.seek(dbxAddress)
!     dbxBuffer = dbxStream.read(dbxTree.TREE_NODE_SIZE)
  
!     count = 0
!     entries = ((self.getEntry(dbxBuffer, 4) >> 8) & 0xff)
  
!     if self.getEntry(dbxBuffer, 2) != 0:
!       self.__readValues(dbxStream, dbxAddress, self.getEntry(dbxBuffer, 2),
!                                    dbxPosition, self.getEntry(dbxBuffer, 5))
!       count += self.getEntry(dbxBuffer, 5)
  
!     for i in range(entries):
!       pos = 6 + i * 3
  
!       if self.getEntry(dbxBuffer, pos) != 0:
!         count += 1
!         value = dbxPosition + count
!         self.dbxValues[value - 1] = self.getEntry(dbxBuffer, pos)
  
!       if self.getEntry(dbxBuffer, pos + 1) != 0:
!         self.__readValues(dbxStream, dbxAddress, self.getEntry(dbxBuffer, pos + 1),
!                                      dbxPosition + count, self.getEntry(dbxBuffer, pos + 2))
!         count += self.getEntry(dbxBuffer, pos + 2)
  
!   def getEntry(self, dbxBuffer, dbxEntry):
!     """Return the n-th entry as a long integer."""
!     return struct.unpack("L", dbxBuffer[dbxEntry * 4:(dbxEntry * 4) + 4])[0]
  
!   def getValue(self, dbxIndex):
!     """Return the address of the n-th message."""
!     return self.dbxValues[dbxIndex]
  
  ###########################################################################
--- 158,204 ----
  
  class dbxTree:
!     """Stands for the tree which stores the messages in a given folder."""
  
!     TREE_NODE_SIZE = 0x27c          # size of a tree node
  
!     def __init__(self, dbxStream, dbxAddress, dbxValues):
!         """Reads the addresses of the stored messages."""
!         self.dbxValues = [i for i in range(dbxValues)]
!         # XXX : silly fix !
!         if dbxAddress > 0:
!             self.__readValues(dbxStream, 0, dbxAddress, 0, dbxValues)
  
!     def __readValues(self, dbxStream, dbxParent, dbxAddress, dbxPosition, dbxValues):
!         dbxStream.seek(dbxAddress)
!         dbxBuffer = dbxStream.read(dbxTree.TREE_NODE_SIZE)
  
!         count = 0
!         entries = ((self.getEntry(dbxBuffer, 4) >> 8) & 0xff)
  
!         if self.getEntry(dbxBuffer, 2) != 0:
!             self.__readValues(dbxStream, dbxAddress, self.getEntry(dbxBuffer, 2),
!                                          dbxPosition, self.getEntry(dbxBuffer, 5))
!             count += self.getEntry(dbxBuffer, 5)
  
!         for i in range(entries):
!             pos = 6 + i * 3
  
!             if self.getEntry(dbxBuffer, pos) != 0:
!                 count += 1
!                 value = dbxPosition + count
!                 self.dbxValues[value - 1] = self.getEntry(dbxBuffer, pos)
  
!             if self.getEntry(dbxBuffer, pos + 1) != 0:
!                 self.__readValues(dbxStream, dbxAddress, self.getEntry(dbxBuffer, pos + 1),
!                                              dbxPosition + count, self.getEntry(dbxBuffer, pos + 2))
!                 count += self.getEntry(dbxBuffer, pos + 2)
  
!     def getEntry(self, dbxBuffer, dbxEntry):
!         """Return the n-th entry as a long integer."""
!         return struct.unpack("L", dbxBuffer[dbxEntry * 4:(dbxEntry * 4) + 4])[0]
  
!     def getValue(self, dbxIndex):
!         """Return the address of the n-th message."""
!         return self.dbxValues[dbxIndex]
  
  ###########################################################################
***************
*** 207,324 ****
  
  class dbxIndexedInfo:
!   """
!     Messages and folders mailboxes contain the "message info" and "folders
!     info" entities.
!     These entities are indexed info sequences. This is their base class.
!   """
  
!   MAX_INDEX = 0x20        # max index
!   DT_NONE   = 0           # data type none
  
!   def __init__(self, dbxStream, dbxAddress):
!     """Reads the indexed infos from the passed stream."""
!     self.dbxBodyLength   = 0L
!     self.dbxObjectLength = 0L
!     self.dbxEntries      = 0L
!     self.dbxCounter      = 0L
!     self.dbxBuffer       = []
!     self.dbxIndexes      = 0L
!     self.dbxBegin        = [0L for i in range(dbxIndexedInfo.MAX_INDEX)]
!     self.dbxLength       = [i  for i in self.dbxBegin]
!     self.dbxAddress       = dbxAddress
!     self.__readIndexedInfo(dbxStream)
  
!   def __readIndexedInfo(self, dbxStream):
!     dbxStream.seek(self.dbxAddress)
!     temp = dbxStream.read(12)
  
!     self.dbxBodyLength   =  self.__getEntry(temp, 1)
!     self.dbxObjectLength =  self.__getEntry(temp, 2) & 0xffff
!     self.dbxEntries      = (self.__getEntry(temp, 2) >> 16) & 0xff
!     self.dbxCounter      = (self.__getEntry(temp, 1) >> 24) & 0xff
!     self.dbxBuffer       =  dbxStream.read(self.dbxBodyLength) # bytes array
  
!     isIndirect           = bool(0)                             # boolean
!     lastIndirect         = 0
!     data                 = self.dbxEntries << 2                # index within dbxBuffer
  
!     for i in range(self.dbxEntries):
!       value      = self.__getEntry(self.dbxBuffer, i)
!       isDirect   = value & 0x80
!       index      = value & 0x7f
!       value    >>= 8
!       if isDirect:
!         self.__setIndex(index, (i << 2) + 1, 3)
!       else:
!         self.__setIndex(index, data + value)
!         if isIndirect:
!           self.__setEnd(lastIndirect, data + value)
!           isIndirect   = bool(1)
!           lastIndirect = index
!       self.dbxIndexes |= 1 << index
  
!     if isIndirect:
!       self.__setEnd(lastIndirect, self.dbxBodyLength)
  
!   def __setIndex(self, dbxIndex, dbxBegin, dbxLength = 0):
!     if dbxIndex < dbxIndexedInfo.MAX_INDEX:
!       self.dbxBegin[dbxIndex] = dbxBegin
!       self.dbxLength[dbxIndex] = dbxLength
  
!   def __setEnd(self, dbxIndex, dbxEnd):
!     if dbxIndex < dbxIndexedInfo.MAX_INDEX:
!       self.dbxLength[dbxIndex] = dbxEnd - self.dbxBegin[dbxIndex]
  
!   def __getEntry(self, dbxBuffer, dbxEntry):
!     return struct.unpack("L", dbxBuffer[dbxEntry * 4:(dbxEntry * 4) + 4])[0]
  
!   def getIndexText(self, dbxIndex):
!     """Returns the description of the given indexed field."""
!     return ""
  
!   def getIndexDataType(self, dbxIndex):
!     """Returns the data type of the given index."""
!     return DT_NONE
  
!   def getValue(self, dbxIndex):
!     """Returns a tuple : (index in buffer of the info, length of the info)."""
!     return (self.dbxBegin[dbxIndex], self.dbxLength[dbxIndex])
  
!   def getValueAsLong(self, dbxIndex):
!     """Returns the indexed info as a long value."""
!     data, length = self.getValue(dbxIndex)
!     value  = 0
!     if data:
!       value = struct.unpack("L", self.dbxBuffer[data:data + 4])[0]
!       if length < 4:
!         value &= (1 << (length << 3)) - 1
!     return value
  
!   def getString(self, dbxIndex):
!     """Returns the indexed info as a string value."""
!     index = self.dbxBegin[dbxIndex]
!     end = index
!     for c in self.dbxBuffer[index:]:
!       if ord(c) == 0: break
!       end += 1
!     return self.dbxBuffer[index:end]
  
!   def getAddress(self):
!     return self.dbxAddress
  
!   def getBodyLength(self):
!     return self.dbxBodyLength
  
!   def getEntries(self):
!     return self.dbxEntries
  
!   def getCounter(self):
!     return self.dbxCounter
  
!   def getIndexes(self):
!     return self.dbxIndexes
  
!   def isIndexed(self, dbxIndex):
!     return self.dbxIndexes & (1 << dbxIndex)
  
  ###########################################################################
--- 207,324 ----
  
  class dbxIndexedInfo:
!     """
!       Messages and folders mailboxes contain the "message info" and "folders
!       info" entities.
!       These entities are indexed info sequences. This is their base class.
!     """
  
!     MAX_INDEX = 0x20        # max index
!     DT_NONE   = 0           # data type none
  
!     def __init__(self, dbxStream, dbxAddress):
!         """Reads the indexed infos from the passed stream."""
!         self.dbxBodyLength   = 0L
!         self.dbxObjectLength = 0L
!         self.dbxEntries      = 0L
!         self.dbxCounter      = 0L
!         self.dbxBuffer       = []
!         self.dbxIndexes      = 0L
!         self.dbxBegin        = [0L for i in range(dbxIndexedInfo.MAX_INDEX)]
!         self.dbxLength       = [i  for i in self.dbxBegin]
!         self.dbxAddress       = dbxAddress
!         self.__readIndexedInfo(dbxStream)
  
!     def __readIndexedInfo(self, dbxStream):
!         dbxStream.seek(self.dbxAddress)
!         temp = dbxStream.read(12)
  
!         self.dbxBodyLength   =  self.__getEntry(temp, 1)
!         self.dbxObjectLength =  self.__getEntry(temp, 2) & 0xffff
!         self.dbxEntries      = (self.__getEntry(temp, 2) >> 16) & 0xff
!         self.dbxCounter      = (self.__getEntry(temp, 1) >> 24) & 0xff
!         self.dbxBuffer       =  dbxStream.read(self.dbxBodyLength) # bytes array
  
!         isIndirect           = bool(0)                             # boolean
!         lastIndirect         = 0
!         data                 = self.dbxEntries << 2                # index within dbxBuffer
  
!         for i in range(self.dbxEntries):
!             value      = self.__getEntry(self.dbxBuffer, i)
!             isDirect   = value & 0x80
!             index      = value & 0x7f
!             value    >>= 8
!             if isDirect:
!                 self.__setIndex(index, (i << 2) + 1, 3)
!             else:
!                 self.__setIndex(index, data + value)
!                 if isIndirect:
!                     self.__setEnd(lastIndirect, data + value)
!                     isIndirect   = bool(1)
!                     lastIndirect = index
!             self.dbxIndexes |= 1 << index
  
!         if isIndirect:
!             self.__setEnd(lastIndirect, self.dbxBodyLength)
  
!     def __setIndex(self, dbxIndex, dbxBegin, dbxLength = 0):
!         if dbxIndex < dbxIndexedInfo.MAX_INDEX:
!             self.dbxBegin[dbxIndex] = dbxBegin
!             self.dbxLength[dbxIndex] = dbxLength
  
!     def __setEnd(self, dbxIndex, dbxEnd):
!         if dbxIndex < dbxIndexedInfo.MAX_INDEX:
!             self.dbxLength[dbxIndex] = dbxEnd - self.dbxBegin[dbxIndex]
  
!     def __getEntry(self, dbxBuffer, dbxEntry):
!         return struct.unpack("L", dbxBuffer[dbxEntry * 4:(dbxEntry * 4) + 4])[0]
  
!     def getIndexText(self, dbxIndex):
!         """Returns the description of the given indexed field."""
!         return ""
  
!     def getIndexDataType(self, dbxIndex):
!         """Returns the data type of the given index."""
!         return DT_NONE
  
!     def getValue(self, dbxIndex):
!         """Returns a tuple : (index in buffer of the info, length of the info)."""
!         return (self.dbxBegin[dbxIndex], self.dbxLength[dbxIndex])
  
!     def getValueAsLong(self, dbxIndex):
!         """Returns the indexed info as a long value."""
!         data, length = self.getValue(dbxIndex)
!         value  = 0
!         if data:
!             value = struct.unpack("L", self.dbxBuffer[data:data + 4])[0]
!             if length < 4:
!                 value &= (1 << (length << 3)) - 1
!         return value
  
!     def getString(self, dbxIndex):
!         """Returns the indexed info as a string value."""
!         index = self.dbxBegin[dbxIndex]
!         end = index
!         for c in self.dbxBuffer[index:]:
!             if ord(c) == 0: break
!             end += 1
!         return self.dbxBuffer[index:end]
  
!     def getAddress(self):
!         return self.dbxAddress
  
!     def getBodyLength(self):
!         return self.dbxBodyLength
  
!     def getEntries(self):
!         return self.dbxEntries
  
!     def getCounter(self):
!         return self.dbxCounter
  
!     def getIndexes(self):
!         return self.dbxIndexes
  
!     def isIndexed(self, dbxIndex):
!         return self.dbxIndexes & (1 << dbxIndex)
  
  ###########################################################################
***************
*** 327,377 ****
  
  class dbxMessageInfo(dbxIndexedInfo):
!   """
!     The message info structure inherits from the index info one. It just
!     defines extra constants which allow to access pertinent info.
!   """
  
!   MI_INDEX           = 0x0                   # index of the message
!   MI_FLAGS           = 0x1                   # the message flags
!   MI_MESSAGE_ADDRESS = 0x4                   # the address of the message
!   MI_SUBJECT         = 0x8                   # the subject of the message
  
!   # label of each indexed info
!   INDEX_LABEL = \
!   [ "message index"                , "flags"                          ,
!     "time message created/send"    , "body lines"                     ,
!     "message address"              , "original subject"               ,
!     "time message saved"           , "message id"                     ,
!     "subject"                      , "sender eMail address and name"  ,
!     "answered to message id"       , "server/newsgroup/message number",
!     "server"                       , "sender name"                    ,
!     "sender eMail address"         , "id 0f"                          ,
!     "message priority"             , "message text length"            ,
!     "time message created/received", "receiver name"                  ,
!     "receiver eMail address"       , "id 15"                          ,
!     "id 16"                        , "id 17"                          ,
!     "id 18"                        , "id 19"                          ,
!     "OE account name"              , "OE account registry key"        ,
!     "message text structure"       , "id 1d"                          ,
!     "id 1e"                        , "id 1f"                           ]
  
!   DT_NONE      = 0                    # index is none
!   DT_INT4      = 1                    # index is a long integer (32 bits)
!   DT_STRING    = 2                    # index is a string
!   DT_DATE_TIME = 3                    # index is date/time
!   DT_DATA      = 4                    # index is data
  
!   # the data type of each index
!   INDEX_DATA_TYPE = \
!   [ DT_INT4  , DT_INT4  , DT_DATE_TIME, DT_INT4  , DT_INT4  , DT_STRING, DT_DATE_TIME, DT_STRING,
!     DT_STRING, DT_STRING, DT_STRING   , DT_STRING, DT_STRING, DT_STRING, DT_STRING   , DT_NONE  ,
!     DT_INT4  , DT_INT4  , DT_DATE_TIME, DT_STRING, DT_STRING, DT_NONE  , DT_INT4     , DT_NONE  ,
!     DT_INT4  , DT_INT4  , DT_STRING   , DT_STRING, DT_DATA  , DT_NONE  , DT_NONE     , DT_NONE   ]
  
!   def getIndexText(self, dbxIndex):
!     return dbxMessageInfo.INDEX_LABEL[dbxIndex]
  
!   def getIndexDataType(self, dbxIndex):
!     return dbxMessageInfo.INDEX_DATA_TYPE[dbxIndex]
  
  ###########################################################################
--- 327,377 ----
  
  class dbxMessageInfo(dbxIndexedInfo):
!     """
!       The message info structure inherits from the index info one. It just
!       defines extra constants which allow to access pertinent info.
!     """
  
!     MI_INDEX           = 0x0                   # index of the message
!     MI_FLAGS           = 0x1                   # the message flags
!     MI_MESSAGE_ADDRESS = 0x4                   # the address of the message
!     MI_SUBJECT         = 0x8                   # the subject of the message
  
!     # label of each indexed info
!     INDEX_LABEL = \
!     [ "message index"                , "flags"                          ,
!       "time message created/send"    , "body lines"                     ,
!       "message address"              , "original subject"               ,
!       "time message saved"           , "message id"                     ,
!       "subject"                      , "sender eMail address and name"  ,
!       "answered to message id"       , "server/newsgroup/message number",
!       "server"                       , "sender name"                    ,
!       "sender eMail address"         , "id 0f"                          ,
!       "message priority"             , "message text length"            ,
!       "time message created/received", "receiver name"                  ,
!       "receiver eMail address"       , "id 15"                          ,
!       "id 16"                        , "id 17"                          ,
!       "id 18"                        , "id 19"                          ,
!       "OE account name"              , "OE account registry key"        ,
!       "message text structure"       , "id 1d"                          ,
!       "id 1e"                        , "id 1f"                           ]
  
!     DT_NONE      = 0                    # index is none
!     DT_INT4      = 1                    # index is a long integer (32 bits)
!     DT_STRING    = 2                    # index is a string
!     DT_DATE_TIME = 3                    # index is date/time
!     DT_DATA      = 4                    # index is data
  
!     # the data type of each index
!     INDEX_DATA_TYPE = \
!     [ DT_INT4  , DT_INT4  , DT_DATE_TIME, DT_INT4  , DT_INT4  , DT_STRING, DT_DATE_TIME, DT_STRING,
!       DT_STRING, DT_STRING, DT_STRING   , DT_STRING, DT_STRING, DT_STRING, DT_STRING   , DT_NONE  ,
!       DT_INT4  , DT_INT4  , DT_DATE_TIME, DT_STRING, DT_STRING, DT_NONE  , DT_INT4     , DT_NONE  ,
!       DT_INT4  , DT_INT4  , DT_STRING   , DT_STRING, DT_DATA  , DT_NONE  , DT_NONE     , DT_NONE   ]
  
!     def getIndexText(self, dbxIndex):
!         return dbxMessageInfo.INDEX_LABEL[dbxIndex]
  
!     def getIndexDataType(self, dbxIndex):
!         return dbxMessageInfo.INDEX_DATA_TYPE[dbxIndex]
  
  ###########################################################################
***************
*** 380,418 ****
  
  class dbxMessage:
!   def __init__(self, dbxStream, dbxAddress):
!     self.dbxAddress = dbxAddress
!     self.dbxText   = ""
!     self.dbxLength = 0L
!     self.__readMessageText(dbxStream)
  
!   def __getEntry(self, dbxBuffer, dbxEntry):
!     if len(dbxBuffer) < (dbxEntry * 4) + 4:
!       return None
!     return struct.unpack("L", dbxBuffer[dbxEntry * 4:(dbxEntry * 4) + 4])[0]
  
!   def __readMessageText(self, dbxStream):
!     address = self.dbxAddress
!     header = ""
  
!     while (address):
!       dbxStream.seek(address)
!       header = dbxStream.read(16)
  
!       self.dbxLength += self.__getEntry(header, 2)
!       address          = self.__getEntry(header, 3)
  
!     pos = ""
!     address = self.dbxAddress
  
!     while (address):
!       dbxStream.seek(address)
!       header  = dbxStream.read(16)
!       pos    += dbxStream.read(self.__getEntry(header, 2))
!       address  = self.__getEntry(header, 3)
  
!     self.dbxText = pos
  
!   def getText(self):
!     return self.dbxText
  
  ###########################################################################
--- 380,418 ----
  
  class dbxMessage:
!     def __init__(self, dbxStream, dbxAddress):
!         self.dbxAddress = dbxAddress
!         self.dbxText   = ""
!         self.dbxLength = 0L
!         self.__readMessageText(dbxStream)
  
!     def __getEntry(self, dbxBuffer, dbxEntry):
!         if len(dbxBuffer) < (dbxEntry * 4) + 4:
!             return None
!         return struct.unpack("L", dbxBuffer[dbxEntry * 4:(dbxEntry * 4) + 4])[0]
  
!     def __readMessageText(self, dbxStream):
!         address = self.dbxAddress
!         header = ""
  
!         while (address):
!             dbxStream.seek(address)
!             header = dbxStream.read(16)
  
!             self.dbxLength += self.__getEntry(header, 2)
!             address          = self.__getEntry(header, 3)
  
!         pos = ""
!         address = self.dbxAddress
  
!         while (address):
!             dbxStream.seek(address)
!             header  = dbxStream.read(16)
!             pos    += dbxStream.read(self.__getEntry(header, 2))
!             address  = self.__getEntry(header, 3)
  
!         self.dbxText = pos
  
!     def getText(self):
!         return self.dbxText
  
  ###########################################################################
***************
*** 421,488 ****
  
  if __name__ == '__main__':
!   import sys
!   import getopt
  
!   try:
!     opts, args = getopt.getopt(sys.argv[1:], 'hp')
!   except getopt.error, msg:
!     print >>sys.stderr, str(msg) + '\n\n' + __doc__
!     sys.exit()
  
!   print_message = False
!   for opt, arg in opts:
!     if opt == '-h':
!       print >>sys.stderr, __doc__
!       sys.exit()
!     elif opt == '-p':
!       print_message = True
  
!   if not args:
!     print "Please enter a directory with dbx files."
!     sys.exit()
  
!   MAILBOX_DIR = args[0]  
  
!   files = [os.path.join(MAILBOX_DIR, file) for file in \
!            os.listdir(MAILBOX_DIR) if os.path.splitext(file)[1] == '.dbx']
  
!   for file in files:
!     try:
!       print
!       print file
  
!       dbx = open(file, "rb", 0)
!       header = dbxFileHeader(dbx)
  
!       print "IS VALID DBX  :", header.isValid()
  
!       if header.isMessages():
!         info = dbxFileInfo(dbx, header.getEntry(dbxFileHeader.FH_FILE_INFO_LENGTH))
!         print "MAILBOX NAME  :", info.getFolderName()
!         print "CREATION TIME :", info.getCreationTime()
  
!         entries = header.getEntry(dbxFileHeader.FH_ENTRIES)
!         address  = header.getEntry(dbxFileHeader.FH_TREE_ROOT_NODE_PTR)
  
!         if address and entries:
!           tree = dbxTree(dbx, address, entries)
  
!           for i in range(entries):
!             address = tree.getValue(i)
!             messageInfo = dbxMessageInfo(dbx, address)
  
!             if messageInfo.isIndexed(dbxMessageInfo.MI_MESSAGE_ADDRESS):
!               messageAddress = messageInfo.getValueAsLong(dbxMessageInfo.MI_MESSAGE_ADDRESS)
!               message        = dbxMessage(dbx, messageAddress)
  
!               if print_message:
!                 print
!                 print "Message :", messageInfo.getString(dbxMessageInfo.MI_SUBJECT)
!                 print "=" * (len(messageInfo.getString(dbxMessageInfo.MI_SUBJECT)) + 9)
!                 print
!                 print message.getText()
  
!     except Exception, (strerror):
!       print strerror
  
!     dbx.close()
--- 421,488 ----
  
  if __name__ == '__main__':
!     import sys
!     import getopt
  
!     try:
!         opts, args = getopt.getopt(sys.argv[1:], 'hp')
!     except getopt.error, msg:
!         print >>sys.stderr, str(msg) + '\n\n' + __doc__
!         sys.exit()
  
!     print_message = False
!     for opt, arg in opts:
!         if opt == '-h':
!             print >>sys.stderr, __doc__
!             sys.exit()
!         elif opt == '-p':
!             print_message = True
  
!     if not args:
!         print "Please enter a directory with dbx files."
!         sys.exit()
  
!     MAILBOX_DIR = args[0]
  
!     files = [os.path.join(MAILBOX_DIR, file) for file in \
!              os.listdir(MAILBOX_DIR) if os.path.splitext(file)[1] == '.dbx']
  
!     for file in files:
!         try:
!             print
!             print file
  
!             dbx = open(file, "rb", 0)
!             header = dbxFileHeader(dbx)
  
!             print "IS VALID DBX  :", header.isValid()
  
!             if header.isMessages():
!                 info = dbxFileInfo(dbx, header.getEntry(dbxFileHeader.FH_FILE_INFO_LENGTH))
!                 print "MAILBOX NAME  :", info.getFolderName()
!                 print "CREATION TIME :", info.getCreationTime()
  
!                 entries = header.getEntry(dbxFileHeader.FH_ENTRIES)
!                 address  = header.getEntry(dbxFileHeader.FH_TREE_ROOT_NODE_PTR)
  
!                 if address and entries:
!                     tree = dbxTree(dbx, address, entries)
  
!                     for i in range(entries):
!                         address = tree.getValue(i)
!                         messageInfo = dbxMessageInfo(dbx, address)
  
!                         if messageInfo.isIndexed(dbxMessageInfo.MI_MESSAGE_ADDRESS):
!                             messageAddress = messageInfo.getValueAsLong(dbxMessageInfo.MI_MESSAGE_ADDRESS)
!                             message        = dbxMessage(dbx, messageAddress)
  
!                             if print_message:
!                                 print
!                                 print "Message :", messageInfo.getString(dbxMessageInfo.MI_SUBJECT)
!                                 print "=" * (len(messageInfo.getString(dbxMessageInfo.MI_SUBJECT)) + 9)
!                                 print
!                                 print message.getText()
  
!         except Exception, (strerror):
!             print strerror
  
!         dbx.close()

Index: smtpproxy.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/smtpproxy.py,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** smtpproxy.py	3 Oct 2003 05:17:43 -0000	1.4
--- smtpproxy.py	16 Dec 2003 05:06:34 -0000	1.5
***************
*** 61,65 ****
     work, but I don't really know.  Richie Hindle suggested something along
     these lines back in September '02.
!    
   o Suggestions?
  
--- 61,65 ----
     work, but I don't really know.  Richie Hindle suggested something along
     these lines back in September '02.
! 
   o Suggestions?
  
***************
*** 74,88 ****
  the smtpproxy can extract the id from the body of the message, if it
  is there.
!                                                         Header	Body
  *** Windows 2000 MUAs ***
  Eudora 5.2 Forward                                         *     *
  Eudora 5.2 Redirect                                              *
  Netscape Messenger (4.7) Forward (inline)                  *     *
! Netscape Messenger (4.7) Forward (quoted) Plain      	         *
! Netscape Messenger (4.7) Forward (quoted) HTML      	         *
! Netscape Messenger (4.7) Forward (quoted) Plain & HTML       	 *       
! Netscape Messenger (4.7) Forward (attachment) Plain 	   *     *	 
! Netscape Messenger (4.7) Forward (attachment) HTML  	   *	 *
! Netscape Messenger (4.7) Forward (attachment) Plain & HTML *  	 *
  Outlook Express 6 Forward HTML (Base64)                          *
  Outlook Express 6 Forward HTML (None)                            *
--- 74,88 ----
  the smtpproxy can extract the id from the body of the message, if it
  is there.
!                                                         Header  Body
  *** Windows 2000 MUAs ***
  Eudora 5.2 Forward                                         *     *
  Eudora 5.2 Redirect                                              *
  Netscape Messenger (4.7) Forward (inline)                  *     *
! Netscape Messenger (4.7) Forward (quoted) Plain                  *
! Netscape Messenger (4.7) Forward (quoted) HTML                   *
! Netscape Messenger (4.7) Forward (quoted) Plain & HTML           *
! Netscape Messenger (4.7) Forward (attachment) Plain        *     *
! Netscape Messenger (4.7) Forward (attachment) HTML         *     *
! Netscape Messenger (4.7) Forward (attachment) Plain & HTML *     *
  Outlook Express 6 Forward HTML (Base64)                          *
  Outlook Express 6 Forward HTML (None)                            *
***************
*** 92,96 ****
  Outlook Express 6 Forward Plain (QP)                             *
  Outlook Express 6 Forward Plain (uuencoded)                      *
! http://www.endymion.com/products/mailman Forward	             *
  M2 (Opera Mailer 7.01) Forward                                   *
  M2 (Opera Mailer 7.01) Redirect                            *     *
--- 92,96 ----
  Outlook Express 6 Forward Plain (QP)                             *
  Outlook Express 6 Forward Plain (uuencoded)                      *
! http://www.endymion.com/products/mailman Forward                     *
  M2 (Opera Mailer 7.01) Forward                                   *
  M2 (Opera Mailer 7.01) Redirect                            *     *
***************
*** 263,267 ****
            then do not forward the command on.  Otherwise forward verbatim.
  
!     Any other commands are merely passed on verbatim to the server.          
      """
  
--- 263,267 ----
            then do not forward the command on.  Otherwise forward verbatim.
  
!     Any other commands are merely passed on verbatim to the server.
      """
  
***************
*** 334,338 ****
              self.blockData = False
          return "%s:%s" % (command, ' '.join(args))
!         
      def onData(self, command, args):
          self.inData = True
--- 334,338 ----
              self.blockData = False
          return "%s:%s" % (command, ' '.join(args))
! 
      def onData(self, command, args):
          self.inData = True
***************
*** 357,361 ****
          self.state = state
          self.imap = imap
!     
      def extractSpambayesID(self, data):
          msg = message_from_string(data)
--- 357,361 ----
          self.state = state
          self.imap = imap
! 
      def extractSpambayesID(self, data):
          msg = message_from_string(data)
***************
*** 493,497 ****
                "different lengths!"
          sys.exit()
!     return servers, proxyPorts    
  
  def CreateProxies(servers, proxyPorts, trainer):
--- 493,497 ----
                "different lengths!"
          sys.exit()
!     return servers, proxyPorts
  
  def CreateProxies(servers, proxyPorts, trainer):

Index: storage.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/storage.py,v
retrieving revision 1.36
retrieving revision 1.37
diff -C2 -d -r1.36 -r1.37
*** storage.py	8 Oct 2003 04:04:35 -0000	1.36
--- storage.py	16 Dec 2003 05:06:34 -0000	1.37
***************
*** 238,242 ****
          self._write_state_key()
          self.db.sync()
!         
      def _write_state_key(self):
          self.db[self.statekey] = (classifier.PICKLE_VERSION,
--- 238,242 ----
          self._write_state_key()
          self.db.sync()
! 
      def _write_state_key(self):
          self.db[self.statekey] = (classifier.PICKLE_VERSION,
***************
*** 248,252 ****
          key."""
          self._write_state_key()
!     
      def _wordinfoget(self, word):
          if isinstance(word, unicode):
--- 248,252 ----
          key."""
          self._write_state_key()
! 
      def _wordinfoget(self, word):
          if isinstance(word, unicode):
***************
*** 340,344 ****
          '''Commit the current transaction - may commit at db or cursor'''
          raise NotImplementedError, "must be implemented in subclass"
!         
      def create_bayes(self):
          '''Create a new bayes table'''
--- 340,344 ----
          '''Commit the current transaction - may commit at db or cursor'''
          raise NotImplementedError, "must be implemented in subclass"
! 
      def create_bayes(self):
          '''Create a new bayes table'''
***************
*** 451,455 ****
  
          import psycopg
!         
          if options["globals", "verbose"]:
              print >> sys.stderr, 'Loading state from',self.db_name,'database'
--- 451,455 ----
  
          import psycopg
! 
          if options["globals", "verbose"]:
              print >> sys.stderr, 'Loading state from',self.db_name,'database'
***************
*** 463,467 ****
              self.db.rollback()
              self.create_bayes()
!         
          if self._has_key(self.statekey):
              row = self._get_row(self.statekey)
--- 463,467 ----
              self.db.rollback()
              self.create_bayes()
! 
          if self._has_key(self.statekey):
              row = self._get_row(self.statekey)
***************
*** 485,489 ****
      It is assumed that the database already exists, and that the mySQL
      server is currently running.'''
!  
      def __init__(self, data_source_name):
          self.table_definition = ("create table bayes ("
--- 485,489 ----
      It is assumed that the database already exists, and that the mySQL
      server is currently running.'''
! 
      def __init__(self, data_source_name):
          self.table_definition = ("create table bayes ("
***************
*** 522,526 ****
  
          import MySQLdb
!         
          if options["globals", "verbose"]:
              print >> sys.stderr, 'Loading state from',self.db_name,'database'
--- 522,526 ----
  
          import MySQLdb
! 
          if options["globals", "verbose"]:
              print >> sys.stderr, 'Loading state from',self.db_name,'database'
***************
*** 535,539 ****
              self.db.rollback()
              self.create_bayes()
!         
          if self._has_key(self.statekey):
              row = self._get_row(self.statekey)
--- 535,539 ----
              self.db.rollback()
              self.create_bayes()
! 
          if self._has_key(self.statekey):
              row = self._get_row(self.statekey)

Index: tokenizer.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/tokenizer.py,v
retrieving revision 1.21
retrieving revision 1.22
diff -C2 -d -r1.21 -r1.22
*** tokenizer.py	15 Dec 2003 09:20:33 -0000	1.21
--- tokenizer.py	16 Dec 2003 05:06:34 -0000	1.22
***************
*** 663,670 ****
      if 3 <= n <= maxword:
          yield word
!         
      elif n >= 3:
          # A long word.
!         
          # Don't want to skip embedded email addresses.
          # An earlier scheme also split up the y in x@y on '.'.  Not splitting
--- 663,670 ----
      if 3 <= n <= maxword:
          yield word
! 
      elif n >= 3:
          # A long word.
! 
          # Don't want to skip embedded email addresses.
          # An earlier scheme also split up the y in x@y on '.'.  Not splitting
***************
*** 816,820 ****
          # http://mail.python.org/pipermail/spambayes-dev/2003-September/001177.html
          yield "filename:<bogus>"
!         
      if 0:   # disabled; see comment before function
          x = msg.get('content-transfer-encoding')
--- 816,820 ----
          # http://mail.python.org/pipermail/spambayes-dev/2003-September/001177.html
          yield "filename:<bogus>"
! 
      if 0:   # disabled; see comment before function
          x = msg.get('content-transfer-encoding')
***************
*** 1174,1178 ****
                  elif valid_habeas == 9:
                      yield "x-habeas-swe:valid"
!                     
          # Subject:
          # Don't ignore case in Subject lines; e.g., 'free' versus 'FREE' is
--- 1174,1178 ----
                  elif valid_habeas == 9:
                      yield "x-habeas-swe:valid"
! 
          # Subject:
          # Don't ignore case in Subject lines; e.g., 'free' versus 'FREE' is



From tim_one at users.sourceforge.net  Tue Dec 16 00:06:36 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Tue Dec 16 00:07:00 2003
Subject: [Spambayes-checkins] spambayes/spambayes/test sb_test_support.py,
	1.1, 1.2 test_programs.py, 1.2, 1.3 test_smtpproxy.py, 1.1, 1.2
Message-ID: <E1AW7Po-0003Gl-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes/test
In directory sc8-pr-cvs1:/tmp/cvs-serv12349/spambayes/test

Modified Files:
	sb_test_support.py test_programs.py test_smtpproxy.py 
Log Message:
Whitespace normalization.


Index: sb_test_support.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/test/sb_test_support.py,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** sb_test_support.py	3 Dec 2003 00:34:17 -0000	1.1
--- sb_test_support.py	16 Dec 2003 05:06:34 -0000	1.2
***************
*** 29,35 ****
          import sb_server
      except ImportError:
!         # Scripts are usually in "spambayes/scripts" (for an 
          # installed SpamBayes, they appear to be in
!         # os.path.join(sys.prefix(), "scripts"), which we may like to 
          # leverage - however, these test scripts are not currently
          # installed).
--- 29,35 ----
          import sb_server
      except ImportError:
!         # Scripts are usually in "spambayes/scripts" (for an
          # installed SpamBayes, they appear to be in
!         # os.path.join(sys.prefix(), "scripts"), which we may like to
          # leverage - however, these test scripts are not currently
          # installed).

Index: test_programs.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/test/test_programs.py,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** test_programs.py	14 Dec 2003 01:12:08 -0000	1.2
--- test_programs.py	16 Dec 2003 05:06:34 -0000	1.3
***************
*** 132,136 ****
      except sb_server.AlreadyRunningException:
          return True
!         
  class TestServer(unittest.TestCase):
      def setUp(self):
--- 132,136 ----
      except sb_server.AlreadyRunningException:
          return True
! 
  class TestServer(unittest.TestCase):
      def setUp(self):
***************
*** 242,246 ****
              self.failUnless(not is_any_sb_server_running(),
                              "Should be no platform mutex held after stopping")
!     
  if __name__=='__main__':
      sb_test_support.unittest_main()
--- 242,246 ----
              self.failUnless(not is_any_sb_server_running(),
                              "Should be no platform mutex held after stopping")
! 
  if __name__=='__main__':
      sb_test_support.unittest_main()

Index: test_smtpproxy.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/test/test_smtpproxy.py,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** test_smtpproxy.py	24 Sep 2003 06:53:37 -0000	1.1
--- test_smtpproxy.py	16 Dec 2003 05:06:34 -0000	1.2
***************
*** 136,140 ****
          print "pushing", repr(data)
          Dibbler.BrighterAsyncChat.push(self, data)
!         
      def recv(self, buffer_size):
          """Asynchat override."""
--- 136,140 ----
          print "pushing", repr(data)
          Dibbler.BrighterAsyncChat.push(self, data)
! 
      def recv(self, buffer_size):
          """Asynchat override."""



From tim_one at users.sourceforge.net  Tue Dec 16 00:06:36 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Tue Dec 16 00:07:01 2003
Subject: [Spambayes-checkins] website/scripts striphtml.py,1.1.1.1,1.2
Message-ID: <E1AW7Po-0003HA-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/website/scripts
In directory sc8-pr-cvs1:/tmp/cvs-serv12349/website/scripts

Modified Files:
	striphtml.py 
Log Message:
Whitespace normalization.


Index: striphtml.py
===================================================================
RCS file: /cvsroot/spambayes/website/scripts/striphtml.py,v
retrieving revision 1.1.1.1
retrieving revision 1.2
diff -C2 -d -r1.1.1.1 -r1.2
*** striphtml.py	19 Sep 2002 08:40:55 -0000	1.1.1.1
--- striphtml.py	16 Dec 2003 05:06:34 -0000	1.2
***************
*** 30,60 ****
      data = fi.read()
      if eheadsearch.search(data) >= 0:
! 	i = eheadsearch.regs[0][1]
! 	head, data = data[:i], data[i:]
      else:
! 	head = ""
      bstart = bodysearch.search(data)
      if bstart < 0:
! 	bstart = 0
      else:
! 	head = head + data[:bstart]
! 	bstart = bstart + len(bodysearch.group(0))
      if bannersearch.search(data) >= 0:
! 	i, j = bannersearch.regs[0]
! 	print "banner", i, j, `data[i:j]`
! 	data = data[:i] + data[j:]
      end = ebodyearch.search(data, bstart)
      if end < 0:
! 	end = len(data)
      body = string.strip(data[bstart:end])
      if titlesearch.search(head) >= 0:
! 	title = titlesearch.group(1)
      elif h1search.search(body) >= 0:
! 	title = h1search.group(1)
      else:
! 	title = ""
      if title:
! 	title = string.join(string.split(title))
! 	fo.write("Title: %s\n" % title)
      fo.write("\n")
      fo.write(body)
--- 30,60 ----
      data = fi.read()
      if eheadsearch.search(data) >= 0:
!         i = eheadsearch.regs[0][1]
!         head, data = data[:i], data[i:]
      else:
!         head = ""
      bstart = bodysearch.search(data)
      if bstart < 0:
!         bstart = 0
      else:
!         head = head + data[:bstart]
!         bstart = bstart + len(bodysearch.group(0))
      if bannersearch.search(data) >= 0:
!         i, j = bannersearch.regs[0]
!         print "banner", i, j, `data[i:j]`
!         data = data[:i] + data[j:]
      end = ebodyearch.search(data, bstart)
      if end < 0:
!         end = len(data)
      body = string.strip(data[bstart:end])
      if titlesearch.search(head) >= 0:
!         title = titlesearch.group(1)
      elif h1search.search(body) >= 0:
!         title = h1search.group(1)
      else:
!         title = ""
      if title:
!         title = string.join(string.split(title))
!         fo.write("Title: %s\n" % title)
      fo.write("\n")
      fo.write(body)
***************
*** 66,79 ****
  def makedirs(dirname):
      if os.path.exists(dirname):
! 	return 1
      head, tail = os.path.split(dirname)
      if head:
! 	if not makedirs(head):
! 	    return 0
      try:
! 	os.mkdir(dirname, 0777)
! 	return 1
      except os.error:
! 	return 0
  
  def main():
--- 66,79 ----
  def makedirs(dirname):
      if os.path.exists(dirname):
!         return 1
      head, tail = os.path.split(dirname)
      if head:
!         if not makedirs(head):
!             return 0
      try:
!         os.mkdir(dirname, 0777)
!         return 1
      except os.error:
!         return 0
  
  def main():
***************
*** 81,105 ****
      prefix = ""
      for o, a in opts:
! 	if o == "-p": prefix = a
      if not args:
! 	strip(sys.stdin, sys.stdout)
      else:
! 	for file in args:
! 	    name, ext = os.path.splitext(file)
! 	    if ext == ".htp":
! 		error("file %s is already an HTML prototype" % name)
! 		continue
! 	    sys.stderr.write("Processing %s ...\n" % file)
! 	    htpname = prefix + name + ".htp"
! 	    dirname = os.path.dirname(htpname)
! 	    if dirname:
! 		if not makedirs(dirname):
! 		    error("can't create directory %s" % dirname)
! 		    continue
! 	    fi = open(file, "r")
! 	    fo = open(htpname, "w")
! 	    strip(fi, fo)
! 	    fi.close()
! 	    fo.close()
  
  if __name__ == '__main__':
--- 81,105 ----
      prefix = ""
      for o, a in opts:
!         if o == "-p": prefix = a
      if not args:
!         strip(sys.stdin, sys.stdout)
      else:
!         for file in args:
!             name, ext = os.path.splitext(file)
!             if ext == ".htp":
!                 error("file %s is already an HTML prototype" % name)
!                 continue
!             sys.stderr.write("Processing %s ...\n" % file)
!             htpname = prefix + name + ".htp"
!             dirname = os.path.dirname(htpname)
!             if dirname:
!                 if not makedirs(dirname):
!                     error("can't create directory %s" % dirname)
!                     continue
!             fi = open(file, "r")
!             fo = open(htpname, "w")
!             strip(fi, fo)
!             fi.close()
!             fo.close()
  
  if __name__ == '__main__':



From tim_one at users.sourceforge.net  Tue Dec 16 00:06:36 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Tue Dec 16 00:07:03 2003
Subject: [Spambayes-checkins] 
	spambayes/spambayes/resources __init__.py, 1.2, 1.3
Message-ID: <E1AW7Po-0003Gf-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes/resources
In directory sc8-pr-cvs1:/tmp/cvs-serv12349/spambayes/resources

Modified Files:
	__init__.py 
Log Message:
Whitespace normalization.


Index: __init__.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/resources/__init__.py,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** __init__.py	5 Sep 2003 01:15:28 -0000	1.2
--- __init__.py	16 Dec 2003 05:06:34 -0000	1.3
***************
*** 6,36 ****
  """
  try:
! 	__file__
  except NameError:
! 	pass
  else:
! 	import os
! 	if os.path.splitext(os.path.basename( __file__ ))[0] == "__init__":
! 		try:
! 			from resourcepackage import package, defaultgenerators
! 			generators = defaultgenerators.generators.copy()
! 			
! 			### CUSTOMISATION POINT
! 			## import specialised generators here, such as for wxPython
! 			#from resourcepackage import wxgenerators
! 			#generators.update( wxgenerators.generators )
! 		except ImportError:
! 			pass
! 		else:
! 			package = package.Package(
! 				packageName = __name__,
! 				directory = os.path.dirname( os.path.abspath(__file__) ),
! 				generators = generators,
! 			)
! 			package.scan(
! 				### CUSTOMISATION POINT
! 				## force true -> always re-loads from external files, otherwise
! 				## only reloads if the file is newer than the generated .py file.
! 				# force = 1, 
! 			)
! 		
--- 6,35 ----
  """
  try:
!     __file__
  except NameError:
!     pass
  else:
!     import os
!     if os.path.splitext(os.path.basename( __file__ ))[0] == "__init__":
!         try:
!             from resourcepackage import package, defaultgenerators
!             generators = defaultgenerators.generators.copy()
! 
!             ### CUSTOMISATION POINT
!             ## import specialised generators here, such as for wxPython
!             #from resourcepackage import wxgenerators
!             #generators.update( wxgenerators.generators )
!         except ImportError:
!             pass
!         else:
!             package = package.Package(
!                     packageName = __name__,
!                     directory = os.path.dirname( os.path.abspath(__file__) ),
!                     generators = generators,
!             )
!             package.scan(
!                     ### CUSTOMISATION POINT
!                     ## force true -> always re-loads from external files, otherwise
!                     ## only reloads if the file is newer than the generated .py file.
!                     # force = 1,
!             )



From tim_one at users.sourceforge.net  Tue Dec 16 00:06:35 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Tue Dec 16 00:07:05 2003
Subject: [Spambayes-checkins] spambayes/contrib mod_spambayes.py,1.2,1.3
Message-ID: <E1AW7Pn-0003GF-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/contrib
In directory sc8-pr-cvs1:/tmp/cvs-serv12349/contrib

Modified Files:
	mod_spambayes.py 
Log Message:
Whitespace normalization.


Index: mod_spambayes.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/contrib/mod_spambayes.py,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** mod_spambayes.py	4 May 2003 03:10:36 -0000	1.2
--- mod_spambayes.py	16 Dec 2003 05:06:33 -0000	1.3
***************
*** 25,29 ****
                  print "text:", s[0:40], "...", s[-40:]
                  return "not authorized"
!        return s
  
  from proxy3_util import *
--- 25,29 ----
                  print "text:", s[0:40], "...", s[-40:]
                  return "not authorized"
!     return s
  
  from proxy3_util import *



From tim_one at users.sourceforge.net  Tue Dec 16 00:06:35 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Tue Dec 16 00:07:07 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000/sandbox dump_props.py,
	1.11, 1.12 extract_bad_msg_from_log.py, 1.1,
	1.2 extract_prop.py, 1.1, 1.2 find_dupe_props.py, 1.1,
	1.2 set_read_flag.py, 1.1, 1.2
Message-ID: <E1AW7Pn-0003G5-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000/sandbox
In directory sc8-pr-cvs1:/tmp/cvs-serv12349/Outlook2000/sandbox

Modified Files:
	dump_props.py extract_bad_msg_from_log.py extract_prop.py 
	find_dupe_props.py set_read_flag.py 
Log Message:
Whitespace normalization.


Index: dump_props.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/sandbox/dump_props.py,v
retrieving revision 1.11
retrieving revision 1.12
diff -C2 -d -r1.11 -r1.12
*** dump_props.py	10 Dec 2003 05:06:23 -0000	1.11
--- dump_props.py	16 Dec 2003 05:06:33 -0000	1.12
***************
*** 140,144 ****
                                      shorten, get_large_props)
      print "%-20s: %s" % (prop_name, prop_repr)
!     
  def usage(driver, extra = None):
      folder_doc = driver.GetFolderNameDoc()
--- 140,144 ----
                                      shorten, get_large_props)
      print "%-20s: %s" % (prop_name, prop_repr)
! 
  def usage(driver, extra = None):
      folder_doc = driver.GetFolderNameDoc()

Index: extract_bad_msg_from_log.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/sandbox/extract_bad_msg_from_log.py,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** extract_bad_msg_from_log.py	19 Aug 2003 13:32:29 -0000	1.1
--- extract_bad_msg_from_log.py	16 Dec 2003 05:06:33 -0000	1.2
***************
*** 21,25 ****
              msg_str = msg_str.replace("\r\n", "\n")
              sys.stdout.write(msg_str)
!         
      inname = sys.argv[1]
  
--- 21,25 ----
              msg_str = msg_str.replace("\r\n", "\n")
              sys.stdout.write(msg_str)
! 
      inname = sys.argv[1]
  

Index: extract_prop.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/sandbox/extract_prop.py,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** extract_prop.py	1 Sep 2003 06:04:35 -0000	1.1
--- extract_prop.py	16 Dec 2003 05:06:33 -0000	1.2
***************
*** 17,21 ****
              propIds = obj.GetIDsFromNames(props, 0)
              prop = mapitags.PROP_TAG( mapitags.PT_UNSPECIFIED, mapitags.PROP_ID(propIds[0]))
!     
      hr, data = item.GetProps((prop,), 0)
      prop_tag, prop_val = data[0]
--- 17,21 ----
              propIds = obj.GetIDsFromNames(props, 0)
              prop = mapitags.PROP_TAG( mapitags.PT_UNSPECIFIED, mapitags.PROP_ID(propIds[0]))
! 
      hr, data = item.GetProps((prop,), 0)
      prop_tag, prop_val = data[0]
***************
*** 35,49 ****
      elif mapitags.PROP_TYPE(prop_tag)==mapitags.PT_ERROR and \
           prop_val in [mapi.MAPI_E_NOT_ENOUGH_MEMORY,'MAPI_E_NOT_ENOUGH_MEMORY']:
!             prop_tag = mapitags.PROP_TAG(mapitags.PT_BINARY, mapitags.PROP_ID(prop_tag))
!             stream = item.OpenProperty(prop_tag,
!                                         pythoncom.IID_IStream,
!                                         0, 0)
!             chunks = []
!             while 1:
!                 chunk = stream.Read(4096)
!                 if not chunk:
!                     break
!                 chunks.append(chunk)
!             prop_val = "".join(chunks)
      outfile.write(prop_val)
  
--- 35,49 ----
      elif mapitags.PROP_TYPE(prop_tag)==mapitags.PT_ERROR and \
           prop_val in [mapi.MAPI_E_NOT_ENOUGH_MEMORY,'MAPI_E_NOT_ENOUGH_MEMORY']:
!         prop_tag = mapitags.PROP_TAG(mapitags.PT_BINARY, mapitags.PROP_ID(prop_tag))
!         stream = item.OpenProperty(prop_tag,
!                                     pythoncom.IID_IStream,
!                                     0, 0)
!         chunks = []
!         while 1:
!             chunk = stream.Read(4096)
!             if not chunk:
!                 break
!             chunks.append(chunk)
!         prop_val = "".join(chunks)
      outfile.write(prop_val)
  

Index: find_dupe_props.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/sandbox/find_dupe_props.py,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** find_dupe_props.py	28 Jul 2003 03:24:16 -0000	1.1
--- find_dupe_props.py	16 Dec 2003 05:06:33 -0000	1.2
***************
*** 102,106 ****
              print details
              sys.exit(1)
!     
          FindDupeProps(driver, folder, args[0], dupe_dict)
      DumpDupes(dupe_dict)
--- 102,106 ----
              print details
              sys.exit(1)
! 
          FindDupeProps(driver, folder, args[0], dupe_dict)
      DumpDupes(dupe_dict)

Index: set_read_flag.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/sandbox/set_read_flag.py,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** set_read_flag.py	17 Jun 2003 02:04:43 -0000	1.1
--- set_read_flag.py	16 Dec 2003 05:06:33 -0000	1.2
***************
*** 1,4 ****
  from __future__ import generators
! # Set items to read/unread 
  
  import pythoncom
--- 1,4 ----
  from __future__ import generators
! # Set items to read/unread
  
  import pythoncom



From tim_one at users.sourceforge.net  Tue Dec 16 00:06:35 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Tue Dec 16 00:07:09 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000/installer crank.py, 1.2,
	1.3 spambayes_addin.py, 1.1, 1.2
Message-ID: <E1AW7Pn-0003Fw-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000/installer
In directory sc8-pr-cvs1:/tmp/cvs-serv12349/Outlook2000/installer

Modified Files:
	crank.py spambayes_addin.py 
Log Message:
Whitespace normalization.


Index: crank.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/installer/crank.py,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** crank.py	10 Aug 2003 07:06:12 -0000	1.2
--- crank.py	16 Dec 2003 05:06:33 -0000	1.3
***************
*** 41,45 ****
          return 1
      return 0
!     
  if __name__=='__main__':
      main()
--- 41,45 ----
          return 1
      return 0
! 
  if __name__=='__main__':
      main()

Index: spambayes_addin.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/installer/spambayes_addin.py,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** spambayes_addin.py	20 Feb 2003 08:16:31 -0000	1.1
--- spambayes_addin.py	16 Dec 2003 05:06:33 -0000	1.2
***************
*** 34,38 ****
              DllUnregisterServer()
              break
!         
          # MS seems to like /automate to run the class factories.
          if string.find(arg, "/automate") > -1:
--- 34,38 ----
              DllUnregisterServer()
              break
! 
          # MS seems to like /automate to run the class factories.
          if string.find(arg, "/automate") > -1:



From tim_one at users.sourceforge.net  Tue Dec 16 00:06:36 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Tue Dec 16 00:07:10 2003
Subject: [Spambayes-checkins] spambayes/scripts sb_dbexpimp.py, 1.3,
	1.4 sb_imapfilter.py, 1.16, 1.17 sb_notesfilter.py, 1.4,
	1.5 sb_pop3dnd.py, 1.3, 1.4 sb_server.py, 1.13, 1.14
Message-ID: <E1AW7Po-0003GQ-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/scripts
In directory sc8-pr-cvs1:/tmp/cvs-serv12349/scripts

Modified Files:
	sb_dbexpimp.py sb_imapfilter.py sb_notesfilter.py 
	sb_pop3dnd.py sb_server.py 
Log Message:
Whitespace normalization.


Index: sb_dbexpimp.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/scripts/sb_dbexpimp.py,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** sb_dbexpimp.py	26 Nov 2003 00:12:44 -0000	1.3
--- sb_dbexpimp.py	16 Dec 2003 05:06:33 -0000	1.4
***************
*** 11,35 ****
      a spambayes database into/from a flat file.  This is useful in a number
      of scenarios.
!     
      Platform portability of database - flat files can be exported and
      imported across platforms (winduhs and linux, for example)
!     
      Database implementation changes - databases can survive database
      implementation upgrades or new database implementations.  For example,
      if a dbm implementation changes between python x.y and python x.y+1...
!     
      Database reorganization - an export followed by an import reorgs an
!     existing database, <theoretically> improving performance, at least in 
      some database implementations
!     
      Database sharing - it is possible to distribute particular databases
      for research purposes, database sharing purposes, or for new users to
      have a 'seed' database to start with.
!     
      Database merging - multiple databases can be merged into one quite easily
      by simply not specifying -n on an import.  This will add the two database
      nham and nspams together (assuming the two databases do not share corpora)
      and for wordinfo conflicts, will add spamcount and hamcount together.
!     
      Spambayes software release migration - an export can be executed before
      a release upgrade, as part of the installation script.  Then, after the
--- 11,35 ----
      a spambayes database into/from a flat file.  This is useful in a number
      of scenarios.
! 
      Platform portability of database - flat files can be exported and
      imported across platforms (winduhs and linux, for example)
! 
      Database implementation changes - databases can survive database
      implementation upgrades or new database implementations.  For example,
      if a dbm implementation changes between python x.y and python x.y+1...
! 
      Database reorganization - an export followed by an import reorgs an
!     existing database, <theoretically> improving performance, at least in
      some database implementations
! 
      Database sharing - it is possible to distribute particular databases
      for research purposes, database sharing purposes, or for new users to
      have a 'seed' database to start with.
! 
      Database merging - multiple databases can be merged into one quite easily
      by simply not specifying -n on an import.  This will add the two database
      nham and nspams together (assuming the two databases do not share corpora)
      and for wordinfo conflicts, will add spamcount and hamcount together.
! 
      Spambayes software release migration - an export can be executed before
      a release upgrade, as part of the installation script.  Then, after the
***************
*** 37,43 ****
      effectively preserve existing training.  This eliminates the need for
      retraining every time a release is installed.
!     
      Others?  I'm sure I haven't thought of everything...
!     
  Usage:
      sb_dbexpimp [options]
--- 37,43 ----
      effectively preserve existing training.  This eliminates the need for
      retraining every time a release is installed.
! 
      Others?  I'm sure I haven't thought of everything...
! 
  Usage:
      sb_dbexpimp [options]
***************
*** 61,75 ****
      Export pickled mybayes.db into mybayes.db.export as a csv flat file
          sb_dbexpimp -e -d mybayes.db -f mybayes.db.export
!         
      Import mybayes.eb.export into a new DBM mybayes.db
          sb_dbexpimp -i -D mybayes.db -f mybayes.db.export
!        
      Export, then import (reorganize) new pickled mybayes.db
          sb_dbexpimp -e -i -n -d mybayes.db -f mybayes.db.export
!         
      Convert a bayes database from pickle to DBM
          sb_dbexpimp -e -d abayes.db -f abayes.export
          sb_dbexpimp -i -D abayes.db -f abayes.export
!         
      Create a new database (newbayes.db) from two
          databases (abayes.db, bbayes.db)
--- 61,75 ----
      Export pickled mybayes.db into mybayes.db.export as a csv flat file
          sb_dbexpimp -e -d mybayes.db -f mybayes.db.export
! 
      Import mybayes.eb.export into a new DBM mybayes.db
          sb_dbexpimp -i -D mybayes.db -f mybayes.db.export
! 
      Export, then import (reorganize) new pickled mybayes.db
          sb_dbexpimp -e -i -n -d mybayes.db -f mybayes.db.export
! 
      Convert a bayes database from pickle to DBM
          sb_dbexpimp -e -d abayes.db -f abayes.export
          sb_dbexpimp -i -D abayes.db -f abayes.export
! 
      Create a new database (newbayes.db) from two
          databases (abayes.db, bbayes.db)
***************
*** 97,101 ****
      # Maintain compatibility with Python 2.2
      True, False = 1, 0
!     
  import spambayes.storage
  from spambayes.Options import options
--- 97,101 ----
      # Maintain compatibility with Python 2.2
      True, False = 1, 0
! 
  import spambayes.storage
  from spambayes.Options import options
***************
*** 126,140 ****
      except IOError, e:
          if e.errno != errno.ENOENT:
!            raise
!        
      nham = bayes.nham;
      nspam = bayes.nspam;
!     
      print "Exporting database %s to file %s" % (dbFN, outFN)
      print "Database has %s ham, %s spam, and %s words" \
              % (nham, nspam, len(words))
!     
      fp.write("%s,%s,\n" % (nham, nspam))
!     
      for word in words:
          wi = bayes._wordinfoget(word)
--- 126,140 ----
      except IOError, e:
          if e.errno != errno.ENOENT:
!             raise
! 
      nham = bayes.nham;
      nspam = bayes.nspam;
! 
      print "Exporting database %s to file %s" % (dbFN, outFN)
      print "Database has %s ham, %s spam, and %s words" \
              % (nham, nspam, len(words))
! 
      fp.write("%s,%s,\n" % (nham, nspam))
! 
      for word in words:
          wi = bayes._wordinfoget(word)
***************
*** 143,147 ****
          word = uquote(word)
          fp.write("%s`%s`%s`\n" % (word, hamcount, spamcount))
!         
      fp.close()
  
--- 143,147 ----
          word = uquote(word)
          fp.write("%s`%s`%s`\n" % (word, hamcount, spamcount))
! 
      fp.close()
  
***************
*** 154,158 ****
              if e.errno != 2:     # errno.<WHAT>
                  raise
!                 
          try:
              os.unlink(dbFN+".dat")
--- 154,158 ----
              if e.errno != 2:     # errno.<WHAT>
                  raise
! 
          try:
              os.unlink(dbFN+".dat")
***************
*** 160,164 ****
              if e.errno != 2:     # errno.<WHAT>
                  raise
!                 
          try:
              os.unlink(dbFN+".dir")
--- 160,164 ----
              if e.errno != 2:     # errno.<WHAT>
                  raise
! 
          try:
              os.unlink(dbFN+".dir")
***************
*** 166,170 ****
              if e.errno != 2:     # errno.<WHAT>
                  raise
!                 
      if useDBM:
          bayes = spambayes.storage.DBDictClassifier(dbFN)
--- 166,170 ----
              if e.errno != 2:     # errno.<WHAT>
                  raise
! 
      if useDBM:
          bayes = spambayes.storage.DBDictClassifier(dbFN)
***************
*** 176,184 ****
      except IOError, e:
          if e.errno != errno.ENOENT:
!            raise
!     
      nline = fp.readline()
      (nham, nspam, junk) = re.split(',', nline)
!  
      if newDBM:
          bayes.nham = int(nham)
--- 176,184 ----
      except IOError, e:
          if e.errno != errno.ENOENT:
!             raise
! 
      nline = fp.readline()
      (nham, nspam, junk) = re.split(',', nline)
! 
      if newDBM:
          bayes.nham = int(nham)
***************
*** 187,204 ****
          bayes.nham += int(nham)
          bayes.nspam += int(nspam)
!     
      if newDBM:
          impType = "Importing"
      else:
          impType = "Merging"
!   
      print "%s database %s using file %s" % (impType, dbFN, inFN)
  
      lines = fp.readlines()
!     
      for line in lines:
          (word, hamcount, spamcount, junk) = re.split('`', line)
          word = uunquote(word)
!        
          try:
              wi = bayes.wordinfo[word]
--- 187,204 ----
          bayes.nham += int(nham)
          bayes.nspam += int(nspam)
! 
      if newDBM:
          impType = "Importing"
      else:
          impType = "Merging"
! 
      print "%s database %s using file %s" % (impType, dbFN, inFN)
  
      lines = fp.readlines()
! 
      for line in lines:
          (word, hamcount, spamcount, junk) = re.split('`', line)
          word = uunquote(word)
! 
          try:
              wi = bayes.wordinfo[word]
***************
*** 208,212 ****
          wi.hamcount += int(hamcount)
          wi.spamcount += int(spamcount)
!                
          bayes._wordinfoset(word, wi)
  
--- 208,212 ----
          wi.hamcount += int(hamcount)
          wi.spamcount += int(spamcount)
! 
          bayes._wordinfoset(word, wi)
  
***************
*** 217,221 ****
      bayes.store()
      print "Finished storing database"
!     
      if useDBM:
          words = bayes.db.keys()
--- 217,221 ----
      bayes.store()
      print "Finished storing database"
! 
      if useDBM:
          words = bayes.db.keys()
***************
*** 223,227 ****
      else:
          words = bayes.wordinfo.keys()
!         
      print "Database has %s ham, %s spam, and %s words" \
             % (bayes.nham, bayes.nspam, len(words))
--- 223,227 ----
      else:
          words = bayes.wordinfo.keys()
! 
      print "Database has %s ham, %s spam, and %s words" \
             % (bayes.nham, bayes.nspam, len(words))

Index: sb_imapfilter.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/scripts/sb_imapfilter.py,v
retrieving revision 1.16
retrieving revision 1.17
diff -C2 -d -r1.16 -r1.17
*** sb_imapfilter.py	15 Dec 2003 09:15:19 -0000	1.16
--- sb_imapfilter.py	16 Dec 2003 05:06:33 -0000	1.17
***************
*** 10,15 ****
      sb_imapfilter [options]
  
! 	note: option values with spaces in them must be enclosed
! 	      in double quotes
  
          options:
--- 10,15 ----
      sb_imapfilter [options]
  
!         note: option values with spaces in them must be enclosed
!               in double quotes
  
          options:
***************
*** 31,35 ****
      Classify inbox, with dbm database
          sb_imapfilter -c -D bayes.db
!         
      Train Spam and Ham, then classify inbox, with dbm database
          sb_imapfilter -t -c -D bayes.db
--- 31,35 ----
      Classify inbox, with dbm database
          sb_imapfilter -c -D bayes.db
! 
      Train Spam and Ham, then classify inbox, with dbm database
          sb_imapfilter -t -c -D bayes.db
***************
*** 53,57 ****
        mail flagged as such, *or* if you set the imap_expunge option to
        True, then this mail will be irretrievably lost.
!     
  To Do:
      o IMAPMessage and IMAPFolder currently carry out very simple checks
--- 53,57 ----
        mail flagged as such, *or* if you set the imap_expunge option to
        True, then this mail will be irretrievably lost.
! 
  To Do:
      o IMAPMessage and IMAPFolder currently carry out very simple checks
***************
*** 181,185 ****
  class IMAPSession(BaseIMAP):
      '''A class extending the IMAP4 class, with a few optimizations'''
!     
      def __init__(self, server, port, debug=0, do_expunge=False):
          try:
--- 181,185 ----
  class IMAPSession(BaseIMAP):
      '''A class extending the IMAP4 class, with a few optimizations'''
! 
      def __init__(self, server, port, debug=0, do_expunge=False):
          try:
***************
*** 212,216 ****
                  raise
          self.logged_in = True
!     
      def logout(self):
          # sign off
--- 212,216 ----
                  raise
          self.logged_in = True
! 
      def logout(self):
          # sign off
***************
*** 230,234 ****
                          self.expunge()
          BaseIMAP.logout(self)  # superclass logout
!         
      def SelectFolder(self, folder):
          '''A method to point ensuing imap operations at a target folder'''
--- 230,234 ----
                          self.expunge()
          BaseIMAP.logout(self)  # superclass logout
! 
      def SelectFolder(self, folder):
          '''A method to point ensuing imap operations at a target folder'''
***************
*** 587,591 ****
                                                    move_opt_name]))
                      msg.Save()
!         return num_trained                
  
      def Filter(self, classifier, spamfolder, unsurefolder):
--- 587,591 ----
                                                    move_opt_name]))
                      msg.Save()
!         return num_trained
  
      def Filter(self, classifier, spamfolder, unsurefolder):
***************
*** 621,629 ****
          self.unsure_folder = IMAPFolder(options["imap", "unsure_folder"])
          self.classifier = classifier
!         
      def Train(self):
          if options["globals", "verbose"]:
              t = time.time()
!             
          total_ham_trained = 0
          total_spam_trained = 0
--- 621,629 ----
          self.unsure_folder = IMAPFolder(options["imap", "unsure_folder"])
          self.classifier = classifier
! 
      def Train(self):
          if options["globals", "verbose"]:
              t = time.time()
! 
          total_ham_trained = 0
          total_spam_trained = 0
***************
*** 657,661 ****
          if total_ham_trained or total_spam_trained:
              self.classifier.store()
!         
          if options["globals", "verbose"]:
              print "Training took %s seconds, %s messages were trained" \
--- 657,661 ----
          if total_ham_trained or total_spam_trained:
              self.classifier.store()
! 
          if options["globals", "verbose"]:
              print "Training took %s seconds, %s messages were trained" \
***************
*** 673,677 ****
          imap.SelectFolder(self.spam_folder.name)
          imap.SelectFolder(self.unsure_folder.name)
!             
          for filter_folder in options["imap", "filter_folders"]:
              # Select the folder to make sure it exists
--- 673,677 ----
          imap.SelectFolder(self.spam_folder.name)
          imap.SelectFolder(self.unsure_folder.name)
! 
          for filter_folder in options["imap", "filter_folders"]:
              # Select the folder to make sure it exists
***************
*** 682,686 ****
              for key in count.keys():
                  count[key] += subcount.get(key, 0)
!  
          if options["globals", "verbose"]:
              if count is not None:
--- 682,686 ----
              for key in count.keys():
                  count[key] += subcount.get(key, 0)
! 
          if options["globals", "verbose"]:
              if count is not None:
***************
*** 689,693 ****
              print "Classifying took", time.time() - t, "seconds."
  
!  
  def run():
      global imap
--- 689,693 ----
              print "Classifying took", time.time() - t, "seconds."
  
! 
  def run():
      global imap
***************
*** 751,755 ****
  
      bdbname = os.path.expanduser(bdbname)
!     
      if options["globals", "verbose"]:
          print "Loading database %s..." % (bdbname),
--- 751,755 ----
  
      bdbname = os.path.expanduser(bdbname)
! 
      if options["globals", "verbose"]:
          print "Loading database %s..." % (bdbname),
***************
*** 758,762 ****
  
      if options["globals", "verbose"]:
!         print "Done."            
  
      if options["imap", "server"]:
--- 758,762 ----
  
      if options["globals", "verbose"]:
!         print "Done."
  
      if options["imap", "server"]:
***************
*** 817,821 ****
  
              imap.logout()
!             
              if sleepTime:
                  time.sleep(sleepTime)
--- 817,821 ----
  
              imap.logout()
! 
              if sleepTime:
                  time.sleep(sleepTime)

Index: sb_notesfilter.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/scripts/sb_notesfilter.py,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** sb_notesfilter.py	10 Nov 2003 20:21:52 -0000	1.4
--- sb_notesfilter.py	16 Dec 2003 05:06:33 -0000	1.5
***************
*** 10,14 ****
      database.  The Notes client must be running when this process is
      executed.
!     
      It requires a Notes folder, named as a parameter, with four
      subfolders:
--- 10,14 ----
      database.  The Notes client must be running when this process is
      executed.
! 
      It requires a Notes folder, named as a parameter, with four
      subfolders:
***************
*** 25,29 ****
      3. Replicate (-r option)
      4. Classify the inbox (-c option)
!         
      Mail that is to be trained as spam should be manually moved to
      that folder by the user. Likewise mail that is to be trained as
--- 25,29 ----
      3. Replicate (-r option)
      4. Classify the inbox (-c option)
! 
      Mail that is to be trained as spam should be manually moved to
      that folder by the user. Likewise mail that is to be trained as
***************
*** 65,69 ****
      which if allowed to continue, would cause Spambayes to tend to
      classify everything as Spam.
!     
      Because there is no programmatic way to determine if a particular
      mail has been previously processed by this classification program,
--- 65,69 ----
      which if allowed to continue, would cause Spambayes to tend to
      classify everything as Spam.
! 
      Because there is no programmatic way to determine if a particular
      mail has been previously processed by this classification program,
***************
*** 80,85 ****
      sb_notesfilter [options]
  
! 	note: option values with spaces in them must be enclosed
! 	      in double quotes
  
          options:
--- 80,85 ----
      sb_notesfilter [options]
  
!         note: option values with spaces in them must be enclosed
!               in double quotes
  
          options:
***************
*** 108,118 ****
      Replicate and classify inbox
          sb_notesfilter -c -d notesbayes -r mynoteserv -l mail.nsf -f Spambayes
!         
      Train Spam and Ham, then classify inbox
          sb_notesfilter -t -c -d notesbayes -l mail.nsf -f Spambayes
!     
!     Replicate, then classify inbox      
          sb_notesfilter -c -d test7 -l mail.nsf -r nynoteserv -f Spambayes
!  
  To Do:
      o Dump/purge notesindex file
--- 108,118 ----
      Replicate and classify inbox
          sb_notesfilter -c -d notesbayes -r mynoteserv -l mail.nsf -f Spambayes
! 
      Train Spam and Ham, then classify inbox
          sb_notesfilter -t -c -d notesbayes -l mail.nsf -f Spambayes
! 
!     Replicate, then classify inbox
          sb_notesfilter -c -d test7 -l mail.nsf -r nynoteserv -f Spambayes
! 
  To Do:
      o Dump/purge notesindex file
***************
*** 159,163 ****
      else:
          firsttime = 0
!         
      docstomove = []
      numham = 0
--- 159,163 ----
      else:
          firsttime = 0
! 
      docstomove = []
      numham = 0
***************
*** 165,174 ****
      numuns = 0
      numdocs = 0
!     
      doc = v.GetFirstDocument()
      while doc:
          nid = doc.NOTEID
          if firsttime:
!            notesindex[nid] = 'never classified'
          else:
              if not notesindex.has_key(nid):
--- 165,174 ----
      numuns = 0
      numdocs = 0
! 
      doc = v.GetFirstDocument()
      while doc:
          nid = doc.NOTEID
          if firsttime:
!             notesindex[nid] = 'never classified'
          else:
              if not notesindex.has_key(nid):
***************
*** 182,186 ****
                  # The com interface returns basic data types as tuples
                  # only, thus the subscript on GetItemValue
!                 
                  try:
                      subj = doc.GetItemValue('Subject')[0]
--- 182,186 ----
                  # The com interface returns basic data types as tuples
                  # only, thus the subscript on GetItemValue
! 
                  try:
                      subj = doc.GetItemValue('Subject')[0]
***************
*** 230,234 ****
      print "   %s classified as ham" % (numham)
      print "   %s classified as unsure" % (numuns)
!     
  
  def processAndTrain(v, vmoveto, bayes, is_spam, notesindex):
--- 230,234 ----
      print "   %s classified as ham" % (numham)
      print "   %s classified as unsure" % (numuns)
! 
  
  def processAndTrain(v, vmoveto, bayes, is_spam, notesindex):
***************
*** 240,244 ****
  
      print "Training %s" % (str)
!     
      docstomove = []
      doc = v.GetFirstDocument()
--- 240,244 ----
  
      print "Training %s" % (str)
! 
      docstomove = []
      doc = v.GetFirstDocument()
***************
*** 253,257 ****
          except:
              body = 'No Body'
!             
          message = "Subject: %s\r\n%s" % (subj, body)
  
--- 253,257 ----
          except:
              body = 'No Body'
! 
          message = "Subject: %s\r\n%s" % (subj, body)
  
***************
*** 270,274 ****
                  # msg is trained as ham, is to be retrained as spam
                  bayes.unlearn(tokens, False)
!   
          bayes.learn(tokens, is_spam)
  
--- 270,274 ----
                  # msg is trained as ham, is to be retrained as spam
                  bayes.unlearn(tokens, False)
! 
          bayes.learn(tokens, is_spam)
  
***************
*** 282,286 ****
  
      print "%s documents trained" % (len(docstomove))
!     
  
  def run(bdbname, useDBM, ldbname, rdbname, foldname, doTrain, doClassify):
--- 282,286 ----
  
      print "%s documents trained" % (len(docstomove))
! 
  
  def run(bdbname, useDBM, ldbname, rdbname, foldname, doTrain, doClassify):
***************
*** 302,306 ****
          notesindex = pickle.load(fp)
          fp.close()
!      
      sess = win32com.client.Dispatch("Lotus.NotesSession")
      try:
--- 302,306 ----
          notesindex = pickle.load(fp)
          fp.close()
! 
      sess = win32com.client.Dispatch("Lotus.NotesSession")
      try:
***************
*** 309,315 ****
          print "Session aborted"
          sys.exit()
!         
      db = sess.GetDatabase("",ldbname)
!     
      vinbox = db.getView('($Inbox)')
      vspam = db.getView("%s\Spam" % (foldname))
--- 309,315 ----
          print "Session aborted"
          sys.exit()
! 
      db = sess.GetDatabase("",ldbname)
! 
      vinbox = db.getView('($Inbox)')
      vspam = db.getView("%s\Spam" % (foldname))
***************
*** 317,331 ****
      vtrainspam = db.getView("%s\Train as Spam" % (foldname))
      vtrainham = db.getView("%s\Train as Ham" % (foldname))
!     
      if doTrain:
          processAndTrain(vtrainspam, vspam, bayes, True, notesindex)
          # for some reason, using inbox as a target here loses the mail
          processAndTrain(vtrainham, vham, bayes, False, notesindex)
!         
      if rdbname:
          print "Replicating..."
          db.Replicate(rdbname)
          print "Done"
!         
      if doClassify:
          classifyInbox(vinbox, vtrainspam, bayes, ldbname, notesindex)
--- 317,331 ----
      vtrainspam = db.getView("%s\Train as Spam" % (foldname))
      vtrainham = db.getView("%s\Train as Ham" % (foldname))
! 
      if doTrain:
          processAndTrain(vtrainspam, vspam, bayes, True, notesindex)
          # for some reason, using inbox as a target here loses the mail
          processAndTrain(vtrainham, vham, bayes, False, notesindex)
! 
      if rdbname:
          print "Replicating..."
          db.Replicate(rdbname)
          print "Done"
! 
      if doClassify:
          classifyInbox(vinbox, vtrainspam, bayes, ldbname, notesindex)
***************
*** 339,343 ****
      pickle.dump(notesindex, fp)
      fp.close()
!     
  
  if __name__ == '__main__':
--- 339,343 ----
      pickle.dump(notesindex, fp)
      fp.close()
! 
  
  if __name__ == '__main__':
***************
*** 390,392 ****
                  pass
      else:
!         print >>sys.stderr, __doc__
\ No newline at end of file
--- 390,392 ----
                  pass
      else:
!         print >>sys.stderr, __doc__

Index: sb_pop3dnd.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/scripts/sb_pop3dnd.py,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** sb_pop3dnd.py	18 Sep 2003 03:58:59 -0000	1.3
--- sb_pop3dnd.py	16 Dec 2003 05:06:33 -0000	1.4
***************
*** 172,176 ****
          return self._flags_iter()
  
!     def _flags_iter(self):    
          if self.deleted:
              yield "\\DELETED"
--- 172,176 ----
          return self._flags_iter()
  
!     def _flags_iter(self):
          if self.deleted:
              yield "\\DELETED"
***************
*** 209,216 ****
      def getSubPart(self, part):
          """Retrieve a MIME sub-message
!         
          @type part: C{int}
          @param part: The number of the part to retrieve, indexed from 0.
!         
          @rtype: Any object implementing C{IMessage}.
          @return: The specified sub-part.
--- 209,216 ----
      def getSubPart(self, part):
          """Retrieve a MIME sub-message
! 
          @type part: C{int}
          @param part: The number of the part to retrieve, indexed from 0.
! 
          @rtype: Any object implementing C{IMessage}.
          @return: The specified sub-part.
***************
*** 243,247 ****
          else:
              print "Tried to set invalid flag", flag, "to", value
!             
      def flags(self):
          """Return the message flags."""
--- 243,247 ----
          else:
              print "Tried to set invalid flag", flag, "to", value
! 
      def flags(self):
          """Return the message flags."""
***************
*** 273,277 ****
          """Body structure data describes the MIME-IMB
          format of a message and consists of a sequence of mime type, mime
!         subtype, parameters, content id, description, encoding, and size. 
          The fields following the size field are variable: if the mime
          type/subtype is message/rfc822, the contained message's envelope
--- 273,277 ----
          """Body structure data describes the MIME-IMB
          format of a message and consists of a sequence of mime type, mime
!         subtype, parameters, content id, description, encoding, and size.
          The fields following the size field are variable: if the mime
          type/subtype is message/rfc822, the contained message's envelope
***************
*** 307,311 ****
          return s
  
!     def body(self):    
          rfc822 = self.as_string()
          bodyRE = re.compile(r"\r?\n(\r?\n)(.*)",
--- 307,311 ----
          return s
  
!     def body(self):
          rfc822 = self.as_string()
          bodyRE = re.compile(r"\r?\n(\r?\n)(.*)",
***************
*** 333,337 ****
      def string_contains(self, whole, sub):
          return whole.find(sub) != -1
!         
      def matches(self, criteria):
          """Return True iff the messages matches the specified IMAP
--- 333,337 ----
      def string_contains(self, whole, sub):
          return whole.find(sub) != -1
! 
      def matches(self, criteria):
          """Return True iff the messages matches the specified IMAP
***************
*** 368,372 ****
                           "SINCE" : (self.since, self.date),
                           }
!                        
          result = True
          test = None
--- 368,372 ----
                           "SINCE" : (self.since, self.date),
                           }
! 
          result = True
          test = None
***************
*** 437,441 ****
          """Add a mailbox change listener."""
          self.listeners.append(listener)
!     
      def removeListener(self, listener):
          """Remove a mailbox change listener."""
--- 437,441 ----
          """Add a mailbox change listener."""
          self.listeners.append(listener)
! 
      def removeListener(self, listener):
          """Remove a mailbox change listener."""
***************
*** 465,469 ****
              if msg.recent:
                  self.recent_count += 1
!     
      def getUIDNext(self, increase=False):
          """Return the likely UID for the next message added to this
--- 465,469 ----
              if msg.recent:
                  self.recent_count += 1
! 
      def getUIDNext(self, increase=False):
          """Return the likely UID for the next message added to this
***************
*** 496,500 ****
          """Return the number of messages with the 'Unseen' flag."""
          return self.unseen_count
!         
      def isWriteable(self):
          """Get the read/write status of the mailbox."""
--- 496,500 ----
          """Return the number of messages with the 'Unseen' flag."""
          return self.unseen_count
! 
      def isWriteable(self):
          """Get the read/write status of the mailbox."""
***************
*** 584,588 ****
                      matches.append(id)
                      break
!         return matches            
  
      def _messagesIter(self, messages, uid):
--- 584,588 ----
                      matches.append(id)
                      break
!         return matches
  
      def _messagesIter(self, messages, uid):
***************
*** 641,645 ****
          # We don't care
          pass
!     
      def flagsChanged(self, newFlags):
          # We don't care
--- 641,645 ----
          # We don't care
          pass
! 
      def flagsChanged(self, newFlags):
          # We don't care
***************
*** 811,815 ****
              date = imaplib.Time2Internaldate(time.time())[1:-1]
              dest_folder.addMessage(msg, (), date)
!             
              # We have to return something, because the client is expecting
              # us to.  We return a short message indicating that a message
--- 811,815 ----
              date = imaplib.Time2Internaldate(time.time())[1:-1]
              dest_folder.addMessage(msg, (), date)
! 
              # We have to return something, because the client is expecting
              # us to.  We return a short message indicating that a message
***************
*** 899,903 ****
      httpServer.register(serverUI)
  
!     return app    
  
  def run():
--- 899,903 ----
      httpServer.register(serverUI)
  
!     return app
  
  def run():

Index: sb_server.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/scripts/sb_server.py,v
retrieving revision 1.13
retrieving revision 1.14
diff -C2 -d -r1.13 -r1.14
*** sb_server.py	14 Dec 2003 01:30:49 -0000	1.13
--- sb_server.py	16 Dec 2003 05:06:33 -0000	1.14
***************
*** 505,509 ****
                  traceback.print_exc(None, stream)
                  details = stream.getvalue()
!                 
                  # Build the header.  This will strip leading whitespace from
                  # the lines, so we add a leading dot to maintain indentation.
--- 505,509 ----
                  traceback.print_exc(None, stream)
                  details = stream.getvalue()
! 
                  # Build the header.  This will strip leading whitespace from
                  # the lines, so we add a leading dot to maintain indentation.
***************
*** 512,516 ****
                  headerName = 'X-Spambayes-Exception'
                  header = Header(dottedDetails, header_name=headerName)
!                 
                  # Insert the header, converting email.Header's '\n' line
                  # breaks to POP3's '\r\n'.
--- 512,516 ----
                  headerName = 'X-Spambayes-Exception'
                  header = Header(dottedDetails, header_name=headerName)
! 
                  # Insert the header, converting email.Header's '\n' line
                  # breaks to POP3's '\r\n'.
***************
*** 676,684 ****
          assert self.platform_mutex is None, "Should not already have the mutex"
          self.platform_mutex = open_platform_mutex()
!     
          # Do whatever we've been asked to do...
          self.createWorkers()
          self.prepared = True
!         
      def buildServerStrings(self):
          """After the server details have been set up, this creates string
--- 676,684 ----
          assert self.platform_mutex is None, "Should not already have the mutex"
          self.platform_mutex = open_platform_mutex()
! 
          # Do whatever we've been asked to do...
          self.createWorkers()
          self.prepared = True
! 
      def buildServerStrings(self):
          """After the server details have been set up, this creates string
***************
*** 713,717 ****
                  big = "spam"
                  small = "ham"
!             if big is not None:                
                  self.warning = "Warning: you have much more %s than %s - " \
                                 "SpamBayes works best with approximately even " \
--- 713,717 ----
                  big = "spam"
                  small = "ham"
!             if big is not None:
                  self.warning = "Warning: you have much more %s than %s - " \
                                 "SpamBayes works best with approximately even " \
***************
*** 919,923 ****
          elif len(args) == 2:
              state.servers = [(args[0], int(args[1]))]
!         
          # Default to listening on port 110 for command-line-specified servers.
          if len(args) > 0 and state.proxyPorts == []:
--- 919,923 ----
          elif len(args) == 2:
              state.servers = [(args[0], int(args[1]))]
! 
          # Default to listening on port 110 for command-line-specified servers.
          if len(args) > 0 and state.proxyPorts == []:



From tim_one at users.sourceforge.net  Tue Dec 16 00:19:10 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Tue Dec 16 00:19:12 2003
Subject: [Spambayes-checkins] spambayes/spambayes classifier.py,1.14,1.15
Message-ID: <E1AW7by-0003ui-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv14339/spambayes

Modified Files:
	classifier.py 
Log Message:
_enhance_wordstream():  Simplify and speed; repaired docstring; now
delivers the last token in the input stream too.  NOT TESTED, though.


Index: classifier.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/classifier.py,v
retrieving revision 1.14
retrieving revision 1.15
diff -C2 -d -r1.14 -r1.15
*** classifier.py	16 Dec 2003 04:59:58 -0000	1.14
--- classifier.py	16 Dec 2003 05:19:08 -0000	1.15
***************
*** 427,434 ****
  
      def _enhance_wordstream(self, wordstream):
!         """Add bigrams to the wordstream.  This wraps the last token
!         to the first one, so a small number of odd tokens might get
!         generated from that, but it shouldn't be significant.  Note
!         that these are *token* bigrams, and not *word* bigrams - i.e.
          'synthetic' tokens get bigram'ed, too.
  
--- 427,435 ----
  
      def _enhance_wordstream(self, wordstream):
!         """Add bigrams to the wordstream.
! 
!         For example, a b c -> a b "a b" c "b c"
! 
!         Note that these are *token* bigrams, and not *word* bigrams - i.e.
          'synthetic' tokens get bigram'ed, too.
  
***************
*** 438,453 ****
  
          If the experimental "Classifier":"x-use_bigrams" option is
!         removed, this function can be removed, too."""
!         p = None
!         while True:
!             try:
!                 if p:
!                     yield p
!                 q = wordstream.next()
!                 if p:
!                     yield "%s %s" % (p, q)
!                 p = q
!             except StopIteration:
!                 break
  
      def _wordinfokeys(self):
--- 439,451 ----
  
          If the experimental "Classifier":"x-use_bigrams" option is
!         removed, this function can be removed, too.
!         """
! 
!         last = None
!         for token in wordstream:
!             yield token
!             if last:
!                 yield "%s %s" % (last, token)
!             last = token
  
      def _wordinfokeys(self):



From tim_one at users.sourceforge.net  Tue Dec 16 00:36:43 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Tue Dec 16 00:36:45 2003
Subject: [Spambayes-checkins] spambayes/spambayes classifier.py,1.15,1.16
Message-ID: <E1AW7sx-0004b9-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv17442/spambayes

Modified Files:
	classifier.py 
Log Message:
learn() and unlearn():  when x-use_bigrams is in effect, the *type* of
wordstream shouldn't make any difference, so don't test it.  In general,
the only assumption about streams made in classifier.py is that they're
iterable objects (generators, lists, doesn't matter).


Index: classifier.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/classifier.py,v
retrieving revision 1.15
retrieving revision 1.16
diff -C2 -d -r1.15 -r1.16
*** classifier.py	16 Dec 2003 05:19:08 -0000	1.15
--- classifier.py	16 Dec 2003 05:36:41 -0000	1.16
***************
*** 211,216 ****
          else that it's definitely not spam.
          """
!         if options["Classifier", "x-use_bigrams"] and \
!            isinstance(wordstream, types.GeneratorType):
              wordstream = self._enhance_wordstream(wordstream)
          self._add_msg(wordstream, is_spam)
--- 211,215 ----
          else that it's definitely not spam.
          """
!         if options["Classifier", "x-use_bigrams"]:
              wordstream = self._enhance_wordstream(wordstream)
          self._add_msg(wordstream, is_spam)
***************
*** 221,226 ****
          Pass the same arguments you passed to learn().
          """
!         if options["Classifier", "x-use_bigrams"] and \
!            isinstance(wordstream, types.GeneratorType):
              wordstream = self._enhance_wordstream(wordstream)
          self._remove_msg(wordstream, is_spam)
--- 220,224 ----
          Pass the same arguments you passed to learn().
          """
!         if options["Classifier", "x-use_bigrams"]:
              wordstream = self._enhance_wordstream(wordstream)
          self._remove_msg(wordstream, is_spam)



From kennypitt at hotmail.com  Tue Dec 16 10:17:16 2003
From: kennypitt at hotmail.com (Kenny Pitt)
Date: Tue Dec 16 10:17:52 2003
Subject: [Spambayes-checkins] spambayes/spambayes OptionsClass.py, 1.19,
	1.20
In-Reply-To: <E1AW78J-0002Se-00@sc8-pr-cvs1.sourceforge.net>
Message-ID: <LAW11-OE26S6xopZz8Q0000b81a@hotmail.com>

I was looking at this same code a little bit yesterday, and one thing
struck me as odd.  The get(), get_option(), and set() functions use
self.conversion_table to translate the requested option, but many of the
other functions such as display_name() don't.  Is there a reason for
that?  If not, I was wondering if it would be easier to just fix
get_option() for case-insensitivity and then have all the other getter
functions call it instead of accessing self._options directly.

Also note that the self.conversion_table translation in get() is already
redundant, as it then calls get_option() which will do the exact same
translation.

Tony Meyer wrote:
> Update of /cvsroot/spambayes/spambayes/spambayes
> In directory sc8-pr-cvs1:/tmp/cvs-serv9453/spambayes
> 
> Modified Files:
> 	OptionsClass.py
> Log Message:
> Option names are always case insensitive, no matter what.
> 
> Index: OptionsClass.py
> ===================================================================
> RCS file: /cvsroot/spambayes/spambayes/spambayes/OptionsClass.py,v
> retrieving revision 1.19
> retrieving revision 1.20
> diff -C2 -d -r1.19 -r1.20
> *** OptionsClass.py	15 Dec 2003 09:20:33 -0000	1.19
> --- OptionsClass.py	16 Dec 2003 04:48:28 -0000	1.20
> ***************
> *** 552,586 ****
>       def display_name(self, sect, opt):
>           '''A name for the option suitable for display to a user.'''
> !         return self._options[sect, opt].display_name()
>[snip]
>       def get_option(self, sect, opt):
> --- 552,586 ----
>       def display_name(self, sect, opt):
>           '''A name for the option suitable for display to a user.'''
> !         return self._options[sect, opt.lower()].display_name()
>[snip]
>       def get_option(self, sect, opt):
> ***************
> *** 588,598 ****
>           if self.conversion_table.has_key((sect, opt)):
>               sect, opt = self.conversion_table[sect, opt]
> !         return self._options[sect, opt]
> 
>       def get(self, sect, opt):
>           '''Get an option value.'''
> !         if self.conversion_table.has_key((sect, opt)):
> !             sect, opt = self.conversion_table[sect, opt]
> !         return self.get_option(sect, opt).get()
> 
>       def __getitem__(self, key):
> --- 588,598 ----
>           if self.conversion_table.has_key((sect, opt)):
>               sect, opt = self.conversion_table[sect, opt]
> !         return self._options[sect, opt.lower()]
> 
>       def get(self, sect, opt):
>           '''Get an option value.'''
> !         if self.conversion_table.has_key((sect, opt.lower())):
> !             sect, opt = self.conversion_table[sect, opt.lower()]
> !         return self.get_option(sect, opt.lower()).get()
> 
>       def __getitem__(self, key):
> ***************
> *** 601,612 ****
>       def set(self, sect, opt, val=None):
>           '''Set an option.'''
> !         if self.conversion_table.has_key((sect, opt)):
> !             sect, opt = self.conversion_table[sect, opt]
>           if self.is_valid(sect, opt, val):
> !             self._options[sect, opt].set(val)
>           else:
>               print >> sys.stderr, ("Attempted to set [%s] %s with
>                                     invalid" " value %s (%s)" %
> !                                   (sect, opt, val, type(val)))
> 
>       def set_from_cmdline(self, arg, stream=None):
> --- 601,612 ----
>       def set(self, sect, opt, val=None):
>           '''Set an option.'''
> !         if self.conversion_table.has_key((sect, opt.lower())):
> !             sect, opt = self.conversion_table[sect, opt.lower()]
>           if self.is_valid(sect, opt, val):
> !             self._options[sect, opt.lower()].set(val)
>           else:
>               print >> sys.stderr, ("Attempted to set [%s] %s with
>                                     invalid" " value %s (%s)" %
> !                                   (sect, opt.lower(), val,
> type(val))) 
> 
>       def set_from_cmdline(self, arg, stream=None):

-- 
Kenny Pitt


From tim_one at users.sourceforge.net  Tue Dec 16 21:05:35 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Tue Dec 16 21:05:39 2003
Subject: [Spambayes-checkins] spambayes/spambayes classifier.py,1.16,1.17
Message-ID: <E1AWR4B-0007Lt-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv27807/spambayes

Modified Files:
	classifier.py 
Log Message:
Implemented the intended "tiling" version of x-use_bigrams.  Tried to
restore most of the speed lost when this option *isn't* in use.

Will add comments later.

Anyone using x-use_bigrams needs to retrain:  synthesized bigrams now
begin with a "bi:" prefix.


Index: classifier.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/classifier.py,v
retrieving revision 1.16
retrieving revision 1.17
diff -C2 -d -r1.16 -r1.17
*** classifier.py	16 Dec 2003 05:36:41 -0000	1.16
--- classifier.py	17 Dec 2003 02:05:33 -0000	1.17
***************
*** 361,404 ****
      def _getclues(self, wordstream):
          mindist = options["Classifier", "minimum_prob_strength"]
-         if not isinstance(wordstream, types.ListType):
-             wordstream = list(wordstream)
- 
-         # clues is now a set so that that duplicates are removed.
-         # Otherwise if a token is a stronger clue than the bigrams
-         # it forms with the previous *and* next tokens, then it
-         # would be used twice.  As is, it is possible for a single
-         # token to be present in two bigram tokens (but in different
-         # positions).
-         clues = Set()  # (distance, prob, word, record) tuples
-         pushclue = clues.add
  
          if options["Classifier", "x-use_bigrams"]:
!             # The tokens list contains 3-tuples of a token,
!             # the bigram it forms with the next token, and the
!             # next token.  This makes it easier to select which
!             # one to use later on.
!             l = wordstream
!             tokens = [(l[i],"%s %s" % (l[i],l[(i+1)%len(l)]),
!                        l[(i+1)%len(l)]) for i in xrange(len(l))]
  
-             # Run through all the triplets and add the strongest
-             # clue from each (note also the comment above explaining
-             # how duplicates are treated).
-             for grams in tokens:
-                 winner = (0, None, None, None) # distance, prob, word, record
-                 for gram in grams:
-                     contestant = self._worddistanceget(gram)
-                     if contestant[0] > winner[0]:
-                         winner = contestant
-                 if winner[0] >= mindist:
-                     pushclue(winner)
          else:
!             for word in wordstream: # Set() not necessary; see above.
!                 contestant = self._worddistanceget(word)
!                 if contestant[0] >= mindist:
!                     pushclue(contestant)
  
-         clues = list(clues)
-         clues.sort()
          if len(clues) > options["Classifier", "max_discriminators"]:
              del clues[0 : -options["Classifier", "max_discriminators"]]
--- 361,403 ----
      def _getclues(self, wordstream):
          mindist = options["Classifier", "minimum_prob_strength"]
  
          if options["Classifier", "x-use_bigrams"]:
!             raw = []
!             push = raw.append
!             pair = None
!             seen = {pair: 1}
!             for i, token in enumerate(wordstream):
!                 if i:
!                     pair = "bi:%s %s" % (last_token, token)
!                 last_token = token
!                 for clue, indices in (token, (i,)), (pair, (i-1, i)):
!                     if clue not in seen:
!                         seen[clue] = 1
!                         tup = self._worddistanceget(clue)
!                         if tup[0] >= mindist:
!                             push((tup, indices))
! 
!             raw.sort()
!             raw.reverse()
!             clues = []
!             push = clues.append
!             seen = {}
!             for tup, indices in raw:
!                 overlap = [i for i in indices if i in seen]
!                 if not overlap:
!                     for i in indices:
!                         seen[i] = 1
!                     push(tup)
!             clues.reverse()
  
          else:
!             clues = []
!             push = clues.append
!             for word in Set(wordstream):
!                 tup = self._worddistanceget(word)
!                 if tup[0] >= mindist:
!                     push(tup)
!             clues.sort()
  
          if len(clues) > options["Classifier", "max_discriminators"]:
              del clues[0 : -options["Classifier", "max_discriminators"]]
***************
*** 444,448 ****
              yield token
              if last:
!                 yield "%s %s" % (last, token)
              last = token
  
--- 443,447 ----
              yield token
              if last:
!                 yield "bi:%s %s" % (last, token)
              last = token
  



From tim_one at users.sourceforge.net  Wed Dec 17 00:43:44 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Wed Dec 17 00:43:48 2003
Subject: [Spambayes-checkins] spambayes/spambayes Options.py, 1.92,
	1.93 classifier.py, 1.17, 1.18
Message-ID: <E1AWUTI-00009W-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv524/spambayes

Modified Files:
	Options.py classifier.py 
Log Message:
Fulfilling a promise to write useful comments for the x-use_bigrams option.


Index: Options.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/Options.py,v
retrieving revision 1.92
retrieving revision 1.93
diff -C2 -d -r1.92 -r1.93
*** Options.py	16 Dec 2003 05:06:34 -0000	1.92
--- Options.py	17 Dec 2003 05:43:42 -0000	1.93
***************
*** 429,446 ****
  
      ("x-use_bigrams", "(EXPERIMENTAL) Use mixed uni/bi-grams scheme", False,
!      """Enabling this option means that SpamBayes will generate both
!      unigrams (words) and bigrams (pairs of words). However, Graham's
!      scheme is also used, where, for each clue triplet of two unigrams and
!      the bigram they make, the clue that is strongest (i.e. has a
!      probability furtherest from 0.5) is used, and the other two are not.
  
       Note that to really test this option you need to retrain with it on,
       so that your database includes the bigrams - if you subsequently turn
!      it off, these tokens will have no effect.
  
!      Note also that you should probably also increase the max_discriminators
!      (Maximum number of extreme words) option if you enable this option;
!      this may need to be doubled or quadrupled to see the benefit from the
!      bigrams.
  
       This option is experimental, and may be removed in a future release.
--- 429,452 ----
  
      ("x-use_bigrams", "(EXPERIMENTAL) Use mixed uni/bi-grams scheme", False,
!      """Generate both unigrams (words) and bigrams (pairs of words).
!      However, extending an idea originally from Gary Robinson, the message
!      is 'tiled' into  non-overlapping unigrams and bigrams, approximating
!      the strongest outcome over all possible tilings.
  
       Note that to really test this option you need to retrain with it on,
       so that your database includes the bigrams - if you subsequently turn
!      it off, these tokens will have no effect.  This option will at least
!      double your database size given the same training data, and will
!      probably at least triple it.
  
!      You may also wish to increase the max_discriminators (maximum number
!      of extreme words) option if you enable this option, perhaps doubling or
!      quadrupling it.  It's not yet clear.  Bigrams create many more hapaxes,
!      and that seems to increase the brittleness of minimalist training
!      regimes; increasing max_discriminators may help to soften that effect.
!      OTOH, max_discriminators defaults to 150 in part because that makes it
!      easy to prove that the chi-squared math is immune from numeric
!      problems.  Increase it too much, and insane results will eventually
!      result (including fatal floating-point exceptions on some boxes).
  
       This option is experimental, and may be removed in a future release.

Index: classifier.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/classifier.py,v
retrieving revision 1.17
retrieving revision 1.18
diff -C2 -d -r1.17 -r1.18
*** classifier.py	17 Dec 2003 02:05:33 -0000	1.17
--- classifier.py	17 Dec 2003 05:43:42 -0000	1.18
***************
*** 359,376 ****
          pass
  
      def _getclues(self, wordstream):
          mindist = options["Classifier", "minimum_prob_strength"]
  
          if options["Classifier", "x-use_bigrams"]:
              raw = []
              push = raw.append
              pair = None
!             seen = {pair: 1}
              for i, token in enumerate(wordstream):
!                 if i:
                      pair = "bi:%s %s" % (last_token, token)
                  last_token = token
                  for clue, indices in (token, (i,)), (pair, (i-1, i)):
!                     if clue not in seen:
                          seen[clue] = 1
                          tup = self._worddistanceget(clue)
--- 359,405 ----
          pass
  
+     # Return list of (prob, word, record) triples, sorted by increasing
+     # prob.  "word" is a token from wordstream; "prob" is its spamprob (a
+     # float in 0.0 through 1.0); and "record" is word's associated
+     # WordInfo record if word is in the training database, or None if it's
+     # not.  No more than max_discriminators items are returned, and have
+     # the strongest (farthest from 0.5) spamprobs of all tokens in wordstream.
+     # Tokens with spamprobs less than minimum_prob_strength away from 0.5
+     # aren't returned.
      def _getclues(self, wordstream):
          mindist = options["Classifier", "minimum_prob_strength"]
  
          if options["Classifier", "x-use_bigrams"]:
+             # This scheme mixes single tokens with pairs of adjacent tokens.
+             # wordstream is "tiled" into non-overlapping unigrams and
+             # bigrams.  Non-overlap is important to prevent a single original
+             # token from contributing to more than one spamprob returned
+             # (systematic correlation probably isn't a good thing).
+ 
+             # First fill list raw with
+             #     (distance, prob, word, record), indices
+             # pairs, one for each unigram and bigram in wordstream.
+             # indices is a tuple containing the indices (0-based relative to
+             # the start of wordstream) of the tokens that went into word.
+             # indices is a 1-tuple for an original token, and a 2-tuple for
+             # a synthesized bigram token.  The indices are needed to detect
+             # overlap later.
              raw = []
              push = raw.append
              pair = None
!             # Keep track of which tokens we've already seen.
!             # Don't use a Set here!  This is an innermost loop, so speed is
!             # important here (direct dict fiddling is much quicker than
!             # invoking Python-level Set methods; in Python 2.4 that will
!             # change).
!             seen = {pair: 1} # so the bigram token is skipped on 1st loop trip
              for i, token in enumerate(wordstream):
!                 if i:   # not the 1st loop trip, so there is a preceding token
!                     # This string interpolation must match the one in
!                     # _enhance_wordstream().
                      pair = "bi:%s %s" % (last_token, token)
                  last_token = token
                  for clue, indices in (token, (i,)), (pair, (i-1, i)):
!                     if clue not in seen:    # as always, skip duplicates
                          seen[clue] = 1
                          tup = self._worddistanceget(clue)
***************
*** 378,395 ****
                              push((tup, indices))
  
              raw.sort()
              raw.reverse()
              clues = []
              push = clues.append
              seen = {}
              for tup, indices in raw:
                  overlap = [i for i in indices if i in seen]
!                 if not overlap:
                      for i in indices:
                          seen[i] = 1
                      push(tup)
              clues.reverse()
  
          else:
              clues = []
              push = clues.append
--- 407,431 ----
                              push((tup, indices))
  
+             # Sort raw, strongest to weakest spamprob.
              raw.sort()
              raw.reverse()
+             # Fill clues with the strongest non-overlapping clues.
              clues = []
              push = clues.append
+             # Keep track of which indices have already contributed to a
+             # clue in clues.
              seen = {}
              for tup, indices in raw:
                  overlap = [i for i in indices if i in seen]
!                 if not overlap: # no overlap with anything already in clues
                      for i in indices:
                          seen[i] = 1
                      push(tup)
+             # Leave sorted from smallest to largest spamprob.
              clues.reverse()
  
          else:
+             # The all-unigram scheme just scores the tokens as-is.  A Set()
+             # is used to weed out duplicates at high speed.
              clues = []
              push = clues.append
***************
*** 443,446 ****
--- 479,484 ----
              yield token
              if last:
+                 # This string interpolation must match the one in
+                 # _getclues().
                  yield "bi:%s %s" % (last, token)
              last = token



From mhammond at users.sourceforge.net  Wed Dec 17 01:31:59 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Wed Dec 17 01:32:02 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 addin.py,1.117,1.118
Message-ID: <E1AWVDz-00029D-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv8186

Modified Files:
	addin.py 
Log Message:
Now we create a temporary mail item to create the 'Spam' field, we need
to create it *before* we hook folder events, else we attempt to filter 
the temp message.


Index: addin.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/addin.py,v
retrieving revision 1.117
retrieving revision 1.118
diff -C2 -d -r1.117 -r1.118
*** addin.py	16 Dec 2003 05:06:32 -0000	1.117
--- addin.py	17 Dec 2003 06:31:57 -0000	1.118
***************
*** 1314,1317 ****
--- 1314,1333 ----
                  folder = msgstore_folder.GetOutlookItem()
                  name = msgstore_folder.GetFQName()
+                 # Ensure the field is created before we hook the folder
+                 # events, else there is a chance our event handler will
+                 # see the temporary message we create.
+                 try:
+                     self.manager.EnsureOutlookFieldsForFolder(msgstore_folder.GetID())
+                 except:
+                     # An exception checking that Outlook's folder has a
+                     # 'spam' field is not fatal, nor really even worth
+                     # telling the user about, nor even worth a traceback
+                     # (as it is likely a COM error).
+                     print "ERROR: Failed to check folder '%s' for " \
+                           "Spam field" % name
+                     etype, value, tb = sys.exc_info()
+                     tb = None # dont want it, and nuke circular ref
+                     traceback.print_exception(etype, value, tb)
+                 # now setup the hook.
                  try:
                      new_hook = DispatchWithEvents(folder.Items, HandlerClass)
***************
*** 1322,1337 ****
                      new_hook.Init(msgstore_folder, self.application, self.manager)
                      new_hooks[msgstore_folder.id] = new_hook
-                     try:
-                         self.manager.EnsureOutlookFieldsForFolder(msgstore_folder.GetID())
-                     except:
-                         # An exception checking that Outlook's folder has a
-                         # 'spam' field is not fatal, nor really even worth
-                         # telling the user about, nor even worth a traceback
-                         # (as it is likely a COM error).
-                         print "ERROR: Failed to check folder '%s' for " \
-                               "Spam field" % name
-                         etype, value, tb = sys.exc_info()
-                         tb = None # dont want it, and nuke circular ref
-                         traceback.print_exception(etype, value, tb)
                      print "SpamBayes: Watching for new messages in folder", name
              else:
--- 1338,1341 ----



From mhammond at users.sourceforge.net  Wed Dec 17 01:45:47 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Wed Dec 17 01:45:49 2003
Subject: [Spambayes-checkins] 
	spambayes/windows/resources sb-started.ico, 1.1,
	1.2 sb-stopped.ico, 1.1, 1.2 sbicon.ico, 1.1, 1.2
Message-ID: <E1AWVRL-0002d6-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/windows/resources
In directory sc8-pr-cvs1:/tmp/cvs-serv10059/resources

Modified Files:
	sb-started.ico sb-stopped.ico sbicon.ico 
Log Message:
Better icons and icon loading code from Kenny Pitt.


Index: sb-started.ico
===================================================================
RCS file: /cvsroot/spambayes/spambayes/windows/resources/sb-started.ico,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
Binary files /tmp/cvsQdZrZe and /tmp/cvsqsZ77l differ

Index: sb-stopped.ico
===================================================================
RCS file: /cvsroot/spambayes/spambayes/windows/resources/sb-stopped.ico,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
Binary files /tmp/cvs6m1BAh and /tmp/cvsUV5bXo differ

Index: sbicon.ico
===================================================================
RCS file: /cvsroot/spambayes/spambayes/windows/resources/sbicon.ico,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
Binary files /tmp/cvszFT2ui and /tmp/cvsalRL9s differ



From mhammond at users.sourceforge.net  Wed Dec 17 01:45:46 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Wed Dec 17 01:45:53 2003
Subject: [Spambayes-checkins] spambayes/windows pop3proxy_tray.py,1.18,1.19
Message-ID: <E1AWVRK-0002d3-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/windows
In directory sc8-pr-cvs1:/tmp/cvs-serv10059

Modified Files:
	pop3proxy_tray.py 
Log Message:
Better icons and icon loading code from Kenny Pitt.


Index: pop3proxy_tray.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/windows/pop3proxy_tray.py,v
retrieving revision 1.18
retrieving revision 1.19
diff -C2 -d -r1.18 -r1.19
*** pop3proxy_tray.py	16 Dec 2003 05:06:34 -0000	1.18
--- pop3proxy_tray.py	17 Dec 2003 06:45:44 -0000	1.19
***************
*** 170,185 ****
              hexe = GetModuleHandle(None)
              icon_flags = win32con.LR_DEFAULTSIZE
!             self.hstartedicon = LoadImage(hexe, 1000, win32con.IMAGE_ICON, 0,
!                                           0, icon_flags)
!             self.hstoppedicon = LoadImage(hexe, 1010, win32con.IMAGE_ICON, 0,
!                                           0, icon_flags)
          else:
              # If we have no icon we fail in all sorts of places - so may as
              # well make it here :)
              icon_flags = win32con.LR_LOADFROMFILE | win32con.LR_DEFAULTSIZE
!             self.hstartedicon = LoadImage(hinst, startedIconPathName, win32con.IMAGE_ICON, 0,
!                                           0, icon_flags)
!             self.hstoppedicon = LoadImage(hinst, stoppedIconPathName, win32con.IMAGE_ICON, 0,
!                                           0, icon_flags)
  
          flags = NIF_ICON | NIF_MESSAGE | NIF_TIP
--- 170,185 ----
              hexe = GetModuleHandle(None)
              icon_flags = win32con.LR_DEFAULTSIZE
!             self.hstartedicon = LoadImage(hexe, 1000, win32con.IMAGE_ICON,
!                                           16, 16, icon_flags)
!             self.hstoppedicon = LoadImage(hexe, 1010, win32con.IMAGE_ICON,
!                                           16, 16, icon_flags)
          else:
              # If we have no icon we fail in all sorts of places - so may as
              # well make it here :)
              icon_flags = win32con.LR_LOADFROMFILE | win32con.LR_DEFAULTSIZE
!             self.hstartedicon = LoadImage(hinst, startedIconPathName, win32con.IMAGE_ICON,
!                                           16, 16, icon_flags)
!             self.hstoppedicon = LoadImage(hinst, stoppedIconPathName, win32con.IMAGE_ICON,
!                                           16, 16, icon_flags)
  
          flags = NIF_ICON | NIF_MESSAGE | NIF_TIP



From anadelonbrin at users.sourceforge.net  Wed Dec 17 03:55:07 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Wed Dec 17 03:55:10 2003
Subject: [Spambayes-checkins] spambayes/spambayes Corpus.py,1.12,1.13
Message-ID: <E1AWXSV-0007e8-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv29377/spambayes

Modified Files:
	Corpus.py 
Log Message:
Remove unused import.

Index: Corpus.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/Corpus.py,v
retrieving revision 1.12
retrieving revision 1.13
diff -C2 -d -r1.12 -r1.13
*** Corpus.py	16 Dec 2003 05:06:34 -0000	1.12
--- Corpus.py	17 Dec 2003 08:55:04 -0000	1.13
***************
*** 92,96 ****
  import sys           # for output of docstring
  import time
- from spambayes import tokenizer
  from spambayes.Options import options
  
--- 92,95 ----



From anadelonbrin at users.sourceforge.net  Wed Dec 17 04:09:55 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Wed Dec 17 04:10:06 2003
Subject: [Spambayes-checkins] spambayes/spambayes Options.py, 1.93,
	1.94 classifier.py, 1.18, 1.19 tokenizer.py, 1.22, 1.23
Message-ID: <E1AWXgp-0008KZ-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv32014/spambayes

Modified Files:
	Options.py classifier.py tokenizer.py 
Log Message:
Add the basis of a new experimental (and highly debatable) option to 'slurp' URLs.

This is based on the urlslurper.py script in the testtools directory, which in turn
was based on Richard Jowsey's URLSlurper.java.

Basically, when the option is enabled, instead of just tokenizing the URLs in a message,
we also retrieve the content at that address (if it's not text, we ignore it).

When classifying, if the message has a 'raw' score in the unsure range, and if the
number of tokens is less than max_discriminators, and adding these 'slurped' tokens
would push the message into the ham/spam range, then they are used.

When training, the tokens are always added (with the current setup).

Improvements to this (note the XXX sections in particular) most welcome.

Index: Options.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/Options.py,v
retrieving revision 1.93
retrieving revision 1.94
diff -C2 -d -r1.93 -r1.94
*** Options.py	17 Dec 2003 05:43:42 -0000	1.93
--- Options.py	17 Dec 2003 09:09:52 -0000	1.94
***************
*** 172,175 ****
--- 172,234 ----
    ),
  
+   # These options are all experimental; it seemed better to put them into
+   # their own category than have several interdependant experimental options.
+   # If this capability is removed, the entire section can go.
+   "URLRetriever" : (
+     ("x-slurp_urls", "Tokenize text content at the end of URLs", False,
+      """If this option is enabled, when a message normally scores in the
+      'unsure' range, and has fewer tokens than the maximum looked at,
+      and contains URLs, then the text at those URLs is obtained and
+      tokenized.  If those tokens result in the message moving to a
+      score outside the 'unsure' range, then they are added to the
+      tokens for the message.  This should be particularly effective
+      for messages that contain only a single URL and no other text.""",
+      BOOLEAN, RESTORE),
+     
+     ("x-cache_expiry_days", "Number of days to store URLs in cache", 7,
+      """This is the number of days that local cached copies of the text
+      at the URLs will be stored for.""",
+      INTEGER, RESTORE),
+ 
+     ("x-cache_directory", "URL Cache Directory", "url-cache",
+      """So that SpamBayes doesn't need to retrieve the same URL
+      over and over again, it stores local copies of the text
+      at the end of the URL.  This is the directory that will be
+      used for those copies.""",
+      PATH, RESTORE),
+ 
+     ("x-only_slurp_base", "Retrieve base url", False,
+      """To try and speed things up, and to avoid following unique URLS, if
+      this option is enabled, SpamBayes will convert the URL to as basic a
+      form it we can.  All directory information is removed and the domain
+      is reduced to the two (or three for those with a country TLD) top-most
+      elements.  For example,
+          http://www.massey.ac.nz/~tameyer/index.html?you=me
+      would become
+          http://massey.ac.nz
+      and
+          http://id.example.com
+      would become http://example.com
+ 
+      This should have two beneficial effects:
+       o It's unlikely that any information could be contained in this 'base'
+         url that could identify the user (unless they have a *lot* of domains).
+       o Many urls (both spam and ham) will strip down into the same 'base' url.
+         Since we have a limited form of caching, this means that a lot fewer
+         urls will have to be retrieved.
+      However, this does mean that if the 'base' url is hammy and the full is
+      spammy, or vice-versa, that the slurp will give back the wrong information.
+      Whether or not this is the case would have to be determined by testing.
+      """,
+      BOOLEAN, RESTORE),
+ 
+     ("x-web_prefix", "Prefix for tokens from web pages", "",
+      """It may be that what is hammy/spammy for you in email isn't from
+      webpages.  You can then set this option (to "web:", for example),
+      and effectively create an independent (sub)database for tokens
+      derived from parsing web pages.""",
+      r"[\S]+", RESTORE),
+   ),
+ 
    # These options control how a message is categorized
    "Categorization" : (

Index: classifier.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/classifier.py,v
retrieving revision 1.18
retrieving revision 1.19
diff -C2 -d -r1.18 -r1.19
*** classifier.py	17 Dec 2003 05:43:42 -0000	1.18
--- classifier.py	17 Dec 2003 09:09:52 -0000	1.19
***************
*** 57,60 ****
--- 57,62 ----
  LN2 = math.log(2)       # used frequently by chi-combining
  
+ slurp_wordstream = []
+ 
  PICKLE_VERSION = 5
  
***************
*** 201,206 ****
              return prob
  
      if options["Classifier", "use_chi_squared_combining"]:
!         spamprob = chi2_spamprob
  
      def learn(self, wordstream, is_spam):
--- 203,234 ----
              return prob
  
+     def slurping_spamprob(self, wordstream, evidence=False):
+         """Do the standard chi-squared spamprob, but if the evidence
+         leaves the score in the unsure range, and we have fewer tokens
+         than max_discriminators, also generate tokens from the text
+         obtained by following http URLs in the message."""
+         h_cut = options["Categorization", "ham_cutoff"]
+         s_cut = options["Categorization", "spam_cutoff"]
+ 
+         # Get the raw score.        
+         prob, clues = self.chi2_spamprob(wordstream, True)
+ 
+         # If necessary, enhance it with the tokens from whatever is
+         # at the URL's destination.
+         if len(clues) < options["Classifier", "max_discriminators"] and \
+            prob > h_cut and prob < s_cut:
+             sprob, sclues = self.chi2_spamprob(slurp_wordstream, True)
+             if sprob < h_cut or sprob > s_cut:
+                 prob = sprob
+                 clues = sclues
+         if evidence:
+             return prob, clues
+         return prob
+ 
      if options["Classifier", "use_chi_squared_combining"]:
!         if options["URLRetriever", "x-slurp_urls"]:
!             spamprob = slurping_spamprob
!         else:
!             spamprob = chi2_spamprob
  
      def learn(self, wordstream, is_spam):
***************
*** 213,216 ****
--- 241,246 ----
          if options["Classifier", "x-use_bigrams"]:
              wordstream = self._enhance_wordstream(wordstream)
+         if options["URLRetriever", "x-slurp_urls"]:
+             wordstream = self._add_slurped(wordstream)
          self._add_msg(wordstream, is_spam)
  
***************
*** 222,225 ****
--- 252,257 ----
          if options["Classifier", "x-use_bigrams"]:
              wordstream = self._enhance_wordstream(wordstream)
+         if options["URLRetriever", "x-slurp_urls"]:
+             wordstream = self._add_slurped(wordstream)
          self._remove_msg(wordstream, is_spam)
  
***************
*** 483,486 ****
--- 515,527 ----
                  yield "bi:%s %s" % (last, token)
              last = token
+ 
+     def _add_slurped(self, wordstream):
+         """Add tokens generated by 'slurping' (i.e. tokenizing
+         the text at the web pages pointed to by URLs in messages)
+         to the wordstream."""
+         for token in wordstream:
+             yield token
+         for token in slurp_wordstream:
+             yield token
  
      def _wordinfokeys(self):

Index: tokenizer.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/tokenizer.py,v
retrieving revision 1.22
retrieving revision 1.23
diff -C2 -d -r1.22 -r1.23
*** tokenizer.py	16 Dec 2003 05:06:34 -0000	1.22
--- tokenizer.py	17 Dec 2003 09:09:52 -0000	1.23
***************
*** 19,22 ****
--- 19,33 ----
      from compatsets import Set
  
+ # XXX At time of writing, these are only necessary for the
+ # XXX experimental url retrieving/slurping code.  If that
+ # XXX gets ripped out, either rip these out, or run
+ # XXX PyChecker over the code.
+ import sys
+ import socket
+ import pickle
+ import urllib2
+ from spambayes import classifier
+ from email import message_from_string
+ # XXX ---- ends ----
  
  from spambayes.Options import options
***************
*** 995,999 ****
          return tokens
  
! crack_urls = URLStripper().analyze
  
  # Nuke HTML <style gimmicks.
--- 1006,1260 ----
          return tokens
  
! DOMAIN_AND_PORT_RE = re.compile(r"([^:/\\]+)(:([\d]+))?")
! HTTP_ERROR_RE = re.compile(r"HTTP Error ([\d]+)")
! URL_KEY_RE = re.compile(r"[\W]")
! 
! class SlurpingURLStripper(URLStripper):
!     def __init__(self):
!         URLStripper.__init__(self)
!         self.setup_done = False
!         self.do_slurp = True
!         
!     def setup(self):
!         # Can't import this at the top because it's circular.
!         # XXX Someone smarter than me, please figure out the right
!         # XXX way to do this.
!         from spambayes.FileCorpus import ExpiryFileCorpus, FileMessageFactory
!         username = options["globals", "proxy_username"]
!         password = options["globals", "proxy_password"]
!         server = options["globals", "proxy_server"]
!         if server:
!             # Build a new opener that uses a proxy requiring authorization
!             proxy_support = urllib2.ProxyHandler({"http" : \
!                                                   "http://%s:%s@%s:%d" % \
!                                                   (username, password,
!                                                    server, port)})
!             opener = urllib2.build_opener(proxy_support,
!                                           urllib2.HTTPHandler)
!         else:
!             # Build a new opener without any proxy information.
!             opener = urllib2.build_opener(urllib2.HTTPHandler)
!          
!         # Install it
!         urllib2.install_opener(opener)
! 
!         # Setup the cache for retrieved urls
!         age = options["URLRetriever", "x-cache_expiry_days"]*24*60*60
!         dir = options["URLRetriever", "x-cache_directory"]
!         if not os.path.exists(dir):
!             # Create the directory.
!             if options["globals", "verbose"]:
!                 print >>sys.stderr, "Creating URL cache directory"
!             os.makedirs(dir)
! 
!         self.urlCorpus = ExpiryFileCorpus(age, FileMessageFactory(),
!                                           dir, cacheSize=20)
!         # Kill any old information in the cache
!         self.urlCorpus.removeExpiredMessages()
! 
!         # Setup caches for unretrievable urls
!         self.bad_url_cache_name = os.path.join(dir, "bad_urls.pck")
!         self.http_error_cache_name = os.path.join(dir, "http_error_urls.pck")
!         if os.path.exists(self.bad_url_cache_name):
!             b_file = file(self.bad_url_cache_name, "r")
!             self.bad_urls = pickle.load(b_file)
!             b_file.close()
!         else:
!             self.bad_urls = {"url:non_resolving": (),
!                         "url:non_html": (),
!                         "url:unknown_error": ()}
!         if os.path.exists(self.http_error_cache_name):
!             h_file = file(self.http_error_cache_name, "r")
!             self.http_error_urls = pickle.load(h_file)
!             h_file.close()
!         else:
!             self.http_error_urls = {}
! 
!     def _save_caches(self):
!         # XXX Note that these caches are never refreshed, which might not
!         # XXX be a good thing long-term (if a previously invalid URL
!         # XXX becomes valid, for example).
!         b_file = file(self.bad_url_cache_name, "w")
!         pickle.dump(self.bad_urls, b_file)
!         b_file.close()
!         h_file = file(self.http_error_cache_name, "w")
!         pickle.dump(self.http_error_urls, h_file)
!         h_file.close()
! 
!     def tokenize(self, m):
!         # XXX A weakness of this is that the text from URLs is
!         # XXX always retrieved, even if it won't be used (if the
!         # XXX raw score is outside unsure, for example).  The
!         # XXX problem is that when tokenizing, we have no idea
!         # XXX what the score of the message should be, and so
!         # XXX if we need the tokens or not.  But when calculating
!         # XXX the spamprob, we have no idea what the content of
!         # XXX the message is - just the tokens we generated from it
!         # XXX (and we can't reverse-engineer the URLs from that).
!         # XXX I've (Tony) played around with various ways to get
!         # XXX around this, but can't really come up with anything
!         # XXX good, apart from moving the decision whether to
!         # XXX recalculate the score 'higher' up (out of classifier's
!         # XXX spamprob()), but then it seems that code in a *lot*
!         # XXX of places will need to be changed to call the new
!         # XXX function; not nice given that this is experimental.
!         # XXX Either someone else will point out a good way to do this
!         # XXX or it can be moved higher up if this ever makes it out
!         # XXX of experimental status.
!         # XXX This might not matter so much because of the local
!         # XXX cache of the 'slurped' content, especially if the cache
!         # XXX isn't set to expire content regularly, and if your ham
!         # XXX (likely) and spam (unlikely) messages tend to have the
!         # XXX same URLs in them, and only unsure change.
!         # XXX Also note that the 'slurped' tokens are *always* trained
!         # XXX on; it would be simple to change/parameterize this.
!         if not self.setup_done:
!             self.setup()
!             self.setup_done = True
!         tokens = URLStripper.tokenize(self, m)
!         if not (options["URLRetriever", "x-slurp_urls"] and \
!            self.do_slurp):
!             return tokens
!         
!         # We don't want to do this recursively and check URLs
!         # on webpages, so we have this little cheat.
!         self.do_slurp = False
! 
!         proto, guts = m.groups()
!         if proto != "http":
!             return tokens
! 
!         assert guts
!         while guts and guts[-1] in '.:;?!/)':
!             guts = guts[:-1]
! 
!         classifier.slurp_wordstream = self.slurp(proto, guts)
!         self.do_slurp = True
!         self._save_caches()
!         return tokens
! 
!     def slurp(self, proto, url):
!         # We generate these tokens:
!         #  url:non_resolving
!         #  url:non_html
!         #  url:http_XXX (for each type of http error encounted,
!         #                for example 404, 403, ...)
!         # And tokenise the received page (but we do not slurp this).
!         # Actually, the special url: tokens barely showed up in my testing,
!         # although I would have thought that they would more - this might
!         # be due to an error, although they do turn up on occasion.  In
!         # any case, we have to do the test, so generating an extra token
!         # doesn't cost us anything apart from another entry in the db, and
!         # it's only two entries, plus one for each type of http error
!         # encountered, so it's pretty neglible.
!         if options["URLRetriever", "x-only_slurp_base"]:
!             url = self._base_url(url)
! 
!         # Check the unretrievable caches
!         for err in self.bad_urls.keys():
!             if url in self.bad_urls[err]:
!                 return [err]
!         if self.http_error_urls.has_key(url):
!             return self.http_error_urls[url]
! 
!         # We check if the url will resolve first
!         mo = DOMAIN_AND_PORT_RE.match(url)
!         domain = mo.group(1)
!         if mo.group(3) is None:
!             port = 80
!         else:
!             port = mo.group(3)
!         try:
!             not_used = socket.getaddrinfo(domain, port)
!         except socket.error:
!             self.bad_urls["url:non_resolving"] += (url,)
!             return ["url:non_resolving"]
! 
!         # If the message is in our cache, then we can just skip over
!         # retrieving it from the network, and get it from there, instead.
!         url_key = URL_KEY_RE.sub('_', url)
!         cached_message = self.urlCorpus.get(url_key)
!         
!         if cached_message is None:
!             # We're going to ignore everything that isn't text/html,
!             # so we might as well not bother retrieving anything with
!             # these extensions.
!             parts = url.split('.')
!             if parts[-1] in ('jpg', 'gif', 'png', 'css', 'js'):
!                 self.bad_urls["url:non_html"] += (url,)
!                 return ["url:non_html"]
!             
!             try:
!                 if options["globals", "verbose"]:
!                     print >>sys.stderr, "Slurping", url
!                 f = urllib2.urlopen("%s://%s" % (proto, url))
!             except (urllib2.URLError, socket.error), details:
!                 mo = HTTP_ERROR_RE.match(str(details))
!                 if mo:
!                     self.http_error_urls[url] = "url:http_" + mo.group(1)
!                     return ["url:http_" + mo.group(1)]
!                 self.bad_urls["url:unknown_error"] += (url,)
!                 return ["url:unknown_error"]
!             
!             # Anything that isn't text/html is ignored
!             content_type = f.info().get('content-type')
!             if content_type is None or \
!                not content_type.startswith("text/html"):
!                 self.bad_urls["url:non_html"] += (url,)
!                 return ["url:non_html"]
! 
!             page = f.read()
!             headers = str(f.info())
!             f.close()
!             fake_message_string = headers + "\r\n" + page
! 
!             # Retrieving the same messages over and over again will tire
!             # us out, so we store them in our own wee cache.
!             message = self.urlCorpus.makeMessage(url_key)
!             message.setPayload(fake_message_string)
!             self.urlCorpus.addMessage(message)
!         else:
!             fake_message_string = cached_message.as_string()
! 
!         msg = message_from_string(fake_message_string)
! 
!         # We don't want to do full header tokenising, as this is
!         # optimised for messages, not webpages, so we just do the
!         # basic stuff.
!         bht = options["Tokenizer", "basic_header_tokenize"]
!         bhto = options["Tokenizer", "basic_header_tokenize_only"]
!         options["Tokenizer", "basic_header_tokenize"] = True
!         options["Tokenizer", "basic_header_tokenize_only"] = True
! 
!         tokens = Tokenizer().tokenize(msg)
!         pf = options["URLRetriever", "x-web_prefix"]
!         tokens = ["%s%s" % (pf, tok) for tok in tokens]
! 
!         # Undo the changes
!         options["Tokenizer", "basic_header_tokenize"] = bht
!         options["Tokenizer", "basic_header_tokenize_only"] = bhto
!         return tokens
! 
!     def _base_url(self, url):
!         # To try and speed things up, and to avoid following
!         # unique URLS, we convert the URL to as basic a form
!         # as we can - so http://www.massey.ac.nz/~tameyer/index.html?you=me
!         # would become http://massey.ac.nz and http://id.example.com
!         # would become http://example.com
!         url += '/'
!         domain, garbage = url.split('/', 1)
!         parts = domain.split('.')
!         if len(parts) > 2:
!             base_domain = parts[-2] + '.' + parts[-1]
!             if len(parts[-1]) < 3:
!                 base_domain = parts[-3] + '.' + base_domain
!         else:
!             base_domain = domain
!         return base_domain
! 
! if options["URLRetriever", "x-slurp_urls"]:
!     crack_urls = SlurpingURLStripper().analyze
! else:
!     crack_urls = URLStripper().analyze
  
  # Nuke HTML <style gimmicks.



From anadelonbrin at users.sourceforge.net  Wed Dec 17 04:16:56 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Wed Dec 17 04:16:59 2003
Subject: [Spambayes-checkins] spambayes/testtools urlslurper.py,1.6,NONE
Message-ID: <E1AWXnc-0000Aw-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/testtools
In directory sc8-pr-cvs1:/tmp/cvs-serv671/testtools

Removed Files:
	urlslurper.py 
Log Message:
Add the basis of a new experimental (and highly debatable) option to 'slurp' URLs.

This is based on the urlslurper.py script in the testtools directory, which in turn
was based on Richard Jowsey's URLSlurper.java.

Basically, when the option is enabled, instead of just tokenizing the URLs in a message,
we also retrieve the content at that address (if it's not text, we ignore it).

When classifying, if the message has a 'raw' score in the unsure range, and if the
number of tokens is less than max_discriminators, and adding these 'slurped' tokens
would push the message into the ham/spam range, then they are used.

This isn't necessary anymore; use the experimental URLRetriever options
instead.

--- urlslurper.py DELETED ---



From tameyer at ihug.co.nz  Wed Dec 17 04:21:45 2003
From: tameyer at ihug.co.nz (Tony Meyer)
Date: Wed Dec 17 04:21:51 2003
Subject: [Spambayes-checkins] spambayes/testtools urlslurper.py,1.6,NONE
In-Reply-To: <1ED4ECF91CDED24C8D012BCF2B034F13047C0D79@its-xchg4.massey.ac.nz>
Message-ID: <1ED4ECF91CDED24C8D012BCF2B034F1304677770@its-xchg4.massey.ac.nz>

Opps.  The comment window had scrolled down and I didn't notice.  Only the
last line should be there in the comments for this.

> -----Original Message-----
> From: spambayes-checkins-bounces@python.org 
> [mailto:spambayes-checkins-bounces@python.org] On Behalf Of Tony Meyer
> Sent: Wednesday, 17 December 2003 10:17 p.m.
> To: spambayes-checkins@python.org
> Subject: [Spambayes-checkins] spambayes/testtools 
> urlslurper.py,1.6,NONE
> 
> 
> Update of /cvsroot/spambayes/spambayes/testtools
> In directory sc8-pr-cvs1:/tmp/cvs-serv671/testtools
> 
> Removed Files:
> 	urlslurper.py 
> Log Message:
> Add the basis of a new experimental (and highly debatable) 
> option to 'slurp' URLs.
> 
> This is based on the urlslurper.py script in the testtools 
> directory, which in turn
> was based on Richard Jowsey's URLSlurper.java.
> 
> Basically, when the option is enabled, instead of just 
> tokenizing the URLs in a message,
> we also retrieve the content at that address (if it's not 
> text, we ignore it).
> 
> When classifying, if the message has a 'raw' score in the 
> unsure range, and if the
> number of tokens is less than max_discriminators, and adding 
> these 'slurped' tokens
> would push the message into the ham/spam range, then they are used.
> 
> This isn't necessary anymore; use the experimental 
> URLRetriever options
> instead.
> 
> --- urlslurper.py DELETED ---
> 
> 
> 
> _______________________________________________
> Spambayes-checkins mailing list
> Spambayes-checkins@python.org
> http://mail.python.org/mailman/listinfo/spambayes-checkins
> 


From mhammond at users.sourceforge.net  Thu Dec 18 00:31:52 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Thu Dec 18 00:31:56 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000/dialogs
	FolderSelector.py, 1.32, 1.33
Message-ID: <E1AWqlM-0000D7-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000/dialogs
In directory sc8-pr-cvs1:/tmp/cvs-serv701

Modified Files:
	FolderSelector.py 
Log Message:
When the 'New Folder' button was used to create a folder, that folder
was not used when you closed the dialog, even though it was selected.
You had to reopen the dialog to select the new folder you created.


Index: FolderSelector.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/dialogs/FolderSelector.py,v
retrieving revision 1.32
retrieving revision 1.33
diff -C2 -d -r1.32 -r1.33
*** FolderSelector.py	16 Dec 2003 05:06:33 -0000	1.32
--- FolderSelector.py	18 Dec 2003 05:31:49 -0000	1.33
***************
*** 641,645 ****
                          # Create a new FolderSpec for this folder, and stash
                          new_spec = FolderSpec(new_folder.GetID(), name)
!                         self.item_map[item[7]] = new_spec
                          # And update the tree with the new item
                          buf, extra = PackTVITEM(handle, None, None, name, None, None, None, None)
--- 641,649 ----
                          # Create a new FolderSpec for this folder, and stash
                          new_spec = FolderSpec(new_folder.GetID(), name)
!                         # The info passed by the notify message appears to
!                         # not have the lparam (even though the docs say it
!                         # does.)  Fetch it
!                         spec_key = self._GetTVItem(handle)[7]
!                         self.item_map[spec_key] = new_spec
                          # And update the tree with the new item
                          buf, extra = PackTVITEM(handle, None, None, name, None, None, None, None)



From anadelonbrin at users.sourceforge.net  Thu Dec 18 01:41:54 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Thu Dec 18 01:41:58 2003
Subject: [Spambayes-checkins] spambayes/spambayes Options.py,1.94,1.95
Message-ID: <E1AWrr8-0002fu-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv10266/spambayes

Modified Files:
	Options.py 
Log Message:
Bring pspam into the modern spambayes world.  update.py, pop.py and scoremsg.py all seem to
work ok for me, with very limited testing.

Let the connection address be either a INET address (in the form "(server,port)", or a UNIX one.

Standardise the options code (move the options into the same place as all the rest, and dump
the pspam only file, which didn't work with the new OptionsClass anyway).  Note that the
options have no nice name or docstring, because I don't really know that I know enough to
fill those out.  They might vanish sometime anyway.

Index: Options.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/Options.py,v
retrieving revision 1.94
retrieving revision 1.95
diff -C2 -d -r1.94 -r1.95
*** Options.py	17 Dec 2003 09:09:52 -0000	1.94
--- Options.py	18 Dec 2003 06:41:52 -0000	1.95
***************
*** 1027,1030 ****
--- 1027,1060 ----
    ),
  
+   "ZODB" : (
+     ("zeo_addr", "", "",
+      """""",
+      IMAP_ASTRING, DO_NOT_RESTORE),
+ 
+     ("event_log_file", "", "",
+      """""",
+      IMAP_ASTRING, RESTORE),
+ 
+     ("folder_dir", "", "",
+      """""",
+      PATH, DO_NOT_RESTORE),
+ 
+     ("ham_folders", "", "",
+      """""",
+      PATH, DO_NOT_RESTORE),
+ 
+     ("spam_folders", "", "",
+      """""",
+      PATH, DO_NOT_RESTORE),
+ 
+     ("event_log_severity", "", 0,
+      """""",
+      INTEGER, RESTORE),
+ 
+     ("cache_size", "", 2000,
+      """""",
+      INTEGER, RESTORE),
+   ),
+ 
    "imapserver" : (
      ("username", "Username", "",



From anadelonbrin at users.sourceforge.net  Thu Dec 18 01:41:54 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Thu Dec 18 01:42:00 2003
Subject: [Spambayes-checkins] spambayes/pspam/pspam database.py, 1.1,
	1.2 profile.py, 1.7, 1.8 options.py, 1.5, NONE
Message-ID: <E1AWrr8-0002fr-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/pspam/pspam
In directory sc8-pr-cvs1:/tmp/cvs-serv10266/pspam/pspam

Modified Files:
	database.py profile.py 
Removed Files:
	options.py 
Log Message:
Bring pspam into the modern spambayes world.  update.py, pop.py and scoremsg.py all seem to
work ok for me, with very limited testing.

Let the connection address be either a INET address (in the form "(server,port)", or a UNIX one.

Standardise the options code (move the options into the same place as all the rest, and dump
the pspam only file, which didn't work with the new OptionsClass anyway).  Note that the
options have no nice name or docstring, because I don't really know that I know enough to
fill those out.  They might vanish sometime anyway.

Index: database.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/pspam/pspam/database.py,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** database.py	4 Nov 2002 04:44:20 -0000	1.1
--- database.py	18 Dec 2003 06:41:52 -0000	1.2
***************
*** 1,3 ****
! from pspam.options import options
  
  import ZODB
--- 1,3 ----
! from spambayes.Options import options
  
  import ZODB
***************
*** 8,17 ****
  
  def logging():
!     os.environ["STUPID_LOG_FILE"] = options.event_log_file
!     os.environ["STUPID_LOG_SEVERITY"] = str(options.event_log_severity)
      zLOG.initialize()
  
  def open():
!     cs = ClientStorage(options.zeo_addr)
!     db = ZODB.DB(cs, cache_size=options.cache_size)
      return db
--- 8,22 ----
  
  def logging():
!     os.environ["STUPID_LOG_FILE"] = options["ZODB", "event_log_file"]
!     os.environ["STUPID_LOG_SEVERITY"] = str(options["ZODB",
!                                                     "event_log_severity"])
      zLOG.initialize()
  
  def open():
!     addr = options["ZODB", "zeo_addr"]
!     if addr and addr[0] == "(" and addr[-1] == ")":
!         s, p = tuple(addr[1:-1].split(',', 1))
!         addr = s, int(p)
!     cs = ClientStorage(addr)
!     db = ZODB.DB(cs, cache_size=options["ZODB", "cache_size"])
      return db

Index: profile.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/pspam/pspam/profile.py,v
retrieving revision 1.7
retrieving revision 1.8
diff -C2 -d -r1.7 -r1.8
*** profile.py	14 Jan 2003 05:38:20 -0000	1.7
--- profile.py	18 Dec 2003 06:41:52 -0000	1.8
***************
*** 10,14 ****
  
  from pspam.folder import Folder
! from pspam.options import options
  
  import os
--- 10,14 ----
  
  from pspam.folder import Folder
! from spambayes.Options import options
  
  import os
***************
*** 50,54 ****
          return "WordInfo(%r, %r)" % (self.spamcount, self.hamcount)
  
! class PMetaInfo(classifier.MetaInfo, Persistent):
      pass
  
--- 50,56 ----
          return "WordInfo(%r, %r)" % (self.spamcount, self.hamcount)
  
! ##class PMetaInfo(classifier.MetaInfo, Persistent):
! ##    pass
! class PMetaInfo(Persistent):
      pass
  

--- options.py DELETED ---



From anadelonbrin at users.sourceforge.net  Thu Dec 18 01:41:54 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Thu Dec 18 01:42:02 2003
Subject: [Spambayes-checkins] spambayes/pspam zeo.bat, NONE, 1.1 README.txt,
	1.1, 1.2 pop.py, 1.5, 1.6 scoremsg.py, 1.3, 1.4 update.py, 1.3,
	1.4 vmspam.ini, 1.1, 1.2
Message-ID: <E1AWrr8-0002fl-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/pspam
In directory sc8-pr-cvs1:/tmp/cvs-serv10266/pspam

Modified Files:
	README.txt pop.py scoremsg.py update.py vmspam.ini 
Added Files:
	zeo.bat 
Log Message:
Bring pspam into the modern spambayes world.  update.py, pop.py and scoremsg.py all seem to
work ok for me, with very limited testing.

Let the connection address be either a INET address (in the form "(server,port)", or a UNIX one.

Standardise the options code (move the options into the same place as all the rest, and dump
the pspam only file, which didn't work with the new OptionsClass anyway).  Note that the
options have no nice name or docstring, because I don't really know that I know enough to
fill those out.  They might vanish sometime anyway.

--- NEW FILE: zeo.bat ---
set STUPID_LOG_FILE=c:\windows\temp\zeospam.log
python c:\python23\lib\site-packages\ZEO\start.py -p 8890 c:\windows\temp\zeospam.fs

Index: README.txt
===================================================================
RCS file: /cvsroot/spambayes/spambayes/pspam/README.txt,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** README.txt	4 Nov 2002 04:44:19 -0000	1.1
--- README.txt	18 Dec 2003 06:41:51 -0000	1.2
***************
*** 18,21 ****
--- 18,22 ----
  vmspam.ini -- a sample configuration file
  zeo.sh -- a script to start a ZEO server
+ zeo.bat -- a script to start a ZEO server on Windows
  
  The code depends on ZODB3, which you can download from

Index: pop.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/pspam/pop.py,v
retrieving revision 1.5
retrieving revision 1.6
diff -C2 -d -r1.5 -r1.6
*** pop.py	20 Jan 2003 20:24:54 -0000	1.5
--- pop.py	18 Dec 2003 06:41:51 -0000	1.6
***************
*** 45,51 ****
  import zLOG
  
! from tokenizer import tokenize
  import pspam.database
! from pspam.options import options
  
  try:
--- 45,51 ----
  import zLOG
  
! from spambayes.tokenizer import tokenize
  import pspam.database
! from spambayes.Options import options
  
  try:
***************
*** 99,103 ****
          parts = line.split()
          if parts[0] != "USER":
!             self.wfile.write("-ERR Invalid command; must specify USER first")
              return False
          user = parts[1]
--- 99,103 ----
          parts = line.split()
          if parts[0] != "USER":
!             self.wfile.write("-ERR Invalid command; must specify USER first\n")
              return False
          user = parts[1]
***************
*** 311,318 ****
      profile = r["profile"]
  
!     log = open("/var/tmp/pop.log", "ab")
      print >> log, "+PROXY start", time.ctime()
  
!     server = POP3ProxyServer(('', options.proxy_port),
                               POP3RequestHandler,
                               profile.classifier,
--- 311,319 ----
      profile = r["profile"]
  
!     log = open("pop.log", "ab")
      print >> log, "+PROXY start", time.ctime()
  
!     server = POP3ProxyServer(('', int(options["pop3proxy",
!                                               "listen_ports"][0])),
                               POP3RequestHandler,
                               profile.classifier,

Index: scoremsg.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/pspam/scoremsg.py,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** scoremsg.py	12 Nov 2002 07:03:20 -0000	1.3
--- scoremsg.py	18 Dec 2003 06:41:51 -0000	1.4
***************
*** 2,14 ****
  """Score a message provided on stdin and show the evidence."""
  
  import ZODB
  from ZEO.ClientStorage import ClientStorage
  
! from tokenizer import tokenize
! 
! import email
! import sys
! 
! import pspam.options
  
  try:
--- 2,14 ----
  """Score a message provided on stdin and show the evidence."""
  
+ import sys
+ import email
+ 
  import ZODB
  from ZEO.ClientStorage import ClientStorage
  
! import pspam.database
! from spambayes.Options import options
! from spambayes.tokenizer import tokenize
  
  try:
***************
*** 20,29 ****
  
  def main(fp):
!     cs = ClientStorage("/var/tmp/zeospam")
!     db = ZODB.DB(cs)
      r = db.open().root()
- 
-     # make sure scoring uses the right set of options
-     pspam.options.mergefile("/home/jeremy/src/vmspam/vmspam.ini")
  
      p = r["profile"]
--- 20,25 ----
  
  def main(fp):
!     db = pspam.database.open()
      r = db.open().root()
  
      p = r["profile"]

Index: update.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/pspam/update.py,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** update.py	12 Nov 2002 07:03:20 -0000	1.3
--- update.py	18 Dec 2003 06:41:51 -0000	1.4
***************
*** 8,12 ****
  import pspam.database
  from pspam.profile import Profile
! from pspam.options import options
  
  try:
--- 8,13 ----
  import pspam.database
  from pspam.profile import Profile
! 
! from spambayes.Options import options
  
  try:
***************
*** 31,45 ****
      if profile is None or rebuild:
          # if there is no profile, create it
!         profile = r["profile"] = Profile(options.folder_dir)
          get_transaction().commit()
  
      # check for new folders of training data
!     for ham in options.ham_folders:
!         p = os.path.join(options.folder_dir, ham)
          if not folder_exists(profile.hams, p):
              profile.add_ham(p)
  
!     for spam in options.spam_folders:
!         p = os.path.join(options.folder_dir, spam)
          if not folder_exists(profile.spams, p):
              profile.add_spam(p)
--- 32,46 ----
      if profile is None or rebuild:
          # if there is no profile, create it
!         profile = r["profile"] = Profile(options["ZODB", "folder_dir"])
          get_transaction().commit()
  
      # check for new folders of training data
!     for ham in options["ZODB", "ham_folders"].split(os.pathsep):
!         p = os.path.join(options["ZODB", "folder_dir"], ham)
          if not folder_exists(profile.hams, p):
              profile.add_ham(p)
  
!     for spam in options["ZODB", "spam_folders"].split(os.pathsep):
!         p = os.path.join(options["ZODB", "folder_dir"], spam)
          if not folder_exists(profile.spams, p):
              profile.add_spam(p)

Index: vmspam.ini
===================================================================
RCS file: /cvsroot/spambayes/spambayes/pspam/vmspam.ini,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** vmspam.ini	4 Nov 2002 04:44:19 -0000	1.1
--- vmspam.ini	18 Dec 2003 06:41:51 -0000	1.2
***************
*** 1,21 ****
! [Train]
  folder_dir: /home/jeremy/Mail
  spam_folders: train/spam
  ham_folders: train/ham
- 
- [Score]
- max_ham: 0.05
- min_spam: 0.99
- 
- [Proxy]
- server: mail.zope.com
- server_port: 110
- proxy_port: 1111
- log_pop_session: true
- log_pop_session_file: /var/tmp/pop.log
- 
- [ZODB]
  zeo_addr: /var/tmp/zeospam
  event_log_file: /var/tmp/zeospam.log
- event_log_severity: 0
- cache_size: 2000
--- 1,6 ----
! [ZODB]
  folder_dir: /home/jeremy/Mail
  spam_folders: train/spam
  ham_folders: train/ham
  zeo_addr: /var/tmp/zeospam
  event_log_file: /var/tmp/zeospam.log



From montanaro at users.sourceforge.net  Thu Dec 18 12:56:15 2003
From: montanaro at users.sourceforge.net (Skip Montanaro)
Date: Thu Dec 18 12:56:19 2003
Subject: [Spambayes-checkins] website windows.ht,1.29,1.30
Message-ID: <E1AX2Nj-0008Ti-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/website
In directory sc8-pr-cvs1:/tmp/cvs-serv32584

Modified Files:
	windows.ht 
Log Message:
fix link


Index: windows.ht
===================================================================
RCS file: /cvsroot/spambayes/website/windows.ht,v
retrieving revision 1.29
retrieving revision 1.30
diff -C2 -d -r1.29 -r1.30
*** windows.ht	16 Nov 2003 12:01:30 -0000	1.29
--- windows.ht	18 Dec 2003 17:56:13 -0000	1.30
***************
*** 74,78 ****
  href="https://sourceforge.net/project/showfiles.php?group_id=61702">Spambayes
  source</a><sup><font size="-2">1</font></sup>, then setup the proxy server
! (<a href="applications.html#pop3">POP3</a> or <a
  href="applications.html#imap">IMAP</a>) which corresponds to the way in
  which you transfer mail from your mail server.
--- 74,78 ----
  href="https://sourceforge.net/project/showfiles.php?group_id=61702">Spambayes
  source</a><sup><font size="-2">1</font></sup>, then setup the proxy server
! (<a href="applications.html#sb_server">POP3</a> or <a
  href="applications.html#imap">IMAP</a>) which corresponds to the way in
  which you transfer mail from your mail server.



From mhammond at users.sourceforge.net  Thu Dec 18 21:20:55 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Thu Dec 18 21:20:59 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 msgstore.py,1.80,1.81
Message-ID: <E1AXAG7-0001OO-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv5281

Modified Files:
	msgstore.py 
Log Message:
Convert COM exceptions to our MsgStore exceptions for GetOutlookItem


Index: msgstore.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/msgstore.py,v
retrieving revision 1.80
retrieving revision 1.81
diff -C2 -d -r1.80 -r1.81
*** msgstore.py	16 Dec 2003 05:06:33 -0000	1.80
--- msgstore.py	19 Dec 2003 02:20:53 -0000	1.81
***************
*** 544,550 ****
  
      def GetOutlookItem(self):
!         hex_item_id = mapi.HexFromBin(self.id[1])
!         hex_store_id = mapi.HexFromBin(self.id[0])
!         return self.msgstore.outlook.Session.GetFolderFromID(hex_item_id, hex_store_id)
  
      def GetMessageGenerator(self, only_filter_candidates = True):
--- 544,553 ----
  
      def GetOutlookItem(self):
!         try:
!             hex_item_id = mapi.HexFromBin(self.id[1])
!             hex_store_id = mapi.HexFromBin(self.id[0])
!             return self.msgstore.outlook.Session.GetFolderFromID(hex_item_id, hex_store_id)
!         except pythoncom.com_error, details:
!             raise MsgStoreExceptionFromCOMException(details)
  
      def GetMessageGenerator(self, only_filter_candidates = True):



From mhammond at users.sourceforge.net  Thu Dec 18 21:23:36 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Thu Dec 18 21:23:39 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 addin.py,1.118,1.119
Message-ID: <E1AXAIi-0001Uw-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv5484

Modified Files:
	addin.py 
Log Message:
* Try and tone down the toolbar message in the log to prevent people
  reporting it as a bug
* Handle situations where Outlook starts up in a confused state, which
  then confused us.


Index: addin.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/addin.py,v
retrieving revision 1.118
retrieving revision 1.119
diff -C2 -d -r1.118 -r1.119
*** addin.py	17 Dec 2003 06:31:57 -0000	1.118
--- addin.py	19 Dec 2003 02:23:34 -0000	1.119
***************
*** 833,837 ****
                          break
                      item.Delete(False)
!                     print "Deleted the dead popup control - re-creating"
                  except pythoncom.com_error, e:
                      print "ERROR: Failed to delete our dead toolbar control"
--- 833,838 ----
                          break
                      item.Delete(False)
!                     print "The above toolbar message is common - " \
!                           "recreating the toolbar..."
                  except pythoncom.com_error, e:
                      print "ERROR: Failed to delete our dead toolbar control"
***************
*** 1245,1249 ****
                                      config.watch_include_sub):
              event_hook = self._GetHookForFolder(folder)
!             if event_hook.use_timer:
                  print "Processing missed spam in folder '%s' by starting a timer" \
                        % (folder.name,)
--- 1246,1258 ----
                                      config.watch_include_sub):
              event_hook = self._GetHookForFolder(folder)
!             # Note event_hook may be none in some strange cases where we
!             # were unable to hook the events for the folder.  This is
!             # generally caused by a temporary Outlook issue rather than a
!             # problem of ours we need to address.
!             if event_hook is None:
!                 manager.LogDebug(0,
!                     "Skipping processing of missed messages in folder '%s', "
!                     "as it is not available" % folder.name)
!             elif event_hook.use_timer:
                  print "Processing missed spam in folder '%s' by starting a timer" \
                        % (folder.name,)
***************
*** 1302,1306 ****
  
      def _GetHookForFolder(self, folder):
!         ret = self.folder_hooks[folder.id]
          assert ret.target == folder
          return ret
--- 1311,1317 ----
  
      def _GetHookForFolder(self, folder):
!         ret = self.folder_hooks.get(folder.id)
!         if ret is None: # we were unable to hook events for this folder.
!             return None
          assert ret.target == folder
          return ret
***************
*** 1312,1317 ****
              existing = self.folder_hooks.get(msgstore_folder.id)
              if existing is None or existing.__class__ != HandlerClass:
-                 folder = msgstore_folder.GetOutlookItem()
                  name = msgstore_folder.GetFQName()
                  # Ensure the field is created before we hook the folder
                  # events, else there is a chance our event handler will
--- 1323,1341 ----
              existing = self.folder_hooks.get(msgstore_folder.id)
              if existing is None or existing.__class__ != HandlerClass:
                  name = msgstore_folder.GetFQName()
+                 try:
+                     folder = msgstore_folder.GetOutlookItem()
+                 except self.manager.message_store.MsgStoreException, details:
+                     # Exceptions here are most likely when the folder is valid
+                     # and available to MAPI, but not via the Outlook.
+                     # One good way to provoke this is to configure Outlook's
+                     # profile so default delivery is set to "None".  Then,
+                     # when you start Outlook, it immediately displays an
+                     # error and terminates.  During this process, the addin
+                     # is initialized, attempts to get the folders, and fails.
+                     print "FAILED to open the Outlook folder '%s' " \
+                           "to hook events" % name
+                     print details
+                     continue
                  # Ensure the field is created before we hook the folder
                  # events, else there is a chance our event handler will



From mhammond at users.sourceforge.net  Thu Dec 18 23:40:28 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Thu Dec 18 23:40:32 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 addin.py,1.119,1.120
Message-ID: <E1AXCRA-0006lx-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv25818

Modified Files:
	addin.py 
Log Message:
Move some of our init code from OnConnection to OnStartupComplete - which
probably is a better place for it, but mainly I am hoping to have an 
impact on the commonly reported:
[ 843521 ] E_FAIL exception watching folders disables SpamBayes


Index: addin.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/addin.py,v
retrieving revision 1.119
retrieving revision 1.120
diff -C2 -d -r1.119 -r1.120
*** addin.py	19 Dec 2003 02:23:34 -0000	1.119
--- addin.py	19 Dec 2003 04:40:26 -0000	1.120
***************
*** 1197,1228 ****
              self.explorers_events = None # create at OnStartupComplete
  
-             if self.manager.never_configured:
-                 import dialogs
-                 dialogs.ShowWizard(0, self.manager)
- 
-             if self.manager.config.filter.enabled:
-                 # A little "sanity test" to help the user.  If our status is
-                 # 'enabled', then it means we have previously managed to
-                 # convince the manager dialog to enable.  If for some reason,
-                 # we no folder definitions but are 'enabled', then it is likely
-                 # something got hosed and the user doesn't know.
-                 if not self.manager.config.filter.spam_folder_id or \
-                    not self.manager.config.filter.watch_folder_ids:
-                     msg = "It appears there was an error loading your configuration\r\n\r\n" \
-                           "Please re-configure SpamBayes via the SpamBayes dropdown"
-                     self.manager.ReportError(msg)
-                 # But continue on regardless.
-                 self.FiltersChanged()
-                 try:
-                     self.ProcessMissedMessages()
-                 except:
-                     print "Error processing missed messages!"
-                     traceback.print_exc()
-             else:
-                 # We should include this fact in the log, as I suspect a
-                 # a number of "it doesn't work" bugs are simply related to not
-                 # being enabled.  The new Wizard should help, but things can
-                 # still screw up.
-                 self.manager.LogDebug(0, "*** SpamBayes is NOT enabled, so will not filter incoming mail. ***")
              if connectMode == constants.ext_cm_AfterStartup:
                  # We are being enabled after startup, which means we don't get
--- 1197,1200 ----
***************
*** 1237,1240 ****
--- 1209,1260 ----
                  "Please re-start Outlook and try again.")
  
+     def OnStartupComplete(self, custom):
+         # Setup all our filters and hooks.  We used to do this in OnConnection,
+         # but a number of 'strange' bugs were reported which I suspect would
+         # go away if done during this later event - and this later place
+         # does seem more "correct" than the initial OnConnection event.
+         if self.manager.never_configured:
+             import dialogs
+             dialogs.ShowWizard(0, self.manager)
+         if self.manager.config.filter.enabled:
+             # A little "sanity test" to help the user.  If our status is
+             # 'enabled', then it means we have previously managed to
+             # convince the manager dialog to enable.  If for some reason,
+             # we no folder definitions but are 'enabled', then it is likely
+             # something got hosed and the user doesn't know.
+             # Note that we could display the config wizard here, but this
+             # has rarely been reported in the wild since the very early
+             # days, so could possibly die.
+             if not self.manager.config.filter.spam_folder_id or \
+                not self.manager.config.filter.watch_folder_ids:
+                 msg = "It appears there was an error loading your configuration\r\n\r\n" \
+                       "Please re-configure SpamBayes via the SpamBayes dropdown"
+                 self.manager.ReportError(msg)
+             # But continue on regardless.
+             self.FiltersChanged()
+             try:
+                 self.ProcessMissedMessages()
+             except:
+                 print "Error processing missed messages!"
+                 traceback.print_exc()
+         else:
+             # We should include this fact in the log, as I suspect a
+             # a number of "it doesn't work" bugs are simply related to not
+             # being enabled.  The new Wizard should help, but things can
+             # still screw up.
+             self.manager.LogDebug(0, "*** SpamBayes is NOT enabled, so will " \
+                                      "not filter incoming mail. ***")
+         # Toolbar and other UI stuff must be setup once startup is complete.
+         explorers = self.application.Explorers
+         if self.manager is not None: # If we successfully started up.
+             # and Explorers events so we know when new explorers spring into life.
+             self.explorers_events = WithEvents(explorers, ExplorersEvent)
+             self.explorers_events.Init(self.manager)
+             # And hook our UI elements to all existing explorers
+             for i in range(explorers.Count):
+                 explorer = explorers.Item(i+1)
+                 explorer = self.explorers_events._DoNewExplorer(explorer)
+                 explorer.OnFolderSwitch()
+ 
      def ProcessMissedMessages(self):
          from time import clock
***************
*** 1421,1436 ****
      def OnAddInsUpdate(self, custom):
          pass
-     def OnStartupComplete(self, custom):
-         # Toolbar and other UI stuff must be setup once startup is complete.
-         explorers = self.application.Explorers
-         if self.manager is not None: # If we successfully started up.
-             # and Explorers events so we know when new explorers spring into life.
-             self.explorers_events = WithEvents(explorers, ExplorersEvent)
-             self.explorers_events.Init(self.manager)
-             # And hook our UI elements to all existing explorers
-             for i in range(explorers.Count):
-                 explorer = explorers.Item(i+1)
-                 explorer = self.explorers_events._DoNewExplorer(explorer)
-                 explorer.OnFolderSwitch()
  
      def OnBeginShutdown(self, custom):
--- 1441,1444 ----



From anadelonbrin at users.sourceforge.net  Fri Dec 19 00:37:49 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Fri Dec 19 00:37:52 2003
Subject: [Spambayes-checkins] spambayes/spambayes tokenizer.py,1.23,1.24
Message-ID: <E1AXDKf-0000YI-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv2120/spambayes

Modified Files:
	tokenizer.py 
Log Message:
Fix a bug where the URLRetriever experimental option wouldn't work with the
proxy setting set.

Index: tokenizer.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/tokenizer.py,v
retrieving revision 1.23
retrieving revision 1.24
diff -C2 -d -r1.23 -r1.24
*** tokenizer.py	17 Dec 2003 09:09:52 -0000	1.23
--- tokenizer.py	19 Dec 2003 05:37:47 -0000	1.24
***************
*** 1024,1027 ****
--- 1024,1031 ----
          password = options["globals", "proxy_password"]
          server = options["globals", "proxy_server"]
+         if server.find(":") != -1:
+             server, port = server.split(':', 1)
+         else:
+             port = 8080
          if server:
              # Build a new opener that uses a proxy requiring authorization



From mhammond at users.sourceforge.net  Fri Dec 19 01:25:17 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Fri Dec 19 01:25:21 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 msgstore.py,1.81,1.82
Message-ID: <E1AXE4b-0002Gn-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv8630

Modified Files:
	msgstore.py 
Log Message:
Allow the test suite to raise an error only a certain number of times.


Index: msgstore.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/msgstore.py,v
retrieving revision 1.81
retrieving revision 1.82
diff -C2 -d -r1.81 -r1.82
*** msgstore.py	19 Dec 2003 02:20:53 -0000	1.81
--- msgstore.py	19 Dec 2003 06:25:15 -0000	1.82
***************
*** 48,55 ****
--- 48,62 ----
  test_suite_failure_request = None
  test_suite_failure = None
+ # Set to the number of times we should fail, or None for all times.
+ test_suite_failure_count = None 
  # Sometimes the test suite will request that we simulate MAPI errors.
  def help_test_suite(checkpoint_name):
+     global test_suite_failure_request, test_suite_failure_count
      if test_suite_running and \
         test_suite_failure_request == checkpoint_name:
+         if test_suite_failure_count:
+             test_suite_failure_count -= 1
+             if test_suite_failure_count==0:
+                 test_suite_failure_request = None
          raise test_suite_failure[0], test_suite_failure[1]
  



From mhammond at users.sourceforge.net  Fri Dec 19 01:27:32 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Fri Dec 19 01:27:35 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 filter.py,1.36,1.37
Message-ID: <E1AXE6m-0002LA-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv8768

Modified Files:
	filter.py 
Log Message:
Handle E_OBJECT_CHANGED exception when filtering a message.  This should
fix bug [ 803798 ] MAPI_E_OBJECT_CHANGED error saving spam score, which is
a dupe of [787676], which was incorrectly marked as fixed (as the fix did
not work)

Don't record in the training database unless we are successful in the
filter - otherwise future attempts to filter will get all screwed up,
as it will think it already was.


Index: filter.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/filter.py,v
retrieving revision 1.36
retrieving revision 1.37
diff -C2 -d -r1.36 -r1.37
*** filter.py	27 Oct 2003 03:34:59 -0000	1.36
--- filter.py	19 Dec 2003 06:27:29 -0000	1.37
***************
*** 13,17 ****
      config = mgr.config.filter
      prob = mgr.score(msg)
-     mgr.stats.RecordClassification(prob)
      prob_perc = prob * 100
      if prob_perc >= config.spam_threshold:
--- 13,16 ----
***************
*** 28,43 ****
      try:
          try:
              if config.save_spam_info:
!                 # Save the score
!                 # Catch this exception, as failing to save the score need not
!                 # be fatal - it may still be possible to perform the move.
!                 msg.SetField(mgr.config.general.field_score_name, prob)
!                 # and the ID of the folder we were in when scored.
!                 # (but only if we want to perform all actions)
!                 # Note we must do this, and the Save, before the
!                 # filter, else the save will fail.
!                 if all_actions:
!                     msg.RememberMessageCurrentFolder()
!                 msg.Save()
          except ms.ReadOnlyException:
              # read-only message - not much we can do!
--- 27,63 ----
      try:
          try:
+             # Save the score
+             # Catch msgstore exceptions, as failing to save the score need
+             # not be fatal - it may still be possible to perform the move.
              if config.save_spam_info:
!                 # The object can sometimes change underneath us (most
!                 # noticably Hotmail, but reported in other cases too).
!                 # Retry 3 times handling ObjectChanged exception.
!                 # Why 3?  Why not!
!                 for i in range(3):
!                     try:
!                         msg.SetField(mgr.config.general.field_score_name, prob)
!                         # and the ID of the folder we were in when scored.
!                         # (but only if we want to perform all actions)
!                         # Note we must do this, and the Save, before the
!                         # filter, else the save will fail.
!                         if all_actions:
!                             msg.RememberMessageCurrentFolder()
!                         msg.Save()
!                         break
!                     except ms.ObjectChangedException:
!                         # Someone has changed the message underneath us.
!                         # The general solution is to re-open the message, and
!                         # try again.  We reach into our knowledge of the
!                         # message to force this.
!                         mgr.LogDebug(1, "Got ObjectChanged changed - " \
!                                         "trying again...")
!                         msg.dirty = False
!                         msg.mapi_object = None # cause it to be re-fetched.
!                 else:
!                     # Give up trying to save the score.
!                     mgr.LogDebug(0, "Got ObjectChanged 3 times in a row - " \
!                                     "giving up!")
!                     msg.dirty = False
          except ms.ReadOnlyException:
              # read-only message - not much we can do!
***************
*** 83,86 ****
--- 103,107 ----
                  raise RuntimeError, "Eeek - bad action '%r'" % (action,)
  
+         mgr.stats.RecordClassification(prob)
          return disposition
      except:



From mhammond at users.sourceforge.net  Fri Dec 19 01:28:00 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Fri Dec 19 01:28:03 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 addin.py,1.120,1.121
Message-ID: <E1AXE7E-0002Lw-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv9020

Modified Files:
	addin.py 
Log Message:
Remove handling of E_OBJECT_CHANGED exception, as it simply did not work.


Index: addin.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/addin.py,v
retrieving revision 1.120
retrieving revision 1.121
diff -C2 -d -r1.120 -r1.121
*** addin.py	19 Dec 2003 04:40:26 -0000	1.120
--- addin.py	19 Dec 2003 06:27:58 -0000	1.121
***************
*** 365,389 ****
          if not self.use_timer or not item.UnRead:
              ms = self.manager.message_store
!             # The object can sometimes change underneath us (most noticably
!             # with hotmail, which presumably has greater 'synchronization'
!             # issues than tightly bound stores.  We accept this error exactly
!             # 2 times (once as we have seen it, and twice as it can't hurt ;)
!             for i in range(3):
!                 try:
!                     msgstore_message = ms.GetMessage(item)
!                     ProcessMessage(msgstore_message, self.manager)
!                     break
!                 except ms.ObjectChangedException:
!                     # try again.
!                     self.manager.LogDebug(1, "Got object changed error - trying again...")
!                     continue
!                 except ms.MsgStoreException, details:
!                     print "Unexpected error fetching message"
!                     traceback.print_exc()
!                     print details
!                     break
!             else:
!                 print "WARNING: Could not filter a message, as we got an " \
!                       "'object changed' exception 3 times!"
          else:
              self._StartTimer()
--- 365,370 ----
          if not self.use_timer or not item.UnRead:
              ms = self.manager.message_store
!             msgstore_message = ms.GetMessage(item)
!             ProcessMessage(msgstore_message, self.manager)
          else:
              self._StartTimer()



From mhammond at users.sourceforge.net  Fri Dec 19 01:28:58 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Fri Dec 19 01:29:00 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 tester.py,1.21,1.22
Message-ID: <E1AXE8A-0002O1-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv9138

Modified Files:
	tester.py 
Log Message:
Ask if you want the slow, non-filter tests run.
Add E_OBJECT_CHANGED tests, as per [ 803798 ] MAPI_E_OBJECT_CHANGED error 
saving spam score


Index: tester.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/tester.py,v
retrieving revision 1.21
retrieving revision 1.22
diff -C2 -d -r1.21 -r1.22
*** tester.py	16 Dec 2003 05:06:33 -0000	1.21
--- tester.py	19 Dec 2003 06:28:55 -0000	1.22
***************
*** 556,560 ****
      msgstore.test_suite_failure_request = None
  
! def _setup_for_mapi_failure(checkpoint, hr):
      assert msgstore.test_suite_running, "msgstore should already know its running"
      assert not msgstore.test_suite_failure, "should already have torn down previous failure"
--- 556,560 ----
      msgstore.test_suite_failure_request = None
  
! def _setup_for_mapi_failure(checkpoint, hr, fail_count = None):
      assert msgstore.test_suite_running, "msgstore should already know its running"
      assert not msgstore.test_suite_failure, "should already have torn down previous failure"
***************
*** 562,576 ****
                           (hr, "testsuite generated error", None, -1)
      msgstore.test_suite_failure_request = checkpoint
  
  def _setup_mapi_notfound_failure(checkpoint):
      _setup_for_mapi_failure(checkpoint, mapi.MAPI_E_NOT_FOUND)
  
! def _do_single_failure_ham_test(driver, checkpoint, hr):
!     _do_single_failure_test(driver, True, checkpoint, hr)
  
! def _do_single_failure_spam_test(driver, checkpoint, hr):
!     _do_single_failure_test(driver, False, checkpoint, hr)
  
! def _do_single_failure_test(driver, is_ham, checkpoint, hr):
      print "-> Testing MAPI error '%s' in %s" % (mapiutil.GetScodeString(hr),
                                                checkpoint)
--- 562,577 ----
                           (hr, "testsuite generated error", None, -1)
      msgstore.test_suite_failure_request = checkpoint
+     msgstore.test_suite_failure_count = fail_count
  
  def _setup_mapi_notfound_failure(checkpoint):
      _setup_for_mapi_failure(checkpoint, mapi.MAPI_E_NOT_FOUND)
  
! def _do_single_failure_ham_test(driver, checkpoint, hr, fail_count = None):
!     _do_single_failure_test(driver, True, checkpoint, hr, fail_count)
  
! def _do_single_failure_spam_test(driver, checkpoint, hr, fail_count = None):
!     _do_single_failure_test(driver, False, checkpoint, hr, fail_count)
  
! def _do_single_failure_test(driver, is_ham, checkpoint, hr, fail_count):
      print "-> Testing MAPI error '%s' in %s" % (mapiutil.GetScodeString(hr),
                                                checkpoint)
***************
*** 583,587 ****
              msg, words = driver.CreateTestMessageInFolder(SPAM, folder)
          try:
!             _setup_for_mapi_failure(checkpoint, hr)
              try:
                  # sleep to ensure filtering.
--- 584,588 ----
              msg, words = driver.CreateTestMessageInFolder(SPAM, folder)
          try:
!             _setup_for_mapi_failure(checkpoint, hr, fail_count)
              try:
                  # sleep to ensure filtering.
***************
*** 610,613 ****
--- 611,616 ----
          _do_single_failure_ham_test(driver, "MAPIMsgStoreMsg.SetField", -2146644781)
          _do_single_failure_ham_test(driver, "MAPIMsgStoreMsg.Save", -2146644781)
+         _do_single_failure_ham_test(driver, "MAPIMsgStoreMsg.Save",
+                                     mapi.MAPI_E_OBJECT_CHANGED, fail_count=1)
          # SetReadState???
          _do_single_failure_spam_test(driver, "MAPIMsgStoreMsg._DoCopyMove", mapi.MAPI_E_TABLE_TOO_BIG)
***************
*** 651,656 ****
          run_failure_tests(manager)
          run_invalid_id_tests(manager)
!         # non-filter tests take alot of time - do them last.
!         run_nonfilter_tests(manager)
          print "*" * 20
          print "Test suite finished without error!"
--- 654,661 ----
          run_failure_tests(manager)
          run_invalid_id_tests(manager)
!         # non-filter tests take alot of time - ask if you want to do them
!         if manager.AskQuestion("Do you want to run the non-filter tests?" \
!                                "\r\n\r\nThese may take some time"):
!             run_nonfilter_tests(manager)
          print "*" * 20
          print "Test suite finished without error!"



From montanaro at users.sourceforge.net  Fri Dec 19 07:00:31 2003
From: montanaro at users.sourceforge.net (Skip Montanaro)
Date: Fri Dec 19 07:00:34 2003
Subject: [Spambayes-checkins] spambayes/spambayes tokenizer.py,1.24,1.25
Message-ID: <E1AXJJ1-000716-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv26954

Modified Files:
	tokenizer.py 
Log Message:
Solved the "backwards breakdown" problem with ip addresses in Received:
headers.  Also tightened up recognition of hostnames and accepted bracketed
or parenthesized ip addresses without requiring a leading space.  Who knows
if it will help, but our databases will be much cleaner.  (Obviously, full
retrain required to take advantage of the change.)


Index: tokenizer.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/tokenizer.py,v
retrieving revision 1.24
retrieving revision 1.25
diff -C2 -d -r1.24 -r1.25
*** tokenizer.py	19 Dec 2003 05:37:47 -0000	1.24
--- tokenizer.py	19 Dec 2003 12:00:28 -0000	1.25
***************
*** 641,646 ****
  """, re.VERBOSE | re.DOTALL)
  
! received_host_re = re.compile(r'from (\S+)\s')
! received_ip_re = re.compile(r'\s[[(]((\d{1,3}\.?){4})[\])]')
  
  message_id_re = re.compile(r'\s*<[^@]+@([^>]+)>\s*')
--- 641,668 ----
  """, re.VERBOSE | re.DOTALL)
  
! # Trailing letter serves to reject "hostnames" which are really ip
! # addresses.  Some spammers forge their apparent ip addresses, so you get
! # Received: headers which look like:
! #   Received: from 199.249.165.175 ([218.5.93.116])
! #       by manatee.mojam.com (8.12.1-20030917/8.12.1) with SMTP id
! #       hBIERsqI018090
! #       for <itinerary@musi-cal.com>; Thu, 18 Dec 2003 08:28:11 -0600
! # "199.249.165.175" is who the spamhaus said it was.  That's really the
! # ip address of the receiving host (manatee.mojam.com), which correctly
! # identified the sender's ip address as 218.5.93.116.
! #
! # Similarly, the more complex character set instead of just \S serves to
! # reject Received: headers where the message bounces from one user to
! # another on the local machine:
! #   Received: (from itin@localhost)
! #       by manatee.mojam.com (8.12.1-20030917/8.12.1/Submit) id hBIEQFxF018044
! #       for skip@manatee.mojam.com; Thu, 18 Dec 2003 08:26:15 -0600
! received_host_re = re.compile(r'from ([a-zA-Z0-9._-]+[a-zA-Z])[)\s]')
! # 99% of the time, the receiving host places the sender's ip address in
! # square brackets as it should, but every once in awhile it turns up in
! # parens.  Yahoo seems to be guilty of this minor infraction:
! #   Received: from unknown (66.218.66.218)
! #       by m19.grp.scd.yahoo.com with QMQP; 19 Dec 2003 04:06:53 -0000
! received_ip_re = re.compile(r'[[(]((\d{1,3}\.?){4})[])]')
  
  message_id_re = re.compile(r'\s*<[^@]+@([^>]+)>\s*')



From mhammond at users.sourceforge.net  Fri Dec 19 07:42:22 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Fri Dec 19 07:42:25 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000/dialogs/resources
	dialogs.rc, 1.42, 1.43
Message-ID: <E1AXJxW-0000Qp-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000/dialogs/resources
In directory sc8-pr-cvs1:/tmp/cvs-serv1587

Modified Files:
	dialogs.rc 
Log Message:
Add WS_SYSMENU to the dialog style - this seems to enable the help
button for win9x as per [ 830865 ] No dialog context-help button on Win9x


Index: dialogs.rc
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/dialogs/resources/dialogs.rc,v
retrieving revision 1.42
retrieving revision 1.43
diff -C2 -d -r1.42 -r1.43
*** dialogs.rc	9 Dec 2003 04:02:43 -0000	1.42
--- dialogs.rc	19 Dec 2003 12:42:20 -0000	1.43
***************
*** 486,490 ****
  
  IDD_GENERAL DIALOGEX 0, 0, 253, 210
! STYLE DS_SETFONT | DS_MODALFRAME | WS_CHILD | WS_VISIBLE | WS_CAPTION
  EXSTYLE WS_EX_CONTEXTHELP
  CAPTION "General"
--- 486,491 ----
  
  IDD_GENERAL DIALOGEX 0, 0, 253, 210
! STYLE DS_SETFONT | DS_MODALFRAME | WS_CHILD | WS_VISIBLE | WS_CAPTION | 
!     WS_SYSMENU
  EXSTYLE WS_EX_CONTEXTHELP
  CAPTION "General"



From mhammond at users.sourceforge.net  Sat Dec 20 00:38:52 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Sat Dec 20 00:38:56 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000/dialogs
	FolderSelector.py, 1.33, 1.34
Message-ID: <E1AXZpE-0005Qk-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000/dialogs
In directory sc8-pr-cvs1:/tmp/cvs-serv20842

Modified Files:
	FolderSelector.py 
Log Message:
Fix [ 860410 ] SpamBayes allows top-level folders to be selected, and also
prevent a single-select dialog from closing with no selection


Index: FolderSelector.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/dialogs/FolderSelector.py,v
retrieving revision 1.33
retrieving revision 1.34
diff -C2 -d -r1.33 -r1.34
*** FolderSelector.py	18 Dec 2003 05:31:49 -0000	1.33
--- FolderSelector.py	20 Dec 2003 05:38:50 -0000	1.34
***************
*** 291,294 ****
--- 291,295 ----
          self.exclude_prop_ids = exclude_prop_ids
          self.in_label_edit = False
+         self.in_check_selections_valid = False
  
      def CompareIDs(self, id1, id2):
***************
*** 423,426 ****
--- 424,429 ----
                                           commctrl.TVGN_CARET, 0)
              except win32gui.error:
+                 h = 0
+             if not h: # nothing selected.
                  return
              info = self._GetTVItem(h)
***************
*** 446,449 ****
--- 449,505 ----
          return ret, check != 0
  
+     def UnselectItem(self, item):
+         if self.single_select:
+             win32gui.SendMessage(self.list,
+                                  commctrl.TVM_SELECTITEM,
+                                  commctrl.TVGN_CARET, 0)
+         else:
+             state = INDEXTOSTATEIMAGEMASK(IIL_UNCHECKED)
+             mask = commctrl.TVIS_STATEIMAGEMASK
+             buf, extra = PackTVITEM(item[0], state, mask,
+                                     None, None, None, None, None)
+             win32gui.SendMessage(self.list, commctrl.TVM_SETITEM,
+                                  0, buf)
+         
+     def _CheckSelectionsValid(self, is_close = False):
+         if self.in_check_selections_valid:
+             return
+         self.in_check_selections_valid = True
+         try:
+             if self.single_select:
+                 if is_close:
+                     # Make sure one is selected.
+                     for ignore in self._YieldCheckedChildren():
+                         break
+                     else:
+                         self.manager.ReportInformation("You must select a folder")
+                         return False
+                 else:
+                     # In a single-select dialog, we can't stop the user selecting
+                     # a 'top-level' folder - we can only stop them closing the
+                     # dialog while it is selected.
+                     return True
+             # For a multi-select dialog, we simply un-check the existing item.
+             # For single-select, we set no item selected.
+             result_valid = True
+             for info, spec in self._YieldCheckedChildren():
+                 try:
+                     folder = self.manager.message_store.GetFolder(spec.folder_id)
+                     parent = folder.GetParent()
+                     valid = parent is not None and parent.GetParent() is not None
+                 except self.manager.message_store.MsgStoreException, details:
+                     print "Eeek - couldn't get the folder to check valid"
+                     valid = False
+                 if not valid:
+                     if result_valid: # are we the first invalid?
+                         self.manager.ReportInformation(
+                             "Please select a child folder - top-level folders " \
+                             "can not be used.")
+                     self.UnselectItem(info)
+                 result_valid = result_valid and valid
+             return result_valid
+         finally:
+             self.in_check_selections_valid = False
+ 
      # Message processing
  #    def GetMessageMap(self):
***************
*** 521,524 ****
--- 577,582 ----
              # Button clicks
              if id == win32con.IDOK:
+                 if not self._CheckSelectionsValid(True):
+                     return
                  self.selected_ids, self.checkbox_state = self.GetSelectedIDs()
                  win32gui.EndDialog(hwnd, id)
***************
*** 527,536 ****
              elif id_name == "IDC_BUT_CLEARALL":
                  for info, spec in self._YieldCheckedChildren():
!                     state = INDEXTOSTATEIMAGEMASK(IIL_UNCHECKED)
!                     mask = commctrl.TVIS_STATEIMAGEMASK
!                     buf, extra = PackTVITEM(info[0], state, mask,
!                                             None, None, None, None, None)
!                     win32gui.SendMessage(self.list, commctrl.TVM_SETITEM,
!                                          0, buf)
              elif id_name == "IDC_BUT_NEW":
                  # Force a new entry in the tree at our location, and begin
--- 585,589 ----
              elif id_name == "IDC_BUT_CLEARALL":
                  for info, spec in self._YieldCheckedChildren():
!                     self.UnselectItem(info)
              elif id_name == "IDC_BUT_NEW":
                  # Force a new entry in the tree at our location, and begin
***************
*** 567,591 ****
  
      def _DoUpdateStatus(self, id, timeval):
!         try:
!             names = []
!             num_checked = 0
!             for info, spec in self._YieldCheckedChildren():
!                 num_checked += 1
!                 if len(names) < 20:
!                     names.append(info[3])
  
!             status_string = "%s%s %d folder" % (self.select_desc_noun,
!                                                 self.select_desc_noun_suffix,
!                                                 num_checked)
!             if num_checked != 1:
!                 status_string += "s"
!             self.SetDlgItemText("IDC_STATUS1", status_string)
!             self.SetDlgItemText("IDC_STATUS2", "; ".join(names))
!         finally:
!             import timer
!             self.timer_id = None
!             timer.kill_timer(id)
  
      def _UpdateStatus(self):
          import timer
          if self.timer_id is not None:
--- 620,648 ----
  
      def _DoUpdateStatus(self, id, timeval):
!         import timer
!         # Kill the timer first to prevent it firing again.
!         self.timer_id = None
!         timer.kill_timer(id)
!         self._CheckSelectionsValid()
!         names = []
!         num_checked = 0
!         for info, spec in self._YieldCheckedChildren():
!             num_checked += 1
!             if len(names) < 20:
!                 names.append(info[3])
  
!         status_string = "%s%s %d folder" % (self.select_desc_noun,
!                                             self.select_desc_noun_suffix,
!                                             num_checked)
!         if num_checked != 1:
!             status_string += "s"
!         self.SetDlgItemText("IDC_STATUS1", status_string)
!         self.SetDlgItemText("IDC_STATUS2", "; ".join(names))
  
      def _UpdateStatus(self):
+         # We have problems with the order of events - we get the notification
+         # events before the new states are available via GetItem.
+         # Therefore, we start a one-shot, immediate timer, which ends up
+         # at the end of the message queue, and we work.
          import timer
          if self.timer_id is not None:
***************
*** 604,609 ****
                  self._UpdateStatus()
              elif code == commctrl.NM_DBLCLK:
!                 if self.single_select: # Only close on double-click for single-select
!                     self.OnOK()
              elif code == commctrl.TVN_ITEMEXPANDING:
                  ignore, ignore, ignore, action, itemOld, itemNew = \
--- 661,668 ----
                  self._UpdateStatus()
              elif code == commctrl.NM_DBLCLK:
!                 # No special dblclick handling - default behaviour is to
!                 # expand/collapse tree, and auto-closing the dialog, even
!                 # when the folder has no children, doesn't really make sense.
!                 pass
              elif code == commctrl.TVN_ITEMEXPANDING:
                  ignore, ignore, ignore, action, itemOld, itemNew = \
***************
*** 668,671 ****
--- 727,731 ----
  
  def Test():
+     single_select =False
      import sys, os
      sys.path.append(os.path.abspath(os.path.join(os.path.dirname(sys.argv[0]), "..")))
***************
*** 677,686 ****
  
      ids = [("0000","0000"),] # invalid ID for testing.
!     d=FolderSelector(0, mgr, ids, single_select = False)
      if d.DoModal() != win32con.IDOK:
          print "Cancelled"
          return
      ids, include_sub = d.GetSelectedIDs()
!     d=FolderSelector(0, mgr, ids, single_select = False, checkbox_state = include_sub)
      d.DoModal()
  
--- 737,746 ----
  
      ids = [("0000","0000"),] # invalid ID for testing.
!     d=FolderSelector(0, mgr, ids, single_select = single_select)
      if d.DoModal() != win32con.IDOK:
          print "Cancelled"
          return
      ids, include_sub = d.GetSelectedIDs()
!     d=FolderSelector(0, mgr, ids, single_select = single_select, checkbox_state = include_sub)
      d.DoModal()
  



From mhammond at users.sourceforge.net  Sat Dec 20 01:25:15 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Sat Dec 20 01:25:18 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 addin.py,1.121,1.122
Message-ID: <E1AXaY7-0006pb-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv26154

Modified Files:
	addin.py 
Log Message:
Include the foldername in many messages, so help track down wierd bugs
from user logs.  Say what we are watching a folder for.


Index: addin.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/addin.py,v
retrieving revision 1.121
retrieving revision 1.122
diff -C2 -d -r1.121 -r1.122
*** addin.py	19 Dec 2003 06:27:58 -0000	1.121
--- addin.py	20 Dec 2003 06:25:12 -0000	1.122
***************
*** 124,128 ****
      import train
      subject = msgstore_message.subject
!     print "Training on message '%s' - " % subject,
      if train.train_message(msgstore_message, False, manager.classifier_data):
          print "trained as good"
--- 124,130 ----
      import train
      subject = msgstore_message.subject
!     print "Training on message '%s' in '%s - " % \
!             (subject,
!              msgstore_message.GetFolder().GetFQName()),
      if train.train_message(msgstore_message, False, manager.classifier_data):
          print "trained as good"
***************
*** 140,144 ****
      import train
      subject = msgstore_message.subject
!     print "Training on message '%s' - " % subject,
      if train.train_message(msgstore_message, True, manager.classifier_data):
          print "trained as spam"
--- 142,148 ----
      import train
      subject = msgstore_message.subject
!     print "Training on message '%s' in '%s - " % \
!             (subject,
!              msgstore_message.GetFolder().GetFQName()),
      if train.train_message(msgstore_message, True, manager.classifier_data):
          print "trained as spam"
***************
*** 193,204 ****
          if manager.config.filter.enabled:
              import filter
              disposition = filter.filter_message(msgstore_message, manager)
!             print "Message '%s' had a Spam classification of '%s'" \
!                   % (msgstore_message.GetSubject(), disposition)
          else:
              print "Spam filtering is disabled - ignoring new message"
      except manager.message_store.NotFoundException:
          manager.LogDebug(1, "ProcessMessage had the message moved out from underneath us")
!     manager.LogDebug(2, "ProcessMessage finished for", msgstore_message)
  
  # Button/Menu and other UI event handler classes
--- 197,212 ----
          if manager.config.filter.enabled:
              import filter
+             # get the foldername before the move operation!
+             folder_name = msgstore_message.GetFolder().GetFQName()
              disposition = filter.filter_message(msgstore_message, manager)
!             print "Message '%s' in '%s' had a Spam classification of '%s'" \
!                   % (msgstore_message.GetSubject(),
!                      folder_name,
!                      disposition)
          else:
              print "Spam filtering is disabled - ignoring new message"
      except manager.message_store.NotFoundException:
          manager.LogDebug(1, "ProcessMessage had the message moved out from underneath us")
!     manager.LogDebug(2, "ProcessMessage finished for", msgstore_message.subject)
  
  # Button/Menu and other UI event handler classes
***************
*** 1297,1301 ****
              self._HookFolderEvents(config.watch_folder_ids,
                                     config.watch_include_sub,
!                                    HamFolderItemsEvent)
              )
          # For spam manually moved
--- 1305,1310 ----
              self._HookFolderEvents(config.watch_folder_ids,
                                     config.watch_include_sub,
!                                    HamFolderItemsEvent,
!                                    "filtering")
              )
          # For spam manually moved
***************
*** 1304,1308 ****
                  self._HookFolderEvents([config.spam_folder_id],
                                         False,
!                                        SpamFolderItemsEvent)
                  )
          for k in self.folder_hooks.keys():
--- 1313,1318 ----
                  self._HookFolderEvents([config.spam_folder_id],
                                         False,
!                                        SpamFolderItemsEvent,
!                                        "incremental training")
                  )
          for k in self.folder_hooks.keys():
***************
*** 1318,1322 ****
          return ret
  
!     def _HookFolderEvents(self, folder_ids, include_sub, HandlerClass):
          new_hooks = {}
          for msgstore_folder in self.manager.message_store.GetFolderGenerator(
--- 1328,1332 ----
          return ret
  
!     def _HookFolderEvents(self, folder_ids, include_sub, HandlerClass, what):
          new_hooks = {}
          for msgstore_folder in self.manager.message_store.GetFolderGenerator(
***************
*** 1363,1367 ****
                      new_hook.Init(msgstore_folder, self.application, self.manager)
                      new_hooks[msgstore_folder.id] = new_hook
!                     print "SpamBayes: Watching for new messages in folder", name
              else:
                  new_hooks[msgstore_folder.id] = existing
--- 1373,1377 ----
                      new_hook.Init(msgstore_folder, self.application, self.manager)
                      new_hooks[msgstore_folder.id] = new_hook
!                     print "SpamBayes: Watching (for %s) in '%s'" % (what, name)
              else:
                  new_hooks[msgstore_folder.id] = existing



From mhammond at users.sourceforge.net  Sat Dec 20 05:38:58 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Sat Dec 20 05:39:01 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000/sandbox dump_props.py,
	1.12, 1.13
Message-ID: <E1AXeVe-0006kL-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000/sandbox
In directory sc8-pr-cvs1:/tmp/cvs-serv25896

Modified Files:
	dump_props.py 
Log Message:
Add -c option, which writes output to the Windows clipboard.  This will
hopefully be useful when the binary is out, so people can easily paste
output into mail/sourceforge.


Index: dump_props.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/sandbox/dump_props.py,v
retrieving revision 1.12
retrieving revision 1.13
diff -C2 -d -r1.12 -r1.13
*** dump_props.py	16 Dec 2003 05:06:33 -0000	1.12
--- dump_props.py	20 Dec 2003 10:38:56 -0000	1.13
***************
*** 4,10 ****
--- 4,12 ----
  import pythoncom
  import os, sys
+ import tempfile
  
  from win32com.mapi import mapi, mapiutil
  from win32com.mapi.mapitags import *
+ import win32clipboard
  
  import mapi_driver
***************
*** 75,79 ****
      return prop_repr
  
! def DumpItemProps(item, shorten, get_large_props):
      all_props = GetAllProperties(item)
      all_props.sort() # sort by first tuple item, which is name :)
--- 77,81 ----
      return prop_repr
  
! def DumpItemProps(item, shorten, get_large_props, stream=None):
      all_props = GetAllProperties(item)
      all_props.sort() # sort by first tuple item, which is name :)
***************
*** 85,116 ****
          prop_repr = FormatPropertyValue(prop_tag, prop_val, item,
                                          shorten, get_large_props)
!         print "%-20s: %s" % (prop_name, prop_repr)
  
! def DumpProps(driver, mapi_folder, subject, include_attach, shorten, get_large):
      hr, data = mapi_folder.GetProps( (PR_DISPLAY_NAME_A,), 0)
      name = data[0][1]
      for item in driver.GetItemsWithValue(mapi_folder, PR_SUBJECT_A, subject):
!         DumpItemProps(item, shorten, get_large)
          if include_attach:
!             print
              table = item.GetAttachmentTable(0)
              rows = mapi.HrQueryAllRows(table, (PR_ATTACH_NUM,), None, None, 0)
              for row in rows:
                  attach_num = row[0][1]
!                 print "Dumping attachment (PR_ATTACH_NUM=%d)" % (attach_num,)
                  attach = item.OpenAttach(attach_num, None,
                                           mapi.MAPI_DEFERRED_ERRORS)
!                 DumpItemProps(attach, shorten, get_large)
!             print
!         print
  
  # Generic table dumper.
! def DumpTable(driver, table, name_query_ob, shorten, large_props):
      cols = table.QueryColumns(TBL_ALL_COLUMNS)
      table.SetColumns(cols, 0)
      rows = mapi.HrQueryAllRows(table, cols, None, None, 0)
!     print "Table has %d rows, each with %d columns" % (len(rows), len(cols))
      for row in rows:
!         print "-- new row --"
          for col in row:
              prop_tag, prop_val = col
--- 87,122 ----
          prop_repr = FormatPropertyValue(prop_tag, prop_val, item,
                                          shorten, get_large_props)
!         print >> stream, "%-20s: %s" % (prop_name, prop_repr)
!     print >> stream, "-- end of item properties --"
  
! def DumpProps(driver, mapi_folder, subject, include_attach, shorten,
!               get_large, stream=None):
      hr, data = mapi_folder.GetProps( (PR_DISPLAY_NAME_A,), 0)
      name = data[0][1]
      for item in driver.GetItemsWithValue(mapi_folder, PR_SUBJECT_A, subject):
!         DumpItemProps(item, shorten, get_large, stream)
          if include_attach:
!             print >> stream
              table = item.GetAttachmentTable(0)
              rows = mapi.HrQueryAllRows(table, (PR_ATTACH_NUM,), None, None, 0)
              for row in rows:
                  attach_num = row[0][1]
!                 print >> stream, \
!                       "Dumping attachment (PR_ATTACH_NUM=%d)" % (attach_num,)
                  attach = item.OpenAttach(attach_num, None,
                                           mapi.MAPI_DEFERRED_ERRORS)
!                 DumpItemProps(attach, shorten, get_large, stream)
!             print >> stream
!         print >> stream
  
  # Generic table dumper.
! def DumpTable(driver, table, name_query_ob, shorten, large_props, stream=None):
      cols = table.QueryColumns(TBL_ALL_COLUMNS)
      table.SetColumns(cols, 0)
      rows = mapi.HrQueryAllRows(table, cols, None, None, 0)
!     print >> stream, \
!           "Table has %d rows, each with %d columns" % (len(rows), len(cols))
      for row in rows:
!         print >> stream, "-- new row --"
          for col in row:
              prop_tag, prop_val = col
***************
*** 122,130 ****
              prop_repr = FormatPropertyValue(prop_tag, prop_val, name_query_ob,
                                              shorten, large_props)
!             print "%-20s: %s" % (prop_name, prop_repr)
  
  # This dumps the raw binary data of the property Outlook uses to store
  # user defined fields.
! def FindAndDumpTableUserProps(driver, table, folder, shorten, get_large_props):
      restriction = (mapi.RES_PROPERTY,
                    (mapi.RELOP_EQ,
--- 128,137 ----
              prop_repr = FormatPropertyValue(prop_tag, prop_val, name_query_ob,
                                              shorten, large_props)
!             print >> stream, "%-20s: %s" % (prop_name, prop_repr)
  
  # This dumps the raw binary data of the property Outlook uses to store
  # user defined fields.
! def FindAndDumpTableUserProps(driver, table, folder, shorten,
!                               get_large_props, stream=None):
      restriction = (mapi.RES_PROPERTY,
                    (mapi.RELOP_EQ,
***************
*** 139,143 ****
      prop_repr = FormatPropertyValue(tag, val, folder,
                                      shorten, get_large_props)
!     print "%-20s: %s" % (prop_name, prop_repr)
  
  def usage(driver, extra = None):
--- 146,150 ----
      prop_repr = FormatPropertyValue(tag, val, folder,
                                      shorten, get_large_props)
!     print >> stream, "%-20s: %s" % (prop_name, prop_repr)
  
  def usage(driver, extra = None):
***************
*** 152,155 ****
--- 159,163 ----
  matching is substring and ignore-case.
  
+ -c - Write output to the clipboard, ready for pasting into an email
  -f - Search for the message in the specified folder (default = Inbox)
  -s - Shorten long property values.
***************
*** 175,179 ****
      import getopt
      try:
!         opts, args = getopt.getopt(sys.argv[1:], "af:snl",
                                     ["dump-folder",
                                      "dump-folder-assoc-contents",
--- 183,187 ----
      import getopt
      try:
!         opts, args = getopt.getopt(sys.argv[1:], "caf:snl",
                                     ["dump-folder",
                                      "dump-folder-assoc-contents",
***************
*** 187,194 ****
--- 195,205 ----
      get_large_props = False
      include_attach = False
+     write_clipboard = False
      dump_folder = dump_folder_assoc_contents = dump_folder_user_props = False
      for opt, opt_val in opts:
          if opt == "-f":
              folder_name = opt_val
+         elif opt == "-c":
+             write_clipboard = True
          elif opt == "--dump-folder":
              dump_folder = True
***************
*** 209,212 ****
--- 220,228 ----
              usage(driver, "Unknown arg '%s'" % opt)
  
+     stream = None
+     if write_clipboard:
+         stream_name = tempfile.mktemp("spambayes")
+         stream = open(stream_name, "w")
+         
      if not folder_name:
          folder_name = "Inbox" # Assume this exists!
***************
*** 229,243 ****
      if is_table_dump:
          if dump_folder:
!             DumpItemProps(folder, shorten, get_large_props)
          if dump_folder_assoc_contents:
              table = folder.GetContentsTable(mapi.MAPI_ASSOCIATED)
!             DumpTable(driver, table, folder, shorten, get_large_props)
          if dump_folder_user_props:
              table = folder.GetContentsTable(mapi.MAPI_ASSOCIATED)
              FindAndDumpTableUserProps(driver, table, folder,
!                                       shorten, get_large_props)
      else:
          DumpProps(driver, folder, subject, include_attach,
!                   shorten, get_large_props)
  
  if __name__=='__main__':
--- 245,268 ----
      if is_table_dump:
          if dump_folder:
!             DumpItemProps(folder, shorten, get_large_props, stream)
          if dump_folder_assoc_contents:
              table = folder.GetContentsTable(mapi.MAPI_ASSOCIATED)
!             DumpTable(driver, table, folder, shorten, get_large_props, stream)
          if dump_folder_user_props:
              table = folder.GetContentsTable(mapi.MAPI_ASSOCIATED)
              FindAndDumpTableUserProps(driver, table, folder,
!                                       shorten, get_large_props, stream)
      else:
          DumpProps(driver, folder, subject, include_attach,
!                   shorten, get_large_props, stream)
!     if write_clipboard:
!         stream.close()
!         stream = open(stream_name, "r")
!         win32clipboard.OpenClipboard()
!         win32clipboard.EmptyClipboard()
!         win32clipboard.SetClipboardText(stream.read())
!         stream.close()
!         os.unlink(stream_name)
!         print "Output successfuly written to the Windows clipboard"
  
  if __name__=='__main__':



From mhammond at users.sourceforge.net  Sat Dec 20 06:17:49 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Sat Dec 20 06:17:54 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000/dialogs processors.py,
	1.13, 1.14
Message-ID: <E1AXf7F-00087P-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000/dialogs
In directory sc8-pr-cvs1:/tmp/cvs-serv31142a

Modified Files:
	processors.py 
Log Message:
If the user attempts to close the Manager dialog while there is a problem
preventing us being enabled, confirm they really want to close it
(The 'general' tab, which is where the problem is displayed may not be
focussed)


Index: processors.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/dialogs/processors.py,v
retrieving revision 1.13
retrieving revision 1.14
diff -C2 -d -r1.13 -r1.14
*** processors.py	16 Dec 2003 05:06:33 -0000	1.13
--- processors.py	20 Dec 2003 11:17:47 -0000	1.14
***************
*** 78,81 ****
--- 78,89 ----
  class CloseButtonProcessor(ButtonProcessor):
      def OnClicked(self, id):
+         problem = self.window.manager.GetDisabledReason()
+         if problem:
+             q = "There appears to be a problem with SpamBayes' configuration" \
+                 "\r\nIf you do not fix this problem, SpamBayes will be" \
+                 " disabled.\r\n\r\n" + problem + \
+                 "\r\n\r\nDo you wish to re-configure?"
+             if self.window.manager.AskQuestion(q):
+                 return
          win32gui.EndDialog(self.window.hwnd, id)
      def GetPopupHelpText(self, ctrlid):



From mhammond at users.sourceforge.net  Sat Dec 20 06:21:11 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Sat Dec 20 06:21:14 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 manager.py,1.94,1.95
Message-ID: <E1AXfAV-0008Fx-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv31490

Modified Files:
	manager.py 
Log Message:
As part of checking our configuration is invalid, make sure the user
hasn't set us up such that either Spam/Unsure folders isn't also being
watched for new messages, as this would confuse us greatly, and may
even explain a few of the bizarre training related bugs.


Index: manager.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/manager.py,v
retrieving revision 1.94
retrieving revision 1.95
diff -C2 -d -r1.94 -r1.95
*** manager.py	16 Dec 2003 05:06:33 -0000	1.94
--- manager.py	20 Dec 2003 11:21:09 -0000	1.95
***************
*** 827,830 ****
--- 827,857 ----
                     "Select the 'Filtering' tab to define this folders."
  
+         # Check that the user hasn't selected the same folder as both
+         # 'Spam' or 'Unsure', and 'Watch' - this would confuse us greatly.
+         ms = self.message_store
+         unsure_folder = None # unsure need not be specified.
+         if config.unsure_folder_id:
+             try:
+                 unsure_folder = ms.GetFolder(config.unsure_folder_id)
+             except ms.MsgStoreException, details:
+                 return "The unsure folder is invalid: %s" % (details,)
+         try:
+             spam_folder = ms.GetFolder(config.spam_folder_id)
+         except ms.MsgStoreException, details:
+             return "The spam folder is invalid: %s" % (details,)
+         if ok_to_enable:
+             for folder in ms.GetFolderGenerator(config.watch_folder_ids,
+                                                 config.watch_include_sub):
+                 bad_folder_type = None
+                 if unsure_folder is not None and unsure_folder == folder:
+                     bad_folder_type = "unsure"
+                     bad_folder_name = unsure_folder.GetFQName()
+                 if spam_folder == folder:
+                     bad_folder_type = "spam"
+                     bad_folder_name = spam_folder.GetFQName()
+                 if bad_folder_type is not None:
+                     return "You can not specify folder '%s' as both the " \
+                            "%s folder, and as being watched." \
+                            % (bad_folder_name, bad_folder_type)
          return None
  



From mhammond at users.sourceforge.net  Sat Dec 20 18:10:52 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Sat Dec 20 18:10:59 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 addin.py,1.122,1.123
Message-ID: <E1AXqFI-0001zo-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv7496

Modified Files:
	addin.py 
Log Message:
When doing a "batch train" (eg, selecting multiple messages and saying
"Delete as" or "Recover from",) the DB was saved in between each and every
message.  Now only saved at the end (which was always the intent)


Index: addin.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/addin.py,v
retrieving revision 1.122
retrieving revision 1.123
diff -C2 -d -r1.122 -r1.123
*** addin.py	20 Dec 2003 06:25:12 -0000	1.122
--- addin.py	20 Dec 2003 23:10:50 -0000	1.123
***************
*** 78,81 ****
--- 78,94 ----
  # niceness doesn't help anyone.
  
+ # As MarkH assumed, and later found to back him up in:
+ # http://www.slipstick.com/dev/comaddins.htm:
+ # On building add-ins for multiple Outlook versions, Randy Byrne writes in
+ # the microsoft.public.office.developer.com.add_ins newsgroup, "The best
+ # practice is to compile your Add-in with OL2000 and MSO9.dll. Then your
+ # add-in will work with both OL2000 and OL2002, and CommandBar events will
+ # work with both versions. If you need to use any specific OL2002 or
+ # Office 10.0 Object Library calls, you can use late binding to address
+ # those issues. The CommandBar Events are the same in both Office
+ # 2000 and Office XP."
+ # So that is what we do: specify the minimum versions of the typelibs we
+ # can work with - ie, Outlook 2000.
+ 
  # win32com generally checks the gencache is up to date (typelib hasn't
  # changed, makepy hasn't changed, etc), but when frozen we dont want to
***************
*** 121,125 ****
  
  # Helper functions
! def TrainAsHam(msgstore_message, manager, rescore = True):
      import train
      subject = msgstore_message.subject
--- 134,138 ----
  
  # Helper functions
! def TrainAsHam(msgstore_message, manager, rescore = True, save_db = True):
      import train
      subject = msgstore_message.subject
***************
*** 137,143 ****
          print "already was trained as good"
      assert train.been_trained_as_ham(msgstore_message, manager.classifier_data)
!     manager.SaveBayesPostIncrementalTrain()
  
! def TrainAsSpam(msgstore_message, manager, rescore = True):
      import train
      subject = msgstore_message.subject
--- 150,157 ----
          print "already was trained as good"
      assert train.been_trained_as_ham(msgstore_message, manager.classifier_data)
!     if save_db:
!         manager.SaveBayesPostIncrementalTrain()
  
! def TrainAsSpam(msgstore_message, manager, rescore = True, save_db = True):
      import train
      subject = msgstore_message.subject
***************
*** 155,159 ****
      assert train.been_trained_as_spam(msgstore_message, manager.classifier_data)
      # And if the DB can save itself incrementally, do it now
!     manager.SaveBayesPostIncrementalTrain()
  
  # Function to filter a message - note it is a msgstore msg, not an
--- 169,174 ----
      assert train.been_trained_as_spam(msgstore_message, manager.classifier_data)
      # And if the DB can save itself incrementally, do it now
!     if save_db:
!         manager.SaveBayesPostIncrementalTrain()
  
  # Function to filter a message - note it is a msgstore msg, not an
***************
*** 620,624 ****
              subject = msgstore_message.GetSubject()
              print "Moving and spam training message '%s' - " % (subject,),
!             TrainAsSpam(msgstore_message, self.manager)
              # Do the new message state if necessary.
              try:
--- 635,639 ----
              subject = msgstore_message.GetSubject()
              print "Moving and spam training message '%s' - " % (subject,),
!             TrainAsSpam(msgstore_message, self.manager, save_db = False)
              # Do the new message state if necessary.
              try:
***************
*** 683,687 ****
                  # Must train before moving, else we lose the message!
                  print "Recovering to folder '%s' and ham training message '%s' - " % (restore_folder.name, subject),
!                 TrainAsHam(msgstore_message, self.manager)
                  # Do the new message state if necessary.
                  try:
--- 698,702 ----
                  # Must train before moving, else we lose the message!
                  print "Recovering to folder '%s' and ham training message '%s' - " % (restore_folder.name, subject),
!                 TrainAsHam(msgstore_message, self.manager, save_db = False)
                  # Do the new message state if necessary.
                  try:



From anadelonbrin at users.sourceforge.net  Sun Dec 21 21:28:28 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Sun Dec 21 21:28:33 2003
Subject: [Spambayes-checkins] spambayes/spambayes oe_mailbox.py,1.3,1.4
Message-ID: <E1AYFo4-00019N-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv4421/spambayes

Modified Files:
	oe_mailbox.py 
Log Message:
Move OE specific stuff out from UserInterface.py to oe_mailbox.py

Index: oe_mailbox.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/oe_mailbox.py,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** oe_mailbox.py	16 Dec 2003 05:06:34 -0000	1.3
--- oe_mailbox.py	22 Dec 2003 02:28:26 -0000	1.4
***************
*** 1,2 ****
--- 1,4 ----
+ from __future__ import generators
+ 
  # This module is part of the spambayes project, which is Copyright 2002-3
  # The Python Software Foundation and is covered by the Python Software
***************
*** 7,14 ****
--- 9,25 ----
  
  __author__ = "Romain Guy"
+ __credits__ = "All the SpamBayes folk"
  
  import binascii
  import os
  import struct
+ import msgs
+ import StringIO
+ import sys
+ 
+ if sys.platform == "win32":
+     import win32api
+     import win32con
+     from win32com.shell import shell, shellcon
  
  ###########################################################################
***************
*** 416,419 ****
--- 427,579 ----
          return self.dbxText
  
+ # This started its SpamBayes life as a private method of the UserInterface
+ # class, but is really a general purpose (Outlook Express) function.
+ def convertToMbox(content):
+     """Check if the given buffer is in a non-mbox format, and convert it
+     into mbox format if so.  If it's already an mbox, return it unchanged.
+     """
+ 
+     dbxStream = StringIO.StringIO(content)
+     header = dbxFileHeader(dbxStream)
+ 
+     if header.isValid() and header.isMessages():
+         file_info_len = dbxFileHeader.FH_FILE_INFO_LENGTH
+         fh_entries = dbxFileHeader.FH_ENTRIES
+         fh_ptr = dbxFileHeader.FH_TREE_ROOT_NODE_PTR
+         
+         info = dbxFileInfo(dbxStream, header.getEntry(file_info_len))
+         entries = header.getEntry(fh_entries)
+         address = header.getEntry(fh_ptr)
+         
+         if address and entries:
+             tree = dbxTree(dbxStream, address, entries)
+             dbxBuffer = ""
+ 
+             for i in range(entries):
+                 address = tree.getValue(i)
+                 messageInfo = dbxMessageInfo(dbxStream, address)
+ 
+                 if messageInfo.isIndexed(dbxMessageInfo.MI_MESSAGE_ADDRESS):
+                     address = dbxMessageInfo.MI_MESSAGE_ADDRESS
+                     messageAddress = messageInfo.getValueAsLong(address)
+                     message = dbxMessage(dbxStream, messageAddress)
+ 
+                     # This fakes up a from header to conform to mbox
+                     # standards.  It would be better to extract this
+                     # data from the message itself, as this will
+                     # result in incorrect tokens.
+                     dbxBuffer += "From spambayes@spambayes.org %s\n%s" \
+                                  % (strftime("%a %b %d %H:%M:%S MET %Y",
+                                              gmtime()), message.getText())
+             content = dbxBuffer
+     dbxStream.close()
+     return content
+ 
+ def OEStoreRoot():
+     """Return the path to the Outlook Express Store Root.
+ 
+     Tested with Outlook Express 5.0 with Windows XP."""
+     if sys.platform != "win32":
+         # AFAIK, there is only a Win32 OE, and a Mac OE.
+         # The Mac OE should be easy enough, but I don't know
+         # where the dbx files are stored (I presume they are in the
+         # same format).
+         raise NotImplementedError
+     
+     reg = win32api.RegOpenKeyEx(win32con.HKEY_USERS, "")
+     user_index = 0
+     while True:
+         # Loop through all the users
+         try:
+             user_name = "%s\\Identities" % \
+                         (win32api.RegEnumKey(reg, user_index),)
+         except win32api.error:
+             break
+         user_index += 1
+         try:
+             user_key = win32api.RegOpenKeyEx(win32con.HKEY_USERS, user_name)
+         except win32api.error:
+             # Not this one
+             continue
+ 
+         identity_index = 0
+         while True:
+             # Loop through all the identities
+             try:
+                 identity_name = win32api.RegEnumKey(user_key,
+                                                     identity_index)
+             except win32api.error:
+                 break
+             identity_index += 1
+             
+             subkey_name = "%s\\%s\\%s" % (user_name, identity_name,
+                                           "Software\\Microsoft\\Outlook " \
+                                           "Express\\5.0")
+             try:
+                 subkey = win32api.RegOpenKeyEx(win32con.HKEY_USERS,
+                                                subkey_name, 0,
+                                                win32con.KEY_READ)
+             except win32api.error:
+                 # Not this user
+                 continue
+ 
+             try:
+                 raw = win32api.RegQueryValueEx(subkey, "Store Root")
+             except win32api.error:
+                 break
+             UserDirectory = shell.SHGetFolderPath \
+                             (0, shellcon.CSIDL_LOCAL_APPDATA, 0, 0)
+             raw = raw[0].replace("%UserProfile%\\Local Settings\\" \
+                                  "Application Data", UserDirectory)
+             return raw
+ 
+ ## For use by the test tools.
+ class OEMsg(msgs.Msg):
+     def __init__(self, guts, id):
+         self.tag = id
+         self.guts = guts
+ 
+ # The iterator yields a stream of Msg objects, taken from a list of
+ # dbx files.
+ class OEMsgStream(msgs.MsgStream):
+     def __init__(self, tag, dbxes, keep=None):
+         msgs.MsgStream.__init__(self, tag, dbxes, keep)
+ 
+     def produce(self):
+         if self.keep is None:
+             for dbx in self.directories:
+                 folder = convertToMbox(file(dbx))
+                 all = folder.split("\nFrom ") # XXX Is this right?
+                 count = 0
+                 for msg in all:
+                     id = "%s::%s" % (dbx, count)
+                     count += 1
+                     yield OEMsg(msg, id)
+             return
+         # We only want part of the msgs.  Shuffle each directory list, but
+         # in such a way that we'll get the same result each time this is
+         # called on the same directory list.
+         for directory in self.directories:
+             folder = convertToMbox(file(dbx))
+             all = folder.split("\nFrom ") # XXX Is this right?
+             random.seed(hash(max(all)) ^ SEED) # reproducible across calls
+             random.shuffle(all)
+             del all[self.keep:]
+             all.sort()  # for consistency with MsgStream
+             count = 0
+             for msg in all:
+                 id = "%s::%s" % (dbx, count)
+                 count += 1
+                 yield OEMsg(msg, id)
+ 
+ class OEHamStream(msgs.HamStream):
+     def __init__(self, tag, dbxes, train=0):
+         msgs.HamStream.__init__(self, tag, dbxes, train)
+ 
+ class OESpamStream(msgs.SpamStream):
+     def __init__(self, tag, dbxes, train=0):
+         msgs.SpamStream.__init__(self, tag, dbxes, train)
+ 
+ 
  ###########################################################################
  ## TEST DRIVER
***************
*** 442,446 ****
          sys.exit()
  
!     MAILBOX_DIR = args[0]
  
      files = [os.path.join(MAILBOX_DIR, file) for file in \
--- 602,606 ----
          sys.exit()
  
!     MAILBOX_DIR = args[0]  
  
      files = [os.path.join(MAILBOX_DIR, file) for file in \
***************
*** 468,485 ****
                      tree = dbxTree(dbx, address, entries)
  
!                     for i in range(entries):
!                         address = tree.getValue(i)
!                         messageInfo = dbxMessageInfo(dbx, address)
  
!                         if messageInfo.isIndexed(dbxMessageInfo.MI_MESSAGE_ADDRESS):
!                             messageAddress = messageInfo.getValueAsLong(dbxMessageInfo.MI_MESSAGE_ADDRESS)
!                             message        = dbxMessage(dbx, messageAddress)
  
!                             if print_message:
!                                 print
!                                 print "Message :", messageInfo.getString(dbxMessageInfo.MI_SUBJECT)
!                                 print "=" * (len(messageInfo.getString(dbxMessageInfo.MI_SUBJECT)) + 9)
!                                 print
!                                 print message.getText()
  
          except Exception, (strerror):
--- 628,645 ----
                      tree = dbxTree(dbx, address, entries)
  
!                 for i in range(entries):
!                     address = tree.getValue(i)
!                     messageInfo = dbxMessageInfo(dbx, address)
  
!                     if messageInfo.isIndexed(dbxMessageInfo.MI_MESSAGE_ADDRESS):
!                         messageAddress = messageInfo.getValueAsLong(dbxMessageInfo.MI_MESSAGE_ADDRESS)
!                         message        = dbxMessage(dbx, messageAddress)
  
!                         if print_message:
!                             print
!                             print "Message :", messageInfo.getString(dbxMessageInfo.MI_SUBJECT)
!                             print "=" * (len(messageInfo.getString(dbxMessageInfo.MI_SUBJECT)) + 9)
!                             print
!                             print message.getText()
  
          except Exception, (strerror):



From anadelonbrin at users.sourceforge.net  Sun Dec 21 21:31:42 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Sun Dec 21 21:31:44 2003
Subject: [Spambayes-checkins] spambayes/spambayes UserInterface.py, 1.38,
	1.39
Message-ID: <E1AYFrC-0001J0-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv5012/spambayes

Modified Files:
	UserInterface.py 
Log Message:
Move OE specific stuff out from UserInterface.py to oe_mailbox.py.

Make PyChecker more happy.

Index: UserInterface.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/UserInterface.py,v
retrieving revision 1.38
retrieving revision 1.39
diff -C2 -d -r1.38 -r1.39
*** UserInterface.py	16 Dec 2003 05:06:34 -0000	1.38
--- UserInterface.py	22 Dec 2003 02:31:39 -0000	1.39
***************
*** 519,562 ****
          DBX one is very quick (just a matter of checking the first few
          bytes), and should not alter the overall performance."""
! 
!         dbxStream = StringIO.StringIO(content)
!         header = oe_mailbox.dbxFileHeader(dbxStream)
! 
!         if header.isValid() and header.isMessages():
!             file_info_len = oe_mailbox.dbxFileHeader.FH_FILE_INFO_LENGTH
!             fh_entries = oe_mailbox.dbxFileHeader.FH_ENTRIES
!             fh_ptr = oe_mailbox.dbxFileHeader.FH_TREE_ROOT_NODE_PTR
! 
!             info = oe_mailbox.dbxFileInfo(dbxStream,
!                                           header.getEntry(file_info_len))
!             entries = header.getEntry(fh_entries)
!             address = header.getEntry(fh_ptr)
! 
!             if address and entries:
!                 tree = oe_mailbox.dbxTree(dbxStream, address, entries)
!                 dbxBuffer = ""
! 
!                 for i in range(entries):
!                     address = tree.getValue(i)
!                     messageInfo = oe_mailbox.dbxMessageInfo(dbxStream,
!                                                             address)
! 
!                     if messageInfo.isIndexed(\
!                         oe_mailbox.dbxMessageInfo.MI_MESSAGE_ADDRESS):
!                         address = oe_mailbox.dbxMessageInfo.MI_MESSAGE_ADDRESS
!                         messageAddress = \
!                                        messageInfo.getValueAsLong(address)
!                         message = oe_mailbox.dbxMessage(dbxStream,
!                                                         messageAddress)
! 
!                         # This fakes up a from header to conform to mbox
!                         # standards.  It would be better to extract this
!                         # data from the message itself, as this will
!                         # result in incorrect tokens.
!                         dbxBuffer += "From spambayes@spambayes.org %s\n%s" \
!                                      % (strftime("%a %b %d %H:%M:%S MET %Y",
!                                                  gmtime()), message.getText())
!                 content = dbxBuffer
!         dbxStream.close()
          return content
  
--- 519,523 ----
          DBX one is very quick (just a matter of checking the first few
          bytes), and should not alter the overall performance."""
!         content = oe_mailbox.convertToMbox(content)
          return content
  
***************
*** 651,654 ****
--- 612,618 ----
          html.mainContent.configFormContent = ""
          html.mainContent.optionsPathname = optionsPathname
+         return self._buildConfigPageBody(html, parm_map)
+ 
+     def _buildConfigPageBody(self, html, parm_map):
          configTable = None
          section = None
***************
*** 716,723 ****
                      # Tim thinks that Yes/No makes more sense than True/False
                      if options.is_boolean(sect, opt):
!                         if val == False:
!                             val = "No"
!                         elif val == True:
                              val = "Yes"
                      newOption.val_label = str(val)
                      newOption.input_box.value = str(val)
--- 680,687 ----
                      # Tim thinks that Yes/No makes more sense than True/False
                      if options.is_boolean(sect, opt):
!                         if val is True:
                              val = "Yes"
+                         elif val is False:
+                             val = "No"
                      newOption.val_label = str(val)
                      newOption.input_box.value = str(val)



From mhammond at users.sourceforge.net  Sun Dec 21 22:13:35 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Sun Dec 21 22:13:38 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 config.py,1.28,1.29
Message-ID: <E1AYGVj-0002xm-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv11382

Modified Files:
	config.py 
Log Message:
Default to background filtering being on for new versions.


Index: config.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/config.py,v
retrieving revision 1.28
retrieving revision 1.29
diff -C2 -d -r1.28 -r1.29
*** config.py	16 Dec 2003 05:06:33 -0000	1.28
--- config.py	22 Dec 2003 03:13:33 -0000	1.29
***************
*** 218,222 ****
          BOOLEAN, RESTORE),
      # Options that allow the filtering to be done by a timer.
!     ("timer_enabled", "Should items be filtered by a timer?", False,
          """Depending on a number of factors, SpamBayes may occasionally miss
          messages or conflict with builtin Outlook rules.  If this option
--- 218,222 ----
          BOOLEAN, RESTORE),
      # Options that allow the filtering to be done by a timer.
!     ("timer_enabled", "Should items be filtered by a timer?", True,
          """Depending on a number of factors, SpamBayes may occasionally miss
          messages or conflict with builtin Outlook rules.  If this option



From mhammond at users.sourceforge.net  Sun Dec 21 22:13:58 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Sun Dec 21 22:14:01 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000/docs
	troubleshooting.html, 1.19, 1.20
Message-ID: <E1AYGW6-0002zb-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000/docs
In directory sc8-pr-cvs1:/tmp/cvs-serv11422/docs

Modified Files:
	troubleshooting.html 
Log Message:
Fix the link to the Frequently Reported Bugs


Index: troubleshooting.html
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/docs/troubleshooting.html,v
retrieving revision 1.19
retrieving revision 1.20
diff -C2 -d -r1.19 -r1.20
*** troubleshooting.html	12 Dec 2003 06:29:56 -0000	1.19
--- troubleshooting.html	22 Dec 2003 03:13:56 -0000	1.20
***************
*** 36,40 ****
  release.</li>
    <li>The online <a
!  href="http://www.sourceforge.net/apps/outlook/bugs.html">Frequently
  Reported Outlook Bugs</a> list has the most common problems people are
  experiencing, and links to more information about the bug.</li>
--- 36,40 ----
  release.</li>
    <li>The online <a
!  href="http://spambayes.sourceforge.net/apps/outlook/bugs.html">Frequently
  Reported Outlook Bugs</a> list has the most common problems people are
  experiencing, and links to more information about the bug.</li>



From mhammond at users.sourceforge.net  Sun Dec 21 22:16:31 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Sun Dec 21 22:16:34 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 addin.py,1.123,1.124
Message-ID: <E1AYGYZ-00035v-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv11813

Modified Files:
	addin.py 
Log Message:
When a frozen executable, we come a mini-installer EXE for the DLL.
See the comments for more information.


Index: addin.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/addin.py,v
retrieving revision 1.123
retrieving revision 1.124
diff -C2 -d -r1.123 -r1.124
*** addin.py	20 Dec 2003 23:10:50 -0000	1.123
--- addin.py	22 Dec 2003 03:16:29 -0000	1.124
***************
*** 1520,1523 ****
--- 1520,1539 ----
  
  if __name__ == '__main__':
+     # woohoo - here is a wicked hack.  If we are a frozen .EXE, then we are
+     # a mini "registration" utility.  However, we still want to register the
+     # DLL, *not* us.  Pretend we are frozen in that DLL.
+     # NOTE: This is only needed due to problems with Inno Setup unregistering
+     # our DLL the 'normal' way, but then being unable to remove the files as
+     # they are in use (presumably by Inno doing the unregister!).  If this
+     # problem ever goes away, so will the need for this to be frozen as
+     # an executable.  In all cases other than as above, 'regsvr32 dll_name'
+     # is still the preferred way of registering our binary.
+     if hasattr(sys, "frozen"):
+         sys.frozendllhandle = win32api.LoadLibrary("outlook_addin.dll")
+         sys.frozen = "dll"
+         # Without this, com registration will look at class.__module__, and
+         # get all confused about the module name holding our class in the DLL
+         OutlookAddin._reg_class_spec_ = "addin.OutlookAddin"
+         # And continue doing the registration with our hacked environment.
      import win32com.server.register
      win32com.server.register.UseCommandLine(OutlookAddin)



From mhammond at users.sourceforge.net  Sun Dec 21 22:17:33 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Sun Dec 21 22:17:35 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 about.html,1.26,1.27
Message-ID: <E1AYGZZ-00038o-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv12051

Modified Files:
	about.html 
Log Message:
Reflect that background filtering now defaults to enabled.


Index: about.html
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/about.html,v
retrieving revision 1.26
retrieving revision 1.27
diff -C2 -d -r1.26 -r1.27
*** about.html	8 Dec 2003 01:00:49 -0000	1.26
--- about.html	22 Dec 2003 03:17:31 -0000	1.27
***************
*** 40,49 ****
  <p><span style="font-weight: bold;">Filtering in the background:</span>
  When Outlook is under load, SpamBayes may occasionally miss some
! messages (as Outlook doesn't tell us they appeared).&nbsp; Also, if you
! use builtin Outlook rules, you may find that occasionally they also
! fail to filter messages (as both SpamBayes and the builtin rules
! "fight" over processing the message).&nbsp; Both of these
! problems can generally be solved by enabling background filtering of
! messages, via the SpamBayes Manager's <span style="font-style: italic;">Advanced</span>
  tab. </p>
  <p><span style="font-weight: bold;">Toolbars stop working:</span> Many
--- 40,51 ----
  <p><span style="font-weight: bold;">Filtering in the background:</span>
  When Outlook is under load, SpamBayes may occasionally miss some
! messages (as Outlook doesn't tell us they appeared), and if you use
! builtin Outlook rules, you may find that occasionally they also
! fail to filter messages (as both SpamBayes and Outlook
! "fight" over processing the message).&nbsp; To overcome these problems,
! SpamBayes defaults to filtering new messages in the background.&nbsp;
! The enabling or disabling of this feature, and the delay before
! processing new mails can be configured via the SpamBayes Manager's <span
!  style="font-style: italic;">Advanced</span>
  tab. </p>
  <p><span style="font-weight: bold;">Toolbars stop working:</span> Many
***************
*** 56,60 ****
  it, then selecting Customize.&nbsp; Our <a
   href="docs/troubleshooting.html">troubleshooting guide</a> has alot
! more information on toolbars.<br>
  </p>
  <h3><a name="Field"></a>Viewing and Using the Spam Score Field</h3>
--- 58,67 ----
  it, then selecting Customize.&nbsp; Our <a
   href="docs/troubleshooting.html">troubleshooting guide</a> has alot
! more information on toolbars.</p>
! <p><span style="font-weight: bold;">Late Breaking Problems:</span> For
! problems reported since this version of SpamBayes was released, see the
! online <a
!  href="http://spambayes.sourceforge.net/apps/outlook/bugs.html">frequently
! reported bugs list</a>.<br>
  </p>
  <h3><a name="Field"></a>Viewing and Using the Spam Score Field</h3>



From anadelonbrin at users.sourceforge.net  Sun Dec 21 23:19:58 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Sun Dec 21 23:20:04 2003
Subject: [Spambayes-checkins] spambayes/spambayes Options.py,1.95,1.96
Message-ID: <E1AYHXy-0005hC-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv21890/spambayes

Modified Files:
	Options.py 
Log Message:
Mark options as experimental/deprecated in options docstrings.



Index: Options.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/Options.py,v
retrieving revision 1.95
retrieving revision 1.96
diff -C2 -d -r1.95 -r1.96
*** Options.py	18 Dec 2003 06:41:52 -0000	1.95
--- Options.py	22 Dec 2003 04:19:56 -0000	1.96
***************
*** 156,172 ****
  
      ("x-search_for_habeas_headers", "Search for Habeas Headers", False,
!      """If true, search for the habeas headers (see http://www.habeas.com)
!      If they are present and correct, this should be a strong ham sign, if
!      they are present and incorrect, this should be a strong spam sign.""",
       BOOLEAN, RESTORE),
  
      ("x-reduce_habeas_headers", "Reduce Habeas Header Tokens to Single", False,
!      """If SpamBayes is set to search for the Habeas headers, nine tokens
!      are generated for messages with habeas headers.  This should be fine,
!      since messages with the headers should either be ham, or result in FN
!      so that we can send them to habeas so they can be sued.  However, to
!      reduce the strength of habeas headers, we offer the ability to reduce
!      the nine tokens to one. (This option has no effect if
!      search_for_habeas_headers is False)""",
       BOOLEAN, RESTORE),
    ),
--- 156,173 ----
  
      ("x-search_for_habeas_headers", "Search for Habeas Headers", False,
!      """(EXPERIMENTAL) If true, search for the habeas headers (see
!      http://www.habeas.com). If they are present and correct, this should
!      be a strong ham sign, if they are present and incorrect, this should
!      be a strong spam sign.""",
       BOOLEAN, RESTORE),
  
      ("x-reduce_habeas_headers", "Reduce Habeas Header Tokens to Single", False,
!      """(EXPERIMENTAL) If SpamBayes is set to search for the Habeas
!      headers, nine tokens are generated for messages with habeas headers.
!      This should be fine, since messages with the headers should either be
!      ham, or result in FN so that we can send them to habeas so they can
!      be sued.  However, to reduce the strength of habeas headers, we offer
!      the ability to reduce the nine tokens to one. (This option has no
!      effect if search_for_habeas_headers is False)""",
       BOOLEAN, RESTORE),
    ),
***************
*** 177,184 ****
    "URLRetriever" : (
      ("x-slurp_urls", "Tokenize text content at the end of URLs", False,
!      """If this option is enabled, when a message normally scores in the
!      'unsure' range, and has fewer tokens than the maximum looked at,
!      and contains URLs, then the text at those URLs is obtained and
!      tokenized.  If those tokens result in the message moving to a
       score outside the 'unsure' range, then they are added to the
       tokens for the message.  This should be particularly effective
--- 178,185 ----
    "URLRetriever" : (
      ("x-slurp_urls", "Tokenize text content at the end of URLs", False,
!      """(EXPERIMENTAL) If this option is enabled, when a message normally
!      scores in the 'unsure' range, and has fewer tokens than the maximum
!      looked at, and contains URLs, then the text at those URLs is obtained
!      and tokenized.  If those tokens result in the message moving to a
       score outside the 'unsure' range, then they are added to the
       tokens for the message.  This should be particularly effective
***************
*** 187,207 ****
      
      ("x-cache_expiry_days", "Number of days to store URLs in cache", 7,
!      """This is the number of days that local cached copies of the text
!      at the URLs will be stored for.""",
       INTEGER, RESTORE),
  
      ("x-cache_directory", "URL Cache Directory", "url-cache",
!      """So that SpamBayes doesn't need to retrieve the same URL
!      over and over again, it stores local copies of the text
!      at the end of the URL.  This is the directory that will be
!      used for those copies.""",
       PATH, RESTORE),
  
      ("x-only_slurp_base", "Retrieve base url", False,
!      """To try and speed things up, and to avoid following unique URLS, if
!      this option is enabled, SpamBayes will convert the URL to as basic a
!      form it we can.  All directory information is removed and the domain
!      is reduced to the two (or three for those with a country TLD) top-most
!      elements.  For example,
           http://www.massey.ac.nz/~tameyer/index.html?you=me
       would become
--- 188,208 ----
      
      ("x-cache_expiry_days", "Number of days to store URLs in cache", 7,
!      """(EXPERIMENTAL) This is the number of days that local cached copies
!      of the text at the URLs will be stored for.""",
       INTEGER, RESTORE),
  
      ("x-cache_directory", "URL Cache Directory", "url-cache",
!      """(EXPERIMENTAL) So that SpamBayes doesn't need to retrieve the same
!      URL over and over again, it stores local copies of the text at the
!      end of the URL.  This is the directory that will be used for those
!      copies.""",
       PATH, RESTORE),
  
      ("x-only_slurp_base", "Retrieve base url", False,
!      """(EXPERIMENTAL) To try and speed things up, and to avoid following
!      unique URLS, if this option is enabled, SpamBayes will convert the URL
!      to as basic a form it we can.  All directory information is removed
!      and the domain is reduced to the two (or three for those with a
!      country TLD) top-most elements.  For example,
           http://www.massey.ac.nz/~tameyer/index.html?you=me
       would become
***************
*** 224,231 ****
  
      ("x-web_prefix", "Prefix for tokens from web pages", "",
!      """It may be that what is hammy/spammy for you in email isn't from
!      webpages.  You can then set this option (to "web:", for example),
!      and effectively create an independent (sub)database for tokens
!      derived from parsing web pages.""",
       r"[\S]+", RESTORE),
    ),
--- 225,232 ----
  
      ("x-web_prefix", "Prefix for tokens from web pages", "",
!      """(EXPERIMENTAL) It may be that what is hammy/spammy for you in email
!      isn't from webpages.  You can then set this option (to "web:", for
!      example), and effectively create an independent (sub)database for
!      tokens derived from parsing web pages.""",
       r"[\S]+", RESTORE),
    ),
***************
*** 471,484 ****
      # LATER:  this option sucked, creating more problems than it solved.
      # It's deprecated, and the support code has gone away.
-     # XXX The "x-" prefix can't be "X-" instead, else it's considered
-     # XXX an invalid option instead of a deprecated one.  That behavior
-     # XXX doesn't match the OptionsClass comments.
  
      ("x-experimental_ham_spam_imbalance_adjustment", "Compensate for unequal numbers of spam and ham", False,
!      """If your training database has significantly more ham than
!      spam, or vice versa, you may start seeing an increase in incorrect
!      classifications (messages put in the wrong category, not just marked
!      as unsure). If so, this option allows you to compensate for this, at
!      the cost of increasing the number of messages classified as "unsure".
  
       Note that the effect is subtle, and you should experiment with both
--- 472,483 ----
      # LATER:  this option sucked, creating more problems than it solved.
      # It's deprecated, and the support code has gone away.
  
      ("x-experimental_ham_spam_imbalance_adjustment", "Compensate for unequal numbers of spam and ham", False,
!      """(DEPRECATED) If your training database has significantly more ham
!      than spam, or vice versa, you may start seeing an increase in
!      incorrect classifications (messages put in the wrong category, not
!      just marked as unsure). If so, this option allows you to compensate
!      for this, at the cost of increasing the number of messages classified
!      as "unsure".
  
       Note that the effect is subtle, and you should experiment with both
***************
*** 487,495 ****
       BOOLEAN, RESTORE),
  
!     ("x-use_bigrams", "(EXPERIMENTAL) Use mixed uni/bi-grams scheme", False,
!      """Generate both unigrams (words) and bigrams (pairs of words).
!      However, extending an idea originally from Gary Robinson, the message
!      is 'tiled' into  non-overlapping unigrams and bigrams, approximating
!      the strongest outcome over all possible tilings.
  
       Note that to really test this option you need to retrain with it on,
--- 486,494 ----
       BOOLEAN, RESTORE),
  
!     ("x-use_bigrams", "Use mixed uni/bi-grams scheme", False,
!      """(EXPERIMENTAL) Generate both unigrams (words) and bigrams (pairs of
!      words). However, extending an idea originally from Gary Robinson, the
!      message is 'tiled' into non-overlapping unigrams and bigrams,
!      approximating the strongest outcome over all possible tilings.
  
       Note that to really test this option you need to retrain with it on,



From anadelonbrin at users.sourceforge.net  Sun Dec 21 23:38:13 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Sun Dec 21 23:38:16 2003
Subject: [Spambayes-checkins] spambayes/spambayes/resources ui.html, 1.28,
	1.29 ui_html.py, 1.27, 1.28
Message-ID: <E1AYHpd-0006KQ-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes/resources
In directory sc8-pr-cvs1:/tmp/cvs-serv24316/spambayes/resources

Modified Files:
	ui.html ui_html.py 
Log Message:
A beginning at a more userfriendly interface to the testing setup.
At the moment to use this, replace the ProxyUserInterface class in sb_server
with the TestTools one (all the old stuff is still available) and go to
http://localhost:8880/cv - it should be straightforward from there.
(I'll post a patch to -dev as well).

Note that the option to use Outlook Express as the source for testing
data doesn't quite work yet.  The 'standard' and 'cache' ones do, from
the testing I've done so far.


Index: ui.html
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/resources/ui.html,v
retrieving revision 1.28
retrieving revision 1.29
diff -C2 -d -r1.28 -r1.29
*** ui.html	5 Nov 2003 13:05:15 -0000	1.28
--- ui.html	22 Dec 2003 04:38:11 -0000	1.29
***************
*** 504,508 ****
  
      <span id="configForm">
!     <form action="changeopts" method="POST">
  
      <p><span id="introduction">This page allows you to change the options that control how
--- 504,508 ----
  
      <span id="configForm">
!     <form id="action_page" action="changeopts" method="POST">
  
      <p><span id="introduction">This page allows you to change the options that control how
***************
*** 567,571 ****
      </form>
  
!     <form action="restoredefaults" method="POST">
      <center>
        <input id="restore" type="submit" name="how" value="Restore Defaults"/>
--- 567,571 ----
      </form>
  
!     <form id="restore_form" action="restoredefaults" method="POST">
      <center>
        <input id="restore" type="submit" name="how" value="Restore Defaults"/>

Index: ui_html.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/resources/ui_html.py,v
retrieving revision 1.27
retrieving revision 1.28
diff -C2 -d -r1.27 -r1.28
*** ui_html.py	3 Oct 2003 18:52:08 -0000	1.27
--- ui_html.py	22 Dec 2003 04:38:11 -0000	1.28
***************
*** 21,121 ****
  �'���� �\036\016t-���\\\024笌����Q��\000�\024G�s\010b\025��RgOS�i�V|�X�Mޠ�G�@�a\
  ZΪ\011�&�e,�<N��_�\"�\023�\005�\035����\024�P,9K���\005iԛs�(\020bNoJ\\\004c�#�b�\015��\
! \012�\031���{0���\012@\026�\014J�AK�\032.�5!�Lp�\037�<\010Q�a`0Xi�%e\022ЃB��˴�W`�\004�6\
! �PXI�d��lr„`\027ˠ���+��\037�ʌ���f\013��Ae��\001\011/\012nPU}C��D\012�@����\
! �;�DoA)5\023Y�ط\002|а\"\032��}\034\031\014\014�`�{b�\0170=@i��=L\000�\\�Y\005w�\037Ц�˄�1X۾\
! ��\032�r�M�L�r\036I�͌+�i�-��T\013\002�\031G\006�\"X\034\023�g\034\010�A��M�\004#��\003�����A�\012�\
! \013�\012�\026���0Y���w-�;Z�aVc�*���x�\025�X*\000�\013�P�4L�_i\035��Ӭ,H\035�&\036+6\
! \031\007w\015��\003�KJn_{\014�ڥ7\032z\003�Z՜1�d�6�t�U�\020L�\"\017V��PB6�Q�ZI�\017�Y{�{>�\
! ���ˀ\015�q׊g=�sw�����*�|�ſ\032z\023��\030�\033�9�˼p\010;�\037*?�d\"��SO\033\003�yy\
! �\035�B\026DO�\016�\024�\020\000U\010��P����S\004z(�,\007�����욉�|�\024���o�x� \014��^\014���$9\
! X�\011\032��]-�t�#���$�B��\001�Yk�кz#H��j�N��(cX̭�7L;.Z\022p�2���E\013�\035�\
! \001���L\000�Y \013�^�}�\0134?�V��\0373��\001�-��,��6�Ŋ�0\017\012�g�B\013l�4j�AhGB\025�$\
! �\003(\032\007�?�\030�\012�\001�\032��y���'\034\034����%��1+0\016\004m�w@���\033b\015F8]�H8XERE0�)_\
! W���\036Yi�\012Jn�AN��\006�͆�@n���%O�Zb\023e�[\013k�,h!\000\0174��0�;-��#�+�S4�\
! \031G����\0119F�K��\016N�#�_{F�����*���Yȓ\004z�\030����*��\005x�\006\034��\032��c\037�\007��\
! �O����\031\031�\"\002-\0028-�~\014�p\005���\021�\026B �7�\003�\010\004@\017]7P�\037x�\006���MY\004E)���;\
! �\015}X���k�t:>\036���j�����V���An\031.��a<h�;!�n�K\032O\012���\003�/a8�\036h\022��\
! �I�U}@��\035�U��\0034T�w��+�\011\010�\005,uQ�%�>\"a�\027�\033\023d&\020�AY�4k�}�\017���^�\
! E2 &��2\031ٵ\0162��\027�.��\005�\016�Ԩ������kx�Mx�Xq��k�zrR\011����Š��\012�[To\
! D�o�Q\\ȯC9\012�\015�\031\017�y\034��\011�8!�υ�f5��`T̩��\0306� ��0Z�f�\"��!�#�͜\
! ��\031�v��k�p\003��=�\002.\033�(\000C���M�\\M\017\003L2%�c(\001z�N1�\012���O\031�\012zl\037\004�Ά)\
! /�0\005� �K(*��\016 +\001u�\014�\0205�\007Q�*�Y��\014�\015��`K���\021�C\"\001'�c#�\005\013�\032l�o%�\
! �\012��@�â�\0012��ǰ�ڰAy��\007h\002���l�\022\007��\035G��\002��%��h&��\007\030\016��W��|��\
! �\022����,*��U��$r)���\026O�@��\013�`-\007��L��\025`�V}��}\034\020� r�X��t�x0�@A\
! ȁ�\013�Oy1���烿�s�݂�<\037}5<�jҏ]��.���׳�\005_a���-!�B�I�\003�/��\000\
! ��a$970E�`�r\010/ �ȁ\014`���\033�\000����jj\014�\011����\022׊\012�\034|�ȸ8@�r\022fb\001�t�\
! \017��Y�l�����G�hf\022�\024Ӳʠ�$b��\005�w���KA�q����\031�\011�d�X �F�\\��\007�\0228\
! [\011_a�<\025�T`�\013�rn��\002F2`\003`\007��JYH���Vz>\011!z)\\���)��l�F��}���R\010I\"�\
! Q��$\021K��T�Ji�\013&0\001�3���p���\027X\033�'�ѯ�=�\021-�:�\\)\026,���I��$�C?W�\
! ��\036���\013�`�~\037\"��˭�Eڽ��@�\007n3��=�[ZZ��_����\015��\025�O��KO頱��\020\
! K\001��=u���^��S�Vq\003�n�~o\0111\011�2�\0203\0100ְ�����\004S++J�;\023�\\)��\005�\002+L�\
! ]z�K��\025�\004V�C\033L\025Ao�7<�6^��~(��?=\035�{:}}��Y�Y�`\026�\021���{5\035j\0157\013Î\
! �>\012��\011&a\024\\\010&E\036\027\033o�Nn_��tV\017��,��\017�{1Po��+��aB�'���\017<6B�y��Ֆ�\
! �z^��&7\034�\006t��;#�u�b;<��\007��Xu�Ңsz�`\\M\011���\020(��8/>A҆u�x,\
! �S�\021D*�\012�I�}���&��mF��>,�{�\032}\032UWW\037MU\030vRe\037\037��'�F�MI�Տ&C�����\
! �\025\031��&�Zr�S g���Λĝ\016�u�N?�8���\"\016U\012\034€�� \003C��H̺\032-�\013��R���\016\
! \023�\0050�0��XK�3��w��Aߙ���\021?��\027E����bA\031���'��P\032\020L3�ύ1�\026\025\005_9�z�\
! T\031��UY��m8�\002\\�CdS�±q�����E�\032�\"�f�ۋ_-;�y�|�\030�E%mj�,��\020��r��\
! ��2&��B�����}=l���S��m��\017�W\034��D6�������\032�J��$��\027�I��{p��\015�\
! T��h4\032�4��`69����䒊: ��\026�N0�ҳr��\033�Y%f��\013\020\021pg\011�P�\010�\\X9�p'��\
! \003�x\003�����KC�c��\010���\012¾�\012Q\004�a��6[iK\015}~�$,��\030�l\037�6jZ�0 i\014�pa<�\
! ����k���O\024i4����I�\024���\031�\021+�6(A�Ǎ1\020�:�z��v�e�G��&FC�\001��\011�\021�\
! 9�\020�J�c\032\033���p��h��F�6�����\001��6��\011%w\030[i\000B7�\027�2��\000\001�R�A��+ eN�\
! �\026�\003�fm\034���]g�\014 \035\017�=�\006W\017\001��\0236d�\015����\007��%A�Y\003�D�9��[��u����\
! +�����̩\017�B�[\007yJ6������؄�1D�9D�걉O)U����l\033�v�ٻ��B�),��\033B\
! �zi0��j �}��:i�\000&\037�b̈r��j&�\024�\032Z#��\031P\035�\002\001���@��vR�yT���\017�k�S\
! �\003g�&\013\016�)҄\"Ӻ��T\003�]5�ԄK]ӡ��I� ��J�X�uy�����Ւ�*3\032d�L\010|qo\
! ~V\026\005�\0023\001\034=/\002~l�*��'�n�\024!���\025k\000��s���&���-�1�+�ϱN�h3�,Ǥ\033�B\
! @�\035\025��U,� �؀\021t�\005�\010��\0200S�\0050C���#T�\025%:�Ze����Bt��6(\015�v\000�{�V��\
! ��os���j���64�\013��/��q�0�\014����c=]��_�Qā�������=\022s�\034B�Z�\035\000\
! R�,�_]uvW^�v��]͉3���\006���0P���`!\034�_�l�]\025����{��\022�dQ\000�t\024K�\017@�\
! �R��3���!X�n�%Y����n���!�\017#�\007h�X���U�M��ޡ�\016��2Ia\036g�,\014[\006�\010�\
! \003�ԟ\014\006\027_�r�����/���L�ʉT\031��ؓ\012��p9\025�f��\012�xΞF\",1��G<$��KV�O\
! x��`{r�N����׹HZv�������V3�4���s\026�\013�\007�\007��Q������Y\022S�#\000�޿�\
! ��q����?��:�d�P��Pty�N�^�\002�Ň�5x�-&fL]=si��/�[���\025�oO��ڽ�\
! \015��*�i\030?/y\007\015��?rn���\031\014~��ڇ&a9�\032���f�O�\021�oҜ*�vˬ\0175P6ͮF�)gf\
! 5޲_\012N�I��\"�@2<\0311��\017�A�U�WY�@�\001������Z��2\036��U|�\012\006T_��NsW\027\025\
! \0109\015Q\000m\032�\035�(2�Vںa_\0321��Մ\034\"��vY��^�'\015�������D}�7zV��^��2;\025�\
! -\003{���={�2�����(\027�1xFʋ\0013�M�5�����F~\026�«O�\004�\024���O���\030���.y�M\
! ��FTY�\020��?F\022\010\024��\015^\035$\003��E��ܴ�\002�A���\006s\014����C�,�貥`9���t�\026\
! \003\027�m��!\016�1g�q\032��Ģǜ�|\013k\0378�)&����K�\033��1Q.�X�ϣk\007i\023֘S\024^�A��\021\
! m�I#ô�\034>vB=,�V��,\027��\027�}�;�C�����\"\020�8f�\021��\031;\031\016O����t<\032u��\
! �t�J���J��\003�\007Q,�\037�\027�͘֏�z��n7\020ы��09__�[\027���\003\01147^���\026\005L�\
! ��1��g\021����\000������/e޼��\014��ᓎ.z��ԯ7��c\0205\012d�'Wh_Ȉ�O� M!�x*\
! �}���a\031\033�&*DH�����pc�=�mH\024�\037\000�\000%�Hl�\010�t�BO���2k���VX�S�M�\
! Y\"���\001@��w���C5�yŒC<E��ʤ������EA\021���uL5�\0124\013l�I�8�+,��\0343\
! ��\"�h��IR�l�� jD|n���5;���l��S�:���z)\022�\007�0+\015��*��\012~Di��[�j7)\
! �H��]G�Tm�]�x���\\t�W��J\002n��v\011\005\025^Յ\002��\033W���\004w�\023Z���A^�\0258#\
! �6%�\0243�z�I�\026x����&ƹ3���\031�ܧ�\011dhc�礮��M\011VU�,�+qOg�쁥��\023�k\
! �G�]�|�z�U����\001\037���4C<��^~�\012��l�\035�gϞ�w#�\024 0�p\032�+�v�\033��$�O�u\
! ���z�2=F\022��l��A\016sr+rf��=զ^t�5�Q#��m��H\0072\016\0155Ģ[�Ư�w:�\007c�.|�\
! ^�4�\004�\007-�\014)��&�b砿\012\036ns.�\022�~��2W�׮�2�\021��$��ě��\033���\025�j�i\026�\
! $R�Q�vM�iRgF���Ӻ�F3�!�g}�є�|�\037v�I���4Ґ��\017|�5���\003�{�\035ݏ&\025\026\
! |���@��\015�yd)�=St�`â�oG��ePC�\030&�\016���s�f�\013\036�U�b�q7̊\\\007�!�yd\
! �\035\015k��#\023I���M\023[m\034�&��\001m\013��kHv��9\004���%�u��4��\0013!\013����֍O*�\014\
! �~�7z6�����\0079\035�����^\033��1R�S��k\012^��vod�l`Um��U;H݁��F��i;ta\
! �����G\014��$� It\006G~$\012��^3Ay\024*���~\025�0\004j\006̌��\017(�hK�\026���e�\016B��F�\
! ɨO1V\037a��<��m\016T���\003���l����N\032��\016�Li�/���\035���`5N���}ml�\017%\0361\
! ­h\014[�s�\022\010߅��\012����\002�\012�r;��\005t���\026@��\000P���F�j�\001]�\016��s@H\017��@\
! \004M\010�D\034O�x�a�����2`ǷFH�0�q�)A���Z���9Л�V^��\020<�\020<@��E���,09\
! �o:�\021v\035y�J�\030\020b*\023��@�?\031�3s]�:\0224T9�?ͭO\010S\0303���{S7��\012�����>K��\
! ����à��\023%i����)\035\02360&�K�\033���\035*_ɪ�\031�}7�y�:B\014�ţn\002'^�*E�\021.!X\
! O:��v�i�R\000�)m3��M�X�\\:����\024�\025���=�\026l!���c:>�\014\000�$´\007����ƭ\023+v\
! ���6+���u�~��u\030��\001VG)�lұ��Elv����\037\006uJ�\020�cKT�G��u���ލ2�\026�)\031\
! �=�i�� ��,͚��\025��h���mS��O�͎�}�T����d?���\006��\037xQ�ա�Ƨ�0�j\
! t�\005)\025���UM5z��x*}g�|:�g���;V�\030+*7�\016�m�S�}�IC��\007\027\031�|�!f�=7�\
! �x��ױ<�=,,���<�V_\002��=\004�*�2S�\036\024��#x�ULʭ�\014���R\025�R]��dT�[P\001\
! U>�w�_^h;\"��B�\017�eJE\016\003]�?����o��mW�\007¨���˷��6�'#:NBL?����\005�\034\
! ռ��+-�\036i��'l�Oh7�-������Y0}U\037g��<},�\016i�}��^\026�[�\016oWA�8��\034\
! L\027�$\031�vXm�l�a�\012��%�y`�46��&O�#\022T���hw�\013�\037:g�j����V+�GL�I|�\
! 0]t���\030JMb�Au�߄����ҭ>�t��\002\016\021�6���\004�T\\�\\�;�u�z����\005�uҔ-gj5\
! >$f���*�\034Or\023/�\035�\03106G揞\015\033@4_*ӧ�ꊕ�4�G�b�'G��\027I�&ܝ}ph��JG\
! ۹H��7��J9�\023�9wXغ�)h/��\030�V_��fK�6��5s���F\021*���.�\030\015��?V+\006���\
! � ��.��,\020'}���֜����TL\\b�k�T\006�ܫ�\021\026n��k�Yʹ�;�<PY��Ǵ\007S�?�\037\
! TC<\030�F��w/�f�Q�B\026��و�U�(� �y6����jG�\024d�R�>�\037���eYDb�v�8�S\
! E�\026�NP?#o�}/D4�T\037��C��e�ڿ\020\002O�ڭ\013\011��.`ֱ�1��ᓮ���nF�|\033�%�\
! %��S\027\014+��5k|�F�\003�\012_\014V�Z������{\024�9\010��g\001^3�4LA\016Cv:d����l|z��s\
! '\032�~�����\020�^��6���5\0308��}�<��\026s\026N\035�\006��u\035�DK���pu�(~̷�\032\037'ME�_\
! LF��\032�\026\001���A�+�\0248������)�{�3%f�F��;�&��\037��\022���X̣��\021���\033k\
! I@QD�Q_�V_��_�cKr")
  ### end
--- 21,121 ----
  �'���� �\036\016t-���\\\024笌����Q��\000�\024G�s\010b\025��RgOS�i�V|�X�Mޠ�G�@�a\
  ZΪ\011�&�e,�<N��_�\"�\023�\005�\035����\024�P,9K���\005iԛs�(\020bNoJ\\\004c�#�b�\015��\
! \012�\031���{0���\012@\026�\014J�AK�\032.�5!�Lp�\037�<\010Q�a`0Xi��I@�����i\021��`\011\026m�\
! ���$Ɇ���\010�.�AQGK�W�1�?��\031�A+�\025\006C����`\002\022^\024�`���b��\024�-�6����\
! w<�ނNj\036�Xqo\005��]E4\000��82\030\030\024�����\037`y��,Cs�\000���\012�\000?�M��\011)c0�}\005\
! �5,劙���+�<�ț\031�qo�D�Z\016\010�82\010\027���=�@ \016��o�%�\020�\035��\\�_\0142W�^ T\
! ��`�%��R�\025�k���b\017�\032�VQ\\��\003,\010�R\001��M\\\000�z�aB\007�J��E�feA�P5�X��8x\
! k >\036\010\\Rr��ch�.���\033hͪ��$K�\021�ìJ�`�\026y�Z\005�\022�q�zTJ2`�����!�g-\
! \\\006l0��\026<�\000������W��K�\014��Л���\000߀́\\�C؁�P��%\023\031O�z�\026x���\013�\010\
! ֱ zzt �H�\000�B��v\012�\025ϟ\"�C�d9�\025\004֯f�L��ˤ8�~;Ż\004a�v�b\020O\016'���O\
! ЀF�j\031�\013\036y��$\031\024�$\015@�Zˇ��\033A��U\033u��D\031�ZnE�a�q͒�;����/Z\030�8\017��\
! \016f\002P�\002Y��B�#]��\001�\0224���\\\015�o\011~eA\025�y.VL�yP�=�\025Z`[_�Q{\011B;\022��&\001\034@\
! �8\030���XUx\015��8d�{,D>��\010��O.9/�Y�a h;�\003r�^�\020k0��D��*�*�AL��2�\
! \015&��J�EPr�\015r\002�4Xm6�\007r�4�/y\002�\022�(+�ZW+fA\013\001x��Ćq�iɥ\035\001=������\011\
! ���M�/�^�uupr�0��3Ҥ<}�U�޿�B�$�3ĠǯlV�O.�\0115����\037��=8��\002\
! .\024\015���\027\021h\021�iA�c\020�+\030�G� �\020�(�\001\037�O_\005 \000z�z�J���4�|�n�\"(J��M��l�\
! �2��/]\033��q�\020O�V\023\015\0244|�\032^N\017��p�d\016�A\013�\011Yt�\\�8R =\025\034P~\011�\001�@�P\025�Lj\
! O��\003�\\o툮��\036��Z��NW\021M@p/`���,��\021\0113�Hߘ 3��\017ʊ�Y;��|��{��-�\001\
! !AF��Ȯ���\015��t��/�v��F\015�̗��^��m��NJ+�_\023֓�JXOO\016\027V\004�V�ݢz#�|\
! s��B~\035�Q�n��x\030��P�Mp\037�\011q~.�7���\004�bN\035�ǰ�Ā���\0125+\0279G\017i\037yo��\014�\
!  ���\\;�\033��\026p�@G\001\030*��nZ�jz\030`�)\001�\035C\011�\023t�\021T\010�\025|�\020V�c� �w6LyY�\
! )�\007�^BQA\014v\000Y\011��e����?��WQΊ�f�n �\007[�}\034�p\035\022\0118�\034\033Q/X\000�`S~+�DVH\
! m\006�\034\026e\016���?�Eֆ\015�c�>@\023�_�e+�8\010$�8R\036\027�>]/q\015F3�e<�h�ͼ���#V���\
! ��<GgQ!���\034'�K�_�xb\006�\006^\000\006k9P?g���\000\003��\003'��8\004�;��\036fKǃ�\004\012B\016l\
! ]�~ʋ\001\006�w<\037�\005���\026�����W�~��\035vaԜ���,�\012��c�o\011!\024�N\032\037\000|!\024\006��\016\
! #ɹ�):\006C�Cx\001�G\016d\000\013f��0\005\000f|�TSc�L\000�}���VT���#E��\001:��0\023\013h�3|�\027\
! �rf�%t��>:F3�\020���U\006\035&\021\003M.0�3��]\012\002�K\014F=�8O�'��\002�4\022��b\036?����J�\
! \012��P�\002�\\�s;�\0250�\001\033\000;X\035W�B*dg���I\010�K��\036Oq\004g�4\012\006�[�\017�BH\022Y��g\
! '�X\012f��VJ#^0�\0118w��F��=����\030=9��~\005��hQױ�J�`i\006�OҴ$�\035��*���\
! h\005�^�\005���\020�m_n�$���\005�\003=p���������W�j��\016�m\030��0\002�]zJ\007��\005�X\012�\
! ��멫M\027_�z�b��\033�wk�{K�I@�1��A���\005\017TE�&�ZYQ\036ߙ\020�J1n.�\024Xa���\003\
! \\�M�\020&�:\035�`�\010z+�����\\�C�O���`������2�2\004�8��å�ܫ�Pk�Y\030v��Q\
! `%O0\011��B0)��x�vn�Bd��z0�e�M~�ߋ�z��_\011�\015�?��~�\021Z��^���>��\
! z��5��4����\031\021��\024��E�;�<ǪӖ\026���\006�jJ0���@1w'\034u�y�\011�6���cɜ�\
! � R\021V�N��cM�4��o3J-�ay�K��Ө���h�°�*��PU=�4\012nJʭ~4\031R�7d�|��\
! ��5\031�Ւ��\0029���w�$�t8�\023w�i�i�|\024q�R�\020\006l�\007\031\030��Fb��h�^�\024���6��,\
! �鄉��Z��Y���g\015��<$G��\011��(\012LĿ\025\013�8}�=�ΆҀ`��}n�\011��(�ʑ�\003��\010\
! \007��b�mñ\026�j\036\"�z\024��C�\027��.J�\030\02694��^�j�\001̛���.*iS�d�凈��S���\
! 1\031Uw\0252�f�\036��a{熞��m�[|ؼ��O'�ɯ�?�?e�\000�UR�$a���M\032�߃�$o(��M\
! F��P�I.\006��q\035$�'�T�\001)}�Pt�\031���\003�ߘ�*1c�^���;K��\"E����q�;�\037P�\
! \033�Wl�M\\\032�\035\003�GP޿T\020�\015W�\"H\016\033���J[j��K%a�<�Pg�ȵQ�r�\001Ic@�\013���n�\
! �^˽�~�H�ѭ��M\"��\017\025�H�Xq��A\011b>n�������\\��,�=��61\032j\015��NЎ\030̱�\
! �WB\034��7�o�[\025Då7Ҵq�\037L\016�����M(���B\003\020�y�ؕ�5�\007X��\017:�^\001)s\022ݷX\
! \036�4k�����:+d\000�x��a7�z\010H���!+o\010��\027M?�V.\011:�\032�&��q��Z&Ϭ\033�>&_y�\
! ̜]fN}\020\024b�:�S�Qu�~V��&̎!*�!\"T�M|J��/\\\034g�\020�k��eE��La��o�\020��K\
! ��V\002��;��I�\0060�\030\026cF�K�U3餠��\032��̀�\010\027\010�e\005B���ZͣB̟|�_۞2\0368\
! �6Yp�N�&\024�ֽW�\032\020W����&\\�\016m}Lb\004)-U��\022��+,�}Ʈ�<T�� \003fB��{�\
! (�\027�\011��y\021�c[W�u?1t�\010��D�X\0030�\025����4���nьQ]Q|�u�E�Id9&�\010\027\002Z�\
! ��N�b\031\006y�\006���/�E�\027����/�\031\022%�\036��(�Y�*Cn�n\026�c\\�AiP�\003H�����\024}}\
! ��5<U\0334Gl���\\\030�}I7�c��dp��_O\035��:~�2�\"\016,''֧<\005푘K�\020�C�r�\000��f\
! ��ꪳ���S\007�jN���L5\030�����]\015\006\013����f��*(�?��{�(%�\002�X\"~\000b�T�\
! ՜Ie\016\017�\002t\033,�r\037\006\037t�O\036\017�~\030�?@��\022�m�jn��\016mtP-�I\012�83da�2�Gp\037��\
! �d0��◫W/�/~ѵ\007e�TN���=ŞT�\\�˩�5+�W��s�4\022a���>�!ٗ_���~��\005\
! \004ۓKvrTu���EҲ�F�_F������14\034���]P7=\000<����oO�̒�J\034\001\010��%���\
! 3ݧ\027O���Q&��ڞ���Kv���\027�/>D���n11c��K��~!ފ5ϯ`}{zt����n�d\
! U�L��y�;h�W��s�/��`���>4\011ˁ����7SҌ8}��T�[e}���iv5�M93��\
! �Rp�Lҥ\026�\004�����\016�\016ʬ���\006�\015��t��\037�ʝ��P7׭��V0��zN't����@�i\
! �\002h�h�\020E�\031���\015�҈\001]�&�\020�F��\032���?i��\027\\�/\0154\017'ꃾѳ\022L�b��٩�l\031�\
! [\004���{�\021\034W��G����3R^\014�\001o���W\034|\0344�\010\026^}\002$�\010\016�~\002\024��\010\014^t��l��6\
! �ʢ����1�@�H\014n�� \031��/*'��\024�\017b]U7�S\000݅�\015��\033�f�E�-\005˩M禓�\030��\
! n��\017q��9���\010�$\026=���[X���O1!���]:�P���r�Ś|\036];H���\034��\015\022��h[G\
! M\032\031�m��\023�a�*|e�\010�`����\037b�4M�\026�\030\007�1\033��\015���px�Fߌ���Ѩ��E�+\
! T��oU\002�\034�<�ba�@��lƴ~���w���^�������޺���\035H���*�����(`\012�u�\
! ��?�\000��\034\004��\037L�\026)��5\005f�\037\016�ttы��~��V\036��Q {?�B�BF�Z�h\012q�S��3\
! ��\015��H4Q!B\022�w\037���\033s�mC���\000H\007(YGb{Gȥs\025z��ݖY�Ld��\"��roz��\022\021\
! D�\014\000rD���K\034��_�\023f\034�)�oU&E��M�U/\012���\014�c��V�Y`\013L��)�]a\001���/\026\
! AE�\007M��e��\004Q#�s�Ou��\011�tf�ǜ��9�T�K��<\020�Yi��TI�W�#J\013��:U�I�F�\
! %O�:��j����S\037��۽��W\022p�4�K(��.\024��߸*݆�|$���*��\017�J���\031��)\
! ���!�\023O2���7%�71Ν\011,nG̨�>�L C\033[='u\035�nJ��jf9_�{:+f\017,\035\035��\\�<�\
! ��\033գ�\012�e�\015��7|�\031��\007���V��fC�0<{���\033a�\000�y��p]\011�����&�:�\033��\
! �K��1��ud+�\017r��[�3����6��{�A�\032Y�n�GF:�qh�!\026�\"6~}��\021=\030Su��J�\
! �&�>h1eHa�7y\024;\007�U�p�s\011�H��Ĕ��v��!���&��'��%�0-\027]��VSO�\030'��\
! �ڴkRL�:3ڔ?��]4�Q\015�?����4�\013���Lz�����4\004~�\013��e�\037��\003��~4��ࣿ\
! \024\006��l��#Kq홢�\004\033\0265;�N/��\033z�0Qv0�\026��6\003�]�\020��\025ӏ�aV�:\020\014��#K�h\
! X�L\035�H2Ugn��j�`7I5\017h[`�]C����! }W/q��}���5\017�\011Y��m�޶n|Q�dh��\
! �ѳaͤ�&7<��`\034����x-��:�\022G^S��F�{#�f\003�j�7��A�\016Dt5�_/Mۡ\013�\015�\
! M�=bx7'�\005I�38�#Qxg��\011ʣPQ�U��\010�!P3`ft�@qD[ʶ�V7g-�v\020��4ZMF}\
! ���\010C��A�ms����\036\030��g��t��v���v8gJ�;}����@��\006�q����kc�~(�\021nE\
! c�b7�\033�@�.\024%V��NN�\024`U���!--��gϷ\000zO\007��$.6�T�\015�Bu\000ݞ\003Bz�<\005\"hB\
! �&�xBś\014�_}�͖\001;>5B��\021o�[N\011\"���\012$t΁�\\���U��A��\001�E.���g��y}�\
! ����ȓV2ǀ�\023S��O\004r��0���ԑ��ʱ�in}B�˜�-Nߛ��NVx}%���Y�.g��\
! \017\037\006E͟(I;D�\017O阰��1�]��h�T�P�JV��X�y�3�\021b�-\036u\0238��V)ҏp\011�z�\011\
! ζ�M˕\002�Li�Ѽl�ƺ��9�Ng�0���\015�\011�`\013��/\037��)e\000�$\021�=�=d}4n�X����\
! �YIM-�;��v���M\017�:J�g����-b�\023\\M��0�S:�\010\034[��?\002l��̶�n���\030L���\001\
! M\033o\007\011��gi��쭸7E��\006o��f*mv\034�뤲�^'�\031��4h����2�\016E6�L��T��,\
! H��lܭj��\013��S�;���\011<�fGݟ�:�XQ��w\000m����VRN�c��\033�y��\017.2t\002��C\
! LO{n*�q]��ey��X\010\017���Z}\032�J�\020�*��LYzP�ϑ�yW�u)�\0004\004O�KU�K��ƴQ�\
! gnA\005T\012\015�2~����Z\014�>(�)U=\014t\035� ӏj��\036>^U\037\014�&�\027/߾v�t\037��8\0321���\002\
! \027\027�uTsc���t{���[��-a�����c�������U}�����9��:����\023|Y��n�]�]\005Y\
! ��Js0]��d0�a�볽�m+��O��n�9�� *�<�\023�HP�{&�݁/\\~蜭��c�j[��\0371\
! q&\023>��сC��(5�]\026֝~\023�:\023K���ҽ�\0228D\024�DCg\023�R�\001s����u\0128��J\027��IS\
! �������5&���s<�M�\034w�f�ؤ�?z6l\000�|�L�\002�KX��|\037Y���\034m��]$��pw6ơ\
! �#+\035m�\"\001[�p�*��N0\010�aa�*���ܪb�Z}us�-q�|G��Y�\032E�\034�S�Tb4\034\016�X�\
! \030�����Է���@�|�\022�[s��~�R1q�a�9c\031�s��OX�����{�9�\034���r��ӈ�\003�\
! Hk�P{p���c�A5ij\023j�\016\012��o�5;��c\025�����^�:j�\007���/4|V;\025� ӷ\\�!�\
! XwX.�\"\022�kSB\035<j�P���\031��{!�٦�V�\036�=P\013��\026�\020x���nH\020�v��\016���\035\016�\
! t\025/�7<:?��X?y.�����`Xtd�[�C5�b�W�b<��\"��_�\015�Ӳ�YQ�P\013��0b\012r\030\
! ��!\033���g��\023��;ш���\030}�\021��zݷ9&����A\004�\025L-���r�5X$���$�.�~��\
! ���c>\017��~i*R��2�;״�\010��v\015r\\\031�P\002M��/�MY1�K\037;1�8b��L4\011r��Η0��\
! �t�z\037��\016��o�XK\002�\"���n�����\002�7Ws")
  ### end



From anadelonbrin at users.sourceforge.net  Sun Dec 21 23:38:13 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Sun Dec 21 23:38:19 2003
Subject: [Spambayes-checkins] spambayes/spambayes TestToolsUI.py,NONE,1.1
Message-ID: <E1AYHpd-0006KT-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv24316/spambayes

Added Files:
	TestToolsUI.py 
Log Message:
A beginning at a more userfriendly interface to the testing setup.
At the moment to use this, replace the ProxyUserInterface class in sb_server
with the TestTools one (all the old stuff is still available) and go to
http://localhost:8880/cv - it should be straightforward from there.
(I'll post a patch to -dev as well).

Note that the option to use Outlook Express as the source for testing
data doesn't quite work yet.  The 'standard' and 'cache' ones do, from
the testing I've done so far.


--- NEW FILE: TestToolsUI.py ---
"""Testing Tools Web Interface

Classes:
    TestToolsUserInterface - Interface class for testing tools.

Abstract:

This module implements a browser based Spambayes user interface for the
various testing tools.  Users may use it to interface with the tools.

The following functions are currently included:
  onCV - cross-validation testing

To do:
 o Add interface to Alex's incremental test setup.
 o Suggestions?
"""

# This module is part of the spambayes project, which is Copyright 2002-3
# The Python Software Foundation and is covered by the Python Software
# Foundation license.

__author__ = "Tony Meyer <ta-meyer@ihug.co.nz>"
__credits__ = "All the Spambayes folk."

try:
    True, False
except NameError:
    # Maintain compatibility with Python 2.2
    True, False = 1, 0

import os
import sys
import cgi
import glob
import random
import StringIO

import ProxyUI
import oe_mailbox
from spambayes import msgs
from spambayes import TestDriver
from spambayes import OptionsClass
from spambayes.Options import options

# These are the options that will be offered on the testing page.
# If the option is None, then the entry is a header and the following
# options will appear in a new box on the configuration page.
# These are also used to generate http request parameters and template
# fields/variables.
testtools_ini_map = (
##    ('General Options', None),
#   Put any general options that we wish to encourage people to test
#   here, for example:
#   ('Classifier',           'max_discriminators'),
    ('Experimental Options', None),
)

# Dynamically add any current experimental/deprecated options.
for opt in options.options(True):
    sect, opt = opt[1:].split(']', 1)
    if opt[:2].lower() == "x-":
        testtools_ini_map += ((sect, opt),)

class TestToolsUserInterface(ProxyUI.ProxyUserInterface):
    """Serves the HTML user interface for the test tools."""
    def onCv(self):
        global testtools_ini_map
        self._writePreamble("CV Test")
        configTable = self.html.configForm.clone()
        del configTable.configTextRow1
        del configTable.configTextRow2
        del configTable.configCbRow1
        del configTable.configRow2
        del configTable.blankRow
        del configTable.folderRow

        # Add some options that are only available via this page.
        # (This makes displaying the options nice and easy, since
        # they're just handled like everything else).
        sect = 'TestToolsUI'
        for newopt in [('source', 'Messages source', 'Standard test setup',
                        'Select the source of the messages to test on.',
                        ('Standard test setup', 'Cache', 'Outlook Express'),
                        False),
                       ('n', 'Number of runs', 10,
                        'Select the number of cross-validation runs.',
                        OptionsClass.INTEGER, False),]:
            options._options[sect, newopt[0]] = OptionsClass.Option(*newopt)
        testtools_ini_map += (('Testing Options', None),
                              ('TestToolsUI', 'source'),
                              ('TestToolsUI', 'n'),)
        
        option_choice = self._buildConfigPageBody(\
            configTable, testtools_ini_map)
        option_choice.action_page.action = "cvresults"
        option_choice.introduction = "Select the options for your test " \
                                     "(these will be run against the " \
                                     "defaults)."
        option_choice.optionsPathname = "memory only"
        del option_choice.restore_form
        del option_choice.adv_button
        option_choice.config_submit.value = "Run Test"
        self.write(option_choice)
        self._writePostamble()

    def onCvresults(self, *args, **kwargs):
        del kwargs["how"]
        self._writePreamble("CV Test Results")
        text = "Display the results of a cross-validation test with the " \
               "current settings against the defaults."
        nsets = options["TestToolsUI", "n"]

        # With defaults first.
        self.write("<p>Testing with defaults...</p>")
        saved = {}
        for opt in options.options(True):
            # Ignore those that have do_not_restore as True
            # (These are predominately storage options, and at least
            # the cache directory ones may be needed later on).
            sect, opt = opt[1:].split(']', 1)
            saved[(sect, opt)] = options[(sect, opt)]
            if not options.no_restore(sect, opt):
                options.set(sect, opt, options.default(sect, opt))
        options["TestToolsUI", "source"] = kwargs["TestToolsUI_source"]
        # XXX Cache this somewhere?  If the testing data isn't changing,
        # XXX and the user is running multiple tests, then it doesn't
        # XXX make much sense to rerun the 'default's test over and over
        # XXX again.
        cv_out, errors = self.timCV(nsets)
##        print errors.read()
        defaults = self.rates(cv_out)

        # Now with specified settings.
        self.write("<p>Testing with selected settings...</p>")
        for opt in options.options(True):
            sect, opt = opt[1:].split(']', 1)
            try:
                value = kwargs["%s_%s" % (sect, opt)]
            except KeyError:
                # Leave as the default.
                pass
            else:
                options.set(sect, opt, value)
        cv_out, errors = self.timCV(nsets)
##        print errors.read()
        current = self.rates(cv_out)

        # Restore the settings.
        for opt in options.options(True):
            sect, opt = opt[1:].split(']', 1)
            options.set(sect, opt, saved[(sect, opt)])

        # Do the comparison.
        comp, errors = self.compare(defaults, current)
##        print errors.read()

        # Output the results
        # XXX This is just what you'd get from running cmp.py
        # XXX at the moment - it could be prettied up a bit.
        comp = comp.read()
        box = self._buildBox('Cross-validation test', None,
                             cgi.escape(comp).replace("\n", "<br />"))
        self.write(box)
        self._writePostamble()

    def timCV(self, nsets):
        # Until we are un-lazy enough to change the code borrowed from
        # timcv.py, just capture the output that normally goes to stdout
        # or stderr and return it.
        cout, cerr = sys.stdout, sys.stderr
        sys.stdout = StringIO.StringIO()
        sys.stderr = StringIO.StringIO()

        if options["TestToolsUI", "source"] == "Standard test setup":
            # Source the test data from the 'standard' test setup,
            # as described in the testtools directory.
            hamdirs  = [options["TestDriver", "ham_directories"] % \
                        i for i in range(1, nsets+1)]
            spamdirs = [options["TestDriver", "spam_directories"] % \
                        i for i in range(1, nsets+1)]
            hstream = msgs.HamStream
            sstream = msgs.SpamStream
        elif options["TestToolsUI", "source"] == "Cache":
            # Source the test data from the cache directories
            # specified in the "Storage" section of the configuration.
            # This means that we have one 'ham' directory and one
            # 'spam' directory (we ignore the unknown one, obviously),
            # but what we really want is n directories of ham and spam.
            # To overcome this without actually moving the files about
            # we have a class that fakes it for us.
            hamdirs  = ["%s%s%s/%s" % (options["Storage", "ham_cache"],
                                       os.pathsep, i, nsets)
                        for i in range(1, nsets+1)]
            spamdirs = ["%s%s%s/%s" % (options["Storage", "spam_cache"],
                                       os.pathsep, i, nsets)
                        for i in range(1, nsets+1)]
            hstream = HamCacheStream
            sstream = SpamCacheStream
        elif options["TestToolsUI", "source"] == "Outlook Express":
            # Source the test data from Outlook Express
            # Pretty crude at the moment (hard coded):
            #   Ignores:
            #     o Deleted Items
            #     o Drafts
            #     o Folders
            #     o Offline
            #     o Outbox
            #     o Pop3uidl
            #   Assumes that anything with 'spam' in the name is spam
            #   (so don't have a folder called "looks like spam"!) and
            #   that anything else is ham.
            # No mixed dbxes!
            # Each dbx is the equivilent of a directory in the 'standard'
            # test setup - so it would be good if each dbx had a roughly
            # equal number of messages, and if there were the same number
            # of ham and spam dbxes.
            dbx_dir = oe_mailbox.OEStoreRoot()
            dbxes = glob.glob(os.path.join(dbx_dir, "*.dbx"))
            spamdirs = []
            hamdirs = []
            for dbx in dbxes:
                if os.path.splitext(os.path.basename(dbx))[0].lower() in \
                   ["deleted items", "drafts", "folders",
                    "offline", "outbox", "pop3uidl",]:
                  continue
                elif dbx.lower().find("spam") == -1:
                    spamdirs.append(dbx)
                else:
                    hamdirs.append(dbx)
            hstream = oe_mailbox.OEHamStream
            sstream = oe_mailbox.OESpamStream

        d = TestDriver.Driver()
        # Train it on all sets except the first.
        h = hstream("%s-%d" % (hamdirs[1], nsets), hamdirs[1:], train=1)
        d.train(hstream("%s-%d" % (hamdirs[1], nsets), hamdirs[1:],
                        train=1),
                sstream("%s-%d" % (spamdirs[1], nsets), spamdirs[1:],
                        train=1))

        # Now run nsets times, predicting pair i against all except pair i.
        for i in range(nsets):
            h = hamdirs[i]
            s = spamdirs[i]
            hamstream = hstream(h, [h], train=0)
            spamstream = sstream(s, [s], train=0)

            if i > 0:
                if options["CV Driver", "build_each_classifier_from_scratch"]:
                    # Build a new classifier from the other sets.
                    d.new_classifier()

                    hname = "%s-%d, except %d" % (hamdirs[0], nsets, i+1)
                    h2 = hamdirs[:]
                    del h2[i]

                    sname = "%s-%d, except %d" % (spamdirs[0], nsets, i+1)
                    s2 = spamdirs[:]
                    del s2[i]

                    d.train(hstream(hname, h2, train=1),
                            sstream(sname, s2, train=1))

                else:
                    # Forget this set.
                    d.untrain(hamstream, spamstream)

            # Predict this set.
            d.test(hamstream, spamstream)
            d.finishtest()

            if i < nsets - 1 and not options["CV Driver",
                                             "build_each_classifier_from_scratch"]:
                # Add this set back in.
                d.train(hamstream, spamstream)
        d.alldone()

        # Other end of the lazy 'capture the output' code.
        sys.stdout.seek(0)
        sys.stderr.seek(0)
        out, err = sys.stdout, sys.stderr
        sys.stdout = cout
        sys.stderr = cerr
        return out, err

    def rates(self, ifile):
        """This is essentially rates.py from the testtools directory."""
        # XXX Stop being lazy and using the remapping cout/cerr cheat
        # XXX at some point.
        cout = sys.stdout
        cerr = sys.stderr
        sys.stdout = StringIO.StringIO()
        sys.stderr = StringIO.StringIO()
        
        interesting = filter(lambda line: line.startswith('-> '), ifile)
        ifile.close()

        ofile = StringIO.StringIO()

        def dump(*stuff):
            msg = ' '.join(map(str, stuff))
            print msg
            print >> ofile, msg

        ntests = nfn = nfp = 0
        sumfnrate = sumfprate = 0.0

        for line in interesting:
            dump(line[:-1])
            fields = line.split()

            # 0      1      2    3    4 5    6                 -5  -4 -3   -2    -1
            #-> <stat> tested 4000 hams & 2750 spams against 8000 hams & 5500 spams
            if line.startswith('-> <stat> tested '):
                ntests += 1
                continue

            #  0      1     2        3
            # -> <stat> false positive %: 0.025
            # -> <stat> false negative %: 0.327272727273
            if line.startswith('-> <stat> false '):
                kind = fields[3]
                percent = float(fields[-1])
                if kind == 'positive':
                    sumfprate += percent
                    lastval = percent
                else:
                    sumfnrate += percent
                    dump('    %7.3f %7.3f' % (lastval, percent))
                continue

            #  0      1 2   3     4         5
            # -> <stat> 1 new false positives
            if len(fields) >= 5 and fields[3] == 'new' and fields[4] == 'false':
                kind = fields[-1]
                count = int(fields[2])
                if kind == 'positives':
                    nfp += count
                else:
                    nfn += count

        dump('total unique false pos', nfp)
        dump('total unique false neg', nfn)
        dump('average fp %', sumfprate / ntests)
        dump('average fn %', sumfnrate / ntests)
        ofile.seek(0)
        sys.stdout = cout
        sys.stderr = cerr
        return ofile

    def compare(self, f1, f2):
        """This is essentially cmp.py from the testtools directory."""
        # XXX Stop being lazy and using the remapping cout/cerr cheat
        # XXX at some point.
        cout, cerr = sys.stdout, sys.stderr
        sys.stdout = StringIO.StringIO()
        sys.stderr = StringIO.StringIO()

        def suck(f):
            fns = []
            fps = []
            hamdev = []
            spamdev = []
            hamdevall = spamdevall = (0.0, 0.0)

            get = f.readline
            while 1:
                line = get()
                if line.startswith('-> <stat> tested'):
                    print line,
                if line.find(' items; mean ') != -1:
                    # -> <stat> Ham distribution for this pair: 1000 items; mean 0.05; sample sdev 0.68
                    # and later "sample " went away
                    vals = line.split(';')
                    mean = float(vals[1].split()[-1])
                    sdev = float(vals[2].split()[-1])
                    val = (mean, sdev)
                    typ = vals[0].split()[2]
                    if line.find('for all runs') != -1:
                        if typ == 'Ham':
                            hamdevall = val
                        else:
                            spamdevall = val
                    elif line.find('all in this') != -1:
                        if typ == 'Ham':
                            hamdev.append(val)
                        else:
                            spamdev.append(val)
                    continue
                if line.startswith('-> '):
                    continue
                if line.startswith('total'):
                    break
                if len(line) == 0:
                    continue
                # A line with an f-p rate and an f-n rate.
                p, n = map(float, line.split())
                fps.append(p)
                fns.append(n)

            # "total unique false pos 0"
            # "total unique false neg 0"
            # "average fp % 0.0"
            # "average fn % 0.0"
            fptot = int(line.split()[-1])
            fntot = int(get().split()[-1])
            fpmean = float(get().split()[-1])
            fnmean = float(get().split()[-1])
            return (fps, fns, fptot, fntot, fpmean, fnmean,
                    hamdev, spamdev, hamdevall, spamdevall)

        def tag(p1, p2):
            if p1 == p2:
                t = "tied          "
            else:
                t = p1 < p2 and "lost " or "won  "
                if p1:
                    p = (p2 - p1) * 100.0 / p1
                    t += " %+7.2f%%" % p
                else:
                    t += " +(was 0)"
            return t

        def mtag(m1, m2):
            mean1, dev1 = m1
            mean2, dev2 = m2
            t = "%7.2f %7.2f " % (mean1, mean2)
            if mean1:
                mp = (mean2 - mean1) * 100.0 / mean1
                t += "%+7.2f%%" % mp
            else:
                t += "+(was 0)"
            t += "     %7.2f %7.2f " % (dev1, dev2)
            if dev1:
                dp = (dev2 - dev1) * 100.0 / dev1
                t += "%+7.2f%%" % dp
            else:
                t += "+(was 0)"
            return t

        def dump(p1s, p2s):
            alltags = ""
            for p1, p2 in zip(p1s, p2s):
                t = tag(p1, p2)
                print "    %5.3f  %5.3f  %s" % (p1, p2, t)
                alltags += t + " "
            print
            for t in "won", "tied", "lost":
                print "%-4s %2d times" % (t, alltags.count(t))
            print

        def dumpdev(meandev1, meandev2):
            for m1, m2 in zip(meandev1, meandev2):
                print mtag(m1, m2)

        (fp1, fn1, fptot1, fntot1, fpmean1, fnmean1,
         hamdev1, spamdev1, hamdevall1, spamdevall1) = suck(f1)

        (fp2, fn2, fptot2, fntot2, fpmean2, fnmean2,
         hamdev2, spamdev2, hamdevall2, spamdevall2) = suck(f2)

        print
        print "false positive percentages"
        dump(fp1, fp2)
        print "total unique fp went from", fptot1, "to", fptot2, tag(fptot1, fptot2)
        print "mean fp % went from", fpmean1, "to", fpmean2, tag(fpmean1, fpmean2)

        print
        print "false negative percentages"
        dump(fn1, fn2)
        print "total unique fn went from", fntot1, "to", fntot2, tag(fntot1, fntot2)
        print "mean fn % went from", fnmean1, "to", fnmean2, tag(fnmean1, fnmean2)

        print
        if len(hamdev1) == len(hamdev2) and len(spamdev1) == len(spamdev2):
            print "ham mean                     ham sdev"
            dumpdev(hamdev1, hamdev2)
            print
            print "ham mean and sdev for all runs"
            dumpdev([hamdevall1], [hamdevall2])


            print
            print "spam mean                    spam sdev"
            dumpdev(spamdev1, spamdev2)
            print
            print "spam mean and sdev for all runs"
            dumpdev([spamdevall1], [spamdevall2])

            print
            diff1 = spamdevall1[0] - hamdevall1[0]
            diff2 = spamdevall2[0] - hamdevall2[0]
            print "ham/spam mean difference: %2.2f %2.2f %+2.2f" % (diff1,
                                                                    diff2,
                                                                    diff2 - diff1)
        else:
            print "[info about ham & spam means & sdevs not available in both files]"

        sys.stdout.seek(0)
        sys.stderr.seek(0)
        out, err = sys.stdout, sys.stderr
        sys.stdout = cout
        sys.stderr = cerr
        return out, err


# The iterator yields a stream of Msg objects from the given
# 'directory'.  The directory is actually the actual directory
# and then an indication of the portion of it that we are after.
# (so that a single directory can be used, a la the caches, rather
# than a nicely split up into sets directory).
class CacheStream(msgs.MsgStream):
    def produce(self):
        # We only want some of the msgs.  Shuffle each directory list, but
        # in such a way that we'll get the same result each time this is
        # called on the same directory list.
        base_check = None
        for directory in self.directories:
            directory, portion = directory.split(os.pathsep)
            # All the directories in the list *must* be the same, and just
            # different sections, because this makes the code easier, and is
            # the desired usage, anyway.
            if base_check is None:
                base_check = directory
            assert directory == base_check

            set_num, nsets = portion.split('/')

            all = os.listdir(directory)
            random.seed(hash(max(all)) ^ msgs.SEED)
            random.shuffle(all)

            set_size = len(all) // int(nsets)
            set_num = int(set_num)
            set = all[set_num*set_size:((set_num+1)*set_size)-1]
            set.sort()
            for fname in set:
                yield msgs.Msg(directory, fname)

class HamCacheStream(CacheStream):
    def __init__(self, tag, directories, train=0):
        if train:
            CacheStream.__init__(self, tag, directories, msgs.HAMTRAIN)
        else:
            CacheStream.__init__(self, tag, directories, msgs.HAMTEST)

class SpamCacheStream(CacheStream):
    def __init__(self, tag, directories, train=0):
        if train:
            CacheStream.__init__(self, tag, directories, msgs.SPAMTRAIN)
        else:
            CacheStream.__init__(self, tag, directories, msgs.SPAMTEST)



From montanaro at users.sourceforge.net  Mon Dec 22 11:55:08 2003
From: montanaro at users.sourceforge.net (Skip Montanaro)
Date: Mon Dec 22 11:55:11 2003
Subject: [Spambayes-checkins] spambayes/spambayes tokenizer.py,1.25,1.26
Message-ID: <E1AYTKm-00054P-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv19474

Modified Files:
	tokenizer.py 
Log Message:
Sendmail annotates the Received: header with "(may be forged)" if it thinks
the sender is forging its identity.  This shows up on infrequent occasions
but is much more strongly associated with spam than ham.  Guarded by
options["Tokenizer", "mine_received_headers"] like the rest of the Received:
header mucking.



Index: tokenizer.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/tokenizer.py,v
retrieving revision 1.25
retrieving revision 1.26
diff -C2 -d -r1.25 -r1.26
*** tokenizer.py	19 Dec 2003 12:00:28 -0000	1.25
--- tokenizer.py	22 Dec 2003 16:55:05 -0000	1.26
***************
*** 1608,1611 ****
--- 1608,1618 ----
          if options["Tokenizer", "mine_received_headers"]:
              for header in msg.get_all("received", ()):
+                 # Sendmail adds a chit to Received: headers if it thinks
+                 # the sender is forging its identity.  That seems to be
+                 # a pretty reliable spam clue.  I'll leave it for others
+                 # to decide if this should be pulled outside the check
+                 # for mine_received_headers.
+                 if header.lower().find('may be forged') != -1:
+                     yield 'received:may be forged'
                  for pat, breakdown in [(received_host_re, breakdown_host),
                                         (received_ip_re, breakdown_ipaddr)]:



From popiel at users.sourceforge.net  Mon Dec 22 18:32:55 2003
From: popiel at users.sourceforge.net (T. Alexander Popiel)
Date: Mon Dec 22 18:32:58 2003
Subject: [Spambayes-checkins] spambayes/testtools regimes.py,1.3,1.4
Message-ID: <E1AYZXj-0001bx-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/testtools
In directory sc8-pr-cvs1:/tmp/cvs-serv6072

Modified Files:
	regimes.py 
Log Message:
Added nonedge regime, corrected comment on expire4months regime.


Index: regimes.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/testtools/regimes.py,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** regimes.py	16 Dec 2003 05:06:34 -0000	1.3
--- regimes.py	22 Dec 2003 23:32:53 -0000	1.4
***************
*** 86,91 ****
  ###
  ### This is a training regime for the incremental.py harness.
! ### It does guess-based training on all messages, followed by
! ### correction to perfect at the end of each group.
  ###
  
--- 86,112 ----
  ###
  ### This is a training regime for the incremental.py harness.
! ### It does perfect training for all messages not already
! ### properly classified with extreme confidence.
! ###
! 
! class nonedge:
!     def __init__(self):
!         pass
! 
!     def group_action(self, which, test):
!         pass
! 
!     def guess_action(self, which, test, guess, actual, msg):
!         if guess[0] != actual:
!             return actual
!         if 0.005 < guess[1] and guess[1] < 0.995:
!             return actual
!         return 0
! 
! 
! ###
! ### This is a training regime for the incremental.py harness.
! ### It does perfect training on all messages, followed by
! ### untraining after 120 groups have gone by.
  ###
  



From mhammond at users.sourceforge.net  Mon Dec 22 22:12:42 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Mon Dec 22 22:12:51 2003
Subject: [Spambayes-checkins] spambayes/windows pop3proxy_tray.py,1.19,1.20
Message-ID: <E1AYcyQ-0002Oh-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/windows
In directory sc8-pr-cvs1:/tmp/cvs-serv9164

Modified Files:
	pop3proxy_tray.py 
Log Message:
Binary version failed to check for most recent version.


Index: pop3proxy_tray.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/windows/pop3proxy_tray.py,v
retrieving revision 1.19
retrieving revision 1.20
diff -C2 -d -r1.19 -r1.20
*** pop3proxy_tray.py	17 Dec 2003 06:45:44 -0000	1.19
--- pop3proxy_tray.py	23 Dec 2003 03:12:40 -0000	1.20
***************
*** 524,531 ****
              latest = fetch_latest_dict()
              SetWaitCursor(0)
!             latest_ver_string = get_version_string(app_name, version_string_key,
!                                                    version_dict=latest)
!             latest_ver_num = get_version_number(app_name, version_number_key,
!                                                 version_dict=latest)
          except:
              self.ShowMessage("Error checking the latest version")
--- 524,536 ----
              latest = fetch_latest_dict()
              SetWaitCursor(0)
!             try:
!                 latest_ver_string = get_version_string(app_name, version_string_key,
!                                                        version_dict=latest)
!                 latest_ver_num = get_version_number(app_name, version_number_key,
!                                                     version_dict=latest)
!             except KeyError:
!                 # "Full Description Binary" not in the version currently on the web
!                 latest_ver_string = "0.1"
!                 latest_ver_num = 0.1
          except:
              self.ShowMessage("Error checking the latest version")



From mhammond at users.sourceforge.net  Mon Dec 22 22:17:03 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Mon Dec 22 22:17:06 2003
Subject: [Spambayes-checkins] spambayes/spambayes Version.py,1.25,1.26
Message-ID: <E1AYd2d-0002bq-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv9943

Modified Files:
	Version.py 
Log Message:
Hack Version.py to reflect a later version for test binaries.  I will
fix this properly soon, but not now.


Index: Version.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/Version.py,v
retrieving revision 1.25
retrieving revision 1.26
diff -C2 -d -r1.25 -r1.26
*** Version.py	11 Dec 2003 18:44:47 -0000	1.25
--- Version.py	23 Dec 2003 03:17:01 -0000	1.26
***************
*** 37,44 ****
          },
          "Outlook" : {
!             "Version":          0.81,
!             "BinaryVersion":    0.81,
              "Description":      "SpamBayes Outlook Addin",
!             "Date":             "September 9, 2003",
              "Full Description": "%(Description)s, version %(Version)s (%(Date)s)",
              "Full Description Binary":
--- 37,44 ----
          },
          "Outlook" : {
!             "Version":          0.85,
!             "BinaryVersion":    0.85,
              "Description":      "SpamBayes Outlook Addin",
!             "Date":             "December 2003",
              "Full Description": "%(Description)s, version %(Version)s (%(Date)s)",
              "Full Description Binary":
***************
*** 51,57 ****
          "POP3 Proxy" : {
              "Version":          0.3,
!             "BinaryVersion":    0.1,
              "Description":      "SpamBayes POP3 Proxy Beta3",
!             "Date":             "September 2003",
              "InterfaceVersion":            0.03,
              "InterfaceDescription":        "SpamBayes POP3 Proxy Web Interface Alpha3",
--- 51,57 ----
          "POP3 Proxy" : {
              "Version":          0.3,
!             "BinaryVersion":    0.2,
              "Description":      "SpamBayes POP3 Proxy Beta3",
!             "Date":             "December 2003",
              "InterfaceVersion":            0.03,
              "InterfaceDescription":        "SpamBayes POP3 Proxy Web Interface Alpha3",



From mhammond at users.sourceforge.net  Mon Dec 22 22:17:43 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Mon Dec 22 22:17:46 2003
Subject: [Spambayes-checkins] spambayes/windows readme_proxy.html,1.2,1.3
Message-ID: <E1AYd3H-0002eB-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/windows
In directory sc8-pr-cvs1:/tmp/cvs-serv10144

Modified Files:
	readme_proxy.html 
Log Message:
Add a couple of links to the sb_server binary readme (which still sucks!)


Index: readme_proxy.html
===================================================================
RCS file: /cvsroot/spambayes/spambayes/windows/readme_proxy.html,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** readme_proxy.html	19 Sep 2003 08:10:25 -0000	1.2
--- readme_proxy.html	23 Dec 2003 03:17:41 -0000	1.3
***************
*** 23,30 ****
  
  <p><em>Note that if you use Microsoft Outlook, you should probably be using the
! Outlook Addin, which was presented as an option at installation time.</em></p>
  
  <p>This SpamBayes application sits between your mail server (generally at
! your ISP) and your mail program. Your mail program still believes
  it is talking to a real mail server, but instead it is talking to
  SpamBayes, which acts as a <span style="font-style: italic;">proxy</span>
--- 23,32 ----
  
  <p><em>Note that if you use Microsoft Outlook, you should probably be using the
! Outlook Addin, which was presented as an option at installation time.</em>
! If you are using Outlook Express or almost any other mail program, 
! you are in the right place</p>
  
  <p>This SpamBayes application sits between your mail server (generally at
! your Internet Service Provider) and your mail program. Your mail program still believes
  it is talking to a real mail server, but instead it is talking to
  SpamBayes, which acts as a <span style="font-style: italic;">proxy</span>
***************
*** 44,47 ****
--- 46,55 ----
  been received, and let you train them as <b>ham</b> or <b>spam</b>.  You
  can also configure SpamBayes via this interface in your web browser.</p>
+ 
+ <p>The <a href="http://spambayes.sourceforge.net/faq.html#using-spambayes">
+ using SpamBayes section in the online FAQ</a> has alot of information
+ which you may find useful, especially considering how little documentation
+ is supplied.  The <a href="http://spambayes.sourceforge.net/">main SpamBayes
+ project pages</a> may also be of interest.</p>
  
  <h3>Getting Started</h3>



From mhammond at users.sourceforge.net  Mon Dec 22 22:19:29 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Mon Dec 22 22:19:34 2003
Subject: [Spambayes-checkins] spambayes/windows spambayes.iss,1.3,1.4
Message-ID: <E1AYd4z-0002kB-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/windows
In directory sc8-pr-cvs1:/tmp/cvs-serv10282

Modified Files:
	spambayes.iss 
Log Message:
Along with checking Outlook isn't running, check Outlook's mail transport
also isn't running, and that an existing sb_server isn't.

Fix uninstall problem - uninstall should be 100% clean, assuming Outlook
isn't running.

Few other minor fixes


Index: spambayes.iss
===================================================================
RCS file: /cvsroot/spambayes/spambayes/windows/spambayes.iss,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** spambayes.iss	4 Dec 2003 03:12:41 -0000	1.3
--- spambayes.iss	23 Dec 2003 03:19:27 -0000	1.4
***************
*** 12,20 ****
  OutputBaseFilename=SpamBayes-Setup
  ShowComponentSizes=no
! ; Note the check for Outlook running has already been done, so no point
! ; having this file tell them to shutdown outlook!
! ; Edit file using Windows 'wordpad'
! ;InfoBeforeFile=installation_notes.rtf
  [Files]
  Source: "py2exe\dist\lib\*.*"; DestDir: "{app}\lib"; Flags: ignoreversion
  Source: "py2exe\dist\bin\python23.dll"; DestDir: "{app}\bin"; Flags: ignoreversion
--- 12,20 ----
  OutputBaseFilename=SpamBayes-Setup
  ShowComponentSizes=no
! UninstallDisplayIcon={app}\sbicon.ico
! 
  [Files]
+ Source: "py2exe\dist\sbicon.ico"; DestDir: "{app}"; Flags: ignoreversion
+ 
  Source: "py2exe\dist\lib\*.*"; DestDir: "{app}\lib"; Flags: ignoreversion
  Source: "py2exe\dist\bin\python23.dll"; DestDir: "{app}\bin"; Flags: ignoreversion
***************
*** 22,26 ****
  Source: "py2exe\dist\lib\PyWinTypes23.dll"; DestDir: "{app}\bin"; Flags: ignoreversion
  
! Source: "py2exe\dist\bin\outlook_addin.dll"; DestDir: "{app}\bin"; Check: InstallingOutlook; Flags: ignoreversion regserver
  Source: "py2exe\dist\bin\outlook_dump_props.exe"; DestDir: "{app}\bin"; Check: InstallingOutlook; Flags: ignoreversion
  Source: "py2exe\dist\docs\outlook\*.*"; DestDir: "{app}\docs\outlook"; Check: InstallingOutlook; Flags: ignoreversion recursesubdirs
--- 22,27 ----
  Source: "py2exe\dist\lib\PyWinTypes23.dll"; DestDir: "{app}\bin"; Flags: ignoreversion
  
! Source: "py2exe\dist\bin\outlook_addin.dll"; DestDir: "{app}\bin"; Check: InstallingOutlook; Flags: ignoreversion
! Source: "py2exe\dist\bin\outlook_addin_register.exe"; DestDir: "{app}\bin"; Check: InstallingOutlook; Flags: ignoreversion
  Source: "py2exe\dist\bin\outlook_dump_props.exe"; DestDir: "{app}\bin"; Check: InstallingOutlook; Flags: ignoreversion
  Source: "py2exe\dist\docs\outlook\*.*"; DestDir: "{app}\docs\outlook"; Check: InstallingOutlook; Flags: ignoreversion recursesubdirs
***************
*** 30,40 ****
  Source: "py2exe\dist\bin\sb_upload.exe"; DestDir: "{app}\bin"; Check: InstallingProxy; Flags: ignoreversion
  Source: "py2exe\dist\docs\sb_server\readme_proxy.html"; DestDir: "{app}\docs\sb_server"; Check: InstallingProxy; Flags: isreadme
  
  [Tasks]
! Name: startup; Description: "Execute SpamBayes each time Windows starts";
  Name: desktop; Description: "Add an icon to the desktop"; Flags: unchecked;
  
  [Run]
! FileName:"{app}\bin\sb_tray.exe"; Description: "Start the server now"; Flags: postinstall skipifdoesntexist nowait
  
  [Icons]
--- 31,54 ----
  Source: "py2exe\dist\bin\sb_upload.exe"; DestDir: "{app}\bin"; Check: InstallingProxy; Flags: ignoreversion
  Source: "py2exe\dist\docs\sb_server\readme_proxy.html"; DestDir: "{app}\docs\sb_server"; Check: InstallingProxy; Flags: isreadme
+ Source: "py2exe\dist\docs\sb_server\*.*"; DestDir: "{app}\docs\sb_server"; Check: InstallingProxy; Flags: recursesubdirs
+ 
+ ; There is a problem attempting to get Inno to unregister our DLL.  If we mark our DLL
+ ; as 'regserver', it installs and registers OK, but at uninstall time, it unregisters
+ ; OK, but Inno is then unable to delete the files.  My guess is Inno loads the DLL,
+ ; calls the function, and unloads the library.  For our executables, this process
+ ; will leave many files in use.
+ ; We get around this by having a little executable purely for registration and
+ ; unregistration.
+ [Run]
+ Filename: "{app}\bin\outlook_addin_register.exe"; StatusMsg: "Registering Outlook Addin"; Check: InstallingOutlook;
+ [UninstallRun]
+ Filename: "{app}\bin\outlook_addin_register.exe"; Parameters: "--unregister"; StatusMsg: "Unregistering Outlook Addin";Check: InstallingOutlook;
  
  [Tasks]
! Name: startup; Description: "Execute SpamBayes each time Windows starts"; Check: InstallingProxy
  Name: desktop; Description: "Add an icon to the desktop"; Flags: unchecked;
  
  [Run]
! FileName:"{app}\bin\sb_tray.exe"; Description: "Start the server now"; Flags: postinstall skipifdoesntexist nowait; Check: InstallingProxy
  
  [Icons]
***************
*** 68,85 ****
  end;
  
! function InitializeSetup(): Boolean;
  begin
      Result := true;
      while Result do begin
!         if not CheckForMutexes('_outlook_mutex_') then
              break;
!           Result := MsgBox(
!               'You must close Outlook before SpamBayes can be installed.' + #13 + #13 +
                'Please close all Outlook Windows (using "File->Exit and Log off"' + #13 +
                'if available) and click Retry, or click Cancel to exit the installation.'+ #13 + #13 +
                'If this message persists after closing all Outlook windows, you may' + #13 +
!               'need to log off from Windows, and try again.',
!               mbConfirmation, MB_RETRYCANCEL) = idRetry;
      end;
      // default our install type.
      if IsOutlookInstalled() then begin
--- 82,126 ----
  end;
  
! function CheckNoAppMutex( mutexName: String; closeMsg: String) : Boolean;
  begin
      Result := true;
      while Result do begin
!         if not CheckForMutexes(mutexName) then
              break;
!         Result := MsgBox(closeMsg, mbConfirmation, MB_RETRYCANCEL) = idRetry;
!     end;
! end;
! 
! function InitializeSetup(): Boolean;
! var
!   closeit: String;
! begin
!     // Check if Outlook is running.
!     closeit:= 'You must close Outlook before SpamBayes can be installed.' + #13 + #13 +
                'Please close all Outlook Windows (using "File->Exit and Log off"' + #13 +
                'if available) and click Retry, or click Cancel to exit the installation.'+ #13 + #13 +
                'If this message persists after closing all Outlook windows, you may' + #13 +
!               'need to log off from Windows, and try again.'
!     Result := CheckNoAppMutex('_outlook_mutex_', closeit);
!     // Check if MAPISP32.EXE is running - if it is, it implies something is screwey
!     // with Outlook.
!     if Result then begin
!       closeit := 'The Outlook mail delivery agent is still running.' + #13 + #13 +
!                  'If you only recently closed Outlook, wait a few seconds and click Retry' + #13 + #13 +
!                  'If this message persists, you may need to log off from Windows, and try again'
!       Result := CheckNoAppMutex('InternetMailTransport', closeit);
!     end;
!     // And finally, the SpamBayes server
!     if Result then begin
!       // Tell them to 'Stop' then 'Exit', so any services are also stopped
!       closeit:= 'An existing SpamBayes server is already running.' + #13 + #13 +
!                 'Please shutdown this server before installing.  If the SpamBayes tray icon' + #13 +
!                 'is running, Right-click it and select "Exit SpamBayes"' + #13 +
!                 'If the Windows Service version of SpamBayes is running, please stop' + #13 +
!                 'it via "Control Panel->Administrative Tools->Services"' + #13 + #13
!                 'If this message persists, you may need to restart Windows'
!       Result := CheckNoAppMutex('SpamBayesServer', closeit);
      end;
+ 
      // default our install type.
      if IsOutlookInstalled() then begin



From mhammond at users.sourceforge.net  Mon Dec 22 22:20:32 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Mon Dec 22 22:20:34 2003
Subject: [Spambayes-checkins] spambayes/windows/py2exe setup_all.py, 1.9,
	1.10
Message-ID: <E1AYd60-0002nN-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/windows/py2exe
In directory sc8-pr-cvs1:/tmp/cvs-serv10586

Modified Files:
	setup_all.py 
Log Message:
Install sbicon.ico, and the images etc referenced by the sb_server readme.
Add a little 'outlook register' utility.


Index: setup_all.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/windows/py2exe/setup_all.py,v
retrieving revision 1.9
retrieving revision 1.10
diff -C2 -d -r1.9 -r1.10
*** setup_all.py	22 Oct 2003 01:02:00 -0000	1.9
--- setup_all.py	23 Dec 2003 03:20:28 -0000	1.10
***************
*** 66,74 ****
      script = os.path.join(sb_top_dir, r"Outlook2000\sandbox\dump_props.py"),
      dest_base = "bin/outlook_dump_props",
  )
  
  service = dict(
      dest_base = "bin/sb_service",
!     modules = ["pop3proxy_service"]
  )
  sb_server = dict(
--- 66,93 ----
      script = os.path.join(sb_top_dir, r"Outlook2000\sandbox\dump_props.py"),
      dest_base = "bin/outlook_dump_props",
+     icon_resources = [(100,  os.path.join(sb_top_dir,
+                                           r"windows\resources\sbicon.ico")),
+                      ],
+ )
+ 
+ # A "register" utility for Outlook.  This should not be necessary, as
+ # 'regsvr32 dllname' does exactly the same thing.  However, Inno Setup
+ # version 4 appears to, upon uninstall, do something that prevents the
+ # files used by the unregister process to be deleted.  Unregistering via
+ # this EXE solves the problem.
+ outlook_addin_register = dict(
+     script = os.path.join(sb_top_dir, r"Outlook2000\addin.py"),
+     dest_base = "bin/outlook_addin_register",
+     icon_resources = [(100,  os.path.join(sb_top_dir,
+                                           r"windows\resources\sbicon.ico")),
+                      ],
  )
  
  service = dict(
      dest_base = "bin/sb_service",
!     modules = ["pop3proxy_service"],
!     icon_resources = [(100,  os.path.join(sb_top_dir,
!                                           r"windows\resources\sbicon.ico")),
!                      ],
  )
  sb_server = dict(
***************
*** 83,87 ****
      dest_base = "bin/sb_tray",
      script = os.path.join(sb_top_dir, "windows", "pop3proxy_tray.py"),
!     icon_resources = [(1000, os.path.join(sb_top_dir, r"windows\resources\sb-started.ico")),
                        (1010, os.path.join(sb_top_dir, r"windows\resources\sb-stopped.ico"))],
  )
--- 102,107 ----
      dest_base = "bin/sb_tray",
      script = os.path.join(sb_top_dir, "windows", "pop3proxy_tray.py"),
!     icon_resources = [(100,  os.path.join(sb_top_dir, r"windows\resources\sbicon.ico")),
!                       (1000, os.path.join(sb_top_dir, r"windows\resources\sb-started.ico")),
                        (1010, os.path.join(sb_top_dir, r"windows\resources\sb-stopped.ico"))],
  )
***************
*** 95,104 ****
  proxy_data_files = [
      ["docs/sb_server", [os.path.join(sb_top_dir, r"windows\readme_proxy.html")]],
  ]
  
  # Default and only distutils command is "py2exe" - save adding it to the
  # command line every single time.
! if len(sys.argv)==1:
!     sys.argv = [sys.argv[0], "py2exe"]
  
  setup(name="SpamBayes",
--- 115,132 ----
  proxy_data_files = [
      ["docs/sb_server", [os.path.join(sb_top_dir, r"windows\readme_proxy.html")]],
+     # this is hacky - the readme uses the Outlook data files.  We need to
+     # consolidate the documents.
+     ["docs/sb_server/docs/images", glob.glob(os.path.join(sb_top_dir, r"Outlook2000\docs\images\*.jpg"))],
+ ]
+ 
+ common_data_files = [
+     ["", [os.path.join(sb_top_dir, r"windows\resources\sbicon.ico")]],
  ]
  
  # Default and only distutils command is "py2exe" - save adding it to the
  # command line every single time.
! if len(sys.argv)==1 or \
!    (len(sys.argv)==2 and sys.argv[1] in ['-q', '-n']):
!     sys.argv.append("py2exe")
  
  setup(name="SpamBayes",
***************
*** 111,117 ****
        console=[sb_server, sb_upload, outlook_dump_props],
        # The taskbar
!       windows=[pop3proxy_tray],
        # and the misc data files
!       data_files = outlook_data_files + proxy_data_files,
        options = {"py2exe" : py2exe_options},
        zipfile = "lib/spambayes.zip",
--- 139,145 ----
        console=[sb_server, sb_upload, outlook_dump_props],
        # The taskbar
!       windows=[pop3proxy_tray, outlook_addin_register],
        # and the misc data files
!       data_files = outlook_data_files + proxy_data_files + common_data_files,
        options = {"py2exe" : py2exe_options},
        zipfile = "lib/spambayes.zip",



From mhammond at users.sourceforge.net  Tue Dec 23 01:00:16 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Tue Dec 23 01:00:43 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000/dialogs/resources
	dialogs.rc, 1.43, 1.44
Message-ID: <E1AYfaa-0000Fv-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000/dialogs/resources
In directory sc8-pr-cvs1:/tmp/cvs-serv729

Modified Files:
	dialogs.rc 
Log Message:
Another attempt at [ 830865 ] No dialog context-help button on Win9x - 
dialogs must not use a "modal frame" on Win9x for this to work apparently!


Index: dialogs.rc
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/dialogs/resources/dialogs.rc,v
retrieving revision 1.43
retrieving revision 1.44
diff -C2 -d -r1.43 -r1.44
*** dialogs.rc	19 Dec 2003 12:42:20 -0000	1.43
--- dialogs.rc	23 Dec 2003 06:00:13 -0000	1.44
***************
*** 58,62 ****
  
  IDD_MANAGER DIALOGEX 0, 0, 275, 260
! STYLE DS_SETFONT | DS_MODALFRAME | WS_POPUP | WS_CAPTION | WS_SYSMENU
  EXSTYLE WS_EX_CONTEXTHELP
  CAPTION "SpamBayes"
--- 58,62 ----
  
  IDD_MANAGER DIALOGEX 0, 0, 275, 260
! STYLE DS_SETFONT | WS_POPUP | WS_CAPTION | WS_SYSMENU
  EXSTYLE WS_EX_CONTEXTHELP
  CAPTION "SpamBayes"
***************
*** 121,125 ****
  
  IDD_DIAGNOSTIC DIALOGEX 0, 0, 183, 98
! STYLE DS_SETFONT | DS_MODALFRAME | WS_POPUP | WS_CAPTION | WS_SYSMENU
  EXSTYLE WS_EX_CONTEXTHELP
  CAPTION "Diagnostics"
--- 121,125 ----
  
  IDD_DIAGNOSTIC DIALOGEX 0, 0, 183, 98
! STYLE DS_SETFONT | WS_POPUP | WS_CAPTION | WS_SYSMENU
  EXSTYLE WS_EX_CONTEXTHELP
  CAPTION "Diagnostics"
***************
*** 553,557 ****
  
  IDD_FILTER_NOW DIALOGEX 0, 0, 244, 185
! STYLE DS_SETFONT | DS_MODALFRAME | WS_POPUP | WS_CAPTION | WS_SYSMENU
  EXSTYLE WS_EX_CONTEXTHELP
  CAPTION "Filter Now"
--- 553,557 ----
  
  IDD_FILTER_NOW DIALOGEX 0, 0, 244, 185
! STYLE DS_SETFONT | WS_POPUP | WS_CAPTION | WS_SYSMENU
  EXSTYLE WS_EX_CONTEXTHELP
  CAPTION "Filter Now"



From mhammond at users.sourceforge.net  Tue Dec 23 01:04:31 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Tue Dec 23 01:04:33 2003
Subject: [Spambayes-checkins] spambayes/windows spambayes.iss,1.4,1.5
Message-ID: <E1AYfeh-0000Ry-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/windows
In directory sc8-pr-cvs1:/tmp/cvs-serv1557

Modified Files:
	spambayes.iss 
Log Message:
Show the Outlook welcome html doc, and update version accordingly


Index: spambayes.iss
===================================================================
RCS file: /cvsroot/spambayes/spambayes/windows/spambayes.iss,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** spambayes.iss	23 Dec 2003 03:19:27 -0000	1.4
--- spambayes.iss	23 Dec 2003 06:04:28 -0000	1.5
***************
*** 5,10 ****
  [Setup]
  AppName=SpamBayes
! AppVerName=SpamBayes 1.1a1
! AppVersion=1.1a1
  DefaultDirName={pf}\SpamBayes
  DefaultGroupName=SpamBayes
--- 5,10 ----
  [Setup]
  AppName=SpamBayes
! AppVerName=SpamBayes Experimental Binary
! AppVersion=0.85
  DefaultDirName={pf}\SpamBayes
  DefaultGroupName=SpamBayes
***************
*** 26,29 ****
--- 26,30 ----
  Source: "py2exe\dist\bin\outlook_dump_props.exe"; DestDir: "{app}\bin"; Check: InstallingOutlook; Flags: ignoreversion
  Source: "py2exe\dist\docs\outlook\*.*"; DestDir: "{app}\docs\outlook"; Check: InstallingOutlook; Flags: ignoreversion recursesubdirs
+ Source: "py2exe\dist\docs\outlook\docs\welcome.html"; DestDir: "{app}\docs\outlook\docs"; Check: InstallingOutlook; Flags: isreadme
  
  Source: "py2exe\dist\bin\sb_server.exe"; DestDir: "{app}\bin"; Check: InstallingProxy; Flags: ignoreversion



From mhammond at users.sourceforge.net  Tue Dec 23 01:05:23 2003
From: mhammond at users.sourceforge.net (Mark Hammond)
Date: Tue Dec 23 01:05:27 2003
Subject: [Spambayes-checkins] spambayes/windows/py2exe setup_all.py, 1.10,
	1.11
Message-ID: <E1AYffX-0000Tv-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/windows/py2exe
In directory sc8-pr-cvs1:/tmp/cvs-serv1757

Modified Files:
	setup_all.py 
Log Message:
default_bayes_customize.ini was going into the wrong directory.


Index: setup_all.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/windows/py2exe/setup_all.py,v
retrieving revision 1.10
retrieving revision 1.11
diff -C2 -d -r1.10 -r1.11
*** setup_all.py	23 Dec 2003 03:20:28 -0000	1.10
--- setup_all.py	23 Dec 2003 06:05:20 -0000	1.11
***************
*** 111,115 ****
      ["docs/outlook/docs", glob.glob(os.path.join(sb_top_dir, r"Outlook2000\docs\*.html"))],
      ["docs/outlook/docs/images", glob.glob(os.path.join(sb_top_dir, r"Outlook2000\docs\images\*.jpg"))],
!     ["docs/outlook", [os.path.join(sb_top_dir, r"Outlook2000\default_bayes_customize.ini")]],
  ]
  proxy_data_files = [
--- 111,115 ----
      ["docs/outlook/docs", glob.glob(os.path.join(sb_top_dir, r"Outlook2000\docs\*.html"))],
      ["docs/outlook/docs/images", glob.glob(os.path.join(sb_top_dir, r"Outlook2000\docs\images\*.jpg"))],
!     ["bin", [os.path.join(sb_top_dir, r"Outlook2000\default_bayes_customize.ini")]],
  ]
  proxy_data_files = [



From montanaro at users.sourceforge.net  Tue Dec 23 23:08:40 2003
From: montanaro at users.sourceforge.net (Skip Montanaro)
Date: Tue Dec 23 23:08:45 2003
Subject: [Spambayes-checkins] spambayes/contrib spamcounts.py,1.4,1.5
Message-ID: <E1AZ0K8-0006ZJ-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/contrib
In directory sc8-pr-cvs1:/tmp/cvs-serv25027/contrib

Modified Files:
	spamcounts.py 
Log Message:
make state key a manifest constant - should have done this ages ago...


Index: spamcounts.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/contrib/spamcounts.py,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** spamcounts.py	12 Dec 2003 20:21:15 -0000	1.4
--- spamcounts.py	24 Dec 2003 04:08:38 -0000	1.5
***************
*** 27,30 ****
--- 27,31 ----
  from spambayes.Options import options
  from spambayes.tokenizer import tokenize
+ from spambayes.storage import STATE_KEY
  
  prog = sys.argv[0]
***************
*** 61,65 ****
      S = options["Classifier", "unknown_word_strength"]
      x = options["Classifier", "unknown_word_prob"]
!     _, ns, nh = db["saved state"]
  
      writer = csv.writer(sys.stdout)
--- 62,66 ----
      S = options["Classifier", "unknown_word_strength"]
      x = options["Classifier", "unknown_word_prob"]
!     _, ns, nh = db[STATE_KEY]
  
      writer = csv.writer(sys.stdout)



From montanaro at users.sourceforge.net  Tue Dec 23 23:08:41 2003
From: montanaro at users.sourceforge.net (Skip Montanaro)
Date: Tue Dec 23 23:08:47 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 tester.py,1.22,1.23
Message-ID: <E1AZ0K8-0006ZL-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv25027/Outlook2000

Modified Files:
	tester.py 
Log Message:
make state key a manifest constant - should have done this ages ago...


Index: tester.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/tester.py,v
retrieving revision 1.22
retrieving revision 1.23
diff -C2 -d -r1.22 -r1.23
*** tester.py	19 Dec 2003 06:28:55 -0000	1.22
--- tester.py	24 Dec 2003 04:08:38 -0000	1.23
***************
*** 19,22 ****
--- 19,24 ----
  import threading
  
+ from spambayes.storage import STATE_KEY
+ 
  import msgstore
  
***************
*** 71,75 ****
          bsddb_error = bsddb.error
      key = bayes.dbm.first()[0]
!     if key not in ["saved state"]:
          yield key, bayes._wordinfoget(key)
      while True:
--- 73,77 ----
          bsddb_error = bsddb.error
      key = bayes.dbm.first()[0]
!     if key != STATE_KEY:
          yield key, bayes._wordinfoget(key)
      while True:
***************
*** 80,84 ****
          except bsddb_error:
              break
!         if key not in ["saved state"]:
              yield key, bayes._wordinfoget(key)
  
--- 82,86 ----
          except bsddb_error:
              break
!         if key != STATE_KEY:
              yield key, bayes._wordinfoget(key)
  



From montanaro at users.sourceforge.net  Tue Dec 23 23:08:41 2003
From: montanaro at users.sourceforge.net (Skip Montanaro)
Date: Tue Dec 23 23:08:49 2003
Subject: [Spambayes-checkins] spambayes/spambayes storage.py,1.37,1.38
Message-ID: <E1AZ0K9-0006ZQ-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv25027/spambayes

Modified Files:
	storage.py 
Log Message:
make state key a manifest constant - should have done this ages ago...


Index: storage.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/storage.py,v
retrieving revision 1.37
retrieving revision 1.38
diff -C2 -d -r1.37 -r1.38
*** storage.py	16 Dec 2003 05:06:34 -0000	1.37
--- storage.py	24 Dec 2003 04:08:38 -0000	1.38
***************
*** 150,153 ****
--- 150,155 ----
  WORD_CHANGED = "C"
  
+ STATE_KEY = 'saved state'
+ 
  class DBDictClassifier(classifier.Classifier):
      '''Classifier object persisted in a caching database'''
***************
*** 157,161 ****
  
          classifier.Classifier.__init__(self)
!         self.statekey = "saved state"
          self.mode = mode
          self.db_name = db_name
--- 159,163 ----
  
          classifier.Classifier.__init__(self)
!         self.statekey = STATE_KEY
          self.mode = mode
          self.db_name = db_name
***************
*** 311,315 ****
  
          classifier.Classifier.__init__(self)
!         self.statekey = "saved state"
          self.db_name = db_name
          self.load()
--- 313,317 ----
  
          classifier.Classifier.__init__(self)
!         self.statekey = STATE_KEY
          self.db_name = db_name
          self.load()



From tim_one at users.sourceforge.net  Wed Dec 24 12:16:41 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Wed Dec 24 12:16:45 2003
Subject: [Spambayes-checkins] spambayes/spambayes/test test_storage.py, 1.4,
	1.5
Message-ID: <E1AZCcj-0001PQ-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes/test
In directory sc8-pr-cvs1:/tmp/cvs-serv5384/spambayes/test

Modified Files:
	test_storage.py 
Log Message:
Removed decoys (overly convoluted spellings; unnecessary methods).


Index: test_storage.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/test/test_storage.py,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** test_storage.py	3 Dec 2003 00:35:16 -0000	1.4
--- test_storage.py	24 Dec 2003 17:16:38 -0000	1.5
***************
*** 2,9 ****
  
  import unittest, os, sys
! try:
!     import cStringIO as StringIO
! except ImportError:
!     import StringIO
  
  import sb_test_support
--- 2,7 ----
  
  import unittest, os, sys
! import tempfile
! import cStringIO as StringIO
  
  import sb_test_support
***************
*** 13,22 ****
  
  class _StorageTestBase(unittest.TestCase):
      StorageClass = None
  
      def setUp(self):
-         import tempfile
          self.db_name = tempfile.mktemp("spambayestest")
!         self.classifier = self.__class__.StorageClass(self.db_name)
  
      def tearDown(self):
--- 11,20 ----
  
  class _StorageTestBase(unittest.TestCase):
+     # Subclass must define a concrete StorageClass.
      StorageClass = None
  
      def setUp(self):
          self.db_name = tempfile.mktemp("spambayestest")
!         self.classifier = self.StorageClass(self.db_name)
  
      def tearDown(self):
***************
*** 29,33 ****
          info = self.classifier._wordinfoget(word)
          if info is None:
!             if expected_ham==0 and expected_spam==0:
                  return
              self.fail("_CheckWordCounts for '%s' got None!")
--- 27,31 ----
          info = self.classifier._wordinfoget(word)
          if info is None:
!             if expected_ham == expected_spam == 0:
                  return
              self.fail("_CheckWordCounts for '%s' got None!")
***************
*** 129,178 ****
  class PickleStorageTestCase(_StorageTestBase):
      StorageClass = PickledClassifier
-     def setUp(self):
-         return _StorageTestBase.setUp(self)
  
  class DBStorageTestCase(_StorageTestBase):
      StorageClass = DBDictClassifier
!     def setUp(self):
!         return _StorageTestBase.setUp(self)
      def tearDown(self):
          self.classifier.db.close()
          _StorageTestBase.tearDown(self)
  
!     def fail_open_best(self, *args):
          from spambayes import dbmstorage
          raise dbmstorage.error("No dbm modules available!")
!     def success(self, *args):
!         self.succeeded = True
      def testNoDBMAvailable(self):
          import tempfile
          from spambayes.storage import open_storage
          DBDictClassifier_load = DBDictClassifier.load
!         DBDictClassifier.load = self.fail_open_best
!         sys_exit = sys.exit
!         sys.exit = self.success
!         # redirect sys.stderr and sys.exit, as storage.py prints
!         # an error to stderr, then attempts to sys.exit
!         # (we probably could just catch SystemExit for sys.exit?)
          sys_stderr = sys.stderr
          sys.stderr = StringIO.StringIO()
-         self.succeeded = False
-         db_name = tempfile.mktemp("nodbmtest")
          try:
!             s = open_storage(db_name, True)
          finally:
              DBDictClassifier.load = DBDictClassifier_load
-             sys.exit = sys_exit
              sys.stderr = sys_stderr
!         if not self.succeeded:
!             self.fail()
          if os.path.isfile(db_name):
              os.remove(db_name)
  
  def suite():
-     # We dont want our base class run
      suite = unittest.TestSuite()
!     suite.addTest(unittest.makeSuite(PickleStorageTestCase))
!     suite.addTest(unittest.makeSuite(DBStorageTestCase))
      return suite
  
--- 127,173 ----
  class PickleStorageTestCase(_StorageTestBase):
      StorageClass = PickledClassifier
  
  class DBStorageTestCase(_StorageTestBase):
      StorageClass = DBDictClassifier
! 
      def tearDown(self):
          self.classifier.db.close()
          _StorageTestBase.tearDown(self)
  
!     def _fail_open_best(self, *args):
          from spambayes import dbmstorage
          raise dbmstorage.error("No dbm modules available!")
! 
      def testNoDBMAvailable(self):
          import tempfile
          from spambayes.storage import open_storage
+ 
+         db_name = tempfile.mktemp("nodbmtest")
          DBDictClassifier_load = DBDictClassifier.load
!         DBDictClassifier.load = self._fail_open_best
!         # Redirect sys.stderr, as open_storage() prints a msg to stderr.
!         # Then it does sys.exit(), which we catch.
          sys_stderr = sys.stderr
          sys.stderr = StringIO.StringIO()
          try:
!             try:
!                 open_storage(db_name, True)
!             except SystemExit:
!                 pass
!             else:
!                 self.fail("expected SystemExit from open_storage() call")
          finally:
              DBDictClassifier.load = DBDictClassifier_load
              sys.stderr = sys_stderr
! 
          if os.path.isfile(db_name):
              os.remove(db_name)
  
  def suite():
      suite = unittest.TestSuite()
!     for cls in (PickleStorageTestCase,
!                 DBStorageTestCase,
!                ):
!         suite.addTest(unittest.makeSuite(cls))
      return suite
  



From tim_one at users.sourceforge.net  Thu Dec 25 13:34:12 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Thu Dec 25 13:34:15 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 export.py,1.6,1.7
Message-ID: <E1AZaJI-0002yu-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv11414

Modified Files:
	export.py 
Log Message:
Brought the usage blurb into line with what the code does.


Index: export.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/export.py,v
retrieving revision 1.6
retrieving revision 1.7
diff -C2 -d -r1.6 -r1.7
*** export.py	3 May 2003 12:21:59 -0000	1.6
--- export.py	25 Dec 2003 18:34:10 -0000	1.7
***************
*** 117,128 ****
  def usage():
      print """ \
! Usage: %s -q [directory]
  
  -q : quiet - don't prompt for confirmation.
  -n : Minimum number of files to aim for in each directory, default=%d
  
! Export the folders defined in the Outlook Plugin to a test directory.
! The directory structure is as defined in the parent README.txt file,
! in the "Standard Test Data Setup" section.
  
  If 'directory' is not specified, '%s' is assumed.
--- 117,128 ----
  def usage():
      print """ \
! Usage: %s [-q] [-n min] [directory]
  
  -q : quiet - don't prompt for confirmation.
  -n : Minimum number of files to aim for in each directory, default=%d
  
! Export Spam and Ham training folders defined in the Outlook Plugin to a
! test directory.  The directory structure is as defined in the parent
! README.txt file, in the "Standard Test Data Setup" section.
  
  If 'directory' is not specified, '%s' is assumed.



From tim_one at users.sourceforge.net  Thu Dec 25 13:40:24 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Thu Dec 25 13:40:28 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 export.py,1.7,1.8
Message-ID: <E1AZaPI-00038m-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv11978

Modified Files:
	export.py 
Log Message:
Made module-global data names uppercase.


Index: export.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/export.py,v
retrieving revision 1.7
retrieving revision 1.8
diff -C2 -d -r1.7 -r1.8
*** export.py	25 Dec 2003 18:34:10 -0000	1.7
--- export.py	25 Dec 2003 18:40:22 -0000	1.8
***************
*** 4,9 ****
  from manager import GetManager
  
! files_per_directory = 400
! default_directory = "..\\testtools\\Data"
  
  def BuildBuckets(manager):
--- 4,9 ----
  from manager import GetManager
  
! FILES_PER_DIRECTORY = 400
! DEFAULT_DIRECTORY = "..\\testtools\\Data"
  
  def BuildBuckets(manager):
***************
*** 17,21 ****
          for msg in folder.GetMessageGenerator():
              num_ham += 1
!     num_buckets = min(num_ham, num_spam)/ files_per_directory
      dirs = []
      for i in range(num_buckets):
--- 17,21 ----
          for msg in folder.GetMessageGenerator():
              num_ham += 1
!     num_buckets = min(num_ham, num_spam)/ FILES_PER_DIRECTORY
      dirs = []
      for i in range(num_buckets):
***************
*** 79,84 ****
  
  def main():
      import getopt
!     try:
          opts, args = getopt.getopt(sys.argv[1:], "qn:")
      except getopt.error, d:
--- 79,86 ----
  
  def main():
+     global FILES_PER_DIRECTORY
      import getopt
! 
!    try:
          opts, args = getopt.getopt(sys.argv[1:], "qn:")
      except getopt.error, d:
***************
*** 91,96 ****
              quiet = 1
          elif opt=='-n':
!             global files_per_directory
!             files_per_directory = int(val)
  
      if len(args) > 1:
--- 93,97 ----
              quiet = 1
          elif opt=='-n':
!             FILES_PER_DIRECTORY = int(val)
  
      if len(args) > 1:
***************
*** 100,104 ****
  
      if len(args)==0:
!         directory = os.path.join(os.path.dirname(sys.argv[0]), default_directory)
      else:
          directory = args[0]
--- 101,105 ----
  
      if len(args)==0:
!         directory = os.path.join(os.path.dirname(sys.argv[0]), DEFAULT_DIRECTORY)
      else:
          directory = args[0]
***************
*** 130,134 ****
  If 'directory' exists, it will be recursively deleted before
  the export (but you will be asked to confirm unless -q is given).""" \
!             % (os.path.basename(sys.argv[0]), files_per_directory, default_directory)
      sys.exit(1)
  
--- 131,135 ----
  If 'directory' exists, it will be recursively deleted before
  the export (but you will be asked to confirm unless -q is given).""" \
!             % (os.path.basename(sys.argv[0]), FILES_PER_DIRECTORY, DEFAULT_DIRECTORY)
      sys.exit(1)
  



From tim_one at users.sourceforge.net  Thu Dec 25 13:41:31 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Thu Dec 25 13:41:35 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 export.py,1.8,1.9
Message-ID: <E1AZaQN-0003Af-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv12163

Modified Files:
	export.py 
Log Message:
Ooops -- deleted a space by accident.


Index: export.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/export.py,v
retrieving revision 1.8
retrieving revision 1.9
diff -C2 -d -r1.8 -r1.9
*** export.py	25 Dec 2003 18:40:22 -0000	1.8
--- export.py	25 Dec 2003 18:41:29 -0000	1.9
***************
*** 82,86 ****
      import getopt
  
!    try:
          opts, args = getopt.getopt(sys.argv[1:], "qn:")
      except getopt.error, d:
--- 82,86 ----
      import getopt
  
!     try:
          opts, args = getopt.getopt(sys.argv[1:], "qn:")
      except getopt.error, d:



From tim_one at users.sourceforge.net  Thu Dec 25 13:49:49 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Thu Dec 25 13:49:52 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 export.py,1.9,1.10
Message-ID: <E1AZaYP-0003bo-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv13759

Modified Files:
	export.py 
Log Message:
Moved common usage()-related code into usage(); add an -h option (just
print usage info).


Index: export.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/export.py,v
retrieving revision 1.9
retrieving revision 1.10
diff -C2 -d -r1.9 -r1.10
*** export.py	25 Dec 2003 18:41:29 -0000	1.9
--- export.py	25 Dec 2003 18:49:47 -0000	1.10
***************
*** 83,102 ****
  
      try:
!         opts, args = getopt.getopt(sys.argv[1:], "qn:")
      except getopt.error, d:
!         print d
!         print
!         usage()
      quiet = 0
      for opt, val in opts:
!         if opt=='-q':
              quiet = 1
!         elif opt=='-n':
              FILES_PER_DIRECTORY = int(val)
  
      if len(args) > 1:
!         print "Only one directory name can be specified"
!         print
!         usage()
  
      if len(args)==0:
--- 83,102 ----
  
      try:
!         opts, args = getopt.getopt(sys.argv[1:], "hqn:")
      except getopt.error, d:
!         usage(d)
      quiet = 0
      for opt, val in opts:
!         if opt == '-h':
!             usage()
!         elif opt == '-q':
              quiet = 1
!         elif opt == '-n':
              FILES_PER_DIRECTORY = int(val)
+         else:
+             assert 0, "internal error on option '%s'" % opt
  
      if len(args) > 1:
!         usage("Only one directory name can be specified.")
  
      if len(args)==0:
***************
*** 116,123 ****
      export(directory)
  
! def usage():
      print """ \
! Usage: %s [-q] [-n min] [directory]
  
  -q : quiet - don't prompt for confirmation.
  -n : Minimum number of files to aim for in each directory, default=%d
--- 116,130 ----
      export(directory)
  
! # Display errormsg (if specified), a blank line, and usage information; then
! # exit with status 1 (usage doesn't return).
! def usage(errormsg=None):
!     if errormsg:
!         print str(errormsg)
!         print
! 
      print """ \
! Usage: %s [-h] [-q] [-n min] [directory]
  
+ -h : help - display this msg and stop
  -q : quiet - don't prompt for confirmation.
  -n : Minimum number of files to aim for in each directory, default=%d



From tim_one at users.sourceforge.net  Thu Dec 25 13:57:58 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Thu Dec 25 13:58:01 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 export.py,1.10,1.11
Message-ID: <E1AZagI-0003tj-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv14975

Modified Files:
	export.py 
Log Message:
Folded long lines.


Index: export.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/export.py,v
retrieving revision 1.10
retrieving revision 1.11
diff -C2 -d -r1.10 -r1.11
*** export.py	25 Dec 2003 18:49:47 -0000	1.10
--- export.py	25 Dec 2003 18:57:56 -0000	1.11
***************
*** 11,18 ****
      config = manager.config
      num_ham = num_spam = 0
!     for folder in store.GetFolderGenerator(config.training.spam_folder_ids, config.training.spam_include_sub):
          for msg in folder.GetMessageGenerator():
              num_spam += 1
!     for folder in store.GetFolderGenerator(config.training.ham_folder_ids, config.training.ham_include_sub):
          for msg in folder.GetMessageGenerator():
              num_ham += 1
--- 11,20 ----
      config = manager.config
      num_ham = num_spam = 0
!     for folder in store.GetFolderGenerator(config.training.spam_folder_ids,
!                                            config.training.spam_include_sub):
          for msg in folder.GetMessageGenerator():
              num_spam += 1
!     for folder in store.GetFolderGenerator(config.training.ham_folder_ids,
!                                            config.training.ham_include_sub):
          for msg in folder.GetMessageGenerator():
              num_ham += 1
***************
*** 58,65 ****
  
      num_spam, num_ham, buckets = BuildBuckets(manager)
!     print "Have %d spam, and %d ham to export, spread over %d directories." \
!           % (num_spam, num_ham, len(buckets))
  
!     for sub in ["Spam", "Ham"]:
          if os.path.exists(os.path.join(directory, sub)):
              shutil.rmtree(os.path.join(directory, sub))
--- 60,67 ----
  
      num_spam, num_ham, buckets = BuildBuckets(manager)
!     print "Have", num_spam, "spam and", num_ham, "ham to export,",
!     print "spread over", len(buckets), "directories."
  
!     for sub in "Spam", "Ham":
          if os.path.exists(os.path.join(directory, sub)):
              shutil.rmtree(os.path.join(directory, sub))
***************
*** 69,79 ****
  
      print "Exporting spam..."
!     num = _export_folders(manager, os.path.join(directory, "Spam"), buckets,
!                           config.training.spam_folder_ids, config.training.spam_include_sub)
      print "Exported", num, "spam messages."
  
      print "Exporting ham..."
!     num = _export_folders(manager, os.path.join(directory, "Ham"), buckets,
!                           config.training.ham_folder_ids, config.training.ham_include_sub)
      print "Exported", num, "ham messages."
  
--- 71,87 ----
  
      print "Exporting spam..."
!     num = _export_folders(manager,
!                           os.path.join(directory, "Spam"),
!                           buckets,
!                           config.training.spam_folder_ids,
!                           config.training.spam_include_sub)
      print "Exported", num, "spam messages."
  
      print "Exporting ham..."
!     num = _export_folders(manager,
!                           os.path.join(directory, "Ham"),
!                           buckets,
!                           config.training.ham_folder_ids,
!                           config.training.ham_include_sub)
      print "Exported", num, "ham messages."
  
***************
*** 99,114 ****
      if len(args) > 1:
          usage("Only one directory name can be specified.")
! 
!     if len(args)==0:
!         directory = os.path.join(os.path.dirname(sys.argv[0]), DEFAULT_DIRECTORY)
!     else:
          directory = args[0]
  
      directory = os.path.abspath(directory)
      print "This program will export your Outlook Ham and Spam folders"
!     print "to the directory '%s'" % (directory,)
      if os.path.exists(directory):
          print "*******"
!         print "WARNING: all existing files in '%s' will be deleted" % (directory,)
          print "*******"
      if not quiet:
--- 107,122 ----
      if len(args) > 1:
          usage("Only one directory name can be specified.")
!     elif args:
          directory = args[0]
+     else:
+         directory = os.path.join(os.path.dirname(sys.argv[0]),
+                                  DEFAULT_DIRECTORY)
  
      directory = os.path.abspath(directory)
      print "This program will export your Outlook Ham and Spam folders"
!     print "to the directory '%s'" % directory
      if os.path.exists(directory):
          print "*******"
!         print "WARNING: all existing files in '%s' will be deleted" % directory
          print "*******"
      if not quiet:
***************
*** 138,142 ****
  If 'directory' exists, it will be recursively deleted before
  the export (but you will be asked to confirm unless -q is given).""" \
!             % (os.path.basename(sys.argv[0]), FILES_PER_DIRECTORY, DEFAULT_DIRECTORY)
      sys.exit(1)
  
--- 146,152 ----
  If 'directory' exists, it will be recursively deleted before
  the export (but you will be asked to confirm unless -q is given).""" \
!             % (os.path.basename(sys.argv[0]),
!                FILES_PER_DIRECTORY,
!                DEFAULT_DIRECTORY)
      sys.exit(1)
  



From tim_one at users.sourceforge.net  Thu Dec 25 14:09:10 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Thu Dec 25 14:09:14 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 export.py,1.11,1.12
Message-ID: <E1AZar8-0004GS-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv16339

Modified Files:
	export.py 
Log Message:
Factored out some duplicated count into new count_messages() function.


Index: export.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/export.py,v
retrieving revision 1.11
retrieving revision 1.12
diff -C2 -d -r1.11 -r1.12
*** export.py	25 Dec 2003 18:57:56 -0000	1.11
--- export.py	25 Dec 2003 19:09:08 -0000	1.12
***************
*** 7,22 ****
  DEFAULT_DIRECTORY = "..\\testtools\\Data"
  
  def BuildBuckets(manager):
      store = manager.message_store
      config = manager.config
      num_ham = num_spam = 0
      for folder in store.GetFolderGenerator(config.training.spam_folder_ids,
                                             config.training.spam_include_sub):
!         for msg in folder.GetMessageGenerator():
!             num_spam += 1
      for folder in store.GetFolderGenerator(config.training.ham_folder_ids,
                                             config.training.ham_include_sub):
!         for msg in folder.GetMessageGenerator():
!             num_ham += 1
      num_buckets = min(num_ham, num_spam)/ FILES_PER_DIRECTORY
      dirs = []
--- 7,29 ----
  DEFAULT_DIRECTORY = "..\\testtools\\Data"
  
+ # Return # of msgs in folder (a MAPIMsgStoreFolder).
+ def count_messages(folder):
+     result = 0
+     for msg in folder.GetMessageGenerator():
+         result += 1
+     return result
+ 
  def BuildBuckets(manager):
      store = manager.message_store
      config = manager.config
+ 
      num_ham = num_spam = 0
      for folder in store.GetFolderGenerator(config.training.spam_folder_ids,
                                             config.training.spam_include_sub):
!         num_spam += count_messages(folder)
      for folder in store.GetFolderGenerator(config.training.ham_folder_ids,
                                             config.training.ham_include_sub):
!         num_ham += count_messages(folder)
! 
      num_buckets = min(num_ham, num_spam)/ FILES_PER_DIRECTORY
      dirs = []
***************
*** 54,57 ****
--- 61,66 ----
      return num
  
+ # This does all the work.  'directory' is the parent directory for the
+ # generated Ham and Spam sub-folders.
  def export(directory):
      print "Loading bayes manager..."



From tim_one at users.sourceforge.net  Thu Dec 25 14:11:12 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Thu Dec 25 14:11:15 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 export.py,1.12,1.13
Message-ID: <E1AZat6-0004KT-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv16582

Modified Files:
	export.py 
Log Message:
Use the integer division operator for integer division.


Index: export.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/export.py,v
retrieving revision 1.12
retrieving revision 1.13
diff -C2 -d -r1.12 -r1.13
*** export.py	25 Dec 2003 19:09:08 -0000	1.12
--- export.py	25 Dec 2003 19:11:09 -0000	1.13
***************
*** 26,30 ****
          num_ham += count_messages(folder)
  
!     num_buckets = min(num_ham, num_spam)/ FILES_PER_DIRECTORY
      dirs = []
      for i in range(num_buckets):
--- 26,30 ----
          num_ham += count_messages(folder)
  
!     num_buckets = min(num_ham, num_spam) // FILES_PER_DIRECTORY
      dirs = []
      for i in range(num_buckets):



From tim_one at users.sourceforge.net  Thu Dec 25 14:15:29 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Thu Dec 25 14:15:37 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 export.py,1.13,1.14
Message-ID: <E1AZaxF-0004Ro-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv16891

Modified Files:
	export.py 
Log Message:
BuildBuckets:  If there aren't enough ham or spam msgs to entirely fill
one bucket, use one bucket anyway, instead of blowing up later with a
mysterious IndexError.  Also changed the computation of dirs to use a
straightforward listcomp.


Index: export.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/export.py,v
retrieving revision 1.13
retrieving revision 1.14
diff -C2 -d -r1.13 -r1.14
*** export.py	25 Dec 2003 19:11:09 -0000	1.13
--- export.py	25 Dec 2003 19:15:27 -0000	1.14
***************
*** 14,17 ****
--- 14,21 ----
      return result
  
+ # Return triple (num_spam_messages,
+ #                num_ham_messages,
+ #                ["Set1", "Set2", ...])
+ # where the list contains one entry for each bucket.
  def BuildBuckets(manager):
      store = manager.message_store
***************
*** 27,33 ****
  
      num_buckets = min(num_ham, num_spam) // FILES_PER_DIRECTORY
!     dirs = []
!     for i in range(num_buckets):
!         dirs.append("Set%d" % (i+1,))
      return num_spam, num_ham, dirs
  
--- 31,38 ----
  
      num_buckets = min(num_ham, num_spam) // FILES_PER_DIRECTORY
!     if num_buckets == 0:
!         num_buckets = 1
! 
!     dirs = ["Set%d" % i for i in range(1, num_buckets + 1)]
      return num_spam, num_ham, dirs
  



From tim_one at users.sourceforge.net  Thu Dec 25 14:21:58 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Thu Dec 25 14:22:02 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 export.py,1.14,1.15
Message-ID: <E1AZb3W-0004cY-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv17442

Modified Files:
	export.py 
Log Message:
We were generating identical useless "Unix From" lines at the start of
each exported msg, like
    From nobody Thu Dec 25 14:15:40 2003
Did no harm beyond wasting bytes, but it was confusing, and we *don't*
generate them when the Outlook plugin is scoring or training a msg.


Index: export.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/export.py,v
retrieving revision 1.14
retrieving revision 1.15
diff -C2 -d -r1.14 -r1.15
*** export.py	25 Dec 2003 19:15:27 -0000	1.14
--- export.py	25 Dec 2003 19:21:56 -0000	1.15
***************
*** 51,55 ****
              # filename is the EID.txt
              try:
!                 msg_text = str(message.GetEmailPackageObject())
              except KeyboardInterrupt:
                  raise
--- 51,57 ----
              # filename is the EID.txt
              try:
!                 # Don't use str(msg) instead -- that inserts an information-
!                 # free "Unix From" line at the top of each msg.
!                 msg_text = message.GetEmailPackageObject().as_string()
              except KeyboardInterrupt:
                  raise
***************
*** 72,76 ****
      manager = GetManager()
      config = manager.config
- 
      num_spam, num_ham, buckets = BuildBuckets(manager)
      print "Have", num_spam, "spam and", num_ham, "ham to export,",
--- 74,77 ----



From tim_one at users.sourceforge.net  Thu Dec 25 14:30:00 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Thu Dec 25 14:30:04 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 export.py,1.15,1.16
Message-ID: <E1AZbBI-0004sA-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv18615

Modified Files:
	export.py 
Log Message:
Create the "standard" reservoir directories under Ham/ and Spam/ too.
They're left empty, but creating them anyway makes life easier for
the various rebalancing scripts.


Index: export.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/export.py,v
retrieving revision 1.15
retrieving revision 1.16
diff -C2 -d -r1.15 -r1.16
*** export.py	25 Dec 2003 19:21:56 -0000	1.15
--- export.py	25 Dec 2003 19:29:58 -0000	1.16
***************
*** 81,85 ****
          if os.path.exists(os.path.join(directory, sub)):
              shutil.rmtree(os.path.join(directory, sub))
!         for b in buckets:
              d = os.path.join(directory, sub, b)
              os.makedirs(d)
--- 81,85 ----
          if os.path.exists(os.path.join(directory, sub)):
              shutil.rmtree(os.path.join(directory, sub))
!         for b in buckets + ["reservoir"]:
              d = os.path.join(directory, sub, b)
              os.makedirs(d)



From tim_one at users.sourceforge.net  Thu Dec 25 14:38:26 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Thu Dec 25 14:38:30 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 export.py,1.16,1.17
Message-ID: <E1AZbJS-0005B5-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv19697

Modified Files:
	export.py 
Log Message:
_export_folders():  "dir" is the name of a Python builtin, so my editor
"colors it funny".  Renamed that argument to "root".  Added block
comment about the purpose of this function.  Removed the choose-a-bucket
function, since it was just a call to random.choice, and it's clearer
to do that inline.


Index: export.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/export.py,v
retrieving revision 1.16
retrieving revision 1.17
diff -C2 -d -r1.16 -r1.17
*** export.py	25 Dec 2003 19:29:58 -0000	1.16
--- export.py	25 Dec 2003 19:38:24 -0000	1.17
***************
*** 37,45 ****
      return num_spam, num_ham, dirs
  
! def ChooseBucket(buckets):
!     import random
!     return random.choice(buckets)
  
- def _export_folders(manager, dir, buckets, folder_ids, include_sub):
      num = 0
      store = manager.message_store
--- 37,49 ----
      return num_spam, num_ham, dirs
  
! # Export the messages from the folders in folder_ids, as text files, into
! # the subdirectories whose names are given in buckets, under the directory
! # 'root' (which is .../Ham or .../Spam).  Each message is placed in a
! # bucket subdirectory chosen at random (among all bucket subdirectories).
! # Returns the total number of .txt files created (== the number of msgs
! # successfully exported).
! def _export_folders(manager, root, buckets, folder_ids, include_sub):
!     from random import choice
  
      num = 0
      store = manager.message_store
***************
*** 47,52 ****
          print "", folder.name
          for message in folder.GetMessageGenerator():
!             sub = ChooseBucket(buckets)
!             this_dir = os.path.join(dir, sub)
              # filename is the EID.txt
              try:
--- 51,55 ----
          print "", folder.name
          for message in folder.GetMessageGenerator():
!             this_dir = os.path.join(root,  choice(buckets))
              # filename is the EID.txt
              try:



From tim_one at users.sourceforge.net  Fri Dec 26 01:57:00 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Fri Dec 26 01:57:05 2003
Subject: [Spambayes-checkins] spambayes/utilities rebal.py,1.2,1.3
Message-ID: <E1AZlu8-0000jS-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/utilities
In directory sc8-pr-cvs1:/tmp/cvs-serv2235/utilities

Modified Files:
	rebal.py 
Log Message:
Beef up docs and comments.  Don't use "dir" as a vrbl name (it's the
name of a builtin function, so is needlessly confusing as a vrbl name).
Spell prefix as "prefix" instead of "pfx".  Always shrink lists from the
"right end", as that's a constant-time operation.  Eliminate massive
duplication of usage info strings.  Repair logic error:  when moving a
file from a Set directory to the reservoir, the file may need to be
renamed (if a file with the original name already existed in the
reservoir -- this is rare).  In that case, the code still stuck the
original name in its list of reservoir file names, which would lead to
a "file not found" error if the second phase tried to move that file out
 of the reservoir again.  migrate() now returns the base name of the moved
file, so its caller can keep track of the actual name.


Index: rebal.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/utilities/rebal.py,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** rebal.py	17 Jan 2003 06:42:53 -0000	1.2
--- rebal.py	26 Dec 2003 06:56:58 -0000	1.3
***************
*** 2,7 ****
  
  """
! rebal.py - rebalance a ham or spam directory, moving files to or from
! a reservoir directory as necessary.
  
  usage: rebal.py [ options ]
--- 2,6 ----
  
  """
! rebal.py - rebalance a ham or spam test directory
  
  usage: rebal.py [ options ]
***************
*** 9,13 ****
     -d     - dry run; display what would be moved, but don't do it [%(DRYRUN)s]
     -r res - specify an alternate reservoir [%(RESDIR)s]
!    -s set - specify an alternate Set pfx [%(SETPFX)s]
     -n num - specify number of files per Set dir desired [%(NPERDIR)s]
     -v     - tell user what's happening [%(VERBOSE)s]
--- 8,12 ----
     -d     - dry run; display what would be moved, but don't do it [%(DRYRUN)s]
     -r res - specify an alternate reservoir [%(RESDIR)s]
!    -s set - specify an alternate Set prefix [%(SETPREFIX)s]
     -n num - specify number of files per Set dir desired [%(NPERDIR)s]
     -v     - tell user what's happening [%(VERBOSE)s]
***************
*** 15,21 ****
     -c     - confirm file moves into Set directory [%(CONFIRM)s]
     -Q     - don't confirm moves; this is independent of -v/-q
  
! The script will work with a variable number of Set directories, but they
! must already exist.
  
  Example:
--- 14,30 ----
     -c     - confirm file moves into Set directory [%(CONFIRM)s]
     -Q     - don't confirm moves; this is independent of -v/-q
+    -h     - display this message and quit
  
! Moves files among the Set subdirectories and a reservoir directory as
! necessary.  You should execute this script from the directory containing your
! Data directory.  By default, the Set1, Set2, ..., and reservoir subdirectories
! under (relative path) Data/Ham/ are rebalanced; this can be changed with the
! -s argument.  The script will work with a variable number of Set directories,
! but they must already exist, and the reservoir directory must also exist.
! 
! It's recommended that you run with the -d (dry run) option first, to see what
! the script would do without actually moving any files.  If, e.g., you
! accidentally mix up spam Sets with your Ham reservoir, it could be very
! difficult to recover from that mistake.
  
  Example:
***************
*** 33,37 ****
  
      rebal.py -n 0
!     rebal.py -n 300
  
  The first run will move all files from the various Data/Ham/Set directories
--- 42,46 ----
  
      rebal.py -n 0
!     rebal.py -n 300 -Q
  
  The first run will move all files from the various Data/Ham/Set directories
***************
*** 56,97 ****
  NPERDIR = 4000
  RESDIR = 'Data/Ham/reservoir'
! SETPFX = 'Data/Ham/Set'
  VERBOSE = True
  CONFIRM = True
  DRYRUN = False
  
! def usage(msg):
!     msg = str(msg)
      if msg:
!         print >> sys.stderr, msg
!     print >> sys.stderr, """\
! usage: rebal.py [ options ]
! options:
!    -d     - dry run; display what would be moved, but don't do it [%(DRYRUN)s]
!    -r res - specify an alternate reservoir [%(RESDIR)s]
!    -s set - specify an alternate Set pfx [%(SETPFX)s]
!    -n num - specify number of files per dir [%(NPERDIR)s]
!    -v     - tell user what's happening [%(VERBOSE)s]
!    -q     - be quiet about what's happening [not %(VERBOSE)s]
!    -c     - confirm file moves into Set directory [%(CONFIRM)s]
!    -Q     - be quiet and don't confirm moves
! """ % globals()
  
- def migrate(f, dir, verbose):
-     """rename f into dir, making sure to avoid name clashes."""
      base = os.path.split(f)[-1]
!     out = os.path.join(dir, base)
      while os.path.exists(out):
          basename, ext = os.path.splitext(base)
          digits = random.randrange(100000000)
!         out = os.path.join(dir, str(digits) + ext)
      if verbose:
          print "moving", f, "to", out
      os.rename(f, out)
  
  def main(args):
      nperdir = NPERDIR
      resdir = RESDIR
!     setpfx = SETPFX
      verbose = VERBOSE
      confirm = CONFIRM
--- 65,101 ----
  NPERDIR = 4000
  RESDIR = 'Data/Ham/reservoir'
! SETPREFIX = 'Data/Ham/Set'
  VERBOSE = True
  CONFIRM = True
  DRYRUN = False
  
! def usage(msg=None):
      if msg:
!         print >> sys.stderr, str(msg)
!     print >> sys.stderr, __doc__ % globals()
! 
! def migrate(f, targetdir, verbose):
!     """Move f into targetdir, renaming if needed to avoid name clashes.
! 
!        The basename of the moved file is returned; this may not be the
!        same as the basename of f, if the file had to be renamed because
!        a file with f's basename already existed in targetdir.
!     """
  
      base = os.path.split(f)[-1]
!     out = os.path.join(targetdir, base)
      while os.path.exists(out):
          basename, ext = os.path.splitext(base)
          digits = random.randrange(100000000)
!         out = os.path.join(targetdir, str(digits) + ext)
      if verbose:
          print "moving", f, "to", out
      os.rename(f, out)
+     return os.path.split(f)[-1]
  
  def main(args):
      nperdir = NPERDIR
      resdir = RESDIR
!     setprefix = SETPREFIX
      verbose = VERBOSE
      confirm = CONFIRM
***************
*** 110,114 ****
              resdir = arg
          elif opt == "-s":
!             setpfx = arg
          elif opt == "-v":
              verbose = True
--- 114,118 ----
              resdir = arg
          elif opt == "-s":
!             setprefix = arg
          elif opt == "-v":
              verbose = True
***************
*** 122,141 ****
              dryrun = True
          elif opt == "-h":
!             usage('')
              return 0
  
      res = os.listdir(resdir)
  
!     dirs = glob.glob(setpfx+"*")
      if dirs == []:
!         print >> sys.stderr, "no directories beginning with", setpfx, "exist."
          return 1
  
      stuff = []
      n = len(res)
!     for dir in dirs:
!         fs = os.listdir(dir)
          n += len(fs)
!         stuff.append((dir, fs))
  
      if nperdir * len(dirs) > n:
--- 126,149 ----
              dryrun = True
          elif opt == "-h":
!             usage()
              return 0
+         else:
+             raise SystemError("internal error on option '%s'" % opt)
  
      res = os.listdir(resdir)
  
!     dirs = glob.glob(setprefix + "*")
      if dirs == []:
!         print >> sys.stderr, "no directories starting with", setprefix, "exist."
          return 1
  
+     # stuff <- list of (directory, files) pairs, where directory is the
+     # name of a Set subdirectory, and files is a list of files in that dir.
      stuff = []
      n = len(res)
!     for d in dirs:
!         fs = os.listdir(d)
          n += len(fs)
!         stuff.append((d, fs))
  
      if nperdir * len(dirs) > n:
***************
*** 144,149 ****
  
      # weak check against mixing ham and spam
!     if (setpfx.find("Ham") >= 0 and resdir.find("Spam") >= 0 or
!         setpfx.find("Spam") >= 0 and resdir.find("Ham") >= 0):
          yn = raw_input("Reservoir and Set dirs appear not to match. "
                         "Continue? (y/n) ")
--- 152,157 ----
  
      # weak check against mixing ham and spam
!     if ((setprefix.find("Ham") >= 0 and resdir.find("Spam") >= 0) or
!         (setprefix.find("Spam") >= 0 and resdir.find("Ham") >= 0)):
          yn = raw_input("Reservoir and Set dirs appear not to match. "
                         "Continue? (y/n) ")
***************
*** 151,183 ****
              return 1
  
!     # if necessary, migrate random files to the reservoir
!     for (dir, fs) in stuff:
!         if nperdir >= len(fs):
              continue
  
          random.shuffle(fs)
          movethese = fs[nperdir:]
          del fs[nperdir:]
          if dryrun:
!             print "would move", len(movethese), "files from", dir, \
                    "to reservoir", resdir
          else:
              for f in movethese:
!                 migrate(os.path.join(dir, f), resdir, verbose)
!         res.extend(movethese)
  
!     # randomize reservoir once so we can just bite chunks from the front
      random.shuffle(res)
  
!     # grow Set* directories from the reservoir
!     for (dir, fs) in stuff:
          if nperdir == len(fs):
              continue
  
!         movethese = res[:nperdir-len(fs)]
!         res = res[nperdir-len(fs):]
          if dryrun:
              print "would move", len(movethese), "files from reservoir", \
!                   resdir, "to", dir
          else:
              for f in movethese:
--- 159,196 ----
              return 1
  
!     # If necessary, migrate random files to the reservoir.
!     for (d, fs) in stuff:
!         if len(fs) <= nperdir:
              continue
  
+         # Retain only nperdir files, moving the rest to reservoir.
          random.shuffle(fs)
          movethese = fs[nperdir:]
          del fs[nperdir:]
          if dryrun:
!             print "would move", len(movethese), "files from", d, \
                    "to reservoir", resdir
+             res.extend(movethese)
          else:
              for f in movethese:
!                 newname = migrate(os.path.join(d, f), resdir, verbose)
!                 res.append(newname)
  
!     # Randomize reservoir once so we can just bite chunks from the end.
      random.shuffle(res)
  
!     # Grow Set* directories from the reservoir as needed.
!     for (d, fs) in stuff:
!         assert len(fs) <= nperdir
          if nperdir == len(fs):
              continue
  
!         numtomove = nperdir - len(fs)
!         assert 0 < numtomove <= len(res)
!         movethese = res[-numtomove:]
!         del res[-numtomove:]
          if dryrun:
              print "would move", len(movethese), "files from reservoir", \
!                   resdir, "to", d
          else:
              for f in movethese:
***************
*** 187,192 ****
                      if not ok.startswith('y'):
                          continue
!                 migrate(os.path.join(resdir, f), dir, verbose)
!         fs.extend(movethese)
  
      return 0
--- 200,204 ----
                      if not ok.startswith('y'):
                          continue
!                 migrate(os.path.join(resdir, f), d, verbose)
  
      return 0



From tim_one at users.sourceforge.net  Fri Dec 26 02:02:26 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Fri Dec 26 02:02:35 2003
Subject: [Spambayes-checkins] spambayes/utilities rebal.py,1.3,1.4
Message-ID: <E1AZlzO-0000wW-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/utilities
In directory sc8-pr-cvs1:/tmp/cvs-serv3555/utilities

Modified Files:
	rebal.py 
Log Message:
migrate():  return the base name of the target file, not the original.


Index: rebal.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/utilities/rebal.py,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** rebal.py	26 Dec 2003 06:56:58 -0000	1.3
--- rebal.py	26 Dec 2003 07:02:22 -0000	1.4
***************
*** 92,96 ****
          print "moving", f, "to", out
      os.rename(f, out)
!     return os.path.split(f)[-1]
  
  def main(args):
--- 92,96 ----
          print "moving", f, "to", out
      os.rename(f, out)
!     return os.path.split(out)[-1]
  
  def main(args):



From tim_one at users.sourceforge.net  Sat Dec 27 20:12:14 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Sat Dec 27 20:13:33 2003
Subject: [Spambayes-checkins] 
	spambayes/testtools incremental.HOWTO.txt, 1.4,
	1.5 sort+group.py, 1.1, 1.2
Message-ID: <E1AaPTa-0006Vr-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/testtools
In directory sc8-pr-cvs1:/tmp/cvs-serv24507

Modified Files:
	incremental.HOWTO.txt sort+group.py 
Log Message:
Sort msgs by full-precision timestamp (not just by day).  Normalize
Received time to UTC first (e.g., I have ISPs in different time zones,
and the order in which I receive msgs across all of them doesn't have
much to do with their ideas of local time).  Use email.Utils to parse
dates instead of hand-rolling our own parser (parsedate_tz knows a lot
about common real-world violations of RFC 8222, and can recover from them);
incidentally, then, the dependence on strptime is gone.  Remove line
ends from the first Received header before trying to parse it, as the
date-time field may be split across physical lines.  Preserve files'
extensions (if any) during renaming.


Index: incremental.HOWTO.txt
===================================================================
RCS file: /cvsroot/spambayes/spambayes/testtools/incremental.HOWTO.txt,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** incremental.HOWTO.txt	5 Nov 2003 13:01:44 -0000	1.4
--- incremental.HOWTO.txt	28 Dec 2003 01:12:11 -0000	1.5
***************
*** 23,33 ****
     sort+group.py for this.  sort+group.py sorts the messages
     into chronological order (by topmost Received header) and
!    then groups them by 24-hour period.
  
!    Note that you need to have either Python 2.3b1 or a *nix
!    version of Python to run the sort+group.py script.
!    Note also that this script will run through *all* the
!    files in the Data directory, not just those in Data/Ham
!    and Data/Spam.
  
  3. Distribute the corpora into multiple sets so you can do
--- 23,35 ----
     sort+group.py for this.  sort+group.py sorts the messages
     into chronological order (by topmost Received header) and
!    then groups them by 24-hour period.  The group number (0123)
!    is the number of full 24-hour periods that elapsed between
!    the time this msg was received and the time the oldest msg
!    found was received.  The id number (004556) is a unique
!    0-based ordinal across all msgs seen, with 000000 given to
!    the oldest msg found.
  
!    Note that this script will run through *all* the files in
!    the Data directory, not just those in Data/Ham and Data/Spam.
  
  3. Distribute the corpora into multiple sets so you can do
***************
*** 67,69 ****
  
  Please, somebody rewrite this file.
!  
--- 69,71 ----
  
  Please, somebody rewrite this file.
! 

Index: sort+group.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/testtools/sort+group.py,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** sort+group.py	28 Feb 2003 00:02:45 -0000	1.1
--- sort+group.py	28 Dec 2003 01:12:11 -0000	1.2
***************
*** 5,9 ****
  ### testing of chronological incremental training.
  
! """Usage: %(program)s
  
  This program has no options!  Muahahahaha!
--- 5,9 ----
  ### testing of chronological incremental training.
  
! """Usage: sort+group.py
  
  This program has no options!  Muahahahaha!
***************
*** 12,97 ****
  import sys
  import os
- import getopt
  import glob
- import re
  import time
- import filecmp
  
! program = sys.argv[0]
! loud = True
! day = 24 * 60 * 60
! dates = {}
  
! def usage(code, msg=''):
!     """Print usage message and sys.exit(code)."""
!     if msg:
!         print >> sys.stderr, msg
!         print >> sys.stderr
!     print >> sys.stderr, __doc__ % globals()
!     sys.exit(code)
  
! def bydate(name1, name2):
!     return cmp(dates[name1], dates[name2])
  
  def main():
      """Main program; parse options and go."""
  
!     global dates
!     dates = {}
!     names = []
!     date_re = re.compile(
!         r";[^0]* (\d{1,2} (?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) \d{2,4})")
!     now = time.mktime(time.strptime(time.strftime("%d %b %Y"), "%d %b %Y"))
!     if loud: print "Scanning everything"
      for name in glob.glob('Data/*/*/*'):
          if loud:
              sys.stdout.write("%-78s\r" % name)
              sys.stdout.flush()
!         fh = file(name, "rb")
!         received = ""
!         line = fh.readline()
!         while line != "\r\n" and line != "\n" and line != "":
!             if line.lower().startswith("received:"):
!                 received = line
!                 line = fh.readline()
!                 while line != "" and (line[0] == " " or line[0] == "\t"):
!                     received += line
!                     line = fh.readline()
!                 break
!             line = fh.readline()
!         fh.close()
!         # Figure out how old the message is
!         date = now
!         try:
!             log = str(received)
!             received = date_re.search(received).group(1)
!             log = "\n" + str(received)
!             date = time.mktime(time.strptime(received, "%d %b %Y"))
!         except:
!             print "Couldn't parse " + name + ":"
!             print log
!             pass
!         dates[name] = date
!         names.append(name)
!     if loud: print ""
  
!     if loud: print "Sorting"
!     names.sort(bydate)
  
!     if loud: print "Renaming first pass"
!     for name in names:
!         dir = os.path.dirname(name)
!         base = os.path.basename(name)
!         os.rename(name, os.path.join(dir, "-"+base))
  
!     if loud: print "Renaming second pass"
!     first = dates[names[0]]
!     for num in range(0, len(names)):
!         name = names[num]
!         dir = os.path.dirname(name)
!         base = os.path.basename(name)
!         group = int((dates[name] - first) // day)
!         os.rename(os.path.join(dir, "-"+base),
!                   os.path.join(dir, "%04d-%06d" % (group, num)))
  
  if __name__ == "__main__":
--- 12,104 ----
  import sys
  import os
  import glob
  import time
  
! from email.Utils import parsedate_tz, mktime_tz
  
! loud = True
! SECONDS_PER_DAY = 24 * 60 * 60
  
! # Scan the file with path fpath for its first Received header, and return
! # a UTC timestamp for the date-time it specifies.  If anything goes wrong
! # (can't find a Received header; can't parse the date), return None.
! # This is the best guess about when we received the msg.
! def get_time(fpath):
!     fh = file(fpath, 'rb')
!     lines = iter(fh)
!     # Find first Received header.
!     for line in lines:
!         if line.lower().startswith("received:"):
!             break
!     else:
!         print "\nNo Received header found."
!         fh.close()
!         return None
!     # Paste on continuation lines, if any.
!     received = line
!     for line in lines:
!         if line[0] in ' \t':
!             received += line
!         else:
!             break
!     fh.close()
!     # RFC 2822 says the date-time field must follow a semicolon at the end.
!     i = received.rfind(';')
!     if i < 0:
!         print "\n" + received
!         print "No semicolon found in Received header."
!         return None
!     # We only want the part after the semicolon.
!     datestring = received[i+1:]
!     # It may still be split across lines (like "Wed, \r\n\t22 Oct ...").
!     datestring = ' '.join(datestring.split())
!     as_tuple = parsedate_tz(datestring)
!     if as_tuple is None:
!         print "\n" + received
!         print "Couldn't parse the date: %r" % datestring
!         return None
!     return mktime_tz(as_tuple)
  
  def main():
      """Main program; parse options and go."""
  
!     from os.path import join, split
! 
!     data = []   # list of (time_received, dirname, basename) triples
!     if loud:
!         print "Scanning everything"
!     now = time.time()
      for name in glob.glob('Data/*/*/*'):
          if loud:
              sys.stdout.write("%-78s\r" % name)
              sys.stdout.flush()
!         when_received = get_time(name) or now
!         data.append((when_received,) + split(name))
  
!     if loud:
!         print ""
!         print "Sorting ..."
!     data.sort()
  
!     # First rename all the files to a form we can't produce in the end.
!     # This is to protect against name clashes in case the files are
!     # already named according to the scheme we use.
!     if loud:
!         print "Renaming first pass ..."
!     for dummy, dirname, basename in data:
!         os.rename(join(dirname, basename),
!                   join(dirname, "-" + basename))
  
!     if loud:
!         print "Renaming second pass ..."
!     earliest = data[0][0]  # timestamp of earliest msg received
!     i = 0
!     for when_received, dirname, basename in data:
!         extension = os.path.splitext(basename)[-1]
!         group = int((when_received - earliest) / SECONDS_PER_DAY)
!         newbasename = "%04d-%06d" % (group, i)
!         os.rename(join(dirname, "-" + basename),
!                   join(dirname, newbasename + extension))
!         i += 1
  
  if __name__ == "__main__":



From tim_one at users.sourceforge.net  Sat Dec 27 21:14:52 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Sat Dec 27 21:16:12 2003
Subject: [Spambayes-checkins] spambayes/utilities rebal.py,1.4,1.5
Message-ID: <E1AaQSC-0008Ho-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/utilities
In directory sc8-pr-cvs1:/tmp/cvs-serv31501

Modified Files:
	rebal.py 
Log Message:
A long time ago, this script was easy to use with a standard test setup.
Then it grew -r and -s options, apparently for non-standard test
setups (unsure), and it became both clumsy and error-prone to use with
a standard test setup then.  Screw that.  Added a new -t option, which
makes it again easy to use with a standard test setup.  -t can't be used
in the same run with -r or -s.  I intended that -r and -s still work,
but since I've never had a use for those apart from long-windedly
getting them to believe a standard test setup is in use, if I broke them
I'll never know that.  Someone who cares about those should ensure they
still work as *they* intended.


Index: rebal.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/utilities/rebal.py,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** rebal.py	26 Dec 2003 07:02:22 -0000	1.4
--- rebal.py	28 Dec 2003 02:14:50 -0000	1.5
***************
*** 7,25 ****
  options:
     -d     - dry run; display what would be moved, but don't do it [%(DRYRUN)s]
-    -r res - specify an alternate reservoir [%(RESDIR)s]
-    -s set - specify an alternate Set prefix [%(SETPREFIX)s]
     -n num - specify number of files per Set dir desired [%(NPERDIR)s]
!    -v     - tell user what's happening [%(VERBOSE)s]
!    -q     - be quiet about what's happening [not %(VERBOSE)s]
!    -c     - confirm file moves into Set directory [%(CONFIRM)s]
!    -Q     - don't confirm moves; this is independent of -v/-q
     -h     - display this message and quit
  
! Moves files among the Set subdirectories and a reservoir directory as
! necessary.  You should execute this script from the directory containing your
! Data directory.  By default, the Set1, Set2, ..., and reservoir subdirectories
! under (relative path) Data/Ham/ are rebalanced; this can be changed with the
! -s argument.  The script will work with a variable number of Set directories,
! but they must already exist, and the reservoir directory must also exist.
  
  It's recommended that you run with the -d (dry run) option first, to see what
--- 7,31 ----
  options:
     -d     - dry run; display what would be moved, but don't do it [%(DRYRUN)s]
     -n num - specify number of files per Set dir desired [%(NPERDIR)s]
!    -t     - top directory, holding Set and reservoir subdirs [%(TOPDIR)s]
! 
!    -v     - tell user what's happening; opposite of -q [%(VERBOSE)s]
!    -q     - be quiet about what's happening; opposite of -v [not %(VERBOSE)s]
! 
!    -c     - confirm file moves into Set directory; opposite of -Q [%(CONFIRM)s]
!    -Q     - don't confirm moves; opposite of -c; independent of -v/-q
! 
     -h     - display this message and quit
  
! If you have a non-standard test setup, you can use -r/-s instead of -t:
!    -r res - specify an alternate reservoir [%(RESDIR)s]
!    -s set - specify an alternate Set prefix [%(SETPREFIX)s]
! 
! Moves files randomly among the Set subdirectories and a reservoir directory to
! leave -n files in each Set directory.  By default, the Set1, Set2, ..., and
! reservoir subdirectories under (relative path) Data/Ham/ are rebalanced; this
! can be changed with the -t option.  The script will work with a variable
! number of Set directories, but they must already exist, and the reservoir
! directory must also exist.
  
  It's recommended that you run with the -d (dry run) option first, to see what
***************
*** 28,52 ****
  difficult to recover from that mistake.
  
! Example:
! 
!     rebal.py -r reservoir -s Set -n 300
! 
! This will move random files between the directory 'reservoir' and the
! various subdirectories prefixed with 'Set', making sure no more than 300
! files are left in the 'Set' directories when finished.
! 
! Example:
! 
! Suppose you want to shuffle your Set files around, winding up with 300 files
! in each one, you can execute:
! 
!     rebal.py -n 0
!     rebal.py -n 300 -Q
! 
! The first run will move all files from the various Data/Ham/Set directories
! to the Data/Ham/reservoir directory.  The second run will randomly parcel
! out 300 files to each of the Data/Ham/Set directories.
  """
  
  import os
  import sys
--- 34,66 ----
  difficult to recover from that mistake.
  
! See the module comments for examples.
  """
  
+ # Examples:
+ #
+ #    rebal.py -n 300
+ #
+ # Moves files among the Set1, Set2, ..., and reservoir directories under
+ # Data/Ham/, leaving 300 files in each Set directory.
+ #
+ #    rebal.py -t Data/Spam -n 300
+ #
+ # The same, but under Data/Spam/.
+ #
+ #    rebal.py -r reservoir -s Set -n 300
+ #
+ # The same, but under the Set1, Set2, ..., and reservoir directories
+ # in the current directory.
+ #
+ # Supposing you want to shuffle your Set files around randomly, winding up
+ # with 300 files in each one, you can execute:
+ #
+ #    rebal.py -n 0
+ #    rebal.py -n 300 -Q
+ #
+ # The first moves all files from the various Data/Ham/Set directories to the
+ # Data/Ham/reservoir directory.  The second run randomly parcels out 300 files
+ # to each of the Data/Ham/Set directories.
+ 
  import os
  import sys
***************
*** 64,69 ****
  # defaults
  NPERDIR = 4000
! RESDIR = 'Data/Ham/reservoir'
! SETPREFIX = 'Data/Ham/Set'
  VERBOSE = True
  CONFIRM = True
--- 78,84 ----
  # defaults
  NPERDIR = 4000
! TOPDIR = os.path.join('Data', 'Ham')
! RESDIR = os.path.join(TOPDIR, 'reservoir')
! SETPREFIX = os.path.join(TOPDIR, 'Set')
  VERBOSE = True
  CONFIRM = True
***************
*** 73,76 ****
--- 88,92 ----
      if msg:
          print >> sys.stderr, str(msg)
+         print >> sys.stderr
      print >> sys.stderr, __doc__ % globals()
  
***************
*** 83,87 ****
      """
  
!     base = os.path.split(f)[-1]
      out = os.path.join(targetdir, base)
      while os.path.exists(out):
--- 99,103 ----
      """
  
!     base = os.path.basename(f)
      out = os.path.join(targetdir, base)
      while os.path.exists(out):
***************
*** 92,107 ****
          print "moving", f, "to", out
      os.rename(f, out)
!     return os.path.split(out)[-1]
  
  def main(args):
      nperdir = NPERDIR
-     resdir = RESDIR
-     setprefix = SETPREFIX
      verbose = VERBOSE
      confirm = CONFIRM
      dryrun = DRYRUN
  
      try:
!         opts, args = getopt.getopt(args, "dr:s:n:vqcQh")
      except getopt.GetoptError, msg:
          usage(msg)
--- 108,122 ----
          print "moving", f, "to", out
      os.rename(f, out)
!     return os.path.basename(out)
  
  def main(args):
      nperdir = NPERDIR
      verbose = VERBOSE
      confirm = CONFIRM
      dryrun = DRYRUN
+     topdir = resdir = setprefix = None
  
      try:
!         opts, args = getopt.getopt(args, "dr:s:t:n:vqcQh")
      except getopt.GetoptError, msg:
          usage(msg)
***************
*** 111,114 ****
--- 126,131 ----
          if opt == "-n":
              nperdir = int(arg)
+         elif opt == "-t":
+             topdir = arg
          elif opt == "-r":
              resdir = arg
***************
*** 131,138 ****
              raise SystemError("internal error on option '%s'" % opt)
  
      res = os.listdir(resdir)
  
      dirs = glob.glob(setprefix + "*")
!     if dirs == []:
          print >> sys.stderr, "no directories starting with", setprefix, "exist."
          return 1
--- 148,171 ----
              raise SystemError("internal error on option '%s'" % opt)
  
+     # Derive setprefix and resdir from topdir, if the latter was given.
+     if topdir is not None:
+         if resdir is not None or setprefix is not None:
+             usage("-t can't be specified with -r or -s")
+             return -1
+         setprefix = os.path.join(topdir, "Set")
+         resdir = os.path.join(topdir, "reservoir")
+     else:
+         if setprefix is None:
+             setprefix = SETPREFIX
+         if resdir is None:
+             resdir = RESDIR
+ 
+     if not os.path.exists(resdir):
+         print >> sys.stderr, "reservoir directory %s doesn't exist" % resdir
+         return 1
      res = os.listdir(resdir)
  
      dirs = glob.glob(setprefix + "*")
!     if not dirs:
          print >> sys.stderr, "no directories starting with", setprefix, "exist."
          return 1
***************
*** 141,145 ****
      # name of a Set subdirectory, and files is a list of files in that dir.
      stuff = []
!     n = len(res)
      for d in dirs:
          fs = os.listdir(d)
--- 174,178 ----
      # name of a Set subdirectory, and files is a list of files in that dir.
      stuff = []
!     n = len(res)    # total number of all files
      for d in dirs:
          fs = os.listdir(d)



From tim_one at users.sourceforge.net  Sat Dec 27 21:58:35 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Sat Dec 27 21:59:57 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 export.py,1.17,1.18
Message-ID: <E1AaR8V-0000zD-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv3370

Modified Files:
	export.py 
Log Message:
INCOMPATIBLE CHANGE:  the -n option now gives the number of Set
subdirectories desired, instead of a number of msgs per Set subdir
"to shoot for".  If you want to run, e.g., 10-fold cross-validation,
you have to have exactly 10 Set folders, and the # of msgs per folder
is of much less importance.  Also added a note recommending to run
rebal.py afterwards.  rebal is the expert in setting up randomized
Set subdirectories, and the export.py script probably should have
stuck to just extracting msgs from Outlook.


Index: export.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/export.py,v
retrieving revision 1.17
retrieving revision 1.18
diff -C2 -d -r1.17 -r1.18
*** export.py	25 Dec 2003 19:38:24 -0000	1.17
--- export.py	28 Dec 2003 02:58:33 -0000	1.18
***************
*** 1,8 ****
! # Exports your ham and spam folders to a standard SpamBayes test directory
  
  import sys, os, shutil
  from manager import GetManager
  
! FILES_PER_DIRECTORY = 400
  DEFAULT_DIRECTORY = "..\\testtools\\Data"
  
--- 1,8 ----
! # Exports your ham and spam folders to a standard SpamBayes test directory.
  
  import sys, os, shutil
  from manager import GetManager
  
! NUM_BUCKETS = 10
  DEFAULT_DIRECTORY = "..\\testtools\\Data"
  
***************
*** 18,22 ****
  #                ["Set1", "Set2", ...])
  # where the list contains one entry for each bucket.
! def BuildBuckets(manager):
      store = manager.message_store
      config = manager.config
--- 18,22 ----
  #                ["Set1", "Set2", ...])
  # where the list contains one entry for each bucket.
! def BuildBuckets(manager, num_buckets):
      store = manager.message_store
      config = manager.config
***************
*** 30,37 ****
          num_ham += count_messages(folder)
  
-     num_buckets = min(num_ham, num_spam) // FILES_PER_DIRECTORY
-     if num_buckets == 0:
-         num_buckets = 1
- 
      dirs = ["Set%d" % i for i in range(1, num_buckets + 1)]
      return num_spam, num_ham, dirs
--- 30,33 ----
***************
*** 73,81 ****
  # This does all the work.  'directory' is the parent directory for the
  # generated Ham and Spam sub-folders.
! def export(directory):
      print "Loading bayes manager..."
      manager = GetManager()
      config = manager.config
!     num_spam, num_ham, buckets = BuildBuckets(manager)
      print "Have", num_spam, "spam and", num_ham, "ham to export,",
      print "spread over", len(buckets), "directories."
--- 69,77 ----
  # This does all the work.  'directory' is the parent directory for the
  # generated Ham and Spam sub-folders.
! def export(directory, num_buckets):
      print "Loading bayes manager..."
      manager = GetManager()
      config = manager.config
!     num_spam, num_ham, buckets = BuildBuckets(manager, num_buckets)
      print "Have", num_spam, "spam and", num_ham, "ham to export,",
      print "spread over", len(buckets), "directories."
***************
*** 105,109 ****
  
  def main():
-     global FILES_PER_DIRECTORY
      import getopt
  
--- 101,104 ----
***************
*** 113,116 ****
--- 108,112 ----
          usage(d)
      quiet = 0
+     num_buckets = NUM_BUCKETS
      for opt, val in opts:
          if opt == '-h':
***************
*** 119,123 ****
              quiet = 1
          elif opt == '-n':
!             FILES_PER_DIRECTORY = int(val)
          else:
              assert 0, "internal error on option '%s'" % opt
--- 115,119 ----
              quiet = 1
          elif opt == '-n':
!             num_buckets = int(val)
          else:
              assert 0, "internal error on option '%s'" % opt
***************
*** 131,134 ****
--- 127,133 ----
                                   DEFAULT_DIRECTORY)
  
+     if num_buckets < 1:
+         usage("-n must be at least 1.")
+ 
      directory = os.path.abspath(directory)
      print "This program will export your Outlook Ham and Spam folders"
***************
*** 140,144 ****
      if not quiet:
          raw_input("Press enter to continue, or Ctrl+C to abort.")
!     export(directory)
  
  # Display errormsg (if specified), a blank line, and usage information; then
--- 139,143 ----
      if not quiet:
          raw_input("Press enter to continue, or Ctrl+C to abort.")
!     export(directory, num_buckets)
  
  # Display errormsg (if specified), a blank line, and usage information; then
***************
*** 150,162 ****
  
      print """ \
! Usage: %s [-h] [-q] [-n min] [directory]
  
  -h : help - display this msg and stop
  -q : quiet - don't prompt for confirmation.
! -n : Minimum number of files to aim for in each directory, default=%d
  
! Export Spam and Ham training folders defined in the Outlook Plugin to a
! test directory.  The directory structure is as defined in the parent
! README.txt file, in the "Standard Test Data Setup" section.
  
  If 'directory' is not specified, '%s' is assumed.
--- 149,163 ----
  
      print """ \
! Usage: %s [-h] [-q] [-n nsets] [directory]
  
  -h : help - display this msg and stop
  -q : quiet - don't prompt for confirmation.
! -n : number of Set subdirectories in the Ham and Spam dirs, default=%d
  
! Export Spam and Ham training folders defined in the Outlook Plugin to a test
! directory.  The directory structure is as defined in the parent
! README-DEVEL.txt file, in the "Standard Test Data Setup" section.  Files are
! distributed randomly among the Set subdirectories.  You should probably use
! rebal.py afterwards to even them out.
  
  If 'directory' is not specified, '%s' is assumed.
***************
*** 165,169 ****
  the export (but you will be asked to confirm unless -q is given).""" \
              % (os.path.basename(sys.argv[0]),
!                FILES_PER_DIRECTORY,
                 DEFAULT_DIRECTORY)
      sys.exit(1)
--- 166,170 ----
  the export (but you will be asked to confirm unless -q is given).""" \
              % (os.path.basename(sys.argv[0]),
!                NUM_BUCKETS,
                 DEFAULT_DIRECTORY)
      sys.exit(1)



From tim_one at users.sourceforge.net  Sun Dec 28 19:35:22 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Sun Dec 28 19:35:26 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 export.py,1.18,1.19
Message-ID: <E1AalNS-00026x-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv7856

Modified Files:
	export.py 
Log Message:
Grrrrr.  The email package can't turn some of my spam into strings if
the MIME headers are removed.  Worm around that, with substantial pain,
but not much new cost.


Index: export.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/export.py,v
retrieving revision 1.18
retrieving revision 1.19
diff -C2 -d -r1.18 -r1.19
*** export.py	28 Dec 2003 02:58:33 -0000	1.18
--- export.py	29 Dec 2003 00:35:20 -0000	1.19
***************
*** 7,10 ****
--- 7,16 ----
  DEFAULT_DIRECTORY = "..\\testtools\\Data"
  
+ import re
+ mime_header_re = re.compile(r"""
+     ^ content- (type | transfer-encoding) : [^\n]* \n
+     ([ \t] [^\n]* \n)*  # suck up adjacent continuation lines
+ """, re.VERBOSE | re.MULTILINE | re.IGNORECASE)
+ 
  # Return # of msgs in folder (a MAPIMsgStoreFolder).
  def count_messages(folder):
***************
*** 33,36 ****
--- 39,90 ----
      return num_spam, num_ham, dirs
  
+ # Return the text of msg (a MAPIMsgStoreMsg object) as a string.
+ # There are subtleties, alas.
+ def get_text(msg):
+     email_object = msg.GetEmailPackageObject()
+     try:
+         # Don't use str(msg) instead -- that inserts an information-
+         # free "Unix From" line at the top of each msg.
+         return email_object.as_string()
+ 
+     except:
+         # Fudge.  GetEmailPackageObject() strips MIME headers by default.
+         # I'm not exactly sure why, but I have some spam with what looks to
+         # be ill-formed MIME, such that the email pkg's .as_string() (or
+         # str() -- same thing, really) gets fatally confused when the MIME
+         # headers are stripped, dying with an internal
+         #
+         #    string payload expected: <type 'list'>
+         #
+         # TypeError.  Ignore the exception and try again.
+         pass
+ 
+     # This is what our ShowClues() does, and that's never had a problem
+     # getting a string from these problem messages.
+     email_object = msg.GetEmailPackageObject(strip_mime_headers=False)
+     text = email_object.as_string()
+ 
+     # If we leave the Content-Type and Content-Transfer-Encoding headers in
+     # now, the email package can get confused when it tries to parse this
+     # string.  So, alas, strip 'em by hand.
+     i = text.find('\n\n')  # boundary between headers and body
+     if i < 0:
+         # no body
+         i = len(text) - 2
+     headers, body = text[:i+2], text[i+2:]
+     ##print 'before:\n', text
+     headers = mime_header_re.sub('', headers) # remove troublesome headers
+     text = headers + body
+     ##print 'after:\n', text
+ 
+     # A sanity check, to make sure the email pkg can still parse this mess.
+     # If it can't, it will raise some exception.  I haven't seen this
+     # happen yet.  Getting into this section is rare (less than 1% of my spam
+     # so far), so the expense doesn't bother me.
+     import email
+     email.message_from_string(text)
+ 
+     return text
+ 
  # Export the messages from the folders in folder_ids, as text files, into
  # the subdirectories whose names are given in buckets, under the directory
***************
*** 50,56 ****
              # filename is the EID.txt
              try:
!                 # Don't use str(msg) instead -- that inserts an information-
!                 # free "Unix From" line at the top of each msg.
!                 msg_text = message.GetEmailPackageObject().as_string()
              except KeyboardInterrupt:
                  raise
--- 104,108 ----
              # filename is the EID.txt
              try:
!                 msg_text = get_text(message)
              except KeyboardInterrupt:
                  raise



From tim_one at users.sourceforge.net  Sun Dec 28 19:57:49 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Sun Dec 28 19:57:53 2003
Subject: [Spambayes-checkins] spambayes/testtools mksets.py,1.1,1.2
Message-ID: <E1AaljB-0002pj-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/testtools
In directory sc8-pr-cvs1:/tmp/cvs-serv10883

Modified Files:
	mksets.py 
Log Message:
Added some comments.


Index: mksets.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/testtools/mksets.py,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** mksets.py	1 Mar 2003 05:13:08 -0000	1.1
--- mksets.py	29 Dec 2003 00:57:47 -0000	1.2
***************
*** 1,8 ****
  #! /usr/bin/env python
  
! ### Redistribute messages among the classic Data/*/Set* directories
! ### based on desired set count, desired with messages
! ### directories based from MH mailboxes ~/Mail/everything and
! ### ~/Mail/spam.
  
  """Usage: %(program)s [OPTIONS] ...
--- 1,6 ----
  #! /usr/bin/env python
  
! # Redistribute messages among the classic Data/{Ham,Spam}/Set* directories,
! # based on desired set count.
  
  """Usage: %(program)s [OPTIONS] ...
***************
*** 14,22 ****
          random number seed
      -n num
!         number of sets
      -g num
!         number of groups
      -m num
!         number of messages per {ham,spam}*group*set
  """
  
--- 12,20 ----
          random number seed
      -n num
!         number of sets; default 5
      -g num
!         max number of groups; default unlimited
      -m num
!         max number of messages per {ham,spam}*group*set; default unlimited
  """
  
***************
*** 33,39 ****
  hamdir = "Data/Ham"
  spamdir = "Data/Spam"
! nsets = 5
! ngroups = None
! nmess = None
  
  def usage(code, msg=''):
--- 31,37 ----
  hamdir = "Data/Ham"
  spamdir = "Data/Spam"
! nsets = 5               # -n
! ngroups = None          # -g
! nmess = None            # -m
  
  def usage(code, msg=''):



From anadelonbrin at users.sourceforge.net  Sun Dec 28 20:01:50 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Sun Dec 28 20:01:53 2003
Subject: [Spambayes-checkins] spambayes NEWTRICKS.txt,1.6,1.7
Message-ID: <E1Aaln4-00031M-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv11595

Modified Files:
	NEWTRICKS.txt 
Log Message:
Add comments from Skip on spambayes-dev.


Index: NEWTRICKS.txt
===================================================================
RCS file: /cvsroot/spambayes/spambayes/NEWTRICKS.txt,v
retrieving revision 1.6
retrieving revision 1.7
diff -C2 -d -r1.6 -r1.7
*** NEWTRICKS.txt	13 Oct 2003 21:44:27 -0000	1.6
--- NEWTRICKS.txt	29 Dec 2003 01:01:46 -0000	1.7
***************
*** 30,33 ****
--- 30,51 ----
    helpful to try stripping punctuation.  (Idea from Paul Sorenson)
  
+   [skip] I tried the first (eliding punctuation from words).  From a testing
+   standpoint it turns out to not be all that useful, I think for a couple
+   reasons:
+ 
+   * There are plenty of other spammy clues in such messages which are
+     sufficient to kick these messages into spam range.  Most of this stuff
+     winds up scoring at 0.95 or above for me.  If they don't score as spam
+     for you, train on a few and see how it does then.
+ 
+   * Training databases full of old-ish mail won't contain many of these
+     sorts of messages, so enabling punctuation removal won't change things
+     very much.
+ 
+   [tony] I tried Skip's patch and got basically the same results, and
+   his reasoning above sounds right for my experience, too.  OTOH, I am
+   getting more of these messages now, so my corpus is changing (they're
+   still classified as spam without this, though).
+  
  - Similarly, some letters get replaced by numbers, e.g.: "V1agra" instead of
    "Viagra".  Mapping numbers to suitable letters might help in some



From tim_one at users.sourceforge.net  Sun Dec 28 20:03:06 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Sun Dec 28 20:03:09 2003
Subject: [Spambayes-checkins] spambayes/testtools mksets.py,1.2,1.3
Message-ID: <E1AaloI-00035C-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/testtools
In directory sc8-pr-cvs1:/tmp/cvs-serv11114

Modified Files:
	mksets.py 
Log Message:
INCOMPATIBLE CHANGE:  the -s option (set random seed) is gone.
sort+group.py recently went thru a lot of work to get the messages named
according to full-precision UTC received time.  This script was ignoring
everything except the "group number", though, and going out of its way
to randomize within a group.  Since the group number is derived from a
notion of number-of-days, message order was effectively randomized within
each 24-hour period.  Now no time-order info is destroyed.


Index: mksets.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/testtools/mksets.py,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** mksets.py	29 Dec 2003 00:57:47 -0000	1.2
--- mksets.py	29 Dec 2003 01:03:04 -0000	1.3
***************
*** 9,14 ****
      -h
          show usage and exit
-     -s num
-         random number seed
      -n num
          number of sets; default 5
--- 9,12 ----
***************
*** 25,29 ****
  import glob
  import shutil
- import random
  
  program = sys.argv[0]
--- 23,26 ----
***************
*** 43,54 ****
      sys.exit(code)
  
- def bybasename(a, b):
-     return cmp(os.path.basename(a).split("-", 2)[0],
-                os.path.basename(b).split("-", 2)[0])
- 
  def distribute(dir):
      files = glob.glob(os.path.join(dir, "*", "*"))
!     random.shuffle(files)
!     files.sort(bybasename)
  
      trash = glob.glob(os.path.join(dir, "Set*"))
--- 40,46 ----
      sys.exit(code)
  
  def distribute(dir):
      files = glob.glob(os.path.join(dir, "*", "*"))
!     files.sort()
  
      trash = glob.glob(os.path.join(dir, "Set*"))
***************
*** 104,108 ****
  
      try:
!         opts, args = getopt.getopt(sys.argv[1:], 'hs:n:g:m:')
      except getopt.error, msg:
          usage(2, msg)
--- 96,100 ----
  
      try:
!         opts, args = getopt.getopt(sys.argv[1:], 'hn:g:m:')
      except getopt.error, msg:
          usage(2, msg)
***************
*** 112,117 ****
              if opt == '-h':
                  usage(0)
-             elif opt == '-s':
-                 random.seed(int(arg))
              elif opt == '-n':
                  nsets = int(arg)
--- 104,107 ----



From tim_one at users.sourceforge.net  Sun Dec 28 20:04:08 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Sun Dec 28 20:04:11 2003
Subject: [Spambayes-checkins] spambayes/testtools mksets.py,1.3,1.4
Message-ID: <E1AalpI-00037u-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/testtools
In directory sc8-pr-cvs1:/tmp/cvs-serv11934

Modified Files:
	mksets.py 
Log Message:
This wasn't checking for the absence of positional arguments unless at
least one option was specified.


Index: mksets.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/testtools/mksets.py,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** mksets.py	29 Dec 2003 01:03:04 -0000	1.3
--- mksets.py	29 Dec 2003 01:04:06 -0000	1.4
***************
*** 100,115 ****
          usage(2, msg)
  
!     if opts:
!         for opt, arg in opts:
!             if opt == '-h':
!                 usage(0)
!             elif opt == '-n':
!                 nsets = int(arg)
!             elif opt == '-g':
!                 ngroups = int(arg)
!             elif opt == '-m':
!                 nmess = int(arg)
!         if args:
!             usage(2, "Positional arguments not allowed")
  
      distribute(hamdir)
--- 100,115 ----
          usage(2, msg)
  
!     if args:
!         usage(2, "Positional arguments not allowed")
! 
!     for opt, arg in opts:
!         if opt == '-h':
!             usage(0)
!         elif opt == '-n':
!             nsets = int(arg)
!         elif opt == '-g':
!             ngroups = int(arg)
!         elif opt == '-m':
!             nmess = int(arg)
  
      distribute(hamdir)



From tim_one at users.sourceforge.net  Sun Dec 28 20:10:22 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Sun Dec 28 20:10:25 2003
Subject: [Spambayes-checkins] spambayes/testtools mksets.py,1.4,1.5
Message-ID: <E1AalvK-0003MF-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/testtools
In directory sc8-pr-cvs1:/tmp/cvs-serv12751

Modified Files:
	mksets.py 
Log Message:
Repair sorting by time received (I introduced a bug there, when removing
the by-day randomization ).


Index: mksets.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/testtools/mksets.py,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** mksets.py	29 Dec 2003 01:04:06 -0000	1.4
--- mksets.py	29 Dec 2003 01:10:20 -0000	1.5
***************
*** 42,46 ****
--- 42,51 ----
  def distribute(dir):
      files = glob.glob(os.path.join(dir, "*", "*"))
+     # Sort by time received, earliest first.  The base names must be such
+     # that sorting by basename accomplishes this; that's true if
+     # sort+group.py was run first.
+     files = [(os.path.basename(f), f) for f in files]
      files.sort()
+     files = [t[-1] for t in files]
  
      trash = glob.glob(os.path.join(dir, "Set*"))



From tim_one at users.sourceforge.net  Sun Dec 28 20:16:30 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Sun Dec 28 20:16:33 2003
Subject: [Spambayes-checkins] spambayes/testtools mksets.py,1.5,1.6
Message-ID: <E1Aam1G-0003a1-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/testtools
In directory sc8-pr-cvs1:/tmp/cvs-serv13748

Modified Files:
	mksets.py 
Log Message:
More comments, and a bit of duplicate code squashing.


Index: mksets.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/testtools/mksets.py,v
retrieving revision 1.5
retrieving revision 1.6
diff -C2 -d -r1.5 -r1.6
*** mksets.py	29 Dec 2003 01:10:20 -0000	1.5
--- mksets.py	29 Dec 2003 01:16:28 -0000	1.6
***************
*** 49,55 ****
      files = [t[-1] for t in files]
  
      trash = glob.glob(os.path.join(dir, "Set*"))
!     for set in range(1, nsets + 1):
!         name = os.path.join(dir, "Set%d" % set)
          try:
              os.makedirs(name)
--- 49,60 ----
      files = [t[-1] for t in files]
  
+     # Make sure all the desired Set directories exist, and a reservoir
+     # directory.  "trash" is left holding a list of the excess Set
+     # directories (if nsets is less than the number of Set directories
+     # that already exist).  Those directories will be removed at the
+     # end, after all the messages they contain have been moved elsewhere.
      trash = glob.glob(os.path.join(dir, "Set*"))
!     for subdir in ["Set%d" % i for i in range(1, nsets+1)] + ["reservoir"]:
!         name = os.path.join(dir, subdir)
          try:
              os.makedirs(name)
***************
*** 60,67 ****
          except:
              pass
-     try:
-         os.makedirs(os.path.join(dir, "reservoir"))
-     except:
-         pass
  
      oldgroup = ""
--- 65,68 ----



From tim_one at users.sourceforge.net  Sun Dec 28 20:27:34 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Sun Dec 28 20:27:38 2003
Subject: [Spambayes-checkins] spambayes/testtools mksets.py,1.6,1.7
Message-ID: <E1AamBy-00040I-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/testtools
In directory sc8-pr-cvs1:/tmp/cvs-serv14887

Modified Files:
	mksets.py 
Log Message:
I'm not sure what the "# of groups" argument is good for here, but
because the code was splitting an entire path on "-", its idea of the
current file's group included the Ham/Set1, Ham/Set2, ..., prefix as
well as the "group number" from the basename.  That doesn't seem right.
Maybe it was, but if so it needed better docs cuz it's not that way
any more <wink>.


Index: mksets.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/testtools/mksets.py,v
retrieving revision 1.6
retrieving revision 1.7
diff -C2 -d -r1.6 -r1.7
*** mksets.py	29 Dec 2003 01:16:28 -0000	1.6
--- mksets.py	29 Dec 2003 01:27:32 -0000	1.7
***************
*** 47,51 ****
      files = [(os.path.basename(f), f) for f in files]
      files.sort()
-     files = [t[-1] for t in files]
  
      # Make sure all the desired Set directories exist, and a reservoir
--- 47,50 ----
***************
*** 70,75 ****
      cmess = 0
      cset = 1
!     for f in files:
!         newgroup = (f.split('-'))[0]
          if newgroup != oldgroup:
              oldgroup = newgroup
--- 69,74 ----
      cmess = 0
      cset = 1
!     for basename, f in files:
!         newgroup = basename.split('-')[0]
          if newgroup != oldgroup:
              oldgroup = newgroup
***************
*** 79,87 ****
          if ((ngroups is not None and cgroups > ngroups) or
              (nmess is not None and cmess > (nmess * nsets))):
!             newname = os.path.join(dir, "reservoir",
!                                    os.path.basename(f))
          else:
!             newname = os.path.join(dir, "Set%d" % cset,
!                                    os.path.basename(f))
              cset = (cset % nsets) + 1
          sys.stdout.write("%-78s\r" % ("Moving %s to %s" % (f, newname)))
--- 78,84 ----
          if ((ngroups is not None and cgroups > ngroups) or
              (nmess is not None and cmess > (nmess * nsets))):
!             newname = os.path.join(dir, "reservoir", basename)
          else:
!             newname = os.path.join(dir, "Set%d" % cset, basename)
              cset = (cset % nsets) + 1
          sys.stdout.write("%-78s\r" % ("Moving %s to %s" % (f, newname)))



From anadelonbrin at users.sourceforge.net  Sun Dec 28 20:46:45 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Sun Dec 28 20:46:49 2003
Subject: [Spambayes-checkins] spambayes CHANGELOG.txt, 1.28,
	1.29 WHAT_IS_NEW.txt, 1.21, 1.22
Message-ID: <E1AamUX-0004or-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv18519

Modified Files:
	CHANGELOG.txt WHAT_IS_NEW.txt 
Log Message:
Start bringing the changelog and what's new file up to date.  Many gaps still!


Index: CHANGELOG.txt
===================================================================
RCS file: /cvsroot/spambayes/spambayes/CHANGELOG.txt,v
retrieving revision 1.28
retrieving revision 1.29
diff -C2 -d -r1.28 -r1.29
*** CHANGELOG.txt	5 Nov 2003 13:06:56 -0000	1.28
--- CHANGELOG.txt	29 Dec 2003 01:46:43 -0000	1.29
***************
*** 1,6 ****
  [Note that all dates are in English, not American format - i.e. day/month/year]
  
! Release 1.1a1
! =============
  Anthony Baxter    05/11/2003  Spell-checked all the HTML and txt files <wink>
  Tony Meyer        30/10/2003  Implement [ 827138 ] Can't display clues/tokens/source for a trained message
--- 1,61 ----
  [Note that all dates are in English, not American format - i.e. day/month/year]
  
! Alpha Release 8
! ===============
! Tim Peters        29/12/2003  Many improvements to the mksets.py testtools script.
! Tim Peters        28/12/2003  Outlook: export.py - the -n option now gives the number of Set subdirectories desired, instead of a number of msgs per Set subdir "to shoot for".
! Tim Peters        28/12/2003  Added a new -t option to rebal.py, may have broken -s and -r options.
! Tim Peters        26/12/2003  Many improvements to the rebal.py testtools script.
! Tim Peters        26/12/2003  Many improvements to the export.py script for Outlook.
! Skip Montanaro    24/12/2003  storage: make state key a manifest constant
! Mark Hammond      23/12/2003  Tray app: Binary version failed to check for most recent version.
! Skip Montanaro    23/12/2003  Sendmail annotates the Received: header with "(may be forged)" if it thinks the sender is forging its identity.  Generate a token for this, if we are mining received headers.
! Tony Meyer        22/12/2003  Move OE specific stuff out from UserInterface.py to oe_mailbox.py.
! Mark Hammond      22/12/2003  Outlook: Default to background filtering being on for new versions.
! Mark Hammond      21/12/2003  Outlook: DWhen doing a "batch train" (eg, selecting multiple messages and saying "Delete as" or "Recover from",) the DB was saved in between each and every message.  Now only saved at the end (which was always the intent)
! Mark Hammond      21/12/2003  Outlook: DAs part of checking our configuration is invalid, make sure the user hasn't set us up such that either Spam/Unsure folders isn't also being watched for new messages
! Skip Montanaro    20/12/2003  Tokenizer: Solved the "backwards breakdown" problem with ip addresses in Received: headers.
! Skip Montanaro    20/12/2003  Tokenizer: Tightened up recognition of hostnames and accepted bracketed or parenthesized ip addresses without requiring a leading space.
! Mark Hammond      19/12/2003  Outlook: Remove handling of E_OBJECT_CHANGED exception, as it simply did not work.
! Mark Hammond      19/12/2003  Fix [ 803798 ] MAPI_E_OBJECT_CHANGED error saving spam score, which is a dupe of [787676], which was incorrectly marked as fixed
! Mark Hammond      19/12/2003  Outlook: Don't record in the training database unless we are successful in the filter - otherwise future attempts to filter will get all screwed up, as it will think it already was
! Mark Hammond      19/12/2003  Outlook: Move some of our init code from OnConnection to OnStartupComplete
! Tony Meyer        18/12/2003  Bring pspam into the modern SpamBayes world.
! Tony Meyer        17/12/2003  Add the basis of a new experimental (and highly debatable) option to 'slurp' URLs.
! Tim Peters        17/12/2003  Implemented the intended "tiling" version of x-use_bigrams.
! Tony Meyer        16/12/2003  Option names are always case insensitive, no matter what.
! Tony Meyer        16/12/2003  Fix a bug in the web interface where the probability would be incorrectly calculated on 'show clues'.
! Tony Meyer        16/12/2003  New experimental option: x-use_bigrams.
! Skip Montanaro    16/12/2003  mboxutils: This change generalizes the DirOfTxtFileMailbox class to assume all non-directory files contain a single message and to recursively descend into subdirectories of the argument directory.
! Tony Meyer        15/12/2003  Add a warning as a temporary solution for Python bug #845560.
! Tony Meyer        15/12/2003  Add the missing code for the Habeas headers tokenizing (and deprecate).
! Mark Hammond      15/12/2003  Fix [ 833439 ] default_bayes_customize.ini is confusing.
! Tim Peters        14/12/2003  Removed support code for the defunct experimental_ham_spam_imbalance_adjustment option
! Mark Hammond      14/12/2003  Fix [ 856628 ] reload(Options) fails in windows binary
! Mark Hammond      14/12/2003  Fix [ 859215 ] "Restore Defaults" causes assertion error at exit.
! Tony Meyer        14/12/2003  ImapUI: When logging in was done by the UI (to show available folders) we assigned the imap_session object to the wrong name
! Mark Hammond      10/12/2003  Outlook: Try and add the Spam field to the 'Unsure' folder in the same way we do for the Spam and watch folders.
! Tony Meyer        04/12/2003  Tray app: Change the default (double-click) behaviour of the tray to "review messages" rather than "display information".
! Tony Meyer        04/12/2003  Tray app: use SetDefaultItem (so the default action is in bold in the menu).
! Mark Hammond      02/12/2003  sb_server was ignoring command-line options; fix.
! Richie Hindle     27/11/2003  Sjoerd's improved version of patch 831388.
! Neale Pickett     27/11/2003  sb_filter now prints each message only once, not once per argument :)
! Tony Meyer        26/11/2003  sb_dbexpimp.py: Import/Export data as utf-8.
! Richie Hindle     26/11/2003  UserInterface.py More robust code for parsing score headers - copes with the presence of logarithms.
! Richie Hindle     26/11/2003  UserInterface.py: More robust code for parsing evidence headers.  Copes with ';' and ': ' being part of a clue.
! Richie Hindle     26/11/2003  Patch [ 831388 ]: Make message.py respect the header_score_digits option.
! Richie Hindle     26/11/2003  Made sb_filter obey the notate_to and notate_subject options.
! Tony Meyer        26/11/2003  As we now use whichdb to figure out what type of file the db is, if we were using windows and python 2.2 we would try and use dbhash instead of db3hash, which is a Bad Thing. Fix this.
! Tony Meyer        26/11/2003  message.py: Encode words in the evidence header as utf-8 if they are unicode.
! Barry A. Warsaw   25/11/2003  sb_xmlrpcserver.py: Make sure that the socket being bound is reusable.
! Barry A. Warsaw   25/11/2003  Change XMLHammie.score() so that the float score is returned directly instead of trying to be wrapped in a Binary object.
! Barry A. Warsaw   25/11/2003  New script: sb_evoscore.py - A shim script between sb_xmlrpcserver.py and Ximian Evolution.
! Skip Montanaro    25/11/2003  Added a makefile to the testtools directory to make using timcv easier.
! Richie Hindle     16/11/2003  Patch [ 842464 ] Correct installation instructions from "setup.py install" to "python setup.py install"
! Skip Montanaro    13/11/2003  sb_filter: add -o/--option command line arg that allows user to set any options value from the command line
! Skip Montanaro    13/11/2003  OptionsClass: Add set_from_cmdline()
! Skip Montanaro    13/11/2003  sb_filter: Allow multiple types of mailboxes to be processed using mboxutils.getmbox. If any mailbox files are given on the command line, the output is always a Unix-style mailbox containing From_ lines.
! Richie Hindle     11/11/2003  notesfilter: The header_x_string options now live in the Headers section, not the Hammie section.
! Richie Hindle     07/11/2003  Fixed an infinite loop when you break the browser connection to sb_server when sb_server is busy training.
  Anthony Baxter    05/11/2003  Spell-checked all the HTML and txt files <wink>
  Tony Meyer        30/10/2003  Implement [ 827138 ] Can't display clues/tokens/source for a trained message
***************
*** 36,42 ****
  Tony Meyer        20/09/2003  Add an advanced word query to the web UI.
  Tony Meyer        20/09/2003  Make the review messages page on the web UI more customizable.
- 
- 1.0 Releases
- ************
  
  Alpha Release 7
--- 91,94 ----

Index: WHAT_IS_NEW.txt
===================================================================
RCS file: /cvsroot/spambayes/spambayes/WHAT_IS_NEW.txt,v
retrieving revision 1.21
retrieving revision 1.22
diff -C2 -d -r1.21 -r1.22
*** WHAT_IS_NEW.txt	5 Dec 2003 04:45:44 -0000	1.21
--- WHAT_IS_NEW.txt	29 Dec 2003 01:46:43 -0000	1.22
***************
*** 10,14 ****
  noted in the "Transition" section.
  
! New in Alpha Release 7
  ======================
  
--- 10,14 ----
  noted in the "Transition" section.
  
! New in Alpha Release 8
  ======================
  
***************
*** 17,137 ****
  --------------------------
  
!  o If you are using a pickle for storage, your 'message info' database
!    would previously still have been a dbm (where available).  This is
!    no longer the case - if you are using a pickle for the statistics
!    database, you have a pickle for everything.  Your old 'message info'
!    database is not converted (and there is no utility provided to do so).
!    If you are using a pickle for storage, you should delete your old
!    'spambayes.messageinfo.db' file before restarting after the upgrade.
!    You should not suffer any ill effects from this, *unless* you are
!    using sb_imapfilter.py.  In that case, you will find that the filter
!    trains and classifies all messages in the folders it examines, even
!    if it has seen them before - this will only occur once, however.
! 
! There should be no other incompatible changes (from 1.0a6) in this release.
! 
!  o The scripts have all moved (in the archive), and their names have been
!    changed.  If you run "setup.py install", it will offer to remove the old
!    ones for you, which we recommend.  In the archive, the scripts are all
!    in a 'scripts' directory, and all the scripts start with the "sb_"
!    prefix, to avoid clashing with similarly named scripts from other
!    packages.  Some name changes go further - "pop3proxy" is now named
!    "sb_server", "hammiefilter" is now named "sb_filter", "hammiecli" is now
!    named "sb_client", "hammiesrv" is now named "sb_xmlrpcserver", "proxytee"
!    is now named "sb_upload", and the experimental "overkill" script is now
!    named "sb_pop3dnd".
! 
! If you were previously using the "hammie.py" script, you will notice that
! it is no longer available.  We recommend that you use either "sb_filter"
! (probably with "sb_mboxtrain"), or use "sb_server" and "sb_upload".  If you
! wish to continue as you were, you can use the "hammie.py" module, which
! will be installed in the "spambayes" package directory, in the same way you
! used the old "hammie.py" script.
! 
!  o All the backwards compatibility code for options which changed names has
!    been removed, which means that you *must* use the correct (new) names.
!    A script (sb_chkopts) is provided which, if you run it, will inform you
!    if you have any invalid names (if will not output anything if there are
!    no problems).
  
! In addition, the values taken by some options have changed, so if you're
! upgrading from a previous version, you may need to update your configuration
! file (.spambayesrc or bayescustomize.ini)
  
-  o The options to put the classification in the subject or recipient list
-    (notate_to and notate_subject) have moved from the "pop3proxy" section
-    to the "Headers" section.
-  o All the "pop3proxy" storage options (where the cache is stored, the
-    number of days before messages expire, and so on) have moved to the
-    "Storage" section.
-  o The "hammie" debug header options have been removed, and you should use
-    the "Headers" evidence header options instead.
  
! Note that pop3proxy (sb_server) and imapfilter users can simply use the web
! interface to check their options and correct any that are wrong.  All
! incorrectly named options in the configuration file will be removed.
  
  Outlook Plugin
  --------------
!  o Change the default for the ham/spam imbalance adjustment option to
!    False - this should make misclassifications for those with large
!    imbalances easier to understand.  Note that we recommend roughly equal
!    numbers of ham and spam are trained.
!  o Add a warning for those with highly imbalanced ham and spam.
!  o Improved the 'Show Clues' results page.
!  o When we fail to add the 'Spam' field to a read-only store (eg, hotmail),
!    complain less loudly.
  
  POP3 Proxy / SMTP Proxy
  -----------------------
!  o An error where a failure message would be printed by
!    the SMTP proxy, even on success, was fixed.
  
  Web Interface
  -------------
!  o The bug which caused the "TypeError" when trying to access
!    the database after setting a configuration option via the
!    interface has been fixed.
  
  POP3 Proxy Service / POP3 Proxy Tray Application
  ------------------------------------------------
  
!  o Both the pop3proxy_service.py and pop3proxy_tray.py
!    scripts are now installed (with "setup.py install") if
!    the user is using Windows.
  
  IMAP Filter
  -----------
!  o Correctly handle IMAP servers that (wrongly) fail to put folder names
!    in quotation marks
!  o Count all messages being classified instead of just the ones from the
!    last folder.
!  o Handle a folder name as a literal when presenting a list to choose from.
!  o Handle IMAP servers that do not pass a blank result line for an empty
!    search.
!  o Fix IMAP over SSL.
  
  General
  -------
!  o Various improvements have been made to the management of the
!    'message info' database.  As outlined above, it will now be
!    stored as a pickle, if your statistics database uses a pickle.
!    In addition, we attempt to close the database when we should,
!    and make sure that we explicitly update it.  This should hopefully
!    go some way to solving the "DB_RUN_RECOVERY" errors that have
!    been regularly reported - we would be interested to hear from
!    you if upgrading to 1.0a7 does appear to solve this problem
!    for you (email spambayes@python.org).
!  o We now try to determine the type of dbm storage used from the
!    file, if one already exists.  This should make the transition
!    between formats a little easier.
!  o Fix sb_xmlrpcserver to work with the renamed (since 1.0a5)
!    scripts.
!  o Fix the sense of include_trained in sb_mboxtrain.
!    
  
  Transition
  ==========
! If you are transitioning from a version older than 1.0a6, please also
  read the notes in the previous release notes (accessible from
  <http://sourceforge.net/project/showfiles.php?group_id=61702>).
--- 17,57 ----
  --------------------------
  
!  o XXX
  
! There should be no other incompatible changes (from 1.0a7) in this release.
  
  
! -------------------
! ** Other changes **
! -------------------
  
  Outlook Plugin
  --------------
!  o 
  
  POP3 Proxy / SMTP Proxy
  -----------------------
!  o 
  
  Web Interface
  -------------
!  o 
  
  POP3 Proxy Service / POP3 Proxy Tray Application
  ------------------------------------------------
  
!  o 
  
  IMAP Filter
  -----------
!  o 
  
  General
  -------
!  o 
  
  Transition
  ==========
! If you are transitioning from a version older than 1.0a7, please also
  read the notes in the previous release notes (accessible from
  <http://sourceforge.net/project/showfiles.php?group_id=61702>).
***************
*** 144,148 ****
  ===================
  The following bugs tracked via the Sourceforge system were fixed:
! 809769, 814322, 816400, 810342, 818552
  
  A URL containing the details of these bugs can be made by appending the
--- 64,68 ----
  ===================
  The following bugs tracked via the Sourceforge system were fixed:
! 
  
  A URL containing the details of these bugs can be made by appending the
***************
*** 161,162 ****
--- 81,110 ----
  No patches tracked via the Sourceforge system were integrated for this
  release.
+ 
+ 
+ Deprecated Options
+ ==================
+ 
+ [add explanation of deprecated options here]
+ 
+ The following options have been deprecated in this release:
+   o [Tokenizer] generate_time_buckets
+   o [Tokenizer] extract_dow
+   o [Classifier] experimental_ham_spam_imbalance_adjustment
+ 
+ 
+ New Experimental Options
+ ========================
+ 
+ [add explanation of experimental options and a pointer to the testing
+ setup here]
+ 
+ The following experimental options have been added in this release:
+   o [Tokenizer] search_for_habeas_headers
+   o [Tokenizer] reduce_habeas_headers
+   o [Classifier] use_bigrams
+   o [URLRetriever] slurp_urls
+   o [URLRetriever] cache_expiry_days
+   o [URLRetriever] cache_directory
+   o [URLRetriever] only_slurp_base
+   o [URLRetriever] web_prefix



From anadelonbrin at users.sourceforge.net  Sun Dec 28 21:43:36 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Sun Dec 28 21:43:40 2003
Subject: [Spambayes-checkins] spambayes/windows/docs - New directory
Message-ID: <E1AanNY-0006fK-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/windows/docs
In directory sc8-pr-cvs1:/tmp/cvs-serv25620/docs

Log Message:
Directory /cvsroot/spambayes/spambayes/windows/docs added to the repository




From anadelonbrin at users.sourceforge.net  Sun Dec 28 21:43:53 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Sun Dec 28 21:43:57 2003
Subject: [Spambayes-checkins] spambayes/windows/docs/images - New directory
Message-ID: <E1AanNp-0006fk-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/windows/docs/images
In directory sc8-pr-cvs1:/tmp/cvs-serv25644/images

Log Message:
Directory /cvsroot/spambayes/spambayes/windows/docs/images added to the repository




From anadelonbrin at users.sourceforge.net  Sun Dec 28 21:45:51 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Sun Dec 28 21:45:54 2003
Subject: [Spambayes-checkins] 
	spambayes/windows/docs/images python.jpg, NONE,
	1.1 review.jpg, NONE, 1.1 review_empty.jpg, NONE,
	1.1 sblogo.jpg, NONE, 1.1 span.jpg, NONE, 1.1 tray.jpg, NONE, 1.1
Message-ID: <E1AanPj-0006kt-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/windows/docs/images
In directory sc8-pr-cvs1:/tmp/cvs-serv25963/windows/docs/images

Added Files:
	python.jpg review.jpg review_empty.jpg sblogo.jpg span.jpg 
	tray.jpg 
Log Message:
Add images for the sb_server (etc) documentation.  Three of these are stolen
from the Outlook documentation, because it was difficult to reference them otherwise.
(The Outlook docs and non-Outlook docs might need a shared images directory, but
for the moment it's only 18k).


--- NEW FILE: python.jpg ---
(This appears to be a binary file; contents omitted.)

--- NEW FILE: review.jpg ---
(This appears to be a binary file; contents omitted.)

--- NEW FILE: review_empty.jpg ---
(This appears to be a binary file; contents omitted.)

--- NEW FILE: sblogo.jpg ---
(This appears to be a binary file; contents omitted.)

--- NEW FILE: span.jpg ---
(This appears to be a binary file; contents omitted.)

--- NEW FILE: tray.jpg ---
(This appears to be a binary file; contents omitted.)



From anadelonbrin at users.sourceforge.net  Sun Dec 28 22:21:41 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Sun Dec 28 22:21:45 2003
Subject: [Spambayes-checkins] spambayes/windows readme_proxy.html,1.4,1.5
Message-ID: <E1AanyP-0007sy-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/windows
In directory sc8-pr-cvs1:/tmp/cvs-serv30310/windows

Modified Files:
	readme_proxy.html 
Log Message:
Mention the "help" link on the web interface.


Index: readme_proxy.html
===================================================================
RCS file: /cvsroot/spambayes/spambayes/windows/readme_proxy.html,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** readme_proxy.html	29 Dec 2003 02:59:54 -0000	1.4
--- readme_proxy.html	29 Dec 2003 03:21:39 -0000	1.5
***************
*** 78,81 ****
--- 78,87 ----
  SpamBayes has for you to review.</p>
  
+ <p>For the most part, SpamBayes just silently works.  When you do need
+ to interact with it, you do so via an interface presented in a web browser.
+ If you need help using the interface, click the "Help" button at the bottom
+ of any page.  This also explains how you can contact the developers, if
+ you wish to do so.</p>
+ 
  <h3>Using the Service</h3>
  <p>There is a Windows service program installed, but it is not configured



From anadelonbrin at users.sourceforge.net  Sun Dec 28 21:59:56 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Sun Dec 28 22:30:06 2003
Subject: [Spambayes-checkins] spambayes/windows readme_proxy.html,1.3,1.4
Message-ID: <E1AandM-00078K-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/windows
In directory sc8-pr-cvs1:/tmp/cvs-serv27410/windows

Modified Files:
	readme_proxy.html 
Log Message:
Improve the non-Outlook documentation for the binary.


Index: readme_proxy.html
===================================================================
RCS file: /cvsroot/spambayes/spambayes/windows/readme_proxy.html,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** readme_proxy.html	23 Dec 2003 03:17:41 -0000	1.3
--- readme_proxy.html	29 Dec 2003 02:59:54 -0000	1.4
***************
*** 20,23 ****
--- 20,25 ----
  </table>
  <h1>SpamBayes Server/Proxy Application</h1>
+ <strong>A spam filter based on statistical analysis of your personal mail.</strong>
+ 
  <h2>Welcome to SpamBayes</h2>
  
***************
*** 25,29 ****
  Outlook Addin, which was presented as an option at installation time.</em>
  If you are using Outlook Express or almost any other mail program, 
! you are in the right place</p>
  
  <p>This SpamBayes application sits between your mail server (generally at
--- 27,36 ----
  Outlook Addin, which was presented as an option at installation time.</em>
  If you are using Outlook Express or almost any other mail program, 
! you are in the right place.</p>
! 
! <p>SpamBayes is an application that provides spam classification based on statistical
! analysis of your personal mail. Unlike many other spam detection systems, SpamBayes
! actually learns what you consider spam, and continually adapts as both your regular
! email and spam patterns change.</p>
  
  <p>This SpamBayes application sits between your mail server (generally at
***************
*** 47,68 ****
  can also configure SpamBayes via this interface in your web browser.</p>
  
! <p>The <a href="http://spambayes.sourceforge.net/faq.html#using-spambayes">
! using SpamBayes section in the online FAQ</a> has alot of information
! which you may find useful, especially considering how little documentation
! is supplied.  The <a href="http://spambayes.sourceforge.net/">main SpamBayes
! project pages</a> may also be of interest.</p>
  
  <h3>Getting Started</h3>
! <p>You should start the <span style="font-style: italic;">SpamBayes Tray
  Icon</span> program, which was installed into your Start menu. This is the
  recommended method of operating SpamBayes.  If you right-click on this
! icon you will be presented with a number of options to control SpamBayes,
! including an option to configure SpamBayes for your system.</p>
  
  <h3>Using the Service</h3>
! <p>There is a windows service program installed, but it is not configured
  by the installation program. If you configure this manually, you
  can continue to use the Tray Icon program, as it will control the
! service instead of running the server internally as it normally does.</p>
  <p>To install the service, perform the following steps:<br/>
  <ul>
--- 54,88 ----
  can also configure SpamBayes via this interface in your web browser.</p>
  
! <p>Please remember that this is free software; please be patient, and
! note that there are plenty of things to improve. There are ways
! you can help, even if you aren't a programmer - you could help with
! this documentation, or <a
!  href="http://spambayes.org/donations.html">make a donation</a>, or
! any number of other ways - please see the <a
!  href="http://spambayes.org">main project
! page</a> for information. The list of <a
!  href="http://spambayes.org/faq.html">Frequently Asked Questions</a>
! may also answer some of yours, especially considering the (current)
! lack of documentation - you may wish to start in the 
! <a href="http://spambayes.sourceforge.net/faq.html#using-spambayes">
! "Using SpamBayes" section</a>.</p>
  
  <h3>Getting Started</h3>
! <p><img src="docs/images/tray.jpg" style="float:right" alt="Tray menu" />
! You should start the <span style="font-style: italic;">SpamBayes Tray
  Icon</span> program, which was installed into your Start menu. This is the
  recommended method of operating SpamBayes.  If you right-click on this
! icon you will be presented with a number of options to control SpamBayes
! (see figure on the right), including an option to configure SpamBayes for
! your system. Double-clicking on the icon will bring up the messages that
! SpamBayes has for you to review.</p>
  
  <h3>Using the Service</h3>
! <p>There is a Windows service program installed, but it is not configured
  by the installation program. If you configure this manually, you
  can continue to use the Tray Icon program, as it will control the
! service instead of running the server internally as it normally does.
! If you don't know what a "Windows service program" is, don't worry about it;
! you don't need to use this.</p>
  <p>To install the service, perform the following steps:<br/>
  <ul>
***************
*** 81,89 ****
    <li>If desired, change the properties of the service to start at boot
  time.</li>
!   <li>Start the service</li>
    <li>Start the <span style="font-style: italic;">SpamBayes Tray Icon</span>
  program, and confirm the server is running. Configure and manage
  spambayes normally.</li>
  </ul>
  </body>
  </html>
--- 101,197 ----
    <li>If desired, change the properties of the service to start at boot
  time.</li>
!   <li>Start the service.</li>
    <li>Start the <span style="font-style: italic;">SpamBayes Tray Icon</span>
  program, and confirm the server is running. Configure and manage
  spambayes normally.</li>
  </ul>
+ 
+ <h3>Configuration</h3>
+ 
+ <p><img alt="Config page" style="float: right; margin: 1em" src="docs/images/config.jpg" />
+ There are two configuration steps that you need to take:</p>
+ <ol>
+ <li><strong>Setup SpamBayes:</strong> SpamBayes comes with default settings
+ that, for the most part, should be suitable for your system.  There is
+ one setting that you will need to fill out, however, which is the address of
+ your mail server (at your ISP, most likely).  Right-click on the tray icon
+ and choose <em>Configure...</em>; this will open up a browser window like the
+ one in the figure on the right.  You can skip over most of the settings here,
+ apart from the top one (shown in the figure).  You need to enter the address
+ of your mail server.  If you don't know the address, you can find it in the
+ settings of your mail client.
+ <br /><br />
+ If you wish to, you can change other settings on this page - to the right
+ of each option is a description of what it does.  There are also some
+ more advanced options available via a button at the end of the page
+ (note that these are not all of the SpamBayes options - see the
+ <a href="http://spambayes.org/faq.html">FAQ</a> for information about how
+ to access other options).
+ <br /><br />
+ Once you are satisfied that you have entered the correct options for your
+ setup, click the <em>Save</em> button at the end of the page.  These
+ values will be saved in your personal configuration file and be used
+ each time SpamBayes is started up.
+ <br /><br />
+ <li><strong>Setup your mail client:</strong> You need to set your
+ mail client to receive mail from <em>localhost</em> rather than your
+ mail server.</li>
+ </ol>
+ 
+ <h3>Training</h3>
+ <p>When you first start using SpamBayes, all your mail will be classified as
+ 'unsure' because SpamBayes doesn't have any preconceived ideas about what
+ looks like good or bad mail.  As soon as you start training the classification
+ will improve, and by the time you've classified even 20 messages of each you'll
+ be seeing quite reasonable results.</p>
+ 
+ <p><img alt="Empty review page" style="float:right; margin: 1em" src="docs/images/review.jpg" />
+ SpamBayes saves a <strong>temporary copy</strong> of all incoming mail so that
+ classification can be independant of whatever mail client (Outlook Express, Eudora, ...)
+ you are using.  You need to run through these messages and tell SpamBayes how to
+ handle mail like that in the future.</p>
+ 
+ <p>If you double-click the tray icon a browser window will open looking like the
+ figure on the right.  For each message listed, you need to choose to either
+ discard (don't train on this message), defer (leave training on this message
+ until later), or train (as either good - <em>ham</em>), or bad - <em>spam</em>).
+ You do this by simply clicking in the circle in the appropriate column; if you
+ wish to change all the messages to the same action, you can simply click the
+ column heading.</p>
+ 
+ <p>You are presented with the subject and sender of each message, but, if this
+ isn't enough information for you to make a decision on the message, you can also
+ view the message text (this is the raw text, so you can't do any damage if the
+ message contains a virus or any other malignant data).  To do this, simply click
+ on the subject of the message.</p>
+ 
+ <p>Once you have chosen the actions you wish to perform on all the displayed
+ messages, click the <em>Train</em> button at the end of the page.  SpamBayes
+ will then update its database to reflect this data.</p>
+ 
+ <p>Note that the messages are split up into the classification that SpamBayes
+ would place the message with current training data (if this is correct, you might
+ choose to <em>Discard</em> the message, rather than train on it - see the 
+ <a href="http://entrian.com/sbwiki">SpamBayes wiki</a> for discussion of training
+ techniques).  You can also see the <em>Tokens</em> that the message contains
+ (the words in the message, plus some additional tokens that SpamBayes generates)
+ and the <em>Clues</em> that SpamBayes used in classifying the message (not all
+ tokens are used in classification).</p>
+ 
+ <p>So that the page isn't overwhelmingly long, messages waiting for review are
+ split by the day they arrived.  You can use the <em>Previous Day</em> or
+ <em>Next Day</em> buttons at the top of the page to move between days.  If mail
+ arrives while the review page is open the new messages will <strong>not</strong>
+ be automatically added to the displayed list; to view the new message, click
+ the <em>Refresh</em> button at the top of the page.</p>
+ 
+ <h2><a name="Your_help_is_needed"></a>Your help is needed!</h2>
+ <p>This is free software. Please offer any help you are able
+ to. In particular, contributions to this documentation are
+ welcome! If you don't know where to start, the
+ <a href="http://spambayes.org/faq.html">FAQ</a> has
+ <a href="http://spambayes.org/faq.html#i-m-not-a-programmer-but-want-to-help-out-what-can-i-do">
+ details about how you can help</a>.
+ </p>
  </body>
  </html>



From anadelonbrin at users.sourceforge.net  Sun Dec 28 22:41:12 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Sun Dec 28 22:41:15 2003
Subject: [Spambayes-checkins] spambayes/spambayes ProxyUI.py, 1.34,
	1.35 UserInterface.py, 1.39, 1.40
Message-ID: <E1AaoHI-00004z-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv303/spambayes

Modified Files:
	ProxyUI.py UserInterface.py 
Log Message:
Improve the 'online' help message for the review page, and add messages
for the stats and home pages.


Index: ProxyUI.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/ProxyUI.py,v
retrieving revision 1.34
retrieving revision 1.35
diff -C2 -d -r1.34 -r1.35
*** ProxyUI.py	16 Dec 2003 05:06:34 -0000	1.34
--- ProxyUI.py	29 Dec 2003 03:41:10 -0000	1.35
***************
*** 179,183 ****
          self._writePreamble("Home")
          self.write(content)
!         self._writePostamble()
  
      def onUpload(self, file):
--- 179,183 ----
          self._writePreamble("Home")
          self.write(content)
!         self._writePostamble(help_topic="home_proxy")
  
      def onUpload(self, file):

Index: UserInterface.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/UserInterface.py,v
retrieving revision 1.39
retrieving revision 1.40
diff -C2 -d -r1.39 -r1.40
*** UserInterface.py	22 Dec 2003 02:31:39 -0000	1.39
--- UserInterface.py	29 Dec 2003 03:41:10 -0000	1.40
***************
*** 870,881 ****
              if topic == "review":
                  helppage.helpheader = "Review Page Help"
!                 helppage.helptext = "This page lists messages that have " \
!                                     "arrived in the last %s days and that " \
!                                     "have not yet been trained.  You should " \
!                                     "go through the messages, correcting " \
!                                     "the classification where necessary, " \
!                                     "and then click train, to train " \
!                                     "SpamBayes with these messages" \
!                                     % (options["Storage", "cache_expiry_days"],)
          self.write(helppage)
          self._writePostamble()
--- 870,977 ----
              if topic == "review":
                  helppage.helpheader = "Review Page Help"
!                 helppage.helptext = """<p>When you first start using
! SpamBayes, all your mail will be classified as 'unsure' because SpamBayes
! doesn't have any preconceived ideas about what good or bad mail looks like.
! As soon as you start training the classification will improve, and by the
! time you've classified even 20 messages of each you'll be seeing quite
! reasonable results.</p>
! 
! <p>SpamBayes saves a <strong>temporary copy</strong> of all incoming mail
! so that classification can be independant of whatever mail client you are
! using. You need to run through these messages and tell SpamBayes how to
! handle mail like that in the future. This page lists messages that have
! arrived in the last %s days and that have not yet been trained. For each
! message listed, you need to choose to either <strong>discard</strong>
! (don't train on this message), <strong>defer</strong> (leave training on
! this message until later), or train (as either good -
! <strong>ham</strong>), or bad - <strong>spam</strong>). You do this by
! simply clicking in the circle in the appropriate column; if you wish to
! change all the messages to the same action, you can simply click the column
! heading.</p>
! 
! <p>You are presented with the subject and sender of each message, but, if
! this isn't enough information for you to make a decision on the message,
! you can also view the message text (this is the raw text, so you can't do
! any damage if the message contains a virus or any other malignant data).
! To do this, simply click on the subject of the message.</p>
! 
! <p>Once you have chosen the actions you wish to perform on all the
! displayed messages, click the <em>Train</em> button at the end of the page.
! SpamBayes will then update its database to reflect this data.</p>
! 
! <p>Note that the messages are split up into the classification that
! SpamBayes would place the message with current training data (if this is
! correct, you might choose to <em>Discard</em> the message, rather than
! train on it - see the <a href="http://entrian.com/sbwiki">SpamBayes wiki
! </a> for discussion of training techniques).  You can also see the
! <em>Tokens</em> that the message contains (the words in the message,
! plus some additional tokens that SpamBayes generates) and the <em>Clues
! </em> that SpamBayes used in classifying the message (not all tokens are
! used in classification).</p>
! 
! <p>So that the page isn't overwhelmingly long, messages waiting for review
! are split by the day they arrived.  You can use the <em>Previous Day</em>
! or <em>Next Day</em> buttons at the top of the page to move between days.
! If mail arrives while the review page is open the new messages will
! <strong>not</strong> be automatically added to the displayed list; to add
! the new message, click the <em>Refresh</em> button at the top of the page.
! </p><hr />""" % (options["Storage", "cache_expiry_days"],)
!             elif topic == "stats":
!                 # This does belong with UserInterface.py, but should
!                 # still probably be loaded from a file or something to
!                 # avoid all this clutter.  Someone come up with the
!                 # best solution! (A pickle?  A single text file? A text
!                 # file per help page in a directory?)
!                 helppage.helpheader = "Statistics Page Help"
!                 helppage.helptext = """<p>SpamBayes keeps track of certain
! information about the messages that are classified.  For your interest,
! this page displays statistics about the messages that have been classified
! and trained so far.</p>
! 
! <p>Currently the page displays information about the
! number of messages that have been classified as good, bad and unsure, how
! many of these were false negatives or positives, and how many messages
! were classified as unsure (and what their correct classification was).</p>
! 
! <p>Note that the data for this page resides in the &quot;message info&quot;
! database that SpamBayes uses, and so only reflects messages since the
! last time this database was created.</p><hr />"""
!             elif topic == "home_proxy":
!                 # Also belongs with UserInterface.py, and probably
!                 # not with the source!
!                 helppage.helpheader = "Home Page Help"
!                 helppage.helptext = """<p>This is the main page for the
! SpamBayes web interface.  You are presented with some information about
! the current status of SpamBayes, and can follow links to review messages
! or alter your configuration.</p>
! 
! <p>If you have messages stored in a mbox or dbx (Outlook Express) file
! that you wish to 'bulk' train, or if you wish to train on a message
! that you type in, you can do this on this page.  Click the
! &quot;Browse&quot; button (or paste the text in, including headers),
! and then click the <em>Train as Ham</em> or <em>Train as Spam</em>
! button.</p>
! 
! <p>Likewise, if you have a message that you wish to classify, you
! can do this.  Either paste the message into the text box, or click
! &quot;Browse&quot; and locate the text file that the message is
! located in.  Click <em>Classify</em>, and you will be taken to a
! page describing the classification of that message.</p>
! 
! <p>If you want to find out information about a word in the statistics
! database that forms the heart of SpamBayes, you can use the &quot;Word
! Query&quot; facility.  Enter in the word that you wish to search for
! and click <em>Tell me about this word</em>.  If you enable the advanced
! find query, you can also search using wildcards or regular expressions.</p>
! 
! <p>You can also search for a specific message in the cache of temporary
! copies of messages that have been proxied.  You might wish to do this if
! you realise that you have incorrectly trained a message and need to correct
! the training.  You can search the subject, headers, or message body, or
! for the SpamBayes ID (which is in the headers of messages that SpamBayes
! proxies).  Messages that are found will be presented in the standard
! review page.  Note that once messages expire from the cache (after %s
! days), you can no longer find them.</p>
! <hr />""" % (options["Storage", "cache_expiry_days"],)
          self.write(helppage)
          self._writePostamble()
***************
*** 891,895 ****
          stats = self._buildBox("Statistics", None, "<br/><br/>".join(stats))
          self.write(stats)
!         self._writePostamble()
  
      def onBugreport(self):
--- 987,991 ----
          stats = self._buildBox("Statistics", None, "<br/><br/>".join(stats))
          self.write(stats)
!         self._writePostamble(help_topic="stats")
  
      def onBugreport(self):



From anadelonbrin at users.sourceforge.net  Sun Dec 28 22:07:36 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Sun Dec 28 22:45:03 2003
Subject: [Spambayes-checkins] spambayes/windows/py2exe setup_all.py, 1.11,
	1.12
Message-ID: <E1Aankm-0007Q4-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/windows/py2exe
In directory sc8-pr-cvs1:/tmp/cvs-serv28516/windows/py2exe

Modified Files:
	setup_all.py 
Log Message:
Copying over the Outlook images wasn't working (here at least) and,
anyway, we have new images that aren't for Outlook now, so fix this up
so that what should be installed is.


Index: setup_all.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/windows/py2exe/setup_all.py,v
retrieving revision 1.11
retrieving revision 1.12
diff -C2 -d -r1.11 -r1.12
*** setup_all.py	23 Dec 2003 06:05:20 -0000	1.11
--- setup_all.py	29 Dec 2003 03:07:34 -0000	1.12
***************
*** 115,121 ****
  proxy_data_files = [
      ["docs/sb_server", [os.path.join(sb_top_dir, r"windows\readme_proxy.html")]],
!     # this is hacky - the readme uses the Outlook data files.  We need to
!     # consolidate the documents.
!     ["docs/sb_server/docs/images", glob.glob(os.path.join(sb_top_dir, r"Outlook2000\docs\images\*.jpg"))],
  ]
  
--- 115,122 ----
  proxy_data_files = [
      ["docs/sb_server", [os.path.join(sb_top_dir, r"windows\readme_proxy.html")]],
!     # note that this includes images that are already in the outlook/docs/images
!     # directory - we need to consolidate the documentation (in terms of
!     # sharing images, if nothing else)
!     ["docs/sb_server/docs/images", glob.glob(os.path.join(sb_top_dir, r"windows\docs\images\*.jpg"))],
  ]
  



From anadelonbrin at users.sourceforge.net  Sun Dec 28 21:59:56 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Sun Dec 28 22:46:06 2003
Subject: [Spambayes-checkins] 
	spambayes/windows/docs/images config.jpg, NONE, 1.1
Message-ID: <E1AandM-00078O-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/windows/docs/images
In directory sc8-pr-cvs1:/tmp/cvs-serv27410/windows/docs/images

Added Files:
	config.jpg 
Log Message:
Improve the non-Outlook documentation for the binary.


--- NEW FILE: config.jpg ---
(This appears to be a binary file; contents omitted.)



From anadelonbrin at users.sourceforge.net  Sun Dec 28 23:46:33 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Sun Dec 28 23:46:37 2003
Subject: [Spambayes-checkins] spambayes CHANGELOG.txt, 1.29,
	1.30 WHAT_IS_NEW.txt, 1.22, 1.23
Message-ID: <E1AapIX-0002A2-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv8306

Modified Files:
	CHANGELOG.txt WHAT_IS_NEW.txt 
Log Message:
Bring these closer to the actual current status.


Index: CHANGELOG.txt
===================================================================
RCS file: /cvsroot/spambayes/spambayes/CHANGELOG.txt,v
retrieving revision 1.29
retrieving revision 1.30
diff -C2 -d -r1.29 -r1.30
*** CHANGELOG.txt	29 Dec 2003 01:46:43 -0000	1.29
--- CHANGELOG.txt	29 Dec 2003 04:46:31 -0000	1.30
***************
*** 3,6 ****
--- 3,7 ----
  Alpha Release 8
  ===============
+ Tony Meyer        29/12/2003  Web interface: Improve the 'online' help message for the review page, and add messages for the stats and home pages.
  Tim Peters        29/12/2003  Many improvements to the mksets.py testtools script.
  Tim Peters        28/12/2003  Outlook: export.py - the -n option now gives the number of Set subdirectories desired, instead of a number of msgs per Set subdir "to shoot for".
***************
*** 9,18 ****
--- 10,26 ----
  Tim Peters        26/12/2003  Many improvements to the export.py script for Outlook.
  Skip Montanaro    24/12/2003  storage: make state key a manifest constant
+ Mark Hammond      23/12/2003  Along with checking Outlook isn't running, check Outlook's mail transport also isn't running, and that an existing sb_server isn't.
+ Mark Hammond      23/12/2003  Fix uninstall problem - uninstall should be 100% clean, assuming Outlook isn't running.
  Mark Hammond      23/12/2003  Tray app: Binary version failed to check for most recent version.
  Skip Montanaro    23/12/2003  Sendmail annotates the Received: header with "(may be forged)" if it thinks the sender is forging its identity.  Generate a token for this, if we are mining received headers.
  Tony Meyer        22/12/2003  Move OE specific stuff out from UserInterface.py to oe_mailbox.py.
  Mark Hammond      22/12/2003  Outlook: Default to background filtering being on for new versions.
+ Tony Meyer        22/12/2003  Web interface: A beginning at a more userfriendly interface to the testing setup.
+ Mark Hammond      22/12/2003  Outlook: When a frozen executable, addin.py becomes a mini-installer EXE for the DLL.
  Mark Hammond      21/12/2003  Outlook: DWhen doing a "batch train" (eg, selecting multiple messages and saying "Delete as" or "Recover from",) the DB was saved in between each and every message.  Now only saved at the end (which was always the intent)
  Mark Hammond      21/12/2003  Outlook: DAs part of checking our configuration is invalid, make sure the user hasn't set us up such that either Spam/Unsure folders isn't also being watched for new messages
+ Mark Hammond      20/12/2003  dump_props.py: Add -c option, which writes output to the Windows clipboard.
+ Mark Hammond      20/12/2003  Outlook: Include the foldername in many messages, so help track down wierd bugs from user logs.  Say what we are watching a folder for.
+ Mark Hammond      20/12/2003  Outlook: Fix [ 860410 ] SpamBayes allows top-level folders to be selected, and also prevent a single-select dialog from closing with no selection
  Skip Montanaro    20/12/2003  Tokenizer: Solved the "backwards breakdown" problem with ip addresses in Received: headers.
  Skip Montanaro    20/12/2003  Tokenizer: Tightened up recognition of hostnames and accepted bracketed or parenthesized ip addresses without requiring a leading space.
***************
*** 22,25 ****
--- 30,34 ----
  Mark Hammond      19/12/2003  Outlook: Move some of our init code from OnConnection to OnStartupComplete
  Tony Meyer        18/12/2003  Bring pspam into the modern SpamBayes world.
+ Mark Hammond      17/12/2003  Tray app: Better icons and icon loading code.
  Tony Meyer        17/12/2003  Add the basis of a new experimental (and highly debatable) option to 'slurp' URLs.
  Tim Peters        17/12/2003  Implemented the intended "tiling" version of x-use_bigrams.
***************
*** 31,41 ****
--- 40,55 ----
  Tony Meyer        15/12/2003  Add the missing code for the Habeas headers tokenizing (and deprecate).
  Mark Hammond      15/12/2003  Fix [ 833439 ] default_bayes_customize.ini is confusing.
+ Mark Hammond      14/12/2003  Move the option loading code to a function, then call this function as the module loads.
  Tim Peters        14/12/2003  Removed support code for the defunct experimental_ham_spam_imbalance_adjustment option
  Mark Hammond      14/12/2003  Fix [ 856628 ] reload(Options) fails in windows binary
  Mark Hammond      14/12/2003  Fix [ 859215 ] "Restore Defaults" causes assertion error at exit.
  Tony Meyer        14/12/2003  ImapUI: When logging in was done by the UI (to show available folders) we assigned the imap_session object to the wrong name
+ Skip Montanaro    10/12/2003  Loosen constraints on HEADER_VALUE regular expression.
+ Skip Montanaro    10/12/2003  Add ability for "x-" options (deprecated, or experimental).
  Mark Hammond      10/12/2003  Outlook: Try and add the Spam field to the 'Unsure' folder in the same way we do for the Spam and watch folders.
  Tony Meyer        04/12/2003  Tray app: Change the default (double-click) behaviour of the tray to "review messages" rather than "display information".
  Tony Meyer        04/12/2003  Tray app: use SetDefaultItem (so the default action is in bold in the menu).
+ Mark Hammond      03/12/2003  For the unittest scritps avoid sys.path munging.
+ Mark Hammond      03/12/2003  Add new test_programs unittest script and support file for unittest scripts.
  Mark Hammond      02/12/2003  sb_server was ignoring command-line options; fix.
  Richie Hindle     27/11/2003  Sjoerd's improved version of patch 831388.
***************
*** 52,55 ****
--- 66,70 ----
  Barry A. Warsaw   25/11/2003  New script: sb_evoscore.py - A shim script between sb_xmlrpcserver.py and Ximian Evolution.
  Skip Montanaro    25/11/2003  Added a makefile to the testtools directory to make using timcv easier.
+ Neale Pickett     18/11/2003  Cleanup sb_filter and sb_mboxtrain.
  Richie Hindle     16/11/2003  Patch [ 842464 ] Correct installation instructions from "setup.py install" to "python setup.py install"
  Skip Montanaro    13/11/2003  sb_filter: add -o/--option command line arg that allows user to set any options value from the command line

Index: WHAT_IS_NEW.txt
===================================================================
RCS file: /cvsroot/spambayes/spambayes/WHAT_IS_NEW.txt,v
retrieving revision 1.22
retrieving revision 1.23
diff -C2 -d -r1.22 -r1.23
*** WHAT_IS_NEW.txt	29 Dec 2003 01:46:43 -0000	1.22
--- WHAT_IS_NEW.txt	29 Dec 2003 04:46:31 -0000	1.23
***************
*** 28,53 ****
  Outlook Plugin
  --------------
!  o 
  
  POP3 Proxy / SMTP Proxy
  -----------------------
!  o 
  
  Web Interface
  -------------
!  o 
  
  POP3 Proxy Service / POP3 Proxy Tray Application
  ------------------------------------------------
! 
!  o 
  
  IMAP Filter
  -----------
!  o 
  
  General
  -------
!  o 
  
  Transition
--- 28,144 ----
  Outlook Plugin
  --------------
!  o Add slightly better statistics.
!  o When installing, along with checking Outlook isn't running, check
!    Outlook's mail transport also isn't running.
!  o Fix uninstall problem - uninstall should be 100% clean, assuming Outlook
!    isn't running.
!  o Default to background filtering being on.
!  o When doing a "batch train" (eg, selecting multiple messages and saying
!    "Delete as" or "Recover from") the database was saved in between each
!    and every message.  Now it is only saved at the end.
!  o Ensure that the Spam and Unsure folders aren't also being watched for
!    new messages.
!  o Include the folder name in many messages, to help track down wierd bugs
!    from user logs, and also say what we are watching a folder for.
!  o Don't allow top-level folders to be selected.
!  o Don't allow a single-select dialog to be closed without a selection.
!  o Add the Spam field to the 'Unsure' folder as is done with the Spam and
!    watched folders.
!  o Improve matters when the default (Outlook message) store is offline.
  
  POP3 Proxy / SMTP Proxy
  -----------------------
!  o smtpproxy is now only a module, not a script.  Use sb_server instead.
!  o sb_server was ignoring command-line options; fix.
  
  Web Interface
  -------------
!  o Add the start of a user-friendly interface to the testing tools that are
!    part of SpamBayes (for testing new features against corpora of ham and
!    spam).
!  o Fix a bug in the web interface where the probability would be
!    incorrectly calculated on 'show clues'.
!  o More robust code for parsing score headers - copes with the presence of
!    logarithms.
!  o More robust code for parsing evidence headers.  Copes with ';' and ': '
!    being part of a clue.
!  o Increased the auth-digest login timeout from one minute to twenty.
!  o Improve the 'Find Message' query on the front page of the web UI.
!  o Add an 'advanced' word query (off by default).
!  o Make the review messages page more customizable.
!  o Provide a partially filled-in bug report message (please use it!).
!  o Add basic statistics information.
!  o Add a basic help system.
!  o Add warning information, for example if the user has imbalanced
!    training, or insufficient training.
!  o Fixed an infinite loop when you break the browser connection to
!    sb_server when sb_server is busy training.
  
  POP3 Proxy Service / POP3 Proxy Tray Application
  ------------------------------------------------
!  o Fix checking for most recent version.
!  o Better icons and icon loading code.
!  o Change the double-click (default) action to "review messages" and
!    display the default in bold.
!  o If a proxy is already running, don't start the service.
  
  IMAP Filter
  -----------
!  o When a message is marked as deleted, it is now marked as read (seen) as
!    well.
!  o If sb_imapfilter.py is run without any switches, just serve the web
!    interface (but don't launch a browser).
! 
! sb_filter
! ---------
!  o Print each message once, not once per argument.
!  o Now obeys the notate_to and notate_subject options.
!  o Added -o/--option command line argument, which allows setting any
!    option via the command line.
!  o Expanded the documentation.
!  o Allow multiple types of mailboxes to be processed using mboxutils.getmbox.
!    If any mailbox files are given on the command line, the output is always
!    a Unix-style mailbox containing From_ lines.
!  o If the -n switch was before the -d/-p switch, then the name wouldn't be
!    used; this has been changed so that the -d/-p name is used wherever the
!    -n switch is.
! 
! Testing Suite
! -------------
!  o Many improvements to the mksets.py script.
!  o Many improvements to the rebal.py script.
!  o Many improvements to the export.py script (for Outlook).
!  o Added a makefile to the testtools directory to make using timcv.py easier.
! 
! Tokenizer
! ---------
!  o Sendmail annotates the Received: header with "(may be forged)" if it
!    thinks the sender is forging its identity.  Generate a token for this,
!    if we are mining received headers.
!  o Solved the "backwards breakdown" problem with IP addresses in Received:
!    headers.
!  o Tightened up recognition of hostnames and accepted bracketed or
!    parenthesized IP addresses without requiring a leading space.
!  o Add the missing code for the Habeas headers tokenizing (and deprecated it).
!  o Removed support code for the defunct experimental_ham_spam_imbalance_adjustment
!    option.
  
  General
  -------
!  o Option names are always case insensitive, no matter what.
!  o Moved the option loading code to a function.
!  o Generalized the DirOfTxtFileMailbox class in mboxutils to assume all
!    non-directory files contain a single message and to recursively descend
!    into subdirectories of the argument directory.
!  o Loosened constraints on HEADER_VALUE regular expression.
!  o Import/Export data as utf-8 with sb_expimp.py.
!  o Fixes to the which_database.py utility script.
!  o Fix bug where if one was using Python 2.2, Windows and bsddb
!    the database would never open correctly.
!  o New script: sb_evoscore.py - A shim script between sb_xmlrpcserver.py
!    and Ximian Evolution.
!  o Fix the pspam scripts, muttrc and spambayes.el so that they work with
!    the current SpamBayes package.
! 
  
  Transition
***************
*** 57,62 ****
  <http://sourceforge.net/project/showfiles.php?group_id=61702>).
  
! Other than the changes listed in "Incompatible changes" at the top of
! this document, there are no known transition issues.
  
  
--- 148,160 ----
  <http://sourceforge.net/project/showfiles.php?group_id=61702>).
  
!  o To take advantage of improvements in the tokenizing engine at the core
!    of SpamBayes, you may wish to do a full retrain; this is not necessary,
!    however.
! 
!  o If you use the scripts in the testtools directory, you should examine
!    how these scripts have changed (some command line switches are altered).
! 
! See also the the changes listed in "Incompatible changes" at the top of
! this document.
  
  
***************
*** 64,68 ****
  ===================
  The following bugs tracked via the Sourceforge system were fixed:
! 
  
  A URL containing the details of these bugs can be made by appending the
--- 162,166 ----
  ===================
  The following bugs tracked via the Sourceforge system were fixed:
! 818871, 833439, 803798, 787676, 860410, 856628, 859215
  
  A URL containing the details of these bugs can be made by appending the
***************
*** 73,90 ****
  Feature Requests Added
  ======================
! No feature requests tracked via the Sourceforge system were added for this
! release.
! 
  
  Patches integrated
  ===================
! No patches tracked via the Sourceforge system were integrated for this
! release.
! 
  
  Deprecated Options
  ==================
  
! [add explanation of deprecated options here]
  
  The following options have been deprecated in this release:
--- 171,203 ----
  Feature Requests Added
  ======================
! The following feature requests tracked via the Sourceforge system were
! added for this release:
! 827138
  
  Patches integrated
  ===================
! The following patches tracked via the Sourceforge system were integrated
! for this release.
! 842464, 831388, 809008, 831388
  
  Deprecated Options
  ==================
  
! SpamBayes now has a method of noting options that are deprecated and which
! will not be available in future releases (it is likely that options will
! only be deprecated for one release before being removed).  Deprecated
! options will not be offered in the graphical interfaces (Outlook plugin
! and web interface), and will be listed in the "What's New" file (this
! file) for each release.
! 
! Deprecated options have the same name as previously, but now begin with
! "x-" (so "extract_dow" is now "x-extract_dow").  You can continue to use
! the original name (eg "extract_dow") in your configuration file, but will
! receive warnings in your log file or console window.  We recommend that you
! examine this output every time you upgrade SpamBayes to ensure that you are
! not using any newly deprecated options.
! 
! Discussion of why any particular option has been deprecated can be found in
! the spambayes-dev archives (at <http://mail.python.org/pipermail/spambayes-dev>).
  
  The following options have been deprecated in this release:
***************
*** 97,110 ****
  ========================
  
! [add explanation of experimental options and a pointer to the testing
! setup here]
  
  The following experimental options have been added in this release:
!   o [Tokenizer] search_for_habeas_headers
!   o [Tokenizer] reduce_habeas_headers
!   o [Classifier] use_bigrams
!   o [URLRetriever] slurp_urls
!   o [URLRetriever] cache_expiry_days
!   o [URLRetriever] cache_directory
!   o [URLRetriever] only_slurp_base
!   o [URLRetriever] web_prefix
--- 210,258 ----
  ========================
  
! SpamBayes now has a method of noting options that are experimental and
! which may be removed or made permanent in future releases (many experimental
! options will only be experimental for one release before being removed or
! fully integrated).  Experimental options will not be offered in the
! graphical interfaces (Outlook plugin and web interface), and will be
! listed in the "What's New" file (this file) for each release.
! 
! Experimental options begin with "x-" (as do deprecated options).  If you
! start using an experimental option and it later becomes permanent you can
! continue to use the "x-" name in your configuration file, but will
! receive warnings in your log file or console window.  We recommend that you
! examine this output every time you upgrade SpamBayes to ensure that you are
! using the correct name for all options.
! 
! Discussion of why experimental options and results from using them can be
! found in the spambayes-dev archives (at
! <http://mail.python.org/pipermail/spambayes-dev>).  Ideally, we would like
! users to test these options out on their mail and let us know the results.
! This can be as simple as turning on the option and emailing
! spambayes@python.org with anacdotal results after a period of time, or the
! full testtools scripts can be used.  For details about using these, please
! read the "README-DEVEL.txt" file that comes with the SpamBayes archive.
! 
! Experimental options are always turned off by default.
  
  The following experimental options have been added in this release:
!   o [Tokenizer] x-search_for_habeas_headers
!   o [Tokenizer] x-reduce_habeas_headers
!     These generate tokens based on the Habeas headers (see
!     <http://habeas.com> for more details).
!   
!   o [Classifier] x-use_bigrams
!     By default, SpamBayes uses unigrams tokens that are basically
!     single words (split on whitespace).  This option enables both unigrams
!     and bigrams (pairs of words), but uses a 'tiling' scheme, where only
!     the set of unigrams and bigrams that have the strongest effect on
!     the message are used.
! 
!   o [URLRetriever] x-slurp_urls
!   o [URLRetriever] x-cache_expiry_days
!   o [URLRetriever] x-cache_directory
!   o [URLRetriever] x-only_slurp_base
!   o [URLRetriever] x-web_prefix
!     If these are used, if a message is scored as 'unsure', and could use
!     more tokens in it's classification, then text from any URLs in the
!     message is retrieved and used, if it makes a difference to the
!     classification.



From anadelonbrin at users.sourceforge.net  Mon Dec 29 00:20:36 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Mon Dec 29 00:20:39 2003
Subject: [Spambayes-checkins] spambayes README.txt,1.61,1.62
Message-ID: <E1AappU-0003Ej-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv12439

Modified Files:
	README.txt 
Log Message:
A couple of very minor improvements suggested by Neal Stoughton.


Index: README.txt
===================================================================
RCS file: /cvsroot/spambayes/spambayes/README.txt,v
retrieving revision 1.61
retrieving revision 1.62
diff -C2 -d -r1.61 -r1.62
*** README.txt	5 Dec 2003 04:45:43 -0000	1.61
--- README.txt	29 Dec 2003 05:20:34 -0000	1.62
***************
*** 77,83 ****
  
  The first thing you need to do is run "python setup.py install" in the
! directory that you expanded the SpamBayes archive into.  This will install
! all the files that you need into the correct locations.  After this, you
! can delete that directory; it is no longer required.
  
  Before you begin
--- 77,85 ----
  
  The first thing you need to do is run "python setup.py install" in the
! directory that you expanded the SpamBayes archive into (to do this, you
! probably need to open up a console window/command prompt/DOS prompt, 
! and navigate to the appropriate directory with the "cd" command).  This
! will install all the files that you need into the correct locations.
! After this, you can delete that directory; it is no longer required.
  
  Before you begin
***************
*** 94,97 ****
--- 96,102 ----
  patterns which appear in ham and spam.  That information is then used
  during the filtering stage.  See the "Training" section below for details.
+ 
+ For more detailed instructions, please read the appropriate section below
+ (if you don't know, you probably want the POP3 Proxy section).
  
  Outlook plug-in



From montanaro at users.sourceforge.net  Mon Dec 29 17:03:41 2003
From: montanaro at users.sourceforge.net (Skip Montanaro)
Date: Mon Dec 29 17:03:51 2003
Subject: [Spambayes-checkins] spambayes NEWTRICKS.txt,1.7,1.8
Message-ID: <E1Ab5UD-0005rc-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv22535

Modified Files:
	NEWTRICKS.txt 
Log Message:
suggest some structural analysis of URLs


Index: NEWTRICKS.txt
===================================================================
RCS file: /cvsroot/spambayes/spambayes/NEWTRICKS.txt,v
retrieving revision 1.7
retrieving revision 1.8
diff -C2 -d -r1.7 -r1.8
*** NEWTRICKS.txt	29 Dec 2003 01:01:46 -0000	1.7
--- NEWTRICKS.txt	29 Dec 2003 22:03:38 -0000	1.8
***************
*** 55,56 ****
--- 55,73 ----
    a "not in dictionary" token.
  
+ - Structural analysis of URLs.  It seems a few things might yield some
+   interesting tokens:
+ 
+     * URLs containing a username portion
+ 
+     * URLs where the server is an ip address instead of a hostname
+ 
+     * URLs which refer to non-standard ports
+ 
+     * URLs in an href attribute which refer to a different web server than
+       the one surrounded by <a href="http://...">http://...</a>
+ 
+   Identity theft scams are increasing in frequency and seem to use one or
+   more of these schemes to hide the true nature of the URLs they attempt to
+   get you to click.
+ 
+ 



From anadelonbrin at users.sourceforge.net  Mon Dec 29 20:55:02 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Mon Dec 29 20:55:06 2003
Subject: [Spambayes-checkins] spambayes/windows autoconfigure.py,1.8,1.9
Message-ID: <E1Ab966-00080M-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/windows
In directory sc8-pr-cvs1:/tmp/cvs-serv30752/windows

Modified Files:
	autoconfigure.py 
Log Message:
Tidy up, and add a little gui for windows users to offer setup.  This should be
usable now - we just need people to test it ;)


Index: autoconfigure.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/windows/autoconfigure.py,v
retrieving revision 1.8
retrieving revision 1.9
diff -C2 -d -r1.8 -r1.9
*** autoconfigure.py	16 Dec 2003 05:06:34 -0000	1.8
--- autoconfigure.py	30 Dec 2003 01:54:59 -0000	1.9
***************
*** 15,23 ****
  
  To do:
!  o Establish which mail client(s) are to be setup.
!  o Locate the appropriate configuration directory
!    (e.g. for Eudora this is probably either the application directory,
!    or c:\documents and settings\username\application data\qualcomm\eudora,
!    i.e. sh_appdata\qualcomm\eudora)
   o This will create some unnecessary proxies in some cases.  For example,
     if I have my client set up to get mail from pop.example.com for the
--- 15,19 ----
  
  To do:
!  o Establish which mail client(s) are installed in a more clever way.
   o This will create some unnecessary proxies in some cases.  For example,
     if I have my client set up to get mail from pop.example.com for the
***************
*** 70,76 ****
  import ConfigParser
  
! # Allow for those without SpamBayes on their PYTHONPATH
! sys.path.insert(-1, os.getcwd())
! sys.path.insert(-1, os.path.dirname(os.getcwd()))
  
  from spambayes import OptionsClass
--- 66,80 ----
  import ConfigParser
  
! try:
!     import win32ui
!     import win32api
!     import win32con
!     from win32com.shell import shell, shellcon
! except ImportError:
!     # The ImportError is delayed until these are needed - if we
!     # did it here, the functions that don't need these would still
!     # fail.  (And having "import win32api" in lots of functions
!     # didn't seem to make much sense).
!     win32api = win32con = shell = shellcon = win32ui = None
  
  from spambayes import OptionsClass
***************
*** 423,432 ****
      """Configure OE to use the SpamBayes POP3 and SMTP proxies, and
      configure SpamBayes to proxy the servers that OE was connecting to."""
      # OE stores its configuration in the registry, not a file.
      key = "Software\\Microsoft\\Internet Account Manager\\Accounts"
  
-     import win32api
-     import win32con
- 
      translate = {("POP3 Server", "POP3 Port") : "pop3proxy",
                   ("SMTP Server", "SMTP Port") : "smtpproxy",
--- 427,438 ----
      """Configure OE to use the SpamBayes POP3 and SMTP proxies, and
      configure SpamBayes to proxy the servers that OE was connecting to."""
+     # Requires win32all to be available (or for someone to write a
+     # Mac version <wink>)
+     if win32api is None:
+         raise ImportError("win32 extensions required")
+ 
      # OE stores its configuration in the registry, not a file.
      key = "Software\\Microsoft\\Internet Account Manager\\Accounts"
  
      translate = {("POP3 Server", "POP3 Port") : "pop3proxy",
                   ("SMTP Server", "SMTP Port") : "smtpproxy",
***************
*** 554,560 ****
  
  def configure_pocomail(unused):
!     import win32api
!     import win32con
! 
      key = "Software\\Poco Systems Inc"
  
--- 560,566 ----
  
  def configure_pocomail(unused):
!     # Requires win32all to be available.
!     if win32api is None:
!         raise ImportError("win32 extensions required")
      key = "Software\\Poco Systems Inc"
  
***************
*** 668,673 ****
      the given mailer, to pass to the configure_* scripts
      above."""
!     import win32api
!     from win32com.shell import shell, shellcon
      if mailer in ["Outlook Express", "PocoMail"]:
          # Outlook Express and PocoMail can be configured without a
--- 674,681 ----
      the given mailer, to pass to the configure_* scripts
      above."""
!     # Requires win32all to be available, until someone
!     # fixes the function to look in the right places for *nix/Mac.
!     if win32api is None:
!         raise ImportError("win32 extensions required")
      if mailer in ["Outlook Express", "PocoMail"]:
          # Outlook Express and PocoMail can be configured without a
***************
*** 680,684 ****
                             "M2" : ("Opera%(sep)sOpera7",),
                             }
!     # We try with the username that the user uses [that's a lot of 'use'!]
      # for Windows, even though that might not be the same as their profile
      # names for mailers.  We can get smarter later.
--- 688,692 ----
                             "M2" : ("Opera%(sep)sOpera7",),
                             }
!     # We try with the username that the user uses
      # for Windows, even though that might not be the same as their profile
      # names for mailers.  We can get smarter later.
***************
*** 711,722 ****
  
  
  if __name__ == "__main__":
      pmail_ini_dir = "C:\\Program Files\\PMAIL\\MAIL\\ADMIN"
-     #configure_eudora(eudora_ini_dir)
-     #configure_mozilla(mozilla_ini_dir)
-     #configure_m2(m2_ini_dir)
-     #configure_outlook_express()
-     #configure_pocomail()
-     #configure_pegasus_mail(pmail_ini_dir)
      for mailer in ["Eudora", "Mozilla", "M2", "Outlook Express", "PocoMail"]:
!         print find_config_location(mailer)
--- 719,749 ----
  
  
+ def offer_to_configure(mailer):
+     """If the mailer appears to be installed, offer to set it up for
+     SpamBayes (and SpamBayes for it)."""
+     # Requires win32all to be available, or someone to write a version
+     # with a different gui.
+     if win32api is None:
+         raise ImportError("win32 extensions required")
+     # At the moment, the test we use to check if the mailer is installed
+     # is whether a valid path to the configuration file can be found.
+     # This is ok, except for those that are setup in the registry - there
+     # will always be a valid path, whether they are installed or not.
+     if find_config_location(mailer) is not None:
+         confirm_text = "Would you like %s setup for SpamBayes, and " \
+                        "SpamBayes setup with your %s settings?\n" \
+                        "(This is alpha software! We recommend that you " \
+                        "only do this if you know how to re-setup %s " \
+                        "if necessary.)" % (mailer, mailer, mailer)
+         ans = win32ui.MessageBox(confirm_text, "Configure?",
+                                  win32con.MB_YESNO)
+         if ans == win32con.IDYES:
+             configure(mailer)
+ 
+ 
  if __name__ == "__main__":
      pmail_ini_dir = "C:\\Program Files\\PMAIL\\MAIL\\ADMIN"
      for mailer in ["Eudora", "Mozilla", "M2", "Outlook Express", "PocoMail"]:
!         #print find_config_location(mailer)
!         #configure(mailer)
!         offer_to_configure(mailer)



From anadelonbrin at users.sourceforge.net  Mon Dec 29 20:57:20 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Mon Dec 29 20:57:23 2003
Subject: [Spambayes-checkins] spambayes/windows/py2exe setup_all.py, 1.12,
	1.13
Message-ID: <E1Ab98K-00086D-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/windows/py2exe
In directory sc8-pr-cvs1:/tmp/cvs-serv31129/windows/py2exe

Modified Files:
	setup_all.py 
Log Message:
Build the little auto-setup application (and install it), too.

If we can convince people to test this and be more sure that it works nicely, then it
could be optionally run at the end of the install process.


Index: setup_all.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/windows/py2exe/setup_all.py,v
retrieving revision 1.12
retrieving revision 1.13
diff -C2 -d -r1.12 -r1.13
*** setup_all.py	29 Dec 2003 03:07:34 -0000	1.12
--- setup_all.py	30 Dec 2003 01:57:18 -0000	1.13
***************
*** 31,35 ****
  py2exe_options = dict(
      packages = "spambayes.resources,encodings",
!     excludes = "win32ui,pywin,pywin.debugger", # pywin is a package, and still seems to be included.
      includes = "dialogs.resources.dialogs", # Outlook dynamic dialogs
      dll_excludes = "dapi.dll,mapi32.dll",
--- 31,35 ----
  py2exe_options = dict(
      packages = "spambayes.resources,encodings",
!     excludes = "pywin,pywin.debugger", # pywin is a package, and still seems to be included.
      includes = "dialogs.resources.dialogs", # Outlook dynamic dialogs
      dll_excludes = "dapi.dll,mapi32.dll",
***************
*** 106,109 ****
--- 106,113 ----
                        (1010, os.path.join(sb_top_dir, r"windows\resources\sb-stopped.ico"))],
  )
+ autoconfigure = dict(
+     dest_base = "bin/setup_server",
+     script = os.path.join(sb_top_dir, "windows", "autoconfigure.py"),
+ )
  
  outlook_data_files = [
***************
*** 140,144 ****
        console=[sb_server, sb_upload, outlook_dump_props],
        # The taskbar
!       windows=[pop3proxy_tray, outlook_addin_register],
        # and the misc data files
        data_files = outlook_data_files + proxy_data_files + common_data_files,
--- 144,148 ----
        console=[sb_server, sb_upload, outlook_dump_props],
        # The taskbar
!       windows=[pop3proxy_tray, outlook_addin_register, autoconfigure],
        # and the misc data files
        data_files = outlook_data_files + proxy_data_files + common_data_files,



From anadelonbrin at users.sourceforge.net  Mon Dec 29 20:57:21 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Mon Dec 29 20:57:26 2003
Subject: [Spambayes-checkins] spambayes/windows spambayes.iss,1.5,1.6
Message-ID: <E1Ab98L-00086I-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/windows
In directory sc8-pr-cvs1:/tmp/cvs-serv31129/windows

Modified Files:
	spambayes.iss 
Log Message:
Build the little auto-setup application (and install it), too.

If we can convince people to test this and be more sure that it works nicely, then it
could be optionally run at the end of the install process.


Index: spambayes.iss
===================================================================
RCS file: /cvsroot/spambayes/spambayes/windows/spambayes.iss,v
retrieving revision 1.5
retrieving revision 1.6
diff -C2 -d -r1.5 -r1.6
*** spambayes.iss	23 Dec 2003 06:04:28 -0000	1.5
--- spambayes.iss	30 Dec 2003 01:57:18 -0000	1.6
***************
*** 31,34 ****
--- 31,35 ----
  Source: "py2exe\dist\bin\sb_tray.exe"; DestDir: "{app}\bin"; Check: InstallingProxy; Flags: ignoreversion
  Source: "py2exe\dist\bin\sb_upload.exe"; DestDir: "{app}\bin"; Check: InstallingProxy; Flags: ignoreversion
+ Source: "py2exe\dist\bin\setup_server.exe"; DestDir: "{app}\bin"; Check: InstallingProxy; Flags: ignoreversion
  Source: "py2exe\dist\docs\sb_server\readme_proxy.html"; DestDir: "{app}\docs\sb_server"; Check: InstallingProxy; Flags: isreadme
  Source: "py2exe\dist\docs\sb_server\*.*"; DestDir: "{app}\docs\sb_server"; Check: InstallingProxy; Flags: recursesubdirs



From anadelonbrin at users.sourceforge.net  Mon Dec 29 21:15:02 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Mon Dec 29 21:15:05 2003
Subject: [Spambayes-checkins] spambayes/spambayes ImapUI.py,1.30,1.31
Message-ID: <E1Ab9PS-0000UV-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv1848/spambayes

Modified Files:
	ImapUI.py 
Log Message:
Quote folder names when displaying them - otherwise if the folder names contained
certain characters it could result in bad html (if the name was ">foo", for example).


Index: ImapUI.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/ImapUI.py,v
retrieving revision 1.30
retrieving revision 1.31
diff -C2 -d -r1.30 -r1.31
*** ImapUI.py	16 Dec 2003 05:06:34 -0000	1.30
--- ImapUI.py	30 Dec 2003 02:15:00 -0000	1.31
***************
*** 43,46 ****
--- 43,47 ----
  
  import re
+ import urllib
  
  import UserInterface
***************
*** 266,269 ****
--- 267,271 ----
          firstRow = True
          for folder in available_folders:
+             folder = urllib.quote(folder)
              folderRow = self.html.configTable.folderRow.clone()
              if firstRow:



From anadelonbrin at users.sourceforge.net  Mon Dec 29 21:20:51 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Mon Dec 29 21:20:54 2003
Subject: [Spambayes-checkins] spambayes/spambayes ImapUI.py,1.31,1.32
Message-ID: <E1Ab9V5-0000iZ-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv2754/spambayes

Modified Files:
	ImapUI.py 
Log Message:
Opps.  I meant cgi.escape, not urllib.quote.


Index: ImapUI.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/ImapUI.py,v
retrieving revision 1.31
retrieving revision 1.32
diff -C2 -d -r1.31 -r1.32
*** ImapUI.py	30 Dec 2003 02:15:00 -0000	1.31
--- ImapUI.py	30 Dec 2003 02:20:49 -0000	1.32
***************
*** 43,47 ****
  
  import re
! import urllib
  
  import UserInterface
--- 43,47 ----
  
  import re
! import cgi
  
  import UserInterface
***************
*** 267,271 ****
          firstRow = True
          for folder in available_folders:
!             folder = urllib.quote(folder)
              folderRow = self.html.configTable.folderRow.clone()
              if firstRow:
--- 267,271 ----
          firstRow = True
          for folder in available_folders:
!             folder = cgi.escape(folder)
              folderRow = self.html.configTable.folderRow.clone()
              if firstRow:



From tim_one at users.sourceforge.net  Tue Dec 30 11:26:34 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Tue Dec 30 11:26:39 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000 msgstore.py,1.82,1.83
Message-ID: <E1AbMhW-0007z8-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000
In directory sc8-pr-cvs1:/tmp/cvs-serv30644/Outlook2000

Modified Files:
	msgstore.py 
Log Message:
Whitespace normalization.


Index: msgstore.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/msgstore.py,v
retrieving revision 1.82
retrieving revision 1.83
diff -C2 -d -r1.82 -r1.83
*** msgstore.py	19 Dec 2003 06:25:15 -0000	1.82
--- msgstore.py	30 Dec 2003 16:26:32 -0000	1.83
***************
*** 49,53 ****
  test_suite_failure = None
  # Set to the number of times we should fail, or None for all times.
! test_suite_failure_count = None 
  # Sometimes the test suite will request that we simulate MAPI errors.
  def help_test_suite(checkpoint_name):
--- 49,53 ----
  test_suite_failure = None
  # Set to the number of times we should fail, or None for all times.
! test_suite_failure_count = None
  # Sometimes the test suite will request that we simulate MAPI errors.
  def help_test_suite(checkpoint_name):



From tim_one at users.sourceforge.net  Tue Dec 30 11:26:35 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Tue Dec 30 11:26:42 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000/sandbox dump_props.py,
	1.13, 1.14
Message-ID: <E1AbMhX-0007zF-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000/sandbox
In directory sc8-pr-cvs1:/tmp/cvs-serv30644/Outlook2000/sandbox

Modified Files:
	dump_props.py 
Log Message:
Whitespace normalization.


Index: dump_props.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/sandbox/dump_props.py,v
retrieving revision 1.13
retrieving revision 1.14
diff -C2 -d -r1.13 -r1.14
*** dump_props.py	20 Dec 2003 10:38:56 -0000	1.13
--- dump_props.py	30 Dec 2003 16:26:33 -0000	1.14
***************
*** 224,228 ****
          stream_name = tempfile.mktemp("spambayes")
          stream = open(stream_name, "w")
!         
      if not folder_name:
          folder_name = "Inbox" # Assume this exists!
--- 224,228 ----
          stream_name = tempfile.mktemp("spambayes")
          stream = open(stream_name, "w")
! 
      if not folder_name:
          folder_name = "Inbox" # Assume this exists!



From tim_one at users.sourceforge.net  Tue Dec 30 11:26:35 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Tue Dec 30 11:26:44 2003
Subject: [Spambayes-checkins] spambayes/Outlook2000/dialogs
	FolderSelector.py, 1.34, 1.35
Message-ID: <E1AbMhX-0007zB-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/Outlook2000/dialogs
In directory sc8-pr-cvs1:/tmp/cvs-serv30644/Outlook2000/dialogs

Modified Files:
	FolderSelector.py 
Log Message:
Whitespace normalization.


Index: FolderSelector.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/Outlook2000/dialogs/FolderSelector.py,v
retrieving revision 1.34
retrieving revision 1.35
diff -C2 -d -r1.34 -r1.35
*** FolderSelector.py	20 Dec 2003 05:38:50 -0000	1.34
--- FolderSelector.py	30 Dec 2003 16:26:32 -0000	1.35
***************
*** 461,465 ****
              win32gui.SendMessage(self.list, commctrl.TVM_SETITEM,
                                   0, buf)
!         
      def _CheckSelectionsValid(self, is_close = False):
          if self.in_check_selections_valid:
--- 461,465 ----
              win32gui.SendMessage(self.list, commctrl.TVM_SETITEM,
                                   0, buf)
! 
      def _CheckSelectionsValid(self, is_close = False):
          if self.in_check_selections_valid:



From tim_one at users.sourceforge.net  Tue Dec 30 11:26:35 2003
From: tim_one at users.sourceforge.net (Tim Peters)
Date: Tue Dec 30 11:26:47 2003
Subject: [Spambayes-checkins] spambayes/spambayes Options.py, 1.96,
	1.97 TestToolsUI.py, 1.1, 1.2 classifier.py, 1.19,
	1.20 oe_mailbox.py, 1.4, 1.5 tokenizer.py, 1.26, 1.27
Message-ID: <E1AbMhX-0007zN-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv30644/spambayes

Modified Files:
	Options.py TestToolsUI.py classifier.py oe_mailbox.py 
	tokenizer.py 
Log Message:
Whitespace normalization.


Index: Options.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/Options.py,v
retrieving revision 1.96
retrieving revision 1.97
diff -C2 -d -r1.96 -r1.97
*** Options.py	22 Dec 2003 04:19:56 -0000	1.96
--- Options.py	30 Dec 2003 16:26:33 -0000	1.97
***************
*** 186,190 ****
       for messages that contain only a single URL and no other text.""",
       BOOLEAN, RESTORE),
!     
      ("x-cache_expiry_days", "Number of days to store URLs in cache", 7,
       """(EXPERIMENTAL) This is the number of days that local cached copies
--- 186,190 ----
       for messages that contain only a single URL and no other text.""",
       BOOLEAN, RESTORE),
! 
      ("x-cache_expiry_days", "Number of days to store URLs in cache", 7,
       """(EXPERIMENTAL) This is the number of days that local cached copies

Index: TestToolsUI.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/TestToolsUI.py,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** TestToolsUI.py	22 Dec 2003 04:38:11 -0000	1.1
--- TestToolsUI.py	30 Dec 2003 16:26:33 -0000	1.2
***************
*** 91,95 ****
                                ('TestToolsUI', 'source'),
                                ('TestToolsUI', 'n'),)
!         
          option_choice = self._buildConfigPageBody(\
              configTable, testtools_ini_map)
--- 91,95 ----
                                ('TestToolsUI', 'source'),
                                ('TestToolsUI', 'n'),)
! 
          option_choice = self._buildConfigPageBody(\
              configTable, testtools_ini_map)
***************
*** 224,228 ****
                     ["deleted items", "drafts", "folders",
                      "offline", "outbox", "pop3uidl",]:
!                   continue
                  elif dbx.lower().find("spam") == -1:
                      spamdirs.append(dbx)
--- 224,228 ----
                     ["deleted items", "drafts", "folders",
                      "offline", "outbox", "pop3uidl",]:
!                     continue
                  elif dbx.lower().find("spam") == -1:
                      spamdirs.append(dbx)
***************
*** 293,297 ****
          sys.stdout = StringIO.StringIO()
          sys.stderr = StringIO.StringIO()
!         
          interesting = filter(lambda line: line.startswith('-> '), ifile)
          ifile.close()
--- 293,297 ----
          sys.stdout = StringIO.StringIO()
          sys.stderr = StringIO.StringIO()
! 
          interesting = filter(lambda line: line.startswith('-> '), ifile)
          ifile.close()

Index: classifier.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/classifier.py,v
retrieving revision 1.19
retrieving revision 1.20
diff -C2 -d -r1.19 -r1.20
*** classifier.py	17 Dec 2003 09:09:52 -0000	1.19
--- classifier.py	30 Dec 2003 16:26:33 -0000	1.20
***************
*** 211,215 ****
          s_cut = options["Categorization", "spam_cutoff"]
  
!         # Get the raw score.        
          prob, clues = self.chi2_spamprob(wordstream, True)
  
--- 211,215 ----
          s_cut = options["Categorization", "spam_cutoff"]
  
!         # Get the raw score.
          prob, clues = self.chi2_spamprob(wordstream, True)
  

Index: oe_mailbox.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/oe_mailbox.py,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** oe_mailbox.py	22 Dec 2003 02:28:26 -0000	1.4
--- oe_mailbox.py	30 Dec 2003 16:26:33 -0000	1.5
***************
*** 441,449 ****
          fh_entries = dbxFileHeader.FH_ENTRIES
          fh_ptr = dbxFileHeader.FH_TREE_ROOT_NODE_PTR
!         
          info = dbxFileInfo(dbxStream, header.getEntry(file_info_len))
          entries = header.getEntry(fh_entries)
          address = header.getEntry(fh_ptr)
!         
          if address and entries:
              tree = dbxTree(dbxStream, address, entries)
--- 441,449 ----
          fh_entries = dbxFileHeader.FH_ENTRIES
          fh_ptr = dbxFileHeader.FH_TREE_ROOT_NODE_PTR
! 
          info = dbxFileInfo(dbxStream, header.getEntry(file_info_len))
          entries = header.getEntry(fh_entries)
          address = header.getEntry(fh_ptr)
! 
          if address and entries:
              tree = dbxTree(dbxStream, address, entries)
***************
*** 480,484 ****
          # same format).
          raise NotImplementedError
!     
      reg = win32api.RegOpenKeyEx(win32con.HKEY_USERS, "")
      user_index = 0
--- 480,484 ----
          # same format).
          raise NotImplementedError
! 
      reg = win32api.RegOpenKeyEx(win32con.HKEY_USERS, "")
      user_index = 0
***************
*** 506,510 ****
                  break
              identity_index += 1
!             
              subkey_name = "%s\\%s\\%s" % (user_name, identity_name,
                                            "Software\\Microsoft\\Outlook " \
--- 506,510 ----
                  break
              identity_index += 1
! 
              subkey_name = "%s\\%s\\%s" % (user_name, identity_name,
                                            "Software\\Microsoft\\Outlook " \
***************
*** 602,606 ****
          sys.exit()
  
!     MAILBOX_DIR = args[0]  
  
      files = [os.path.join(MAILBOX_DIR, file) for file in \
--- 602,606 ----
          sys.exit()
  
!     MAILBOX_DIR = args[0]
  
      files = [os.path.join(MAILBOX_DIR, file) for file in \

Index: tokenizer.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/tokenizer.py,v
retrieving revision 1.26
retrieving revision 1.27
diff -C2 -d -r1.26 -r1.27
*** tokenizer.py	22 Dec 2003 16:55:05 -0000	1.26
--- tokenizer.py	30 Dec 2003 16:26:33 -0000	1.27
***************
*** 1037,1041 ****
          self.setup_done = False
          self.do_slurp = True
!         
      def setup(self):
          # Can't import this at the top because it's circular.
--- 1037,1041 ----
          self.setup_done = False
          self.do_slurp = True
! 
      def setup(self):
          # Can't import this at the top because it's circular.
***************
*** 1061,1065 ****
              # Build a new opener without any proxy information.
              opener = urllib2.build_opener(urllib2.HTTPHandler)
!          
          # Install it
          urllib2.install_opener(opener)
--- 1061,1065 ----
              # Build a new opener without any proxy information.
              opener = urllib2.build_opener(urllib2.HTTPHandler)
! 
          # Install it
          urllib2.install_opener(opener)
***************
*** 1142,1146 ****
             self.do_slurp):
              return tokens
!         
          # We don't want to do this recursively and check URLs
          # on webpages, so we have this little cheat.
--- 1142,1146 ----
             self.do_slurp):
              return tokens
! 
          # We don't want to do this recursively and check URLs
          # on webpages, so we have this little cheat.
***************
*** 1201,1205 ****
          url_key = URL_KEY_RE.sub('_', url)
          cached_message = self.urlCorpus.get(url_key)
!         
          if cached_message is None:
              # We're going to ignore everything that isn't text/html,
--- 1201,1205 ----
          url_key = URL_KEY_RE.sub('_', url)
          cached_message = self.urlCorpus.get(url_key)
! 
          if cached_message is None:
              # We're going to ignore everything that isn't text/html,
***************
*** 1210,1214 ****
                  self.bad_urls["url:non_html"] += (url,)
                  return ["url:non_html"]
!             
              try:
                  if options["globals", "verbose"]:
--- 1210,1214 ----
                  self.bad_urls["url:non_html"] += (url,)
                  return ["url:non_html"]
! 
              try:
                  if options["globals", "verbose"]:
***************
*** 1222,1226 ****
                  self.bad_urls["url:unknown_error"] += (url,)
                  return ["url:unknown_error"]
!             
              # Anything that isn't text/html is ignored
              content_type = f.info().get('content-type')
--- 1222,1226 ----
                  self.bad_urls["url:unknown_error"] += (url,)
                  return ["url:unknown_error"]
! 
              # Anything that isn't text/html is ignored
              content_type = f.info().get('content-type')



From montanaro at users.sourceforge.net  Tue Dec 30 16:08:02 2003
From: montanaro at users.sourceforge.net (Skip Montanaro)
Date: Tue Dec 30 16:08:09 2003
Subject: [Spambayes-checkins] spambayes/testtools Makefile,1.2,1.3
Message-ID: <E1AbR5u-0006Ge-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/testtools
In directory sc8-pr-cvs1:/tmp/cvs-serv24050a

Modified Files:
	Makefile 
Log Message:
build $(TRIAL)s.txt before $(BASE)s.txt in most cases on the assumption that
errors are more likely to turn up in experimental code, so you'll see them
that much earlier.



Index: Makefile
===================================================================
RCS file: /cvsroot/spambayes/spambayes/testtools/Makefile,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** Makefile	25 Nov 2003 02:03:03 -0000	1.2
--- Makefile	30 Dec 2003 21:08:00 -0000	1.3
***************
*** 47,51 ****
  all : $(BASE)-$(TRIAL).txt
  
! $(BASE)-$(TRIAL).txt : $(BASE)s.txt $(TRIAL)s.txt
  	$(CMP) $(BASE)s.txt $(TRIAL)s.txt > $(BASE)-$(TRIAL).txt
  
--- 47,51 ----
  all : $(BASE)-$(TRIAL).txt
  
! $(BASE)-$(TRIAL).txt : $(TRIAL)s.txt $(BASE)s.txt
  	$(CMP) $(BASE)s.txt $(TRIAL)s.txt > $(BASE)-$(TRIAL).txt
  



From anadelonbrin at users.sourceforge.net  Tue Dec 30 17:41:52 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Tue Dec 30 17:41:55 2003
Subject: [Spambayes-checkins] spambayes/spambayes Stats.py,1.3,1.4
Message-ID: <E1AbSYi-0002NS-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv9133/spambayes

Modified Files:
	Stats.py 
Log Message:
Output plurals correctly in stats information.

We printed out false positive numbers in the false negatives section,
and vice versa.  Fix.


Index: Stats.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/Stats.py,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** Stats.py	11 Dec 2003 18:44:00 -0000	1.3
--- Stats.py	30 Dec 2003 22:41:49 -0000	1.4
***************
*** 1,5 ****
  #! /usr/bin/env python
  
! """message.py - Core Spambayes classes.
  
  Classes:
--- 1,5 ----
  #! /usr/bin/env python
  
! """Stats.py - Spambayes statistics class.
  
  Classes:
***************
*** 64,72 ****
                  self.cls_spam += 1
                  if m.t == 0:
!                     self.fn += 1
              elif m.c == 'h':
                  self.cls_ham += 1
                  if m.t == 1:
!                     self.fp += 1
              elif m.c == 'u':
                  self.cls_unsure += 1
--- 64,72 ----
                  self.cls_spam += 1
                  if m.t == 0:
!                     self.fp += 1
              elif m.c == 'h':
                  self.cls_ham += 1
                  if m.t == 1:
!                     self.fn += 1
              elif m.c == 'u':
                  self.cls_unsure += 1
***************
*** 95,109 ****
              }
          format_dict.update(self.__dict__)
!         push("SpamBayes has processed %(num_seen)d messages - " \
               "%(cls_ham)d (%(perc_ham).0f%%) good, " \
               "%(cls_spam)d (%(perc_spam).0f%%) spam " \
               "and %(cls_unsure)d (%(perc_unsure)d%%) unsure." % format_dict)
!         push("%(trn_ham)d message(s) were manually " \
!              "classified as good (with %(fp)d " \
!              "being false positives)." % format_dict)
!         push("%(trn_spam)d message(s) were manually " \
!              "classified as spam (with %(fn)d " \
!              "being false negatives)." % format_dict)
!         push("%(trn_unsure_ham)d unsure message(s) were manually " \
               "identified as good, and %(trn_unsure_spam)d as spam." \
               % format_dict)
--- 95,124 ----
              }
          format_dict.update(self.__dict__)
!         # Figure out plurals
!         for num, key in [(self.total, "sp1"), (self.trn_ham, "sp2"),
!                          (self.trn_spam, "sp3"),
!                          (self.trn_unsure_ham, "sp4"),
!                          (self.fp, "sp5"), (self.fn, "sp6")]:
!             if num == 1:
!                 format_dict[key] = ''
!             else:
!                 format_dict[key] = 's'
!         for num, key in [(self.fp, "wp1"), (self.fn, "wp2")]:
!             if num == 1:
!                 format_dict[key] = 'was a'
!             else:
!                 format_dict[key] = 'were'
!             
!         push("SpamBayes has processed %(num_seen)d message%(sp1)s - " \
               "%(cls_ham)d (%(perc_ham).0f%%) good, " \
               "%(cls_spam)d (%(perc_spam).0f%%) spam " \
               "and %(cls_unsure)d (%(perc_unsure)d%%) unsure." % format_dict)
!         push("%(trn_ham)d message%(sp2)s were manually " \
!              "classified as good (%(fp)d %(wp1)s false positive%(sp5)s)." \
!              % format_dict)
!         push("%(trn_spam)d message%(sp3)s were manually " \
!              "classified as spam (%(fn)d %(wp2)s false negative%(sp6)s)." \
!              % format_dict)
!         push("%(trn_unsure_ham)d unsure message%(sp4)s were manually " \
               "identified as good, and %(trn_unsure_spam)d as spam." \
               % format_dict)



From anadelonbrin at users.sourceforge.net  Tue Dec 30 17:57:26 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Tue Dec 30 17:57:30 2003
Subject: [Spambayes-checkins] spambayes/spambayes ImapUI.py, 1.32,
	1.33 ProxyUI.py, 1.35, 1.36
Message-ID: <E1AbSnm-0003KS-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv12788/spambayes

Modified Files:
	ImapUI.py ProxyUI.py 
Log Message:
As Richie pointed out, the status message was only updated when the
state was recreated.

Fix this so that it's recreated each time the home page is displayed.


Index: ImapUI.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/ImapUI.py,v
retrieving revision 1.32
retrieving revision 1.33
diff -C2 -d -r1.32 -r1.33
*** ImapUI.py	30 Dec 2003 02:20:49 -0000	1.32
--- ImapUI.py	30 Dec 2003 22:57:23 -0000	1.33
***************
*** 126,129 ****
--- 126,130 ----
      def onHome(self):
          """Serve up the homepage."""
+         state.buildStatusStrings()
          stateDict = self.classifier.__dict__.copy()
          stateDict["warning"] = ""

Index: ProxyUI.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/ProxyUI.py,v
retrieving revision 1.35
retrieving revision 1.36
diff -C2 -d -r1.35 -r1.36
*** ProxyUI.py	29 Dec 2003 03:41:10 -0000	1.35
--- ProxyUI.py	30 Dec 2003 22:57:23 -0000	1.36
***************
*** 158,161 ****
--- 158,162 ----
      def onHome(self):
          """Serve up the homepage."""
+         state.buildStatusStrings()
          stateDict = state.__dict__.copy()
          stateDict.update(state.bayes.__dict__)



From anadelonbrin at users.sourceforge.net  Tue Dec 30 17:57:25 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Tue Dec 30 17:57:32 2003
Subject: [Spambayes-checkins] spambayes/scripts sb_server.py,1.14,1.15
Message-ID: <E1AbSnl-0003KO-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/scripts
In directory sc8-pr-cvs1:/tmp/cvs-serv12788/scripts

Modified Files:
	sb_server.py 
Log Message:
As Richie pointed out, the status message was only updated when the
state was recreated.

Fix this so that it's recreated each time the home page is displayed.


Index: sb_server.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/scripts/sb_server.py,v
retrieving revision 1.14
retrieving revision 1.15
diff -C2 -d -r1.14 -r1.15
*** sb_server.py	16 Dec 2003 05:06:33 -0000	1.14
--- sb_server.py	30 Dec 2003 22:57:23 -0000	1.15
***************
*** 688,705 ****
          self.proxyPortsString = ', '.join(map(_addressPortStr, self.proxyPorts))
  
!     def createWorkers(self):
!         """Using the options that were initialised in __init__ and then
!         possibly overridden by the driver code, create the Bayes object,
!         the Corpuses, the Trainers and so on."""
!         print "Loading database...",
!         if self.isTest:
!             self.useDB = True
!             options["Storage", "persistent_storage_file"] = \
!                         '_pop3proxy_test.pickle'   # This is never saved.
!         filename = options["Storage", "persistent_storage_file"]
!         filename = os.path.expanduser(filename)
!         self.bayes = storage.open_storage(filename, self.useDB)
! 
!         # Status message
          nspam = self.bayes.nspam
          nham = self.bayes.nham
--- 688,694 ----
          self.proxyPortsString = ', '.join(map(_addressPortStr, self.proxyPorts))
  
!     def buildStatusStrings(self):
!         """Build the status message(s) to display on the home page of the
!         web interface."""
          nspam = self.bayes.nspam
          nham = self.bayes.nham
***************
*** 726,729 ****
--- 715,733 ----
                             "will classify all messages as 'unsure', " \
                             "ready for you to train."
+ 
+     def createWorkers(self):
+         """Using the options that were initialised in __init__ and then
+         possibly overridden by the driver code, create the Bayes object,
+         the Corpuses, the Trainers and so on."""
+         print "Loading database...",
+         if self.isTest:
+             self.useDB = True
+             options["Storage", "persistent_storage_file"] = \
+                         '_pop3proxy_test.pickle'   # This is never saved.
+         filename = options["Storage", "persistent_storage_file"]
+         filename = os.path.expanduser(filename)
+         self.bayes = storage.open_storage(filename, self.useDB)
+ 
+         self.buildStatusStrings()
  
          # Don't set up the caches and training objects when running the self-test,



From anadelonbrin at users.sourceforge.net  Tue Dec 30 20:05:35 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Tue Dec 30 20:05:39 2003
Subject: [Spambayes-checkins] spambayes/scripts sb_pop3dnd.py,1.4,1.5
Message-ID: <E1AbUnn-0001mX-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/scripts
In directory sc8-pr-cvs1:/tmp/cvs-serv6846/scripts

Modified Files:
	sb_pop3dnd.py 
Log Message:
Clean out some cruft that isn't necessary with the latest version of twisted (1.1)

Add two new Message classes, one for messages that are stored in memory,
and one for messages that are re-generated each time the message is loaded.

Start our UIDs at 1, not 0, because Eudora likes this more.

Don't override the imported name "message".

Add a new folder - INBOX - this holds any messages from SpamBayes to the user.
(Having INBOX as an alias for Spam wasn't working well, and being able to communicate
within the confines of the mailer is nice, too).

Don't let the user set so much via the command line.  Use a config file, you lazy person.
(And I'll put in Skip's -o/--option flag at some point, so it can be done if really necessary).
(Not having the option there means that there is less trouble later on if -d/-D are changed
to -p and -d or something like that).

This actually appears to work with Eudora at the moment!

Index: sb_pop3dnd.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/scripts/sb_pop3dnd.py,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** sb_pop3dnd.py	16 Dec 2003 05:06:33 -0000	1.4
--- sb_pop3dnd.py	31 Dec 2003 01:05:32 -0000	1.5
***************
*** 116,119 ****
--- 116,120 ----
  from twisted.internet.defer import maybeDeferred
  from twisted.internet.protocol import ServerFactory
+ from twisted.protocols.imap4 import IMessage
  from twisted.protocols.imap4 import parseNestedParens, parseIdList
  from twisted.protocols.imap4 import IllegalClientResponse, IAccount
***************
*** 122,130 ****
  from twisted.protocols.imap4 import IMailboxListener, collapseNestedLists
  
! # Provide for those that don't have spambayes on their PYTHONPATH
! sys.path.insert(-1, os.path.dirname(os.getcwd()))
! 
  from spambayes.Options import options
- from spambayes.message import Message
  from spambayes.tokenizer import tokenize
  from spambayes import FileCorpus, Dibbler
--- 123,128 ----
  from twisted.protocols.imap4 import IMailboxListener, collapseNestedLists
  
! from spambayes import message
  from spambayes.Options import options
  from spambayes.tokenizer import tokenize
  from spambayes import FileCorpus, Dibbler
***************
*** 146,166 ****
  
  
! class IMAPFileMessage(FileCorpus.FileMessage):
!     '''IMAP Message that persists as a file system artifact.'''
  
!     def __init__(self, file_name, directory):
!         """Constructor(message file name, corpus directory name)."""
!         FileCorpus.FileMessage.__init__(self, file_name, directory)
!         self.id = file_name
!         self.directory = directory
!         self.date = imaplib.Time2Internaldate(time.time())[1:-1]
          self.clear_flags()
  
      # IMessage implementation
!     def getHeaders(self, negate, names):
          """Retrieve a group of message headers."""
          headers = {}
-         if not isinstance(names, tuple):
-             names = (names,)
          for header, value in self.items():
              if (header.upper() in names and not negate) or names == ():
--- 144,160 ----
  
  
! class IMAPMessage(message.Message):
!     '''IMAP Message base class.'''
!     __implements__ = (IMessage,)
  
!     def __init__(self, date):
!         message.Message.__init__(self)
!         self.date = date
          self.clear_flags()
  
      # IMessage implementation
!     def getHeaders(self, negate, *names):
          """Retrieve a group of message headers."""
          headers = {}
          for header, value in self.items():
              if (header.upper() in names and not negate) or names == ():
***************
*** 197,201 ****
          s.seek(0)
          return s
-         #return file(os.path.join(self.directory, self.id), "r")
  
      def getSize(self):
--- 191,194 ----
***************
*** 207,210 ****
--- 200,207 ----
          return self.id
  
+     def isMultipart(self):
+         """Indicate whether this message has subparts."""
+         return False
+     
      def getSubPart(self, part):
          """Retrieve a MIME sub-message
***************
*** 216,219 ****
--- 213,217 ----
          @return: The specified sub-part.
          """
+         raise NotImplementedError
  
      # IMessage implementation ends
***************
*** 321,416 ****
          return rfc822[:bmatch.start(2)]
  
-     def on(self, date1, date2):
-         "contained within the date"
-         raise NotImplementedError
-     def before(self, date1, date2):
-         "before the date"
-         raise NotImplementedError
-     def since(self, date1, date2):
-         "within or after the date"
-         raise NotImplementedError
- 
-     def string_contains(self, whole, sub):
-         return whole.find(sub) != -1
- 
-     def matches(self, criteria):
-         """Return True iff the messages matches the specified IMAP
-         criteria."""
-         match_tests = {"ALL" : [(True, True)],
-                        "ANSWERED" : [(self.answered, True)],
-                        "DELETED" : [(self.deleted, True)],
-                        "DRAFT" : [(self.draft, True)],
-                        "FLAGGED" : [(self.flagged, True)],
-                        "NEW" : [(self.recent, True), (self.seen, False)],
-                        "RECENT" : [(self.recent, True)],
-                        "SEEN" : [(self.seen, True)],
-                        "UNANSWERED" : [(self.answered, False)],
-                        "UNDELETED" : [(self.deleted, False)],
-                        "UNDRAFT" : [(self.draft, False)],
-                        "UNFLAGGED" : [(self.flagged, False)],
-                        "UNSEEN" : [(self.seen, False)],
-                        "OLD" : [(self.recent, False)],
-                        }
-         complex_tests = {"BCC" : (self.string_contains, self.get("Bcc")),
-                          "SUBJECT" : (self.string_contains, self.get("Subject")),
-                          "CC" : (self.string_contains, self.get("Cc")),
-                          "BODY" : (self.string_contains, self.body()),
-                          "TO" : (self.string_contains, self.get("To")),
-                          "TEXT" : (self.string_contains, self.as_string()),
-                          "FROM" : (self.string_contains, self.get("From")),
-                          "SMALLER" : (operator.lt, len(self.as_string())),
-                          "LARGER" : (operator.gt, len(self.as_string())),
-                          "BEFORE" : (self.before, self.date),
-                          "ON" : (self.on, self.date),
-                          "SENTBEFORE" : (self.before, self.get("Date")),
-                          "SENTON" : (self.on, self.get("Date")),
-                          "SENTSINCE" : (self.since, self.get("Date")),
-                          "SINCE" : (self.since, self.date),
-                          }
  
!         result = True
!         test = None
!         header = None
!         header_field = None
!         for c in criteria:
!             if match_tests.has_key(c) and test is None and header is None:
!                 for (test, result) in match_tests[c]:
!                     result = result and (test == result)
!             elif complex_tests.has_key(c) and test is None and header is None:
!                 test = complex_tests[c]
!             elif test is not None and header is None:
!                 result = result and test[0](test[1], c)
!                 test = None
!             elif c == "HEADER" and test is None:
!                 # the only criteria that uses the next _two_ elements
!                 header = c
!             elif test is None and header is not None and header_field is None:
!                 header_field = c
!             elif header is not None and header_field is not None and test is None:
!                 result = result and self.string_contains(self.get(header_field), c)
!                 header = None
!                 header_field = None
!         return result
! """
! Still to do:
!       <message set>  Messages with message sequence numbers
!                      corresponding to the specified message sequence
!                      number set
!       UID <message set>
!                      Messages with unique identifiers corresponding to
!                      the specified unique identifier set.
  
-       KEYWORD <flag> Messages with the specified keyword set.
-       UNKEYWORD <flag>
-                      Messages that do not have the specified keyword
-                      set.
  
!       NOT <search-key>
!                      Messages that do not match the specified search
!                      key.
  
!       OR <search-key1> <search-key2>
!                      Messages that match either search key.
! """
  
  
--- 319,344 ----
          return rfc822[:bmatch.start(2)]
  
  
! class DynamicIMAPMessage(IMAPMessage):
!     """An IMAP Message that may change each time it is loaded."""
!     def __init__(self, func):
!         date = imaplib.Time2Internaldate(time.time())[1:-1]
!         IMAPMessage.__init__(self, date)
!         self.func = func
!         self.load()
!     def load(self):
!         self.setPayload(self.func(body=True, headers=True))
  
  
! class IMAPFileMessage(IMAPMessage, FileCorpus.FileMessage):
!     '''IMAP Message that persists as a file system artifact.'''
  
!     def __init__(self, file_name, directory):
!         """Constructor(message file name, corpus directory name)."""
!         date = imaplib.Time2Internaldate(time.time())[1:-1]
!         IMAPMessage.__init__(self, date)
!         FileCorpus.FileMessage.__init__(self, file_name, directory)
!         self.id = file_name
!         self.directory = directory
  
  
***************
*** 452,456 ****
          # UIDs are required to be strictly ascending.
          if len(self.storage.keys()) == 0:
!             self.nextUID = 0
          else:
              self.nextUID = long(self.storage.keys()[-1]) + 1
--- 380,384 ----
          # UIDs are required to be strictly ascending.
          if len(self.storage.keys()) == 0:
!             self.nextUID = 1
          else:
              self.nextUID = long(self.storage.keys()[-1]) + 1
***************
*** 474,482 ****
          return reply
  
!     def getUID(self, message):
          """Return the UID of a message in the mailbox."""
          # Note that IMAP messages are 1-based, our messages are 0-based
          d = self.storage
!         return long(d.keys()[message - 1])
  
      def getFlags(self):
--- 402,410 ----
          return reply
  
!     def getUID(self, msg):
          """Return the UID of a message in the mailbox."""
          # Note that IMAP messages are 1-based, our messages are 0-based
          d = self.storage
!         return long(d.keys()[msg - 1])
  
      def getFlags(self):
***************
*** 528,536 ****
          return answer
  
!     def addMessage(self, message, flags=(), date=None):
          """Add the given message to this mailbox."""
          msg = self.storage.makeMessage(self.getUIDNext(True))
          msg.date = date
!         msg.setPayload(message.read())
          self.storage.addMessage(msg)
          self.store(MessageSet(long(msg.id), long(msg.id)), flags, 1, True)
--- 456,464 ----
          return answer
  
!     def addMessage(self, content, flags=(), date=None):
          """Add the given message to this mailbox."""
          msg = self.storage.makeMessage(self.getUIDNext(True))
          msg.date = date
!         msg.setPayload(content.read())
          self.storage.addMessage(msg)
          self.store(MessageSet(long(msg.id), long(msg.id)), flags, 1, True)
***************
*** 599,603 ****
                  # Non-existant message.
                  continue
!             msg.load()
              yield (id, msg)
  
--- 527,533 ----
                  # Non-existant message.
                  continue
!             # Load the message, if necessary
!             if hasattr(msg, "load"):
!                 msg.load()
              yield (id, msg)
  
***************
*** 629,632 ****
--- 559,631 ----
  
  
+ class SpambayesInbox(SpambayesMailbox):
+     """A special mailbox that holds status messages from SpamBayes."""
+     def __init__(self, id):
+         IMAPMailbox.__init__(self, "INBOX", "spambayes", id)
+         self.UID_validity = id
+         self.nextUID = 1
+         self.unseen_count = 0
+         self.recent_count = 0
+         self.storage = {}
+         self.createMessages()
+ 
+     def buildStatusMessage(self, body=False, headers=False):
+         """Build a message containing the current status message.
+ 
+         If body is True, then return the body; if headers is True
+         return the headers.  If both are true, then return both
+         (and insert a newline between them).
+         """
+         msg = []
+         if headers:
+             msg.append("Subject:SpamBayes Status")
+             msg.append('From:"SpamBayes" <no-reply@localhost>')
+             if body:
+                 msg.append('\r\n')
+         if body:
+             state.buildStatusStrings()
+             msg.append(state.warning or "SpamBayes operating correctly.")
+         return "\r\n".join(msg)
+ 
+     def createMessages(self):
+         """Create the special messages that live in this mailbox."""
+         state.buildStatusStrings()
+         about = 'Subject: About SpamBayes\r\n' \
+                  'From: "SpamBayes" <no-reply@localhost>\r\n\r\n' \
+                  'See <http://spambayes.org>.\r\n'
+         date = imaplib.Time2Internaldate(time.time())[1:-1]
+         msg = IMAPMessage(date)
+         msg.setPayload(about)
+         self.addMessage(msg)
+         msg = DynamicIMAPMessage(self.buildStatusMessage)
+         self.addMessage(msg)
+         # XXX Add other messages here, for example
+         # XXX statistics
+         # XXX information from sb_server homepage about number
+         # XXX   of messages classified etc.
+ 
+     def isWriteable(self):
+         """Get the read/write status of the mailbox."""
+         return False
+ 
+     def addMessage(self, msg, flags=(), date=None):
+         """Add the given message to this mailbox."""
+         msg.id = self.getUIDNext(True)
+         self.storage[msg.id] = msg
+         d = defer.Deferred()
+         reactor.callLater(0, d.callback, self.storage.keys().index(msg.id))
+         return d
+ 
+     def expunge(self):
+         """Remove all messages flagged \\Deleted."""
+         # Mailbox is read-only.
+         return []
+ 
+     def store(self, messages, flags, mode, uid):
+         """Set the flags of one or more messages."""
+         # Mailbox is read-only.
+         return {}
+ 
+ 
  class Trainer(object):
      """Listens to a given mailbox and trains new messages as spam or
***************
*** 648,652 ****
      def newMessages(self, exists, recent):
          # We don't get passed the actual message, or the id of
!         # the message, of even the message number.  We just get
          # the total number of new/recent messages.
          # However, this function should be called _every_ time
--- 647,651 ----
      def newMessages(self, exists, recent):
          # We don't get passed the actual message, or the id of
!         # the message, or even the message number.  We just get
          # the total number of new/recent messages.
          # However, this function should be called _every_ time
***************
*** 663,678 ****
      """Account for Spambayes server."""
  
!     def __init__(self, id, ham, spam, unsure):
          MemoryAccount.__init__(self, id)
          self.mailboxes = {"SPAM" : spam,
                            "UNSURE" : unsure,
!                           "TRAIN_AS_HAM" : ham}
  
      def select(self, name, readwrite=1):
          # 'INBOX' is a special case-insensitive name meaning the
!         # primary mailbox for the user; we interpret this as an alias
!         # for 'spam'
!         if name.upper() == "INBOX":
!             name = "SPAM"
          return MemoryAccount.select(self, name, readwrite)
  
--- 662,676 ----
      """Account for Spambayes server."""
  
!     def __init__(self, id, ham, spam, unsure, inbox):
          MemoryAccount.__init__(self, id)
          self.mailboxes = {"SPAM" : spam,
                            "UNSURE" : unsure,
!                           "TRAIN_AS_HAM" : ham,
!                           "INBOX" : inbox}
  
      def select(self, name, readwrite=1):
          # 'INBOX' is a special case-insensitive name meaning the
!         # primary mailbox for the user; for our purposes this contains
!         # special messages from SpamBayes.
          return MemoryAccount.select(self, name, readwrite)
  
***************
*** 859,863 ****
  
  def setup():
!     # Setup app, boxes, trainers and account
      proxyListeners = []
      app = Application("SpambayesIMAPServer")
--- 857,862 ----
  
  def setup():
!     # Setup state, app, boxes, trainers and account
!     state.createWorkers()
      proxyListeners = []
      app = Application("SpambayesIMAPServer")
***************
*** 869,872 ****
--- 868,872 ----
      ham_train_box = SpambayesMailbox("TrainAsHam", 2,
                                       options["Storage", "ham_cache"])
+     inbox = SpambayesInbox(3)
  
      spam_trainer = Trainer(spam_box, True)
***************
*** 876,880 ****
  
      user_account = SpambayesAccount(options["imapserver", "username"],
!                                     ham_train_box, spam_box, unsure_box)
  
      # add IMAP4 server
--- 876,881 ----
  
      user_account = SpambayesAccount(options["imapserver", "username"],
!                                     ham_train_box, spam_box, unsure_box,
!                                     inbox)
  
      # add IMAP4 server
***************
*** 886,890 ****
  
      # add POP3 proxy
-     state.createWorkers()
      for (server, serverPort), proxyPort in zip(state.servers,
                                                 state.proxyPorts):
--- 887,890 ----
***************
*** 916,927 ****
          elif opt == '-b':
              launchUI = True
-         elif opt == '-d':   # dbm file
-             state.useDB = True
-             options["Storage", "persistent_storage_file"] = arg
-         elif opt == '-D':   # pickle file
-             state.useDB = False
-             options["Storage", "persistent_storage_file"] = arg
-         elif opt == '-u':
-             state.uiPort = int(arg)
  
      # Let the user know what they are using...
--- 916,919 ----



From anadelonbrin at users.sourceforge.net  Tue Dec 30 21:59:54 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Tue Dec 30 22:00:00 2003
Subject: [Spambayes-checkins] spambayes/scripts sb_pop3dnd.py,1.5,1.6
Message-ID: <E1AbWaQ-0006SX-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/scripts
In directory sc8-pr-cvs1:/tmp/cvs-serv24824/scripts

Modified Files:
	sb_pop3dnd.py 
Log Message:
Update comments.

Fix fetching an envelope (twisted changed from uppercase to lowercase)

Handle storing no flags.

Update the RETR'ing of messages to reflect what sb_server currently does.

<shock> This appears to work with Outlook Express now.  I think that's the
first time that it's fully worked!

Index: sb_pop3dnd.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/scripts/sb_pop3dnd.py,v
retrieving revision 1.5
retrieving revision 1.6
diff -C2 -d -r1.5 -r1.6
*** sb_pop3dnd.py	31 Dec 2003 01:05:32 -0000	1.5
--- sb_pop3dnd.py	31 Dec 2003 02:59:51 -0000	1.6
***************
*** 18,22 ****
  via the web interface, and you are ready to go.  Good messages will appear
  as per normal, but you will also have two new incoming folders, one for
! spam and one for ham.
  
  To train SpamBayes, use the spam folder, and the 'train_as_ham' folder.
--- 18,22 ----
  via the web interface, and you are ready to go.  Good messages will appear
  as per normal, but you will also have two new incoming folders, one for
! spam and one for unsure messages.
  
  To train SpamBayes, use the spam folder, and the 'train_as_ham' folder.
***************
*** 38,52 ****
  This SpamBayes application is designed to work with Outlook Express, and
  provide the same sort of ease of use as the Outlook plugin.  Although the
! majority of development and testing has been done with Outlook Express,
! any mail client that supports both IMAP and POP3 should be able to use this
! application - if the client enables the user to work with an IMAP account
! and POP3 account side-by-side (and move messages between them), then it
! should work equally as well as Outlook Express.
  
  This module includes the following classes:
   o IMAPFileMessage
   o IMAPFileMessageFactory
   o IMAPMailbox
   o SpambayesMailbox
   o Trainer
   o SpambayesAccount
--- 38,55 ----
  This SpamBayes application is designed to work with Outlook Express, and
  provide the same sort of ease of use as the Outlook plugin.  Although the
! majority of development and testing has been done with Outlook Express and
! Eudora, any mail client that supports both IMAP and POP3 should be able to
! use this application - if the client enables the user to work with an IMAP
! account and POP3 account side-by-side (and move messages between them),
! then it should work equally as well.
  
  This module includes the following classes:
+  o IMAPMessage
+  o DynamicIMAPMessage
   o IMAPFileMessage
   o IMAPFileMessageFactory
   o IMAPMailbox
   o SpambayesMailbox
+  o SpambayesInbox
   o Trainer
   o SpambayesAccount
***************
*** 61,65 ****
   o Message flags are currently not persisted, but should be.  The
     IMAPFileMessage class should be extended to do this.  The same
!    goes for the 'internaldate' of the message.
   o The RECENT flag should be unset at some point, but when?  The
     RFC says that a message is recent if this is the first session
--- 64,69 ----
   o Message flags are currently not persisted, but should be.  The
     IMAPFileMessage class should be extended to do this.  The same
!    goes for the 'internaldate' of the message.  These could be put
!    in the message info database, no doubt.
   o The RECENT flag should be unset at some point, but when?  The
     RFC says that a message is recent if this is the first session
***************
*** 80,83 ****
--- 84,92 ----
     that we kick off, and if it dies, we should die too.  Need to figure
     out how to do this in twisted.
+  o Apparently, twisted.internet.app is deprecated, and we should
+    use twisted.application instead.  Need to figure out what that means!
+  o We could have a distinction between messages classified as spam
+    and messages to train as spam.  At the moment we force users into
+    the 'incremental training' system available with the Outlook plug-in.
   o Suggestions?
  """
***************
*** 158,163 ****
          headers = {}
          for header, value in self.items():
!             if (header.upper() in names and not negate) or names == ():
!                 headers[header.upper()] = value
          return headers
  
--- 167,172 ----
          headers = {}
          for header, value in self.items():
!             if (header.lower() in names and not negate) or names == ():
!                 headers[header.lower()] = value
          return headers
  
***************
*** 547,551 ****
              elif mode == 1:
                  value = True
!             for flag in flags:
                  if flag == '(' or flag == ')':
                      continue
--- 556,560 ----
              elif mode == 1:
                  value = True
!             for flag in flags or (): # flags might be None
                  if flag == '(' or flag == ')':
                      continue
***************
*** 591,594 ****
--- 600,604 ----
          """Create the special messages that live in this mailbox."""
          state.buildStatusStrings()
+         # This about message could have a bit more content!
          about = 'Subject: About SpamBayes\r\n' \
                   'From: "SpamBayes" <no-reply@localhost>\r\n\r\n' \
***************
*** 604,607 ****
--- 614,620 ----
          # XXX information from sb_server homepage about number
          # XXX   of messages classified etc.
+         # XXX one with a link to the configuration page
+         # XXX   (or maybe even the configuration page itself,
+         # XXX    in html!)
  
      def isWriteable(self):
***************
*** 738,746 ****
      """
  
!     intercept_message = 'From: "Spambayes" <no-reply@localhost>\n' \
!                         'Subject: Spambayes Intercept\n\nA message ' \
!                         'was intercepted by Spambayes (it scored %s).\n' \
!                         '\nYou may find it in the Spam or Unsure ' \
!                         'folder.\n\n.\n'
  
      def __init__(self, clientSocket, serverName, serverPort, spam, unsure):
--- 751,763 ----
      """
  
!     # This message could be a bit more informative - it could at least
!     # say whether it's the spam or unsure folder.  It could give
!     # information about who the message was from, or what the subject
!     # was, if people thought that would be a good idea.
!     intercept_message = 'From: "Spambayes" <no-reply@localhost>\r\n' \
!                         'Subject: Spambayes Intercept\r\n\r\nA message ' \
!                         'was intercepted by Spambayes (it scored %s).\r\n' \
!                         '\r\nYou may find it in the Spam or Unsure ' \
!                         'folder.\r\n\r\n'
  
      def __init__(self, clientSocket, serverName, serverPort, spam, unsure):
***************
*** 789,817 ****
          pass it through.  If the result is an unsure or spam, move it
          to the appropriate IMAP folder."""
          # Use '\n\r?\n' to detect the end of the headers in case of
          # broken emails that don't use the proper line separators.
          if re.search(r'\n\r?\n', response):
              # Break off the first line, which will be '+OK'.
              ok, messageText = response.split('\n', 1)
  
!             prob = state.bayes.spamprob(tokenize(messageText))
!             if prob < options["Categorization", "ham_cutoff"]:
!                 # Return the +OK and the message with the header added.
!                 state.numHams += 1
!                 return ok + "\n" + messageText
!             elif prob > options["Categorization", "spam_cutoff"]:
!                 dest_folder = self.spam_folder
!                 state.numSpams += 1
!             else:
!                 dest_folder = self.unsure_folder
!                 state.numUnsure += 1
!             msg = StringIO.StringIO(messageText)
!             date = imaplib.Time2Internaldate(time.time())[1:-1]
!             dest_folder.addMessage(msg, (), date)
  
!             # We have to return something, because the client is expecting
!             # us to.  We return a short message indicating that a message
!             # was intercepted.
!             return ok + "\n" + self.intercept_message % (prob,)
          else:
              # Must be an error response.
--- 806,880 ----
          pass it through.  If the result is an unsure or spam, move it
          to the appropriate IMAP folder."""
+         # XXX This is all almost from sb_server!  We could just
+         # XXX extract that out into a function and call it here.
+ 
          # Use '\n\r?\n' to detect the end of the headers in case of
          # broken emails that don't use the proper line separators.
          if re.search(r'\n\r?\n', response):
+             # Remove the trailing .\r\n before passing to the email parser.
+             # Thanks to Scott Schlesier for this fix.
+             terminatingDotPresent = (response[-4:] == '\n.\r\n')
+             if terminatingDotPresent:
+                 response = response[:-3]
+ 
              # Break off the first line, which will be '+OK'.
              ok, messageText = response.split('\n', 1)
  
!             try:
!                 msg = message.SBHeaderMessage()
!                 msg.setPayload(messageText)
!                 # Now find the spam disposition and add the header.
!                 (prob, clues) = state.bayes.spamprob(msg.asTokens(),\
!                                  evidence=True)
  
!                 msg.addSBHeaders(prob, clues)
! 
!                 # Check for "RETR" or "TOP N 99999999" - fetchmail without
!                 # the 'fetchall' option uses the latter to retrieve messages.
!                 if (command == 'RETR' or
!                     (command == 'TOP' and
!                      len(args) == 2 and args[1] == '99999999')):
!                     cls = msg.GetClassification()
!                     dest_folder = None
!                     if cls == options["Headers", "header_ham_string"]:
!                         state.numHams += 1
!                         headers = []
!                         for name, value in msg.items():
!                             header = "%s: %s" % (name, value)
!                             headers.append(re.sub(r'\r?\n', '\r\n', header))
!                         body = re.split(r'\n\r?\n', messageText, 1)[1]
!                         messageText = "\r\n".join(headers) + "\r\n\r\n" + body
!                     elif prob > options["Categorization", "spam_cutoff"]:
!                         dest_folder = self.spam_folder
!                         state.numSpams += 1
!                     else:
!                         dest_folder = self.unsure_folder
!                         state.numUnsure += 1
!                     if dest_folder:
!                         msg = StringIO.StringIO(msg.as_string())
!                         date = imaplib.Time2Internaldate(time.time())[1:-1]
!                         dest_folder.addMessage(msg, (), date)
! 
!                         # We have to return something, because the client
!                         # is expecting us to.  We return a short message
!                         # indicating that a message was intercepted.
!                         messageText = self.intercept_message % (prob,)
!             except:
!                 stream = cStringIO.StringIO()
!                 traceback.print_exc(None, stream)
!                 details = stream.getvalue()
!                 detailLines = details.strip().split('\n')
!                 dottedDetails = '\n.'.join(detailLines)
!                 headerName = 'X-Spambayes-Exception'
!                 header = Header(dottedDetails, header_name=headerName)
!                 headers, body = re.split(r'\n\r?\n', messageText, 1)
!                 header = re.sub(r'\r?\n', '\r\n', str(header))
!                 headers += "\n%s: %s\r\n\r\n" % (headerName, header)
!                 messageText = headers + body
!                 print >>sys.stderr, details
!             retval = ok + "\n" + messageText
!             if terminatingDotPresent:
!                 retval += '.\r\n'
!             return retval
          else:
              # Must be an error response.



From anadelonbrin at users.sourceforge.net  Tue Dec 30 22:32:40 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Tue Dec 30 22:32:43 2003
Subject: [Spambayes-checkins] spambayes/scripts sb_server.py,1.15,1.16
Message-ID: <E1AbX68-0007pU-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/scripts
In directory sc8-pr-cvs1:/tmp/cvs-serv30093/scripts

Modified Files:
	sb_server.py 
Log Message:
When we stopped sb_server and then restarted, we didn't init the state, so it
wouldn't work.  Fix that.

Index: sb_server.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/scripts/sb_server.py,v
retrieving revision 1.15
retrieving revision 1.16
diff -C2 -d -r1.15 -r1.16
*** sb_server.py	30 Dec 2003 22:57:23 -0000	1.15
--- sb_server.py	31 Dec 2003 03:32:37 -0000	1.16
***************
*** 851,854 ****
--- 851,855 ----
  
  def prepare():
+     state.init()
      state.prepare()
      # Launch any SMTP proxies.  Note that if the user hasn't specified any



From anadelonbrin at users.sourceforge.net  Tue Dec 30 23:07:39 2003
From: anadelonbrin at users.sourceforge.net (Tony Meyer)
Date: Tue Dec 30 23:07:43 2003
Subject: [Spambayes-checkins] website faq.txt,1.54,1.55
Message-ID: <E1AbXdy-0000az-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/website
In directory sc8-pr-cvs1:/tmp/cvs-serv2286

Modified Files:
	faq.txt 
Log Message:
General tidy-up and bring a few things more up-to-date.

Index: faq.txt
===================================================================
RCS file: /cvsroot/spambayes/website/faq.txt,v
retrieving revision 1.54
retrieving revision 1.55
diff -C2 -d -r1.54 -r1.55
*** faq.txt	11 Dec 2003 02:56:07 -0000	1.54
--- faq.txt	31 Dec 2003 04:07:36 -0000	1.55
***************
*** 41,50 ****
  for good mail).  It's best to train on recent email, because your interests
  and the nature of what spam looks like change over time.  Once you've
! collected a fair portion of each (anything is better than nothing, but it
! helps to have a couple hundred of each), you can tell SpamBayes, "Here's my
  ham and my spam".  It will then process that mail and save information about
  different patterns which appear in ham and spam.  That information is then
! used during the filtering stage.  See the "Command-line training" section
! below for details.
  
  When SpamBayes filters your email, it compares each unclassified message
--- 41,48 ----
  for good mail).  It's best to train on recent email, because your interests
  and the nature of what spam looks like change over time.  Once you've
! collected a fair portion of each, you can tell SpamBayes, "Here's my
  ham and my spam".  It will then process that mail and save information about
  different patterns which appear in ham and spam.  That information is then
! used during the filtering stage.
  
  When SpamBayes filters your email, it compares each unclassified message
***************
*** 72,80 ****
    details.
  
! * Donate money to the Python Software Foundations.  For more 
    information, including why you would want to donate to the PSF,
    please see our `donations page`_.
  
! * Investigate some of the commercial programs based on the SpamBayes code.
    This should give you some additional benefits like support or greater 
    ease-of-use.
--- 70,78 ----
    details.
  
! * Donate money to the `Python Software Foundation`_.  For more 
    information, including why you would want to donate to the PSF,
    please see our `donations page`_.
  
! * Investigate some of the commercial `programs based on the SpamBayes code`_.
    This should give you some additional benefits like support or greater 
    ease-of-use.
***************
*** 82,86 ****
--- 80,86 ----
  .. _the PSA license: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/*checkout*/spambayes/spambayes/LICENSE.txt
  .. _I'm not a programmer but still want to help: #i-m-not-a-programmer-but-want-to-help-out-what-can-i-do
+ .. _Python Software Foundation: http://www.python.org/psf/
  .. _donations page: donations.html
+ .. _programs based on the SpamBayes code: related.html
  
  What online resources are available?
***************
*** 103,107 ****
  
  5. The `SpamBayes bugs list`_ receives copies of all the bug, patch,
!    support requests and feature request reports that are submitted via the
     `sourceforge`_ system.  This is generally only of interest to developers
     (you can use the sourceforge system to monitor any individual bugs that
--- 103,107 ----
  
  5. The `SpamBayes bugs list`_ receives copies of all the bug, patch,
!    support request and feature request reports that are submitted via the
     `sourceforge`_ system.  This is generally only of interest to developers
     (you can use the sourceforge system to monitor any individual bugs that
***************
*** 114,120 ****
  ::
  
!     site:mail.python.org pop3proxy -checkins
  
! would search for messages which mention pop3proxy but exclude checkin
  messages.
  
--- 114,120 ----
  ::
  
!     site:mail.python.org sb_server -checkins
  
! would search for messages which mention sb_server but exclude checkin
  messages.
  
***************
*** 130,138 ****
  ------------------------------------
  
! Unless you are using the Outlook plugin, you must have a recent version of
  Python installed on your computer, version 2.2 or later.  (Don't ask about
  backporting it to earlier versions of Python.  It's almost a certainty this
  won't happen.) If you need to install Python on your system, check the
! `Python download page`_ for the version appropriate to your computer You
  also need version 2.4.3 or above of the Python "email" package.  If you're
  running Python 2.2.2 or above, then you already have this.  If not, you can
--- 130,142 ----
  ------------------------------------
  
! Unless you want to run from the source code, all you need is the
! SpamBayes installer.  At present, unless you want to use the Outlook
! plug-in, you must run from source.  This will change in the near future.
! 
! If you want to run from source, you must have a recent version of
  Python installed on your computer, version 2.2 or later.  (Don't ask about
  backporting it to earlier versions of Python.  It's almost a certainty this
  won't happen.) If you need to install Python on your system, check the
! `Python download page`_ for the version appropriate to your computer. You
  also need version 2.4.3 or above of the Python "email" package.  If you're
  running Python 2.2.2 or above, then you already have this.  If not, you can
***************
*** 158,163 ****
     give it messages, tell it whether those messages are ham or spam, and it
     adjusts its probabilities accordingly.  How to train it is covered below.
!    By default it lives in a file called "hammie.db" or (for the Outlook
!    plugin) "default_bayes_database".
  
  2. The tokenizer/classifier.  This is the core engine of the system.  The
--- 162,167 ----
     give it messages, tell it whether those messages are ham or spam, and it
     adjusts its probabilities accordingly.  How to train it is covered below.
!    By default it lives in a file called "hammie.db", "statistics_database.db"
!    or (for the Outlook plugin) "default_bayes_database".
  
  2. The tokenizer/classifier.  This is the core engine of the system.  The
***************
*** 231,238 ****
     the web.  You can upload emails to it for training or classification,
     query the probabilities database ("How many valid emails *really* contain
!    the word Viagra") find particular messages, and most importantly, train
     it on the emails you've received.  When you start using the system,
!    unless you train it using the Hammie script it will classify most things
!    as Unsure, and often make mistakes.  But it keeps copies of all the
     emails it's seen, and through the web interface you can train it by going
     through a list of all the emails you've received and checking a Ham/Spam
--- 235,242 ----
     the web.  You can upload emails to it for training or classification,
     query the probabilities database ("How many valid emails *really* contain
!    the word Viagra?") find particular messages, and most importantly, train
     it on the emails you've received.  When you start using the system,
!    (unless you train it with an existing collection) it will classify most
!    things as Unsure, and often make mistakes.  But it keeps copies of all the
     emails it's seen, and through the web interface you can train it by going
     through a list of all the emails you've received and checking a Ham/Spam
***************
*** 243,253 ****
     it's very quick and easy.
  
! 6. The Outlook plug-in.  For Outlook 2000 and Outlook XP (2002) users (not
     Outlook Express) this lets you manage the whole thing from within
!    Outlook.  You set up a Ham folder and a Spam folder, and train it simply
!    by dragging messages into those folders.  Alternatively there are buttons
!    to do the same thing.  And it integrates into Outlook's filtering system
!    to make it easy to file all the suspected spam into its own folder, for
!    instance.
  
  7. The filter script.  This does three jobs: command-line training, procmail
--- 247,257 ----
     it's very quick and easy.
  
! 6. The Outlook plug-in.  For Outlook (2000, 2002 (XP), or 2003) users (not
     Outlook Express) this lets you manage the whole thing from within
!    Outlook.  You tell the plug-in which folders to watch for new mail, and
!    where to put messages it is unsure about, or considers spam, and it takes
!    care of everything else for you.  It also has a nice graphical interface
!    for training, or you can set it up to train any messages you move into
!    particular folders.
  
  7. The filter script.  This does three jobs: command-line training, procmail
***************
*** 353,360 ****
  
  Users limited to POP3/IMAP communications to the server can use the POP3_ or
! IMAP_ proxies which are part of the SpamBayes source.
  
! .. _POP3: http://spambayes.sf.net/applications.html#sb_server
! .. _IMAP: http://spambayes.sf.net/applications.html#imap
  
  
--- 357,364 ----
  
  Users limited to POP3/IMAP communications to the server can use the POP3_ or
! proxy or IMAP_ filter which are part of the SpamBayes source.
  
! .. _POP3: applications.html#sb_server
! .. _IMAP: applications.html#imap
  
  
***************
*** 435,447 ****
  -----------------------------------------------------
  
! Previous versions of the binary had a number of problems with various
! versions of Outlook/Windows.  However, to our knowledge, the current version
! should work with any combination of Windows/Outlook versions. Please let us
! know if this is not the case.  The `troubleshooting guide`_ for the Outlook
! plugin contains the most up-to-date help for working around known problems.
! A number of people have used the plugin with a beta version of Outlook 2003.
! If you fall into that category, note that you must have applied all the
! technical refreshes released by Microsoft to use the plugin successfully.
! Better yet, upgrade to the final version now that it's available.
  
  .. _troubleshooting guide: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/*checkout*/spambayes/spambayes/Outlook2000/docs/troubleshooting.html?rev=HEAD&content-type=text/html
--- 439,446 ----
  -----------------------------------------------------
  
! To our knowledge, the current version of the plug-in should work with any
! version of Windows and Outlook 2000 or above. The `troubleshooting guide`_
! for the Outlook plugin contains the most up-to-date help for working around
! known problems.
  
  .. _troubleshooting guide: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/*checkout*/spambayes/spambayes/Outlook2000/docs/troubleshooting.html?rev=HEAD&content-type=text/html
***************
*** 825,829 ****
  web interface and the Outlook plug-in let you view the clues that make
  up the message.  If you still can't figure out the reason why, you can
! ask the mailing list for advice.
  
  
--- 824,829 ----
  web interface and the Outlook plug-in let you view the clues that make
  up the message.  If you still can't figure out the reason why, you can
! ask the mailing list for advice - but make sure you include the spam
! clues/tokens listing in your message!
  
  
***************
*** 852,861 ****
     They should not be close together (say, 0.4 and 0.6).
  
! 2. Have you trained on a reasonable number of hams and spams?  You should
     train on 10 to 20 of each to start with just to get a decent base.  After
     that, you should be able to train on just mistakes and messages
     classified as unsure.
  
! 3. Check to be sure you haven't made any classification mistakes (trained
     spams as hams or vice versa).  If so, you could really confuse things and
     should move incorrectly classified messages to their correct locations
--- 852,864 ----
     They should not be close together (say, 0.4 and 0.6).
  
! 2. It is quite important that you have trained on roughly equal numbers of
!    ham and spam (don't go above a 2::1 ratio, for example).
! 
! 3. Have you trained on a reasonable number of hams and spams?  You should
     train on 10 to 20 of each to start with just to get a decent base.  After
     that, you should be able to train on just mistakes and messages
     classified as unsure.
  
! 4. Check to be sure you haven't made any classification mistakes (trained
     spams as hams or vice versa).  If so, you could really confuse things and
     should move incorrectly classified messages to their correct locations
***************
*** 867,871 ****
  ---------------------------------------------------------
  
! Because training from scratch is a very rare occurrence, and because
  deleting all your training information is something you don't want to do by
  accident, there isn't an option for this.  However, you can quite simply do
--- 870,877 ----
  ---------------------------------------------------------
  
! If you're using the Outlook plug-in, you can simply use the "Training"
! tab of the SpamBayes Manager, and tick the "Rebuild entire database" box.
! 
! Otherwise, because training from scratch is a very rare occurrence, and as
  deleting all your training information is something you don't want to do by
  accident, there isn't an option for this.  However, you can quite simply do



From richiehindle at users.sourceforge.net  Wed Dec 31 09:09:38 2003
From: richiehindle at users.sourceforge.net (Richie Hindle)
Date: Wed Dec 31 09:09:41 2003
Subject: [Spambayes-checkins] spambayes/spambayes Dibbler.py,1.11,1.12
Message-ID: <E1Abh2Y-0006gz-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes
In directory sc8-pr-cvs1:/tmp/cvs-serv25721

Modified Files:
	Dibbler.py 
Log Message:
Fix for 842984: If webbrowser.open_new() fails, print a message
saying "Please point your web browser at http://localhost:8880/"
rather than bombing out.


Index: Dibbler.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/Dibbler.py,v
retrieving revision 1.11
retrieving revision 1.12
diff -C2 -d -r1.11 -r1.12
*** Dibbler.py	16 Dec 2003 05:06:34 -0000	1.11
--- Dibbler.py	31 Dec 2003 14:09:36 -0000	1.12
***************
*** 690,694 ****
  
      if launchBrowser:
!         webbrowser.open_new("http://localhost:%d/" % context._HTTPPort)
      asyncore.loop(map=context._map)
  
--- 690,698 ----
  
      if launchBrowser:
!         try:
!             url = "http://localhost:%d/" % context._HTTPPort
!             webbrowser.open_new(url)
!         except webbrowser.Error, e:
!             print "\n%s.\nPlease point your web browser at %s." % (e, url)
      asyncore.loop(map=context._map)
  



From richiehindle at users.sourceforge.net  Wed Dec 31 09:27:36 2003
From: richiehindle at users.sourceforge.net (Richie Hindle)
Date: Wed Dec 31 09:27:39 2003
Subject: [Spambayes-checkins] spambayes/spambayes/resources ui.html, 1.29,
	1.30 ui_html.py, 1.28, 1.29
Message-ID: <E1AbhJw-0007Sq-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/spambayes/resources
In directory sc8-pr-cvs1:/tmp/cvs-serv28690

Modified Files:
	ui.html ui_html.py 
Log Message:
Default to twenty search results rather than just one.

Made the search form do a GET rather than a POST.  Previously,
going Back to the search results would show an "Expired" page
(at least in IE6).  It now runs the search again, which is almost as
bad but at least you see the page.  We'll need a proper page expiry
mechanism eventually.


Index: ui.html
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/resources/ui.html,v
retrieving revision 1.29
retrieving revision 1.30
diff -C2 -d -r1.29 -r1.30
*** ui.html	22 Dec 2003 04:38:11 -0000	1.29
--- ui.html	31 Dec 2003 14:27:34 -0000	1.30
***************
*** 410,414 ****
  <p class="uiHtmlHeader">findMessage</p>
  
!     <form id="findMessage" action='review' method='post'>
      <table>
      <tr>
--- 410,414 ----
  <p class="uiHtmlHeader">findMessage</p>
  
!     <form id="findMessage" action='review' method='GET'>
      <table>
      <tr>
***************
*** 433,437 ****
          <br/>
          <span id="max_results_label">Maximum results</span>
!         <input id="max_results" name='max_results' value='1' type='text' size='4'/>
          <br/>
      </td>
--- 433,437 ----
          <br/>
          <span id="max_results_label">Maximum results</span>
!         <input id="max_results" name='max_results' value='20' type='text' size='4'/>
          <br/>
      </td>

Index: ui_html.py
===================================================================
RCS file: /cvsroot/spambayes/spambayes/spambayes/resources/ui_html.py,v
retrieving revision 1.28
retrieving revision 1.29
diff -C2 -d -r1.28 -r1.29
*** ui_html.py	22 Dec 2003 04:38:11 -0000	1.28
--- ui_html.py	31 Dec 2003 14:27:34 -0000	1.29
***************
*** 93,121 ****
  �ڴkRL�:3ڔ?��]4�Q\015�?����4�\013���Lz�����4\004~�\013��e�\037��\003��~4��ࣿ\
  \024\006��l��#Kq홢�\004\033\0265;�N/��\033z�0Qv0�\026��6\003�]�\020��\025ӏ�aV�:\020\014��#K�h\
! X�L\035�H2Ugn��j�`7I5\017h[`�]C����! }W/q��}���5\017�\011Y��m�޶n|Q�dh��\
! �ѳaͤ�&7<��`\034����x-��:�\022G^S��F�{#�f\003�j�7��A�\016Dt5�_/Mۡ\013�\015�\
! M�=bx7'�\005I�38�#Qxg��\011ʣPQ�U��\010�!P3`ft�@qD[ʶ�V7g-�v\020��4ZMF}\
! ���\010C��A�ms����\036\030��g��t��v���v8gJ�;}����@��\006�q����kc�~(�\021nE\
! c�b7�\033�@�.\024%V��NN�\024`U���!--��gϷ\000zO\007��$.6�T�\015�Bu\000ݞ\003Bz�<\005\"hB\
! �&�xBś\014�_}�͖\001;>5B��\021o�[N\011\"���\012$t΁�\\���U��A��\001�E.���g��y}�\
! ����ȓV2ǀ�\023S��O\004r��0���ԑ��ʱ�in}B�˜�-Nߛ��NVx}%���Y�.g��\
! \017\037\006E͟(I;D�\017O阰��1�]��h�T�P�JV��X�y�3�\021b�-\036u\0238��V)ҏp\011�z�\011\
! ζ�M˕\002�Li�Ѽl�ƺ��9�Ng�0���\015�\011�`\013��/\037��)e\000�$\021�=�=d}4n�X����\
! �YIM-�;��v���M\017�:J�g����-b�\023\\M��0�S:�\010\034[��?\002l��̶�n���\030L���\001\
! M\033o\007\011��gi��쭸7E��\006o��f*mv\034�뤲�^'�\031��4h����2�\016E6�L��T��,\
! H��lܭj��\013��S�;���\011<�fGݟ�:�XQ��w\000m����VRN�c��\033�y��\017.2t\002��C\
! LO{n*�q]��ey��X\010\017���Z}\032�J�\020�*��LYzP�ϑ�yW�u)�\0004\004O�KU�K��ƴQ�\
! gnA\005T\012\015�2~����Z\014�>(�)U=\014t\035� ӏj��\036>^U\037\014�&�\027/߾v�t\037��8\0321���\002\
! \027\027�uTsc���t{���[��-a�����c�������U}�����9��:����\023|Y��n�]�]\005Y\
! ��Js0]��d0�a�볽�m+��O��n�9�� *�<�\023�HP�{&�݁/\\~蜭��c�j[��\0371\
! q&\023>��сC��(5�]\026֝~\023�:\023K���ҽ�\0228D\024�DCg\023�R�\001s����u\0128��J\027��IS\
! �������5&���s<�M�\034w�f�ؤ�?z6l\000�|�L�\002�KX��|\037Y���\034m��]$��pw6ơ\
! �#+\035m�\"\001[�p�*��N0\010�aa�*���ܪb�Z}us�-q�|G��Y�\032E�\034�S�Tb4\034\016�X�\
! \030�����Է���@�|�\022�[s��~�R1q�a�9c\031�s��OX�����{�9�\034���r��ӈ�\003�\
! Hk�P{p���c�A5ij\023j�\016\012��o�5;��c\025�����^�:j�\007���/4|V;\025� ӷ\\�!�\
! XwX.�\"\022�kSB\035<j�P���\031��{!�٦�V�\036�=P\013��\026�\020x���nH\020�v��\016���\035\016�\
! t\025/�7<:?��X?y.�����`Xtd�[�C5�b�W�b<��\"��_�\015�Ӳ�YQ�P\013��0b\012r\030\
! ��!\033���g��\023��;ш���\030}�\021��zݷ9&����A\004�\025L-���r�5X$���$�.�~��\
! ���c>\017��~i*R��2�;״�\010��v\015r\\\031�P\002M��/�MY1�K\037;1�8b��L4\011r��Η0��\
! �t�z\037��\016��o�XK\002�\"���n�����\002�7Ws")
  ### end
--- 93,121 ----
  �ڴkRL�:3ڔ?��]4�Q\015�?����4�\013���Lz�����4\004~�\013��e�\037��\003��~4��ࣿ\
  \024\006��l��#Kq홢�\004\033\0265;�N/��\033z�0Qv0�\026��6\003�]�\020��\025ӏ�aV�:\020\014��#K�h\
! X�L\035�H2Ugn��j�`7I5\017h[`�]C����! }W/q��}��wȮ�\016��w�\033\037T8\031��fo�l\
! X����\015\017r:\027G�㳽&^\013c�\016�đה�����Ǯ�����ͫv��\003\021]���+�v��v�d�|�\
! \030�MIzA��\004��H\024�٭f��(TTq��&�a\010��\031\035�\037P\033і�-�խY˼\035��#mV�Q�b�\036\
! o�N�]\036��6\007�����Hx6�Hw�j'\015_m�s�Կ�\027�k�\016txo�\032�{_߾6F�\022�\030�V4�\
! -vùQ\011��BQb\005���tK\001V\005j�\035��\002:{�|\013��t\000(N�b�O5ـ.T\007��9 �\007�S �&�l\
! \"�'T�ɰ����l\031��S#$|\030�ָ� ��i�@B�\034��u+�\\E\010\036D\010\036 Y�\"]L~\026���7\035�\
! \010��<i%s\014�?1�\011�D ן\014Ù�.H\035\011\032�\034۟��'�)�������\033�d��W�{H���rv���\
! aP�����C���\011�?\030�٥ݍFN�\016��d������<C\035!\006��Q7�\023/n�\"�\010�\020�'��l\
! �۴\\)�ϔ�\031��&m�K.�c�tv\012�\012J���P\013�\020���1\035�R\006\000O\022aڃ�C�G�։\025;JM_�\
! ����s?h�B���\000��\024y6����\"6;��t�\017�:�C���%*�#��:�lk�F\031x�����\036д\
! �v��Z�fM�ފ{S��i�)h����f�ѾN*���u��Q[L���\017�(��Pd��T�O5�ς�\
! \012��ݪ�\032�Pm<��3i>��3jv��\031�C�\025��~\007ж�)��k%�T<v����G���\"C/�\034<��\
! ��\"\035���Z������\\O�է���\017��\022�̔�\007��\034\011�w�_�r\013@C�D�T\025�T�hL\033�|�\
! \026T@���/�\030ڞ��Ő�b�R��@��\0172������U��0j2}���k�M�Q���\021�\017n/pq\
! A_G57f�JK�Gj��\012�\022\026���?��k�\030L_���XO��z��\034|_;��\005��\026���U�5N\
! �4\007�U(I\006�\035V�>ۋڶ\002���\011�&�C�\015���S:�\004U�gR�\035����ٺ�=f��\025��\021\023g\
! 2�#�\037\0358���R��ea��7��3�t��(ݫ*�CD�M4t6�*U\0330��΍^�����t�l�4e�\
! �Z�\017�Yc���<ǣ���q\007i\006�͚��g�\006\020͗��)����)�������Ѧ��Eҿ\011wgc\034�9\
! ���v.\022��\015w�RN�\004��\035\026��b\012�˭*F��W7��\022��wt͜e��Q��a:�K%F����Պ\001\
! ��m�H}��\034>\013��')��5G)�\007+\025\023�\030\006�3��=�����۽\017k�����q�\017*�I?��<P��\
! �\017�\007w�?�\037TC<;�F�`+�\006[�\023�8V!�\011Yxf��V\035�؃|���\027\032>���R��[���~�\
! ;,�e\021�uڵ)�\016\0365Z�RB���?��\020�lS}�S\017�\036���j�C\010<�kw7$�S��Y����\016�O�\
! ���;\036�\037�m��<��y�M]0,:2���\032U1�+|1��k\021��/�\006�iY�(F�\005��x\0301\0059\014�\
! 鐍��ó��\011VםhD��c�����z���\034\023~�`� ���\012�\026[�q9u�\032,��uj\022m�g���U\
! ��1�\007k|�4\025)~S\031�k�Z\004�W�\0069��R(�&�ۗ񦬘��]\034�no&�\0049~c�K��sf\
! :c���V\007��p�%\001E\021�F}�Z}��\001�UV�")
  ### end



From richiehindle at users.sourceforge.net  Wed Dec 31 20:56:14 2003
From: richiehindle at users.sourceforge.net (Richie Hindle)
Date: Wed Dec 31 20:56:18 2003
Subject: [Spambayes-checkins] spambayes/utilities hammer.py,NONE,1.1
Message-ID: <E1Abs4M-0002OL-00@sc8-pr-cvs1.sourceforge.net>

Update of /cvsroot/spambayes/spambayes/utilities
In directory sc8-pr-cvs1:/tmp/cvs-serv9186

Added Files:
	hammer.py 
Log Message:
Hammers the core SpamBayes code, repeatedly training and
classifying using faked-up messages.


--- NEW FILE: hammer.py ---
#! /usr/bin/env python

# Part of the SpamBayes project.  Released under the Python Software
# Foundation license; see http://www.python.org/

import os, sys, re, random, textwrap

from spambayes import storage
from spambayes import tokenizer

__author__ = "Richie Hindle <richie@entrian.com>"

headerTemplate = """To: %(To)s
From: %(From)s
Subject: %(Subject)s
Date: %(Date)s

"""

[...1219 lines suppressed...]
 tools. IT IS UP TO YOU NOW !


 ORDER YOUR REPORTS TODAY AND GET STARTED ON YOUR ROAD

 TO FREEDOM.


 THERE IS NO NEED TO RESPOND TO THIS E-MAIL IF YOU DO NOT WISH TO

 RECEIVE FURTHER CORRESPONDENCE. THIS IS A ONETIME E-MAIL.


 Happy Mailing!
 ______________"""
} ]


if __name__ == '__main__':
    hammer()