[spambayes-bugs] [ spambayes-Patches-1177008 ] messageinfo_use_database option and none as db type

[SourceForge.net]

Patches item #1177008, was opened at 2005-04-05 14:22
Message generated for change (Comment added) made by fufsource
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=498105&aid=1177008&group_id=61702

Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Michal Vitecek (fufsource)
Assigned to: Tony Meyer (anadelonbrin)
Summary: messageinfo_use_database option and none as db type

Initial Comment:
hello,
this patch adds 1) new option
'messageinfo_use_database' so it can be defined
independently of the storage database type used, 2)
'none' as database type which acts as empty database
without stores

this all is to help desperate spambayes users get rid
of the ever occuring messageinfo database corruption
(on our firm's mail server it's getting corrupted
almost every 5 minutes).

thanks.

----------------------------------------------------------------------

>Comment By: Michal Vitecek (fufsource)
Date: 2005-04-06 13:15

Message:
Logged In: YES 
user_id=698198

i know this is rather a hack then a nice patch. however the
corrupted database problem hasn't been solved for quite a
long time and there might be users (like me) who put
spambyes' sb_server on their firm's mailserver and need to
have it running as stably as possible. i know it's not a
solution, just a workaround some people might use and which
will probably never be put in the CVS :)

steps to get the messageinfo_db corrupted:
1) install spambayes on a mail/pop3 server where the pop3
serves ~90k message retrievals (RETR commands)/day
2) use sb_upload.py each 10 mins to learn sb_server ham/spam
messages
3) within 5-60mins messageinfo_db gets corrupted

background:
- the mailserver is a single cpu pIII at 800, linux kernel 2.4.
- the corruption seems to be python version independent
(occured on 2.3.[3-5], occurs on 2.4.1)
- pybsddb is 4.3.0
- 1.0.x spambayes branch seems to be better than 1.1.x
(corruption seem to occur in longer intervals)

the corruption seems to occur only in onRetr() method:
Traceback (most recent call last):
  File "/usr/bin/sb_server.py", line 567, in onRetr
    msg.addSBHeaders(prob, clues)
  File
"/usr/lib/python2.4/site-packages/spambayes/message.py",
line 452, in addSBHeaders
    self.RememberClassification(disposition)
  File
"/usr/lib/python2.4/site-packages/spambayes/message.py",
line 401, in RememberClassification
    self.modified()
  File
"/usr/lib/python2.4/site-packages/spambayes/message.py",
line 377, in modified
    self.message_info_db.store_msg(self)
  File
"/usr/lib/python2.4/site-packages/spambayes/message.py",
line 181, in store_msg
    self.db[msg.getDBKey()] = attributes
  File "/usr/lib/python2.4/shelve.py", line 130, in __setitem__
    self.dict[key] = f.getvalue()
  File "/usr/lib/python2.4/bsddb/__init__.py", line 218, in
__setitem__
    self.db[key] = value
DBRunRecoveryError: (-30982, 'DB_RUNRECOVERY: Fatal error,
run database recovery -- fatal region error detected; run
recovery')

another thing is that the database handle is not probably
closed when such error is caught causing spambayes to eat
file handles like crazy.

thanks.

----------------------------------------------------------------------

Comment By: Tony Meyer (anadelonbrin)
Date: 2005-04-06 05:58

Message:
Logged In: YES 
user_id=552329

I'm +0 on adding the new option (this would have to be
1.1a2, as it's too late for 1.1a1 now).  I don't really
think that there's much advantage to it, and it's used in
many places (the patch doesn't cover the Outlook plug-in,
for example, and I don't see why there is a key in
storage.py where "none" is DBDictClassifier) so is a very
involved change.

I'm -1 on adding a "none" type.  Outlook and sb_imapfitler
really need that database to exist and work correctly. 
sb_server can manage without at the moment, but that may
change in the future.  

It would be much better to figure out *why* your messageinfo
database gets corrupted all the time and fix that problem,
rather than just hide the problem.  If you can outline steps
to get it corrupted, please let me know.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=498105&aid=1177008&group_id=61702