[Spambayes] memory consumption...

[Tim Peters]

[Anthony Baxter]
> For my standard test, using the graham scheme, memory consumption's blown
> out considerably in the last couple of days. timcv -n 10 used to consume
> about 50M - it's now about 75M. This unfortunately pushes my system into
> swap :-( is there anything obvious to change to make it happy again?

If you haven't changed the set of msgs you're using, and neither have you
changed the set of options you're using, then that's a mystery.

Nothing has changed in the codebase in "the last couple of days" that would
account for this, unless you have a great many messages that the email pkg
just can't parse.  If that's the case, then a little over 4 days ago changes
were checked in to fall back to the raw text instead of giving up.

BTW, I also have 256MB RAM on my test machine, and of course it goes to disk
a lot when running my test:  it's opening 34,000 files multiple times.
timcv is carefully arranged to minimize that burden after the initial pain
of initial training on 9/10ths of all msgs.

Whether 50-75M "is excessive" can't be guessed without knowing how many
messages you're using and the full set of options you're running with.  If,
e.g., you didn't used to save binary pickles, but started to, memory use is
bound to blow up (the under-the-covers pickle memo dict is huge).