[Mailman-Developers] message data storage

[Charlie Clark]

Hi Donn,

nice to see you on this list!

> At the end of April, one of our new lists was up to 6500 archived 
> messages for the month.  It takes close to 4 minutes CPU time to archive 
> a message in a situation like that.  The average delivery rate to achieve 
> 6500 messages in 30 days is 1 per 6 minutes, so the system obviously 
> cannot work.

wow, that is a lot of traffic and a sluggish system. But I don't see how 
6500 mails can generate a 68 MB unless there are a lot of attachments.

> Working with the current system (as opposed to the Next Generation 
> archiver), I'm tempted to think about a semi-permanent per-archive 
> archiver and queue.  The archiver would exit when its queue has been 
> empty for some short interval, but any extra messages that arrive during 
> that time get the database for free.  I'd be interested in any other 
> ideas.

As someone else has noted apart from the performance issue there is also 
one of usability: searching 6500 messages is a proverbial needle in a 
haystack. I don't know what the Next Generation archiver is but I guess 
it's a move towards a more sophisticated persistance system which might 
allow things like full-text searches. If I understand you correctly you 
want to archive mails directly as they come in and keep the archiver / 
db-connection open pretty much all the time. This would seem about the best 
solution in the short term. Anything else sounds like: database to pass the 
memory issue to something which is designed to handle it. Our own 
experience with lists with a large member base is that Mailman isn't that 
efficient at dealing with them which is why we're using an RDBMS adapter 
and letting the RDBMS be the muscle while Mailman remains the brains.

Novice question? How easy would it be just to dump to a database rather 
than using Python storage?

Charlie