[Mailman-Developers] message data storage

[Donn Cave]

Quoth Charlie Clark <charlie at begeistert.org>:
...
| wow, that is a lot of traffic and a sluggish system. But I don't see how 
| 6500 mails can generate a 68 MB unless there are a lot of attachments.

Even with attachments it takes some doing.  The articles database
actually tends to be double the size of the .txt archive for the
same month.  Not only this particular list.

| As someone else has noted apart from the performance issue there is also 
| one of usability: searching 6500 messages is a proverbial needle in a 
| haystack. I don't know what the Next Generation archiver is but I guess 
| it's a move towards a more sophisticated persistance system which might 
| allow things like full-text searches. If I understand you correctly you 
| want to archive mails directly as they come in and keep the archiver / 
| db-connection open pretty much all the time. This would seem about the best 
| solution in the short term. Anything else sounds like: database to pass the 
| memory issue to something which is designed to handle it. Our own 
| experience with lists with a large member base is that Mailman isn't that 
| efficient at dealing with them which is why we're using an RDBMS adapter 
| and letting the RDBMS be the muscle while Mailman remains the brains.
|
| Novice question? How easy would it be just to dump to a database rather 
| than using Python storage?

I don't know if anyone has a clear picture of this next generation
system, it's just the subject of a dormant mailing list
http://rogue.amk.ca/mailman/listinfo/ng-arch

An external database might be a part of that picture, for all I know.
I would expect that if a database really would pay off, it would take
some rewriting anyway.  It would be trivial to use a database for storage,
but that would only slow things down.  Really need to find a way to use
the database more inside the threading algorithm.

	Donn Cave, donn at u.washington.edu