[Mailman-Developers] Requirements for a new archiver

[Chuq Von Rospach]

On Oct 29, 2003, at 6:22 PM, J C Lawrence wrote:

> cycbufs implement a filesystem-based heap with pool semantics.  
> (There's
> a fair bit of literature on that space in the OS and application realm)
> As such they are specifically tuned for the case where the number of
> calls to malloc() are of a similar magnitude to the calls to free().
> This makes sense in a netnews world where news articles expire
> regularly, and in general as much data is added to the spool as is
> removed from it.
>
> Does that model really apply to list archives?  It doesn't for me.  I
> may be unusual in this regard, but I generally consider list archives 
> as
> one-way systems: messages go in and never come out.
>

and in general, you're mostly right. Deletions out of archives are 
pretty minimal. But I think cycbufs still make a lot of sense as a way 
to reduce design complexity needed to avoid using up potentially 
infinite numbers of inodes, and the performance and design complexity 
inherent in building a storage structure around a typical unix 
filesystem.

It's just so much less hassle on any number of levels dealing with 50 
100 megabyte files than it is a directory structure with 500 megabytes 
of messages spread around 100,000 individual files. whether it's 
backups and restores, migrating data to a new server, etc, etc etc, you 
make life much simpler. And god help you if you're updating that 
structure when the system crashes and you have to fsck and put it back 
together again.