[Mailman-Developers] Requirements for a new archiver

[Peter C. Norton]

On Thu, Oct 30, 2003 at 05:00:48AM +0100, Brad Knowles wrote:
> 	SGIs XFS on Irix does a pretty good job, with hashed directory 
> structures, and an extent-based journaling filesystem.  Regretfully, 
> I don't think that all of these features are fully supported under 
> the Linux version of XFS, and that work has basically ground to a 
> halt with the lay-offs of all the key SGI people who had been working 
> on XFS.  Veritas VxFS also does a good job in this area.

[ A cursory google search indicates that hashed dirs, extents, and
journalling are all in linux xfs.  I can't imagine an unsupported
feature making its way into the filesystem that SGI is putting on its
latest and greatest systems, but if you know about this, please share ]

In the case of a one-file-per-message approach, my experience with
vxfs is that it creates a rather slow filesystem when you get your
filesystem to the point of haing with a few hundred thousand small
files (lots of wasted space in the extents and I believe, though I may
be wrong, that there were lots of metadata lookups through multiple
layers of indirections slowing things down).  

However reiserfs was built to handle a mix of lots of small files, ala
maildir or mh spools.  

I'm not too current on current bsd going-ons, but I'd bet that ffs2
has something to offer in this arena, too, since it looks like it
almost does extent-based allocation now.

> 	Kirk McKusick and Eric Allman agree with you that this is a 
> proper filesystem problem that should be solved at the filesystem 
> level (at least, that's what they've said to me when I brought this 
> issue up to them), and they feel you should not attempt to solve 
> filesystem problems with "tricks" like INN timecaf/timehash cycbufs.

Err... then to relate this to a prior post, why not just use maildirs
on filesystems that are engineered to handle that sort of thing?
 
> 	However, while that's nice in theory, that doesn't necessarily 
> help us here in the real world.

Unless you are using a filesystem that works for this, right?  Like
xfs, vxfs, reiserfs, and probably ffs2.  I believe that linux's ext3
has support for hashing directories (or soon will - I don't precisely
know as I've been focusing on other things)

-Peter

-- 
The 5 year plan:
In five years we'll make up another plan.
Or just re-use this one.