[Mailman-Developers] Re: Future of pipermail?

[Chuq Von Rospach]

At 12:49 AM -0500 11/22/00, Bill Bumgarner wrote:
>This is where i disagree *very* strongly-- maybe not with the implementation
>choice [DBI], but with the reasoning behind it.
>
>I don't think archival should be treated as a database centric operation.  The
>concept of archival falls very naturally into a static hierarchy of
>collections/directories containing resoruces/files with a bit of 
>additional meta
>information associated with some resources.   This is exactly the kind of
>information archive that a web server is *designed* to optimally 
>serve.   Adding
>extra layers here or abstracting to a DBI really doesn't buy us much.

um, except you seem to be defining a database-centric operation using 
a filesystem as a data store. It's still database-centric.

>Alone, a basic filesystem served webserver gives us:
>
>     - effecient access to archives
>
>     - basic per-site, per-list authentication
>
>     - [with little addition] unified access/passwords between lists, etc...
>
>     - almost *zero* overhead with *very little* implementation cost

which are all as true with a database based system. Except I don't 
agree withyou on the overhead and implementation costs on your end. 
Ask anyone who manages a decent-sized NNTP system -- filesystem 
centric storage systems don't scale to large data sets well at all. 
Databases do....

>I feel *very strongly* that the archival solution-- whether it be 
>raw messages or
>decoded messages-- should be centric to storing files in directories 
>and serving
>files from directories.

And I disagree, since you're tying to a single technology that works 
for some cases, but isn't a general solution, and limits other 
options. For instance, it's fairly easy to implement a lot of 
searching via DBI. it's a lot harder using a filesystem store.

>The second reason I feel strongly that moving to a DBI based 
>interface wouldn't
>present that much of an advantage is that most people that need to 
>actually store
>the data in a database are going to have their own requirements surrounding
>decoding, storage, indexing, and presentation of said database 
>related content.
>There are few *real* standards in terms of the storage of multimedia [MIME]
>content into a database environment and, as such, the developer will 
>likely have
>to rip the data out of whatever our implementation prefers and into their own
>storage subsystem.

I don't see this as an issue. You have to define an architecture either way.

>This is *not* to say that the DBI approach isn't the right way to go;  if a
>generic DBI->filesystem, DBI->WebDAV, DBI->DB capable API were put 
>together and
>was relatively hidden from the user and casual developer, it might 
>be a huge win.

which is exactly what I'm arguing. And DBI is a well-known interface 
that ias easily accessible to anyone who wants to write an interface 
-- if the architecture is done right. And it's portable off of Unix, 
which is an issue for Mailman.

>  > Truly. And if we can support BLOBs in DBI, well, we don't have to
>>  write anything to disk and can generate an entire message out of a
>>  DBI database -- portable to any decent database.
>
>But an order of magnitude less effecient than downloading the BLOB 
>off of disk via
>a webserver!

Not necessarily. And is the efficiency important? A lot of time is 
wasted in computer development optimizing the wrong things....

>Generic access with simple access control is what *most* users/administrators
>want  *most* of the time.   More complex/abstracted/portable access 
>is less of a
>requirement and *a lot* of the people with such requirements also have other
>issues-- real or imagined-- that dictate that they really just want Mailman to
>hand the stuff to them as quickly/easily as possible and be done with it.

that's changing, rapidly. Generic, text-only e-mail was what most 
users/administrators wanted a year ago, too. If you implement to what 
they WANTED, by the time it's written, they won't want it any more. 
You have to figure out what's possible and what they're going to want 
when you're done writing it. you don't want to write obsolete stuff 
because you implement what they wanted last time.

>I feel strongly that abstraction is key, but that we should also 
>provide decent,
>production quality, implementations of solutions to the very same 
>set of problems
>for which we build the gneric abstracted/modularized APIs.

agreed.

>If Mailman is not fully functional "out of the box", then people 
>will ignore it.
>However, if it isn't also flexible enough to be integrated into their weird
>environments (because every server on the web has weirdness), 
>they'll bitch and
>moan until they find something else that doesn't solve their problem to B&M
>about....

it has to be functional out of the box, but we have to make sure we 
define "functional" properly, too. You can't be missing features but 
doing the "kitchen sink" think just in case is wrong, too.

-- 
Chuq Von Rospach - Plaidworks Consulting (mailto:chuqui@plaidworks.com)
Apple Mail List Gnome (mailto:chuq@apple.com)

The vet said it was behavioral, but I prefer to think of it as genetic.
It cuts down on the liability -- Get Fuzzy