[Mailman-Developers] Re: Future of pipermail?

[Chuq Von Rospach]

At 11:23 PM -0500 11/21/00, Bill Bumgarner wrote:

>- archival of messages is a lot more than just writing the bodies to 
>a web server and then generating some kind of automatic TOC/index.

agreed completely. I'd take it a step further and say it probably 
shouldn't generate indexes at all, but that indexes should be 
generated when a user wants to access the archives, dynamically. 
That's probably the single major weakness of mhonarc.

>  MIME based email content is not only a reality, but it is often the default.

and since AOL 6.0 for windows no longer has any provision for 
text-only operation, you now have no choice but to deal with MIME in 
some way. they no lnoger HAVE the option to turn MIME off. So the 
server has to figure it out.

And IMHO, from my research, we're currently making the switch to 
style-capable e-mail in the mainstream. A year from now -- lists that 
don't support it reasonably (and I don't believe "strip it to the 
text part and throw the rest away" will qualify as "reasonable" -- 
it's a transition stopgap) wll be looked on negatively. People want 
enclosures and HTML, because the tools to deal with them are 
maturing. Looking forward, it's going t be standard, and that means 
all of the tools need to deal with them properly, and taht includes 
displaying them, transporting them, and protecting users from them as 
needed.

>  Mailman needs to more gracefully handle mime-types;  automatic 
>filters would be one solution [very desirable for things like a pure 
>discussion list] and automatic decoding/filing/handling of 
>attachments is another solution [and very desirable for things like 
>project discussion lists, etc].

agreed, and it's been talked about here. We need to properly support 
it, plus we also need the ability to filter adn edit it -- to strip 
.exe files, for instance, to store HTML and graphics on the server to 
show properly in the archives, to generate what I call TOC-digests 
(simply a list of pointers into the archives), and other stuff.

>- message rewriting is a really powerful concept and should be an 
>abstract and first class part of the Mailman core processing 
>pipeline.

It's something I'm doing pre-design work on now, in fact.

>- the current solution for creating an archive is lacking-- that 
>doesn't mean it doesn't work.

It's very much an older-technology base -- where it's falling behind 
a rapidly changing technology. A year ago, pipermail was great. Now, 
it's not. Fortunately, I think we're close to the end of the rapid 
evolution of e-mail, or at least where we can take a breath, figure 
out reality and catch up a bit. But it's been really interesting 
trying to NOTICE all of the changes, much less manage them...

>  It works and it works very well, but it is *not* modular, does 
>*not* follow the design patterns of the rest of the tool and makes 
>it *very* difficult to slap in replacement models for archival (has 
>this changed since, say, may?

A concept I've been sort of beating into the ground, but if there's a 
weakness in open source software, it tends to be written by 
programmers used to the "I want this, therefore I shall write it", 
and interfacing stuff to other pieces of similar stuff is low on the 
priority list, or missing. And the future of the software world are 
those damn C and I words I keep using...

this is not a criticism, but a recognition that the paradigms are 
changing. I'm seeing these concepts adopted into open source as well 
as commercial software (probably faster in much of the open source 
world now, in fact) -- but the way we wrote stuff a year ago isn't 
how we'll write stuff in a year. We're seeing what Apple wanted to do 
with OpenDoc actually starting to happen, IMHO. Pluggable 
extensibility. Object-oriented systems, not object oriented software.

>- for the archival of plain text messages, WebDAV is overkill [as 
>Chuq mentions].  However, as soon as you move to archiving mime 
>attachments, it quickly becomes extremely advantageous to archive 
>various properties with the archived message pieces.

but you can do that with a lot less overhead in MySQL by doing a 
focussed database. In fact, you could program a system to do this via 
DBI that'd work in any DBI-capable environment, so users could roll 
their own based on what they've already adopted. unless WebDAV gives 
us enough extra capabilities to be worthy of the specialization, my 
argument is (and will be) we program to a more general API (like 
DBI), so that we work in many environments, and if someone wants, 
they can program a DBI->WebDAV interface to attach to it. This way, 
we get DB, MySQL, PostGres, Oracle, ODBC, yadayada more or less for 
free, giving us functionality across multiple environments that users 
can tailor. If we program just to WebDAV, we don't get that.

So it's choosing what the appropriate interfaces are that's as 
important as having interfaces. you don't program to a technology 
unless you have to -- you program to an interface that enables 
technologies. (image: this is chuqui. this is a dead horse. This is 
chuqui holding a whip...)

>- ....restoring decoded attachments and reencoding back to their 
>original state with their original headers is an extremely cool 
>feature.

Truly. And if we can support BLOBs in DBI, well, we don't have to 
write anything to disk and can generate an entire message out of a 
DBI database -- portable to any decent database.

>- if we are to manage the complexity associated with the integration 
>of numerous technologies, it is only going to happen through well 
>refined and highly modular APIs....

agreed. and to make ti clear, I'm not arguing against WebDAV. I'm 
arguing that for something like this, you define the interface and 
see if you can build it in a way that you don't JUST get WebDAV, but 
support at a more abstracted level that gets you a range of supported 
technologies (and future capability for that yet discovered) for an 
incrementally greater amount of work. the trick is to find the right 
abstractions and the proper technology layer to attach that to.
-- 
Chuq Von Rospach - Plaidworks Consulting (mailto:chuqui@plaidworks.com)
Apple Mail List Gnome (mailto:chuq@apple.com)

The vet said it was behavioral, but I prefer to think of it as genetic.
It cuts down on the liability -- Get Fuzzy