[Mailman-Developers] Re: Future of pipermail?

[J C Lawrence]

On Wed, 22 Nov 2000 00:49:46 -0500 
Bill Bumgarner <bbum@codefab.com> wrote:

> I don't think archival should be treated as a database centric
> operation.  The concept of archival falls very naturally into a
> static hierarchy of collections/directories containing
> resoruces/files with a bit of additional meta information
> associated with some resources.  This is exactly the kind of
> information archive that a web server is *designed* to optimally
> serve.  Adding extra layers here or abstracting to a DBI really
> doesn't buy us much.

While true in general, this ignores the benefits and ease of
dynamically generated reports on archives.  

Now, perhaps I should talk a little about what I'm spending my
copious spare time on lately (other than attempting to chase
heisenbugs that crash machines for unknowable reasons).

Among other things I'm attempting to meld the basic concepts of
WikiWiki, mail archives, and advogato/slashdot-style commentary
systems into a form where the basic tenets of Wiki noun definition
are retained such that every individual comment and archive page has
a dynamically generated/assigned WikiName and there is a moderately
sound karma/trust net running as a meta layer atop this as an
abstracted report layer.  The fact that further abstractions can be
made, and that rule-based reports/abstractions can be created as
dynamically filled out WikiPages on the other side of this is just,
well, really nice.

  I realise that's pretty turgid.  Without spending considerable
  time and verbiage I don't want to spend right now I'm not sure I
  can do better.  If you're interested in details, and I don't mean
  just casually, please ask away.

But, getting back to the point, to do this properly (ie so that its
scalable and reasonably system efficient), especially given the fact
that any given data item (be it an archived message, a comment, a
report, or whatever) has at least presentation modes each of which
has very different surrounding capabilities, the apparancy is that
I'm going to have to shove everything into a set of SQL tables.

<shrug>

The current half-implementation is a mix of SQL and static files
(mostly wrapped around my current PHPLib template based PHP4+MHonArc
based archiving setup).

> WebDAV adds the ability to do advanced locking, easy meta
> information storage, etc... but-- most importantly-- does not take
> away the very effecient presentation of data naturally present
> within a filesystem of stuff served by a web server.

I'm actually considering WebDav slightly longer term as a potential
Wiki-ish posting method.  I'm not at all convinced its a good way to
go, but I'm willing to look into it.

> As well, a filesystem centric storage/presentation solution--
> webdav or raw filesystem-- solves *most* peoples archiving
> problems *most* of the time.

Very true.

> I feel *very strongly* that the archival solution-- whether it be
> raw messages or decoded messages-- should be centric to storing
> files in directories and serving files from directories.

In the general case, I agree.

> This is *not* to say that the DBI approach isn't the right way to
> go; if a generic DBI->filesystem, DBI->WebDAV, DBI->DB capable API
> were put together and was relatively hidden from the user and
> casual developer, it might be a huge win.

I'd like to see something trivial:

  The archiver deconstructs messages into a stasndardised and well
  documented templatised form on to the file system.

  A standard CGI-like toolset is then provided to reconstruct those
  fragments back into a message (please resend this message to me, I 
  didn't get it the first time), into an HTML page (web archives),
  etc.  This is done by having standard small scripts which do
  >whatever< by assembling simple call sequences to the toolset.

  End users may redefine this by re-writing said scripts, acessing
  the deconstructions or reconstructions as they wise, to then do
  anything they want with the resulting archives.  This could be SQL 
  inserts, report generation, or black voodoo, it really doesn't
  matter, and more importantly, Mailman, and Mailman's archiver
  doesn't need to know about it, only the user does.

  The other nice thing is that we don't do it by defining APIs
  (which then reuires programmers to do useful things with), but by
  defining small useful atomic tools that may be assembled/ordered
  in interesting fashions wihtou having to subject the users to
  Python, or programming much beyond writing shell scripts and
  editing canned templates.

> But an order of magnitude less effecient than downloading the BLOB
> off of disk via a webserver!

<koh> Greenspun  <kof>

> Generic access with simple access control is what *most*
> users/administrators want *most* of the time.  More
> complex/abstracted/portable access is less of a requirement and *a
> lot* of the people with such requirements also have other issues--
> real or imagined-- that dictate that they really just want Mailman
> to hand the stuff to them as quickly/easily as possible and be
> done with it.

Precisely.

-- 
J C Lawrence                                       claw@kanga.nu
---------(*)                        : http://www.kanga.nu/~claw/
--=| A man is as sane as he is dangerous to his environment |=--