[Mailman-Developers] Re: Requirements for a new archiver

[J C Lawrence]

On Tue, 28 Oct 2003 16:46:02 -0500 
J C Lawrence <J> wrote:

> While I see your concern for spending a whole lot of time investing
> and specialising in a netnews core for Mailman, I think the fear is
> displaced.  Mailman already bidirectionally gates to netnews.  The
> changes required would be comparatively small for a significant gain
> in feature-set for the average non-geek case (better web archives,
> posting from web archives, more flexible message store, ability to
> have archive URL placed in broadcast messages, etc).

I wrote this badly.

I'm looking at this in a rather abstract light, and haven't really
stated that fact.  From here this is all a question of abstraction
models.  Currently Mailman doesn't have a clean or well defined
abstraction model among the list processor, the archives, the message
sort, the web presentation layers, or search engine supports.  Gaining a
clean and well defined abstraction layer that does the Right Things
would do a whole bunch of useful things, like allow for far easier and
cleaner plugin-style approaches to any of the core components:

  Message store

  Search support

  Web archives.

  Message access.

Right now those four are a somewhat incestuous mess and pulling them
apart into a clean model is a bitch for an external integrator.  Harking
back to the earlier process queues model for Mailman 3.0, the same
abstraction and division gains happen again.

  We can implement (say) Twisted as a trivial message store.  Really
  what we're doing however is stating that Mailman can use any store you
  want as long as messages can be sent to it via process pipe, SMTP, or
  NNTP.

  We can implement a CGI-based web interface to that NNTP store.  Really
  what we're saying however is that Mailman will provide a web-based
  interface for archive browsing and message posting to its own default
  message store, or any NNTP-based message store you implement.

  We can implement a plugin layer for external search engines as we no
  know (control) the unique primary retrieval key for messages in the
  default store (and thus the default URL etc).  Or, if you wish, what
  we're really saying is that Mailman's archives can be integrated with
  any external search engine that can be handed the retrieval key and
  can index the message appropriately (via web scrape, custom store
  access, NNTP, whatever).

Sure, we can grow up and feature-fluff the search side later.  I'm a
little less concerned with that at this point.  I'm more concerned with
gaining a clean abstraction model among the store/retrieve/present
components of the archives.

Sure, we could also decompose the messages into SQL structures.  Having
done this, I'll briefly note that it is not a trivial task on several
scores (MIME, data partitioning, etc).  This isn't to say that its
horrible, merely not trivial.  Picking up Twisted's store is about as
close as you can get to trivial.  You don't get all the MIME
decomposition etc that you might get with an SQL store, but having
adopted the abstraction model it is then much simpler for someone to
write an SQL-based message store that Mailman can talk to cleanly

  In an OpenSource world smaller shorter feedback loops based on simpler
  iterative steps are almost always the better choice.

-- 
J C Lawrence                
---------(*)                Satan, oscillate my metallic sonatas. 
claw at kanga.nu               He lived as a devil, eh?		  
http://www.kanga.nu/~claw/  Evil is a name of a foeman, as I live.