[Mailman-Developers] (no subject)

[J C Lawrence]

On Mon, 11 Dec 2000 19:27:06 -0800 
Chuq Von Rospach <chuqui@plaidworks.com> wrote:

> At 7:16 PM -0800 12/11/00, J C Lawrence wrote:
>> Not exactly.  My architecture has the ability to create messages
>> internally that are then passed back thru the processing system.

> oh, yeah. duh.

<snicker>

>> I kinda like the way you think.

> that should scare you...

According to my wife you should be terrified about now. 

FWLIW I'm working on the following leading notes:

--<cut>--

Assumption: The localhost is Unix-like.  

---

ObTheme: All config files should be human readable unless those
files are dynamically created and contain data which will be easily
and automatically recreated.

ObTheme: Unless a data set is inherently private to Mailman, Mailman
will not mandate a storage format or location for that data set, and
will allow that data set to be the result of a locally defined
abitrary replaceable process.

ObTheme: Every single program or process may be replaced with
something else, as long as that other thing accepts the same inputs,
generates outputs within spec, and performs a somewhat similar
function.

---

There are basically three approaches to scalability in use for this
sort of application:

  1) Using multiple simultaneous processes/threads to parallelise
     a given task.

  2) Using multiple systems running parallel to parallelise a given
     task.

  3) Using multiple systems, each one dedicated to some portion(s)
     or sub-set of the overall task (might be all working in
     parallel on the entire problem (lock contention! failure
     modes!)).

The intent is to be able to transparently support all three models
on a per-list basis or per-installation basis or some arbitrary mix
of the two (some sections of the problem for some lists handled by
dedicated systems, other sections of the problem for all the other
lists handled either by a different pool of systems or processes).

---

Observation: MLMs are primarily IO bound devices, and are
specifically IO bound on output.  Internal processing on mail
servers, even given crypto authentication and expensive membership
generation processes (eg heavy SQL DB joins etc) are an order of
magnitude smaller problem than just getting the outbound mail off
the system.

Consider a mid-size list of 1K members.  It is a busy list and
receives 500 messages a day, each of which is exploded to all 1K
members:

  -- That's 500 authentication cycles per day.
  -- That's 500 membership list generations.
  -- That's 500,000 outbound messages
  -- That's 500,000/MAX_RCPT_TOs SMTP transactions

Even given a MAX_RCPT_TOS of 500 (a bit large in my mind) that's 1K
high latency multi-process SMTP transactions versus 500 crytps
crypts or SQL queries.

---

Observation: In the real of MLM installations there are two end
points to the scalability problem:

  1) Sites with lists with very large numbers of members
  2) Sites with large numbers of lists which have few members.

Sites with large numbers of lists with large numbers of members (and
presumably large numbers of messages per list) are the pessimal
case, and is not one Mailman is currently targeting to solve.

The first case MLM is oubound bounnd.  The second case may be local
storage IO bound as it spends significant time walking local
filesystems during queue processing which the outbound IO rates are
comparitively small (and unbursty).  Possibly.

SourceForge falls into the second case.

---

Observation: Traffic bursts are bad.  Minimally the MLM should
attempt to smooth out delivery rates to a given MTA to be no higher
than N messages/time.  This doesn't mean the MLM doesn't deliver
mail quickly, just that in the case of a mail burst (suddenly
20Million messages sitting in the outbound queue), that the MLM will
give the MTA the opportunity to try and react intelligently rather
than overwhelming it near instantly with all 20M messages dumped in
the MTA spool over 30 seconds while the spool filesystem gags.

---

There are five basic transition points for a message passing thru a
mailing list server:

  1) Receipt of message by local MTA
  2) Receipt by list server
  3) Approval/editing/moderation 
  4) Processing of message and emission of any resultant message(s)
  5) Delivery of message to MTA for final delivery.

#1 is significant only because we can can rely on the MTA to
distinguish between valif list-related addresses and non-list
addresses.

#2 is just that.  The message is received by the MLM and put
somewhere where it eill later be processed.  The intent is that this
is a lightweight LDA process that does nothing but write queue
files.  The MLM's business is to make life as easy as possible on
the MTA.  This is part of that.

#3 Mainly occurs for moderation, and encludes editing, approval,
authentication, and any other requisite steps.  The general purpose
of this step is to determine what (if any) subsequent processing
there will be of this message .

#4 Any requisite processing on the message occurs, and any messages
generated by that processing are placed int he outbound queue.

#5 An equivalent to the current queue runner process empties the
queue by creating SMTP transations for the entries in the queue.

The basic view I'm taking os the list server is that it is a staged
sequence of processes, each invokved distinctly, orchestrated in the
background by cron.  

Note: Bounce processing and request processing re not detailed at
this point as their rate of occurance outside of DoS attacks is
comparitively low and are far cheaper than list broadcasts in
general.

---

List processing is a sequence of accepting a message, performing
various operations on it which cause state changes to the message
and the list processing system, and optionally emitting some number
of messages at the end.

As such this lends itself to process queues and process pipes.

---

We don't want an over-arching API, or the attempt to solve the
entire peoblem with either one hammer, or one sort of hammer.  The
intent is to build something that the end user/SysAdm can adapt to
his local installation without either stretching or breaking the
model, and without needint to build an installation which is
necessarily structurally very different from either the very light
weight single machine small list system, or the larger
EGroups/Topica equivalent.

By using process queues based on cannonical names in known
filesystem locations and pre-defined data exchange formats between
processes we can make the processes themselves arbitrary black boxes
so long as they accept the appropriate inputs and generate the
expected output.

--<cut>--

-- 
J C Lawrence                                       claw@kanga.nu
---------(*)                        : http://www.kanga.nu/~claw/
--=| A man is as sane as he is dangerous to his environment |=--