[Mailman-Users] load-balance 2.1.x

[David Lee]

On Fri, 4 Jul 2008, Mark Sapiro wrote:

> David Lee wrote:
> >
> >(This is in relation to 2.0.9, but I would probably move to 2.1.11.)
> >
> >I've been reading FAQ 4.75 ("How do I set up ... to load-balance ...") and
> >Barry's archived email of 2006/11/29 "Load-balancing mailman between two
> >servers".
> >
> >They talk about setting up multiple "slices" for "qfiles".  But it's not
> >clear how this is done.  I suspect I need to copy the "QRUNNERS"
> >definition across from Defaults.py to mm_cfg.py, and then do some sort of
> >edit.  And that this edit should somehow specify both a power-of-2 number
> >of slices (same on both machines), and also some sort of specification of
> >which slices this machine should run (different on both machines).  But
> >what is that edit?  Is it changing the "1" for one or more of the QRUNNER
> >entries (which?) to something like "1:2" (machine1) and "2:2" (machine2)?
> >
> >Have I missed something in the doucmentation about this particular aspect?
>
>
> You need to do more than just edit the QRUNNERS list in mm_cfg.py . If
> you just change the 1's to 2's, mailmanctl will start two of each
> runner on each machine, a 0:2 runner and a 1:2 runner (slices are
> numbered from 0).
>
> You also need to modify mailmanctl, so that on one machine it starts
> only the 0:2 sliced runners and on the other only the 1:2 sliced
> runners.

Thanks for your reply.  Much appreciated.

I'm starting from the ground here, bootstrapping myself through the
process, with very little knowledge other than what is in the publicly
available FAQ (e.g. 4.75) and Barry's email thread.

So if I "need to do more than just edit the QRUNNERS", what are those
extras that I need to do?

My guess of changing "1" to "1:2" was simply a guess in the absence of any
(known) documentation?  So let's backtrack on that. Could you let me know
a reasonable starting point for a specification, please?

You say that I "also need to modify mailmanctl".  But what should I modify
and how?

There's also sorts of words such as "queue runners" and "slices" and
"qfiles".  But what are these mysterious concepts and where can I read
about them (in a "bootstrap" kind of way)?

As I say, I'm happy to keep notes and to try to write them up for the
benefit of others.


> If you are having separate mm_cfg.py files per machine (see below), it
> would be fairly simple to modify the structure of the QRUNNERS list to
> specify not only the number of slices, but which slices to start. This
> would be done by changing the start_all_runners() function in
> mailmanctl.
>
> [...]
> Having separate mm_cfg.py files per machine might be tricky. I have no
> experience with this, so I'm probably overlooking things, but my
> initial thought is it would be easy to just create a symlink from
> Mailman/mm_cfg.py to a file outside the shared file system, but this
> is problematic. What will happen is the first process that imports
> mm_cfg will read it from the symlinked file, but will create
> mm_cfg.pyc in the Mailman/ directory and subsequent processes will
> just read the shared mm_cfg.pyc. You can work around this by also
> having a compiled mm_cfg.pyc symlinked from Mailman/mm_cfg.pyc, but
> you have to be careful, because if you edit mm_cfg.py and then just
> import it, Python will recompile it and put the new mm_cfg.pyc in
> Mailman/mm_cfg.pyc removing the symlink.


OK, so rewind on my earlier attempts at guessing what I might need to do.

Starting again, what would be "best practice" for this?

I currently run a simple single-machine self-contained Mailman server.

What steps (e.g. what edits to what files) do I take to adjust that (with
reasonable simplicity) into a resilient pair (Mailman cluster) using (for
example) Linux-HA/heartbeat and a NetApp NFS file server.

(I realise that the WWW aspect is "out of scope" for this discussion.
That's fine.)

There's some Mailman data (list configs, archives) which needs to be
available to both machines in the cluster and so "factored out" onto the
NetApp NFS box.  But what data (e.g. pathnames) is this?

There's some config. things I need to change (these mysterious "queue
runner", "slice", ... concepts?).  But what, and where and how?

(My guess at per-machine "mm_cfg.py" was simply a guess.  And possibly too
wild for comfort.  And I'd rather do things the tame way!  If there's a
better way than my guess, I'll follow it.)



> I'm not aware of anything beyond what you'll find with the above and
> similar google searches.

Thus it seems difficult for me (or others) to bootstrap myself up to the
requisite level even to understand the concepts, let alone be able then to
map them into details in config files (or, worse, edits to scripts).

But I would be happy to be guided through this, and then to try to
document it for the sake of others.

-- 

:  David Lee                                I.T. Service          :
:  Senior Systems Programmer                Computer Centre       :
:  UNIX Team Leader                         Durham University     :
:                                           South Road            :
:  http://www.dur.ac.uk/t.d.lee/            Durham DH1 3LE        :
:  Phone: +44 191 334 2752                  U.K.                  :