[Mailman-Developers] Features requested...

[Harald Meland]

[Barry A. Warsaw]

> >>>>> "SH" == Steven Hazel <cherub@azrael.dyn.cheapnet.net> writes:
> 
>     Me>    The problem here is that we don't actually keep that
>     Me> information around in the database.  This is bogus, and is
>     Me> something that I hope will get fixed when we have a real User
>     Me> database (don't ask when ;)
> 
>     SH> What are the current plans for this?  I'd like to help.  I
>     SH> should have some time over the coming holidays.
> 
> Well, there are a number of questions (if it was an easy hack it'd
> already be done :).

This is my cue, I guess ;)

> First, do we want to use a real OODB underneath (such as ZODB), or do
> we cook our own?

I tried using ZODB at first, but found that my try at that scaled very
poorly (The FileStorage file was ~160MB even before there were 2000
entries in my database, and the time needed to open the database
seemed to scale with the size of it...).  Dunno if this was due to me
being boneheaded when implementing stuff, or to actual problems with
ZODB, but I kinda lost my spirit for it anyway :-\

So, I surfed to Parnassos, searched the vault there, and came up with
a copy of bplustree.py.  Implemented purely in Python, as opposed to
ZODB, which relies on some C Zope modules.  The disk space used per
record seems very reasonable.  Open time appears to be pretty much
unrelated to file size.  Record access time seems fast enough.  One
problem, though -- this module does not handle multiple simultanous
writes, so we have to be careful with locking -- which *could* make
the user db into a bottleneck.

[ As we put Mailman into production with the ~3.500 lists we have here
  at uio.no, we discovered that the current locking code didn't work
  very well on a system that got so busy it started swapping -- so I
  rewrote the entire file locking module to be more robust, and added
  some code to have the "post" mail injection script exit with
  temporary error codes if it couldn't get a lock on the list within a
  reasonable amount of time -- leaving it to the MTA to try injecting
  the mail again later.  Things have been running smooth ever since. ]

A fresh entry (created to be added as member of LIST-A) in the
resulting user db have these attributes:

  {'admin_for': {},
   'admin_pwd': None,
   'aliases': {'ADDRESS@DOMAIN': 1},
   'member_of': {'LIST-A': 1},
   'member_pwd': 'PASSWORD',
   'preferred_address': 'ADDRESS@DOMAIN'}

Currently 'aliases' and 'preferred_address' isn't used, but the
intention is to use them for associating several mail addresses with
the same user account (probably by some means of email confirmation).

'member_pwd' is stored in cleartext, while 'admin_pwd' (when set) is
stored as a MD5 digest.

An "username" field has been added to all login pages -- currently
we're running with this as a text entry field, but I'll probably
install some changes to use a <SELECT> menu of the list's admins
instead (mainly to reduce the amount of typing an admin has to do when
logging in, but also partly due to the fact that it's common for a
browser to SUBMIT on pressing Enter in a text field iff it's the only
one in the form).


Logins are still done per list (but with your own admin password), as
the cookie secret business has been a bit complicated by all this --
I'm planning to have a cron job generating random list- and
admin-independent secrets ever so often, and use these when issuing
new cookies -- thus removing the complications and making cross-list
single login viable.


It is all rather non-generalized (and part of the code
is... uhmmm... not quite as clean as could be wished for), but it has
been working fine for over a month now (modulo the various bugfixes
I've made along the way, of course).  I agree with those wanting a
generalized API to use whatever database backend appropriate -- but
with all these local changes I'm feeling rather removed from the rest
of Mailman development already, so I don't think I'm likely to start
working on that soonish.


I'm not sure how it's best to proceed with all this from here -- I
think the stuff I've made could prove usable for others beside me, and
I'd sure love to get my CVS tree closer to being in sync with the rest
of you.  However, if I just go ahead and check stuff in, that might
mean pushing 1.2 a fair bit away.

So, for now, I've made my current development tree (which, Right Now,
isn't quite the code we have in production here; I've been holding off
installing changes since I started implementing the login-with-menu
stuff mentioned above.  As some list admins might not want to be
listed as list admins this openly, I've asked for opinions before
installing that change) available at <URL:
http://www.uio.no/~hmeland/tmp/mailman-userdb/>, or if one prefers to
download it all as a single .tar.gz: <URL:
http://www.uio.no/~hmeland/tmp/mailman-userdb.tar.gz>.  This tree
should be up-to-date with current CVS mailman.  I'd appreciate it a
lot if anyone have the time to have a look at it tell me what they
think.

I'll try explaining all this in a more structured fashion later, but
please don't hesitate to ask any specific questions you might have.

[ Apart from the user database stuff, the tree has a fair bit of other
  new stuff in it as well.

  Perhaps most notable is "meta-members", i.e. special list members
  which are dynamically expanded to real mail addresses at list post
  time.  Currently there is only one type of meta-member, MailmanList,
  which expands to the appropriate set of members (digest or regular
  members, based on whether what kind of post this is) on the given
  list.  Other types are planned.

  Other nice stuff is the ability to name lists with full email
  addresses internally (useful when housing lists in several mail
  domains), the use of a cache for building the list overview pages,
  and (useful when having lots of lists, to avoid loading _all_ of
  them to build the overview), the "this list is busy, return incoming
  mail to MTA" hack, and bin/move_to has been extended so it can
  create a copy of list X with another name Y -- to help renaming
  lists internally. ]

-- 
Harald