[Mailman-Developers] Looking at performance again

[Barry Warsaw]

On Wed, 2003-05-14 at 12:41, M.-A. Lemburg wrote:

> Apart from the fact that you'll have tough times when loading
> 100k members into memory, the use of a real database is a
> tradeoff:
> 
> Mailman has lots of code which is built in a way that assumes
> membership data access is fast. With a few 1000 members, things
> can still fit nicely into memory, so that's a valid assumption.
> However, with a few 100k members you really want to be more
> careful and only load data in chunks into memory.
> 
> Mailman needs to be redesigned in a couple of places for that
> to work. One example is the membership admin interface, another
> is personalization.

I totally agree.  I've been fleshing out some api's for mm3 that attempt
to address these exact issues.  I'm not ready to show them to the world
yet, but I'm thinking exactly along these lines.

> I don't think that multi-threading would gain any performance.
> It would make more sense to have Mailman use multiple SMTP
> backends for delivery (MTA clustering).

Here's one place where it would help: with multiple threads, we won't
need to keep multiple copies of the databases in memory.  Maybe chunking
and caching can help with this instead of moving to a threading model.  

> BTW, is Mailman thread-safe ?

Depends.  If we use one thread per qrunner, then I think so.

> > - Does our "NFS-safe" locks impose too much complexity and overhead to
> >   be worth it?  Does anybody actually /use/ Mailman over NFS?  Don't
> >   we sorta suspect the LockFile implementation anyway?  Would we be
> >   better off using kernel file locks, or thread locks if we go to a MT
> >   model?
> 
> You may want to have a look at mx.Misc.FileLock (in egenix-mx-base).
> That's a portable file locking mechanism which is not the
> fastest, but fast enough for most cases.

I'll look: in the meantime, how does it do locking?

> > Okay, now I'm rambling.  What is the lowest hanging fruit that we might
> > be able to attack?  I'm up for any other ideas people have.
> 
> MTA clustering support. Basically just do round-robin
> delivery to a list of SMTP hosts.

I'm pretty sure there are folks on this list that are doing this.  We
may want to make it easier to configure though.

-Barry