[Mailman-Developers] Some notes on my stuff...

[Thomas Wouters]

On Sun, Jul 22, 2001 at 10:15:15PM -0700, Chuq Von Rospach wrote:

> For those that don't remember, my big mailman machine (Sun E250, mailman
> 2.0.5, sendmail) was maxing out at about 625 messages processed a day -- and
> we were seeing peak backlogs of 4 hours between receiving and processing.
> Not good.

For what it's worth, we did some performance comparisons between the Sun
E450's we have (single- and dual-CPU, 1G to 2G RAM, with separate LVD SCSI
disks and with a hardware RAID array in RAID5 mode (also LVD SCSI)) and the
1Ghz P3's with 1G RAM but with U160 SCSI (twice the theoretical speed of
'normal' LVD SCSI) and software RAID 5. The intels (running Linux and
FreeBSD) were so much more faster, especially in disk i/o, it was just
scary. And that wasn't even factoring in the price issue. And there's U320
too, which we haven't tested with yet.

The U450's are for sale, by the way. They're lend out to HAL2001, where
they'll be on show, too :) (http://www.hal2001.org)

> And it's probably safe to say you should ALWAYS assume that disk I/O is a
> problem, and rule it out before you start finding other problems... Because
> it's something that will make all of the other problems worse, but not
> necessarily easy to find unless you look for it specifically.

Amen to that. We used to use BSDI a lot, and we always installed it from CD,
using an old external SCSI CD player. It took several hours to do the
install that way, but we always did it that way because that was the way we
did it, and we assumed it was normal for it to take that long. And then one
of the new servers arrived with an IDE CD player in it, 32 speed or so, and
using that, the install was done in 20 minutes. Talk about embarrasing :)
Nowadays, we do PEX installs of FreeBSD and Linux, to fully configured and
customized machines, in 15 minutes :P

We stopped using BSDI before we came to U160, but switching one I/O-bound
application from BSDI 4.x on 'normal' UWD SCSI (40Mb/s) to FreeBSD on U160
SCSI (160Mb/s) really did quadruple the performance, much to all of our
amazement. 

> We're now talking about upgrading all of the email machines to RAID 1+0
> down the road, just to build a system that maximizes the disk performance,
> since it's now clear I've been underestimating it's impact (even though I
> thought I wasn't...)

Picking the right filesystem, if you have a choice, and tuning it is also
important. Not sure howmuch there is to tune about Sun's UFS implementation,
but FreeBSD has some tunables, and Linux has several filesystems to choose
from. FreeBSD also desperately needs tuning if you're going to use it as an
NFS server, by the way.

To bring it back on-topic: I concur that disk i/o is very important in
Mailman, and is probably the chief bottleneck (for those who have
performance problems) and in fact all of the crashes our main list machine
has, are disk i/o related. The machine was also a batched-SMTP machine, so
it ran a *lot* of sendmail. 

-- 
Thomas Wouters <thomas@xs4all.net>

Hi! I'm a .signature virus! copy me into your .signature file to help me spread!