[Mailman-Developers] [scr] Worm Klez.E immunity

[Ron Jarrell]

At 03:34 PM 4/25/02 -0700, you wrote:

>Fabulous.  This goes to my mailing list, apparently from Barry.
>
>F***ing virus writers.  I want them strung up by and with their privates.

We ended up spending upwards of $150K to build a redundant, and fast, 
central email virus scanning solution to solve the problem.  About 1300 
nodes worth of email gets fed through a pair of redundant load balancers 
into one of four duplicated dedicated scanning/routing engines.  The vast 
majority of it gets "intercepted" and handed off to a central mail server 
(so mail to schmuck@schmuckssmtplesspc gets properly rerouted to 
schmuck@arealmailserver).  The rest then gets redistributed to end client 
machines based on some ldap-based routing rules.  (Such as mail to our 
listserv machine, or mail going through the mailman machine.)  We also scan 
outbound from the central servers, and anyone else who uses our outbound relay.

It was expensive as hell (although a lot of fun, in retrospect, after all 
the screaming at "You're *how* busy?" vendors whose stuff tended to fall 
apart under our loads) but well worth it.  Combining the hardware, with 
efforts by our support staff to contact, and delouse, nodes confirmed as 
sending high quantities of virus email, etc, has paid off.  We went from an 
average of 30,000 virus detections a day when the school year (and the 
production project) began, to now we're seeing about 3,000.

Based on historical call records to our central group, and support cost 
analyses, and that fact that we've now intercepted about 1.8 million 
viruses, we estimate we've over a million dollars in staff time.  That 
translates into what our group calls M$ (Management Dollars).  So, 
basically, everyone agrees that we've paid out the purchase price of the 
complex...

<siderant>
Now if we could just get more vendors to understand that a big university 
is not at all comparable to even huge isps.  We're, effectively, bigger 
than they are.  We get vendors telling us "Oh, look at the benchmark 
numbers on our product; we can handle 150,000 users on one 
box."  Yes.  They can.  If only 5% of them call in at any time, use 33.6K 
modems, check their mail on the average of twice a day, send at most 2 1K 
or less notes a day, and generally receive no more than 3 <1K notes, the 
stuff works fine.

Then they come here, where out of 86,000 user accounts I'll have 35,000 
active in any given day, and about 70,000 active in a seven day period, 
with some of them checking email about every 4 seconds (2.1 million pop 
checks a day on average), and about 12,000 of them plugged into a switched 
10meg port that backs onto our gig backbone, and their box bursts into flame...
</siderant>