help! advocacy resources needed fast

[Geoff Gerrietts]

I feel like I'm misrepresenting things, in the siege of sudden panic,
so I'll clarify a little.

First, we've been talking about changing site architecture for
something approaching a year now. It's always been an outside thing.
"We'd like to do..." but the cost is staggering, we know it's not
going to happen.

Over the last few weeks, some people in the business end of the world
made us privy to some estimates they had, based on deals they were
cooking, angles they were working, and present trend lines. They
estimate that our traffic will not just double (as it has been doing
yearly), but multiply by something like a factor of 10. Even when
adjusted for the optimism of a business projection, this would destroy
our performance. This prompted a call to emergency action: all those
things we've been talking about are going to happen very soon because
now they need to.

We have something approaching 30 servers in play, many of them
commodity-level boxes (PC architectures with varied processor speeds,
in the 1GB RAM range, big beefy database server), and at peak traffic
times, response becomes unacceptably slow. The bottleneck is in Zope;
some portion of this could be blamed on the dynamic nature and
complexity of our pages, but some portion is pushing DTML through a
Python parser on every request.

We have spent a year refactoring key components, and building caching
solutions to minimize the impact of load. We're still having some
problems, and the problems look to increase.

So, yesterday it was sprung on us: we need to find a solution. By next
Monday we need to have direction, because we need to have something in
place to handle this ramp up. Even if none of the deals go through, we
need to be able to handle something like double our current load by
summer.

I'm caught flat-footed, because I'm still looking at developing
technologies, considering options. I haven't seen an architecture
shift as a viable option. The guy who's pushing Java, he's got a nice
easy answer, because Sun's marketing budget makes his case for him. I
have a harder time: anecdotal evidence about increase in programmer
productivity, short turnaround times, shallow learning curve.

And when it comes right down to it, we need examples of sites doing
the same kind of highly-dynamic, high-volume business we're doing, in
some python toolkit, and succeeding. For the most part, sites with
more traffic than ours that use python seem to use it in an auxiliary
role: Google's core is C++, and I don't know what Yahoo's using, but
it's not /all/ Python.

And I need numbers, something to recommend the final solution I want
to push as most viable in Python beyond "a bunch of Python advocates
think it's quality shit". Or, I should say, I think I need numbers.

As dire as I've made things look, every engineer who touches the
Python code, wants to work in Python. We like Python, we're good at
Python, and Python has proven to be our engineering department's
competitive advantage. So, it's not like I'm standing out there alone.
It's just that I'm the guy nominated most likely to come up with an
architecture we could live with.

I hear lots of "probably mod_python" and now some "check out SCGI",
but is anyone doing page templating with these things? I'm gonna do
research, but people with answers please don't be shy, my eyes are
only so good, and I still have "real work" that needs doing.

--G.

-- 
Geoff Gerrietts             "If life were measured by accomplishments, 
<geoff at gerrietts net>     most of us would die in infancy." 
http://www.gerrietts.net/       --A.P. Gouthey