[Chicago] Optimiztion, was Re: guido @ google

[Peter Fein]

Peter Harkins wrote:
> Were I an employer, I would prefer to hire web programmer to write in
> Python. Writing in Python is simply several times faster than writing in
> C. If Python is too slow*, I'd just take the money I don't spend hiring
> several C developers and cluster a couple dozen boxes.

This is basically the strategy taken by my company.  A few comments:

1. Optimizing python code is a lot easier than I expected.  I was able
to achieve around a 10x speed up to a critical code block in around a
day's worth of work.  This was mostly pure-python modifications -
replacing lists with tuples (16% faster for slicing!), and removing some
redundant calculations, but not changing the core algorithm. Knowing
something about intepreter internals (even though I've never written a
line of CPython) helps - for example, generator comprehensions are
*much* faster than lambda + imap. When your done with the obvious
things, run it through hotshot.

2. Building a cluster can be a lot more complicated than just buying
more boxes, though it depends on your app.  And don't forget the costs
beyond initial purchase - administration and rack space are not free.
That said, you can scale your investment much more gradually -
programmers come in units of tens of thousands of dollars, while
hardware comes in units of hundreds of dollars.

3. Asking your boss to cut you a check for every machine you don't need
to buy b/c you've optimized your code is unlikely to yield satisfactory
results. ;)