Using Python for processing of large datasets (convincing managment)

[Cameron Laird]

In article <3D2A078A.7040502 at ob_scure.dk>,
Thomas Jensen  <spam at ob_scure.dk> wrote:
			.
			.
			.
>I have, on purpose, not described the workings of this program in very 
>great detail, since my original post was more about the general idea of 
>using Python for this kind of job. Having easy access to distributed 
>computations is merely a bonus and, if nothing else, a buzz-word to 
>mention to managment *hint*.
>Furthermore, the ability to scale the application simply gives a good 
>feeling, even if it is *never* needed.
			.
			.
			.
>Satisfying, perhaps, but could you elaborate on scalable?
>
>I simply fail to see how it is that distributed computing is so bad? 
>Everybody seems to think that once you make something distributed, every 
>other optimization posibility simply disapear?
>I never said distributed computing was a priority or even would be a 
>part of the first version. I *is* a design goal however, that should we 
>one day, after all other optimizations in the world, using SQL, need 
>more speed, we can do so by adding machines/CPUs (be it DB servers or 
>application servers).
			.
			.
			.
I call SQL noodling "scalable" in the sense that good
SQL queries can be hosted on bigger and bigger servers.
We know how to do that--it's a commercial reality.

I *like* distributed computing.  I've spent much of the
last eighteen months promoting SOAP, XML-RPC, and CORBA.
Your mention of Linda and its descendants, including
T-Spaces, thrilled me.  HOWEVER, I rarely recommend
distribution for performance objectives, for reasons
that have mostly appeared already in this thread.  Com-
mercial applications (as opposed to scientific ones)
just don't find success that way.

Your situation might be an exception.  It's hard to know.
The computations you describe--DB retrievals, elementary
statistics, ...--sound to me like ones that I've seen
most successfully hosted on conventional architectures.
-- 

Cameron Laird <Cameron at Lairds.com>
Business:  http://www.Phaseit.net
Personal:  http://starbase.neosoft.com/~claird/home.html