Using Python for processing of large datasets (convincing managment)

[Paul Rubin]

Thomas Jensen <spam at ob_scure.dk> writes:
> I gotta look into this.
> However I am uncertain as how to structure my program.
> One of the tasks of the program will be to calculate the standard
> deviation of rows of daily values (which are the result of another
> calculation, etc). I was planning on using lists and tuples like this:
>      [(date, value), (date, value), ...]
> How well will this perform i wonder? Since lists and tuples are Python
> structures, won't they still be "slow" to traverse?

Lists and tuples are fast to traverse (they're just vectors in memory)
but I don't see their relevance if by "rows" you mean database rows.
You have to iterate over the rows and compute the SD.  I expect the
time for that will be mostly taken by database operations.