[Numpy-discussion] newbie question - large dataset

[Steve Staneff]

Many thanks to all those who responded to my question.  Not being a real programmer, sometimes it's good to be reminded 
of the basics - such as timing various sections of code.  It turns out that most of the time in the loop was spent on 
two of the functions.  I've rewritten the first to be pre-calculated (a real oversight on my part), and looked at 
improving the second as much as possible.

Some of the other time in the loop is being spent going back to the db server for data, and I'll look into using 
pytables as a way of speeding that up (possibly by initializing temporary tables on each of my client machines).

Thanks again,

Steve

> On 4/7/07, Steve Staneff <staneff at constructiondatares.com> wrote:
> 
>>Hi,
>>
>>I'm looking for a better solution to managing a very large calculation.
>>Set A is composed of tuples a, each of the form a = [float, string]; set B
>>is composed of tuples of similar structure (b = [float, string]).  For
>>each possible combination of a and b I'm calculating c, of the form c =
>>f(a,b) = [g(a[0], b[0]), h(a[1], b[1])] where g() and h() are non-trivial
>>functions.
>