[Numpy-discussion] how to build a series of arrays as I go?

[Robert Kern]

On Tue, Mar 18, 2008 at 4:25 AM, Chris Withers <chris at simplistix.co.uk> wrote:
> Robert Kern wrote:
>  > Appending to a list is almost always better than growing an array by
>  > concatenation. If you have a real need for speed, though, there are a
>  > few tricks you can do at the expense of complexity.
>
>  I don't for this project but I might in future, where can I read about this?

There was a thread on one of the scipy lists several years ago, I
think. Before April 2005 certainly because I found a message from
myself referencing it.

Basically, if you are constructing a 1D array by appending individual
elements, the stdlib's array module is actually quite useful. It uses
the same preallocation strategy as lists. You then use
numpy.fromstring(buffer(pyarray), dtype=whatever) to create the numpy
array.

If you are building up a 1D array by chunks instead of individual
elements, it probably depends on the type of the chunks. If the chunks
are already arrays, I believe that appending the chunks to a list and
using hstack() will be the best. If the chunks are still lists,
probably .extend()ing the accumulator list is probably best.

For (N>1)D arrays, append to lists.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco