[Numpy-discussion] Creating an ndarray from an iterable over sequences

[Oscar Benjamin]

On Tue, Jan 21, 2014 at 07:34:19AM +0100, Dr. Leo wrote:
> Hi,
> 
> I would like to write something like:
> 
> In [25]: iterable=((i, i**2) for i in range(10))
> 
> In [26]: a=np.fromiter(iterable, int32)
> ---------------------------------------------------------------------------
> ValueError                                Traceback (most recent call
> last)
> <ipython-input-26-5bcc2e94dbca> in <module>()
> ----> 1 a=np.fromiter(iterable, int32)
> 
> ValueError: setting an array element with a sequence.
> 
> 
> Is there an efficient way to do this?
> 
> Creating two 1-dimensional arrays first is costly as one has to
> iterate twice over the data. So the only way I see is creating an
> empty [10,2] array and filling it row by row. This is memory-efficient
> but slow. List comprehension is vice versa.

You could use itertools:

>>> from itertools import chain
>>> g = ((i, i**2) for i in range(10))
>>> import numpy
>>> numpy.fromiter(chain.from_iterable(g), numpy.int32).reshape(-1, 2)
array([[ 0,  0],
       [ 1,  1],
       [ 2,  4],
       [ 3,  9],
       [ 4, 16],
       [ 5, 25],
       [ 6, 36],
       [ 7, 49],
       [ 8, 64],
       [ 9, 81]], dtype=int32)


Oscar