[Numpy-discussion] fromiter shape argument -- was Re: For loop tips

[Tim Hochberg]

Torgil Svensson wrote:
>> Yes. fromiter(iterable, dtype, count) works.
>>     
>
> Oh. Thanks. I probably had too old documentation to see this (15
> June). If it's not updated since I'll give Travis a rest about this,
> at least until 1.0 is released :)
>   
Actually I just knew 'cause I wrote it. I don't see a docstring for 
fromiter, although I though I wrote one. Maybe I just forgot?

>> Regardless, L is only iterated over once.
>>     
>
> How can this be true? If no size is given, mustn't numpy either loop
> over L twice or build an internal representation on which it'll
> iterate or copy in chunks?
>   
Well, it can't in general loop over L twice since the only method that L 
is guaranteed to have is next(); that's the extent of the iterator 
protocol. What it does is allocate an initial chunk of memory (the size 
of which I forget -- I did some tuning) and start filling it up. Once 
it's full, it does a realloc, which expands the existing chunk or 
memory, if possible, or returns a new, larger, chunk of memory with the 
data copied into it. Then we iterate on L some more until we fill up the 
new larger chunk, in which case we go get another one, etc. This is 
exactly how list.append works, although in that case the chunk of data 
is acutally a chunk of pointers to objects.

-tim
>
> I just found out that this works
>   
>>>> import numpy,itertools
>>>> rec_dt=numpy.dtype(">i4,S10,f8")
>>>> rec_iter=itertools.cycle([(1,'s',4.0),(5,'y',190.0),(2,'h',-8)])
>>>> numpy.fromiter(rec_iter,rec_dt,10).view(recarray)
>>>>         
> recarray([(1, 's', 4.0), (5, 'y', 190.0), (2, 'h', -8.0), (1, 's', 4.0),
>        (5, 'y', 190.0), (2, 'h', -8.0), (1, 's', 4.0), (5, 'y', 190.0),
>        (2, 'h', -8.0), (1, 's', 4.0)],
>       dtype=[('f0', '>i4'), ('f1', '|S10'), ('f2', '<f8')])
>
> but what's wrong with this?
>
>   
>>>> d2_dt=numpy.dtype("4f8")
>>>> d2_iter=itertools.cycle([(1.0,numpy.nan,-1e10,14.0)])
>>>> numpy.fromiter(d2_iter,d2_dt,10)
>>>>         
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
> TypeError: a float is required
>   
>>>> numpy.__version__
>>>>         
> '1.0b4'
>
> //Torgil
>
>
>
> On 8/30/06, Tim Hochberg <tim.hochberg at ieee.org> wrote:
>   
>> Torgil Svensson wrote:
>>     
>>>>    return uL,asmatrix(fromiter((idx[x] for x in L),dtype=int))
>>>>
>>>>         
>>> Is it possible for fromiter to take an optional shape (or count)
>>> argument in addition to the dtype argument?
>>>       
>> Yes. fromiter(iterable, dtype, count) works.
>>
>>     
>>> If both is given it could
>>> preallocate memory and we only have to iterate over L once.
>>>
>>>       
>> Regardless, L is only iterated over once. In general you can't rewind
>> iterators, so that's a requirement. This is accomplished by doing
>> successive overallocation similar to the way appending to a list is
>> handled. By specifying the count up front you save a bunch of reallocs,
>> but no iteration.
>>
>> -tim
>>
>>
>>
>>     
>>> //Torgil
>>>
>>> On 8/29/06, Keith Goodman <kwgoodman at gmail.com> wrote:
>>>
>>>       
>>>> On 8/29/06, Torgil Svensson <torgil.svensson at gmail.com> wrote:
>>>>
>>>>         
>>>>> something like this?
>>>>>
>>>>> def list2index(L):
>>>>>    uL=sorted(set(L))
>>>>>    idx=dict((y,x) for x,y in enumerate(uL))
>>>>>    return uL,asmatrix(fromiter((idx[x] for x in L),dtype=int))
>>>>>
>>>>>           
>>>> Wow. That's amazing. Thank you.
>>>>
>>>> -------------------------------------------------------------------------
>>>> Using Tomcat but need to do more? Need to support web services, security?
>>>> Get stuff done quickly with pre-integrated technology to make your job easier
>>>> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>>>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>>>> _______________________________________________
>>>> Numpy-discussion mailing list
>>>> Numpy-discussion at lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>>>>
>>>>
>>>>         
>>> -------------------------------------------------------------------------
>>> Using Tomcat but need to do more? Need to support web services, security?
>>> Get stuff done quickly with pre-integrated technology to make your job easier
>>> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>>> _______________________________________________
>>> Numpy-discussion mailing list
>>> Numpy-discussion at lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>>>
>>>
>>>
>>>       
>>
>> -------------------------------------------------------------------------
>> Using Tomcat but need to do more? Need to support web services, security?
>> Get stuff done quickly with pre-integrated technology to make your job easier
>> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>> _______________________________________________
>> Numpy-discussion mailing list
>> Numpy-discussion at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>>
>>     
>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>
>
>