[Numpy-discussion] load from text files Pull Request Review

[Derek Homeier]

On 02.09.2011, at 11:45PM, Christopher Jordan-Squire wrote:
>>> 
>>> and unfortunately it's for 1D-arrays only).
>> 
>> That's not bad for this use -- make a row a struct dtype, and you've got
>> a 1-d array anyway -- you can optionally convert to a 2-d array after
>> the fact.
>> 
>> I don't know why I didn't think of using fromiter() when I build
>> accumulator.  Though what I did is a bit more flexible -- you can add
>> stuff later on, too, you don't need to do it allat once.
>> 
> 
> I'm unsure how to use fromiter for missing data. It sounds like a
> potential solution when no data is missing, though.

Strange I haven't thought about it before either; I guess for record arrays it 
comes more natural to view them as a collection of 1D arrays. 
However, you'd need to construct a list or something of ncolumn iterators from 
the input - should not be too hard; but then 
how do you feed the ncolumn fromiter() instances synchronously from that?? 
As far as I can see there is no way to make them read one item at a time, 
row by row. Then there are additional complications with multi-D dtypes, 
and in your case, especially datetime instances, but the problem that all columns 
have to be read in in parallel really seems to be the showstopper here. 
Of course for "flat" 2D arrays of data (all the same dtype) this would work with 
simply reshaping the array - that's probably even the most common use case for 
loadtxt, but that method lacks way too much generality for my taste.
Back to accumulator, I suppose. 

Cheers,
						Derek
--
----------------------------------------------------------------
Derek Homeier          Centre de Recherche Astrophysique de Lyon
ENS Lyon                                      46, Allée d'Italie
69364 Lyon Cedex 07, France                  +33 1133 47272-8894
----------------------------------------------------------------