[SciPy-user] scipy.io.read_array: NaN in data file

[Pierre GM]

On Mar 10, 2009, at 12:22 PM, Dharhas Pothina wrote:
>
> so does np.genfromtxtx also deal with missing values in a file?

Yep:
 >>> data = StringIO.StringIO("""#
1999,4,12,
1999,5,,,
1999,6,12,34
""")
 >>> a = np.genfromtxt(data, delimiter=",", usemask=True)
 >>> a
masked_array(data =
  [[1999.0 4.0 12.0 --]
  [1999.0 5.0 -- --]
  [1999.0 6.0 12.0 34.0]],
              mask =
  [[False False False  True]
  [False False  True  True]
  [False False False False]],
        fill_value = 1e+20)

Looks like the first 2 columns are YYYY and MM: you can use  
scikits.timeseries.tsfromtxt for that, with a special converter to  
transform the first 2 columns into a datearray:
 >>> dconv=lambda y,m: Date('M', year=y, month=m)



> i.e something like:
> 1999,1,22,42
> 1999,2,18,23
> 1999,3,,22
> 1999,4,12,
> 1999,5,,,
> 1999,6,12,34
>
> I've worked out how to do this using np.loadtxt by defining  
> conversions for each column buts its pretty cumbersome and looks  
> like spagetti in the code.
>
> - dharhas
>
>>>> Pierre GM <pgmdevlist at gmail.com> 3/10/2009 11:14 AM >>>
> With the SVN version of Numpy:
>
>>>> import numpy as np
>>>> import StringIO
>
>>>> a = np.genfromtxtx(StringIO.StringIO("1, NaN"), delimiter=",")
>
> If you want to output a MaskedArray:
>>>> a = np.genfromtxt(StringIO.StringIO("1, NaN"), delimiter=",",
>                       missing="NaN", usemask=True)
>>>> isinstance(a, np.ma.MaskedArray)
> True
>
> On Mar 10, 2009, at 11:57 AM, Erik Granstedt wrote:
>
>> Hello,
>>
>> I am using scipy.io.read_array to read in values from data files to
>> arrays.  The data files occasionally contain "NaN"s, and I would like
>> the returned array to also contain "NaN"s.  I've tried calling
>> read_array with:
>>
>> scipy
>> .io.read_array(file('read_array_test.dat','r'),missing=float('NaN'))
>>
>> but this still seems to convert the "NaN"s to 0.0
>>
>> Is there a way to get it to return "NaN"s in the array instead of
>> converting them to 0.0 ?
>>
>> Thanks,
>>
>> -Erik
>> _______________________________________________
>> SciPy-user mailing list
>> SciPy-user at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>
> _______________________________________________
> SciPy-user mailing list
> SciPy-user at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
> _______________________________________________
> SciPy-user mailing list
> SciPy-user at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user