[AstroPy] Changing dtype when copying masked table throws TypeError

[Aldcroft, Thomas]

On Fri, Aug 21, 2015 at 10:41 AM, Aldcroft, Thomas <
aldcroft at head.cfa.harvard.edu> wrote:

>
>
> On Tue, Jun 16, 2015 at 6:07 PM, Gregory Simonian <
> simonian.7 at buckeyemail.osu.edu> wrote:
>
>> Hi all,
>>
>> I have a Table with several columns which were read in as strings because
>> they had a combination of numeric and non-numeric characters. After
>> cleaning out the non-numeric characters, I'd like to convert the dtype of
>> the column from string to int.
>>
>> Looking at the astropy documentation for modifying tables [0], under the
>> caveats section it states:
>>
>> "The data type for a column cannot be changed in place. In order to do
>> this you must make a copy of the table with the column type changed
>> appropriately."
>>
>> So I try making a new table (the table is small, so I don't mind the
>> temporary memory hit), but I run into the error:
>> "TypeError: Cannot set fill value of string with array of dtype int64"
>> When checking the fill value of the column (which is masked from the
>> reader), it's 'N/A', which makes sense that it wouldn't work. But when I
>> change it to '0', I still get the same problem. I managed to find a minimal
>> working example as:
>>
>> import numpy as np
>> t = Table({'a': ['1']}, masked=True)
>> t['a'].set_fill_value('0')
>> u = Table(t, names=['a'], dtype=[np.int])
>> > TypeError: Cannot set fill value of string with array of dtype int64
>>
>> When debugging the exception, it seems like the Table constructor doesn't
>> convert the fill value of masked tables when generating a new table, which
>> seems to defeat the ability to change the dtype of masked columns. I was
>> wondering if other people have run by this, and if there's a more
>> straightforward way of changing the dtype of a column. This is with Astropy
>> 1.0.3 on Python 2.7.10 through Anaconda on CentOS 6.6.
>>
>> When checking to see if this was reported, I ran into issue #3809 [1],
>> which might end up being the ultimate solution to my problem. It seems like
>> a workaround would be:
>>
>> temp_column = np.ma.asanyarray(t['a'], np.int)
>> del(t['a'])
>> t['a'] = temp_column
>>
>>
> Hi Gregory,
>
> Sorry, your nice bug report fell through the cracks and didn't get a
> response.  You've basically understood the issues correctly, and I just put
> in a pull request that fixes this:
>
> https://github.com/astropy/astropy/pull/4088/files
>


Just to close out this thread, #4088 has been merged.


>
>
> In your workaround you could have done something slightly simpler:
>
>  temp_column = t['a'].astype(np.int)
>
> Cheers,
> Tom
>
>
>
>
>> Thanks!
>>
>>                                             Gregory Simonian
>>
>> [0] http://docs.astropy.org/en/stable/table/modify_table.html#caveats
>> [1] https://github.com/astropy/astropy/issues/3809
>>
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org
>> http://mail.scipy.org/mailman/listinfo/astropy
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150825/8031ec75/attachment.html>