[AstroPy] Error when opening Vizier FITS table with astropy Table.read

[Thomas Robitaille]

Hi Phil,

Could you open an issue in the GitHub repository to make sure we don't
lose track of this problem?

Thanks!
Tom


On 31 May 2016 at 21:08, Phil Hodge <hodge at stsci.edu> wrote:
> It looks as if astropy.io.fits still doesn't accept an empty (or null)
> string as a value for an integer column in an ASCII table.  If you just need
> to see the data values, you can use the tables package in IRAF as a
> workaround:
>
> --> tprint J_ApJS_186_111_table4_mod.fits[1] row=1-3
> #  Table J_ApJS_186_111_table4_mod.fits[1]  Tue 15:57:58 31-May-2016
>
> # row 2MASS             Name                     Ap 3.6mag e_3.6mag f_3.6mag
> #                                               pix    mag mag
>
>     1 J04034930+2610520 HBC 358 A+B+C         INDEF   9.15 0.02
>     2 J04034997+2620382 XEST 06-006           INDEF  INDEF INDEF o
>     3 J04035084+2610531 HBC 359               INDEF   9.33 0.02
>
> # row 4.5mag e_4.5mag f_4.5mag 5.8mag e_5.8mag f_5.8mag 8.0mag e_8.0mag
> f_8.0mag
> #        mag      mag             mag      mag mag      mag
>
>     1   9.11     0.02            9.05     0.03 9.05     0.03
>     2  INDEF    INDEF o         INDEF    INDEF o INDEF    INDEF o
>     3   9.26     0.02            9.23     0.03 9.23     0.03
>
> # row Date
> #     "Y/M/D"
>
>     1 2005 Feb 20
>     2
>     3 2005 Feb 20
>
> A printed value of INDEF means the table value was not defined.
>
> Phil
>
>
>
> On 05/31/2016 03:42 PM, Phil Hodge wrote:
>>
>> The keyword (and ones like it) should be TNULL3, not TBNUL3.  You can fix
>> this by downloading the FITS file and then opening the file as follows:
>>
>> import astropy.io.fits as fits
>> fd = fits.open("J_ApJS_186_111_table4.dat.fits")
>>
>> Then modify the table header:
>>
>> hdr = fd[1].header
>> hdr.rename_keyword("tbnul3", "tnull3")
>> hdr.rename_keyword("tbnul4", "tnull4")
>> hdr.rename_keyword("tbnul5", "tnull5")
>> hdr.rename_keyword("tbnul7", "tnull7")
>> hdr.rename_keyword("tbnul8", "tnull8")
>> hdr.rename_keyword("tbnul10", "tnull10")
>> hdr.rename_keyword("tbnul13", "tnull13")
>> hdr.rename_keyword("tbnul14", "tnull14")
>>
>> And save it to a new file:
>>
>> fd.writeto("J_ApJS_186_111_table4_mod.fits")  # or some other name
>>
>> Phil
>>
>>
>>
>> On 05/31/2016 03:25 PM, Eric Jensen wrote:
>>>
>>>
>>> On May 31, 2016, at 2:56 PM, Paul Kuin <npkuin at gmail.com
>>> <mailto:npkuin at gmail.com>> wrote:
>>>
>>>> Have you tried to read the fits file with astropy.io.fits ? What do you
>>>> get then for the table?
>>>
>>>
>>> It opens the file without error, and I can examine the header. But as
>>> soon as I try to access any part of the data structure, I get the same
>>> ValueError about TNULL3 shown below.
>>>
>>>
>>>
>>>>
>>>> On Tue, May 31, 2016 at 7:07 PM, Eric Jensen <ejensen1 at swarthmore.edu
>>>> <mailto:ejensen1 at swarthmore.edu>> wrote:
>>>>
>>>>     Hi all,
>>>>
>>>>     I’m trying to read a FITS table associated with an ApJ paper, and
>>>>     getting an error I don’t understand.  The table in question
>>>>     doesn’t have a standard ASCII/CDS version with the full header
>>>>     that astropy handles very well (in my experience) with
>>>>     format=‘ascii.cds’, so I was trying to use the FITS version
>>>>     linked here:
>>>>
>>>>
>>>> http://cdsarc.u-strasbg.fr/viz-bin/nph-Cat/fits?J%2FApJS%2F186%2F111/table4.dat
>>>>
>>>>     Looking at that URL, it would appear that Vizier may be doing an
>>>>     on-the-fly conversion to FITS?  But my hope was that using that
>>>>     would take into account the information in the Vizier README, and
>>>>     thus generate a nice table structure in Python without my having
>>>>     to identify columns by hand.  The plain-text version
>>>> (http://cdsarc.u-strasbg.fr/vizier/ftp/cats/J/ApJS/186/111/table4.dat)
>>>>     doesn’t have header info, though such info is present in the
>>>>     separate README file.
>>>>
>>>>     However, when I read the FITS file like this:
>>>>
>>>>     from astropy.table import Table
>>>>     datafile = 'J_ApJS_186_111_table4.dat.fits'
>>>>     t2 = Table.read(datafile, format='fits')
>>>>
>>>>     I get this error:
>>>>
>>>>     ValueError: invalid literal for int() with base 10: ' '; the
>>>>     header may be missing the necessary TNULL3 keyword or the table
>>>>     contains invalid data
>>>>
>>>>     Looking at the FITS header, there is indeed no TNULL3 keyword,
>>>>     but there is a TBNUL3 keyword:
>>>>
>>>>     TBNUL3 = '        '           / NULL (undefined) value
>>>>
>>>>     So presumably that keyword has the needed info, but not in the
>>>>     field astropy is expecting.  Is there any way to work around this?
>>>>
>>>>     Possibly related - reading in that file in its text version,
>>>>     specifying format=‘cds’ and readme=‘ReadMe’ doesn’t work
>>>>     correctly either, because the ReadMe has a combined entry for
>>>>     Tables 4 and 5, and says in a note that the third field only
>>>>     applies to Table 5.  So this could just be a pathological case
>>>>     for this particular file, but I’m curious to know if anyone has
>>>>     encountered anything like this or has any recommendations.
>>>>
>>>>     Thanks in advance for your help with this,
>>>>
>>>>     Eric
>>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> AstroPy mailing list
>>> AstroPy at scipy.org
>>> https://mail.scipy.org/mailman/listinfo/astropy
>>
>>
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org
>> https://mail.scipy.org/mailman/listinfo/astropy
>
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> https://mail.scipy.org/mailman/listinfo/astropy