[AstroPy] Error when opening Vizier FITS table with astropy Table.read

[Phil Hodge]

It looks as if astropy.io.fits still doesn't accept an empty (or null) 
string as a value for an integer column in an ASCII table.  If you just 
need to see the data values, you can use the tables package in IRAF as a 
workaround:

--> tprint J_ApJS_186_111_table4_mod.fits[1] row=1-3
#  Table J_ApJS_186_111_table4_mod.fits[1]  Tue 15:57:58 31-May-2016

# row 2MASS             Name                     Ap 3.6mag e_3.6mag f_3.6mag
#                                               pix    mag mag

     1 J04034930+2610520 HBC 358 A+B+C         INDEF   9.15 0.02
     2 J04034997+2620382 XEST 06-006           INDEF  INDEF INDEF o
     3 J04035084+2610531 HBC 359               INDEF   9.33 0.02

# row 4.5mag e_4.5mag f_4.5mag 5.8mag e_5.8mag f_5.8mag 8.0mag e_8.0mag 
f_8.0mag
#        mag      mag             mag      mag mag      mag

     1   9.11     0.02            9.05     0.03 9.05     0.03
     2  INDEF    INDEF o         INDEF    INDEF o INDEF    INDEF o
     3   9.26     0.02            9.23     0.03 9.23     0.03

# row Date
#     "Y/M/D"

     1 2005 Feb 20
     2
     3 2005 Feb 20

A printed value of INDEF means the table value was not defined.

Phil


On 05/31/2016 03:42 PM, Phil Hodge wrote:
> The keyword (and ones like it) should be TNULL3, not TBNUL3.  You can 
> fix this by downloading the FITS file and then opening the file as 
> follows:
>
> import astropy.io.fits as fits
> fd = fits.open("J_ApJS_186_111_table4.dat.fits")
>
> Then modify the table header:
>
> hdr = fd[1].header
> hdr.rename_keyword("tbnul3", "tnull3")
> hdr.rename_keyword("tbnul4", "tnull4")
> hdr.rename_keyword("tbnul5", "tnull5")
> hdr.rename_keyword("tbnul7", "tnull7")
> hdr.rename_keyword("tbnul8", "tnull8")
> hdr.rename_keyword("tbnul10", "tnull10")
> hdr.rename_keyword("tbnul13", "tnull13")
> hdr.rename_keyword("tbnul14", "tnull14")
>
> And save it to a new file:
>
> fd.writeto("J_ApJS_186_111_table4_mod.fits")  # or some other name
>
> Phil
>
>
>
> On 05/31/2016 03:25 PM, Eric Jensen wrote:
>>
>> On May 31, 2016, at 2:56 PM, Paul Kuin <npkuin at gmail.com 
>> <mailto:npkuin at gmail.com>> wrote:
>>
>>> Have you tried to read the fits file with astropy.io.fits ? What do 
>>> you get then for the table?
>>
>> It opens the file without error, and I can examine the header. But as 
>> soon as I try to access any part of the data structure, I get the 
>> same ValueError about TNULL3 shown below.
>>
>>
>>
>>>
>>> On Tue, May 31, 2016 at 7:07 PM, Eric Jensen 
>>> <ejensen1 at swarthmore.edu <mailto:ejensen1 at swarthmore.edu>> wrote:
>>>
>>>     Hi all,
>>>
>>>     I’m trying to read a FITS table associated with an ApJ paper, and
>>>     getting an error I don’t understand.  The table in question
>>>     doesn’t have a standard ASCII/CDS version with the full header
>>>     that astropy handles very well (in my experience) with
>>>     format=‘ascii.cds’, so I was trying to use the FITS version
>>>     linked here:
>>>
>>> http://cdsarc.u-strasbg.fr/viz-bin/nph-Cat/fits?J%2FApJS%2F186%2F111/table4.dat
>>>
>>>     Looking at that URL, it would appear that Vizier may be doing an
>>>     on-the-fly conversion to FITS?  But my hope was that using that
>>>     would take into account the information in the Vizier README, and
>>>     thus generate a nice table structure in Python without my having
>>>     to identify columns by hand.  The plain-text version
>>> (http://cdsarc.u-strasbg.fr/vizier/ftp/cats/J/ApJS/186/111/table4.dat)
>>>     doesn’t have header info, though such info is present in the
>>>     separate README file.
>>>
>>>     However, when I read the FITS file like this:
>>>
>>>     from astropy.table import Table
>>>     datafile = 'J_ApJS_186_111_table4.dat.fits'
>>>     t2 = Table.read(datafile, format='fits')
>>>
>>>     I get this error:
>>>
>>>     ValueError: invalid literal for int() with base 10: ' '; the
>>>     header may be missing the necessary TNULL3 keyword or the table
>>>     contains invalid data
>>>
>>>     Looking at the FITS header, there is indeed no TNULL3 keyword,
>>>     but there is a TBNUL3 keyword:
>>>
>>>     TBNUL3 = '        '           / NULL (undefined) value
>>>
>>>     So presumably that keyword has the needed info, but not in the
>>>     field astropy is expecting.  Is there any way to work around this?
>>>
>>>     Possibly related - reading in that file in its text version,
>>>     specifying format=‘cds’ and readme=‘ReadMe’ doesn’t work
>>>     correctly either, because the ReadMe has a combined entry for
>>>     Tables 4 and 5, and says in a note that the third field only
>>>     applies to Table 5.  So this could just be a pathological case
>>>     for this particular file, but I’m curious to know if anyone has
>>>     encountered anything like this or has any recommendations.
>>>
>>>     Thanks in advance for your help with this,
>>>
>>>     Eric
>>>
>>
>>
>>
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org
>> https://mail.scipy.org/mailman/listinfo/astropy
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> https://mail.scipy.org/mailman/listinfo/astropy