[AstroPy] Help with WindowsError: [Error 8]

[Erik Bray]

On 03/12/2014 02:31 PM, Zachary Robinson wrote:
> Sure.
>
> WARNING: File may have been truncated: actual file length (9947040) is smaller
> than the expected size (10736640) [astropy.io.fits.file]
> Filename: C:\Documents and Settings\zach\Desktop\H0638308.FIT
> No.    Name         Type      Cards   Dimensions   Format
> 0    PRIMARY     PrimaryHDU      34   ()
> 1                BinTableHDU    221   892800R x 4C   [J, J, I, I]
>
> This is the output after running fits.info <http://fits.info>().

As I suspected, it sounds like the file is corrupted.  When that "File may have 
been truncated" warning occurs pyfits doesn't outright *crash* because it may 
still be possible to get some information out of the file.  But it still means 
all bets are off.

For what it's worth, I did open an issue to try to improve reading data from 
corrupt files, so that it can at least be partially read (it sounds like most of 
your file is at least there, and I think we can do better than just crashing 
with an error from the OS): https://github.com/spacetelescope/PyFITS/issues/50


Erik

> Zach
>
>
>
> On Wed, Mar 12, 2014 at 1:00 PM, Erik Bray <embray at stsci.edu
> <mailto:embray at stsci.edu>> wrote:
>
>     On 03/11/2014 02:49 PM, Zachary Robinson wrote:
>      > Perry,
>      >
>      > The FITS file is 3.66MB. The dimensions of the table are 892800Rx4C
>
>     This raises the question, however, of what the formats are of the columns.
>     Could you please post the entire header(s) of this file and the output of
>     fits.info <http://fits.info>() on the file?
>
>     Erik
>
>      >> On Mar 11, 2014, at 1:09 PM, Perry Greenfield <stsci.perry at gmail.com
>     <mailto:stsci.perry at gmail.com>> wrote:
>      >>
>      >> How big is the table?
>      >>
>      >> It's quite possible you are running into a 32-bit limitation,
>     particularly if the table is 2GB or bigger.
>      >>
>      >> Perry
>      >>
>      >>
>      >>> On Mar 11, 2014, at 2:06 PM, Zachary Robinson wrote:
>      >>>
>      >>> Hello,
>      >>>
>      >>> I'm currently a graduate student working on a project for my adviser.
>     The data we are using has always been in CSV format and we have a bunch of
>     python programs using these CSV formatted files.  However, a small portion
>     of the oldest data we have is in FITS formatted files.  I've been put in
>     charge of extracting the data from these FITS files and putting them into
>     CSV files so we can use our programs we wrote. I read online that astropy
>     has the ability to do this so I've downloaded it and have started to learn
>     it using the documentation center.
>      >>>
>      >>> However, I have experienced an issue while trying to learn the basics
>     commands of the table data portion of FITS file handling.  My code so far is
>     (infil is the location of the FITS files on my computer):
>      >>>
>      >>> HDUlist = fits.open(infil)
>      >>> HDUlist.info()
>      >>> tbdata = HDUlist[1].data
>      >>> print(tbdata[1])
>      >>>
>      >>> When I try to run the program, I get this error message:
>      >>>
>      >>> Traceback (most recent call last):
>      >>>   File "C:\Documents and Settings\zach\Desktop\Dropbox\Probabilistic
>     SPE Model\GOES\Python Programs\Astropy and FIT Files Tutorial\Reading FITS
>     Files.py", line 52, in <module>
>      >>>     tbdata = HDUlist[1].data
>      >>>   File "C:\Program
>     Files\Python27\lib\site-packages\astropy\utils\misc.py", line 279, in __get__
>      >>>     val = self._fget(obj)
>      >>>   File "C:\Program
>     Files\Python27\lib\site-packages\astropy\io\fits\hdu\table.py", line 285, in
>     data
>      >>>     data = self._get_tbdata()
>      >>>   File "C:\Program
>     Files\Python27\lib\site-packages\astropy\io\fits\hdu\table.py", line 93, in
>     _get_tbdata
>      >>>     self._data_offset)
>      >>>   File "C:\Program
>     Files\Python27\lib\site-packages\astropy\io\fits\hdu\base.py", line 509, in
>     _get_raw_data
>      >>>     return self._file.readarray(offset=offset, dtype=code, shape=shape)
>      >>>   File "C:\Program
>     Files\Python27\lib\site-packages\astropy\io\fits\file.py", line 257, in
>     readarray
>      >>>     shape=shape).view(np.ndarray)
>      >>>   File "C:\Program
>     Files\Python27\lib\site-packages\numpy\core\memmap.py", line 237, in __new__
>      >>>     mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=start)
>      >>> WindowsError: [Error 8] Not enough storage is available to process this
>     command
>      >>>
>      >>> Can anyone explain to me what this error is and how I can go about
>     fixing it? From what I've read online so far, I'm thinking it has something
>     to do with the mmap but I don't know what that is.
>      >>>
>      >>> I'm running this code in Python 2.7 on a 32-bit WinXP machine.
>      >>>
>      >>> Thanks a lot for the help,
>      >>> Zach
>      >>> _______________________________________________
>      >>> AstroPy mailing list
>      >>> AstroPy at scipy.org <mailto:AstroPy at scipy.org>
>      >>> http://mail.scipy.org/mailman/listinfo/astropy
>      >>
>      > _______________________________________________
>      > AstroPy mailing list
>      > AstroPy at scipy.org <mailto:AstroPy at scipy.org>
>      > http://mail.scipy.org/mailman/listinfo/astropy
>      >
>
>     _______________________________________________
>     AstroPy mailing list
>     AstroPy at scipy.org <mailto:AstroPy at scipy.org>
>     http://mail.scipy.org/mailman/listinfo/astropy
>
>