[AstroPy] Advice on reading FITS file with many HDUs

[David Kirkby]

Thanks for the feedback, Perry. Yes, point taken about asking too much from
FITS.  I was hoping there might be a simple fix, but my plan B is to save
the 45K-2 HDUs in an HDF5 file instead. That's a bit more hassle but
probably worth it.

David

On Sat Jan 17 2015 at 6:33:32 PM Perry Greenfield <stsci.perry at gmail.com>
wrote:

> My first reaction is that you really are asking a lot of the fits module
> (45K headers!). If it was only to read the first two, we could consider
> some option to that module not to locate all headers in the file, but just
> the first N, but asking for random access basically requires reading most
> of them anyway.
>
> My second reaction is this is a poor use of FITS. Isn't there something
> more efficient you could do with storing the information in a FITS file? I
> know, I know, someone else decided this (usually) and you don't  have any
> control over that. Still, it makes me wonder seeing things like FITS taken
> to this extreme level.
>
> Really, this seems a better job for CFITSIO (I'd be curious to see how
> much faster it is though since it does have to run through most of the file
> to find the header locations as well, but will avoid the overhead of
> creating 45K objects).
>
> Cheers, Perry
>
> On Jan 17, 2015, at 8:29 PM, David Kirkby wrote:
>
> > I am reading a ~700 Mb file containing ~45K HDUs and looking for advice
> on how to speed things up. I typically only want to read the first 2 HDUs
> right after opening the file, and then a small number of randomly selected
> HDUs while my program runs. The first 2 HDUs are the largest, but still
> only represent about10% of the total file size.
> >
> > The following command takes about 42 seconds:
> >
> >   fits.open('LSST_i.fits',mode='readonly',memmap=False)
> >
> > Changing to memap=True makes no difference but leads to "Too many open
> files" if I try to read too many HDUs after opening the file.
> >
> > Is this what I should expect? Any suggestions for opening the file and
> reading a small number of HDUs faster? If necessary, I can change the
> format of the file I am reading.
> >
> > In case it helps, the file I am testing with can be downloaded from:
> >
> >   http://srs.slac.stanford.edu/DataCatalog/?experiment=LSST-
> DESC&folder=12915647
> >
> > thanks,
> > David
> > _______________________________________________
> > AstroPy mailing list
> > AstroPy at scipy.org
> > http://mail.scipy.org/mailman/listinfo/astropy
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150118/50f0947c/attachment.html>