[AstroPy] reading one line from many small fits files

[John K. Parejko]

Follow up on this:

Erin's suggestion to use fitsio gave me a factor of more than 10 improvement in speed. I was quite astonished at how much faster it was, so I've written up a short example, and attached it. On my laptop (13" macbook pro, OS X 10.6.8, regular HDD), the code produces the following:

$ python fits_tester.py
fitsio version: 0.9.0
pyfits version: 3.0.6
Single pass: fitsio took  1.14109 seconds.
Single pass: pyfits took 14.64361 seconds.

One of the problems with the pyfits version is that I don't know how to efficiently get at row(n) of a pyfits object in a form that can be directly ingested into an ndarray. If there is a way to make the pyfits version significantly faster just by calling pyfits differently, I'm all ears.

Looking at the profiles for the runs (output to .prof files), it looks like pyfits is doing a lot of object creation and destruction in the background, which may be what's killing it.

Anyway, there does seem to be a major difference in speed here, even in what is probably the most favorable configuration for pyfits, with it running last and thus having files potentially cached.

Assuming this difference isn't just me, is way to get these speed improvements merged into pyfits?

John


-------------- next part --------------
A non-text attachment was scrubbed...
Name: fits_tester.py
Type: text/x-python-script
Size: 3497 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/astropy/attachments/20120802/84c40d28/attachment.bin>