[AstroPy] PyFITS memory usage

[Erik Bray]

On 05/13/2011 12:58 PM, G Jones wrote:
> Hello,
> I am seeing what looks like a memory leak when using PyFITS. It may not
> actually be a leak, but I am curious if anyone can explain the behavior.
> I am processing PSRFITS files which are ~40 GB and have a table with the
> data in it. Since the files are so large, I open them with memmap=True.
> I would expect the following simple script to have a constant memory
> footprint, but instead I see continuously growing memory usage.
>
> import pyfits
>
> hdul = pyfits.open('myfile.fits',mode="readonly", memmap=True)
> primary,secondary = hdul
> for k in range(100):
> x = secondary.data[k].field('DATA').astype('float32')
>
>
> I can see that the data is being read in from disk as it should be, but
> I don't expect the memory usage to increase, since x is constantly
> overwritten. The versions of programs I am using are:
> RHEL 5 x86_64 (2.6.18-194.32.1.el5)
> python 2.7.1
> numpy 2.0.0.dev
> pyfits 2.4.0


There is a known memory leak in PyFITS 2.4.0 (and possibly below).  See 
http://trac6.assembla.com/pyfits/ticket/49.  This has been fixed in the 
current development version, and the fix should be in the next release.

If you can patch your PyFITS installation, I have a patch for the memory 
leak issue that applies cleanly to 2.4.0.

Though enough people have had this issue, that maybe we could release a 
2.4.1 with this fix in the meantime?  That might not be a bad idea.

Erik