[AstroPy] PyFITS memory usage

[Taro Sato]

On Fri, May 13, 2011 at 1:58 PM, G Jones <glenn.caltech at gmail.com> wrote:
> Hello,
> I am seeing what looks like a memory leak when using PyFITS. It may not
> actually be a leak, but I am curious if anyone can explain the behavior. I
> am processing PSRFITS files which are ~40 GB and have a table with the data
> in it. Since the files are so large, I open them with memmap=True. I would
> expect the following simple script to have a constant memory footprint, but
> instead I see continuously growing memory usage.
>
> import pyfits
>
> hdul = pyfits.open('myfile.fits',mode="readonly", memmap=True)
> primary,secondary = hdul
> for k in range(100):
>   x = secondary.data[k].field('DATA').astype('float32')
>
>
> I can see that the data is being read in from disk as it should be, but I
> don't expect the memory usage to increase, since x is constantly
> overwritten. The versions of programs I am using are:


What if you do:

for k in range(100):
  ...
  del x

to sort of force memory release within the loop?  Not sure how the
refcounting/garbage collection works in Python anymore, but I've been
successful in the past with this strategy, when I had tight memory
budget.






-- 
Taro Sato
Department of Astronomy & Physics
Saint Mary's University             email: taro at ap.smu.ca
Halifax, NS B3H 3C3                 phone: (902)420-5027
Canada                                web: http://ap.smu.ca/~taro