[AstroPy] PyFITS memory usage
Taro Sato
taro at ap.smu.ca
Fri May 13 13:08:02 EDT 2011
On Fri, May 13, 2011 at 1:58 PM, G Jones <glenn.caltech at gmail.com> wrote:
> Hello,
> I am seeing what looks like a memory leak when using PyFITS. It may not
> actually be a leak, but I am curious if anyone can explain the behavior. I
> am processing PSRFITS files which are ~40 GB and have a table with the data
> in it. Since the files are so large, I open them with memmap=True. I would
> expect the following simple script to have a constant memory footprint, but
> instead I see continuously growing memory usage.
>
> import pyfits
>
> hdul = pyfits.open('myfile.fits',mode="readonly", memmap=True)
> primary,secondary = hdul
> for k in range(100):
> x = secondary.data[k].field('DATA').astype('float32')
>
>
> I can see that the data is being read in from disk as it should be, but I
> don't expect the memory usage to increase, since x is constantly
> overwritten. The versions of programs I am using are:
What if you do:
for k in range(100):
...
del x
to sort of force memory release within the loop? Not sure how the
refcounting/garbage collection works in Python anymore, but I've been
successful in the past with this strategy, when I had tight memory
budget.
--
Taro Sato
Department of Astronomy & Physics
Saint Mary's University email: taro at ap.smu.ca
Halifax, NS B3H 3C3 phone: (902)420-5027
Canada web: http://ap.smu.ca/~taro
More information about the AstroPy
mailing list