[AstroPy] Fwd: PyFITS Too many open files

[Chris Beaumont]

(Forgot to reply to list as well!)

Just a guess, but I think the problem is that pyfits is opening files in
"memory map" mode -- that is, the array isn't actually loaded into RAM, but
rather kept on disk as long as possible. This is nice when working with big
files, but probably involves numpy keeping an internal open fille handle,
even after you close the fits file handle (either explicitly or by exiting
the with block).

If that's the case, then passing "memmap=False" to pyfits.open should solve
the problem (but watch out for increased RAM usage!)

chris


On Sat, Jun 15, 2013 at 1:29 PM, Nicholas Earl <nchlsearl at gmail.com> wrote:

> Sorry, but it seems I sent this back to only Nice Nell, and not the entire
> list!
>
> -----
>
> Thanks for the reply. I was under the impression that the with command
> automatically closed the file? Regardless, I went ahead and wrote the
> script up like this
>
> import pyfits
>
> for filename in os.listdir(directory):
>  kep_id = filename[4:13]
> fits = pyfits.open(directory + "/" + filename, ignore_missing_end=True)
>  time = fits[1].data["TIME"]
> energy = fits[1].data["ENERGY"]
>  fits.close()
>
> Unfortunately, I get the same error.
>
> Nicholas Earl
> Graduate Research Assistant
> San Diego State University
>
>
> On Fri, Jun 14, 2013 at 10:08 PM, Nico Nell <nico.nell at gmail.com> wrote:
>
>> You need to close the files once you have retrieved the data...
>>
>> See here:
>> http://pythonhosted.org/pyfits/api_docs/api_hdulists.html#pyfits.HDUList.close
>>
>> ~Nick
>>
>>
>> On Fri, Jun 14, 2013 at 10:51 PM, Nicholas Earl <nchlsearl at gmail.com>wrote:
>>
>>> Hello everyone,
>>>
>>> I'm having an issue with opening and retrieving information for a large
>>> amount of fits files. Mainly, I keep running into a mmap.error: [Errno
>>> 24] Too many open files error. The code I'm using is:
>>>
>>> import pyfits
>>>
>>> for filename in os.listdir(directory):
>>>  kep_id = filename[4:13]
>>>
>>> with pyfits.open(directory + "/" + filename, ignore_missing_end=True) as
>>> fits:
>>>  time = fits[1].data["TIME"]
>>> energy = fits[1].data["ENERGY"]
>>>
>>> Any ideas would be greatly appreciated!
>>>
>>> Nicholas Earl
>>> Graduate Research Assistant
>>> San Diego State University
>>>
>>> _______________________________________________
>>> AstroPy mailing list
>>> AstroPy at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/astropy
>>>
>>>
>>
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>
>


-- 
************************************
Chris Beaumont
Graduate Student
Institute for Astronomy
University of Hawaii at Manoa
2680 Woodlawn Drive
Honolulu, HI 96822
www.ifa.hawaii.edu/~beaumont
************************************



-- 
************************************
Chris Beaumont
Graduate Student
Institute for Astronomy
University of Hawaii at Manoa
2680 Woodlawn Drive
Honolulu, HI 96822
www.ifa.hawaii.edu/~beaumont
************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20130615/bd7ddc1a/attachment.html>