[AstroPy] PyFITS Too many open files

[Nicholas Earl]

Thanks for the ideas! It seems that memmap=False solves the problem,
although I can see that it does begin using a large percentage of the
memory -- I can only imagine how much if I let it run on the entire set of
directories... Is there not any way to flush it all after I close the file?

I am using python 2.7.5, and PyFITs 3.1.2.

Thanks again, everyone, you're awesome.

Nicholas Earl
Graduate Research Assistant
San Diego State University


On Sat, Jun 15, 2013 at 11:01 AM, Nico Nell <nico.nell at gmail.com> wrote:

> Sorry -- I should have read the blip of code better.
>
> So, what version of python and pyfits are you using? How many files are
> you opening?
>
> What happens if you add 'memmap=False' in each of your pyfits open()
> lines. The pyfits mmap stuff is relatively new so maybe there is an issue.
>
> ~Nick
>
>
> On Sat, Jun 15, 2013 at 11:29 AM, Nicholas Earl <nchlsearl at gmail.com>wrote:
>
>> Sorry, but it seems I sent this back to only Nice Nell, and not the
>> entire list!
>>
>> -----
>>
>> Thanks for the reply. I was under the impression that the with command
>> automatically closed the file? Regardless, I went ahead and wrote the
>> script up like this
>>
>> import pyfits
>>
>> for filename in os.listdir(directory):
>>  kep_id = filename[4:13]
>> fits = pyfits.open(directory + "/" + filename, ignore_missing_end=True)
>>  time = fits[1].data["TIME"]
>> energy = fits[1].data["ENERGY"]
>>  fits.close()
>>
>> Unfortunately, I get the same error.
>>
>> Nicholas Earl
>> Graduate Research Assistant
>> San Diego State University
>>
>>
>> On Fri, Jun 14, 2013 at 10:08 PM, Nico Nell <nico.nell at gmail.com> wrote:
>>
>>> You need to close the files once you have retrieved the data...
>>>
>>> See here:
>>> http://pythonhosted.org/pyfits/api_docs/api_hdulists.html#pyfits.HDUList.close
>>>
>>> ~Nick
>>>
>>>
>>> On Fri, Jun 14, 2013 at 10:51 PM, Nicholas Earl <nchlsearl at gmail.com>wrote:
>>>
>>>> Hello everyone,
>>>>
>>>> I'm having an issue with opening and retrieving information for a large
>>>> amount of fits files. Mainly, I keep running into a mmap.error: [Errno
>>>> 24] Too many open files error. The code I'm using is:
>>>>
>>>> import pyfits
>>>>
>>>> for filename in os.listdir(directory):
>>>>  kep_id = filename[4:13]
>>>>
>>>> with pyfits.open(directory + "/" + filename, ignore_missing_end=True)
>>>> as fits:
>>>>  time = fits[1].data["TIME"]
>>>> energy = fits[1].data["ENERGY"]
>>>>
>>>> Any ideas would be greatly appreciated!
>>>>
>>>> Nicholas Earl
>>>> Graduate Research Assistant
>>>> San Diego State University
>>>>
>>>> _______________________________________________
>>>> AstroPy mailing list
>>>> AstroPy at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/astropy
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20130615/4fe72d32/attachment.html>