[AstroPy] PyFITS Too many open files

[Anthony Smith]

Hi all,

I haven't been looking into this properly, but if you copy the arrays, then
surely it frees the files to be closed? E.g.,

time = fits[1].data["TIME"].copy()

rather than

time = fits[1].data["TIME"]

Worth a try...

Anthony



2013/6/15 Nico Nell <nico.nell at gmail.com>

> You can try running the python garbage collector after each file close
> with gc.collect().
>
> For the memmap issue consider submitting a bug report if you have time:
> https://github.com/spacetelescope/PyFITS/issues?state=open
>
> ~Nick
>
>
> On Sat, Jun 15, 2013 at 4:21 PM, Nicholas Earl <nchlsearl at gmail.com>wrote:
>
>> Thanks for the ideas! It seems that memmap=False solves the problem,
>> although I can see that it does begin using a large percentage of the
>> memory -- I can only imagine how much if I let it run on the entire set of
>> directories... Is there not any way to flush it all after I close the file?
>>
>> I am using python 2.7.5, and PyFITs 3.1.2.
>>
>> Thanks again, everyone, you're awesome.
>>
>> Nicholas Earl
>> Graduate Research Assistant
>> San Diego State University
>>
>>
>> On Sat, Jun 15, 2013 at 11:01 AM, Nico Nell <nico.nell at gmail.com> wrote:
>>
>>> Sorry -- I should have read the blip of code better.
>>>
>>> So, what version of python and pyfits are you using? How many files are
>>> you opening?
>>>
>>> What happens if you add 'memmap=False' in each of your pyfits open()
>>> lines. The pyfits mmap stuff is relatively new so maybe there is an issue.
>>>
>>> ~Nick
>>>
>>>
>>> On Sat, Jun 15, 2013 at 11:29 AM, Nicholas Earl <nchlsearl at gmail.com>wrote:
>>>
>>>> Sorry, but it seems I sent this back to only Nice Nell, and not the
>>>> entire list!
>>>>
>>>> -----
>>>>
>>>> Thanks for the reply. I was under the impression that the with command
>>>> automatically closed the file? Regardless, I went ahead and wrote the
>>>> script up like this
>>>>
>>>> import pyfits
>>>>
>>>> for filename in os.listdir(directory):
>>>>  kep_id = filename[4:13]
>>>> fits = pyfits.open(directory + "/" + filename, ignore_missing_end=True)
>>>>  time = fits[1].data["TIME"]
>>>> energy = fits[1].data["ENERGY"]
>>>>  fits.close()
>>>>
>>>> Unfortunately, I get the same error.
>>>>
>>>> Nicholas Earl
>>>> Graduate Research Assistant
>>>> San Diego State University
>>>>
>>>>
>>>> On Fri, Jun 14, 2013 at 10:08 PM, Nico Nell <nico.nell at gmail.com>wrote:
>>>>
>>>>> You need to close the files once you have retrieved the data...
>>>>>
>>>>> See here:
>>>>> http://pythonhosted.org/pyfits/api_docs/api_hdulists.html#pyfits.HDUList.close
>>>>>
>>>>> ~Nick
>>>>>
>>>>>
>>>>> On Fri, Jun 14, 2013 at 10:51 PM, Nicholas Earl <nchlsearl at gmail.com>wrote:
>>>>>
>>>>>> Hello everyone,
>>>>>>
>>>>>> I'm having an issue with opening and retrieving information for a
>>>>>> large amount of fits files. Mainly, I keep running into a mmap.error:
>>>>>> [Errno 24] Too many open files error. The code I'm using is:
>>>>>>
>>>>>> import pyfits
>>>>>>
>>>>>> for filename in os.listdir(directory):
>>>>>>  kep_id = filename[4:13]
>>>>>>
>>>>>> with pyfits.open(directory + "/" + filename, ignore_missing_end=True)
>>>>>> as fits:
>>>>>>  time = fits[1].data["TIME"]
>>>>>> energy = fits[1].data["ENERGY"]
>>>>>>
>>>>>> Any ideas would be greatly appreciated!
>>>>>>
>>>>>> Nicholas Earl
>>>>>> Graduate Research Assistant
>>>>>> San Diego State University
>>>>>>
>>>>>> _______________________________________________
>>>>>> AstroPy mailing list
>>>>>> AstroPy at scipy.org
>>>>>> http://mail.scipy.org/mailman/listinfo/astropy
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20130617/c95f8843/attachment.html>