File Closing Problem in 2.3 and 2.4, Not in 2.5

[John Machin]

On 9/01/2007 8:25 AM, Carroll, Barry wrote:
>> -----Original Message-----
>> From: python-list-bounces+barry.carroll=psc.com at python.org [mailto:python-
>> list-bounces+barry.carroll=psc.com at python.org] On Behalf Of John Machin
>> Sent: Saturday, January 06, 2007 11:09 PM
>> To: python-list at python.org
>> Subject: Re: File Closing Problem in 2.3 and 2.4, Not in 2.5
>>
>> Martin v. Löwis wrote:
>>> Carroll, Barry schrieb:
>>>> What I want to know is:
>>>>
>>>> * has anyone else encountered a problem like this, * how was the
>>>> problem corrected, * can the fix be retro-fitted to 2.5 and 2.4?
>>> From your description, I suspect an error in your code. Your description
>>> indicates that you don't expect to have more than five files open
>>> simultaneously. Yet, the error message "Too many open files" occurs when
>>> you open many more files (in the order of hundreds of files).
>>>
>>> It is very unlikely that there is a bug in Python where it would fail to
>>> close a file when .close() is explicitly invoked on it (as your
>>> description suggests that you do), so if you get that error message, it
>>> can only mean that you fail to close some files.
>> I don't understand:  the OP's description suggests nothing of the sort
>> to me. What he said was:
>> """
>> In this way, a tree of Parser instances is created, each with a single
>> open file object.  (BTW, recursive and circular references are not
>> allowed.)  When each Parser instance comes to the end of its table, the
>> instance is explicitly destroyed, presumably destroying any objects it
>> holds, AND closing its open file.
>> """
>> which I interpret as: he is doing del parser_instance, and *presuming*
>> (incorrectly) that attributes of parser_instance (including an open
>> file object) are magically whisked away instantly, instead of
>> later/maybe. He later says he explicitly closed the files, which fixed
>> what he alleges (incorrectly) to be a bug.
>>
>> To the OP:
>> (1) The del statement doesn't "destroy" anything. It unbinds the name
>> from the object in the current namespace, and decrements the object's
>> reference count. Only if the reference count is then zero will the
>> janitor be called in.
>> (2) Check the reference count on the parser_instance just before you
>> del it. You could be retaining a reference somewhere.
>> (3) Explicitly close all non-lightweight objects like files (even
>> read-only ones) and sockets rather than hoping they will go away.
>>
>> HTH,
>> John
> 
> John:
> 
> Thank you.  I was afraid I had been totally misunderstood.  We have retained the explicit file close in the current version of the Parser code.  
> 
> I still have a question, however.  Python 2.3 and 2.4 both produce the "Too many open files" error on Windows XP Pro, but Python 2.5 does not.  Running on the 2.5 interpreter, our program process all the table invocations with no errors, a total of over 650 file opens.  Also, Python 2.3 on RedHat Fedora core 1 runs our program to completion with no errors.  (I know, we are running pretty ancient SW here. Constraints unrelated to this question make that necessary.)  Do you know why the error would appear on some combinations of OS and interpreter and not on others?
> 
> Thanks again for your help.
> 
> Regards,
>  
> Barry

Hi Barry,

Please always reply on-list if the communication is non-private -- and 
meaningful :-)

I don't know; here are some things you could check out:

(1) Windows XP versus *x: The limit on open file handles is 512 in 
Windows XP; write yourself a reference-preserving [hint: append to a 
list] multiple file opener on *x and see how many you get open. I have 
vague recollections of the number 1024 being mentioned.

(2) Python 2.4 versus 2.5: Read the "what's new" doc, fire up your 
googler, ...

IMHO, the attitude should be to program defensively so that you need 
neither to know nor care about such implementation-specific concerns. 
Operating systems these days give you enough file handles to do any 
reasonable job. It's many moons since MS-DOS 2.n with its max 20 open 
file handles made folks implement LRU caches for file handles.

Simple rules about unwanted objects include:
(1) If an object has a free() or close() method, then call it!
(2) Unbind *all* references to the object.

Have you checked out my point 2 (holding multiple references to your 
Parser objects)?

Cheers,
John