File Closing Problem in 2.3 and 2.4, Not in 2.5

[Carroll, Barry]

> -----Original Message-----
> From: python-list-bounces+barry.carroll=psc.com at python.org [mailto:python-
> list-bounces+barry.carroll=psc.com at python.org] On Behalf Of John Machin
> Sent: Saturday, January 06, 2007 11:09 PM
> To: python-list at python.org
> Subject: Re: File Closing Problem in 2.3 and 2.4, Not in 2.5
> 
> Martin v. Löwis wrote:
> > Carroll, Barry schrieb:
> > > What I want to know is:
> > >
> > > * has anyone else encountered a problem like this, * how was the
> > > problem corrected, * can the fix be retro-fitted to 2.5 and 2.4?
> >
> > From your description, I suspect an error in your code. Your description
> > indicates that you don't expect to have more than five files open
> > simultaneously. Yet, the error message "Too many open files" occurs when
> > you open many more files (in the order of hundreds of files).
> >
> > It is very unlikely that there is a bug in Python where it would fail to
> > close a file when .close() is explicitly invoked on it (as your
> > description suggests that you do), so if you get that error message, it
> > can only mean that you fail to close some files.
> 
> I don't understand:  the OP's description suggests nothing of the sort
> to me. What he said was:
> """
> In this way, a tree of Parser instances is created, each with a single
> open file object.  (BTW, recursive and circular references are not
> allowed.)  When each Parser instance comes to the end of its table, the
> instance is explicitly destroyed, presumably destroying any objects it
> holds, AND closing its open file.
> """
> which I interpret as: he is doing del parser_instance, and *presuming*
> (incorrectly) that attributes of parser_instance (including an open
> file object) are magically whisked away instantly, instead of
> later/maybe. He later says he explicitly closed the files, which fixed
> what he alleges (incorrectly) to be a bug.
> 
> To the OP:
> (1) The del statement doesn't "destroy" anything. It unbinds the name
> from the object in the current namespace, and decrements the object's
> reference count. Only if the reference count is then zero will the
> janitor be called in.
> (2) Check the reference count on the parser_instance just before you
> del it. You could be retaining a reference somewhere.
> (3) Explicitly close all non-lightweight objects like files (even
> read-only ones) and sockets rather than hoping they will go away.
> 
> HTH,
> John

John:

Thank you.  I was afraid I had been totally misunderstood.  We have retained the explicit file close in the current version of the Parser code.  

I still have a question, however.  Python 2.3 and 2.4 both produce the "Too many open files" error on Windows XP Pro, but Python 2.5 does not.  Running on the 2.5 interpreter, our program process all the table invocations with no errors, a total of over 650 file opens.  Also, Python 2.3 on RedHat Fedora core 1 runs our program to completion with no errors.  (I know, we are running pretty ancient SW here. Constraints unrelated to this question make that necessary.)  Do you know why the error would appear on some combinations of OS and interpreter and not on others?

Thanks again for your help.

Regards,
 
Barry
barry.carroll at psc.com
541-302-1107
________________________
We who cut mere stones must always be envisioning cathedrals.

-Quarry worker's creed