File Closing Problem in 2.3 and 2.4, Not in 2.5 (Final report)

[Carroll, Barry]

Hi, Gabriel,

> -----Original Message-----
> From: Gabriel Genellina [mailto:gagsl-py at yahoo.com.ar]
> Sent: Tuesday, January 09, 2007 4:06 PM
> To: python-list at python.org
> Subject: RE: File Closing Problem in 2.3 and 2.4, Not in 2.5 (Final
> report)
> 
> At Tuesday 9/1/2007 20:31, Carroll, Barry wrote:
> 
> >I've spent about a day investigating our "too many open files" error.  I
> >found the following:
> >
> >         1. Windows XP allows a Python 2.5 script to open 509 concurrent
> >            files.
> 
> And do you actually need so many open files simultaneously?

No. In fact, that was the original point of my post.  We expected that the maximum number of file objects (open and otherwise) would never be more than five or six.  We believed (apparently incorrectly) that destroying the table parser object would destroy its associated file object and close the referenced file.  Here is the code that does this:

##########
    def unloadtable(self, log=True):
        """Unload current table and pop prior processor from call stack.
        """

        # SNIP

        # force clean up for unloaded table
        self.processor = None
        gc.collect()
        # pop prior processor from stack
        self.processor = self.pstack.pop()

        # SNIP

##########

The first assumption was correct: the file object is destroyed.  The second assumption is apparently incorrect: the file handle is not released when the associated object is destroyed.  These 'orphaned' file handles build up and when the OS handle limit is reached, "too many open files" errors show up.

> Try to close them explicitly when you finish working on them - do
> not rely on GC for closing files. This has *always* been the
> recomended practice (except maybe, inside a short script that
> finishes quickly).

This is what we have done.  The unloadtable function now looks like this:


##########
    def unloadtable(self, log=True):
        """Unload current table and pop prior processor from call stack.
        """

        # SNIP

        # force clean up for unloaded table
        self.processor.source.close()
        self.processor = None
        gc.collect()
        # pop prior processor from stack
        self.processor = self.pstack.pop()

        # SNIP

##########

> On Python 2.5 you can use the new `with` statement.
> 

Not only that, the file errors don't appear when we run our program/table combination under 2.5.  I can't explain the difference, unless the work done to implement the 'with' statement has an impact outside the scope of that feature.  In any event, it does us no good.  Other constraints on the test system require us to stick with Python 2.3, at least for the present.  

> 
> --
> Gabriel Genellina
> Softlab SRL
> 
> __________________________________________________
> Preguntá. Respondé. Descubrí.
> Todo lo que querías saber, y lo que ni imaginabas,
> está en Yahoo! Respuestas (Beta).
> ¡Probalo ya!
> http://www.yahoo.com.ar/respuestas
> 


Thanks for your interest.  
 
Barry
barry.carroll at psc.com
541-302-1107
________________________
We who cut mere stones must always be envisioning cathedrals.

-Quarry worker's creed