[Tutor] Too many open files

[dman]

On Mon, Dec 03, 2001 at 03:30:08PM -0600, Burchill, Scott B. wrote:
| I have a small program which uses recursion while dealing with a flat file
| database.  I am opening and closing a number of files repeatedly during this
| recursion and I feel like I have done the closing needed but I am still
| being faced with a "too many files open" error.

To debug this (you probably are forgetting to close some files, or
closing it too late) you could make a dict and store a reference to
all the files you open in it (as keys, the value doesn't matter).
When you close a file, remove it from the dict (you can increment a
counter if you want to report how many you closed).  Then, when you
get the exception, iterate over the keys of the dict and print out the
name (and length) to see what files you do have open.  For example :

debug_files = {}
debug_closed = 0
import random
def recurse() :
    i = random.random()
    try :
        f = open( "/tmp/" + str(i) , "w" )
    except OSError , err :
        print err
        files = debug_files.keys()
        print "Open files : %d" % len( files )
        print "Closed files : %d" % debug_closed
        for file in files :
            print file.name
            # now close it just to be nice before we exit
            file.close()
        import sys
        sys.exit( 1 )
    debug_files[ f ] = None
    # just a little magic to simulate program logic
    if i % 2 :
        f.close()
        debug_closed += 1
        del debug_files[ f ]
    recurse()
    

| My questions:  Is there a variable I can reference which holds the number of
| open files?  How can I find out what the maximum number of open
| files is for my system?

Read the kernel source?  (heh, not for Solaris)  The bash manpage
shows that 'ulimit' doesn't display this.  Here's an idea though :

>>> l = [ ]
>>> i = 0
>>> while 1 :
...   l.append( open( "/tmp/" + str(i) , "w" ) )
...   i += 1
... 
Traceback (most recent call last):
  File "<stdin>", line 2, in ?
IOError: [Errno 24] Too many open files: '/tmp/1021'
>>> print len( l )
1021
>>>

Looks like it is 1021 for Linux 2.4.10, x86.  Actually, I bet it is
1024 (nice round number), but python already has stdin, stdout, and
stderr open that I didn't account for.

| I am running as follows:
| Python 2.1.1 (#1, Sep  4 2001, 12:16:58)
| [GCC 3.0] on sunos5

How's gcc 3.0 holding up for you?

-D 

-- 

"GUIs normally make it simple to accomplish simple actions and
impossible to accomplish complex actions."
    --Doug Gwyn  (22/Jun/91 in comp.unix.wizards)