FDs will be closed after exception automatically in python2.7?

[Cameron Simpson]

On 10Jun2019 12:47, Peter J. Holzer <hjp-python at hjp.at> wrote:
>On 2019-06-10 18:18:07 +0800, lampahome wrote:
>> I confused will fd will be close after exception automatically?
>>
>> Like:
>> try:
>> fd=open("file","w+")
>> fd.get() //any useless function of fd
>> except Exception:
>> print 'hi'
>
>Please try to preserve indentation when sending code to the list.
>Indentation is part of the Python syntax and if you smash everything to
>the left margin your program is not only hard to read but syntactically
>wrong.
>
>To answer your question:
>
>No, an exception will not automatically close open file descriptors
>(which file descriptors should it close and why?).
>
>However, while handling an exception the variable holding the file
>descriptor may go out of scope (for example if you leave a "with"
>block[1] or return from a function). In that case the file descriptor
>will be closed (in the case of a with block immediately, otherwise when
>the garbage collector gets around to it).

We want to be more precise here.

By using the term "file descriptor" above you've conflated 2 things: the 
Python file object and the OS file descriptor. They're different things 
with different behaviours.

What you get from "open()" is a python "file object". And what the OP is 
getting in his example code is also a file object. And as you say, when 
all references to that are removed (variable goes out of scope), the 
Python interpreter will close the file.

So far so good.

However the term "file descriptor", at least in POSIX (UNIX and Linux), 
is _not_ the same as a Python "file object". Instead, it is the integer 
you get from a OS level file open such as returned by os.open(). It is 
_not_ managed by the Python interpreter and does not get closed when 
nobody references it, because it is not an object.

Now, for the OP's edification: when you use Python's open() function you 
get a file object, which includes a reference to the _underlying_ OS 
level file descriptor. When that object get recovered _it_ will close 
the file descriptor as part of the close operation.

However, if you do an OS level file open which returns _just_ the file 
descriptor, this does not happen because there's no object cleanup and 
nothing to close the descriptor, because nothing is managing it.

Example code:

    #!/usr/bin/python
    import os
    def open_and_fail():
        fd = os.open('/dev/null',os.O_RDONLY)
        raise RuntimeError("blam")
    try:
        open_and_fail()
    except RuntimeError:
        pass
    os.system("lsof -p "+str(os.getpid()))

Here's some code which does an OS level open, getting a fie descriptor.  
The opening function raises an exception. After we've caught it, we run 
"lsof" on our own process. here's it on my Mac:

    [~]fleet*1> /usr/bin/python fd.py
    COMMAND   PID    USER   FD   TYPE DEVICE  SIZE/OFF      NODE NAME
    Python  97417 cameron  cwd    DIR    1,5      9214    934731 /Users/cameron
    Python  97417 cameron  txt    REG    1,5     25152 199261280 /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
    Python  97417 cameron  txt    REG    1,5   2633024 199261275 /System/Library/Frameworks/Python.framework/Versions/2.7/Python
    Python  97417 cameron  txt    REG    1,5     52832 199261366 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_locale.so
    Python  97417 cameron  txt    REG    1,5    643792 199264919 /usr/lib/dyld
    Python  97417 cameron  txt    REG    1,5 560200432 200219840 /private/var/db/dyld/dyld_shared_cache_x86_64
    Python  97417 cameron    0u   CHR  16,47    0t1529      1991 /dev/ttys047
    Python  97417 cameron    1u   CHR  16,47    0t1529      1991 /dev/ttys047
    Python  97417 cameron    2u   CHR  16,47    0t1529      1991 /dev/ttys047
    Python  97417 cameron    3r   CHR    3,2       0t0       305 /dev/null

See the last line. That is file descriptor 3, attached to /dev/null. It 
is still open.

When your programme exits the OS will clean this up, but until then the 
_descriptor_ remains open.

If you rewrote this with plain Python "open("/dev/null")" instead of the 
os.open you would not see the file descriptor lying around because the 
close of the _file object_ would have cleaned up the related OS file 
descriptor.

Cheers,
Cameron Simpson <cs at cskk.id.au>