[IronPython] Last change needed for working tempfiles

[Martin (gzlist)]

As mentioned in the release thread[1] tempfiles nearly work in
IronPython 2.6.1 RC 1 now, they go away when explicitly closed or when
the process ends, thanks to adding support[2] for the os.O_TEMPORARY
flag. The current bug[3] is actually fixed as stated in IronPython
2.6.1 RC 1 as can be seen by the following testcase:

def test_named_temporary_file_is_removed_by_close():
    """Temporary files should be removed from the filesystem when closed"""
    import os, tempfile
    tf = tempfile.NamedTemporaryFile()
    f = tf.file
    try:
        tf.close()
        if os.path.exists(tf.name): raise AssertionError(
            "Temporary file not removed on close")
    except:
        f.close()
        raise

However, the larger problem with tempfiles isn't quite done yet, as
can be seen by the following failing case:

def test_named_temporary_file_is_removed_by_gc():
    """Unreferenced temporary files should be removed from the filesystem"""
    import os, tempfile, weakref, gc
    tf = tempfile.NamedTemporaryFile(dir=".")
    wf = weakref.ref(tf.file)
    try:
        fname = tf.name
        del tf
        gc.collect()
        if os.path.exists(fname): raise AssertionError(
            "Temporary file not removed on deletion")
    except:
        f = wf()
        if f is not None:
            f.close()
        raise

Both those tests are from my mini-suite[4] of issues I've needed to
workaround. The underlying problem is in Src\IronPython.Modules\nt.cs
here:

public static object fdopen(CodeContext/*!*/ context, int fd, string
mode, int bufsize) {
    // check for a valid file mode...
    PythonFile.ValidateMode(mode);

    PythonContext pythonContext = PythonContext.GetContext(context);
    PythonFile pf = pythonContext.FileManager.GetFileFromId(pythonContext, fd);
    return pf;
}

A change like this should prevent the file from being kept alive:

     PythonFile pf = pythonContext.FileManager.GetFileFromId(pythonContext, fd);
+    pythonContext.FileManager.Remove(pf);
     return pf;

However, will that trip the following assertion? Some code might rely on this.

    f = os.fdopen(fd)
    assert fd == f.fileno()

A better fix would be something like:

     PythonFile pf = pythonContext.FileManager.GetFileFromId(pythonContext, fd);
+    pythonContext.FileManager.Weaken(fd);
     return pf;

That needs some new methods in the underlying mappings to turn the
strong reference into a weak one though.

There may be some code that does variations of the following:

IronPython 2.6.1 (2.6.10920.0) on .NET 2.0.50727.3082
Type "help", "copyright", "credits" or "license" for more information.
>>> import tempfile
>>> for _ in range(4096):
...   f = tempfile.NamedTemporaryFile()
...
Traceback (most recent call last):
  File "C:\Program Files\IronPython 2.6\Lib\tempfile.py", line 444, in
NamedTemporaryFile
  File "C:\Program Files\IronPython 2.6\Lib\tempfile.py", line 228, in
_mkstemp_inner
OSError: [Errno -2146233079] HybridMapping is full

Which might still run into problems depending on how the GC operates,
but isn't very likely.

(I'd have posted this straight on the tracker rather than to list, but
it doesn't seem to do wiki markup and having it eat spaces is rather
deadly for example Python code).

Martin


[1]: http://lists.ironpython.com/pipermail/users-ironpython.com/2010-February/012116.html
[2]: http://lists.ironpython.com/pipermail/users-ironpython.com/2009-October/011457.html
[3]: http://ironpython.codeplex.com/WorkItem/View.aspx?WorkItemId=23936
[4]: http://float.endofinternet.org/bazaar/simple/test_noncpython/test_noncpython.py