Behavior of tempfile temp files when scripts killed, interpreter crashes, server crashes?

[eryk sun]

On Wed, Jul 27, 2016 at 1:58 PM, Malcolm Greene <python at bdurham.com> wrote:
> Can someone share their OS specific experience in working with tempfile
> generated temp files under these conditions?
>
> 1. Script killed by another process
> 2. Interpreter crashes
> 3. Server crashes (sudden loss of power)
> 4. Other application termination conditions ???
>
> I'm curious which scenarios result in temp files not being automatically
> deleted after use and what technique you're using to cleanup temp files
> left over after these conditions (without affecting legitimate temp
> files present from the current session)?
>
> Do any OS's support a type of temp file that truly gets automatically
> deleted in all of the above scenarios?

Linux has the O_TMPFILE open() flag [1]. This creates an anonymous
file that gets automatically deleted when the last open file
descriptor is closed. If the file isn't opened O_EXCL, then you can
make it permanent by linking it back into the filesystem. For example:

    >>> fd = os.open('/tmp', os.O_TMPFILE | os.O_RDWR, 0o600)
    >>> tmp_dir_fd = os.open('/tmp', os.O_DIRECTORY)
    >>> os.write(fd, b'spam')
    4
    >>> os.link('/proc/self/fd/%d' % fd, 'tempfile', dst_dir_fd=tmp_dir_fd)
    >>> os.close(fd); os.close(tmp_dir_fd)
    >>> open('/tmp/tempfile').read()
    'spam'

tempfile.TemporaryFile uses the O_TMPFILE flag when it's available.
Since it also uses O_EXCL, the files it creates cannot be made
permanent via os.link.

[1]: http://man7.org/linux/man-pages/man2/open.2.html

Windows CreateFile [2] has two related flags:
FILE_FLAG_DELETE_ON_CLOSE to ensure that a temporary file gets deleted
and FILE_ATTRIBUTE_TEMPORARY, which avoids writing the file to disk as
long as there's enough available cache memory. The Windows C runtime
and Python make these available as O_TEMPORARY and O_SHORT_LIVED,
respectively [3].

tempfile.NamedTemporaryFile uses O_TEMPORARY. tempfile.TemporaryFile
is an alias for NamedTemporary, since files can't be anonymous on
Windows.

You can delete the temporary file, because it's opened with delete
sharing, but it won't actually get unlinked from the directory until
the last handle is closed. The only benefit to deleting the file is
getting exclusive access, but you can already control that in Windows
via the dwShareMode parameter of CreateFile or the shflag parameter of
_wsopen_s. Unfortunately Python doesn't provide a high-level API for
setting the sharing mode, so if you need to set it you're stuck with
using _winapi.CreateFile and msvcrt.open_osfhandle, PyWin32, or
ctypes.

It's possible to work around the delete-on-close flag by opening the
file a 2nd time, closing the original handle, and then calling
SetFileInformationByHandle [4] to unset the file's delete disposition.

[2]: https://msdn.microsoft.com/en-us/library/aa363858
[3]: https://msdn.microsoft.com/en-us/library/w64k0ytk
[4]: https://msdn.microsoft.com/en-us/library/aa365539