[Python-ideas] Implementation of shutil.move

[Devin Jeanpierre]

This doesn't completely solve the race condition:

`os.open(path, O_EXCL | O_CREAT)` opens an fd for a file location. If
a file was already at that location, it fails, if not, it succeeds
with an fd.
So if os.move() used this fd to actually write the new data, there
would be no race condition in terms of file creation/deletion. This
is, as far as I can tell, what was suggested by your linked thread.

However, in your suggestion it does not do that: it does opens a new
fd to a new file, and then does precisely what it did before. During
the time in-between the os.open and the shutil.move(), somebody can
delete the created file, and write a new one, or whatever. If they do
that, then any such changes they make will be lost because shutil.move
will steamroller them and overwrite the file.

Devin

On Fri, Aug 12, 2011 at 7:59 AM, David Townshend <aquavitae69 at gmail.com> wrote:
> The shutil.move function uses os.rename to move files on the same file
> system. On unix, this function will overwrite an existing destination, so
> the obvious approach is
> if not os.path.exists(dst):
>     shutil.move(src, dst)
> But this could result in race conditions if dst is created after
> os.path.exists and before shutil.move.  From my research, it seems that this
> is a limitation in the unix c library, but it should be possible to avoid it
> through a workaround (pieced together
> from http://bytes.com/topic/python/answers/555794-safely-renaming-file-without-overwriting).
>  This involves some fairly low-level work, so I propose adding a new move2
> function to shutil, which raises an error if dst exists and locking it if it
> doesn't:
> def move2(src, dst):
>     try:
>         fd = os.open(dst, os.O_EXCL | os.O_CREAT)
>     except OSError:
>         raise Error('Destination exists')
>     try:
>         move(src, dst)
>     finally:
>         os.close(fd)
> This could be optimised by using shutil.move code rather than just calling
> it, but the idea is that an attempt is made to create dst with exclusive
> access. If this fails, then it means that the file exists, but if it passes,
> then dst is locked so no other process can create it.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
>