[Python-ideas] Implementation of shutil.move

[David Townshend]

My understanding of os.O_EXCL is that it locks the file from changes by any
other process.  It appears from a quick test, though, that this is not the
case. Perhaps the second suggestion in the linked thread (using link/unlink)
would work better, since this situation only arises on unix.

I like the idea of a context manager for locking, but I'm not sure how that
would work in this case...

On Fri, Aug 12, 2011 at 3:53 PM, Devin Jeanpierre <jeanpierreda at gmail.com>wrote:

> This doesn't completely solve the race condition:
>
> `os.open(path, O_EXCL | O_CREAT)` opens an fd for a file location. If
> a file was already at that location, it fails, if not, it succeeds
> with an fd.
> So if os.move() used this fd to actually write the new data, there
> would be no race condition in terms of file creation/deletion. This
> is, as far as I can tell, what was suggested by your linked thread.
>
> However, in your suggestion it does not do that: it does opens a new
> fd to a new file, and then does precisely what it did before. During
> the time in-between the os.open and the shutil.move(), somebody can
> delete the created file, and write a new one, or whatever. If they do
> that, then any such changes they make will be lost because shutil.move
> will steamroller them and overwrite the file.
>
> Devin
>
> On Fri, Aug 12, 2011 at 7:59 AM, David Townshend <aquavitae69 at gmail.com>
> wrote:
> > The shutil.move function uses os.rename to move files on the same file
> > system. On unix, this function will overwrite an existing destination, so
> > the obvious approach is
> > if not os.path.exists(dst):
> >     shutil.move(src, dst)
> > But this could result in race conditions if dst is created after
> > os.path.exists and before shutil.move.  From my research, it seems that
> this
> > is a limitation in the unix c library, but it should be possible to avoid
> it
> > through a workaround (pieced together
> > from
> http://bytes.com/topic/python/answers/555794-safely-renaming-file-without-overwriting
> ).
> >  This involves some fairly low-level work, so I propose adding a new
> move2
> > function to shutil, which raises an error if dst exists and locking it if
> it
> > doesn't:
> > def move2(src, dst):
> >     try:
> >         fd = os.open(dst, os.O_EXCL | os.O_CREAT)
> >     except OSError:
> >         raise Error('Destination exists')
> >     try:
> >         move(src, dst)
> >     finally:
> >         os.close(fd)
> > This could be optimised by using shutil.move code rather than just
> calling
> > it, but the idea is that an attempt is made to create dst with exclusive
> > access. If this fails, then it means that the file exists, but if it
> passes,
> > then dst is locked so no other process can create it.
> > _______________________________________________
> > Python-ideas mailing list
> > Python-ideas at python.org
> > http://mail.python.org/mailman/listinfo/python-ideas
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110812/9efda842/attachment.html>