[Python-ideas] Implementation of shutil.move

[Devin Jeanpierre]

> open(), in my opinon, already behaves as it should.

open(..., 'w') doesn't, it overwrites the target. For copying an
individual file, you'd want os.open(..., O_EXCL | O_CREAT), which is a
cross-platform, race-condition-free way of creating and writing to a
single file provided it didn't exist before.


> From what I've read, there are several ways of ensuring that these functions
> fail if destination exists depending on the platform and filesystem, but
> there is no uniform way to do it. One approach would be to try all possible
> methods and hope that at least one works, with a simple "if os.exists(dst):
> fail" fallback.  The documentation would state that "An exception occurs if
> the destination exists.  This check is done is as safe a way possible to
> avoid race conditions where the system supports it."

There are two attitudes to take when using the don't-overwrite argument:

1. User cares about safety and race conditions, is glad that this
function tries to be safe.

2. User doesn't care about safety or race conditions, as User doesn't
have shell scripts creating files at inconvenient times and tempting
fate.

In case 1, having it silently revert to the unsafe version is very
bad, and potentially damaging. In case 2, the user doesn't care about
the safe version. In fact, User 2 is probably using os.path.exists
already.

If it can't do things safely, it shouldn't do them at all.

Devin

On Mon, Aug 15, 2011 at 6:07 AM, David Townshend <aquavitae69 at gmail.com> wrote:
> I''ve done some more reading and this is where I've got to:
> shutil.move() uses either os.rename() or shutil.copyfile() (via
> shutil.copytree()) depending on the situation. shutil.copyfile() uses
> open().  So to implement a safe move function it would seem to be necessary
> to do the same to copyfile(), and possibly open().
> open(), in my opinon, already behaves as it should.  It would be possible to
> add a slightly safer implementation by writing to a temporary file first,
> but this would not always be desired or even possible. Perhaps an
> alternative function could be added if the idea is popular enough?
> I would expect copyfile(), like move(), to fail if the destination exists.
>  This is not the current behaviour, so both functions could benefit from
> this.
> From what I've read, there are several ways of ensuring that these functions
> fail if destination exists depending on the platform and filesystem, but
> there is no uniform way to do it. One approach would be to try all possible
> methods and hope that at least one works, with a simple "if os.exists(dst):
> fail" fallback.  The documentation would state that "An exception occurs if
> the destination exists.  This check is done is as safe a way possible to
> avoid race conditions where the system supports it."  An additional measure
> of safety on copyfile() would be to write to a temporary file first, then
> use move. This would allow rollback in case of failure during the copy, but
> as with open(), its not always the most appropriate approach.
> Adding new copyfile() and move() functions would mean also mean adding new
> copy(), copy2() and copytree() functions, perhaps as copy3() and
> copytree2(). This seems to be getting rather messy - three slightly
> different copy functions, so it might still be better to add an optional
> argument to these.  Alternatively, a new module could be added dedicated to
> safe file operations.
> On Sun, Aug 14, 2011 at 5:23 PM, David Townshend <aquavitae69 at gmail.com>
> wrote:
>>
>> Sorry, yes. That is what I meant.
>>
>> On Aug 14, 2011 5:12 PM, "Devin Jeanpierre" <jeanpierreda at gmail.com>
>> wrote:
>> >> Why do you think that move on remote file systems use copy? From past
>> >> experience and recent tests I can confirm that shutil.move() uses
>> >> rename
>> >> on remote CIFS and NFS file systems
>> >
>> > I believe what he meant to say was "if you move from one filesystem to
>> > another".
>> >
>> > shutil.move tries to do a copy and delete if rename fails with an
>> > OSError.
>> >
>> > Devin
>> >
>> > On Sun, Aug 14, 2011 at 10:39 AM, Christian Heimes <lists at cheimes.de>
>> > wrote:
>> >> Am 14.08.2011 16:09, schrieb David Townshend:
>> >>> It seems there's a second problem too - move on remote file systems
>> >>> use copy
>> >>> rather than rename, so changing the implementation means changing it
>> >>> for
>> >>> copy too, which is more difficult. Maybe the best option is to try to
>> >>> apply
>> >>> some sort of locking mechanism, but I can't see how right now.
>> >>
>> >> Why do you think that move on remote file systems use copy? From past
>> >> experience and recent tests I can confirm that shutil.move() uses
>> >> rename
>> >> on remote CIFS and NFS file systems.
>> >>
>> >>
>> >> _______________________________________________
>> >> Python-ideas mailing list
>> >> Python-ideas at python.org
>> >> http://mail.python.org/mailman/listinfo/python-ideas
>> >>
>> > _______________________________________________
>> > Python-ideas mailing list
>> > Python-ideas at python.org
>> > http://mail.python.org/mailman/listinfo/python-ideas
>
>