[Python-ideas] shutil zero-copy and exotic filesystems

[Giampaolo Rodola']

Whops, I hit "send" too soon. Sorry about the messed up message.

On Tue, May 29, 2018 at 10:56 AM, Giampaolo Rodola' <g.rodola at gmail.com>
wrote:

> Hello,
> I've been working on a patch which speeds up shutil.copy* operations for
> all 3 major platforms (Linux, Windows, OSX):
> https://bugs.python.org/issue33671
> Since the speedup is quite consistent I'd love to see this merged in, but
> considering shutil.copy* is quite crucial I wanted to hear other folk's
> opinion first. Attached patch attempts to use platform-specific zero-copy
> syscalls [1] by default and fallbacks on using plain read() / write()
> variant in case of immediate failure. In theory this should work fine, in
> practice I haven't tested it on exotic (e.g. network) filesystems. In order
> to diminish risks of breakage I think if it would make sense to:
> - add a global shutil.NO_ZEROCOPY variable defaulting to False
> - add a "no_zerocopy" argument to all functions involving a copy
> (copyfile(). copy(), copy2(), copytree(), move())
>
> Thoughts?
>
> [1] sendfile() (Linux), fcopyfile() (OSX), CopyFileW (Windows)
>
>
>
>
> since the matter is a bit sensitive in terms of potential breakage on
> exotic / untested (e.g. network) filesystems I want to raise some attention
> about:
> https://bugs.python.org/issue33671
> Attached patch attempts to use platform-specific zero-copy syscalls by
> default and fallbacks on using plain read() / write() copy in case of
> immediate failure.  In order to diminish risks I think it would make sense
> to:
> - add a global shutil.NO_ZEROCOPY variable defaulting to False
> - add a "no_zerocopy" argument to all functions involving a copy
> (copyfile(). copy(), copy2(), copytree(), move())
>
> Thoughts?
>
> --
> Giampaolo - http://grodola.blogspot.com
>
>
>


-- 
Giampaolo - http://grodola.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180529/8343f6f0/attachment.html>