Remove directory tree without following symlinks

[eryk sun]

On Sun, Apr 24, 2016 at 5:42 AM, Albert-Jan Roskam
<sjeik_appie at hotmail.com> wrote:
> Aww, I kinda forgot about that already, but I came across this last
> year [1]. Apparently, shutil.rmtree(very_long_path) failed under Win 7,
> even with the "silly prefix". I believe very_long_path was a
> Python2-str.
> [1]
> https://mail.python.org/pipermail/python-list/2015-June/693156.html

Python 2's str branch of the os functions gets implemented on Windows
using the [A]NSI API, such as FindFirstFileA and FindNextFileA to
implement listdir(). Generally the ANSI API is a light wrapper around
the [W]ide-character API. It simply decodes byte strings to UTF-16 and
calls the wide-character function (or a common internal function).

IIRC, in Windows 7, byte strings are decoded using a per-thread buffer
with size MAX_PATH (260), so prefixing the path with "\\?\" won't
help. You have to use the wide-character API. Windows 10, on the other
hand, decodes using a dynamically allocated buffer, so you can usually
get away with using a long byte string. But not with Python 2
os.listdir(), which uses a stack-allocated MAX_PATH+5 buffer in the
str branch. For example:

Python 2 os.mkdir works:

    >>> path = os.path.normpath('//?/C:/Temp/long/' + 'a' * 255)
    >>> os.makedirs(path)

but os.listdir requires unicode:

    >>> os.listdir(path)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: must be (buffer overflow), not str
    >>> os.listdir(path.decode('mbcs'))
    []

Also, the str branch of listdir appends "/*.*", with a forward slash,
so it's incompatible with the "\\?\" prefix, even for short paths:

    >>> os.listdir(r'\\?\C:\Temp')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    WindowsError: [Error 123] The filename, directory name, or volume
    label syntax is incorrect: '\\\\?\\C:\\Temp/*.*'

> It seems useful if shutil or os.path would automatically prefix paths
> with "\\?\". It is rarely really needed, though. (in my case it was
> needed to copy a bunch of MS Outlook .msg files, which automatically
> get the subject line as the filename, and perhaps the first sentence
> of the mail of the mail has no subject).

I doubt a change like that would get backported to 2.7. Recently there
was a lengthy discussion about adding an __fspath__ protocol to Python
3. Possibly this can be automatically handled in the __fspath__
implementation of pathlib.WindowsPath and the DirEntry type returned
by os.scandir.