[issue24672] shutil.rmtree fails on non ascii filenames

[Jason R. Coombs]

Jason R. Coombs added the comment:

I'm afraid I need to re-open this issue.

Although passing unicode names to rmtree fixes the issue on Windows systems, it causes problems on Linux systems where LC_ALL=C. Consider this script:

#################################
# encoding: utf-8

from __future__ import unicode_literals

import os
import shutil

os.mkdir('temp')

with open('temp/Слава Україні.html'.encode('utf-8'), 'w'):
    pass

print(os.listdir(b'temp')[0])

shutil.rmtree('temp')
#################################

Invoked thus, a UnicodeDecodeError occurs:

vagrant at trusty:/vagrant$ LC_ALL=C python2.7 issue24672.py 
Слава Україні.html
Traceback (most recent call last):
  File "issue24672.py", line 15, in <module>
    shutil.rmtree('temp')
  File "/usr/lib/python2.7/shutil.py", line 241, in rmtree
    fullname = os.path.join(path, name)
  File "/usr/lib/python2.7/posixpath.py", line 80, in join
    path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1: ordinal not in range(128)


This is the same error seen trying to rmtree an extraction of Sphinx (a package containing an offending non-ascii character)::

vagrant at trusty:/vagrant$ wget 'https://files.pythonhosted.org/packages/b2/d5/bb4bf7fbc2e6b85d1e3832716546ecd434632d9d434a01efe87053fe5f25/Sphinx-1.5.1.tar.gz' -O - | tar xz 
--2016-12-20 19:07:21--  https://files.pythonhosted.org/packages/b2/d5/bb4bf7fbc2e6b85d1e3832716546ecd434632d9d434a01efe87053fe5f25/Sphinx-1.5.1.tar.gz
Resolving files.pythonhosted.org (files.pythonhosted.org)... 151.101.33.63
Connecting to files.pythonhosted.org (files.pythonhosted.org)|151.101.33.63|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4397246 (4.2M) [binary/octet-stream]
Saving to: ‘STDOUT’

100%[========================================================>] 4,397,246   2.06MB/s   in 2.0s   

2016-12-20 19:07:23 (2.06 MB/s) - written to stdout [4397246/4397246]

vagrant at trusty:/vagrant$ LC_ALL=C python2.7 -c "import shutil; shutil.rmtree(u'Sphinx-1.5.1')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib/python2.7/shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib/python2.7/shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib/python2.7/shutil.py", line 241, in rmtree
    fullname = os.path.join(path, name)
  File "/usr/lib/python2.7/posixpath.py", line 80, in join
    path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xcc in position 8: ordinal not in range(128)


Is the solution to call rmtree with unicode on Windows, but with bytes when on Python 2 and Linux? What else can be done?

----------
resolution: wont fix -> 
status: closed -> open

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue24672>
_______________________________________