[Python-Dev] Issue #8610: Set default file system encoding to ascii on error?

[M.-A. Lemburg]

Victor Stinner wrote:
> Python 3.0 introduced PyUnicode_DecodeFSDefault() and 
> PyUnicode_DecodeFSDefaultAndSize() functions. These functions fallback to 
> UTF-8 if getting the file system encoding failed or the encoding is unknown 
> (there is not nl_langinfo(CODESET) function).
> 
> UTF-8 is not a good choice for the fallback because it's incompatible with 
> other encodings like Latin1. I would like to fallback to ASCII on error which 
> is compatible with all encodings (thanks to surrogateescape).
> 
> I would like to ensure that sys.getfilesystemencoding() result cannot be None, 
> because Python 3.2 uses it on Unix to decode os.environb and to encode 
> filenames in subprocess. I can implement a fallback for os.environb and 
> subprocess (and other functions calling sys.getfilesystemencoding()), but I 
> prefer to have a reliable sys.getfilesystemencoding() function.
> 
> This change doesn't concern Windows and Mac OS X because the encoding is 
> hardcoded (mbcs, utf-8). On Unix, I don't know in which case nl_langinfo() can 
> fail. Empty environment is not an error: nl_langinfo(CODESET) returns "ascii". 
> I think that few (or no) user would notice this change.

+1 on that change.

The UTF-8 fallback has these major problems:

 * it hides errors by always having the Unicode-bytes conversion
   succeed

 * it can cause applications to write files and create
   directories with wrongly encoded names (e.g. use UTF-8 on
   a Latin-1 file system)

Together with [issue8622] Add PYTHONFSENCODING environment variable:

    http://bugs.python.org/issue8622

which reduces the Python3 reliance on encoding guess work,
the change would make Python3 more user friendly and reduce the
number of bummers user run into when waking up in the all-new
Unicode world of Python3.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 07 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2010-04-23: Released mxODBC.Zope.DA 2.0.1        http://zope.egenix.com/

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/