[Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

[M.-A. Lemburg]

Victor Stinner wrote:
> Le mardi 28 juin 2011 à 16:02 +0200, M.-A. Lemburg a écrit :
>> How about a more radical change: have open() in Py3 default to
>> opening the file in binary mode, if no encoding is given (even
>> if the mode doesn't include 'b') ?
> 
> I tried your suggested change: Python doesn't start.

No surprise there: it's an incompatible change, but one that undoes
a wart introduced in the Py3 transition. Guessing encodings should
be avoided whenever possible.

> sysconfig uses the implicit locale encoding to read sysconfig.cfg, the
> Makefile and pyconfig.h. I think that it is correct to use the locale
> encoding for Makefile and pyconfig.h, but maybe not for sysconfig.cfg.
> 
> Python require more changes just to run "make". I was able to run "make"
> by using encoding='utf-8' in various functions (of distutils and
> setup.py). I didn't try the test suite, I expect too many failures.

This demonstrates that Python's stdlib is still not being explicit
about the encoding issues. I suppose that things just happen to work
because we mostly use ASCII files for configuration and setup.

> --
> 
> Then I tried my suggestion (use "utf-8" by default): Python starts
> correctly, I can build it (run "make") and... the full test suite pass
> without any change. (I'm testing on Linux, my locale encoding is UTF-8.)

I bet it would also with "ascii" in most cases. Which then just
means that the Python build process and test suite is not a good
test case for choosing a default encoding.

Linux is also a poor test candidate for this, since most user setups
will use UTF-8 as locale encoding. Windows, OTOH, uses all sorts of
code page encodings (usually not UTF-8), so you are likely to hit
the real problem cases a lot easier.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jun 29 2011)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/