Notice: While Javascript is not essential for this website, your interaction with the content will be limited. Please turn Javascript on for the full experience.

PEP 597 -- Add PYTHONTEXTENCODING environment variable

PEP:597
Title:Add PYTHONTEXTENCODING environment variable
Author:Inada Naoki <songofacandy at gmail.com>
Status:Draft
Type:Standards Track
Created:05-Jun-2019
Python-Version:3.9

Abstract

Currently, TextIOWrapper uses locale.getpreferredencoding(False) (hereinafter called "locale encoding") when encoding is not specified.

This PEP proposes adding PYTHONTEXTENCODING environment variable to override the default text encoding since Python 3.9.

The goal of this PEP is providing "UTF-8 by default" experience to Windows users, because macOS, Linux, Android, iOS users use UTF-8 by default already.

Motivation

UTF-8 is the best encoding for saving unicode text

String in Python 3 is unicode. Encoding valid unicode strings with UTF-8 should not fail.

On the other hand, most locale encoding used in Windows can not save all valid unicode string. It will cause UnicodeEncodeError or it may not round-trip. User may lost their data in such case.

UTF-8 is the best encoding for saving text when user don't specify any encoding.

Using UTF-8 by default is easier on new programmers

Python is one of the most popular first programming languages.

New programmers may not know about encoding. When they download text data written in UTF-8 from the Internet, they are forced to learn about encoding.

Popular text editors like VS Code or Atom use UTF-8 by default. Even Microsoft Notepad uses UTF-8 by default since the Windows 10 May 2019 Update. (Note that Python 3.9 will be released in 2021.)

Additionally, the default encoding of Python source files is UTF-8. We can assume new Python programmers who don't know about encoding use editors which use UTF-8 by default.

It would be nice if new programmers are not forced to learn about encoding until they need to handle text files encoded in encoding other than UTF-8.

People assume it is always UTF-8

Package authors using macOS or Linux may forget that the default encoding is not always UTF-8.

For example, long_description = open("README.md").read() in setup.py is a common mistake. If there is at least one emoji or any other non-ASCII character in the README.md file, many Windows users cannot install the package due to a UnicodeDecodeError.

Consistent with default encoding

Python has sys.defaultencoding() which is always "UTF-8". str.encode() uses "UTF-8" when encoding is omitted.

Using "UTF-8" for text files are consistent with it. It makes Python more easy to learn language.

Specification

PYTHONTEXTENCODING environment variable

PYTHONTEXTENCODING environment variable can be used to specify the default text encoding.

Unlike PYTHONIOENCODING, it doesn't accept error handler. PYTHONIOENCODING support it because changing error handler of stdio was difficult. But it is not true for regular files.

sys.gettextencoding()

When PYTHONTEXTENCODING is specified, this function return it.

When it is not specified, this function returns locale.getpreferredencoding(False).

encoding="locale" option

TextIOWrapper now accepts encoding="locale" option.

"locale" is not real encoding or alias. This is just a shortcut of encoding=locale.getpreferredencoding(False).

Changes in stdlibs

TextIOWrapper uses sys.gettextencoding() where locale.getpreferredencoding(False) is used.

But stdin, stdout, and stderr continue to respect locale encoding as well. PYTHONIOENCODING can be used to override thier encoding.

Pipes and TTY should use the "locale" encoding. UTF-8 mode [1] can be used to override these encoding:

  • subprocess and os.popen use the "locale" encoding because the subprocess will use the locale encoding.
  • getpass.getpass uses the "locale" encoding when using TTY.

All other code using the default encoding are not modified. They can be overridden by PYTHONTEXTENCODING. This is an incomplete list:

  • lzma.open, gzip.open, bz2.open, ZipFile.read_text
  • socket.makefile
  • tempfile.TemporaryFile, tempfile.NamedTemporaryFile
  • trace.CoverageResults.write_results_file

Rationale

Why not just enable UTF-8 mode by default?

This PEP is not mutually exclusive to UTF-8 mode.

If we enable UTF-8 mode by default, even people using Windows will forget the default encoding is not always UTF-8. More scripts will be written assuming the default encoding is UTF-8.

So changing the default encoding of text files to UTF-8 would be better even if UTF-8 mode is enabled by default at some point.

Why is "locale" not an alias codec?

For backward compatibility, io.TextIOWrapper calls locale.getpreferredencoding(False) every time when encoding="locale" is specified.

It respects changing locale after Python startup.

Why not change std(in|out|err) encoding too?

Even when the locale encoding is not UTF-8, there can be many UTF-8 text files. These files could be downloaded from the Internet or written by modern text editors.

On the other hand, terminal encoding is assumed to be the same as locale encoding. And other tools are assumed to read and write the locale encoding as well.

std(in|out|err) are likely to be connected to a terminal or other tools. So the locale encoding should be respected.

Anyway, PYTHONIOENCODING can be used to change these encodings.

Reference Implementation

To be written.

Rejected Ideas

Change the default text encoding

Previous version of this PEP tried to change the default encoding to UTF-8.

But we should have deprecation period long enough. Between the deprecation period, users can not change the default text encoding.

And there are many difficulity there:

  • Omitting encoding option is very common.
    • If we raise DeprecationWarning always, it will be too noisy.
    • We can not assume how user use it. Complicated heuritics may be needed to raise DeprecationWarning only when it is really needed.
  • Users of legacy systems may dismiss warning.
    • They may not check the warning.
    • They may upgrade Python from 2.7 after 2020.

Additionally, Microsoft is improving UTF-8 support of Windows 10 recently.

There are no public plan for future UTF-8 support yet. But Python may be able to change the default encoding without painful deprecation period in the future.

Open Issues

Easy way to set PYTHONTEXTENCODING

UTF-8 is the best encoding for new users. But setting environment variables is not easy enough to new users.

It would be helpfule if Python on Windows can provide easy way to set PYTHONTEXTENCODING=UTF-8 even after Python is installed.

Commandline option

If there is reasonable use case for changing default text encoding per process, command line option should be considered.

C-API

The default text encoding should be able to configured from C. This will be considered when writing reference Implementation.

Additionally, C-API like PySys_GetTextEncoding() should be considered too.

Source: https://github.com/python/peps/blob/master/pep-0597.rst