Notice: While Javascript is not essential for this website, your interaction with the content will be limited. Please turn Javascript on for the full experience.

PEP 540 -- Add a new UTF-8 mode

PEP: 540
Title: Add a new UTF-8 mode
Author: Victor Stinner <victor.stinner at gmail.com>
BDFL-Delegate: INADA Naoki
Status: Draft
Type: Standards Track
Created: 5-January-2016
Python-Version: 3.7

Abstract

Add a new UTF-8 mode, disabled by default, to ignore the locale and force the usage of the UTF-8 encoding.

Basically, UTF-8 mode behaves as Python 2: it "just works" and doesn't bother users with encodings, but it can produce mojibake. The UTF-8 mode can be configured as strict to prevent mojibake.

A new -X utf8 command line option and a PYTHONUTF8 environment variable are added to control the UTF-8 mode. The POSIX locale enables the UTF-8 mode.

Rationale

"It's not a bug, you must fix your locale" is not an acceptable answer

Since Python 3.0 was released in 2008, the usual answer to users getting Unicode errors is to ask developers to fix their code to handle Unicode properly. Most applications and Python modules were fixed, but users kept reporting Unicode errors regularly: see the long list of issues in the Links section below.

In fact, a second class of bugs comes from a locale which is not properly configured. The usual answer to such a bug report is: "it is not a bug, you must fix your locale".

Technically, the answer is correct, but from a practical point of view, the answer is not acceptable. In many cases, "fixing the issue" is a hard task. Moreover, sometimes, the usage of the POSIX locale is deliberate.

A good example of a concrete issue are build systems which create a fresh environment for each build using a chroot, a container, a virtual machine or something else to get reproducible builds. Such a setup usually uses the POSIX locale. To get 100% reproducible builds, the POSIX locale is a good choice: see the Locales section of reproducible-builds.org .

UNIX users don't expect Unicode errors, since the common command lines tools like cat , grep or sed never fail with Unicode errors. These users expect that Python 3 "just works" with any locale and won't bother them with encodings. From their point of the view, the bug is not their locale, it's obviously Python 3.

Since Python 2 handles data as bytes, it's rarer in Python 2 compared to Python 3 to get Unicode errors. It also explains why users also perceive Python 3 as the root cause of their Unicode errors.

Some users expect that Python 3 just works with any locale and so don't bother with mojibake, whereas some developers are working hard to prevent mojibake and so expect that Python 3 fails early before creating it.

Since different group of users have different expectations, there is no silver bullet which solves all issues at once. Last but not least, backward compatibility should be preserved whenever possible.

Locale and operating system data

Python uses an encoding called the "filesystem encoding" to decide how to encode and decode data from/to the operating system:

  • file content
  • command line arguments: sys.argv
  • standard streams: sys.stdin , sys.stdout , sys.stderr
  • environment variables: os.environ
  • filenames: os.listdir(str) for example
  • pipes: subprocess.Popen using subprocess.PIPE for example
  • error messages: os.strerror(code) for example
  • user and terminal names: os , grp and pwd modules
  • host name, UNIX socket path: see the socket module
  • etc.

At startup, Python calls setlocale(LC_CTYPE, "") to use the user LC_CTYPE locale and then store the locale encoding as the "filesystem error". It's possible to get this encoding using sys.getfilesystemencoding() . In the whole lifetime of a Python process, the same encoding and error handler are used to encode and decode data from/to the operating system.

The os.fsdecode() and os.fsencode() functions can be used to decode and encode operating system data. These functions use the filesystem error handler: sys.getfilesystemencodeerrors() .

Note

In some corner cases, the current LC_CTYPE locale must be used instead of sys.getfilesystemencoding() . For example, the time module uses the current LC_CTYPE locale to decode timezone names.

The POSIX locale and its encoding

The following environment variables are used to configure the locale, in this preference order:

  • LC_ALL , most important variable
  • LC_CTYPE
  • LANG

The POSIX locale, also known as "the C locale", is used:

  • if the first set variable is set to "C"
  • if all these variables are unset, for example when a program is started in an empty environment.

The encoding of the POSIX locale must be ASCII or a superset of ASCII.

On Linux, the POSIX locale uses the ASCII encoding.

On FreeBSD and Solaris, nl_langinfo(CODESET) announces an alias of the ASCII encoding, whereas mbstowcs() and wcstombs() functions use the ISO 8859-1 encoding (Latin1) in practice. The problem is that os.fsencode() and os.fsdecode() use Python codec of the locale encoding. For example, if command line arguments are decoded by mbstowcs() and encoded back by os.fsencode() , an UnicodeEncodeError exception is raised instead of retrieving the original byte string.

To fix this issue, from Python 3.4, a check is made to see if mbstowcs() really uses the ASCII encoding if the the LC_CTYPE uses the the POSIX locale and nl_langinfo(CODESET) returns "ASCII" (or an alias to ASCII). If not (the effective encoding is not ASCII), Python uses its own ASCII codec instead of using mbstowcs() and wcstombs() functions for operating system data .

See the POSIX locale (2016 Edition) .

POSIX locale used by mistake

In many cases, the POSIX locale is not really expected by users who get it by mistake. Examples:

  • program started in an empty environment
  • User forcing LANG=C to get messages in English
  • LANG=C used for bad reasons, without being aware of the ASCII encoding
  • SSH shell
  • Linux installed with no configured locale
  • chroot environment, Docker image, container, ... with no locale is configured
  • User locale set to a non-existing locale, typo in the locale name for example

C.UTF-8 and C.utf8 locales

Some UNIX operating systems provide a variant of the POSIX locale using the UTF-8 encoding:

  • Fedora 25: "C.utf8" or "C.UTF-8"
  • Debian (eglibc 2.13-1, 2011), Ubuntu: "C.UTF-8"
  • HP-UX: "C.utf8"

It was proposed to add a C.UTF-8 locale to glibc: glibc C.UTF-8 proposal .

It is not planned to add such locale to BSD systems.

Popularity of the UTF-8 encoding

Python 3 uses UTF-8 by default for Python source files.

On Mac OS X, Windows and Android, Python always use UTF-8 for operating system data. For Windows, see PEP 529 [2] : "Change Windows filesystem encoding to UTF-8".

On Linux, UTF-8 became the de facto standard encoding, replacing legacy encodings like ISO 8859-1 or ShiftJIS. For example, using different encodings for filenames and standard streams is likely to create mojibake, so UTF-8 is now used everywhere .

The UTF-8 encoding is the default encoding of XML and JSON file formats. As of January 2017, UTF-8 was used in more than 88% of web pages (HTML, Javascript, CSS, etc.).

See utf8everywhere.org for more general information on the UTF-8 codec.

Note

Some applications and operating systems (especially Windows) use Byte Order Markers (BOM) to indicate the used Unicode encoding: UTF-7, UTF-8, UTF-16-LE, etc. BOM are not well supported and are rarely used in Python.

Old data stored in different encodings and surrogateescape

Even if UTF-8 became the de facto standard, there are still systems in the wild which don't use UTF-8. And there are a lot of data stored in different encodings. For example, an old USB key using the ext3 filesystem with filenames encoded to ISO 8859-1.

The Linux kernel and libc don't decode filenames: a filename is used as a raw array of bytes. The common solution to support any filename is to store filenames as bytes and don't try to decode them. When displayed to stdout, mojibake is displayed if the filename and the terminal don't use the same encoding.

Python 3 promotes Unicode everywhere including filenames. A solution to support filenames not decodable from the locale encoding was found: the surrogateescape error handler ( PEP 383 [4] ), store undecodable bytes as surrogate characters. This error handler is used by default for operating system data , for example, by os.fsdecode() and os.fsencode() (except on Windows which uses the strict error handler).

Standard streams

Python uses the locale encoding for standard streams: stdin, stdout and stderr. The strict error handler is used by stdin and stdout to prevent mojibake.

The backslashreplace error handler is used by stderr to avoid Unicode encode errors when displaying non-ASCII text. It is especially useful when the POSIX locale is used, because this locale usually uses the ASCII encoding.

The problem is that operating system data like filenames are decoded using the surrogateescape error handler ( PEP 383 [4] ). Displaying a filename to stdout raises a Unicode encode error if the filename contains an undecoded byte stored as a surrogate character.

Python 3.6 now uses surrogateescape for stdin and stdout if the POSIX locale is used: issue #19977 . The idea is to pass through operating system data even if it creates mojibake, because most UNIX applications work like that. Most UNIX applications store filenames as bytes, usually because bytes are first-citizen class in the used programming language, whereas Unicode is badly supported.

Note

The encoding and/or the error handler of standard streams can be overridden with the PYTHONIOENCODING environment variable.

Proposal

Changes

Add a new UTF-8 mode, disabled by default, to ignore the locale and force the usage of the UTF-8 encoding with the surrogateescape error handler, instead using the locale encoding (with strict or surrogateescape error handler depending on the case).

Basically, the UTF-8 mode behaves as Python 2: it "just works" and doesn't bother users with encodings, but it can produce mojibake. It can be configured as strict to prevent mojibake: the UTF-8 encoding is used with the strict error handler for inputs and outputs, but the surrogateescape error handler is still used for operating system data .

A new -X utf8 command line option and a PYTHONUTF8 environment variable are added to control the UTF-8 mode. The UTF-8 mode is enabled by using -X utf8 or PYTHONUTF8=1 . It can be configured as strict by using -X utf8=strict or PYTHONUTF8=strict . Other option values fail with an error.

The POSIX locale enables the UTF-8 mode. In this case, the UTF-8 mode can be explicitly disabled by -X utf8=0 or PYTHONUTF8=0 .

Options priority for the UTF-8 mode:

  • PYTHONLEGACYWINDOWSFSENCODING
  • -X utf8
  • PYTHONUTF8
  • POSIX locale

For example, PYTHONUTF8=0 python3 -X utf8 enables the UTF-8 mode, whereas LC_ALL=C python3.7 -X utf8=0 disables the UTF-8 mode and uses the encoding of the POSIX locale.

Encodings used by open() , highest priority first:

  • encoding and errors parameters (if set)
  • UTF-8 mode
  • os.device_encoding(fd)
  • os.getpreferredencoding(False)

Encoding and error handler

The UTF-8 mode changes the default encoding and error handler used by open() , os.fsdecode() , os.fsencode() , sys.stdin , sys.stdout and sys.stderr :

Function Default UTF-8 mode or POSIX locale UTF-8 Strict mode
open() locale/strict UTF-8/surrogateescape UTF-8 /strict
os.fsdecode(), os.fsencode() locale/surrogateescape UTF-8 /surrogateescape UTF-8 /surrogateescape
sys.stdin, sys.stdout locale/strict UTF-8/surrogateescape UTF-8 /strict
sys.stderr locale/backslashreplace UTF-8 /backslashreplace UTF-8 /backslashreplace

By comparison, Python 3.6 uses:

Function Default POSIX locale
open() locale/strict locale/strict
os.fsdecode(), os.fsencode() locale/surrogateescape locale/surrogateescape
sys.stdin, sys.stdout locale/strict locale/ surrogateescape
sys.stderr locale/backslashreplace locale/backslashreplace

The UTF-8 mode uses the surrogateescape error handler instead of the strict mode for convenience: the idea is that data not encoded to UTF-8 are passed through "Python" without being modified, as raw bytes.

The PYTHONIOENCODING environment variable has priority over the UTF-8 mode for standard streams. For example, PYTHONIOENCODING=latin1 python3 -X utf8 uses the Latin1 encoding for stdin, stdout and stderr.

Encoding and error handler on Windows

On Windows, the encodings and error handlers are different:

Function Default Legacy Windows FS encoding UTF-8 mode UTF-8 Strict mode
open() mbcs/strict mbcs/strict UTF-8/surrogateescape UTF-8 /strict
os.fsdecode(), os.fsencode() UTF-8/surrogatepass mbcs/replace UTF-8/surrogatepass UTF-8/surrogatepass
sys.stdin, sys.stdout UTF-8/surrogateescape UTF-8/surrogateescape UTF-8/surrogateescape UTF-8/strict
sys.stderr UTF-8/backslashreplace UTF-8/backslashreplace UTF-8/backslashreplace UTF-8/backslashreplace

By comparison, Python 3.6 uses:

Function Default Legacy Windows FS encoding
open() mbcs/strict mbcs/strict
os.fsdecode(), os.fsencode() UTF-8/surrogatepass mbcs/replace
sys.stdin, sys.stdout UTF-8/surrogateescape UTF-8/surrogateescape
sys.stderr UTF-8/backslashreplace UTF-8/backslashreplace

The "Legacy Windows FS encoding" is enabled by setting the PYTHONLEGACYWINDOWSFSENCODING environment variable to 1 as specified in PEP 529 .

Enabling the legacy Windows filesystem encoding disables the UTF-8 mode (as -X utf8=0 ).

If stdin and/or stdout is redirected to a pipe, sys.stdin and/or sys.output use mbcs encoding by default rather than UTF-8. But with the UTF-8 mode, sys.stdin and sys.stdout always use the UTF-8 encoding.

There is no POSIX locale on Windows. The ANSI code page is used to the locale encoding, and this code page never uses the ASCII encoding.

Rationale

UTF-8 mode is disabled by default in order to keep hard Unicode errors when encoding or decoding operating system data fails and preserve backward compatibility. In addition, users will be better prepared for mojibake if it is their responsibility to explicitly enable UTF-8 mode than they would be if it was enabled by default .

UTF-8 mode should be used on systems known to be configured with UTF-8 where most applications speak UTF-8. It prevents Unicode errors if the user overrides a locale by mistake or if a Python program is started with no locale configured (and so with the POSIX locale).

Most UNIX applications handle operating system data as bytes, so the LC_ALL , LC_CTYPE and LANG environment variables have a limited impact on how these data are handled by the application.

The UTF-8 mode should help make Python more interoperable with other UNIX applications on the system assuming that UTF-8 is used everywhere and that users expect UTF-8.

Ignoring LC_ALL , LC_CTYPE and LANG environment variables in Python is more convenient, since they are more commonly misconfigured by mistake (configured to use an encoding different than UTF-8, whereas the system uses UTF-8), rather than being misconfigured by intent.

Expected mojibake and surrogate character issues

The UTF-8 mode only affects code running directly in Python, especially code written in pure Python. The other code, called "external code" here, is not aware of this mode. Examples:

  • C libraries called by Python modules like OpenSSL
  • The application code when Python is embedded in an application

In the UTF-8 mode, Python uses the surrogateescape error handler which stores bytes not decodable from UTF-8 as surrogate characters.

If the external code uses the locale and the locale encoding is UTF-8, it should work fine.

External code using bytes

If the external code processes data as bytes, surrogate characters are not an issue since they are only used inside Python. Python encodes back surrogate characters to bytes at the edges, before calling external code.

The UTF-8 mode can produce mojibake since Python and external code don't both of invalid bytes, but it's a deliberate choice. The UTF-8 mode can be configured as strict to prevent mojibake and fail early when data is not decodable from UTF-8 or not encodable to UTF-8.

External code using text

If the external code uses text API, for example using the wchar_t* C type, mojibake should not occur, but the external code can fail on surrogate characters.

Use Cases

The following use cases were written to highlight the impact of the chosen encodings and error handlers on concrete examples.

The "Always work" results were written to prove the benefit of having a UTF-8 mode which works with any data and any locale, compared to the existing old Python versions.

The "Mojibake" column shows that ignoring the locale causes a practical issue: the UTF-8 mode produces mojibake if the terminal doesn't use the UTF-8 encoding.

List a directory into stdout

Script listing the content of the current directory into stdout:

import os
for name in os.listdir(os.curdir):
    print(name)

Result:

Python Always works? Mojibake?
Python 2 Yes Yes
Python 3 No No
Python 3.5, POSIX locale Yes Yes
UTF-8 mode Yes Yes
UTF-8 Strict mode No No

"No" means that the script can fail on decoding or encoding a filename depending on the locale or the filename.

To be able to always work, the program must be able to produce mojibake. Mojibake is more user friendly than an error with a truncated or empty output.

For example, using a directory which contains a file called b'xxx\xff' (the byte 0xFF is invalid in UTF-8).

Default and UTF-8 Strict mode fail on print() with an encode error:

$ python3.7 ../ls.py
Traceback (most recent call last):
  File "../ls.py", line 5, in <module>
    print(name)
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcff' ...

$ python3.7 -X utf8=strict ../ls.py
Traceback (most recent call last):
  File "../ls.py", line 5, in <module>
    print(name)
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcff' ...

UTF-8 mode, POSIX locale, Python 2 and the UNIX ls command work but display mojibake:

$ python3.7 -X utf8 ../ls.py
xxx�

$ LC_ALL=C /python3.6 ../ls.py
xxx�

$ python2 ../ls.py
xxx�

$ ls
'xxx'$'\377'

List a directory into a text file

Similar to the previous example, except that the listing is written into a text file:

import os
names = os.listdir(os.curdir)
with open("/tmp/content.txt", "w") as fp:
    for name in names:
        fp.write("%s\n" % name)

Result:

Python Always works? Mojibake?
Python 2 Yes Yes
Python 3 No No
Python 3.5, POSIX locale No No
UTF-8 mode Yes Yes
UTF-8 Strict mode No No

"Yes" implies that mojibake can be produced. "No" means that the script can fail on decoding or encoding a filename depending on the locale or the filename. Typical error:

$ LC_ALL=C python3 test.py
Traceback (most recent call last):
  File "test.py", line 5, in <module>
    fp.write("%s\n" % name)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128)

Display Unicode characters into stdout

Very basic example used to illustrate a common issue, display the euro sign (U+20AC: €):

print("euro: \u20ac")

Result:

Python Always works? Mojibake?
Python 2 No No
Python 3 No No
Python 3.5, POSIX locale No No
UTF-8 mode Yes Yes
UTF-8 Strict mode Yes Yes

The UTF-8 and UTF-8 Strict modes will always encode the euro sign as UTF-8. If the terminal uses a different encoding, we get mojibake.

Replace a word in a text

The following script replaces the word "apple" with "orange". It reads input from stdin and writes the output into stdout:

import sys
text = sys.stdin.read()
sys.stdout.write(text.replace("apple", "orange"))

Result:

Python Always works? Mojibake?
Python 2 Yes Yes
Python 3 No No
Python 3.5, POSIX locale Yes Yes
UTF-8 mode Yes Yes
UTF-8 Strict mode No No

Producer-consumer model using pipes

Let's say that we have a "producer" program which writes data into its stdout and a "consumer" program which reads data from its stdin.

On a shell, such programs are run with the command:

producer | consumer

The question is if these programs will work with any data and any locale. UNIX users don't expect Unicode errors, and so expect that such programs "just work".

If the producer only produces ASCII output, no error should occur. Let's say the that the producer writes at least one non-ASCII character (at least one byte in the range 0x80..0xff ).

To simplify the problem, let's say that the consumer has no output (doesn't write results into a file or stdout).

A "Bytes producer" is an application which cannot fail with a Unicode error and produces bytes into stdout.

Let's say that a "Bytes consumer" does not decode stdin but stores data as bytes: such a consumer always works. Common UNIX command line tools like cat , grep or sed are in this category. Many Python 2 applications are also in this category.

"Python producer" and "Python consumer" are a producer and consumer implemented in Python.

Bytes producer, Bytes consumer

It always works, but it is out of the scope of this PEP since it doesn't involve Python.

Python producer, Bytes consumer

Python producer:

print("euro: \u20ac")

Result:

Python Always works? Mojibake?
Python 2 No No
Python 3 No No
Python 3.5, POSIX locale No No
UTF-8 mode Yes Yes
UTF-8 Strict mode No No

The question here is not if the consumer is able to decode the input, but if Python is able to produce its output. So it's similar to the Display Unicode characters into stdout case.

UTF-8 modes work with any locale since the consumer doesn't try to decode its stdin.

Bytes producer, Python consumer

Python consumer:

import sys
text = sys.stdin.read()
result = text.replace("apple", "orange")
# ignore the result

Result:

Python Always works? Mojibake?
Python 2 Yes Yes
Python 3 No No
Python 3.5, POSIX locale Yes Yes
UTF-8 mode Yes Yes
UTF-8 Strict mode No No

Python 3 fails on decoding stdin depending on the input and the locale.

Python producer, Python consumer

Python producer:

print("euro: \u20ac")

Python consumer:

import sys
text = sys.stdin.read()
result = text.replace("apple", "orange")
# ignore the result

Result, using the same Python version for the producer and the consumer:

Python Always works? Mojibake?
Python 2 No No
Python 3 No No
Python 3.5, POSIX locale No No
UTF-8 mode Yes Yes
UTF-8 Strict mode No No

This case combines a Python producer with a Python consumer, so the result is the subset of Python producer, Bytes consumer and Bytes producer, Python consumer .

Backward Compatibility

The main backward incompatible change is that the UTF-8 encoding is now used by default if the locale is POSIX. Since the UTF-8 encoding is used with the surrogateescape error handler, encoding errors should not occur and so the change should not break applications.

The more likely source of trouble comes from external libraries. Python can successfully decode data from UTF-8, but a library using the locale encoding can fail to encode the decoded text back to bytes. Hopefully, encoding text in a library is a rare operation. Very few libraries expect text, most libraries expect bytes and even manipulate bytes internally.

The PEP only changes the default behaviour if the locale is POSIX. For other locales, the default behaviour is unchanged.

Alternatives

Don't modify the encoding of the POSIX locale

A first version of the PEP did not change the encoding and error handler used for the POSIX locale.

The problem is that adding the -X utf8 command line option or setting the PYTHONUTF8 environment variable is not possible in some cases, or at least not convenient.

Moreover, many users simply expect that Python 3 behaves like Python 2: it doesn't bother them with encodings and "just works" in all cases. These users don't worry about mojibake, or even expect mojibake because of complex documents using multiple incompatibles encodings.

Always use UTF-8

Python already always uses the UTF-8 encoding on Mac OS X, Android and Windows. Since UTF-8 became the de facto encoding, it makes sense to always use it on all platforms with any locale.

The risk is to introduce mojibake if the locale uses a different encoding, especially for locales other than the POSIX locale.

Force UTF-8 for the POSIX locale

An alternative to always using UTF-8 in any case is to only use UTF-8 when the LC_CTYPE locale is the POSIX locale.

PEP 538 [1] "Coercing the legacy C locale to C.UTF-8" by Nick Coghlan proposes to implement that using the C.UTF-8 locale.

Use the strict error handler for operating system data

Using the surrogateescape error handler for operating system data creates surprising surrogate characters. No Python codec (except for utf-7 ) accepts surrogates so encoding text coming from the operating system is likely to raise an error. The problem is that the error comes late, very far from where the data was read.

The strict error handler can be used instead to decode ( os.fsdecode() ) and encode ( os.fsencode() ) operating system data and raise encoding errors as soon as possible. Using it helps find bugs more quickly.

The main drawback of this strategy is that it doesn't work in practice. Python 3 is designed on top on Unicode strings. Most functions expect Unicode and produce Unicode. Even if many operating system functions have two flavors, bytes and Unicode, the Unicode flavor is used in most cases. There are good reasons for that: Unicode is more convenient in Python 3 and using Unicode helps to support the full Unicode Character Set (UCS) on Windows (even if Python now uses UTF-8 since Python 3.6, see the PEP 528 [3] and the PEP 529 [2] ).

For example, if os.fsdecode() uses utf8/strict , os.listdir(str) fails to list filenames of a directory if a single filename is not decodable from UTF-8. As a consequence, shutil.rmtree(str) fails to remove a directory. Undecodable filenames, environment variables, etc. are simply too common to make this alternative viable.

Prior Art

Perl has a -C command line option and a PERLUNICODE environment variable to force UTF-8: see perlrun . It is possible to configure UTF-8 per standard stream, on input and output streams, etc.

Source: https://github.com/python/peps/blob/master/pep-0540.txt