[Python-checkins] bpo-41894: Fix UnicodeDecodeError while loading native module (GH-22466)

[Miss Skeleton (bot)]

https://github.com/python/cpython/commit/47ca6799725bb4c40953bb26ebcd726d1d766361
commit: 47ca6799725bb4c40953bb26ebcd726d1d766361
branch: 3.8
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2020-10-14T19:11:08-07:00
summary:

bpo-41894: Fix UnicodeDecodeError while loading native module (GH-22466)


When running in a non-UTF-8 locale, if an error occurs while importing a
native Python module (say because a dependent share library is missing),
the error message string returned may contain non-ASCII code points
causing a UnicodeDecodeError.

PyUnicode_DecodeFSDefault is used for buffers which may contain
filesystem  paths. For consistency with os.strerror(),
PyUnicode_DecodeLocale is used for buffers which contain system error
messages. While the shortname parameter is always encoded in ASCII
according to PEP 489, it is left decoded using PyUnicode_FromString to
minimize the changes and since it should not affect the decoding (albeit
_potentially_ slower).

In dynload_hpux, since the error buffer contains a message generated
from a static ASCII string and the module filesystem path,
PyUnicode_DecodeFSDefault is used instead of PyUnicode_DecodeLocale as
is used elsewhere.

* bpo-41894: Fix bugs in dynload error msg handling

For both dynload_aix and dynload_hpux, properly handle the possibility
that decoding strings may return NULL and when such an error happens,
properly decrement any previously decoded strings and return early.

In addition, in dynload_aix, ensure that we pass the decoded string
*object* pathname_ob to PyErr_SetImportError instead of the original
pathname buffer.

Co-authored-by: Serhiy Storchaka <storchaka at gmail.com>
(cherry picked from commit 2d2af320d94afc6561e8f8adf174c9d3fd9065bc)

Co-authored-by: Kevin Adler <kadler at us.ibm.com>

files:
A Misc/NEWS.d/next/Core and Builtins/2020-10-02-11-35-33.bpo-41894.ffmtOt.rst
M Python/dynload_aix.c
M Python/dynload_hpux.c
M Python/dynload_shlib.c

diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-10-02-11-35-33.bpo-41894.ffmtOt.rst b/Misc/NEWS.d/next/Core and Builtins/2020-10-02-11-35-33.bpo-41894.ffmtOt.rst
new file mode 100644
index 0000000000000..571f5dae1a4a1
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2020-10-02-11-35-33.bpo-41894.ffmtOt.rst	
@@ -0,0 +1,3 @@
+When loading a native module and a load failure occurs, prevent a possible
+UnicodeDecodeError when not running in a UTF-8 locale by decoding the load
+error message using the current locale's encoding.
diff --git a/Python/dynload_aix.c b/Python/dynload_aix.c
index b3ff8e288c1a6..25fe19e601acb 100644
--- a/Python/dynload_aix.c
+++ b/Python/dynload_aix.c
@@ -144,10 +144,16 @@ aix_loaderror(const char *pathname)
         ERRBUF_APPEND(message[i]);
         ERRBUF_APPEND("\n");
     }
-    errbuf[strlen(errbuf)-1] = '\0';            /* trim off last newline */
-    pathname_ob = PyUnicode_FromString(pathname);
-    errbuf_ob = PyUnicode_FromString(errbuf);
-    PyErr_SetImportError(errbuf_ob, NULL, pathname);
+    /* Subtract 1 from the length to trim off trailing newline */
+    errbuf_ob = PyUnicode_DecodeLocaleAndSize(errbuf, strlen(errbuf)-1, "surrogateescape");
+    if (errbuf_ob == NULL)
+        return;
+    pathname_ob = PyUnicode_DecodeFSDefault(pathname);
+    if (pathname_ob == NULL) {
+        Py_DECREF(errbuf_ob);
+        return;
+    }
+    PyErr_SetImportError(errbuf_ob, NULL, pathname_ob);
     Py_DECREF(pathname_ob);
     Py_DECREF(errbuf_ob);
     return;
diff --git a/Python/dynload_hpux.c b/Python/dynload_hpux.c
index e59d00435ec7d..9fa003c944a7b 100644
--- a/Python/dynload_hpux.c
+++ b/Python/dynload_hpux.c
@@ -37,9 +37,20 @@ dl_funcptr _PyImport_FindSharedFuncptr(const char *prefix,
         char buf[256];
         PyOS_snprintf(buf, sizeof(buf), "Failed to load %.200s",
                       pathname);
-        PyObject *buf_ob = PyUnicode_FromString(buf);
+        PyObject *buf_ob = PyUnicode_DecodeFSDefault(buf);
+        if (buf_ob == NULL)
+            return NULL;
         PyObject *shortname_ob = PyUnicode_FromString(shortname);
-        PyObject *pathname_ob = PyUnicode_FromString(pathname);
+        if (shortname_ob == NULL) {
+            Py_DECREF(buf_ob);
+            return NULL;
+        }
+        PyObject *pathname_ob = PyUnicode_DecodeFSDefault(pathname);
+        if (pathname_ob == NULL) {
+            Py_DECREF(buf_ob);
+            Py_DECREF(shortname_ob);
+            return NULL;
+        }
         PyErr_SetImportError(buf_ob, shortname_ob, pathname_ob);
         Py_DECREF(buf_ob);
         Py_DECREF(shortname_ob);
diff --git a/Python/dynload_shlib.c b/Python/dynload_shlib.c
index c51f97abd2863..23e58ed046e30 100644
--- a/Python/dynload_shlib.c
+++ b/Python/dynload_shlib.c
@@ -105,7 +105,7 @@ _PyImport_FindSharedFuncptr(const char *prefix,
         const char *error = dlerror();
         if (error == NULL)
             error = "unknown dlopen() error";
-        error_ob = PyUnicode_FromString(error);
+        error_ob = PyUnicode_DecodeLocale(error, "surrogateescape");
         if (error_ob == NULL)
             return NULL;
         mod_name = PyUnicode_FromString(shortname);
@@ -113,7 +113,7 @@ _PyImport_FindSharedFuncptr(const char *prefix,
             Py_DECREF(error_ob);
             return NULL;
         }
-        path = PyUnicode_FromString(pathname);
+        path = PyUnicode_DecodeFSDefault(pathname);
         if (path == NULL) {
             Py_DECREF(error_ob);
             Py_DECREF(mod_name);