[New-bugs-announce] [issue41894] UnicodeDecodeError during load failure in non-UTF-8 locale

[Kevin]

New submission from Kevin <kadler at us.ibm.com>:

If a native module fails to load, the dynload code will call PyUnicode_FromString on the error message to give back to the user. This can cause a UnicodeDecodeError if the locale is not a UTF-8 locale and the error message contains non-ASCII code points.

While Linux systems almost always use a UTF-8 locale by default nowadays, AIX systems typically use non-UTF-8 locales by default. We encountered an issue where a customer did not have libbz2 installed, causing a load failure when bz2 tried to import _bz2 when running in an Italian locale:

$ LC_ALL=it_IT python3 -c 'import bz2'        
Traceback (most recent call last): 
 File "<string>", line 1, in <module> 
 File "/QOpenSys/pkgs/lib/python3.6/bz2.py", line 21, in <module> 
   from _bz2 import BZ2Compressor, BZ2Decompressor 
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe8 in position 161: invalid continuation byte

After switching to a UTF-8 locale, the problem goes away:

$ LC_ALL=IT_IT python3 -c 'import bz2'   
Traceback (most recent call last): 
 File "<string>", line 1, in <module> 
 File "/QOpenSys/pkgs/lib/python3.6/bz2.py", line 21, in <module> 
   from _bz2 import BZ2Compressor, BZ2Decompressor 
ImportError:    0509-022 Impossibile caricare il modulo /QOpenSys/pkgs/lib/python3.6/lib-dynload/_bz2.so. 
       0509-150   Il modulo dipendente libbz2.so non è stato caricato. 
       0509-022 Impossibile caricare il modulo libbz2.so. 
       0509-026 Errore di sistema: Un file o una directory nel nome percorso non esiste. 
       0509-022 Impossibile caricare il modulo /QOpenSys/pkgs/lib/python3.6/lib-dynload/_bz2.so. 
       0509-150   Il modulo dipendente /QOpenSys/pkgs/lib/python3.6/lib-dynload/_bz2.so non è stato caricato.


While this conceivably affects any Unix-like platform, the only system I can recreate it on is AIX and IBM i PASE. As far as I can tell, on Linux you will always get something like "error while loading shared libraries: libbz2.so.1.0: cannot open shared object file: No such file or directory". Even though there seems to be some translations in GLIBC, I have been unable to get them to be used on either Fedora or Ubuntu.

----------
components: Interpreter Core
messages: 377713
nosy: kadler
priority: normal
severity: normal
status: open
title: UnicodeDecodeError during load failure in non-UTF-8 locale
type: behavior
versions: Python 3.10, Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 3.9

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue41894>
_______________________________________