[pypy-issue] Issue #3165: PyPy should set `errors="surrogatepass"` when decoding numpy arrays to unicode scalars (pypy/pypy)
Eric Wieser
issues-reply at bitbucket.org
Tue Feb 4 11:18:08 EST 2020
New issue 3165: PyPy should set `errors="surrogatepass"` when decoding numpy arrays to unicode scalars
https://bitbucket.org/pypy/pypy/issues/3165/pypy-should-set-errors-surrogatepass-when
Eric Wieser:
See [https://github.com/numpy/numpy/issues/15363](https://github.com/numpy/numpy/issues/15363), where the bug was reported in numpy.
The fix in numpy was to replace calls to `PyUnicode_DecodeUTF32` with calls to `PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND, ...)`.
However, it seems that PyPI does not use our code there, and has their own implementation. Assuming `PyUnicode_FromKindAndData` is not available, the equivalent python code would be `some_bytes.decode('utf-32-le', errors='surrogatepass')`.
More information about the pypy-issue
mailing list