[issue44170] ShareableList cannot safely handle multibyte utf-8 characters

[Huw Jones]

New submission from Huw Jones <huw at huwcbjones.co.uk>:

I've experienced a UnicodeDecodeError when adding unicode strings that contain multibye utf-8 characters into a shareable list.
My observation is that ShareableList chunks the list of strings before sending it over the process boundary, however this chunking process is not multibyte aware and will chunk in the middle of multibyte characters.
On the other end, this results in the ShareableList throwing a UnicodeDecodeError when it fails to decode not-a-full multibyte utf-8 character.

>From running the attached MWE, I see that the string is sent in two chunks, the first being b'Boom \xf0\x9f\x92\xa5 \xf0\x9f\x92\xa5 \xf0' which clearly splits the 4 bytes of the 💥 character into the first byte and remaining 3 bytes.

----------
components: Library (Lib)
files: shareablelist_mwe.py
messages: 393868
nosy: huwcbjones
priority: normal
severity: normal
status: open
title: ShareableList cannot safely handle multibyte utf-8 characters
type: crash
versions: Python 3.8
Added file: https://bugs.python.org/file50049/shareablelist_mwe.py

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue44170>
_______________________________________