[New-bugs-announce] [issue43333] utf8 in BytesGenerator

[Darcy Beurle]

New submission from Darcy Beurle <darcy.beurle at gmail.com>:

I have some emails that I'm importing from an XML format according to rfc822. Some of these have some encoding other than ascii. I create the message with the default policy:

message = email.message_from_string(
                        # Extract text from xml
                        message_name.find("property_string").text,
                        policy=email.policy.default)

Then I want to convert this to bytes so I can append it to an IMAP folder using the imap_tools package:

mailbox.append(email.as_bytes(),
               "INBOX",
               dt=None,
               flag_set=(imap_tools.MailMessageFlags.SEEN))

Which then leads to the following output:

line 405, in parse_goldmine_output
    email.as_bytes(),
  File "/usr/lib64/python3.9/email/message.py", line 178, in as_bytes
    g.flatten(self, unixfrom=unixfrom)
  File "/usr/lib64/python3.9/email/generator.py", line 116, in flatten
    self._write(msg)
  File "/usr/lib64/python3.9/email/generator.py", line 181, in _write
    self._dispatch(msg)
  File "/usr/lib64/python3.9/email/generator.py", line 218, in _dispatch
    meth(msg)
  File "/usr/lib64/python3.9/email/generator.py", line 276, in _handle_multipart
    g.flatten(part, unixfrom=False, linesep=self._NL)
  File "/usr/lib64/python3.9/email/generator.py", line 116, in flatten
    self._write(msg)
  File "/usr/lib64/python3.9/email/generator.py", line 181, in _write
    self._dispatch(msg)
  File "/usr/lib64/python3.9/email/generator.py", line 218, in _dispatch
    meth(msg)
  File "/usr/lib64/python3.9/email/generator.py", line 436, in _handle_text
    super(BytesGenerator,self)._handle_text(msg)
  File "/usr/lib64/python3.9/email/generator.py", line 253, in _handle_text
    self._write_lines(payload)
  File "/usr/lib64/python3.9/email/generator.py", line 155, in _write_lines
    self.write(line)
  File "/usr/lib64/python3.9/email/generator.py", line 410, in write
    self._fp.write(s.encode('ascii', 'surrogateescape'))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 41-43: ordinal not in range(128)


If I change the line:

self._fp.write(s.encode('ascii', 'surrogateescape'))

to:

self._fp.write(s.encode('utf8', 'surrogateescape'))

then it writes the email body with the strange characters (same as in the xml). I'm not sure how to proceed. Those emails should be able to be processed, but the bytes writer doesn't seem to inherit the utf8 encoding from anywhere (e.g. if a utf8 policy is used).

----------
components: email
messages: 387749
nosy: barry, darcy.beurle, r.david.murray
priority: normal
severity: normal
status: open
title: utf8 in BytesGenerator
type: crash
versions: Python 3.9

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue43333>
_______________________________________