Email parsing and unicode/utf8

[Thomas Jollans]

Hi,

I just stumbled over some curious behaviour of the stdlib email parsing
APIs which accept strings rather than bytes. It appears that you can't
parse an 8-bit UTF-8 message you have as a str without first encoding it.

The docs
<https://docs.python.org/3/library/email.parser.html#feedparser-api> do
mention some problems (which I saw after the fact):

> class email.parser.FeedParser(_factory=None, *, policy=policy.compat32)
> 
>     Works like BytesFeedParser except that the input to the feed() method must be a string. This is of limited utility, since the only way for such a message to be valid is for it to contain only ASCII text or, if utf8 is True, no binary attachments.
> 
>     Changed in version 3.3: Added the policy keyword.

Okay, cool - let's try parsing a message with text only (no attachments,
no BINARYMIME), with a UTF-8 Content-Type, and a policy with utf8=True.

Python 3.7.1rc2 (default, Oct 14 2018, 15:27:05)
[GCC 8.2.1 20180831 [gcc-8-branch revision 264010]] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import email.parser, email.policy
>>> pol = email.policy.SMTPUTF8
>>> pol.utf8
True
>>> pol.cte_type
'8bit'
>>> msg = '''MIME-Version: 1.0
... Content-Type: text/plain; charset="utf-8"
... Content-Transfer-Encoding: 8bit
... Subject: ¿Will it parse? Нет.
...
... ¡This message contains two (٢) non-ASCII characters!
... '''
>>> fp = email.parser.FeedParser(policy=pol)
>>> fp.feed(msg)
>>> msg_obj = fp.close()
>>> msg_obj
<email.message.EmailMessage object at 0x7ff028012e10>
>>> print(msg_obj.get_content())
�This message contains two (\u0662) non-ASCII characters!

>>> print(msg_obj['Subject'])
¿Will it parse? Нет.

I don't know WHAT it's doing with the body there... It doesn't look like
utf8 mode actually did anything. Interesting that the subject header
survived! Maybe this is what the utf8=True does?

>>> email.policy.default.utf8
False
>>> fp2 = email.parser.FeedParser(policy=email.policy.default)
>>> fp2.feed(msg)
>>> msg_obj2 = fp2.close()
>>> print(msg_obj2['Subject'])
¿Will it parse? Нет.

Nope. Apparently, contrary to what my reading of the docs suggests, the
utf8 flag does nothing at all when parsing.

Just to check that this was in fact a perfectly valid email:

>>> bfp = email.parser.BytesFeedParser(policy=pol)
>>> bfp.feed(msg.encode('utf-8'))
>>> msg_objb = bfp.close()
>>> print(msg_objb.get_content())
¡This message contains two (٢) non-ASCII characters!

>>> print(msg_objb['Subject'])
¿Will it parse? Нет.

BytesFeedParser is happy.

Question: Is this a bug? Am I missing something? Does the clause in the
docs about utf8 mean anything?

Cheers
Thomas