[Mailman-Developers] Proposal: option for UTF-8 emails without base64 encoding

[Petr Hroudný]

Hi all,

due to Python defaults, mailman exhibits strange behaviour when
processing UTF-8 emails.
When no header/footer is configured, mailman passes UTF-8 emails in
original form, i.e. 8bit.
However, when either header or footer is configured in mailman, it
uses Python's libraries to
add them and as a side effect it converts 8bit emails into 7bit base64
encoded ones.

This is highly undesirable in some cases. For instance, mailinglist
might be used to distribute
trouble tickets or other content which is expected to be easily
parsable by automated text-based
utilities. With base64, emails grow in size by 33 % and such emails
are getting much higher spam
scores since base64 is typically used by spammers to obfuscate the
payload. There are of course
much more reasons for not using base64 as the primary encoding method
for UTF-8 email.

The fix is quite simple and is already widely used by other projects.
All that needs to be done is
to redefine Python's UTF-8 charset properties, i.e. in every place
where you have

from email.Charset import Charset

you need to add:

email.Charset.add_charset('utf-8',email.Charset.SHORTEST, None, None)

With such setting, mailman will keep the 8bit encoding also when it's
adding header/footer and won't downconvert to 7bit+base64. So I'd like
to propose the above addition, at least as a configurable option
if there's any fear that enabling it by default could cause some problems.

Thanks, Petr