[Mailman-Developers] Proposal: option for UTF-8 emails without base64 encoding

[Barry Warsaw]

On May 1, 2009, at 3:01 AM, Petr Hroudný wrote:

> due to Python defaults, mailman exhibits strange behaviour when
> processing UTF-8 emails.
> When no header/footer is configured, mailman passes UTF-8 emails in
> original form, i.e. 8bit.
> However, when either header or footer is configured in mailman, it
> uses Python's libraries to
> add them and as a side effect it converts 8bit emails into 7bit base64
> encoded ones.

Just to be clear, we're talking about headers and footers added by  
appending the text to the main body, right?  When headers and footers  
are attached via MIME, Mailman always does the right thing?

> This is highly undesirable in some cases. For instance, mailinglist
> might be used to distribute
> trouble tickets or other content which is expected to be easily
> parsable by automated text-based
> utilities. With base64, emails grow in size by 33 % and such emails
> are getting much higher spam
> scores since base64 is typically used by spammers to obfuscate the
> payload. There are of course
> much more reasons for not using base64 as the primary encoding method
> for UTF-8 email.

One option of course would be to simply not add headers and footers  
for these types of mailing lists.  Another option, which I proposed in  
bug 373083, is to add an option to force a mailing list to always  
attach headers and footers as MIME.  You risk some users not seeing  
them because their MUA doesn't display them inline and they don't know  
how to click on the attachments, but I think that's an acceptable risk  
for the minority of mailing lists that care about this.  At least for  
MM2.1.

> The fix is quite simple and is already widely used by other projects.
> All that needs to be done is
> to redefine Python's UTF-8 charset properties, i.e. in every place
> where you have
>
> from email.Charset import Charset
>
> you need to add:
>
> email.Charset.add_charset('utf-8',email.Charset.SHORTEST, None, None)
>
> With such setting, mailman will keep the 8bit encoding also when it's
> adding header/footer and won't downconvert to 7bit+base64. So I'd like
> to propose the above addition, at least as a configurable option
> if there's any fear that enabling it by default could cause some  
> problems.

I'll note that you don't need to change Mailman at all do to this.   
You simply need to add this to your mm_cfg.py file and I'll bet it  
will just work for you.

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 832 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20090514/59020ae6/attachment.pgp>