[Mailman-Developers] Mailman3: decorating non-ascii message templates

[Aurélien Bompard]

Hey all,

I first thought about opening a bug for this, but I think it needs a
small discussion first.
In Mailman3 the message templates are stored on disk (welcome.txt,
footer-generic.txt, ...)
However, unless I missed something, there is no hint about the encoding
of those files.
As a result, when I try the decorate() function (from
mailman.handlers.decorate) on a non-ascii file, it crashes with a
classic UnicodeDecodeError: 'ascii' codec can't decode byte [...].
Note: if the file contains no string to replace (like $fqdn_listname),
it is passed through unchanged and it works.

So how should we deal with this? I think that the TemplateLoader in
mailman.app.templates should return unicode strings, because that's the
closest to the moment when files are read, and unicode conversion should
happen on the "external borders" of the application. Thus the
TemplateLoader's get() method seems to be the right place.
I see two options:

- We require that all template files are stored in either ascii or
utf-8. That's the easiest way to go, and we just decode the text after
getting the file.

- We use the fact that our Language entities contain encoding values.
When the template is loaded from an internal URL containing the
language, we add the corresponding encoding to the result metadata (to
be retrieved with the info() method) and use that to decode the
contents. This means that templates in non-localized directories still
have to be ascii-only, and the same goes for templates retrieved from
non-internal URLs (not starting with mailman://). It's more complex but
it may seem more natural to the administrator, since he won't have to
force UTF-8 encoding when editing a file. We must also think of not
making it too hard for Postorius, which will probably only get UTF-8
posted from the webpage (since it's always displayed in UTF-8 IIRC).

I would vote for the easy way for just requiring UTF-8 encoding, but I'd
like to hear your thoughts on this.

Thanks,
Aurélien
-- 
http://aurelien.bompard.org    ~~~~~~    xmpp:aurelien at bompard.org
Tell me and I will forget. Show me and I will remember. Involve me and
I will understand. -- Chinese proverb


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/mailman-developers/attachments/20131014/b98527dd/attachment.sig>