[Mailman-Developers] Mailman3: decorating non-ascii message templates

[Stephen J. Turnbull]

Barry Warsaw writes:

 > >So how should we deal with this? I think that the TemplateLoader in
 > >mailman.app.templates should return unicode strings, because that's the
 > >closest to the moment when files are read, and unicode conversion should
 > >happen on the "external borders" of the application.
 > 
 > Completely agreed.  We need to convert to unicode at the edges and
 > treat strings internally as unicode. 

+1

 > >Thus the TemplateLoader's get() method seems to be the right place.  I see
 > >two options:
 > >
 > >- We require that all template files are stored in either ascii or
 > >utf-8. That's the easiest way to go, and we just decode the text after
 > >getting the file.
 > >
 > >- We use the fact that our Language entities contain encoding values.

 > I would vote for UTF-8 for all files, internal or external, but
 > maybe there are some languages for which this will cause problems.

I don't think there are, any more.  There are some programmers and
translators for whom it will be annoying, and maybe the Chinese
government will refuse to use Mailman (they have some requirement that
software use GB18030, which is Red Unicode).

(BTW, I've always been in favor of making UTF the default for text
files using programs, all the way back to PEP 263.)

 > We do have a `charset` variable in the config file for languages,

Which is useless for the languages where we would actually care
because there are multiple charsets in common use in those languages,
and any setting will surely screw up some users.

 > that some of the charsets in MM2.1 are not UTF-8, but I'm not sure
 > if any of them are UTF-8 incompatible.

I'll take a look but it will have to be after I get back from the
Mentor Summit.  Wish you were here!

Steve