[Mailman-Users] Encoding problem with 2.15 to 2.18 upgrade with Finnish

[Mark Sapiro]

On 01/07/2016 01:20 PM, Mark Sapiro wrote:
> 
> You can keep the Debian package with it's utf-8 character sets and
> programmatically change all your lists so they work. In MM 2.1.19 I
> changed the character sets for Russian and Romanian from koi8-r and
> iso-8859-2 respectively to utf-8 because I was convinced that the former
> character sets were not appropriate for those languages. In so doing, I
> augmented Mailman's version updater to identify strings in list
> attributes that were not valid utf-8 and convert them from the former
> character set.
> 
> This is an imperfect process. It does guarantee that all the strings in
> list attributes are valid utf-8 encodings, but not necessarily that they
> are 'correct'.
> 
> I am working on a stand-alone script to do this. I will post it later today.


There is now a first cut at this script. The functions that do the
recoding are lifted straight from Mailman/versions.py so they have been
fairly well exercised. I didn't just import them because I wanted the
caller to know if they had changed anything and that required a minor
addition to the top level function to return a flag.

The script is at <https://www.msapiro.net/scripts/recode_list> (mirrored
at <http://fog.ccsf.edu/~msapiro/scripts/recode_list>)

Install the script in Mailman's bin/ directory and run it with the -h
option for full info. It uses Python's argparse function and as such
requires python 2.7.

In your case, to convert all Finnish preferred_language lists strings
from iso-8859-1 encoding to utf-8

recode_list --language=fi

would suffice. There are options to specify other than iso-8859-1 to
utf-8 and to produce output about skipped or unchanged lists, but the
defaults should be good for you and will report which lists were
changed. In no case does it report the actual changes, just what lists
were changed.

I recommend making backups of your lists/*/config.pck files before
running this in case something goes wrong.

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan