[Mailman-Users] Dealing with multiple charsets (list messages and web archive)

[Stefan Förster]

Hello world,

sometimes living outside of the U.S., or better said, living otside
"us-ascii" can be a real pain. Let me explain, considerning a default
installation for a list with default language "de". The LC_DESCRIPTIONS
dictionary will include:

add_language('de',    _('German'),              'iso-8859-1')

Now users are starting to post to the list and encode their messages
in:

ISO-8859-1
ISO-8859-15
UTF-8

And ofc, the ISO-encodings will use 8 bit transport encoding as well
as QP. This leads to nasty side effect #1: Messages not posted in
ISO-88599-1 will contain the list footer in an attachment.

When mail gets archived, the HTML header of the archive pages shows
the encoding specified in the list preferences, leading to nasty side
effect #2: Messages encoded in ISO-8859-15 and UTF-8 will show broken
non-ascii symbols.  Even worse, since non-ascii characters in message
headers may be encoded in a different charset than the one used in the
message body, you can end up with a message which show two sorts of
broken non-ascii chars (given headers are encoded in ISO-8859-15 and
message body in UTF-8).

What I'm looking for is a way to:

1. Examine an incoming message and determine wether it contains MIME
   parts with a content-type of text/plain.
2. Take those parts and do a charset conversion to the lists charset.
3. Add the list footer, header etc. in the lists charset.
4. Attach all parts of the original message which are not test/plain
   at the end of the message.

This way, I could set a lists encoding to "UTF-8" by adding to
mm_cfg.py:

def _(s):
    return s
add_language('de',    _('German'),       'utf-8')
del _

Users could post with whatever encoding they prefer, add attachments
of any type. The messages sent by Mailman would alway look like:

1. Message sent by user, recoded (if necessary) to UTF-8
2. List footer with UTF-8 encoding
3. Attachments like application/pdf

Is there a way to do this? If yes, can archives be rebuilt, too?


Ciao
Stefan

P.S: I don't know if the above is understandable, English is not my
first language. Feel free to ask for clarification.
-- 
Stefan Förster     http://www.incertum.net/     Public Key: 0xBBE2A9E9
FdI #320: Softwarewartung - Flags putzen, Strukturen nachrichten und Variablen
mit Öl auffüllen. (Hans Bonfigt)