[Mailman-Users] UnicodeDecodeError with Mailman 2.1 and Python 2.6

[Stephen J. Turnbull]

Mark Sapiro writes:

 > I don't know what you are grepping, but if it's the mbox, you shouldn't
 > be looking for "\xea", you should be looking for "ê".

At least on recent BSD-based systems "\xea" is a well-defined escape
sequence, interpreted as the hexadecimal representation of a byte.
Dunno about GNU or proprietary systems.  (POSIX.2)

 > > Can I patched pipermail.py or HyperDatabase.py (or ???) in some way to
 > > work around this? I have LANG=en_US.UTF-8 and LC_TIME=en_DK.UTF8 in my
 > > shell environment: does that make a difference?
 > 
 > Probably not.

Actually, yes, it may.  If you previously had LANG=en_US.ISO8859-1 (or
similar), then Python's default encoding may have allowed all bytes.
On the other hand, 0xEA is not a legal byte in modern UTF-8 (it's out
of the range of legal Unicode scalars as a leading byte and it can't
be a trailing byte).

 > > This used to work just fine,

s/just fine/incorrectly but conveniently for the sysadmin/. :-)

I suppose it's possible that a Python upgrade wiped out a patch or
configuration that told Python to use a Latin-N default encoding, so
it reverted to ASCII.

I suspect that Mailman's copy of the email libraries has also evolved
quite a bit since 2.1.9 (I think that's what you upgraded from?), and
if it was a Mailman provided by the OS vendor, all bets are off.  Who
knows what patches they may have applied.