[Bug 1415406] [NEW] Message excerpt corruption on admindb Web UI

[Yasuhito FUTATSUKI at POEM]

Public bug reported:

Some messages held on admindb cannot display correctly becase of partial Unicode conversion error or
incomplete multi-byte character on mm_cfg.ADMINDB_PAGE_TEXT_LIMIT boundary.

Message character corruption has been occured in conditions below.

(1) Message charset/encoding is charset/encoding of multi-byte charaters.
(2) Message charset/encoding differs from web display charset/encoding.
(3) Message contains character that cannot convert to Unicode by using Python codec.
or
(3') Message body size exceeds mm_cfg.ADMINDB_PAGE_TEXT_LIMIT in bytes after decoding mime
     and cut down along multi-byte charater's byte sequence.

Under these conditions, Unicode error occur in converting message charset/encoding and message
has remained not to convert charset/encoding.

A patch attached below solves (3) by using decode/encode with 'replace' error handling scheme,
and (3') by rounding on character boundary not to exceeds the limit in bytes after 
character/encoding converted.

Note: Even If Message charset/encoding is same as web display charset/encoding, condition (3')
may produce invalid html, but patch below don't fix it.

** Affects: mailman
     Importance: Undecided
         Status: New

** Attachment added: "patch-admindb-py.txt"
   https://bugs.launchpad.net/bugs/1415406/+attachment/4307114/+files/patch-admindb-py.txt

-- 
You received this bug notification because you are a member of Mailman
Coders, which is subscribed to GNU Mailman.
https://bugs.launchpad.net/bugs/1415406

Title:
  Message excerpt corruption on admindb Web UI

To manage notifications about this bug go to:
https://bugs.launchpad.net/mailman/+bug/1415406/+subscriptions