[Mailman-Users] Digests stop being sent after a few days if theMailman interface is changed to Japanese

[Mark Sapiro]

S. Patrick Eaton wroye:
>
>I've been having a number of problems with the distribution of digest
>messages that I am hoping someone here can help me with.
>
>The first is that, unless the Mailman administrative interface is
>changed to Japanese, any Japanese characters included in digest
>messages get changed into question mark characters when the digest is
>sent out, even if those characters appeared correctly in the message
>that was sent out to non-digest subscribers. If the interface is
>changed to Japanese, this does not occur; Japanese characters then
>appear in the digests the way they should.


This occurs with 'plain' format digests because they are produced in
the character set of the list's language which for English is
us-ascii. It does not occur with the MIME format digest because in
that digest, the individual messages are separate MIME parts that
retain the original character set and encoding.

You could try setting the digest members' digest format to MIME
(uncheck plain in Membership Management... -> Membership List for
digest members and set Digest options -> mime_is_default_digest to
MIME so new members get MIME digests by default.

This may or may not be satisfactory for any particular digest member
depending on the member's MUA. Some MUAs including those on smart
phones do not handle MIME format digests well.


>This, however, inevitably runs into another problem: When the
>interface is changed to Japanese, the distribution of digests
>continues for a while, but inevitably it stops at some point and does
>not resume until the interface is changed back to English, at which
>point all of the messages that had not been distributed in digests
>since the last digest was sent get included in the next digest
>message, which can be quite massive if the digests have not been
>getting sent out regularly for a while.
>
[...]
>
>I have encountered this very same problem in two Mailman
>installations, one running Mailman 2.1.9 and another running Mailman
>2.1.12, on two entirely separate hosted environments, so I am
>reasonably sure that the problem is specific to Mailman and not to
>installation or environment-related issues.


There should be a message in Mailman's error log saying "send_digests()
failed:" with a specific exception. Beginning with Mailman 2.1.14, a
traceback from the exception is also logged.

The exception itself occurs because there is some message among those
accumulated for the digest (in the lists/LISTNAME/digest.mbox file)
that is causing some exception to be thrown in processing the messages
for the digest. It is not possible to say what the problem is without
seeing the exception and the offending message or digest.mbox.

This problem used to also stop list delivery when the digest.mbox
reached the threshold size that would trigger a digest, but that bug
has been fixed. However,your situation is more difficult. Ideally,
Mailman would just remove the offending message from the digest and
store it aside somewhere, but doing this is not straightforward,and it
currently isn't done, so what happens is the exception aborts the
digest processing, but the digest.mbox is unchanged, so the exception
continues each time there is an attempt to produce a digest.

In your situation, the exception depends on the list's language so
changing the language allows a digest to be produced and the
digest.mbox removed so the problem doesn't recur even if you change
the language back until another offending message arrives.

If you are able to report the exception from the error log, please do.
Also, if you are able and willing to send me (off list) a copy of an
offending lists/LISTNAME/digest.mbox file or put it on the web
somewhere and post or send me a link, I will try to find just what it
is about certain messages that causes this.

A possible avoidance is to change Mailman's character set for English
from us-ascii to utf-8 and leave the list's language set to English.
To do this, you would add the line

add_language('en', 'English (USA)', 'utf-8')

to mm_cfg.py. You could possibly also avoid this by changing Mailman's
character set for Japanese from euc-jp to utf-8, but this is more
difficult as it also requires transliterating the Japanese templates
and message catalog.

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan