[Mailman-Users] UnicodeDecodeError with Mailman 2.1 and Python 2.6

[David Magda]

On Tue, September 1, 2015 22:43, Mark Sapiro wrote:
> On 09/01/2015 06:55 PM, David Magda wrote:
>>
>> The message in question seems to be from October 2005, and it has the
>> "Subject: Você [
]” header. Running under Debian 5 (Python 2.5), archive
>> processing wasn’t an issue: messages were coming into the mbox and the
>> HTML page was being generated until August 25. I then did the upgrade on
>> that day.
>>
>> The mbox then no longer has messages until September 1, the day when I
>> did “arch —wipe”, even though the mail.log shows messages coming in (and
>> going out AFAICT).
>
>
> So what has probably happened is that archiving has been broken since
> the upgrade on Aug 25. There should be a an error message with traceback
> in Mailman's 'error' log for each message that wasn't archived. More
> importantly, the unarchived messages should all be in Mailman's 'shunt'
> queue and running Mailman's bin/unshunt should process them for the
> archive.

The last error message that is in the 'error' log is from Aug 24:

mailman at mail1:~$ ls -lrth logs/
total 16M
-rw-rw-r-- 1 mailman  mailman 1.2K Mar  2  2014 fromusenet
-rw-rw-r-- 1 mailman  mailman  63K Mar 18 09:57 locks
-rw-rw-r-- 1 www-data mailman 3.1K Mar 31 18:04 mischief
-rw-rw-r-- 1 mailman  mailman 808K Aug 24 08:00 smtp-failure
-rw-rw-r-- 1 mailman  mailman 354K Aug 24 08:14 bounce
-rw-rw-r-- 1 www-data mailman 1.1M Aug 24 13:14 error
-rw-rw-r-- 1 mailman  mailman 584K Aug 31 10:45 qrunner
-rw-rw-r-- 1 www-data mailman 1.6M Sep  1 12:09 vette
-rw-rw-r-- 1 www-data mailman 278K Sep  2 09:03 subscribe
-rw-rw-r-- 1 mailman  mailman 3.8M Sep  2 09:48 smtp
-rw-rw-r-- 1 mailman  mailman 2.7M Sep  2 09:48 post

>> So would there be anything besides a “—wipe” that could otherwise be
>> done to prevent this archive breakage? Attempt a “cleanarch" first?
>> Seems strange that a message from 2005 could stop archive processing for
>> current years/months.
>
> Presumably, you did some kind of bin/arch process on Aug 25. The answer
> is don't do that. Or if you didn't do that, then the answer may lie in
> whatever changed in Python.

Not that I'm aware of, unless the Debian package runs it in some way
pre-/post-upgrade. I didn't start looking at bin/arch until I noticed the
breakage.

> You should never be running bin/arch without --wipe unless you are
> importing archives from elsewhere to be added to an existing archive,
> and even in that case, it might be preferable to stop Mailman,
> concatenate the existing Mailman mbox and import mbox into a combined
> mbox and run 'bin/arch --wipe'. Then after success, start Mailman again.
>
> You should also consider running the contrib/mmdsr script from the
> Mailman source distro or if not that, somehow looking a Mailman's error
> log frequently for problems to discover them sooner.

We have a few more hosts to upgrade, so I'll have to look more closely at
this. However some of these hosts have quite a few lists, so I'm not sure
how practical it will be examine every one to see if they're working
properly.