[Mailman-Users] utf-8 error with all lists

[Jason Morrill]

Realizing I didn't respond to the list. So scroll down for my response. Then come back here to continue reading. 

Now that I have my Python-fu working I'm retesting Mark's suggestion:

webserv:~# python
Python 2.7.13 (default, Sep 26 2018, 18:42:22)
[GCC 6.3.0 20170516] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import string
>>> string.lowercase
'abcdefghijklmnopqrstuvwxyz'
>>>

It appears that lowercase does return an apparent properly lowercased string of letters. So, I'm uncertain why the hand-edited code would function differently.

Thanks,

  Jason Morrill

  IT Director

  Child and Family Agency

  860-443-2896 x1422

  http://www.childandfamilyagency.org/

________________________________________
From: Jason Morrill
Sent: Monday, May 13, 2019 10:29 AM
To: Mark Sapiro
Subject: RE: [Mailman-Users] utf-8 error with all lists

Mark, Thanks for the suggestions and link to the remainder of that prior message thread.

I too get the same results related to Python and lowercase.

Now, I'm not a Python programmer but can certainly stumble through to sort things out. Here's what I'm seeing:

webserv:~# python
Python 2.7.13 (default, Sep 26 2018, 18:42:22)
[GCC 6.3.0 20170516] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import string
>>> lowercase
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'lowercase' is not defined
>>> string
<module 'string' from '/usr/lib/python2.7/string.pyc'>
>>>

Per your suggestion I modified /var/lib/mailman/Mailman/Archiver/pipermail.py to set lowercase = 'abcdefghijklmnopqrstuvwxyz'

And indeed the issue worked itself out. No more errors and I was able to clear the qfiles/shunt folder too.

Per the Python docs, and your info, the string returned by lowercase should be a-z. And it depends upon the locale of the server. Something I've never purposefully changed.

In Debian it looks at /etc/locale.gen

In that file I see my locale set to be
en_US ISO-8859-1

Could that somehow be related to this issue?

Thanks,

  Jason Morrill

  IT Director

  Child and Family Agency

  860-443-2896 x1422

  http://www.childandfamilyagency.org/

________________________________________
From: Mailman-Users [mailman-users-bounces+morrillj=childandfamilyagency.org at python.org] on behalf of Mark Sapiro [mark at msapiro.net]
Sent: Saturday, May 11, 2019 2:13 PM
To: mailman-users at python.org
Subject: Re: [Mailman-Users] utf-8 error with all lists

On 5/10/19 12:37 PM, Jason Morrill wrote:
> Since the beginning of May I've had a problem with all messages going through Mailman. They are being delivered to the lists properly but copies are staying in qfiles/shunt
>
> The errors appearing in the Mailman error.1 log file indicate
> "
> May 05 10:16:29 2019 (1167) Uncaught runner exception: 'utf8' codec can't decode byte 0xaa in position 26: invalid start byte
...
>   File "/var/lib/mailman/Mailman/Archiver/pipermail.py", line 63, in fixAuthor
>     while i>0 and (L[i-1][0] in lowercase or
> UnicodeDecodeError: 'utf8' codec can't decode byte 0xaa in position 26: invalid start byte
>
> May 05 10:16:29 2019 (1167) SHUNTING: 1557065787.914554+25c37ef6e662cc98d41ac9d45855fe04f151e11a
>
> "
>
> This is on Debian 9.6 with Python 2.7.13 and Mailman 2.1.23 , all installed from the Debian package manager.
>
> I've spent several hours hunting around for a potential solution. Then stumbled upon this:
> https://mail.python.org/pipermail/mailman-users/2019-March/084243.html


If you read the entire thread (which is split in the archive with the
second part beginning at
<https://mail.python.org/pipermail/mailman-users/2019-March/084266.html>),
you will see that I think the issue is that whatever changed in your
system at the beginning of May is causing the Python

from string import lowercase

in /var/lib/mailman/Mailman/Archiver/pipermail.py to import a lowercase
object which should be the string 'abcdefghijklmnopqrstuvwxyz', but in
fact is longer and has non-ascii accented characters following 'z'.

The OP in that thread couldn't verify that that was the case, but I am
convinced that it is.

There doesn't seem to be a Debian patch to mailman 1:2.1.23-1+deb9u4
that would affect this so it is apparently something in Python or the OS.

I am certain that if you start with the base
/var/lib/mailman/Mailman/Archiver/pipermail.py and change

from string import lowercase

at line 13 to

lowercase = 'abcdefghijklmnopqrstuvwxyz'

the problem will be fixed.

--
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan
------------------------------------------------------
Mailman-Users mailing list Mailman-Users at python.org
https://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: https://mail.python.org/mailman/options/mailman-users/morrillj%40childandfamilyagency.org