[Mailman-Users] Charset problem. cmd_confirm assumes ascii code.

[Osmo Kujala]

Hi, Sorry about previous mail. I tried replying by using
a very stupid editor. I tried canceling the
the message but there wasn't that option. Lets try again.

I'm continuing my monolog(ue):

> Hello

> We have problem with confirm mail when mail body contains
> characters out of ascii code (range(128)). Mailman version
> is 2.1.5 . I think Mailman could leave the body unread
> when Subject contains the confirm code or at least it should
> stop to "end" line.

In the chart of IncomingRunner.py I can see why this is
not the case. I thought confirm is one of the jobs of
command processor like join, leave and request, when even
unicode processing would succeed better...

> If Mailman wants to check through the body
> it would be fair not to bug with the non-ascii codes.
> People have auto-signatures with different character codes and
> they may have automatic copy of the original mail as citation
> (quotation). Mailmans request for confirmation contains non-ascii
> code if the language code of the list is such.

> Excerpt of the error log:

> Apr 26 12:53:40 2006 (11216) Uncaught runner exception: 'ascii' codec can't decode byte 0xe4 in position 3: ordinal not in range(128)
> Apr 26 12:53:40 2006 (11216) Traceback (most recent call last):
>   File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 111, in _oneloop
>     self._onefile(msg, msgdata)
>   File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 167, in _onefile
>     keepqueued = self._dispose(mlist, msg, msgdata)
>   File "/usr/lib/mailman/Mailman/Queue/CommandRunner.py", line 237, in _dispose
>     res.process()
>   File "/usr/lib/mailman/Mailman/Queue/CommandRunner.py", line 110, in process
>     stop = self.do_command(cmd, args)
>   File "/usr/lib/mailman/Mailman/Queue/CommandRunner.py", line 135, in do_command
>     return self.do_command(cmd, args)
>   File "/usr/lib/mailman/Mailman/Queue/CommandRunner.py", line 137, in do_command
>     return handler.process(self, args)
>   File "/usr/lib/mailman/Mailman/Commands/cmd_confirm.py", line 86, in process
>     if line.lstrip() == match:
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 3: ordinal not in range(128)

> Apr 26 12:53:40 2006 (11216) SHUNTING: 1146045219.7761891+6b2f2afceea57f26e6a9302e6a615bbd8266d493


> Copy of the shunted file:

> /usr/lib~mailman/bin/dumpdb -p /var/spool/mailman/shunt/1146045219.7761891+6b2f2afceea57f26e6a9302e6a615bbd8266d493.pck

> [----- start pickle file -----]
> <----- start object 1 ----->
> >From KUJALA at jylk.jyu.fi Wed Apr 26 12:53:39 2006
> Received: from posti1.jyu.fi (posti1.jyu.fi [130.234.4.32])
>         by lists.jyu.fi (8.13.1/8.13.1) with ESMTP id k3Q9rdOt029242
>         for <ok-testi-request at lists.jyu.fi>; Wed, 26 Apr 2006 12:53:39 +0300
> Received: from localhost (localhost.localdomain [127.0.0.1])
>         by posti1.jyu.fi (8.13.6/8.13.4) with ESMTP id k3Q9rdAc030411
>         for <ok-testi-request at lists.jyu.fi>; Wed, 26 Apr 2006 12:53:39 +0300
> Received: from jylk.jyu.fi (jylk.jyu.fi [130.234.4.141])
>         by posti1.jyu.fi (8.13.6/8.13.6) with ESMTP id k3Q9rcUj030404
>         for <ok-testi-request at lists.jyu.fi>; Wed, 26 Apr 2006 12:53:38 +0300
> Received: from jylk.jyu.fi by jylk.jyu.fi (PMDF V4.3-7 #2482)
>         id <01M1PGKIPOB4ECWZZ9 at jylk.jyu.fi>; Wed, 26 Apr 2006 12:54:35 EET
> Date: Wed, 26 Apr 2006 12:54:11 +0200 (EET)
> From: Osmo Kujala <KUJALA at jylk.jyu.fi>
> Subject: Re: confirm 8e7af57ba7a721f68c96b0111f988ad36c8804f7
> In-reply-to: Your message dated "Wed, 26 Apr 2006 12:52:38 +0300"
>         <mailman.0.1146045158.29214.ok-testi at lists.jyu.fi>
> To: ok-testi-request at lists.jyu.fi
> Message-id: <01M1PQ8OVNMYECWZZ9 at jylk.jyu.fi>
> X-Envelope-to: ok-testi-request at lists.jyu.fi
> MIME-version: 1.0
> Content-type: TEXT/PLAIN; CHARSET=ISO-8859-1
> Content-transfer-encoding: 8BIT
> X-Virus-Scanned: amavisd-new at cc.jyu.fi
> X-Spam-Status: No, hits=-5.794 required=5 tests=[ALL_TRUSTED=-3.3, AWL=0.105,
>         BAYES_00=-2.599]
> X-Spam-Level:


> end

> 123ä
> 23456ö ÖÄ


> <----- start object 2 ----->
> {   '_parsemsg': False,
>     'lang': 'en',
>     'listname': 'ok-testi',
>     'received_time': 1146045219.7761891,
>     'torequest': 1,
>     'version': 3,
>     'whichq': '/var/spool/mailman/commands'}
> [----- end pickle file -----]


> What's especially nasty is that the waiting time for confirmation
> stops with this error and the entry disappears from the list.
> So it's hard to find out what happened.

> I tried to find help from mailman-users archives. Found some
> identical cases from years 2003 and 2004 by searching keywords
> "ascii ordinal confirm" but no solution to the problem.

http://www.mail-archive.com/mailman-users@python.org/msg20140.html

http://www.mail-archive.com/mailman-users@python.org/msg22621.html

...

>From Mailman-developers:

http://www.mail-archive.com/mailman-developers@python.org/msg07411.html

Osmo