[Mailman-Users] Messages shunted with "TypeError: decodingUnicode is not supported"

[Mark Sapiro]

Josh Clark wrote:

>Thanks again, Mark, this is all *very* appreciated.
>
>> I'm interested in how many of these addresses are u'...', and how  
>> they got to be that way.
>
>
>Of the 1639 e-mail addresses, 349 are u'...' I don't have any great  
>ideas about what might be out of the ordinary about my installation. 


Ist it the case that the u'...' ones all have at least one upper case
character in the local part and the rest don't?

 
>I've never made any mods to the Mailman scripts. I *have* used custom  
>html templates (charset=utf-8), but I don't think that this would  
>affect the character encoding on the server side, would it?


This is not an encoding issue. It's complicated, but unicode is an
abstract representation of characters. Unicode entities are encoded
into strings of characters (or bytes in the case of some encodings
that represent a character as multiple bytes) using some character
encoding like us-ascii or utf-8 or iso-8859-1, etc., and strings can
be decoded into unicode.

The issue here is not one of the character encoding of the email
address. It's that some of the email addresses (or at least the 'case
preserved' addresses) are stored as unicode objects instead of strings.


>If you'd  
>like to take a look:
><http://mailman.globalmoxie.com/mailman/listinfo/moxiemail>
>
>However, the majority of subscriptions come via a script that I set up  
>to send subscribe commands via email to the list. It accepts an email  
>address in a web form and sends it along as a subscribe command to  
>moxiemail-requests. Here's a sample (with domains changed to  
>example.com):
>
>------
>Return-Path: <support at example.com>
>X-Original-To: moxiemail-request at example.com
>Delivered-To: moxiemail-request at example.com
>Received: by houdini.example.com (Postfix, from userid 536)
>	id 26E3A312869A; Tue, 28 Oct 2008 14:38:07 -0500 (CDT)
>From: support at example.com
>To: moxiemail-request at example.com
>Subject: subscribe address=test at example.com
>Mime-Version: 1.0
>Content-Transfer-Encoding: 7bit
>Content-Type: text/plain; charset=US-ASCII; format=flowed
>Message-Id: <20081028193807.26E3A312869A at houdini.example.com>
>Date: Tue, 28 Oct 2008 14:38:07 -0500 (CDT)
>
>subscribe address=test at example.com
>------
>
>I'm happy to do any tests or troubleshooting that might be helpful to  
>get at the root problem.


Thanks. I'll try to think of what to do. One thing, if you trust me to
respect the confidentiality of your list membership, you can send me
off-list a copy of lists/moxiemail/config.pck.


>> The attached patch to Decorate.py (applied after removing the other  
>> one) will work around the problem.
>
>Excellent, thanks! A quick question before I try it: I've now tried to  
>send this message a few times; is it possible that these attempts were  
>successfully mailed to some portion of the list (e.g., some of the non- 
>unicode addresses)? If it's an indicator, the last address that showed  
>up with the "bin/withlist" commands that you suggested was also the  
>same address that showed up in the message footer for the debug patch  
>that you originally sent (and, I presume, the address that caused the  
>error).
>
>(If it's likely that some significant number of people have gotten  
>several copies of the message, I just want to add an apology for the  
>repeats before mailing it again. Either way, I'll give the patch a  
>spin in the next 24 hours.)


When we had the debug patch, there appeared to be only one entry in the
debug log. If that mail had been successfully delivered, the address
that caused the failure would have been preceeded by debug log entries
for the successful ones.

In any case, each failure resulted in a .pck file in the shunt queue.
Those files should still be there. If you dump them with bin/dumpdb,
you will see the message as the first object in the file and the
metadata as the second object. The metadata should contain an
'undelivered' list containing all those who didn't get that message.

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan