[Mailman-Users] UnicodeDecodeError accessing into web admin interface

[Mark Sapiro]

On 11/24/2014 05:15 PM, Ferriol wrote:
> 
> El 24/11/14 a les 21:10, Mark Sapiro ha escrit:
>>
>> This is probably an English language list so it's Mailman character set
>> is us-ascii, and there is something (?) that has non-ascii, probably
>> utf-8 because 0xc3 is a utf-8 leadin, encoded text.
>>
>> Finding exactly what is problematic.
>>
>> The additional Python and environment information from the above error
>> log entry may help.
> This is not an English list but uses a latin alphabet is in Catalan that
> uses tittles and is latin_1 encoding


Is the lists' preferred_language Catalan or English (USA)?

If it is Catalan, the web UI and all the i18n strings should be utf-8
encoded and you shouldn't be seeing this issue.

Or are you saying that the lists preferred_language is English (USA),
but you are entering latin_1 encoded text in the various fields in the
web UI or via config_list?

In any case, we shouldn't be throwing exceptions, even if you are
entering non-ascii in an English list. I'll look into this further.


> Some full log entries below


OK.


>> And the full traceback from this may help too.
> $  config_list -o config listname
> Traceback (most recent call last):
>   File "/usr/sbin/config_list", line 362, in <module>
>     main()
>   File "/usr/sbin/config_list", line 355, in main
>     do_output(listname, outfile)
>   File "/usr/sbin/config_list", line 128, in do_output
>     do_list_categories(mlist, k, None, outfp)
>   File "/usr/sbin/config_list", line 198, in do_list_categories
>     outfp.write(NL.join(lines).replace('"', '\\"'))
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in
> position 9: ordinal not in range(128)


Unfortunately, that doesn't help too much, but thanks anyway.


>> Also, does this occur with all lists or only one?
> In 4 lists. In one on General options, topics and non-digest options,
> other one general options and non-digest options and the other two only
> in general options. May be has the same information in some value...
>>         output = unicode(output, errors='replace')
>>         for i in range(len(self.cells[row])):
>>             output = output + unicode(
>>                 self.FormatCell(row, i, indent + 2), errors='replace')
>>         output = output.encode(errors='replace')
>>
> I tried this and doesn't work, this is the log entry:
...
> admin(27848):   File "/var/lib/mailman/Mailman/htmlformat.py", line 192,
> in FormatRow
> admin(27848):     output = output.encode(errors='replace')
> admin(27848): TypeError: encode() takes no keyword arguments


Sorry about that. The keyword argument errors='replace' requires Python
2.7. Anyway, you got the information another way.


>> Alternatively, you could replace the above two lines with
>>
>>         from Mailman.Logging.Syslog import syslog
>>         try:
>>             for i in range(len(self.cells[row])):
>>                 output = output + self.FormatCell(row, i, indent + 2)
>>         except UnicodeDecodeError:
>>             syslog('error', 'UnicodeDecodeError: %s', output)
>>
>> which should result in the problem bit being omitted from the page, and
>> Mailman's error log containing the html for the part of the page leading
>> up to the problem.
> 
> This works perfect, with this I can find all values with rare characters.


OK. good.


> I'm reseting this values with config_list. I prepare a file with
> value='' p.ex.
> # -*- python -*-
> # -*- coding: utf-8 -*-
> info=''
> welcome_msg=''
> msg_footer=''
> topics=''
> 
> and then I do config_list -i file listname
> 
> 
> Thanks, I solved the problem with your help ! Thanks for your work !


Thanks for the info you provided, and I'm glad you were able to find the
problem strings.

I will look into making Mailman behave more gracefully in cases like this.

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan