[Moin-user] Problem with encoding URLS

[ml at mherrn.de]

Hi Paul,

thanks for your help.


> > When trying to access this page the apache log tells me:
> > 
> > -----/-----
> > mod_wsgi (pid=31988): Exception occurred processing WSGI script
> > '/home/user/www/moin/moin.wsgi'.
> > Traceback (most recent call last):
> >    File "/usr/lib/python2.7/dist-packages/werkzeug/wsgi.py", line 567, in
> > __call__
> >      cleaned_path = cleaned_path.encode(sys.getfilesystemencoding())
> > UnicodeEncodeError: 'ascii' codec can't encode character u'\\xf6' in
> > position 2: ordinal not in range(128)
> > -----/-----
> > 
> > What is going on here? Do I have a misconfiguration in moinmoin? Why is it
> > trying to encode it in ASCII?
> 
> Here, I imagine that your locale setting isn't helping. What do you get at the 
> Python prompt if you call sys.getfilesystemencoding() ?

As the user I running the moin instance, I get:
'UTF-8'

> 
> You may need to look at your system's default locale and/or the user's locale, 
> I guess. 

The users locale is:

LANG=en_US.UTF-8
LANGUAGE=en_US:en
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES=POSIX
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

So that should all be well.
I have now done some tests and wrote a minimal WSGI script to print out some variables:

----/----
import locale
import getpass
import os

def application(environ, start_response):
    status = '200 OK'
    output = ""
    output += "User: "+getpass.getuser()+"\n"
    output += "defaultlocale: "+str(locale.getdefaultlocale())+"\n"
    output += "prefencoding: "+locale.getpreferredencoding()+"\n"
    output += "LANG: "+os.getenv('LANG')+"\n"
    output += "LC_ALL: "+str(os.getenv('LC_ALL'))+"\n"
    output += "LC_CTYPE: "+str(os.getenv('LC_CTYPE'))+"\n"
    output += "LC_MESSAGES: "+str(os.getenv('LC_MESSAGES'))+"\n"

    response_headers = [('Content-type', 'text/plain'),
                        ('Content-Length', str(len(output)))]
    start_response(status, response_headers)

    return [output]
----/----

And the output is:

----/----
User: moinuser
defaultlocale: (None, None)
prefencoding: ANSI_X3.4-1968
LANG: C
LC_ALL: None
LC_CTYPE: None
LC_MESSAGES: None
----/----

When running these lines on the console I get:

----/----
User: moinuser
defaultlocale: ('en_US', 'UTF-8')
prefencoding: UTF-8
LANG: en_US.UTF-8
LC_ALL: None
LC_CTYPE: None
LC_MESSAGES: POSIX
----/----


So obviously the locale settings are gone.

I have googled around a bit and it seems that I have to explicitly set 
the locale settings to WSGIDaemonProcess:

----/----
WSGIDaemonProcess moin user=moinuser group=moinuser processes=5 threads=10 maximum-requests=1000 umask=0007 lang='de_DE.UTF-8' locale='de_DE.UTF-8'
----/----

and everything is well then!


> I see someone else has experienced this recently, too:
> 
> https://moinmo.in/MoinMoinBugs/1.9.8NonAsciiURL-UnicodeEncodeError

I have added my solution to the Discussion paragraph on that page.

Thanks for your help!