[Python-Dev] [Python-3000] New proposition for Python3 bytes filename issue

[M.-A. Lemburg]

On 2008-09-30 18:46, Guido van Rossum wrote:
> On Tue, Sep 30, 2008 at 8:20 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>> In the end, I think it's better not to be clever and just return
>> the filenames that cannot be decoded as bytes objects in os.listdir().
> 
> Unfortunately that's going to break most code that is using
> os.listdir(), so it's hardly an improved experience.

Right, but this also signals a problem to the application and the
application is in the best position to determine a proper
work-around.

>> Passing those to open() will then open the files as expected, in most
>> other cases the application will have to provide explicit conversions
>> in whatever way best fits the application.
> 
> In most cases the app will try to concatenate a pathname given as a
> string and then it will fail.

True, and that's the right thing to do in those cases.
The application will have to deal with the problem, e.g. convert
the path to bytes and retry the joining, or convert the bytes string
to Latin-1 and then convert the result back to bytes (using Latin-1)
for passing it to open() (which will of course only work if there are
no non-Latin-1 characters in the path dir), or apply a different
filename encoding based on the path and then retry to convert the
bytes filename into Unicode, or ask the user what to do, etc.

There are many possibilities to solve the problem, apply a work-around,
or inform the user of ways to correct it.

>> Also note that os.listdir() isn't the only source of filesnames. You
>> often read them from a file, a database, some socket, etc, so letting
>> the application decide what to do is not asking too much, IMHO.
> 
> In all those cases, the code that reads them is responsible for
> picking an encoding or relying on a default encoding, and the
> resulting filenames are always expressed as text, not bytes. I don't
> think it's the same at all.

What I was trying to say is that you run into the same problem
in other places as well. Trying to have os.listdir() implement
some strategy is not going to solve the problem at large.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 30 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611