os.walk the apostrophe and unicode

[Peter Otten]

Rod Person wrote:

> Ok...so after reading all the replies in the thread, I thought I would
> be easier to send a general reply and include some links to screenshots.
> 
> As Peter mention, the logic thing to do would be to fix the file name
> to what I actually thought it was and if this was for work that
> probably what I would have done, but since I want to understand what's
> going on I decided to waste time on that.
> 
> I have to admit, I didn't think the file system was utf-8 as seeing what
> looked to be an apostrophe sent me down the road of why is this
> apostrophe screwed up instead of "ah this must be unicode".
> 
> But doing a simple ls of that directory show it is unicode but the
> replacement of the offending character.
> 
> http://rodperson.com/graphics/uc/ls.png

Have you set LANG to something that implies ASCII?

$ touch Todd’s ähnlich üblich löblich
$ ls
ähnlich  löblich  Todd’s  üblich
$ LANG=C ls
Todd???s  l??blich  ??hnlich  ??blich
$ python3 -c 'import os; print(os.listdir())'
['Todd’s', 'üblich', 'ähnlich', 'löblich']
$ LANG=C python3 -c 'import os; print(os.listdir())'
['Todd\udce2\udc80\udc99s', '\udcc3\udcbcblich', '\udcc3\udca4hnlich', 
'l\udcc3\udcb6blich']
$ LANG=en_US.utf-8 python3 -c 'import os; print(os.listdir())'
['Todd’s', 'üblich', 'ähnlich', 'löblich']

For file names Python resorts to surrogates whenever a byte does not 
translate into a character in the advertised encoding.
 
> I am in fact using Python 3.5. I may be lacking in unicode skills but I
> do have the sense enough to know the version of Python I am invoking.

I've made so many "stupid errors" myself that I always consider them first 
;)

> So I included this screenshot of that so the version of Python and the
> files list returned by os.walk
> 
> http://rodperson.com/graphics/uc/files.png
> 
> So the fact that it shows as a string and not bytes in the debugger was
> throwing me for a loop, in my log section I was trying to determine if
> it was unicode decode it...if not don't do anything which wasn't working
> 
> http://rodperson.com/graphics/uc/log_section.png
> 
> 
> 
> 
> On Sun, 25 Jun 2017 10:47:18 +0200
> Peter Otten <__peter__ at web.de> wrote:
> 
>> Steve D'Aprano wrote:
>> 
>> > On Sun, 25 Jun 2017 04:57 pm, Peter Otten wrote:
>> 
>> >> if everything worked correctly? Though I don't understand why the
>> >> OP doesn't see
>> >> 
>> >> '06 - Toddâ\x80\x99s Song (Post-Spiderland Song in Progress).flac'
>> >> 
>> >> which is the repr() that I get.
>> > 
>> > That's mojibake and is always wrong :-)
>> 
>> Yes, that's my very point.
>> 
>> > I'm not sure how you got that.
>> 
>> I took the OP's string at face value and pasted it into the
>> interpreter:
>> 
>> # python 3.4
>> >>> '06 - Todd\xe2\x80\x99s Song (Post-Spiderland Song in
>> >>> Progress).flac'
>> '06 - Toddâ\x80\x99s Song (Post-Spiderland Song in Progress).flac'
>> 
>> > Something to do with an accidental decode to Latin-1?
>> 
>> If the above filename is the only one or one of a few that seem
>> broken, and other non-ascii filenames look OK the OP's
>> toolchain/filesystem may work correctly and the odd name might have
>> been produced elsewhere, e. g. by copying an already messed-up
>> freedb.org entry.
>> 
>> [Heureka]
>> 
>> However, the most likely explanation is that the filename is correct
>> and that the OP is not using Python 3 as he claims but Python 2.
>> 
>> Yes, it took that long for me to realise ;) Python 2 is slowly
>> sinking into oblivion...
>> 
> 
> 
>