Python 3.1.1 bytes decode with replace bug

[Dave Angel]

Joe wrote:
>> For the reason BK explained, the important difference is that I ran in
>> the IDLE shell, which handles screen printing of unicode better ;-)
>>     
>
> Something still does not seem right here to me.
>
> In the example above the bytes were decoded to 'UTF-8' with the
>   
*nope*  you're decoding FROM utf-8 to unicode.
> replace option so any characters that were not UTF-8 were replaced and
> the resulting string is '\ufffdabc' as BK explained.  I understand
> that the replace worked.
>
> Now consider this:
>
> Python 3.1.1 (r311:74483, Aug 17 2009, 16:45:59) [MSC v.1500 64 bit
> (AMD64)] on
> win32
> Type "help", "copyright", "credits" or "license" for more information.
>   
>>>> s = '\ufffdabc'
>>>> print(s)
>>>>         
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "p:\SW64\Python.3.1.1\lib\encodings\cp437.py", line 19, in
> encode
>     return codecs.charmap_encode(input,self.errors,encoding_map)[0]
> UnicodeEncodeError: 'charmap' codec can't encode character '\ufffd' in
> position
> 0: character maps to <undefined>
>   
>>>> import sys
>>>> sys.getdefaultencoding()
>>>>         
> 'utf-8'
>
> This too fails for the exact same reason (and doesn't invole decode).
>
> In the original example I decoded to UTF-8 and in this example the
> default encoding is UTF-8 so why is cp437 being used?
>
> Thanks in advance for your assistance!
>
>
>   
Benjamin had it right, but you still don't understand what he said.

The problem in your original example, and in the current one, is not in 
decode(), but in encode(), which is implicitly called by print(), when 
needed to convert from Unicode to some byte format of the console.  Take 
your original example:

>>>>>  b'\x80abc'.decode('utf-8', 'replace')


The decode() is explicit, and converts *FROM* utf8 string to a unicode 
one.  But since you're running in a debugger, there's an implicit print, 
which is converting unicode into whatever your default console encoding 
is.  That calls encode() (or one of its variants,  charmap_encode(), on 
the unicode string.  There is no relationship between the two steps.

In your current example, you're explicitly doing the print(), but still 
have the same implicit encoding to cp437, which gets the equivalent 
error.  That's the encoding that your Python 3.x is choosing for the 
stdout console, based on country-specific Windows settings.  In the US, 
that implicit encoding is ASCII.  I don't know how to override it 
generically, but I know it's possible to replace stdout with a wrapper 
that does your preferred encoding.  You probably want to keep cp437, but 
change the error handling to ignore.  Or if this is a one-time problem, 
I suspect you could do the encoding manually, to a byte array, then 
print that.

DaveA