Stripping unencodable characters from a string

[Serhiy Storchaka]

On 05.05.15 21:19, Paul Moore wrote:
> I want to write a string to an already-open file (sys.stdout, typically). However, I *don't* want encoding errors, and the string could be arbitrary Unicode (in theory). The best way I've found is
>
>      data = data.encode(file.encoding, errors='replace').decode(file.encoding)
>      file.write(data)
>
> (I'd probably use backslashreplace rather than replace, but that's a minor point).
>
> Is that the best way? The multiple re-encoding dance seems a bit clumsy, but it was the best I could think of.

There are flaws in this approach.

1) file.encoding can be None (StringIO) or absent (general file-like 
object, that implements only write()).

2) When the encoding is UTF-16, UTF-32, UTF-8-SIG, the output will 
contain superfluous byte order marks.

This is not easy problem and there is no simple solution. In particular 
cases you can create TextIOWrapper(file.buffer, 'w', 
encoding=file.encoding, errors='replace', newline=file.newlines, 
write_through=True) and write to it, but be aware of limitations.