Fwd: Lossless bulletproof conversion to unicode (backslashing)

[anatoly techtonik]

On Wed, May 27, 2015 at 2:47 PM, anatoly techtonik <techtonik at gmail.com> wrote:
> On Wed, May 27, 2015 at 2:35 PM, Laura Creighton <lac at openend.se> wrote:
>> I am missing something.  Why do you need unicode at all?  Why can you
>> not just keep your binary data as binary data?
>
> Good question. From the SCons code I see that we need unicode, because
> we switched to io.StringIO which is advertised as the future (and Python 3
> way of doing things, because Python 3 doesn't have non-unicode StringIO).
>
> A really deep and exhaustive answer.
> advertisement (first link on StringIO vs io.StringIO):
> http://stackoverflow.com/questions/3410309/what-is-the-difference-between-stringio-and-io-stringio-in-python2-7
> peaceful details
> https://bitbucket.org/scons/scons/commits/05d5af305a5d
> gory consequences
> https://bitbucket.org/scons/scons/pull-request/235/fix-tree-all-print-when-build-tree
>
>> I feel like I must be missing something obvious here ...
>
> Not that obvious as it appears.

And the short answer is that we need unicode because we are printing this
information to the stdout, and stdout is opened in text mode at least on
Windows, and without explicit conversion, Python will try to decode stuff
as being `ascii` and fail anyway.
-- 
anatoly t.