Everything you did not want to know about Unicode in Python 3

[Ian Kelly]

On Tue, May 13, 2014 at 5:19 AM, alister
<alister.nospam.ware at ntlworld.com> wrote:
> I am only an amateur python coder which is why I asked if I am missing
> something
>
> I could not see any reason to be using the shutil module if all that the
> programm is doing is opening a file, reading it & then printing it.
>
> is it python that causes the issue, the shutil module or just the OS not
> liking the data it is being sent?
>
> an explanation of why this approach is taken would be much appreciated.

No, that part is perfectly fine.  This is exactly what the shutil
module is meant for: providing shell-like operations.  Although in
this case the copyfileobj function is quite simple (have yourself a
look at the source -- it just reads from one file and writes to the
other in a loop), in general the Pythonic thing is to avoid
reinventing the wheel.

And since it's so simple, it shouldn't be hard to see that the use of
the shutil module has nothing to do with the Unicode woes here.  The
crux of the issue is that a general-purpose command like cat typically
can't know the encoding of its input and can't assume anything about
it. In fact, there may not even be an encoding; cat can be used with
binary data.  The only non-destructive approach then is to copy the
binary data straight from the source to the destination with no
decoding steps at all, and trust the user to ensure that the
destination will be able to accommodate the source encoding.  Because
Python 3 presents stdin and stdout as text streams however, it makes
them more difficult to use with binary data, which is why Armin sets
up all that extra code to make sure his file objects are binary.