[Python-ideas] Add encoding attribute to bytes

[Terry Reedy]

Nick Coghlan wrote:
> Terry Reedy wrote:
>>> As for the usefulness, I often have to work with proprietary
>>> communication
>>> protocols between computer and devices, and there the bytes have no
>>> encoding
>>> whatsoever
>> Random bits? It seems to me that protocol means some sort of encoding,
>> formatting, or structuring, some sort of agreed on interpretation, even
>> if private.
> 
> This is true, but the encoding scheme *isn't* a property of the binary
> data in and of itself. It's metadata about it that guides the
> application as to how the stream should be interpreted.
> 
> For a lot of the things I've done in the past, I haven't cared at all
> about the encoding of binary data - I've just been schlepping bits from
> point A to point B and back without caring what they actually *meant*.
> Other times I didn't have to guess or pass any metadata around because
> the comms port was hardwired to a particular device that only knew one
> way of communicating - the definition of the protocol was implicit in
> the implementation of the interface software.
> 
> In fact, one of the key features typically desired in a communications
> protocol is for it to be content neutral: you push binary data in one
> end and get the same binary data out of the other end. Peer applications
> using the channel to communicate with each other don't need to care what
> the channel is doing with the data, but equally importantly, the
> software implementing the comms channel doesn't need to know how to
> interpret the bits it is transporting*.
> 
> For other applications, the Unicode encoding might be important to know.
> Some will care more about the MIME type, or use some other defined
> binary encoding (what is the Unicode encoding of an sqlite or bsddb
> database file?). Other applications may be interested in a proprietary
> binary format that is formally defined solely by the code that knows how
> to read and write it.
> 
> Can bytes be used to store encoded Unicode data? Sure they can. But they
> can be used for a whole host of other things as well, so burdening them
> with an attribute that is occasional helpful, but more often dead weight
> or even outright misleading would be a mistake.
> 
> Cheers,
> Nick.
> 
> * Sometimes a bit more coupling makes sense when there are engineering
> advantages to be had, but this is usually an application specific thing
> (e.g. IP has a protocol field that identifies different application
> layer protocols such as TCP, UDP and ESP which have different network
> performance expectations, This allows IP network routers to apply
> different rules without having to peek inside the payload of each IP packet)

Your experience has been different from mine. Thanks for the exposition.
I can see why you prefer metadata to either be in the stream itself or 
as part of a wrapper object.

Terry Jan Reedy