Sending binary pickled data through TCP

[MRAB]

David Hirschfield wrote:
> I have a pair of programs which trade python data back and forth by
> pickling up lists of objects on one side (using
> pickle.HIGHEST_PROTOCOL), and sending that data over a TCP socket
> connection to the receiver, who unpickles the data and uses it.
>
> So far this has been working fine, but I now need a way of separating
> multiple chunks of pickled binary data in the stream being sent back and
> forth.
>
> Questions:
>
> Is it safe to do what I'm doing? I didn't think there was anything
> fundamentally wrong with sending binary pickled data, especially in the
> closed, safe environment these programs operate under...but maybe I'm
> making a poor assumption?
>
> I was going to separate the chunks of pickled data with some well-formed
> string, but couldn't that string potentially randomly appear in the
> pickled data? Do I just pick an extremely
> unlikely-to-be-randomly-generated string as the separator? Is there some
> string that will definitely NEVER show up in pickled binary data?
>
> I thought about base64 encoding the data, and then decoding on the
> opposite side (like what xmlrpclib does), but that turns out to be a
> very expensive operation, which I want to avoid, speed is of the essence
> in this situation.
>
> Is there a reliable way to determine the byte count of some pickled
> binary data? Can I rely on len(<pickled data>) == bytes?
>
Instead of communicating directly with the TCP socket, you could talk
to it via an object which precedes each chunk with a byte count, and if
you're working with multiple streams of picked data, then each chunk
could also have an identifier which specified which stream it belonged
to.