Threading and consuming output from processes

[Donn Cave]

Quoth Jack Orenstein <jao at geophile.com>:
[ ... re alternatives to threads ]
| Thanks for your replies. The streams that I need to read contain
| pickled data. The select call returns files that have available input,
| and I can use read(file_descriptor, max) to read some of the input
| data. But then how can I convert the bytes just read into a stream for
| unpickling? I somehow need to take the bytes arriving for a given file
| descriptor and buffer them until the unpickler has enough data to
| return a complete unpickled object.
|
| (It would be nice to do this without copying the bytes from one place
| to another, but I don't even see how do solve the problem with
| copying.)

Note that the file object copies bytes from one place to another,
via C library stdio.  If we could only see the data in those
stdio buffers, it would be possible to use file objects with
select() in more applications.  (Though not with pickle.)  Since
input very commonly needs to be buffered for various reasons, we
end up writing our own buffer code, all because stdio has no
standard function that tells you how much data is in a buffer.

But unpickling consumes an I/O stream, as you observe, so as a
network data protocol by itself, it's unsuitable for use with
select.  I think the only option would be a packet protocol -
a count field followed by the indicated amount of pickle data.
I suppose I would copy the received data into a StringIO object,
and unpickle that when all the data has been received.

Incidentally, I think I read here yesterday, someone held a book
about Python programming up to some ridicule for suggesting that
pickles would be a good way to send data around on the network.
The problem with this was supposed to have something to do with
"overloading".  I have no idea what he was talking about, but you
might be interested in this issue.

	Donn Cave, donn at drizzle.com