Threading and consuming output from processes

[Jack Orenstein]

I asked:

>> I am developing a Python program that submits a command to each node
>> of a cluster and consumes the stdout and stderr from each. I want all
>> the processes to run in parallel, so I start a thread for each
>> node. There could be a lot of output from a node, so I have a thread
>> reading each stream, for a total of three threads per node. (I could
>> probably reduce to two threads per node by having the process thread
>> handle stdout or stderr.)

Simon Wittber said:

 > In the past, I have used the select module to manage asynchronous
 > IO operations.
 >
 > I pass the select.select function a list of file-like objects, and it
 > returns a list of file-like objects which are ready for reading and
 > writing.

Donn Cave said:

> As I see another followup has already mentioned, the classic
> "pre threads" solution to multiple I/O sources is the select(2)
> function, ...

Thanks for your replies. The streams that I need to read contain
pickled data. The select call returns files that have available input,
and I can use read(file_descriptor, max) to read some of the input
data. But then how can I convert the bytes just read into a stream for
unpickling? I somehow need to take the bytes arriving for a given file
descriptor and buffer them until the unpickler has enough data to
return a complete unpickled object.

(It would be nice to do this without copying the bytes from one place
to another, but I don't even see how do solve the problem with
copying.)

Jack