How to perform a nonblocking read from a process

[sturlamolden]

On Jun 4, 8:45 am, rdab... at gmail.com wrote:

> I've to admit I'm a newbie to this kind of programming...
> what if I have to run thousands of these commands...it doesn't make
> sense to create
> thousands of threads..
> Is there a way that above mentioned piece of code be made to worked...


Are you planning on doing thousands of simultaneous asynchronous I/O
calls? How many processes are you communicating with? Take a look at
the twisted framwork (twistedmatrix.org) or perhaps i/o completion
ports on Windows (e.g. win32file.CreateIoCompletionPort in PyWin32).
But if you need that king of scalability, I suggest you start by
reconsidering the design.

If you are communicating with only one process, you don't need
thousands of threads, just one. Keep the thread idle until you need it
again. Here is a simple worker thread that allows you to do multiple,
different function calls in a background thread (you can shut it down
by passing None to setTask).


import threading
import Queue

class WorkerThread(threading.Thread):

    def __init__(self):
        self.taskQueue = Queue.Queue(0)
        self.resQueue = Queue.Queue(0)
        self.setDaemon(True)

    def run(self):
        while 1:
            fun, args, kwarg = self.taskQueue.get()
            if (fun is None): break
            self.resQueue.put(fun(*args,**kwarg))

    def setTask(self, fun, *args, **kwarg):
        self.taskQueue.set((fun, args, kwarg))

    def getResult(self):
        return self.resQueue.get()

    def probeResult(self):
        return not self.resQueue.empty()


Rewrite of your examples:

from subprocess import *
p2 = Popen('python',stdin=PIPE,stdout=PIPE,universal_newlines=True)
worker = WorkerThread()
for i in range(10):
    worker.setTask(p2.stdin.write, 'print 10'+'\n')
    worker.setTask(p2.stdout.readline)
    worker.getResult() # wait for write to finish
    o,e = worker.getResult() # wait for read to finish
    print o,e

from subprocess import *
p2 = Popen('python',stdin=PIPE,stdout=PIPE,universal_newlines=True)
writer = WorkerThread()
reader = WorkerThread()
for i in range(10):
    writer.setTask(p2.stdin.write, 'print 10'+'\n')
    reader.setTask(p2.stdout.readline)
    o,e = reader.getResult() # wait for read to finish
    print o,e