Multiplexing 2 streams with asyncio

Paul Moore p.f.moore at gmail.com
Fri Jan 30 13:45:09 EST 2015


I'm trying to get to grips with asyncio. I *think* it's a reasonable fit for my problem, but I'm not really sure - so if the answer is "you shouldn't be doing that, then that's fair enough :-)

What I am trying to do is, given 2 files (the stdout and stderr from a subprocess.Popen object, as it happens), I want to wait on data coming in from either file, and write it to the appropriate one of sys.stdout or sys.stderr, process the data somehow, and then go back to waiting. Sorry if that description's a little vague. Basically I'm trying to write a version of Popen.communicate() that echoes the output to the standard streams *as well* as capturing it.

This would normally be handled on Unix with something like a select loop, and with threads on Windows. But is it something I could use an asyncio event loop for? It seems like I could set up reader callbacks on the 2 streams, and then run the event loop, but I can't work out from the documentation how to do this. Can anyone help with some pointers?

The reasons I want to try this rather than use the traditional approach are:

1. To learn about asyncio, which looks really cool but I've no idea how to start with even something simple using it :-(
2. Because on Windows I'd have to use threads, whereas asyncio uses IO completion ports behind the scenes (I think) which are probably a lot more lightweight.

I know there's a Process abstraction in asyncio, but I don't want to use that directly, because I need my code to be interoperable with existing code that uses Popen.

Thanks for any help.
Paul



More information about the Python-list mailing list