Direct interaction with subprocess - the curse of blocking I/O

[Pascal Chambon]

Thank you all for the comments

>you might want something like Expect.

Yes "Expect" deals with such things, unfortunately it's posix only (due to the PTY module requirement...); whereas I'd like to find generic ways (i.e at least windows/linux/mac recipes)

> The latter is inherently tricky (which is why C's popen() lets you connect
> to stdin or stdout but not both). You have to use either multiple threads,
> select/poll, or non-blocking I/O.
>
> If the child's output is to the console, it should presumably be the
> former, i.e. piping stdin but allowing the child to inherit stdout, in
> which case, where's the problem? Or are you piping its stdout via Python
> for the hell of it
It's well a too-way piping that I'd like to get ; but actually, even  a 
single-way piping is error-prone : if I try to send data to the child's 
stdin, and this child never wants to receive that data (for whatever 
reason), the parent thread will be forever stuck.
I can use multiple threads, but it doesn't fully solve the problem, 
because having even a single thread stuck might prevent the process from 
terminating, on some operating systems... I'd need a way to deblock I/O 
blocking threads (but even closing the pipe isn't sure to do it properly 
- the closing might be delayed by the I/O operations wating).


 >I would guess the architectural differences are so great that an attempt
 >to "do something simple" is going to involve architecture-specific code.
 >and I personally wouldn't have it any other way.  Simulating a shell
 >with hooks on its I/O should be so complicated that a "script kiddie"
 >has trouble writing a Trojan.

I'm not sure I understand the security hole there - if a script kiddie 
manages to make you execute his program, he doesn't need complex I/O 
redirections to harm you, simply destroying files randomly on your hard 
drive should fit his needs shouldn't it. :?


> > I met the issue : select() works only on windows ...
>   

>No it doesn't. It works only on sockets on Windows, on Unix/Linux it works 
>with all file descriptors <http://docs.python.org/library/select.html>.


Indeed, I mixed up my words there... >_<


 >If you are willing to have a wxPython dependency, wx.Execute handles 
non-blocking i/o with processes on all supported platforms more 
discussion of python's blocking i/o issues here:
 >  
http://groups.google.com/group/comp.lang.python/browse_thread/thread/a037349e18f99630/60886b8beb55cfbc?q=#60886b8beb55cfbc

Very interesting link, that's exactly the problem... and it seems no 
obvious solution comes out of it, except, of course, going down to the 
OS' internals...
File support is really weak in python I feel (stdio is a little 
outdated...), it'd need a complete blocking/non-blocking, 
locking/non-locking stream API straight in the stdlib...
I'm already working on locking classes, I'll explore the opportunity for 
a higher level "file" object too whenever possible (a new filesystem api 
is in progress here at europython2009, it might be the occasion).


regards,
Pascal



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20090702/7d97340c/attachment-0001.html>