[Chicago] capturing output from subprocesses

[Noel Thomas Taylor]

Hi Jess,

It's great to be making progress on this problem. I tried the code you 
sent and it also worked for me under Linux (but see below!)

You wrote that you weren't sure what I was getting at with the "timeout" 
concept. The idea is that the child process should be producing output 
fairly regularly. If some number of seconds go by during which it does not 
produce output, the parent should decide that the child is stuck and 
should kill it. Adding the timeout to the select call makes this possible. 
I don't know a better way.

A couple of questions: I notice you don't import the pty module anywhere, 
nor do you use os.forkpty(). I guess just opening the file in 
"/dev/pty..." makes it a pseudo-terminal? Isn't this a little dangerous? I 
would have thought the pty module would make things safer all around, but 
maybe it's not necessary at all?

Also, the code you sent unfortunately doesn't work under Irix. On that OS, 
the child's termination signal will never be picked up, and it will run 
indefinitely. The timeout in the select call is ineffective.

I mentioned that I'd been using Noah Spurrier's pexpect module, which 
contains a lot of the functionality I'm looking for. Originally, pexpect 
had the exact same problem under Irix that your code does, but I got in 
touch with Noah and he made a hack for it.

In his email to me he wrote:
   "Irix is a weird platform. It seems to have no reliable way to test if
    a child process is alive or not without blocking... I managed to find
    a hack for this, but it adds a 2 second timeout to every call to
    read_nonblocking(). That means you can't have a timeout less than 3
    seconds. That's probably not a big deal in general."

I orignally sent this email two days ago with a copy of "pexpect.py" 
attached to it, but it got held up by the list administrator for being too 
large, so rather than wait to see if it goes through, I'm canceling that 
original mail and sending this one instead, which has a link to the code 
instead of the code itself:

   http://people.cs.uchicago.edu/~nttaylor/pexpect.py

If you have time, please visit the above site and take a look. There's a 
good bit of code there, but I wonder if you might look over the part that 
specifically addresses the Irix hack (just search for the word 'irix' in 
the source). My ultimate goal is to understand exactly what's going on, 
and I think I stand a better chance with your code, which is much shorter 
and more focused on this one specific problem, than with pexpect, which is 
meant to be a much more general. If I could eventually make my own program 
work with a short piece of code that I throroughly understand, that is 
preferable to using the pexpect module, which is more of a black box.

And the short-term goal is to incorporate Noah's Irix hack into the code 
that you sent me. What do you think? I'm working on getting you a guest 
account on our Irix box for testing, in case you don't have one available.

with thanks for all your help,

Noel Taylor


> On Fri, 11 Nov 2005, Jess Balint wrote:
>
>> Ok, I've attached a pty version. I've tested it on Linux and Solaris
>> and it seems to work well. I left your select timeout in there for
>> illustration, but I'm still not sure what you were getting at with
>> that. I set the select timeout to '3' and the ticker.py to sleep the
>> number of seconds as the incrementer. This way you see a few 'ticks'
>> in the output before the parent kills it. Let me know if this one
>> works ok and makes sense.
>> 
>> Jess
>> 
>> On 11/11/05, Jess Balint <jbalint at gmail.com> wrote:
>>>> The child process in the real application I want to use this for is
>>>> written in fortran, so I can't use the techniques you mentioned.
>>>> 
>>>> I had already sought out second and third opinions on this, so now I am
>>>> really convinced that there is no way to get the effect I want without
>>>> using pseudo-terminals.
>>> 
>>> Yes, I think that's the only thing you have influence on at this
>>> point. Using forkpty() and connecting it to the pipe might work.
>>> Another option might be to use an LD_PRELOAD with an init routine that
>>> calls setbuf(). I am trying to find a simple, non-intrusive way to do
>>> this, I will let you know if I come up with something.
>>> 
>>>>> The non-blocking io is not relevant to any of this.
>>>> 
>>>> Too bad. If I can ask you one more thing: what is the proper purpose of
>>>> using fcntl to set a file descriptor to non-blocking IO if not to achieve
>>>> the behavior I'm looking for in this problem?
>>> 
>>> Non-blocking IO is something that allows a read() or write() call to
>>> return when a file descriptor is not ready. So if you are reading from
>>> a network connection and nothing has come through, a normal read()
>>> call would block (and wait for something). A non-blocking attribute
>>> added to the socket will allow a read() call to return so you can do
>>> other stuff until it is ready.
>>> 
>>> Jess
>>> 
>