[Baypiggies] Subprocess: Common problem/common pattern

[Glen Jarvis]

Thanks, Jeremy!

Lots of options... Plenty for me to read and absorb now :)

Cheers,



Glen

On Sat, Oct 2, 2010 at 10:27 AM, Jeremy Fishman
<jeremy.r.fishman at gmail.com>wrote:

> As Luca mentioned, there is pexpect.
>
> If you don't want an external library dependency, and the underlying
> process produces interactive output (and doesn't explicitly flush) the
> simplest solution is to open a pseudo terminal.  There are several ways to
> interact with the terminal, documented at
> http://docs.python.org/library/pty.html, but here's one way I've coded
> before:
>
> #!/usr/bin/python
> import os
> import pty
> import subprocess as sp
> import sys
>
> # spawn subprocess with input/output connected to a pseudo-terminal
> master, slave = pty.openpty()
> tstP = sp.Popen(['/usr/bin/python','foo.py'], stdin=slave, stdout=slave,
>                 stderr=sp.STDOUT, close_fds=True)
> # read an write on the terminal file descriptors
> sys.stdout.write(os.read(master, 1024))
> # we happily execute thanks to the pseudo-terminal
> os.write(master, 'bar\n')
> sys.stdout.write(os.read(master, 4096))
>
> (foo.py)
>
>  #!/usr/bin/python
> print "hello world"
> foo = raw_input('? ')
> print 'foo', foo
>
>
>  - Jeremy
>
>
> --
> Jeremy R. Fishman
> Developer @ Quantcast
> jeremy.r.fishman at gmail.com
>
> On Sat, Oct 2, 2010 at 10:14 AM, Glen Jarvis <glen at glenjarvis.com> wrote:
>
>> What is needed is the input as it's being generated.
>>
>> For example, the C-program counts to 10, with a second delay between each
>> count.
>>
>> The python program, when it calls the C program, will have a 10 second
>> delay before it sees anything. Ideally, the python program can be printing
>> the results of the C program as they are being generated.
>>
>>
>> Thanks for asking for clarification...
>>
>>
>> Cheers,
>>
>>
>>
>> Glen
>>
>> On Sat, Oct 2, 2010 at 10:05 AM, joshua kaderlan <jkaderlan at yahoo.com>wrote:
>>
>>> Reading this, it's not entirely clear what the problem is. Is it that
>>> subprocess.Popen buffers output, rather than returning it immediately the
>>> way os.system() does?
>>>
>>>
>>> *From:* Glen Jarvis <glen at glenjarvis.com>
>>> *To:* Baypiggies <baypiggies at python.org>
>>> *Sent:* Sat, October 2, 2010 9:44:05 AM
>>> *Subject:* [Baypiggies] Subprocess: Common problem/common pattern
>>>
>>> I've now seen a common problem come up several times. And, I imagine
>>> there is a common solution to this problem that I don't know.
>>>
>>> About a year ago, I wrote an automatic script that would automatically do
>>> an 'svn export' of certain branches of a tree depending upon what menu
>>> option the customer chose. When I did the svn export, I used
>>> subprocess.Popen.
>>>
>>> The pattern was similar to the following:
>>>
>>> print """This output is being buffered so I can read the version number.
>>>
>>>     ....  I'm not stuck, just busy exporting files....
>>>
>>> """
>>>
>>> .....
>>>
>>> process = subprocess.Popen(['svn', 'export', repository,
>>> repo_name], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> stdoutdata, stderrdata = process.communicate()
>>>
>>>
>>> I printed "please wait" and then printed the data given when the process
>>> was done (stdoutdata). It wasn't ideal but, it was sufficient for the time.
>>> If I were to have gone for the best fix, I would probably have learned the
>>> API for subversion to integrate directly into python.
>>>
>>> However, another BayPIGgie is having the same issue. He is using a
>>> command to start a CD burner from the command line and wants to print the
>>> output as it is being created from the command line.
>>>
>>> I can see that to solve this we wouldn't use the communicate()
>>> convenience function. A few 'hackish' ways that may solve this, but I'm
>>> looking for the common pattern that is used when other pythonista run up
>>> against this problem. I also want to ensure that I don't have a 'hack' that
>>> causes a deadlock only to discover this much later after I've implemented
>>> the pattern a few times.
>>>
>>> To help keep the conversation more focused, I've created two tiny test
>>> programs for a test case:
>>> 1) A C command line program that we have no control over changing within
>>> python, and
>>> 2) A Python program that calls that the c-program (baypiggies_demo):
>>>
>>> Compile command line so output is same as expected by Python program:
>>> gcc -pedantic baypiggies_demo.c -o baypiggies_demo
>>>
>>>
>>> ---- start of c program: File: baypiggies_demo.c ---
>>> #include <stdio.h>
>>> #include <unistd.h>
>>>
>>> int
>>> main()
>>> {
>>>     int             i = 0;
>>>
>>>     printf("Silly output for monitoring...\n");
>>>     for (i = 0; i <= 10; i++) {
>>>         printf("Counting... %d\n", i);
>>>         sleep(1);
>>>     }
>>> }
>>> --- end of c program ---
>>>
>>>
>>>
>>> --- start of python program to demonstrate. File baypiggies.py ---
>>> import subprocess
>>>
>>> print "Just waiting...."
>>>
>>> process = subprocess.Popen(['./baypiggies_demo'],
>>>                     stdout=subprocess.PIPE)
>>>
>>> stdoutdata, stderrdata = process.communicate()
>>>
>>> print "Well, NOW I get it.. :( "
>>> print stdoutdata
>>> --- end baypiggies.py --
>>>
>>>
>>>
>>> Has anyone else ran into this before? What's the classic pattern used?
>>>
>>>
>>> Thanks in advance,
>>>
>>>
>>> Glen
>>> --
>>> Whatever you can do or imagine, begin it;
>>> boldness has beauty, magic, and power in it.
>>>
>>> -- Goethe
>>>
>>>
>>>
>>
>>
>> --
>> Whatever you can do or imagine, begin it;
>> boldness has beauty, magic, and power in it.
>>
>> -- Goethe
>>
>> _______________________________________________
>> Baypiggies mailing list
>> Baypiggies at python.org
>> To change your subscription options or unsubscribe:
>> http://mail.python.org/mailman/listinfo/baypiggies
>>
>
>


-- 
Whatever you can do or imagine, begin it;
boldness has beauty, magic, and power in it.

-- Goethe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/baypiggies/attachments/20101002/4857e914/attachment-0001.html>