[Baypiggies] Subprocess: Common problem/common pattern

[Glen Jarvis]

What is needed is the input as it's being generated.

For example, the C-program counts to 10, with a second delay between each
count.

The python program, when it calls the C program, will have a 10 second delay
before it sees anything. Ideally, the python program can be printing the
results of the C program as they are being generated.


Thanks for asking for clarification...


Cheers,



Glen

On Sat, Oct 2, 2010 at 10:05 AM, joshua kaderlan <jkaderlan at yahoo.com>wrote:

> Reading this, it's not entirely clear what the problem is. Is it that
> subprocess.Popen buffers output, rather than returning it immediately the
> way os.system() does?
>
>
> *From:* Glen Jarvis <glen at glenjarvis.com>
> *To:* Baypiggies <baypiggies at python.org>
> *Sent:* Sat, October 2, 2010 9:44:05 AM
> *Subject:* [Baypiggies] Subprocess: Common problem/common pattern
>
> I've now seen a common problem come up several times. And, I imagine there
> is a common solution to this problem that I don't know.
>
> About a year ago, I wrote an automatic script that would automatically do
> an 'svn export' of certain branches of a tree depending upon what menu
> option the customer chose. When I did the svn export, I used
> subprocess.Popen.
>
> The pattern was similar to the following:
>
> print """This output is being buffered so I can read the version number.
>
>     ....  I'm not stuck, just busy exporting files....
>
> """
>
> .....
>
> process = subprocess.Popen(['svn', 'export', repository,
> repo_name], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
> stdoutdata, stderrdata = process.communicate()
>
>
> I printed "please wait" and then printed the data given when the process
> was done (stdoutdata). It wasn't ideal but, it was sufficient for the time.
> If I were to have gone for the best fix, I would probably have learned the
> API for subversion to integrate directly into python.
>
> However, another BayPIGgie is having the same issue. He is using a command
> to start a CD burner from the command line and wants to print the output as
> it is being created from the command line.
>
> I can see that to solve this we wouldn't use the communicate() convenience
> function. A few 'hackish' ways that may solve this, but I'm looking for the
> common pattern that is used when other pythonista run up against this
> problem. I also want to ensure that I don't have a 'hack' that causes a
> deadlock only to discover this much later after I've implemented the pattern
> a few times.
>
> To help keep the conversation more focused, I've created two tiny test
> programs for a test case:
> 1) A C command line program that we have no control over changing within
> python, and
> 2) A Python program that calls that the c-program (baypiggies_demo):
>
> Compile command line so output is same as expected by Python program:
> gcc -pedantic baypiggies_demo.c -o baypiggies_demo
>
>
> ---- start of c program: File: baypiggies_demo.c ---
> #include <stdio.h>
> #include <unistd.h>
>
> int
> main()
> {
>     int             i = 0;
>
>     printf("Silly output for monitoring...\n");
>     for (i = 0; i <= 10; i++) {
>         printf("Counting... %d\n", i);
>         sleep(1);
>     }
> }
> --- end of c program ---
>
>
>
> --- start of python program to demonstrate. File baypiggies.py ---
> import subprocess
>
> print "Just waiting...."
>
> process = subprocess.Popen(['./baypiggies_demo'],
>                     stdout=subprocess.PIPE)
>
> stdoutdata, stderrdata = process.communicate()
>
> print "Well, NOW I get it.. :( "
> print stdoutdata
> --- end baypiggies.py --
>
>
>
> Has anyone else ran into this before? What's the classic pattern used?
>
>
> Thanks in advance,
>
>
> Glen
> --
> Whatever you can do or imagine, begin it;
> boldness has beauty, magic, and power in it.
>
> -- Goethe
>
>
>


-- 
Whatever you can do or imagine, begin it;
boldness has beauty, magic, and power in it.

-- Goethe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/baypiggies/attachments/20101002/41862674/attachment-0001.html>