[Baypiggies] Subprocess: Common problem/common pattern

[Glen Jarvis]

Thanks Minesh... :)

I especially like your third choice. I didn't know this recipe was in the
Cookbook. Exactly what I was asking for. I've yanked the cookbook off the
shelf (and send my high praise to Alex for documenting this in advance as I
knew he know the answer to this)... I'll try to find the recipe in the
cookbook. I've also noticed a substantial amount of code sent by James
Tatum:

http://code.activestate.com/recipes/576957-asynchronous-subprocess-using-asyncore/

Now, I suddenly have a lot to review :)

Cheers,


Glen

On Sat, Oct 2, 2010 at 10:23 AM, Minesh B. Amin <mamin at mbasciences.com>wrote:

>  Hi Glen,
>
> The are three general ways:
>
> + commands.getstatusoutput(...)
>      when you:
>         * do not mind waiting for the command to finish
>         * are interested in the stdout+stderr at the end (when the command
> is finished)
>
> + pexpect < <http://www.noah.org/wiki/Pexpect#Overview>
> http://www.noah.org/wiki/Pexpect#Overview >
>      when you:
>         * want to interact with the launched command; e.g. enter login,
> password, answer Y/N
>
>      Thanks to pexpect, most of these interactions may be automated; i.e.
>      the python script may take over responsibility of processing "login",
> "password" requests.
>
>      Any interaction not automated may be passed on to the "human".
>
> + The solution "Capturing the Output and Error Streams from a
>      Unix Shell Command" from the book "Python Cookbook"
>
>     when you:
>       * want to simply consume the stderr/stdout of the launched command in
> real-time
>
> Hope this helps.
>
> Cheers!
> Minesh
>
>
> On Sat, 2010-10-02 at 09:44 -0700, Glen Jarvis wrote:
>
> I've now seen a common problem come up several times. And, I imagine there
> is a common solution to this problem that I don't know.
>
>
>
>  About a year ago, I wrote an automatic script that would automatically do
> an 'svn export' of certain branches of a tree depending upon what menu
> option the customer chose. When I did the svn export, I used
> subprocess.Popen.
>
>
>
>  The pattern was similar to the following:
>
>
>
>  print """This output is being buffered so I can read the version number.
>
>      ....  I'm not stuck, just busy exporting files....
>
> """
>
> .....
>
> process = subprocess.Popen(['svn', 'export', repository,
> repo_name], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>
>
>  stdoutdata, stderrdata = process.communicate()
>
>
>
>
>
>  I printed "please wait" and then printed the data given when the process
> was done (stdoutdata). It wasn't ideal but, it was sufficient for the time.
> If I were to have gone for the best fix, I would probably have learned the
> API for subversion to integrate directly into python.
>
>
>
>  However, another BayPIGgie is having the same issue. He is using a
> command to start a CD burner from the command line and wants to print the
> output as it is being created from the command line.
>
>
>
>  I can see that to solve this we wouldn't use the communicate()
> convenience function. A few 'hackish' ways that may solve this, but I'm
> looking for the common pattern that is used when other pythonista run up
> against this problem. I also want to ensure that I don't have a 'hack' that
> causes a deadlock only to discover this much later after I've implemented
> the pattern a few times.
>
>
>
>  To help keep the conversation more focused, I've created two tiny test
> programs for a test case:
>
>  1) A C command line program that we have no control over changing within
> python, and
>
>  2) A Python program that calls that the c-program (baypiggies_demo):
>
>
>
>  Compile command line so output is same as expected by Python program:
>
>  gcc -pedantic baypiggies_demo.c -o baypiggies_demo
>
>
>
>
>  ---- start of c program: File: baypiggies_demo.c ---
>
>  #include <stdio.h>
>
>  #include <unistd.h>
>
>
>
>  int
>
>  main()
>
>  {
>
>      int             i = 0;
>
>
>
>      printf("Silly output for monitoring...\n");
>
>      for (i = 0; i <= 10; i++) {
>
>          printf("Counting... %d\n", i);
>
>          sleep(1);
>
>      }
>
>  }
>
>  --- end of c program ---
>
>
>
>
>
>
>
>  --- start of python program to demonstrate. File baypiggies.py ---
>
>  import subprocess
>
>
>
>  print "Just waiting...."
>
>
>
>  process = subprocess.Popen(['./baypiggies_demo'],
>
>                      stdout=subprocess.PIPE)
>
>
>
>  stdoutdata, stderrdata = process.communicate()
>
>
>
>  print "Well, NOW I get it.. :( "
>
>  print stdoutdata
>
>  --- end baypiggies.py --
>
>
>
>
>
>
>
>  Has anyone else ran into this before? What's the classic pattern used?
>
>
>
>
>
>  Thanks in advance,
>
>
>
>
>
>  Glen
>
>  --
> Whatever you can do or imagine, begin it;
> boldness has beauty, magic, and power in it.
>
> -- Goethe
>
>  _______________________________________________
> Baypiggies mailing list
> Baypiggies at python.org
> To change your subscription options or unsubscribe:http://mail.python.org/mailman/listinfo/baypiggies
>
>
>   --
> MBA Sciences, Inc (www.mbasciences.com)
> Tel #: 650-938-4306
> Email: mamin at mbasciences.com
>
>


-- 
Whatever you can do or imagine, begin it;
boldness has beauty, magic, and power in it.

-- Goethe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/baypiggies/attachments/20101002/6324b149/attachment.html>