Interprocess communication woes

[Nick Craig-Wood]

Murali <murali.dhanakoti at gmail.com> wrote:
>  After some investigation, I found out that this problem had nothing to
>  do with my GUI app not getting refreshed and I was able to reproduce
>  this problem with normal python scripts. Here is one such script
> 
>  #File test.py
>  from subprocess import Popen
>  from subprocess import PIPE
>  import sys
>  if __name__ == '__main__':
>   prog = sys.argv[1]
>   proc = Popen(prog, shell = True, stdout = PIPE, bufsize = 0)
>   out = proc.stdout
>   while proc.poll() is None:
>     print out.readline()
> 
>  Run this program as follows
>  $ test.py "ping -c 10 www.google.com"
> 
>  Now, you would see the responses as if you just launched ping on the
>  command line. Now, lets look at another program. Here is a simple C++
>  program that prints numbers 1 to 10 at the passage of every second
>  (sort of a stopwatch)
> 
>  #include <stdio.h>
>  #include <stdlib.h>
>  #include <sys/time.h>
>  main ( )
>  {
>   timeval t1, t2;
>   gettimeofday(&t1, NULL);
>   int prev = -1;
>   int cur = 0;
>   while (true)
>   {
>     gettimeofday(&t2,NULL);
>     if(t2.tv_sec - t1.tv_sec > 10)
>       break;
>     else
>     {
>       cur = t2.tv_sec-t1.tv_sec;
>       if (cur != prev)
>       {
>         printf("%d\r\n",cur);
>         prev = cur;
>       }
>     }
>   }
>  }
> 
>  if you would build this program and call it lets say timer ($ g++ -o
>  timer timer.cpp)  and run it with our python script like this
> 
>  $python test.py "./timer"
> 
>  you would see that every time you run the program your results vary
>  and on top of this the stdout of the timer program gets displayed all
>  at once presumably when the timer program has completed execution.
> 
>  Why this discrepancy between the ping and timer programs? Is my
>  test.py script correct? Is there a better or a preferred method for
>  doing interprocess communication in Python.

Buffering is your problem.

If you add a fflush(stdout); after the printf(...); you'll find the
c++ program works as you expect.

It is just a fact of life of the C stdio system.  If it is connected
to a terminal then it will turn off buffering.  If it is connected
anything else (eg a pipe via subprocess) then it will buffer stuff as
you've seen.

So you can

a) modify the c++ prog to add fflush() in or use setvbuf()
b) use the pexpect module - http://pexpect.sourceforge.net/
c) use the pty module (unix only)

The pexpect module will connect to the subprogram with pseudo-ttys,
fooling the program, and the C library, into thinking that it is
speaking to a terminal and turn off buffering.  Pexpect doesn't work
on windows.

The fact that ping works is because it uses fflush() - you can see
this if you "ltrace" it.

-- 
Nick Craig-Wood <nick at craig-wood.com> -- http://www.craig-wood.com/nick