[IPython-dev] ERROR can not use fork(), exec(), dup2() to communicate with child process

[Andy Davidson]

Hi MinRK

I am new to Python system programming. In my first experiments I use
subprocess. Reading the document it appears subprocess it great for a
setting up traditional unix style pipe like for example calling ³ls²

In the notebook I aussem
! Uname ­a

Works using subprocess.

The subprocess  documentation suggests that you need to call communicate()
and that this function will not return until the child sends EOF or
terminates. My child will never exit or send EOF. it will keep sending me
real time data. I want to process the data one line at a time. Here is my
psudo code

While (True):
Data = readALineFromChild() # this should block until data becomes available
Graph(data)



Thanks

Andy


> 
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: 
> http://mail.scipy.org/pipermail/ipython-dev/attachments/20141002/aefaa3fc/atta
> chment-0001.html 
> 
> ------------------------------
> 
> Message: 3
> Date: Thu, 2 Oct 2014 17:19:18 -0700
> From: MinRK <benjaminrk at gmail.com>
> Subject: Re: [IPython-dev] ERROR can not use fork(), exec(), dup2() to
> communicate with child process
> To: IPython developers list <ipython-dev at scipy.org>
> Message-ID:
> <CAHNn8BXxCej6VL4XkjguOSXtgnxpsNCJXHhKGNLSBoYbiu48-g at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
> 
> why use execlp instead of subprocess.Popen, which allows capturing stdout,
> stderr, etc.?
> ?
> 
> On Thu, Oct 2, 2014 at 4:17 PM, Andy Davidson <Andy at santacruzintegration.com
>>  wrote:
> 
>>  Hi
>> 
>> 
>>  I want to write a notebook that receives data from another process. Bellow
>>  is my python code. It forks a child process and reads the data from the
>>  child using a pipe. This code works if I run it in python. When I try to
>>  execute from iPython notebook I get the following error. Any idea what the
>>  problems is ?
>> 
>> 
>>  If pipes do not work, can I use a socket? The other process would send me
>>  data by writing, my notebook needs behave like a server.
>> 
>> 
>>  Any comments or suggestions would be greatly appreciated.
>> 
>> 
>>  Thanks
>> 
>> 
>>  Andy
>> 
>> 
>>  P.s. I am running headless on a remote machine and use an ssh tunnel to
>>  run the notebook from my local browser. My other notebooks works fine
>> 
>> 
>> 
>>  [ec2-user at ip-172-31-14-107 ~]$ IPYTHON_OPTS="notebook --pylab inline
>>  --no-browser --port=7000" /root/spark/bin/pyspark
>> 
>>  /root/spark/conf/spark-env.sh: line 21: ulimit: open files: cannot modify
>>  limit: Operation not permitted
>> 
>>  [NotebookApp] Using existing profile dir:
>>  u'/home/ec2-user/.ipython/profile_default'
>> 
>>  [NotebookApp] Serving notebooks from /home/ec2-user
>> 
>>  [NotebookApp] The IPython Notebook is running at: http://127.0.0.1:7000/
>> 
>>  [NotebookApp] Use Control-C to stop this server and shut down all kernels.
>> 
>>  [NotebookApp] Using MathJax from CDN:
>>  http://cdn.mathjax.org/mathjax/latest/MathJax.js
>> 
>>  [NotebookApp] Kernel started: 71f07d5c-2db1-465f-9d86-da29f8711a0b
>> 
>>  [NotebookApp] Connecting to: tcp://127.0.0.1:54119
>> 
>>  [NotebookApp] Connecting to: tcp://127.0.0.1:50649
>> 
>>  [NotebookApp] Connecting to: tcp://127.0.0.1:35516
>> 
>>  [IPKernelApp] To connect another client to this kernel, use:
>> 
>>  [IPKernelApp] --existing kernel-71f07d5c-2db1-465f-9d86-da29f8711a0b.json
>> 
>>  Assertion failed: ok (bundled/zeromq/src/mailbox.cpp:79)
>> 
>>  [NotebookApp] Kernel 71f07d5c-2db1-465f-9d86-da29f8711a0b failed to
>>  respond to heartbeat
>> 
>> 
>> 
>>  import sys
>>  import os
>> 
>>  def child(pipeout):
>>      os.dup2(pipeout, sys.stdout.fileno())
>>      os.close(pipeout)
>>      os.execlp("./test.sh", "abc")
>> 
>> 
>>  def parent():
>>      pipein, pipeout = os.pipe()
>>      if os.fork() == 0:
>>          os.close(pipein)
>>          child(pipeout)
>>      else:
>>          os.close(pipeout)
>>          counter = 1
>>          pipein = os.fdopen(pipein)
>>          while True:
>>              json = pipein.readline()[:-1]
>>              print 'parent: %s' % (json)
>> 
>>  parent()
>> 
>>  $ cat test.sh
>> 
>>  #!/bin/bash
>> 
>> 
>>  script_name=$0
>> 
>>  args=$@
>> 
>>  current=`pwd`
>> 
>>  echo $script_name $args
>> 
>>  echo "XYZ: "$XYZ
>> 
>>  echo "PATH: "$PATH
>> 
>>  echo "current directory: $current"
>> 
>> 
>>  _______________________________________________
>>  IPython-dev mailing list
>>  IPython-dev at scipy.org
>>  http://mail.scipy.org/mailman/listinfo/ipython-dev
>> 
>> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20141003/883aab7c/attachment.html>