[IPython-dev] Launching new kernels by forking existing kernel

[Jason Grout]

On 5/8/12 2:04 PM, MinRK wrote:
>
>
> On Tue, May 8, 2012 at 11:33 AM, Brian Granger <ellisonbg at gmail.com
> <mailto:ellisonbg at gmail.com>> wrote:
>
>     Jason,
>
>     Whenever we fork processes using pyzmq, we have to create a new
>     context and sockets after the fork.  IOW, pyzmq context and sockets
>     cannot move over  fork boundary.  I think this means that it will be
>     essentially impossible to do what you are wanting to.
>
>
> I think that depends on what he means by this:
>
>>  Also, is it easy to redirect a kernel's zmq channels (after we fork, I
> presume we need to hook the new kernel up to separate channels with the
> client.)
>
> If there is some kernel-launching facility that is forking off
> processes, and those forked processes then create the context and
> sockets for the kernel, then this should work just fine.  What won't
> work is passing context and/or sockets across the fork boundary.
>
> This is the behavior we currently have:
>
> 1. KernelManager writes connection file.
> 2. KM starts Kernel via subprocess.Popen, passing the connection file
> via the command-line.
>
> If you replace step 2 with:
>
> 2a. KM sends special request to KernelStarter, including connection file
> (either via network, or in-process, I'm not sure of the architecture).
> 2b. KernelStarter starts kernel via *fork*, passing connection file, and
> being careful to avoid the subprocess inheriting zmq objects.

That's similar to how we do things now (wrapping ipython 0.10).  We have 
a managing process that receives a message, forks, and then the worker 
creates a new context (actually, with exactly the same zmq socket). 
What I'm trying to do is see if this model fits into the framework of 
the launchers already built into IPython (I'm trying to reuse the 
architecture and design of IPython as much as is reasonable, and 
contribute back where I can).

So if I understand things correctly, the managing process (the one that 
is forking, if everything is local) is the controller, and the forked 
worker is an engine.  The code in apps/launcher.py looks very much like 
the sorts of things I want to do---launch a controller or some engines. 
  Is that where I should begin experimenting, making a subclass of 
LocalProcessLauncher?

Thanks for you patience while I find my way around the code.

Jason