[IPython-dev] Launching new kernels by forking existing kernel

[MinRK]

On Tue, May 8, 2012 at 2:02 PM, Jason Grout <jason-sage at creativetrax.com>wrote:

> On 5/8/12 2:04 PM, MinRK wrote:
> >
> >
> > On Tue, May 8, 2012 at 11:33 AM, Brian Granger <ellisonbg at gmail.com
> > <mailto:ellisonbg at gmail.com>> wrote:
> >
> >     Jason,
> >
> >     Whenever we fork processes using pyzmq, we have to create a new
> >     context and sockets after the fork.  IOW, pyzmq context and sockets
> >     cannot move over  fork boundary.  I think this means that it will be
> >     essentially impossible to do what you are wanting to.
> >
> >
> > I think that depends on what he means by this:
> >
> >>  Also, is it easy to redirect a kernel's zmq channels (after we fork, I
> > presume we need to hook the new kernel up to separate channels with the
> > client.)
> >
> > If there is some kernel-launching facility that is forking off
> > processes, and those forked processes then create the context and
> > sockets for the kernel, then this should work just fine.  What won't
> > work is passing context and/or sockets across the fork boundary.
> >
> > This is the behavior we currently have:
> >
> > 1. KernelManager writes connection file.
> > 2. KM starts Kernel via subprocess.Popen, passing the connection file
> > via the command-line.
> >
> > If you replace step 2 with:
> >
> > 2a. KM sends special request to KernelStarter, including connection file
> > (either via network, or in-process, I'm not sure of the architecture).
> > 2b. KernelStarter starts kernel via *fork*, passing connection file, and
> > being careful to avoid the subprocess inheriting zmq objects.
>
> That's similar to how we do things now (wrapping ipython 0.10).  We have
> a managing process that receives a message, forks, and then the worker
> creates a new context (actually, with exactly the same zmq socket).
> What I'm trying to do is see if this model fits into the framework of
> the launchers already built into IPython (I'm trying to reuse the
> architecture and design of IPython as much as is reasonable, and
> contribute back where I can).
>
> So if I understand things correctly, the managing process (the one that
> is forking, if everything is local) is the controller, and the forked
> worker is an engine.  The code in apps/launcher.py looks very much like
> the sorts of things I want to do---launch a controller or some engines.
>  Is that where I should begin experimenting, making a subclass of
> LocalProcessLauncher?
>

You can definitely try subclassing the LocalProcessLauncher to use fork
instead of subprocess.

I'm not sure how valuable that will be, given how much if the
ProcessLauncher stuff is based on passing cl-args to a subprocess, but it's
worth a look.


>
> Thanks for you patience while I find my way around the code.
>
> Jason
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20120508/0fe10ad8/attachment.html>