[IPython-dev] MultiKernelManager vs. ipcluster

[MinRK]

On Fri, Jun 1, 2012 at 2:07 PM, Jason Grout <jason-sage at creativetrax.com>wrote:

> Hi everyone,
>
> As mentioned yesterday, we've been exploring ways to implement the Sage
> cell server using the IPython capabilities better.  This message is a
> call for comments, as well as a few questions about the directions of
> things like ipcluster.
>
> Our goals are to:
>
> * start multiple kernels/engines very quickly (e.g., through forking a
> master kernel)
> * connect single kernels to single web sessions
> * be able to transfer files back and forth between the executing
> environment and the web browser
> * do things as securely as is reasonable (e.g., workers are running on a
> firewalled virtual instance, etc.)
> * be able to use an enterprise-level web server, such as Google App Engine.
>
>
> We've explored two approaches, and have somewhat-barely-working partial
> proofs-of-concepts of each:
>
> IPCLUSTER
> ---------
>
> We implemented a forking engine factory which is run on the worker
> computer.  A cluster can start up new engines by sshing into the worker
> computer and sending a message to the forking engine factory.  A new
> engine starts and then registers with the controller.  We're thinking it
> might be best to implement a new scheduler that would take messages,
> check to see if the session id matches up with one of the
> currently-running engines, and send the message to the appropriate
> engine if so.  If the message id does not match up to a currently
> running engine, then the scheduler would start up a new engine (can the
> scheduler do this?)
>

Very little of the current scheduler's code would be useful for this.  the
scheduler object is for load-balanced execution, and all of its logic is
involved in deciding what engines should get tasks.  You seem to be looking
for a pattern more like MUX (direct-routing).  For this pattern, IPython
just uses a pure-C MonitoredQueue function from pyzmq.  It sounds like you
want that is initially identical to the zmq_queue function, but on
unroutable requests, it starts a new engine and then routes the request
there.


>
> The client would basically be the zmq/socket.io bridge, translating
> messages to/from the controller and the browser.  I guess we could
> implement one client per session, or we could implement one overall
> client that would route cluster output to the appropriate browser.
>

Again, most of the Client code is involved in dealing with multiple
engines, and interleaving asynchronous requests.  So I would do one of:

* 1 Client per Server process, and 1 light View-like object per session
* build a much lighter Client that is more akin to the KernelManager, which
only talks to one engine.  It's possible this should just be a
KernelManager subclass.


>
> MULTI FORKING KERNEL MANAGER
> ----------------------------
>
> This approach stays much closer to the current IPython notebook
> architecture.  A MultiKernelManager starts up kernels using a
> ForkingKernelManager (based on Min's gist the other day).  Each kernel
> sets up a connection to a specific set of websocket channels through a
> tornado-based bridge.  We still have to implement things like separation
> of the forked kernels (on a separate ssh account somewhere) and the
> tornado handler, and things like that.
>
>
> THOUGHTS
> --------
> It seems that the multikernel manager is much lighter-weight, but we'd
> have to implement a lot of the enterprise-level functionality that the
> cluster already has.  On the other hand, the ipcluster approach really
> does more than we need, so, in a sense, we need to trim it back.  We're
> not asking you to make a decision for us, obviously, but it would be
> valuable to hear any comments or suggestions.
>
> SPECIFIC QUESTIONS
> ------------------
>
> 1. Brian, why did you decide to make a new multikernel manager instead
> of trying to leverage the ipcluster functionality to execute multiple
> engines?
>
> 2. Some time ago, on this list or on a pull request, there was some
> discussion about the difference between kernels and engines (or the lack
> of difference).  Are there some plans to simplify the ipcluster
> architecture a bit to merge the concepts of engines and kernels?
>

Engines and Kernels really are the exact same thing (as of a recent PR,
consolidating the two Kernel classes that had diverged during simultaneous
development).

The IPython Kernel is a single object defined in IPython.zmq.ipkernel

But there are two ways to start a Kernel:

1. IPKernelApp (used by KernelManagers, and `ipython kernel`).
2. IPEngineApp (used by `ipengine`)

At the simplest level, an Engine is a Kernel that connects instead of binds
(for consolidation of connection points at the Controller).

The Kernel object itself does not know whether it is binding or connecting
(in fact, it can already do both at the same time).

All of the practical differences at this point are due to the different
Application classes, which have a different set of command-line options
(IPKernelApp has all the expected shell options, like pylab, etc., while
the EngineApp has some options general to the IPython.parallel apps, like
forwarding logging over zmq).

The remaining differences are configuration details, and ultimately the two
entry points will probably be merged as well, and there will no longer be a
distinction beyond:

Binding Kernel: `ipython kernel `
Engine: `ipython kernel --connect`
Both: `ipython kernel --connect --bind`



> 3. Any idea about how much overhead the cluster introduces for lots of
> short computations involving lots of output?  We'll test this too, but
> I'm curious if there was thought towards this use-case in the design.
>

I haven't done performance tests with output before, but I would expect it
to do fairly well.  It's a  direct PUB-SUB channel up to the Clients (one
pure-C zmq_device hop at the Controller).


>
> Thanks,
>
> Jason
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20120601/d3c8c06c/attachment.html>