[IPython-dev] Enhancement proposal for kernel messaging protocol

[Brian Granger]

> I wrote a PR for adding new option to messaging protocol [1]_.
> Bussonnier Matthias and Thomas Kluyver suggested to discuss about
> possible extensions for current messaging protocol, and then make an
> IPEP before working on these extensions.  So, let me start the
> discussion here.

Great, thanks for getting this started.

>
> 1. Protocol introspection.
>
> Currently, messaging protocol has no introspection mechanism.  It is
> not a problem when the messaging protocol is used only for internal
> communication.  However, there are various software using IPython via
> messaging protocol.  Also, IPython version of the kernel and client
> can be different, for example when used over ssh connection.
> Therefore, we need some introspection mechanism to inform messaging
> protocol supported by connecting kernel.

I agree with this assessment.

> One way to inform supported messaging protocol is to add version
> number to the messaging protocol.  There are several options for
> version numbering:
>
> a. Protocol version = IPython version.
>
>    This may be the simplest way to do it.  But in development version,
>    supported protocol will be unclear.

The problem with this is that the message protocol is quite stable
over the ipython versions.  This is why the notebook format has its
own version numbers.

> b. Semantic Versioning [2]_.
>
>    This is some what close to how notebook version works (right?).
>    Simple explanation: It takes the form of `x.y.z`.  Patch version
>    `z` is incremented for bug fix.  Minor version `y` is incremented
>    for backward compatible changes.  Major version `x` is incremented
>    for backwards incompatible changes.
>
>    For the messaging protocol, the next version number will be 1.1.0
>    if we think the current protocol is public.  It will be 0.2.0 if we
>    think the current protocol is still in development.
>
> There are several ways to let client know the version number.

I would use the exact same convention that we do in the notebook format.  X.Y:

X = major versions, incompatible
Y = minor versions, compatible

> c. Add `version_request` / `version_reply` messaging protocol.

Yes, this is what I would prefer.

> d. Add `version` slot to the message format.  It can be in
>    the top level, `header` or `metadata`.
> e. Send version number during "hand shake" (but there is no
>    such stage for the current messaging protocol, I suppose?).
>
>
> 2. New optional `unique` key to `history_request`.
>
> I propose [1]_ to add this new boolean key `unique` to
> `history_request` [3]_.  When this key is specified and true,
> `history_reply` contains only entries of unique input.  This is useful
> for searching history using `hist_access_type: 'search'`.  This
> operation can be done in client side.  However, sending large number
> of entries over messaging protocol is time consuming.  One case this
> `unique` key can be critical is when using `history_request` in an
> interactive UI to search IPython history in as-you-type manner
> ("QuickSilver style"; for example, my Emacs client EIN [4]_ has such
> UI).  For such kind of UI speed is critical and it is preferred to be
> done in the kernel side.
>
> (As you can see, this is the reason why I made the PR in the first
> place!)

Yes, I can see that this would be a good thing to have.

>
> 3. Messaging protocol to pass structured data to kernel.
>
> Currently it is not possible to pass structured data such as dict from
> client to kernel.  You can generate Python code and use
> `execute_request`, but it is difficult to do this properly in
> non-Python client such as Javascript client.  Note that it is
> already possible to pass structured data to client from kernel
> by using JSON repr.
>
> (I think I saw some argument related to this issue tracker, but
> I can't find it now.  Please post the link if you know.)
>
> There are several ways to solve this problem:
>
> a. Add RPC-like message `call_request` / `call_reply`.  Client can
>    execute a short code to register "methods" in kernel to use this
>    RPC.  For example `__import__('mylib.ipyutils').register()`.
>
> b. Add `override_variables` slot to `execute_request`.  This slot
>    takes a dict and the value in the dict will override the value
>    correspond to its key.  For example, client can execute `func(x)`
>    with `override_variables: {'x': 1}`.
>
>    The problem is that it contaminates user's namespace.  To solve this
>    problem, we can add another `namespace` slot to `execute_request`.
>    This way, client can import its supporting library without
>    contaminating user's namespace and execute its functions using
>    `override_variables`.  For example, client can issue `execute_request`
>    like this::
>
>      {'code' "import mylib; mylib.ipyutils.func(x)",
>       'namespace': 'mylib',
>       'override_variables': {'x': 1}}
>
>    Merit of this approach comparing to the other one (`call_request`)
>    is that `namespace` can be used for another purpose.  For example,
>    notebook client can have separated namespace for each worksheet,
>    to avoid namespace collision.  Sharing data between namespaces can
>    be easily done by using singleton object.  Other possible application
>    is editor integration.  For example, you have associate one namespace
>    to each file (= python module) opened in the editor.

Even though we will eventually need this, I don't think we are ready
to tackle it yet.  This will happen as part of the work on interactive
widgets that will hopefully get started in Jan.

But, for the first two of these enhancements, I think they sound great
and don't need an IPEP.  A separate PR for each would be great :)

Cheers,

Brian

> .. [1] https://github.com/ipython/ipython/pull/2609
> .. [2] http://semver.org/
> .. [3] http://ipython.org/ipython-doc/dev/development/messaging.html#history
> .. [4] http://github.com/tkf/emacs-ipython-notebook
>
> ---
> Takafumi Arakaki
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev



--
Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu and ellisonbg at gmail.com