[IPython-dev] api between kernel, web server and frontend

[Ondrej Certik]

Hi,

we were in Berkeley with Mateusz and we discussed the API between the
computational kernel, web server and the javascript frontend (and
other frontends), so I wanted to share our thoughts.

The main workhorse is a kernel, that has two connections:

1) feed, that publishes all changes to a given session, anyone (who
knows the session UID) can subscribe to it and observe the whole
session (e.g. ipython session)
2) request/response channel, which is used for sending python code for
evaluation and some other minor things. The results are published in
the feed.

The API is done using json, and it is described here:

http://github.com/ellisonbg/pyzmq/blob/master/examples/kernel/message_spec.rst

and demo implementation of this is here:

http://github.com/ellisonbg/pyzmq/tree/master/examples/kernel/

This demo implementation is using 0MQ (http://www.zeromq.org/) for the
transport layer (i.e. for sending the json messages). 0MQ is probably
the best library for sending/receiving messages over the net, but a
big disadvantage is that it probably can't be used on the google app
engine and it creates a dependency. But it's just the transport layer,
in principle it can be replaced with anything else.



So the above is the main logic for handling computational sessions,
with many people (frontends) doing simultaneous calculations in the
same session and one kernel (that however can dispatch the actual
little calculations to many cores, e.g. do it in parallel). Note that
this kernel has no database, no user accounts, nothing. All that it
has is a session with UID (so you need to create the session somehow,
but that's it) and it has a namespace for the running session, that's
it.

Now we need a web server, that would use the above API to communicate
with the kernel, and this web server would have a database with user
accounts, it would also have a database with user worksheets and cells
and it would expose this over HTTP. The problem is that using HTTP and
the current web technologies pose some limitations currently to what
can be done. We talked with Mateusz how to do that in the car to Reno
and we agreed that the best way is to use a JSONRPC, so the web server
would define a Python class with some functionality, and then the
javascript frontend (running in the browser) would use methods of this
class to do things (create worksheets, get users, create cells,
evaluate cells, get the feed from the kernel, ...). The JS frontend
would have to periodically call some method on the server to get the
feed, as we don't know how to "subscribe" to it using AJAX. Maybe in
couple years, this would be simple using web sockets.

I would be interested in any feedback and discussion.

We now need to write a demo implementation of the above web server and
a client that uses JSONRPC to do the above. The client can (should) be
command line based and it will only use JSONRPC for all the
communication. It will then be quite trivial (trivial in terms of the
logic) to anyone to write a javascript client, as JSONRPC works nice
with javascript, or adapt codenode or the sage notebook frontends to
use this JSONRPC.


For testing purposes, I would really like to have a demo
implementation that runs on the google app engine (both the web server
and the kernel). Then anyone can run a simple client, that connects to
it (and we will also have a simple javascript client, so you would
just go to the demo page and it would work). For that to happen, we
need to figure out some other transport layer than 0MQ, something that
works on the app engine. This can also be the only option for people
that don't want/can't to install 0MQ for some reason. Any ideas?

Ondrej