[IPython-dev] Project idea: Automatic lab notebook for iPython

[Thomas Kluyver]

Hi Peter,

I think this is the kind of question that a lot of us are interested in,
and I'd be keen to see support for this integrated with IPython notebooks.
I won't have much time to help with this for a few months, but I'll give a
brief run down of the architecture, so you can see where your tools might
fit in.

- The notebook server is an HTTP server - at present, most users run it
locally, but multi-user servers are definitely planned.
- When you open a notebook in the browser, the server starts a
corresponding 'kernel', a Python interpreter communicating over ZMQ.
- As you interact with the notebook, messages like 'execute this code' and
'get tab completions for this line' go from the browser to the kernel, and
replies come back.
- When you save a notebook, javascript running in the browser serialises
the current state of your input and output cells, and sends it to the
server as JSON. The server dumps it to a file, or in the future to a
database.

We'll be happy to answer any questions you have.

Thanks,
Thomas


On 17 April 2013 00:38, Peter Macko <pmacko at eecs.harvard.edu> wrote:

> Hi iPython developers,
>
> Here is a new project idea: automatic lab notebook for iPython and
> iPython Notebook, which would keep track of how each of your output
> files was produced, linking this "history" (or a "lineage") of an object
> across different iPython sessions and different iPython notebooks, and
> storing it persistently. This is frequently referred to in the Computer
> Science literature as "provenance."
>
> It will enable you to ask questions like "what did I do to produce this
> plot?" - and for example, it will tell you that you downloaded the input
> data set on Monday from such and such website, you ran all these
> commands to process the data on Tuesday, and then produced this plot on
> Thursday from a different iPython session. Note that this goes beyond
> (and is complementary in purpose to) iPython Notebook, since the history
> of a file is tracked across different sessions and Notebooks, and when
> you ask a question, you will get only the relevant information,
> suppressing any additional things that you did that are unrelated to the
> file in which you are interested.
>
> We are in touch with computational scientists all the way from
> bioinformatics to physics that are very interested in this feature! We
> met their needs partially by developing a cross-platform, multi-lingual
> library (https://code.google.com/p/core-provenance-library/) that they
> can use to annotate their Python (and non-Python) scripts in order to
> track the lineage of their objects.
>
> Our vision is that this will be all done fully automatically, without
> requiring the users to manually annotate their scripts. But
> unfortunately neither of us who are involved in this project has the
> resources or the knowledge of the iPython code-base to tackle this
> challenge. We need your help to make this happen! We have some ideas
> about how we might go about this, but we need someone who knows more
> about iPython to talk them over and to spearhead the actual development.
> Please let us know if you can help!
>
> Thank you,
>
> Peter Macko
>
> Harvard School of Engineering and Applied Sciences
> 33 Oxford St.
> Cambridge, MA 02138
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20130417/76309805/attachment.html>