[IPython-dev] ipython with sync

Tue Sep 10 16:09:01 EDT 2013

On Tue, Sep 10, 2013 at 12:54 PM, Fernando Perez <fperez.net at gmail.com> wrote:
> Hi William,
>
> On Tue, Sep 10, 2013 at 9:11 AM, William Stein <wstein at gmail.com> wrote:
>
>> 5. Open the ipynb file in multiple browsers, and see that changes in
>> one appear in the other, including moving cells around, creating new
>> cells, editing markdown (the rendered version appears elsewhere), etc.
>>   Anything that sets the notebook.dirty flag in IPython causes a sync
>> (evaluating a cell that creates no output doesn't set this flag, at
>> least in 1.0.0, which is a bug in IPython, I guess).
>
> Wow, very impressive!  At the very first cell I saw some glitches: I
> simply typed '1' in the first empty cell and ran it; it produced '1'
> as output but the input got deleted and copied to the next (new) cell.
> Weird.  It didn't happen again.

You should have seen had bad the overall sync infrastructure was in
May... I've fixed a lot of issues with it, but it's still not perfect
in all cases, and I'm using some new features for the IPython stuff.
Hopefully the above sort of race condition should be very rare.  If
you can find anything reproducible, please let me know.

> Overall it seems to work, though for now I've only tested it very
> lightly.  The startup problem I saw yesterday seems gone.
>
>> Since this is all very new and the first (I guess) realtime sync
>> implementation on top of IPython, there are probably a lot of issues.
>>  Note that if you click the "i" info button to the right, you'll get a
>> link to the standard IPython
>
> My first question is: to what extent does this rely on your
> surrounding node.js machinery and to what extent is it something
> self-contained that could be used by the current IPython design?  This
> is obviously something of high interest to us (and oft-requested by
> users), so it would be good to understand what parts of the design
> would make sense to upstream and what parts are deeply intertwined
> with the entire sagemath architecture.

I just quickly wrote up a rough draft of a blog post, which I had
planned to post at http://www.blogger.com, but evidently that
(Google-owned) site is broken for login, for me at least.     So just
see the 1-page "How It Works" section here:

   http://wstein.org/tmp/2013-09-10-104600-ipython.html

>> The other thing of interest is a little Python script called
>> "ipython-notebook", which I wrote.  It basically makes it easy to run
>> an IPython notebook server as a daemon, get the port it is running on,
>> etc.  It's pretty simple but satisfies my use case, and has
>> JSON-output, to make it web friendly.    As I've written it, my script
>> passes several base_url options through by default, which are needed
>> for cloud.sagemath.  Anyway, I've attached it to this email (with a
>> BSD license) in case there is any interest.
>
> On that front, there are two related projects worth mentioning:
>
> https://github.com/UnataInc/ipydra: along the lines of yours but more elaborate.
>
> https://github.com/ptone/jiffylab: docker-based full containerization
> of a notebook+shell.
>
>> Regarding the monkey patching in 4 above, the right thing to do would
>> be to explain exactly what hooks/changes in the IPython html client I
>> need in order to do sync, etc., make sure these makes sense to the
>> IPython devs, and send a pull request (or have a coding sprint in
>> Seattle or Berkeley?).  As an example, in order to do sync
>> *efficiently*, I have to be able to set a given cell from JSON -- it's
>> critical to do this in place when possible, since the overhead of
>> creating a new cell is huge (due probably to the overhead of creating
>> CodeMirror editors); however, the fromJSON method in IPython assumes
>> that the cell is brand new -- it would be nice to add an option to
>> make a cell fromJSON without assuming it is empty.
>
> Yup, this seems like a sensible idea and I can imagine us using it in
> many other places, actually.
>
>> The ultimate outcome of this could be a clean well-defined way of
>> doing sync for IPython notebooks using any third-party sync
>> implementation.  IPython might provide their own sync service and
>> there are starting to be others available these days -- e.g., Google
>> has one: https://developers.google.com/drive/realtime/, and maybe
>> Guido van Rosum helped write one for Dropbox recently?
>
> Thanks for that Google link, I hadn't seen this one.  I remember
> they'd opened up an older implementation of their Wave stuff, but when
> I looked at it, it was a huge Java monster so it didn't look like we
> could reuse any of it.  It would be interesting to consider relying on
> an external service for syncing as an option, but honestly it seems to
> me that in the end we should put that into the IPython server itself.
> If nothing else, for latency reasons in locally hosted servers I can
> imagine it making a big difference.

It might be possible to do both, with the right design and hooks.   I
know you guys like to really think things through carefully and get
things right in a long-term general and flexible way.

>
>> Subdirectories:   I noticed, incidentally, that the wakari version of
>> the IPython notebook server allows one to load ipynb files that are in
>> any subdirectory, whereas the standard IPython notebook server
>> doesn't.  For cloud.sagemath, I just spawn a new IPython notebook
>> server for each directory that a user accesses files in right now.
>> This seems cludgy, so I'm interested in the situation regarding adding
>> support for subdirectories.
>
> See Thomas' response.  We hope to have this merged before too long,
> and it will be a huge improvement for everyone.
>
> Cheers,
>
> f
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev

-- 
William Stein
Professor of Mathematics
University of Washington
http://wstein.org