[IPython-dev] Notebook websocket change

William Stein wstein at gmail.com
Wed Jan 14 19:55:21 EST 2015 

On Wed, Jan 14, 2015 at 4:50 PM, MinRK <benjaminrk at gmail.com> wrote:
> The PR is 2321, switching IPython to use SockJS instead of WebSockets. I'm
> not sure how informative it is, since things have moved around a lot since
> then, and it is from a time when we were shipping more things in external,
> rather than accepting them as dependencies. I think a similar patch today
> would be a great deal simpler, especially now that it doesn't need to
> include merging sockets. I also had to include a hack to workaround the fact
> that SockJS didn't allow empty messages, and we were sending empty messages
> as part of the initial handshake (also not true anymore).
>
> -MinRK

Thanks!  I'll carefully think through a range of options.  Another
might be to just plug into your code to send/receive messages, and
just send/receive them over my existing websocket connection, so there
is just a single websocket connection, shared across ipython and smc.

 -- William

>
>
> On Wed, Jan 14, 2015 at 4:12 PM, William Stein <wstein at gmail.com> wrote:
>>
>> On Thu, Jan 8, 2015 at 11:18 AM, MinRK <benjaminrk at gmail.com> wrote:
>> > Some time ago, I made a PR to IPython switching to socket.io, which
>> > required
>> > the same basic changes since it only allowed a single connection per
>> > page,
>> > so I know it's not too hard. We may start to see more pressure as hosted
>> > instances, and reconsider the fallback in the future, but at this point,
>> > we
>> > don't feel that pressure. Wind proper websockets to be sufficiently
>> > preferable that we will wait as long as we can before appeasing old,
>> > broken,
>> > or misconfigured network environments that prevent websockets from
>> > working.
>> >
>> > The majority of notebook users seem to be running as a desktop app via
>> > localhost, where websockets are rarely a problem (The nightmare of
>> > Sophos
>> > AV/firewall on Windows excluded).
>>
>> Do you have a pointer to the above pull request?
>>
>> In my experience battling with websockets and the cloud over the last
>> few years, it's not just an issue of websockets versus fallbacks --
>> there's also issues with the robustness of the websocket connection,
>> how it's maintained,  automatically reconnecting if the connection has
>> problems, heartbeats, etc.   If you only use IPython locally then none
>> of this is a problem.  However, if you use IPython over a complicated
>> network that drops packets sometimes, flakes out, etc., it can be.   I
>> get fairly regular emails from people having trouble specifically with
>> the robustness of the iPython connection inside of SMC.    Libraries
>> like socket.io and engine.io [1] seem to have pretty good debuged and
>> thought out code for dealing with these situations (and for fallbacks
>> in case websockets doesn't work).
>>
>> Since this use case is only a tiny part of the total IPython notebook
>> usage, it makes sense for me to take this on myself.   I will
>> definitely highly prioritize trying to hack on IPython in order to get
>> it to use engine.io+primus, since I have to deal with this websocket
>> robustness situation to improve my user's experience.   For example,
>> I'm getting fairly regular complaints from Randy Leveque and his
>> colleagues at UW.     So I'd greatly appreciate a link to that pull
>> request, and of course I'll share anything I do here.
>>
>> [1] https://github.com/Automattic/engine.io
>>
>> >
>> > -MinRK
>> >
>> > On Thu, Jan 8, 2015 at 10:55 AM, William Stein <wstein at gmail.com> wrote:
>> >>
>> >> On Thu, Jan 8, 2015 at 10:19 AM, Brian Granger <ellisonbg at gmail.com>
>> >> wrote:
>> >> > So far we have felt that it isn't worth the complexity cost for us to
>> >> > support people that deliberately choose to break the internet. It is
>> >> > not
>> >> > like WebSockets is some crazy, insecure, unsupported hack over a
>> >> > weird
>> >> > port.
>> >> > It is just different bytes going over the same HTTP/HTTPS socket
>> >> > connection
>> >> > and is essentially a formal standard (I know you know this - this is
>> >> > our
>> >> > messaging to companies who complain).
>> >>
>> >> I know -- that's why I switched SMC to pure websockets for a while.
>> >> Then I received complaints at a rate of about 2 very desperate users
>> >> per week, and these were the people who cared enough to complain
>> >> repeatedly.
>> >>
>> >> > Our situation is a bit different though than SMC because most of our
>> >> > current
>> >> > users run the notebook on their own computers. This may be something
>> >> > that we
>> >> > eventually want to rethink as more people run this in the cloud.
>> >> > However, I
>> >> > would like to push back on the companies choosing to break the
>> >> > internet
>> >> > really hard before giving in. I am guessing that in most cases,
>> >> > WebSockets
>> >> > are broken in enterprise setting not because of some important
>> >> > deliberate
>> >> > choice, but rather because people don't understand it and are using
>> >> > default
>> >> > settings that disable them (I could be wrong though)....
>> >>
>> >> In my experience, for some people that's exactly the problem.  For one
>> >> person I helped it turned out their specific problem was the Windows
>> >> install they had access to had websockets explicitly disabled --
>> >> perhaps that was an IT policy.  For other people the problem is
>> >> further along the stack (e.g., possibly involving caching).
>> >>
>> >> Anyway, this change so that "there is a single websocket connection
>> >> between the server and client. " makes implementing an alternative to
>> >> WebSocket as a fallback dramatically easier, which I greatly
>> >> appreciate.
>> >>
>> >> > Our situation is a bit different though than SMC because most of our
>> >> > current
>> >> > users run the notebook on their own computers. This may be something
>> >> > that we
>> >>
>> >> Just curious -- how do you know that most people using IPython aren't
>> >> already using it through SMC?
>> >>
>> >> William
>> >>
>> >> >
>> >> > Cheers,
>> >> >
>> >> > Brian
>> >> >
>> >> >
>> >> >
>> >> > On Thu, Jan 8, 2015 at 10:10 AM, William Stein <wstein at gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> On Thu, Jan 8, 2015 at 10:04 AM, MinRK <benjaminrk at gmail.com> wrote:
>> >> >> > Alternative notebook frontend authors:
>> >> >> >
>> >> >> > There is an upcoming change to websocket connections to the
>> >> >> > notebook
>> >> >> > server.
>> >> >> > Instead of three websockets per notebook, one for each zmq
>> >> >> > channel,
>> >> >> > there is
>> >> >> > a single websocket connection between the server and client. The
>> >> >> > channel
>> >> >> > associated with each message is identified by a channel key in the
>> >> >> > message
>> >> >> > dict itself.
>> >> >> >
>> >> >> > Relevant changes:
>> >> >> >
>> >> >> > url is /channels instead of /shell, etc.
>> >> >> > when sending a message, add a channel key with the channel name
>> >> >> > ('shell',
>> >> >> > 'stdin')
>> >> >> > when receiving a message, check the channel key for routing
>> >> >> >
>> >> >> > -MinRK
>> >> >>
>> >> >> Thanks.
>> >> >>
>> >> >> Of related interest, what the situation involving providing a
>> >> >> fallback
>> >> >> to websockets?
>> >> >>
>> >> >> I tried to only support websockets connections with SageMathCloud
>> >> >> for
>> >> >> a few weeks, and immediately ran into many complaints from users who
>> >> >> couldn't connect as a result.   This is even with https and so on --
>> >> >> it's just a sad fact that in some corporate or otherwise constrained
>> >> >> environments that websockets don't work.   By far the best approach
>> >> >> I
>> >> >> found was to use Primus + Engine.io, which uses websockets if
>> >> >> possible, but will fallback to polling.   So this is what SMC does
>> >> >> now, and everything works, even for people that don't have
>> >> >> websockets
>> >> >> as an option... except for IPython-in-SMC of course, which doesn't
>> >> >> work.
>> >> >>
>> >> >> Another unrelated issue is that I'm now hardcoding making this
>> >> >> change
>> >> >> around line 171 of IPython/html/notebookapp.py, since I want to
>> >> >> serve
>> >> >> the purely static html of IPython from a completely different web
>> >> >> server:
>> >> >>
>> >> >>             #static_url_prefix = url_path_join(base_url,'/static/'),
>> >> >>             static_url_prefix = '/static/ipython/',
>> >> >>
>> >> >> (Sorry for derailing the thread.)
>> >> >>
>> >> >>  -- William
>> >> >>
>> >> >> >
>> >> >> >
>> >> >> > _______________________________________________
>> >> >> > IPython-dev mailing list
>> >> >> > IPython-dev at scipy.org
>> >> >> > http://mail.scipy.org/mailman/listinfo/ipython-dev
>> >> >> >
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> William Stein
>> >> >> Professor of Mathematics
>> >> >> University of Washington
>> >> >> http://wstein.org
>> >> >> _______________________________________________
>> >> >> IPython-dev mailing list
>> >> >> IPython-dev at scipy.org
>> >> >> http://mail.scipy.org/mailman/listinfo/ipython-dev
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Brian E. Granger
>> >> > Cal Poly State University, San Luis Obispo
>> >> > @ellisonbg on Twitter and GitHub
>> >> > bgranger at calpoly.edu and ellisonbg at gmail.com
>> >> >
>> >> > _______________________________________________
>> >> > IPython-dev mailing list
>> >> > IPython-dev at scipy.org
>> >> > http://mail.scipy.org/mailman/listinfo/ipython-dev
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> William Stein
>> >> Professor of Mathematics
>> >> University of Washington
>> >> http://wstein.org
>> >> _______________________________________________
>> >> IPython-dev mailing list
>> >> IPython-dev at scipy.org
>> >> http://mail.scipy.org/mailman/listinfo/ipython-dev
>> >
>> >
>> >
>> > _______________________________________________
>> > IPython-dev mailing list
>> > IPython-dev at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/ipython-dev
>> >
>>
>>
>>
>> --
>> William Stein
>> Professor of Mathematics
>> University of Washington
>> http://wstein.org
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
>
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>



-- 
William Stein
Professor of Mathematics
University of Washington
http://wstein.org

More information about the IPython-dev mailing list