[IPython-dev] Proposal: soft moratorium on re-architecting for 5.0

[Brian Granger]

> One of the principles to which we attribute the success of the project so
> far is that we solve the problems in front of us and restructure when we
> need to, rather than rushing into building grand architecture. I think that
> we need a cycle to digest the architectural changes we've recently made
> before we dive into another round. In particular, the talk of refactoring
> all our frontend machinery concerns me. I don't want to stop us from
> planning this, but I think it would be a mistake to try to do it for 5.0.

I do think there are part of our code base where conservatism is
useful and beneficial: the message specification, notebook document
format, kernel spec, etc. Many features and APIs associated with the
ipython kernel are also evolving in a pretty conservative manner. The
server side of the notebook is reasonably stable, although the
real-time collab and ongoing security efforts will require significant
changes to parts of it.

However, I think the project would be deeply hurt by applying this
conservatism broadly or to the notebook frontend in particular.

>
> There are a lot of important features that we have been saying for some time
> we will work on - like multi cell selection, find & replace across the
> notebook, improved tab completion, and an HTML console interface. We've
> deferred these tasks while we worked on the architecture, and it's starting
> to be embarrassing that the interface is not moving forwards quicker. There
> would also be less impetus for people to turn to competing projects if we
> sanded some of the rough edges off the notebook interface.

I agree that we have lots of rough edge that are adversely affecting
our users today. These range from bugs, usability problems, basic
features that are lacking, and new, highly-anticipated features.
However, the development challenges we have seen in trying to
implement these things strongly point to continuing with a deep
architectural surgery:

* Off the top of my head, between Cal Poly, Berkeley, Google,
Bloomberg and Continuum I can count 8-ish existing staff being
allocated to work full time on the notebook. With upcoming new hires
at Cal Poly and Berkeley, this number will go over >10 staff working
primarily on the notebook.
* Our existing monolithic, tightly-coupled frontend code makes it
impossible for more than 1-2 main lines of frontend work to move
forward at the same time.
* We have numerous third part projects and companies (kbase, Rodeo,
Hydrogen, Sage, IBM, Quantopian, DataRobot, O'Reilly, etc.) who are
taking our frontend code and using it to build customized frontend
applications. Our code base is making this extremely difficult for
these groups. Because of this, each of these groups is having to
maintain and develop forks, rather than working with us to build a
common set of flexible components. This means that everyone is
resolving the same problems in ways that our users don't benefit from.

Deep changes in the frontend code are needed, precisely because there
is no way we can build the user-focused features with this many people
using our currently code base. Even something like the HTML console
interface will be dramatically easier - if not trivial - once the code
base is refactored into better-designed, loosely coupled components.

> On that note, I'd like us to try doing 'user Fridays', where we pretend that
> we can't change the Jupyter/IPython code, and work on making interesting
> notebooks, extensions, and other parts of the ecosystem. Every time I work
> on such things, it highlights problems I wasn't aware of, or rough edges I
> hadn't fully appreciated. Having a semi-official mechanism to encourage us
> to do this in regular small chunks would be valuable in keeping us focussed
> on users rather than just architecture.

I agree this is useful for all of us to do on a continual basis. I
just finished teaching for ten weeks and most of my time was spent in
doing this type of stuff. It was wonderful and very instructive.

> The moratorium I propose would be left to our judgement to implement - it's
> a principle rather than a rule. I'd make an exception for the conversion to
> Typescript, because Matthias has already spent significant time on that, and
> I know he's champing at the bit to finish it. Widgets would be exempt,
> because we've made it very clear that they're liable to change
> significantly, and they shouldn't much affect development of the notebook
> itself.
>
> Thanks,
> Thomas
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>



-- 
Brian E. Granger
Cal Poly State University, San Luis Obispo
@ellisonbg on Twitter and GitHub
bgranger at calpoly.edu and ellisonbg at gmail.com