[SciPy-Dev] ways of storing notebook tutorials?

[Matthias Geier]

OK, nbsphinx has already been mentioned, so I don't have to (I'm its
author, BTW).

I just wanted to reinforce what has been said about not storing cell
outputs in the repository.
That's a good idea!

Notebooks can be "cleaned" automatically, see:
https://nbsphinx.readthedocs.io/en/0.4.3/usage.html#Using-Notebooks-with-Git

It's also possible to have most of the repository without outputs, but
only one branch containing executed notebooks. I've written about this
recently: https://mg.readthedocs.io/git-jupyter.html

Even without output, Jupyter notebooks are a bit annoying in version
control, because all strings are quoted in the JSON representation and
there are often plenty of escaped characters.
To avoid this, something like jupytext is great, but it adds quite
some of complexity for users.
Ideally, the native Jupyter notebook format should be changed, but I
don't know if that's likely to happen. FWIW, I've made a suggestion
(https://jupyter-format.readthedocs.io/), but I guess that's a topic
for the far future ...

Regarding what Ralf said:

> (a) have a "download as notebook" link in the html docs

That's easy, just look at the nbsphinx docs, e.g.
https://nbsphinx.readthedocs.io/en/0.4.3/code-cells.html.
At the top of the page there is a download link to the notebook on
Github and a link for launching it on Binder.
Arbitrary such links can be created, see
https://nbsphinx.readthedocs.io/en/0.4.3/prolog-and-epilog.html.

cheers,
Matthias

On Wed, Oct 9, 2019 at 2:26 PM Andrew Nelson <andyfaff at gmail.com> wrote:
>
> The overview for nbsphinx is at https://nbsphinx.readthedocs.io/en/0.4.3/
>
> On Wed, 9 Oct 2019, 12:57 Ralf Gommers, <ralf.gommers at gmail.com> wrote:
>>
>>
>>
>> On Tue, Oct 8, 2019 at 8:27 AM Andrew Nelson <andyfaff at gmail.com> wrote:
>>>
>>> Another alternative is to use `jupyter-sphinx` and `nbsphinx` sphinx extensions. You can include a link to the unexecuted notebook in RST (e.g. https://github.com/refnx/refnx/blob/master/doc/index.rst), and when the doc build occurs it'll execute the notebook + output as part of the run.
>>
>>
>> That looks like the closest fit so far. https://refnx.readthedocs.io/en/latest/getting_started.html looks nice. The parts your setup doesn't seem to do yet are (a) have a "download as notebook" link in the html docs, and (b) get rid of notebook output (e.g. https://raw.githubusercontent.com/refnx/refnx/master/doc/getting_started.ipynb contains the png images) in the repo. Both of those may not be too hard to add.
>>
>>>
>>>
>>> On Tue, 8 Oct 2019, 17:06 Matthew Brett, <matthew.brett at gmail.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> On Tue, Oct 8, 2019 at 3:55 PM Ralf Gommers <ralf.gommers at gmail.com> wrote:
>>>> >
>>>> >
>>>> >
>>>> > On Tue, Oct 8, 2019 at 3:22 PM Eric Larson <larson.eric.d at gmail.com> wrote:
>>>> >>
>>>> >> I've used sphinx-gallery in a number of projects (am now a maintainer). Basically you write Python code and it renders to html with links to downloadable notebooks in the page, automatic linking/cross-refs to function and class defs, nice index pages, etc.
>>>> >
>>>> >
>>>> > That doesn't really answer my question I think. The starting point is: notebooks are nice to write a long-form tutorial in, ReST is not. And sphinx-gallery is nice for a set of independent small to medium sized examples, but not the best fit for a long tutorial on a whole module (e.g. http://scipy.github.io/devdocs/tutorial/optimize.html).
>>>>
>>>> Right - I think sphinx-gallery is the right choice for these small
>>>> examples and where you want to develop the tutorial in text.  It
>>>> solves the ReST output nicely.
>>>>
>>>> >>
>>>> >>
>>>> >> Eric
>>>> >>
>>>> >>
>>>> >> On Tue, Oct 8, 2019 at 9:20 AM Matthew Brett <matthew.brett at gmail.com> wrote:
>>>> >>>
>>>> >>> Hi,
>>>> >>>
>>>> >>> I use Jupytext all the time now, and I prefer the RMarkdown input
>>>> >>> format to Jupytext.
>>>> >>>
>>>> >>> Jupytext can convert .ipynb files to .Rmd files, and saves notebooks
>>>> >>> as both .ipynb and .Rmd.
>>>> >>>
>>>> >>> The workflow is pretty good - after installing Jupytext, you open your
>>>> >>> .Rmd file in the notebook interface, and run anything you want, in the
>>>> >>> standard interface.
>>>> >>>
>>>> >>> Then you save, if you need to, which saves to both .ipynb and .Rmd.
>>>> >>>
>>>> >>> You edit the .Rmd using your nice text editor.
>>>> >>>
>>>> >>> Then you can reload the notebook in Jupyter, with just a browser
>>>> >>> refresh, to see the changes, and to run any extra code you've added.
>>>> >>>
>>>> >>> I then store the .Rmd files only, in version control, keeping the
>>>> >>> .ipynb as outputs from the final build step.
>>>> >
>>>> >
>>>> > Thanks, that sounds like a reasonable workflow. Seems like we can roundtrip notebook -> plain text format -> notebook.
>>>>
>>>> Right - exactly - and you can do that interactively, so you can save
>>>> from the Jupyter interface, load in a text editor, save in the text
>>>> editor and just reload the page in Jupyter to get the result.  It will
>>>> even keep the outputs as long as you haven't done very substantial
>>>> edits.
>>>>
>>>> > We'll still need to integrate that into the built html docs then, so the plain text format should be ReST. That doesn't seem supported yet: https://jupytext.readthedocs.io/en/latest/using-cli.html#testing-the-round-trip-conversion. Or am I missing something?
>>>>
>>>> No, that is (currently) a problem that I haven't been working on -
>>>> because I'm using Markdown output documents like jupyter-book and
>>>> bookdown. One of my collaborators uses notebooks a lot and I was
>>>> going to suggest to him that we have a separate site for the notebooks
>>>> and the main docs, or at least, we build them separately, and perhaps
>>>> combine them after building to HTML.    Maybe jupyter-book would help
>>>> here?
>>
>>
>> If we would start over, something like jupyter-book may be a better option. I'd prefer to avoid a major overhaul of all tutorials though, because there's a *lot* of content written in reST files already. Ideally we keep what we have in reST, and add the ability to write new ones in notebooks, which could be downloaded standalone but also integrate nicely with the existing Sphinx html build.
>>
>>>> Chris Holdgraf is the maintainer, and he's very responsive.
>>
>>
>> Thanks for the suggestion. Yes Chris is great, may ask him for advice.
>>
>> Cheers,
>> Ralf
>>
>>
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev