From junkatar at vivistar.com  Thu Jun  3 13:39:58 2021
From: junkatar at vivistar.com (Jonathan Hess)
Date: Thu, 3 Jun 2021 10:39:58 -0700
Subject: [IPython-dev] saving base64 encoded content separately
Message-ID: <BBACD121-D776-4CD3-AA1A-0556C88D0F7D@vivistar.com>

[The archives show this list has become very quiet.  If developers have moved to another forum, care to tell me where?]

I'm looking for an option or extension that would, for .ipynb files, replace base64 encoded data with references to an external "native" file of the same data.

For example, if an ipynb file has a matplotlib produced png image, I need that image stored in an external png file.

Yes, integral base64 encoding is nice from network, efficiency, and file management aspects.

Unfortunately, security conscious organization that have established review processes simply will not allow base64 content -- it is too easy to hide content or malware that way from their established (read unchangeable) processes.

I've been reading up on:

  https://ipython.readthedocs.io/en/stable/config/extensions/index.html
  https://ipython.readthedocs.io/en/stable/config/shell_mimerenderer.html

But, before I dive off into the trial and error time sync of just trying things:

A) Has anyone already done an extension like this?  My websearches have failed so far but I would be surprized if no-one has asked about such a thing before.

B) If A) is no, anyone have suggestions or guidance on making such an extension?

Finally (and possibly not relevant to this forum), with such a feature or extension for ipython, I would expect end users to be able to control it thru a Jupyter Lab configuration setting.

Thanks for reading this far,
Jonathan Hess

From chris.barker at noaa.gov  Thu Jun  3 14:21:36 2021
From: chris.barker at noaa.gov (Chris Barker)
Date: Thu, 3 Jun 2021 11:21:36 -0700
Subject: [IPython-dev] saving base64 encoded content separately
In-Reply-To: <BBACD121-D776-4CD3-AA1A-0556C88D0F7D@vivistar.com>
References: <BBACD121-D776-4CD3-AA1A-0556C88D0F7D@vivistar.com>
Message-ID: <CALGmxEJEhutGj27bANSY0UWOn+wBeG4CzFOh87hhVMfg54-9Wg@mail.gmail.com>

I have no idea if there's a new list -- but you're right, I haven't gotten
a note from this list in ages.

There's a whole pile of Project Jupyter community groups:

https://jupyter.org/community

maybe the discourse channel for the notebook is the way to go:

https://discourse.jupyter.org/c/notebook/31

but on to your problem:

There is a real challenge here -- the goal of .ipynb files is to store the
whole notebook in a single file, and that is JSON format, which requires
base64 or similar to embed binary data.

Yes, it would be fairly simple to have it reference an external file
instead, but then you'd have a bundle of files that needed to be kept
together. We could put them all in a single zip file, but that may well run
afoul of other overzealous security scanners :-(

I wonder if one could use BSON, which has a Binary data type, but is
otherwise a pretty one-for-one match with JSON.

Of course, maybe the security folks won't like that either -- frankly, you
either have arbitrary binary data or you don't :-)

-CHB



On Thu, Jun 3, 2021 at 11:03 AM Jonathan Hess <junkatar at vivistar.com> wrote:

> [The archives show this list has become very quiet.  If developers have
> moved to another forum, care to tell me where?]
>
> I'm looking for an option or extension that would, for .ipynb files,
> replace base64 encoded data with references to an external "native" file of
> the same data.
>
> For example, if an ipynb file has a matplotlib produced png image, I need
> that image stored in an external png file.
>
> Yes, integral base64 encoding is nice from network, efficiency, and file
> management aspects.
>
> Unfortunately, security conscious organization that have established
> review processes simply will not allow base64 content -- it is too easy to
> hide content or malware that way from their established (read unchangeable)
> processes.
>
> I've been reading up on:
>
>   https://ipython.readthedocs.io/en/stable/config/extensions/index.html
>   https://ipython.readthedocs.io/en/stable/config/shell_mimerenderer.html
>
> But, before I dive off into the trial and error time sync of just trying
> things:
>
> A) Has anyone already done an extension like this?  My websearches have
> failed so far but I would be surprized if no-one has asked about such a
> thing before.
>
> B) If A) is no, anyone have suggestions or guidance on making such an
> extension?
>
> Finally (and possibly not relevant to this forum), with such a feature or
> extension for ipython, I would expect end users to be able to control it
> thru a Jupyter Lab configuration setting.
>
> Thanks for reading this far,
> Jonathan Hess
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at python.org
> https://mail.python.org/mailman/listinfo/ipython-dev
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/ipython-dev/attachments/20210603/2a2d2e9f/attachment.html>

From Mark.Voorhies at ucsf.edu  Thu Jun  3 14:18:56 2021
From: Mark.Voorhies at ucsf.edu (Voorhies, Mark)
Date: Thu, 3 Jun 2021 18:18:56 +0000
Subject: [IPython-dev] saving base64 encoded content separately
In-Reply-To: <BBACD121-D776-4CD3-AA1A-0556C88D0F7D@vivistar.com>
References: <BBACD121-D776-4CD3-AA1A-0556C88D0F7D@vivistar.com>
Message-ID: <a81215f345ab752cb6332e5d876a3b528938d9c3.camel@ucsf.edu>

On Thu, 2021-06-03 at 10:39 -0700, Jonathan Hess wrote:
> [The archives show this list has become very quiet.  If developers have moved to another forum, care to tell me where?]
> 
> I'm looking for an option or extension that would, for .ipynb files, replace base64 encoded data with references to an external "native" file of the same data.
> 
> For example, if an ipynb file has a matplotlib produced png image, I need that image stored in an external png file.
> 

<snip>

> A) Has anyone already done an extension like this?  My websearches have failed so far but I would be surprized if no-one has asked about such a thing before.
> 
> B) If A) is no, anyone have suggestions or guidance on making such an extension?
> 

The original HTML/XHTML export from the qtconsole had an option for externally linked images: c.f. commit aae8597d63e in the ipython repo.

That implementation depended on Qt to do the heavy lifting for HTML export, so it's not directly applicable to Jupyter notebooks.

I _would_ think that you could do something similar by adding a new HTML template to jupyter nbconvert.

--Mark

From takowl at gmail.com  Sat Jun  5 06:54:40 2021
From: takowl at gmail.com (Thomas Kluyver)
Date: Sat, 5 Jun 2021 11:54:40 +0100
Subject: [IPython-dev] saving base64 encoded content separately
In-Reply-To: <BBACD121-D776-4CD3-AA1A-0556C88D0F7D@vivistar.com>
References: <BBACD121-D776-4CD3-AA1A-0556C88D0F7D@vivistar.com>
Message-ID: <CAOvn4qjyHyqu0uxNfX49ThgqxFMjRAU_R=jX-PZc977heCc5TA@mail.gmail.com>

Hi Jonathan,

The notebook file format is really about Jupyter, rather than IPython (the
project split into two parts; IPython provides the Python execution backend
for Jupyter). You might want to try on the Jupyter Discourse board:
https://discourse.jupyter.org/

I experimented several years ago with 'exploding' a notebook file into a
directory (https://github.com/takluyver/nbexplode ). I don't expect it to
be a directly usable solution, but it might have useful ideas or pieces for
you.

Best wishes,
Thomas

On Thu, 3 Jun 2021 at 19:02, Jonathan Hess <junkatar at vivistar.com> wrote:

> [The archives show this list has become very quiet.  If developers have
> moved to another forum, care to tell me where?]
>
> I'm looking for an option or extension that would, for .ipynb files,
> replace base64 encoded data with references to an external "native" file of
> the same data.
>
> For example, if an ipynb file has a matplotlib produced png image, I need
> that image stored in an external png file.
>
> Yes, integral base64 encoding is nice from network, efficiency, and file
> management aspects.
>
> Unfortunately, security conscious organization that have established
> review processes simply will not allow base64 content -- it is too easy to
> hide content or malware that way from their established (read unchangeable)
> processes.
>
> I've been reading up on:
>
>   https://ipython.readthedocs.io/en/stable/config/extensions/index.html
>   https://ipython.readthedocs.io/en/stable/config/shell_mimerenderer.html
>
> But, before I dive off into the trial and error time sync of just trying
> things:
>
> A) Has anyone already done an extension like this?  My websearches have
> failed so far but I would be surprized if no-one has asked about such a
> thing before.
>
> B) If A) is no, anyone have suggestions or guidance on making such an
> extension?
>
> Finally (and possibly not relevant to this forum), with such a feature or
> extension for ipython, I would expect end users to be able to control it
> thru a Jupyter Lab configuration setting.
>
> Thanks for reading this far,
> Jonathan Hess
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at python.org
> https://mail.python.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/ipython-dev/attachments/20210605/3b6e19de/attachment.html>