From lkb.teichmann at gmail.com Wed Oct 1 03:39:17 2014 From: lkb.teichmann at gmail.com (Martin Teichmann) Date: Wed, 1 Oct 2014 09:39:17 +0200 Subject: [IPython-dev] IPython inputhook, higher rate? In-Reply-To: References: Message-ID: Hi Thomas, > Thanks for looking into this. We would be interested in having a better way > to integrate event loops with getting input in the terminal. It's probably > not practical to push that straight into core Python, and even if it was, it > would be some time before we could scrap our inputhook functionality in > favour of a new feature in Python 3.5. To me it's a bit weird to say that we shouldn't push that straight into core Python - IPython is working around this broken functionality since more than a decade, so how long do we have to wait until something is not considered "straight" anymore? Apparently IPython is not the only one having this problem, as you mentioned others have also already written a library to circumvent this problem in the standard library. As for compatibility: I actually think that the way IPython is handling the problem - mostly hiding all the ugly details behind a nice API - would make it possible to have the new and old version nicely in parallel, especially as the new version has already mostly been written, for the case if IPython is not running in the terminal. Greetings Martin From bussonniermatthias at gmail.com Wed Oct 1 04:57:52 2014 From: bussonniermatthias at gmail.com (Matthias BUSSONNIER) Date: Wed, 1 Oct 2014 10:57:52 +0200 Subject: [IPython-dev] IPython inputhook, higher rate? In-Reply-To: References: Message-ID: <1DD61257-3A66-42F6-A523-35F7DEDED5FC@gmail.com> Le 1 oct. 2014 ? 09:39, Martin Teichmann a ?crit : > Hi Thomas, > >> Thanks for looking into this. We would be interested in having a better way >> to integrate event loops with getting input in the terminal. It's probably >> not practical to push that straight into core Python, and even if it was, it >> would be some time before we could scrap our inputhook functionality in >> favour of a new feature in Python 3.5. > > To me it's a bit weird to say that we shouldn't push that straight into core > Python - IPython is working around this broken functionality since more > than a decade, so how long do we have to wait until something is not > considered "straight" anymore? Apparently IPython is not the only one > having this problem, as you mentioned others have also already written > a library to circumvent this problem in the standard library. I think that the point of Thomas is that our code need to be cleaned up. We are currently preparing to split IPython into different repo so trying to at that into IPython **now** is probably not a good time. What I can suggest (other can chime in and say water they think is a good idea) is to write an IPep[1] or an issue on github that target IPython 4.0 or later. It will allow us to organize and decide if/when we want that. > As for compatibility: I actually think that the way IPython is handling > the problem - mostly hiding all the ugly details behind a nice API - > would make it possible to have the new and old version nicely in > parallel, especially as the new version has already mostly been written, > for the case if IPython is not running in the terminal. The other issue is that we still support Python 2.7, thus anything that targets 3.x only is dangerous and subject to controversy. Though the discussion to drop 2.7 support for some functionality do arise more and more often. This might be a case where maybe dropping 2.7 might help, or where such a feature can push toward a drop of 2. support. It's not my area of expertise so I don't know. From the discussion on Python bug tracker with Guido, and reading in between lines, I also would say that having such a feature in IPython might help to push this into Python itself later. And having async console looks cool. > > Greetings > > Martin > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev [1] https://github.com/ipython/ipython/wiki/IPEPs:-IPython-Enhancement-Proposals, don't worry much more informal than Python one. -------------- next part -------------- An HTML attachment was scrubbed... URL: From oren.a4 at gmail.com Wed Oct 1 07:35:29 2014 From: oren.a4 at gmail.com (Oren) Date: Wed, 1 Oct 2014 14:35:29 +0300 Subject: [IPython-dev] How to paste large code text using %cpaste in ipython? Message-ID: I am having the same problem as the one described in the following link http://stackoverflow.com/questions/23765349/how-to-paste-large-code-text-using-cpaste-in-ipython "I'm trying to paste code into ipython using %cpaste, but whenever the pasted code is too large (almost 100-150 lines of code), it stops actually reading the input buffer and therefore gives me an error similar to: File "", line 133 return(SyntaxError: unexpected EOF while parsing If it matters, I'm using a bash inside an emacs and running ipython3 -i from there. *EDIT:* interestingly if I paste half of the code, hit enter, paste the second half, press enter, then --, then enter, it processes it properly. " Is there a way to fix it? Thank You, Oren. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lkb.teichmann at gmail.com Wed Oct 1 10:47:04 2014 From: lkb.teichmann at gmail.com (Martin Teichmann) Date: Wed, 1 Oct 2014 16:47:04 +0200 Subject: [IPython-dev] Asyncio extensions for IPython In-Reply-To: References: Message-ID: Hi, > I noticed you've put it under a GPL license. Much of this ecosystem is BSD ... I didn't think much about this, so I relicensed. Hopefully one day this extension will go into IPython. Greetings Martin From takowl at gmail.com Wed Oct 1 12:44:47 2014 From: takowl at gmail.com (Thomas Kluyver) Date: Wed, 1 Oct 2014 09:44:47 -0700 Subject: [IPython-dev] IPython inputhook, higher rate? In-Reply-To: References: Message-ID: Hi Martin, On 1 October 2014 00:39, Martin Teichmann wrote: > To me it's a bit weird to say that we shouldn't push that straight into > core > Python - IPython is working around this broken functionality since more > than a decade, so how long do we have to wait until something is not > considered "straight" anymore? Apparently IPython is not the only one > having this problem, as you mentioned others have also already written > a library to circumvent this problem in the standard library. > InputHook has been around for a long time, but only recently has there been a standardised event loop interface in the stdlib that it could use instead. And integrating input() with that is presumably not trivial, so it makes sense to work out the kinks in a separate package, not in a Python builtin function. > As for compatibility: I actually think that the way IPython is handling > the problem - mostly hiding all the ugly details behind a nice API - > would make it possible to have the new and old version nicely in > parallel, especially as the new version has already mostly been written, > for the case if IPython is not running in the terminal. It probably could, but we'll probably still have to maintain the old API for quite some time; this is true whether it's a separate package depending on asyncio or a new Python feature. The real value for us would be being able to drop the inputhook stuff altogether, which seems unlikely for the foreseeable future. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Wed Oct 1 13:03:20 2014 From: takowl at gmail.com (Thomas Kluyver) Date: Wed, 1 Oct 2014 10:03:20 -0700 Subject: [IPython-dev] How to paste large code text using %cpaste in ipython? In-Reply-To: References: Message-ID: Hi Oren, We have very limited control over what happens when you paste in the terminal, because it looks just like that text is being typed in very quickly. I can't see how that would cause the symptoms you describe, though. Thomas On 1 October 2014 04:35, Oren wrote: > I am having the same problem as the one described in the following link > > > > http://stackoverflow.com/questions/23765349/how-to-paste-large-code-text-using-cpaste-in-ipython > > "I'm trying to paste code into ipython using %cpaste, but whenever the > pasted code is too large (almost 100-150 lines of code), it stops actually > reading the input buffer and therefore gives me an error similar to: > > File "", line 133 > return(SyntaxError: unexpected EOF while parsing > > If it matters, I'm using a bash inside an emacs and running ipython3 -i from > there. > > *EDIT:* interestingly if I paste half of the code, hit enter, paste the > second half, press enter, then --, then enter, it processes it properly. > > " > > Is there a way to fix it? > > Thank You, > Oren. > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Wed Oct 1 13:03:54 2014 From: takowl at gmail.com (Thomas Kluyver) Date: Wed, 1 Oct 2014 10:03:54 -0700 Subject: [IPython-dev] Asyncio extensions for IPython In-Reply-To: References: Message-ID: Thanks Martin! Thomas On 1 October 2014 07:47, Martin Teichmann wrote: > Hi, > > > I noticed you've put it under a GPL license. Much of this ecosystem is > BSD ... > > I didn't think much about this, so I relicensed. Hopefully one day this > extension will go into IPython. > > Greetings > > Martin > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.m.bray at gmail.com Wed Oct 1 13:25:57 2014 From: erik.m.bray at gmail.com (Erik Bray) Date: Wed, 1 Oct 2014 13:25:57 -0400 Subject: [IPython-dev] IPython inputhook, higher rate? In-Reply-To: References: Message-ID: Hi Martin, On Tue, Sep 30, 2014 at 11:05 AM, Martin Teichmann wrote: > Hi, > > I just saw this thread about why and how the python > inputhook works, and how to integrate other event loops > with IPython, using said input hooks. > > The actual problem is that the python input routine blocks > while it reads from the console. This is completely > unnecessary, as the underlying GNU readline library does > not require a block at all. Down in the python interpreter, > something like the following happens (it's in C in original). > > while True: > PyOS_InputHook() > select(stdin, timeout=100) > if something_in_stdin: > call_GNU_readline_to_read_one_char() > > so, python defines its own event loop! As one can see > very clearly, from the IPython standpoint this is completely > stupid: we want our own event loop to run (eg, the Qt > event loop), and once something arrives at stdin, call > GNU readline to process whatever arrived. > > I already wrote the necessary changes and already put > up some patch on the python issue tracker: > http://bugs.python.org/issue22412 I'm sorry your patch got rejected--it looks like something you put a lot of thought into! That said, reading over the thread, I agree with the rationale and it seemed to come with a "for now" attached to it, as in, it's not like it's a bad idea. And indeed Python's REPL event loop has been controversial for a long time now (though to say it's not needed is also incorrect at least on some major platforms...) Relatedly, if you like horror stories, you should see the time I had to reimplement raw_input and replace it in the builtins module since it would fall over and *always* block if sys.stdin was replaced: https://gist.github.com/embray/3dc1dcb0d264bce912fb (Fortunately this issue is fixed on Python 3 at least.) I agree that having something like your asyncio console in a third-party library sounds interesting, and might also look nice as part of IPython (though I defer to the IPython/Jupyter devs on that of course ;) Erik From fperez.net at gmail.com Wed Oct 1 14:26:10 2014 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 1 Oct 2014 11:26:10 -0700 Subject: [IPython-dev] Asyncio extensions for IPython In-Reply-To: References: Message-ID: On Wed, Oct 1, 2014 at 7:47 AM, Martin Teichmann wrote: > > I noticed you've put it under a GPL license. Much of this ecosystem is > BSD ... > > I didn't think much about this, so I relicensed. Hopefully one day this > extension will go into IPython. > Much appreciated Martin! -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Wed Oct 1 14:28:18 2014 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 1 Oct 2014 11:28:18 -0700 Subject: [IPython-dev] py.test plugin for running Notebook cells as unit tests In-Reply-To: References: Message-ID: Great! Could you list it in the extensions index? https://github.com/ipython/ipython/wiki/Extensions-Index Cheers f On Tue, Sep 30, 2014 at 3:23 PM, Andrea Zonca wrote: > Hi, > I wrote a py.test plugin that discovers and runs unit tests written > as cells of IPython notebooks, it also supports fixtures, see: > > https://github.com/zonca/pytest-ipynb > http://zonca.github.io/2014/09/unit-tests-ipython-notebook.html > > Cheers, > Andrea > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Wed Oct 1 20:22:46 2014 From: takowl at gmail.com (Thomas Kluyver) Date: Wed, 1 Oct 2014 17:22:46 -0700 Subject: [IPython-dev] IPython 2.3 released Message-ID: IPython 2.3, a bugfix release for the 2.x series, has been released. The most important fix is for a bug that broke the rendering of maths when notebooks were converted to HTML using nbconvert. As usual, a number of other bugfixes have been backported from master for this release. You can see the full list here: http://ipython.org/ipython-doc/2/whatsnew/github-stats-2.0.html#issues-closed-in-2-3-0 Thanks, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Thu Oct 2 01:57:06 2014 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 1 Oct 2014 22:57:06 -0700 Subject: [IPython-dev] Heads up: shutting down hipchat Message-ID: Hi all, given we seem to be pretty happy using Gitter, I'm shutting down the hipchat accounts for everyone so we start paying for that. Just a heads-up in case you wonder why you get an email from hipchat about your user being deleted... cheers f -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail -------------- next part -------------- An HTML attachment was scrubbed... URL: From hughesadam87 at gmail.com Thu Oct 2 15:05:34 2014 From: hughesadam87 at gmail.com (Adam Hughes) Date: Thu, 2 Oct 2014 15:05:34 -0400 Subject: [IPython-dev] Resetting widget application view/state Message-ID: Hi, Following the helpful discussion on widgets last month, we've made good progress with our spectroscopy widgets and are very happy! One issue we're having is that we want the user to be able to change which datateset he/she is piping is working with. Different datasets have different default values, for example spectral units, start and end ranges etc... Right now, our application's view and model are completely separate; however, when the user chooses to work with a new dataset, we'd like the view to reset itself based on the new ranges for some of these attributes. We were hoping there was a smart way to do this! Even if it's a total hack, that's fine. Here is the notebook: http://nbviewer.ipython.org/urls/raw.github.com/hugadams/pyuvvis/master/pyuvvis/interact/ipynbs/prototypes/widgets_slicing.ipynb I've attached a ss of the ouput The _filename_changed trait is where this new functionality should go. Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: widget_ss.png Type: image/png Size: 70255 bytes Desc: not available URL: From hughesadam87 at gmail.com Thu Oct 2 15:07:02 2014 From: hughesadam87 at gmail.com (Adam Hughes) Date: Thu, 2 Oct 2014 15:07:02 -0400 Subject: [IPython-dev] Resetting widget application view/state In-Reply-To: References: Message-ID: Just to clarigy, the "Units" and "Sampling" widgets are the ones that ought to reset their defaults based on the new dataset, which contains attributes for its units and sampling is determined by the range of the index values. On Thu, Oct 2, 2014 at 3:05 PM, Adam Hughes wrote: > Hi, > > Following the helpful discussion on widgets last month, we've made good > progress with our spectroscopy widgets and are very happy! > > One issue we're having is that we want the user to be able to change which > datateset he/she is piping is working with. Different datasets have > different default values, for example spectral units, start and end ranges > etc... > > Right now, our application's view and model are completely separate; > however, when the user chooses to work with a new dataset, we'd like the > view to reset itself based on the new ranges for some of these attributes. > We were hoping there was a smart way to do this! Even if it's a total > hack, that's fine. > > Here is the notebook: > > > http://nbviewer.ipython.org/urls/raw.github.com/hugadams/pyuvvis/master/pyuvvis/interact/ipynbs/prototypes/widgets_slicing.ipynb > > I've attached a ss of the ouput > > The _filename_changed trait is where this new functionality should go. > > Thanks > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Andy at SantaCruzIntegration.com Thu Oct 2 19:17:34 2014 From: Andy at SantaCruzIntegration.com (Andy Davidson) Date: Thu, 02 Oct 2014 16:17:34 -0700 Subject: [IPython-dev] ERROR can not use fork(), exec(), dup2() to communicate with child process Message-ID: Hi I want to write a notebook that receives data from another process. Bellow is my python code. It forks a child process and reads the data from the child using a pipe. This code works if I run it in python. When I try to execute from iPython notebook I get the following error. Any idea what the problems is ? If pipes do not work, can I use a socket? The other process would send me data by writing, my notebook needs behave like a server. Any comments or suggestions would be greatly appreciated. Thanks Andy P.s. I am running headless on a remote machine and use an ssh tunnel to run the notebook from my local browser. My other notebooks works fine [ec2-user at ip-172-31-14-107 ~]$ IPYTHON_OPTS="notebook --pylab inline --no-browser --port=7000" /root/spark/bin/pyspark /root/spark/conf/spark-env.sh: line 21: ulimit: open files: cannot modify limit: Operation not permitted [NotebookApp] Using existing profile dir: u'/home/ec2-user/.ipython/profile_default' [NotebookApp] Serving notebooks from /home/ec2-user [NotebookApp] The IPython Notebook is running at: http://127.0.0.1:7000/ [NotebookApp] Use Control-C to stop this server and shut down all kernels. [NotebookApp] Using MathJax from CDN: http://cdn.mathjax.org/mathjax/latest/MathJax.js [NotebookApp] Kernel started: 71f07d5c-2db1-465f-9d86-da29f8711a0b [NotebookApp] Connecting to: tcp://127.0.0.1:54119 [NotebookApp] Connecting to: tcp://127.0.0.1:50649 [NotebookApp] Connecting to: tcp://127.0.0.1:35516 [IPKernelApp] To connect another client to this kernel, use: [IPKernelApp] --existing kernel-71f07d5c-2db1-465f-9d86-da29f8711a0b.json Assertion failed: ok (bundled/zeromq/src/mailbox.cpp:79) [NotebookApp] Kernel 71f07d5c-2db1-465f-9d86-da29f8711a0b failed to respond to heartbeat import sys import os def child(pipeout): os.dup2(pipeout, sys.stdout.fileno()) os.close(pipeout) os.execlp("./test.sh", "abc") def parent(): pipein, pipeout = os.pipe() if os.fork() == 0: os.close(pipein) child(pipeout) else: os.close(pipeout) counter = 1 pipein = os.fdopen(pipein) while True: json = pipein.readline()[:-1] print 'parent: %s' % (json) parent() $ cat test.sh #!/bin/bash script_name=$0 args=$@ current=`pwd` echo $script_name $args echo "XYZ: "$XYZ echo "PATH: "$PATH echo "current directory: $current" -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjaminrk at gmail.com Thu Oct 2 20:19:18 2014 From: benjaminrk at gmail.com (MinRK) Date: Thu, 2 Oct 2014 17:19:18 -0700 Subject: [IPython-dev] ERROR can not use fork(), exec(), dup2() to communicate with child process In-Reply-To: References: Message-ID: why use execlp instead of subprocess.Popen, which allows capturing stdout, stderr, etc.? ? On Thu, Oct 2, 2014 at 4:17 PM, Andy Davidson wrote: > Hi > > > I want to write a notebook that receives data from another process. Bellow > is my python code. It forks a child process and reads the data from the > child using a pipe. This code works if I run it in python. When I try to > execute from iPython notebook I get the following error. Any idea what the > problems is ? > > > If pipes do not work, can I use a socket? The other process would send me > data by writing, my notebook needs behave like a server. > > > Any comments or suggestions would be greatly appreciated. > > > Thanks > > > Andy > > > P.s. I am running headless on a remote machine and use an ssh tunnel to > run the notebook from my local browser. My other notebooks works fine > > > > [ec2-user at ip-172-31-14-107 ~]$ IPYTHON_OPTS="notebook --pylab inline > --no-browser --port=7000" /root/spark/bin/pyspark > > /root/spark/conf/spark-env.sh: line 21: ulimit: open files: cannot modify > limit: Operation not permitted > > [NotebookApp] Using existing profile dir: > u'/home/ec2-user/.ipython/profile_default' > > [NotebookApp] Serving notebooks from /home/ec2-user > > [NotebookApp] The IPython Notebook is running at: http://127.0.0.1:7000/ > > [NotebookApp] Use Control-C to stop this server and shut down all kernels. > > [NotebookApp] Using MathJax from CDN: > http://cdn.mathjax.org/mathjax/latest/MathJax.js > > [NotebookApp] Kernel started: 71f07d5c-2db1-465f-9d86-da29f8711a0b > > [NotebookApp] Connecting to: tcp://127.0.0.1:54119 > > [NotebookApp] Connecting to: tcp://127.0.0.1:50649 > > [NotebookApp] Connecting to: tcp://127.0.0.1:35516 > > [IPKernelApp] To connect another client to this kernel, use: > > [IPKernelApp] --existing kernel-71f07d5c-2db1-465f-9d86-da29f8711a0b.json > > Assertion failed: ok (bundled/zeromq/src/mailbox.cpp:79) > > [NotebookApp] Kernel 71f07d5c-2db1-465f-9d86-da29f8711a0b failed to > respond to heartbeat > > > > import sys > import os > > def child(pipeout): > os.dup2(pipeout, sys.stdout.fileno()) > os.close(pipeout) > os.execlp("./test.sh", "abc") > > > def parent(): > pipein, pipeout = os.pipe() > if os.fork() == 0: > os.close(pipein) > child(pipeout) > else: > os.close(pipeout) > counter = 1 > pipein = os.fdopen(pipein) > while True: > json = pipein.readline()[:-1] > print 'parent: %s' % (json) > > parent() > > $ cat test.sh > > #!/bin/bash > > > script_name=$0 > > args=$@ > > current=`pwd` > > echo $script_name $args > > echo "XYZ: "$XYZ > > echo "PATH: "$PATH > > echo "current directory: $current" > > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From claresloggett at gmail.com Fri Oct 3 00:42:54 2014 From: claresloggett at gmail.com (Clare Sloggett) Date: Fri, 3 Oct 2014 14:42:54 +1000 Subject: [IPython-dev] Multi-user IPython Notebook Message-ID: Hi all, I am wondering if anyone knows the time for the multi-user IPython Notebook - i.e. the concept where, as I understand it, it will run as a daemon and use linux accounts for login. I'm really looking forward to this. I can see it mentioned in a couple of places: https://github.com/ipython/ipython/wiki/IPEP-3:-Multiuser-support-in-the-notebook https://github.com/ipython/ipython/wiki/Roadmap:-IPython#overview-of-2013-2014 But the timeline is just "2014". Is there any more updated info? I'm doing some work at the moment that would be unnecessary if this were out so if it's coming out soon, I will wait! Thanks, Clare -------------- next part -------------- An HTML attachment was scrubbed... URL: From ssanderson at quantopian.com Fri Oct 3 00:51:39 2014 From: ssanderson at quantopian.com (ssanderson) Date: Thu, 2 Oct 2014 21:51:39 -0700 (PDT) Subject: [IPython-dev] Multi-user IPython Notebook In-Reply-To: References: Message-ID: <1412311899323-5073521.post@n6.nabble.com> The multi-user notebook project now goes by the name jupyterhub. It lives at https://github.com/jupyter/jupyterhub/tree/master/jupyterhub. I'm not sure what it's status is w/r/t stability and concrete/public APIs, but it's definitely usable already. -- View this message in context: http://python.6.x6.nabble.com/Multi-user-IPython-Notebook-tp5073520p5073521.html Sent from the IPython - Development mailing list archive at Nabble.com. From benjaminrk at gmail.com Fri Oct 3 01:15:29 2014 From: benjaminrk at gmail.com (MinRK) Date: Thu, 2 Oct 2014 22:15:29 -0700 Subject: [IPython-dev] Multi-user IPython Notebook In-Reply-To: <1412311899323-5073521.post@n6.nabble.com> References: <1412311899323-5073521.post@n6.nabble.com> Message-ID: JupyterHub is basically functional, right now. We are still working on some configuration, with respect to running the server as a limited user. But if you run the server as root and you want to provide access to existing system users, it works. We also have working examples using GitHub OAuth for user login instead of system passwords, as well as running user code in docker containers . It?s pretty rough right now, and there?s a fair amount of UI and error handling to deal with (and documenting how to use it, importantly), but it would be great to get some feedback! -MinRK ? On Thu, Oct 2, 2014 at 9:51 PM, ssanderson wrote: > The multi-user notebook project now goes by the name jupyterhub. It lives > at > https://github.com/jupyter/jupyterhub/tree/master/jupyterhub. I'm not > sure > what it's status is w/r/t stability and concrete/public APIs, but it's > definitely usable already. > > > > -- > View this message in context: > http://python.6.x6.nabble.com/Multi-user-IPython-Notebook-tp5073520p5073521.html > Sent from the IPython - Development mailing list archive at Nabble.com. > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bobd at stanford.edu Fri Oct 3 01:44:07 2014 From: bobd at stanford.edu (Bob Dougherty) Date: Thu, 2 Oct 2014 22:44:07 -0700 (PDT) Subject: [IPython-dev] Multi-user IPython Notebook In-Reply-To: References: <1412311899323-5073521.post@n6.nabble.com> Message-ID: <634869803.9727722.1412315047475.JavaMail.zimbra@stanford.edu> It's definitely usable-- we're running our class tutorials through JupyterHub using the default pam authentication module. We pre-configured linux accounts on the server for each student. Over 20 students have been accessing the notebook tutorials through a JubyterHub process that's been running for a few weeks straight and it seems to be working fine for them. It's all running on a dedicated VM as root and are using ssl for security. cheers, bob -- Robert F. Dougherty, PhD Research Director Stanford Center for Cognitive and Neurobiological Imaging 70 Jordan Hall * Stanford CA 94305 * 650-725-0051 http://www.stanford.edu/~bobd ----- Original Message ----- From: "MinRK" To: "IPython developers list" Sent: Thursday, October 2, 2014 10:15:29 PM Subject: Re: [IPython-dev] Multi-user IPython Notebook JupyterHub is basically functional, right now. We are still working on some configuration, with respect to running the server as a limited user. But if you run the server as root and you want to provide access to existing system users, it works. We also have working examples using GitHub OAuth for user login instead of system passwords, as well as running user code in docker containers . It?s pretty rough right now, and there?s a fair amount of UI and error handling to deal with (and documenting how to use it, importantly), but it would be great to get some feedback! -MinRK ? On Thu, Oct 2, 2014 at 9:51 PM, ssanderson < ssanderson at quantopian.com > wrote: The multi-user notebook project now goes by the name jupyterhub. It lives at https://github.com/jupyter/jupyterhub/tree/master/jupyterhub . I'm not sure what it's status is w/r/t stability and concrete/public APIs, but it's definitely usable already. -- View this message in context: http://python.6.x6.nabble.com/Multi-user-IPython-Notebook-tp5073520p5073521.html Sent from the IPython - Development mailing list archive at Nabble.com. _______________________________________________ IPython-dev mailing list IPython-dev at scipy.org http://mail.scipy.org/mailman/listinfo/ipython-dev _______________________________________________ IPython-dev mailing list IPython-dev at scipy.org http://mail.scipy.org/mailman/listinfo/ipython-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From ssanderson at quantopian.com Fri Oct 3 02:26:45 2014 From: ssanderson at quantopian.com (ssanderson) Date: Thu, 2 Oct 2014 23:26:45 -0700 (PDT) Subject: [IPython-dev] Multi-user IPython Notebook In-Reply-To: References: <1412311899323-5073521.post@n6.nabble.com> Message-ID: <1412317605582-5073526.post@n6.nabble.com> I wasn't aware of the existence of jupyter/oauthenticator and jupyter/dockerspawner. Is the plan for future extensions of this sort to be broken out into their own separate repositories? I would have naively expected all the "official" jupyterhub spawners to be part of the main repo. -- View this message in context: http://python.6.x6.nabble.com/Multi-user-IPython-Notebook-tp5073520p5073526.html Sent from the IPython - Development mailing list archive at Nabble.com. From bussonniermatthias at gmail.com Fri Oct 3 02:44:58 2014 From: bussonniermatthias at gmail.com (Matthias BUSSONNIER) Date: Fri, 3 Oct 2014 08:44:58 +0200 Subject: [IPython-dev] Multi-user IPython Notebook In-Reply-To: <1412317605582-5073526.post@n6.nabble.com> References: <1412311899323-5073521.post@n6.nabble.com> <1412317605582-5073526.post@n6.nabble.com> Message-ID: Le 3 oct. 2014 ? 08:26, ssanderson a ?crit : > I wasn't aware of the existence of jupyter/oauthenticator and > jupyter/dockerspawner. Is the plan for future extensions of this sort to be > broken out into their own separate repositories? I would have naively > expected all the "official" jupyterhub spawners to be part of the main repo. It's not because it's in a different repo that it is not official. And yes we will try to work in separate repo if possible. It is easier. At install time, it is easy to just ask other repo to be requirements. We will also split IPython into smaller part/many repo. -- M > > > > -- > View this message in context: http://python.6.x6.nabble.com/Multi-user-IPython-Notebook-tp5073520p5073526.html > Sent from the IPython - Development mailing list archive at Nabble.com. > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev From ssanderson at quantopian.com Fri Oct 3 04:24:14 2014 From: ssanderson at quantopian.com (ssanderson) Date: Fri, 3 Oct 2014 01:24:14 -0700 (PDT) Subject: [IPython-dev] Interactive, Graphical, Diffable Pstats Viewer for IPython Notebook Message-ID: <1412324654616-5073531.post@n6.nabble.com> Hi all, I got frustrated this week trying to work with python's built-in pstats module for displaying cProfile results. The result of that frustration is now https://github.com/ssanderson/pstats-view, a small module for displaying and diffing cProfile runs in IPython Notebook. I was pleasantly surprised at how easy the `interactive` interface in the notebook made it to replicate most of the command-line functionality in a GUI. Anyway, I figured I'd share in case anyone else shares my dislike of the default pstats module. -Scott -- View this message in context: http://python.6.x6.nabble.com/Interactive-Graphical-Diffable-Pstats-Viewer-for-IPython-Notebook-tp5073531.html Sent from the IPython - Development mailing list archive at Nabble.com. From doug.blank at gmail.com Fri Oct 3 06:34:08 2014 From: doug.blank at gmail.com (Doug Blank) Date: Fri, 3 Oct 2014 06:34:08 -0400 Subject: [IPython-dev] Multi-user IPython Notebook In-Reply-To: <634869803.9727722.1412315047475.JavaMail.zimbra@stanford.edu> References: <1412311899323-5073521.post@n6.nabble.com> <634869803.9727722.1412315047475.JavaMail.zimbra@stanford.edu> Message-ID: I concur with Bob that it is in a state that is very usable. I've been teaching my Programming Languages course (mostly in Scheme) from jupyterhub and ipython master git (yes, a bit risky, not the recommended mode of operation. But great rewards!). All of the assignments, notes, and materials have been straight from notebooks. See my public notebooks here [1]. We hope to package up that public page code to share---jupyterhub doesn't currently come with a means to serve rendered pages, but has been suggested [2]. I'm now helping our Physics department get their own server setup. As part of that project, we are making extensive notes to help those non-sys admin types be able to get up and running. Will share notes those when available. If anyone would like to take a test drive, please contact me and we'll see about making you an account. -Doug [1] - http://jupyter.cs.brynmawr.edu/hub/dblank/public [2] - https://github.com/jupyter/jupyterhub/issues/17 On Fri, Oct 3, 2014 at 1:44 AM, Bob Dougherty wrote: > It's definitely usable-- we're running our class tutorials through > JupyterHub using the default pam authentication module. We pre-configured > linux accounts on the server for each student. Over 20 students have been > accessing the notebook tutorials through a JubyterHub process that's been > running for a few weeks straight and it seems to be working fine for them. > It's all running on a dedicated VM as root and are using ssl for security. > > cheers, > bob > > -- > Robert F. Dougherty, PhD > Research Director > Stanford Center for Cognitive and Neurobiological Imaging > 70 Jordan Hall * Stanford CA 94305 * 650-725-0051 > http://www.stanford.edu/~bobd > > ------------------------------ > *From: *"MinRK" > *To: *"IPython developers list" > *Sent: *Thursday, October 2, 2014 10:15:29 PM > *Subject: *Re: [IPython-dev] Multi-user IPython Notebook > > > JupyterHub is basically > functional, right now. We are still working on some configuration, with > respect to running the server as a limited user. But if you run the server > as root and you want to provide access to existing system users, it works. > We also have working examples using GitHub OAuth > for user login instead of > system passwords, as well as running user code in docker containers > . > > It?s pretty rough right now, and there?s a fair amount of UI and error > handling to deal with (and documenting how to use it, importantly), but it > would be great to get some feedback! > > -MinRK > ? > > On Thu, Oct 2, 2014 at 9:51 PM, ssanderson > wrote: > >> The multi-user notebook project now goes by the name jupyterhub. It >> lives at >> https://github.com/jupyter/jupyterhub/tree/master/jupyterhub. I'm not >> sure >> what it's status is w/r/t stability and concrete/public APIs, but it's >> definitely usable already. >> >> >> >> -- >> View this message in context: >> http://python.6.x6.nabble.com/Multi-user-IPython-Notebook-tp5073520p5073521.html >> Sent from the IPython - Development mailing list archive at Nabble.com. >> _______________________________________________ >> IPython-dev mailing list >> IPython-dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/ipython-dev >> > > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From claresloggett at gmail.com Fri Oct 3 08:09:55 2014 From: claresloggett at gmail.com (Clare Sloggett) Date: Fri, 3 Oct 2014 22:09:55 +1000 Subject: [IPython-dev] Multi-user IPython Notebook In-Reply-To: References: <1412311899323-5073521.post@n6.nabble.com> <634869803.9727722.1412315047475.JavaMail.zimbra@stanford.edu> Message-ID: Thanks MinRK and everyone, I didn't realise this - great news. I'll try it out! On 3 October 2014 20:34, Doug Blank wrote: > I concur with Bob that it is in a state that is very usable. I've been > teaching my Programming Languages course (mostly in Scheme) from jupyterhub > and ipython master git (yes, a bit risky, not the recommended mode of > operation. But great rewards!). All of the assignments, notes, and > materials have been straight from notebooks. See my public notebooks here > [1]. We hope to package up that public page code to share---jupyterhub > doesn't currently come with a means to serve rendered pages, but has been > suggested [2]. > > I'm now helping our Physics department get their own server setup. As part > of that project, we are making extensive notes to help those non-sys admin > types be able to get up and running. Will share notes those when available. > > If anyone would like to take a test drive, please contact me and we'll see > about making you an account. > > -Doug > > [1] - http://jupyter.cs.brynmawr.edu/hub/dblank/public > [2] - https://github.com/jupyter/jupyterhub/issues/17 > > On Fri, Oct 3, 2014 at 1:44 AM, Bob Dougherty wrote: > >> It's definitely usable-- we're running our class tutorials through >> JupyterHub using the default pam authentication module. We pre-configured >> linux accounts on the server for each student. Over 20 students have been >> accessing the notebook tutorials through a JubyterHub process that's been >> running for a few weeks straight and it seems to be working fine for them. >> It's all running on a dedicated VM as root and are using ssl for security. >> >> cheers, >> bob >> >> -- >> Robert F. Dougherty, PhD >> Research Director >> Stanford Center for Cognitive and Neurobiological Imaging >> 70 Jordan Hall * Stanford CA 94305 * 650-725-0051 >> http://www.stanford.edu/~bobd >> >> ------------------------------ >> *From: *"MinRK" >> *To: *"IPython developers list" >> *Sent: *Thursday, October 2, 2014 10:15:29 PM >> *Subject: *Re: [IPython-dev] Multi-user IPython Notebook >> >> >> JupyterHub is basically >> functional, right now. We are still working on some configuration, with >> respect to running the server as a limited user. But if you run the server >> as root and you want to provide access to existing system users, it works. >> We also have working examples using GitHub OAuth >> for user login instead of >> system passwords, as well as running user code in docker containers >> . >> >> It?s pretty rough right now, and there?s a fair amount of UI and error >> handling to deal with (and documenting how to use it, importantly), but it >> would be great to get some feedback! >> >> -MinRK >> ? >> >> On Thu, Oct 2, 2014 at 9:51 PM, ssanderson >> wrote: >> >>> The multi-user notebook project now goes by the name jupyterhub. It >>> lives at >>> https://github.com/jupyter/jupyterhub/tree/master/jupyterhub. I'm not >>> sure >>> what it's status is w/r/t stability and concrete/public APIs, but it's >>> definitely usable already. >>> >>> >>> >>> -- >>> View this message in context: >>> http://python.6.x6.nabble.com/Multi-user-IPython-Notebook-tp5073520p5073521.html >>> Sent from the IPython - Development mailing list archive at Nabble.com. >>> _______________________________________________ >>> IPython-dev mailing list >>> IPython-dev at scipy.org >>> http://mail.scipy.org/mailman/listinfo/ipython-dev >>> >> >> >> _______________________________________________ >> IPython-dev mailing list >> IPython-dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/ipython-dev >> >> >> _______________________________________________ >> IPython-dev mailing list >> IPython-dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/ipython-dev >> >> > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rgbkrk at gmail.com Fri Oct 3 10:54:43 2014 From: rgbkrk at gmail.com (Kyle Kelley) Date: Fri, 3 Oct 2014 09:54:43 -0500 Subject: [IPython-dev] Multi-user IPython Notebook In-Reply-To: <1412317605582-5073526.post@n6.nabble.com> References: <1412311899323-5073521.post@n6.nabble.com> <1412317605582-5073526.post@n6.nabble.com> Message-ID: > I wasn't aware of the existence of jupyter/oauthenticator and > jupyter/dockerspawner. That's probably because these were only developed in the last couple of weeks and are still in formation. :) We're still ironing out the best way to handle containerized notebook servers. It seems it is probably time to announce the sandbox project that has been in quiet development. ;) > Is the plan for future extensions of this sort to be > broken out into their own separate repositories? I would have naively > expected all the "official" jupyterhub spawners to be part of the main repo. There may come a day, but the OAuthenticator and the Dockerspawner are both regarded as example of custom auth and spawning at this point. We'll see what happens! -- Kyle On Fri, Oct 3, 2014 at 1:26 AM, ssanderson wrote: > I wasn't aware of the existence of jupyter/oauthenticator and > jupyter/dockerspawner. Is the plan for future extensions of this sort to > be > broken out into their own separate repositories? I would have naively > expected all the "official" jupyterhub spawners to be part of the main > repo. > > > > -- > View this message in context: > http://python.6.x6.nabble.com/Multi-user-IPython-Notebook-tp5073520p5073526.html > Sent from the IPython - Development mailing list archive at Nabble.com. > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -- Kyle Kelley (@rgbkrk ; http://lambdaops.com) -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.yee at gmail.com Fri Oct 3 11:27:00 2014 From: raymond.yee at gmail.com (Raymond Yee) Date: Fri, 3 Oct 2014 08:27:00 -0700 Subject: [IPython-dev] [IPythokon-dev] Multi-user IPython Noteboo Message-ID: Lo On Oct 3, 2014 7:54 AM, "Kyle Kelley" wrote: > > I wasn't aware of the existence of jupyter/oauthenticator and > > jupyter/dockerspawner. > > That's probably because these were only developed in the last couple of > weeks and are still in formation. :) We're still ironing out the best way > to handle containerized notebook servers. It seems it is probably time to > announce the sandbox project that has been in quiet development. ;) > > > Is the plan for future extensions of this sort to be > > broken out into their own separate repositories? I would have naively > > expected all the "official" jupyterhub spawners to be part of the main > repo. > > There may come a day, but the OAuthenticator and the Dockerspawner are > both regarded as example of custom auth and spawning at this point. We'll > see what happens! > > -- Kyle > > > On Fri, Oct 3, 2014 at 1:26 AM, ssanderson > wrote: > >> I wasn't aware of the existence of jupyter/oauthenticator and >> jupyter/dockerspawner. Is the plan for future extensions of this sort to >> be >> broken out into their own separate repositories? I would have naively >> expected all the "official" jupyterhub spawners to be part of the main >> repo. >> >> >> >> -- >> View this message in context: >> http://python.6.x6.nabble.com/Multi-user-IPython-Notebook-tp5073520p5073526.html >> Sent from the IPython - Development mailing list archive at Nabble.com. >> _______________________________________________ >> IPython-dev mailing list >> IPython-dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/ipython-dev >> > > > > -- > Kyle Kelley (@rgbkrk ; http://lambdaops.com) > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ssanderson at quantopian.com Fri Oct 3 11:44:30 2014 From: ssanderson at quantopian.com (ssanderson) Date: Fri, 3 Oct 2014 08:44:30 -0700 (PDT) Subject: [IPython-dev] Multi-user IPython Notebook In-Reply-To: References: <1412311899323-5073521.post@n6.nabble.com> <1412317605582-5073526.post@n6.nabble.com> Message-ID: <1412351070200-5073572.post@n6.nabble.com> Kyle Kelley wrote > It seems it is probably time to announce the sandbox project that has been > in quiet development. ;) Interesting. Your sandbox project sounds very similar to a project we have going over at Quantopian; we're building a notebook-based financial research environment for our users. We're also using Docker containers for notebook sandboxing and performance isolation. Currently we're managing deployments of single-user servers with a homegrown prototype, but in the next few weeks we're going to investigate switching to Jupyterhub. The DockerSpawner looks very much like what I had sketched out in a to-do list of custom extensions we would have had to write ourselves :). I'd love to compare notes at some point on what you guys have planned vs our roadmap; we might be able to avoid some duplicated work that way. -Scott -- View this message in context: http://python.6.x6.nabble.com/Multi-user-IPython-Notebook-tp5073520p5073572.html Sent from the IPython - Development mailing list archive at Nabble.com. From rgbkrk at gmail.com Fri Oct 3 12:25:15 2014 From: rgbkrk at gmail.com (Kyle Kelley) Date: Fri, 3 Oct 2014 11:25:15 -0500 Subject: [IPython-dev] Multi-user IPython Notebook In-Reply-To: <1412351070200-5073572.post@n6.nabble.com> References: <1412311899323-5073521.post@n6.nabble.com> <1412317605582-5073526.post@n6.nabble.com> <1412351070200-5073572.post@n6.nabble.com> Message-ID: Scott, How are you currently locking down the Docker containers for your users? How many do you run per host? On Fri, Oct 3, 2014 at 10:44 AM, ssanderson wrote: > Kyle Kelley wrote > > It seems it is probably time to announce the sandbox project that has > been > > in quiet development. ;) > > Interesting. Your sandbox project sounds very similar to a project we have > going over at Quantopian; we're building a notebook-based financial > research > environment for our users. We're also using Docker containers for notebook > sandboxing and performance isolation. Currently we're managing deployments > of single-user servers with a homegrown prototype, but in the next few > weeks > we're going to investigate switching to Jupyterhub. The DockerSpawner > looks > very much like what I had sketched out in a to-do list of custom extensions > we would have had to write ourselves :). I'd love to compare notes at some > point on what you guys have planned vs our roadmap; we might be able to > avoid some duplicated work that way. > > -Scott > > > > -- > View this message in context: > http://python.6.x6.nabble.com/Multi-user-IPython-Notebook-tp5073520p5073572.html > Sent from the IPython - Development mailing list archive at Nabble.com. > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -- Kyle Kelley (@rgbkrk ; http://lambdaops.com) -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Fri Oct 3 13:37:57 2014 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 3 Oct 2014 10:37:57 -0700 Subject: [IPython-dev] Multi-user IPython Notebook In-Reply-To: References: <1412311899323-5073521.post@n6.nabble.com> <634869803.9727722.1412315047475.JavaMail.zimbra@stanford.edu> Message-ID: On Fri, Oct 3, 2014 at 3:34 AM, Doug Blank wrote: > I'm now helping our Physics department get their own server setup. As part > of that project, we are making extensive notes to help those non-sys admin > types be able to get up and running. Will share notes those when available. This is great! Please do share those when it makes sense, I'm sure many would find them useful. Best f -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail -------------- next part -------------- An HTML attachment was scrubbed... URL: From ssanderson at quantopian.com Fri Oct 3 14:18:50 2014 From: ssanderson at quantopian.com (ssanderson) Date: Fri, 3 Oct 2014 11:18:50 -0700 (PDT) Subject: [IPython-dev] Multi-user IPython Notebook In-Reply-To: References: <1412311899323-5073521.post@n6.nabble.com> <1412317605582-5073526.post@n6.nabble.com> <1412351070200-5073572.post@n6.nabble.com> Message-ID: <1412360330179-5073582.post@n6.nabble.com> Kyle, Currently we're spinning up new containers manually over SSH using Fabric. The container we're running is conceptually pretty similar to the scipyserver container in docker-notebook . At the moment we're not doing any sort of proxying, so each single user server just gets its own unique URL and password, which is supplied to its user. As far as how many containers we run per host, we're still playing around with different instance types and resource limits, so we haven't settled on a specific number. One of the challenges for us is that different research activities have radically different performance profiles. For example, querying historical financial data is mostly IO-bound, analyses of large datasets are often constrained by RAM, and backtesting with in-memory data is primarily CPU-bound (in my dream world, we can farm out the CPU-bound tasks to pool of worker containers using IPython Parallel, but that's a ways off). That said, I think our current ballpark is to run on the order of 10 containers per host. -- View this message in context: http://python.6.x6.nabble.com/Multi-user-IPython-Notebook-tp5073520p5073582.html Sent from the IPython - Development mailing list archive at Nabble.com. From Andy at SantaCruzIntegration.com Fri Oct 3 14:22:40 2014 From: Andy at SantaCruzIntegration.com (Andy Davidson) Date: Fri, 03 Oct 2014 11:22:40 -0700 Subject: [IPython-dev] ERROR can not use fork(), exec(), dup2() to communicate with child process Message-ID: Hi MinRK I am new to Python system programming. In my first experiments I use subprocess. Reading the document it appears subprocess it great for a setting up traditional unix style pipe like for example calling ?ls? In the notebook I aussem ! Uname ?a Works using subprocess. The subprocess documentation suggests that you need to call communicate() and that this function will not return until the child sends EOF or terminates. My child will never exit or send EOF. it will keep sending me real time data. I want to process the data one line at a time. Here is my psudo code While (True): Data = readALineFromChild() # this should block until data becomes available Graph(data) Thanks Andy > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://mail.scipy.org/pipermail/ipython-dev/attachments/20141002/aefaa3fc/atta > chment-0001.html > > ------------------------------ > > Message: 3 > Date: Thu, 2 Oct 2014 17:19:18 -0700 > From: MinRK > Subject: Re: [IPython-dev] ERROR can not use fork(), exec(), dup2() to > communicate with child process > To: IPython developers list > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > why use execlp instead of subprocess.Popen, which allows capturing stdout, > stderr, etc.? > ? > > On Thu, Oct 2, 2014 at 4:17 PM, Andy Davidson > wrote: > >> Hi >> >> >> I want to write a notebook that receives data from another process. Bellow >> is my python code. It forks a child process and reads the data from the >> child using a pipe. This code works if I run it in python. When I try to >> execute from iPython notebook I get the following error. Any idea what the >> problems is ? >> >> >> If pipes do not work, can I use a socket? The other process would send me >> data by writing, my notebook needs behave like a server. >> >> >> Any comments or suggestions would be greatly appreciated. >> >> >> Thanks >> >> >> Andy >> >> >> P.s. I am running headless on a remote machine and use an ssh tunnel to >> run the notebook from my local browser. My other notebooks works fine >> >> >> >> [ec2-user at ip-172-31-14-107 ~]$ IPYTHON_OPTS="notebook --pylab inline >> --no-browser --port=7000" /root/spark/bin/pyspark >> >> /root/spark/conf/spark-env.sh: line 21: ulimit: open files: cannot modify >> limit: Operation not permitted >> >> [NotebookApp] Using existing profile dir: >> u'/home/ec2-user/.ipython/profile_default' >> >> [NotebookApp] Serving notebooks from /home/ec2-user >> >> [NotebookApp] The IPython Notebook is running at: http://127.0.0.1:7000/ >> >> [NotebookApp] Use Control-C to stop this server and shut down all kernels. >> >> [NotebookApp] Using MathJax from CDN: >> http://cdn.mathjax.org/mathjax/latest/MathJax.js >> >> [NotebookApp] Kernel started: 71f07d5c-2db1-465f-9d86-da29f8711a0b >> >> [NotebookApp] Connecting to: tcp://127.0.0.1:54119 >> >> [NotebookApp] Connecting to: tcp://127.0.0.1:50649 >> >> [NotebookApp] Connecting to: tcp://127.0.0.1:35516 >> >> [IPKernelApp] To connect another client to this kernel, use: >> >> [IPKernelApp] --existing kernel-71f07d5c-2db1-465f-9d86-da29f8711a0b.json >> >> Assertion failed: ok (bundled/zeromq/src/mailbox.cpp:79) >> >> [NotebookApp] Kernel 71f07d5c-2db1-465f-9d86-da29f8711a0b failed to >> respond to heartbeat >> >> >> >> import sys >> import os >> >> def child(pipeout): >> os.dup2(pipeout, sys.stdout.fileno()) >> os.close(pipeout) >> os.execlp("./test.sh", "abc") >> >> >> def parent(): >> pipein, pipeout = os.pipe() >> if os.fork() == 0: >> os.close(pipein) >> child(pipeout) >> else: >> os.close(pipeout) >> counter = 1 >> pipein = os.fdopen(pipein) >> while True: >> json = pipein.readline()[:-1] >> print 'parent: %s' % (json) >> >> parent() >> >> $ cat test.sh >> >> #!/bin/bash >> >> >> script_name=$0 >> >> args=$@ >> >> current=`pwd` >> >> echo $script_name $args >> >> echo "XYZ: "$XYZ >> >> echo "PATH: "$PATH >> >> echo "current directory: $current" >> >> >> _______________________________________________ >> IPython-dev mailing list >> IPython-dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/ipython-dev >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjaminrk at gmail.com Fri Oct 3 14:26:52 2014 From: benjaminrk at gmail.com (MinRK) Date: Fri, 3 Oct 2014 11:26:52 -0700 Subject: [IPython-dev] ERROR can not use fork(), exec(), dup2() to communicate with child process In-Reply-To: References: Message-ID: You don?t need to read the whole output at once. You can get one line at a time. Or, you can use pexpect to create the subprocess, which offers nice timeout, pattern-expecting waiting for output. Depending on context, IPython will use one or the other of these for !. -MinRK ? On Fri, Oct 3, 2014 at 11:22 AM, Andy Davidson < Andy at santacruzintegration.com> wrote: > Hi MinRK > > I am new to Python system programming. In my first experiments I use > subprocess. Reading the document it appears subprocess it great for a > setting up traditional unix style pipe like for example calling ?ls? > > In the notebook I aussem > ! Uname ?a > > Works using subprocess. > > The subprocess documentation suggests that you need to call communicate() > and that this function will not return until the child sends EOF or > terminates. My child will never exit or send EOF. it will keep sending me > real time data. I want to process the data one line at a time. Here is my > psudo code > > While (True): > Data = readALineFromChild() # this should block until data becomes > available > Graph(data) > > > > Thanks > > Andy > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://mail.scipy.org/pipermail/ipython-dev/attachments/20141002/aefaa3fc/attachment-0001.html > > ------------------------------ > > Message: 3 > Date: Thu, 2 Oct 2014 17:19:18 -0700 > From: MinRK > Subject: Re: [IPython-dev] ERROR can not use fork(), exec(), dup2() to > communicate with child process > To: IPython developers list > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > why use execlp instead of subprocess.Popen, which allows capturing stdout, > stderr, etc.? > ? > > On Thu, Oct 2, 2014 at 4:17 PM, Andy Davidson < > Andy at santacruzintegration.com > > wrote: > > > Hi > > > I want to write a notebook that receives data from another process. Bellow > is my python code. It forks a child process and reads the data from the > child using a pipe. This code works if I run it in python. When I try to > execute from iPython notebook I get the following error. Any idea what the > problems is ? > > > If pipes do not work, can I use a socket? The other process would send me > data by writing, my notebook needs behave like a server. > > > Any comments or suggestions would be greatly appreciated. > > > Thanks > > > Andy > > > P.s. I am running headless on a remote machine and use an ssh tunnel to > run the notebook from my local browser. My other notebooks works fine > > > > [ec2-user at ip-172-31-14-107 ~]$ IPYTHON_OPTS="notebook --pylab inline > --no-browser --port=7000" /root/spark/bin/pyspark > > /root/spark/conf/spark-env.sh: line 21: ulimit: open files: cannot modify > limit: Operation not permitted > > [NotebookApp] Using existing profile dir: > u'/home/ec2-user/.ipython/profile_default' > > [NotebookApp] Serving notebooks from /home/ec2-user > > [NotebookApp] The IPython Notebook is running at: http://127.0.0.1:7000/ > > [NotebookApp] Use Control-C to stop this server and shut down all kernels. > > [NotebookApp] Using MathJax from CDN: > http://cdn.mathjax.org/mathjax/latest/MathJax.js > > [NotebookApp] Kernel started: 71f07d5c-2db1-465f-9d86-da29f8711a0b > > [NotebookApp] Connecting to: tcp://127.0.0.1:54119 > > [NotebookApp] Connecting to: tcp://127.0.0.1:50649 > > [NotebookApp] Connecting to: tcp://127.0.0.1:35516 > > [IPKernelApp] To connect another client to this kernel, use: > > [IPKernelApp] --existing kernel-71f07d5c-2db1-465f-9d86-da29f8711a0b.json > > Assertion failed: ok (bundled/zeromq/src/mailbox.cpp:79) > > [NotebookApp] Kernel 71f07d5c-2db1-465f-9d86-da29f8711a0b failed to > respond to heartbeat > > > > import sys > import os > > def child(pipeout): > os.dup2(pipeout, sys.stdout.fileno()) > os.close(pipeout) > os.execlp("./test.sh", "abc") > > > def parent(): > pipein, pipeout = os.pipe() > if os.fork() == 0: > os.close(pipein) > child(pipeout) > else: > os.close(pipeout) > counter = 1 > pipein = os.fdopen(pipein) > while True: > json = pipein.readline()[:-1] > print 'parent: %s' % (json) > > parent() > > $ cat test.sh > > #!/bin/bash > > > script_name=$0 > > args=$@ > > current=`pwd` > > echo $script_name $args > > echo "XYZ: "$XYZ > > echo "PATH: "$PATH > > echo "current directory: $current" > > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > > > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Fri Oct 3 14:32:34 2014 From: takowl at gmail.com (Thomas Kluyver) Date: Fri, 3 Oct 2014 11:32:34 -0700 Subject: [IPython-dev] ERROR can not use fork(), exec(), dup2() to communicate with child process In-Reply-To: References: Message-ID: On 3 October 2014 11:22, Andy Davidson wrote: > The subprocess documentation suggests that you need to call communicate() > and that this function will not return until the child sends EOF or > terminates. You don't actually *need* to use communicate(), it's just advised for most cases to avoid deadlocks. If you need to e.g. read streaming data one line at a time, you can use Popen.stdout.read(). Where you need to be careful is if the process writes to both stdout and stderr, and you only read stdout, it will fill up the stderr buffer and then block trying to write more stderr, while you're waiting for output on stdout that never appears. Similarly, if you're sending it data on stdin, it can get stuck waiting for input while you're waiting for output. communicate() handles this for you, but if necessary you can handle it yourself. Depending on your process, you might want to pipe stderr to /dev/null to discard it, or merge it with stdout so you read both at once. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From ellisonbg at gmail.com Fri Oct 3 14:57:08 2014 From: ellisonbg at gmail.com (Brian Granger) Date: Fri, 3 Oct 2014 11:57:08 -0700 Subject: [IPython-dev] Multi-user IPython Notebook In-Reply-To: <1412360330179-5073582.post@n6.nabble.com> References: <1412311899323-5073521.post@n6.nabble.com> <1412317605582-5073526.post@n6.nabble.com> <1412351070200-5073572.post@n6.nabble.com> <1412360330179-5073582.post@n6.nabble.com> Message-ID: (Kyle, working at Rackspace, might not volunteer this information in this context) I highly recommend looking at Rackspace's OnMetal Servers for performance critical things that you want to cost contain. Cheers, Brian On Fri, Oct 3, 2014 at 11:18 AM, ssanderson wrote: > Kyle, > > Currently we're spinning up new containers manually over SSH using Fabric. > The container we're running is conceptually pretty similar to the > scipyserver container in docker-notebook > . At the moment we're not > doing any sort of proxying, so each single user server just gets its own > unique URL and password, which is supplied to its user. > > As far as how many containers we run per host, we're still playing around > with different instance types and resource limits, so we haven't settled on > a specific number. One of the challenges for us is that different research > activities have radically different performance profiles. For example, > querying historical financial data is mostly IO-bound, analyses of large > datasets are often constrained by RAM, and backtesting with in-memory data > is primarily CPU-bound (in my dream world, we can farm out the CPU-bound > tasks to pool of worker containers using IPython Parallel, but that's a ways > off). That said, I think our current ballpark is to run on the order of 10 > containers per host. > > > > -- > View this message in context: http://python.6.x6.nabble.com/Multi-user-IPython-Notebook-tp5073520p5073582.html > Sent from the IPython - Development mailing list archive at Nabble.com. > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev -- Brian E. Granger Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub bgranger at calpoly.edu and ellisonbg at gmail.com From one at kentran.net Fri Oct 3 16:09:42 2014 From: one at kentran.net (Kenneth Tran) Date: Fri, 3 Oct 2014 13:09:42 -0700 Subject: [IPython-dev] Multi-user IPython Notebook In-Reply-To: References: <1412311899323-5073521.post@n6.nabble.com> <1412317605582-5073526.post@n6.nabble.com> <1412351070200-5073572.post@n6.nabble.com> <1412360330179-5073582.post@n6.nabble.com> Message-ID: Are there any instructions on how to get JupyterHub work on Windows using Active Directory or using Online Microsoft accounts? Fernando mentioned that we can plug-in a custom authenticator (or something like this). I want to know more details. Thanks. -Ken On Fri, Oct 3, 2014 at 11:57 AM, Brian Granger wrote: > (Kyle, working at Rackspace, might not volunteer this information in > this context) > > I highly recommend looking at Rackspace's OnMetal Servers for > performance critical things that you want to cost contain. > > Cheers, > > Brian > > On Fri, Oct 3, 2014 at 11:18 AM, ssanderson > wrote: > > Kyle, > > > > Currently we're spinning up new containers manually over SSH using > Fabric. > > The container we're running is conceptually pretty similar to the > > scipyserver container in docker-notebook > > . At the moment we're not > > doing any sort of proxying, so each single user server just gets its own > > unique URL and password, which is supplied to its user. > > > > As far as how many containers we run per host, we're still playing around > > with different instance types and resource limits, so we haven't settled > on > > a specific number. One of the challenges for us is that different > research > > activities have radically different performance profiles. For example, > > querying historical financial data is mostly IO-bound, analyses of large > > datasets are often constrained by RAM, and backtesting with in-memory > data > > is primarily CPU-bound (in my dream world, we can farm out the CPU-bound > > tasks to pool of worker containers using IPython Parallel, but that's a > ways > > off). That said, I think our current ballpark is to run on the order > of 10 > > containers per host. > > > > > > > > -- > > View this message in context: > http://python.6.x6.nabble.com/Multi-user-IPython-Notebook-tp5073520p5073582.html > > Sent from the IPython - Development mailing list archive at Nabble.com. > > _______________________________________________ > > IPython-dev mailing list > > IPython-dev at scipy.org > > http://mail.scipy.org/mailman/listinfo/ipython-dev > > > > -- > Brian E. Granger > Cal Poly State University, San Luis Obispo > @ellisonbg on Twitter and GitHub > bgranger at calpoly.edu and ellisonbg at gmail.com > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ellisonbg at gmail.com Fri Oct 3 16:11:46 2014 From: ellisonbg at gmail.com (Brian Granger) Date: Fri, 3 Oct 2014 13:11:46 -0700 Subject: [IPython-dev] Multi-user IPython Notebook In-Reply-To: References: <1412311899323-5073521.post@n6.nabble.com> <1412317605582-5073526.post@n6.nabble.com> <1412351070200-5073572.post@n6.nabble.com> <1412360330179-5073582.post@n6.nabble.com> Message-ID: No but we are accepting PRs! On Fri, Oct 3, 2014 at 1:09 PM, Kenneth Tran wrote: > Are there any instructions on how to get JupyterHub work on Windows using > Active Directory or using Online Microsoft accounts? > > Fernando mentioned that we can plug-in a custom authenticator (or something > like this). I want to know more details. > > Thanks. > -Ken > > > On Fri, Oct 3, 2014 at 11:57 AM, Brian Granger wrote: >> >> (Kyle, working at Rackspace, might not volunteer this information in >> this context) >> >> I highly recommend looking at Rackspace's OnMetal Servers for >> performance critical things that you want to cost contain. >> >> Cheers, >> >> Brian >> >> On Fri, Oct 3, 2014 at 11:18 AM, ssanderson >> wrote: >> > Kyle, >> > >> > Currently we're spinning up new containers manually over SSH using >> > Fabric. >> > The container we're running is conceptually pretty similar to the >> > scipyserver container in docker-notebook >> > . At the moment we're not >> > doing any sort of proxying, so each single user server just gets its own >> > unique URL and password, which is supplied to its user. >> > >> > As far as how many containers we run per host, we're still playing >> > around >> > with different instance types and resource limits, so we haven't settled >> > on >> > a specific number. One of the challenges for us is that different >> > research >> > activities have radically different performance profiles. For example, >> > querying historical financial data is mostly IO-bound, analyses of large >> > datasets are often constrained by RAM, and backtesting with in-memory >> > data >> > is primarily CPU-bound (in my dream world, we can farm out the CPU-bound >> > tasks to pool of worker containers using IPython Parallel, but that's a >> > ways >> > off). That said, I think our current ballpark is to run on the order >> > of 10 >> > containers per host. >> > >> > >> > >> > -- >> > View this message in context: >> > http://python.6.x6.nabble.com/Multi-user-IPython-Notebook-tp5073520p5073582.html >> > Sent from the IPython - Development mailing list archive at Nabble.com. >> > _______________________________________________ >> > IPython-dev mailing list >> > IPython-dev at scipy.org >> > http://mail.scipy.org/mailman/listinfo/ipython-dev >> >> >> >> -- >> Brian E. Granger >> Cal Poly State University, San Luis Obispo >> @ellisonbg on Twitter and GitHub >> bgranger at calpoly.edu and ellisonbg at gmail.com >> _______________________________________________ >> IPython-dev mailing list >> IPython-dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/ipython-dev > > > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -- Brian E. Granger Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub bgranger at calpoly.edu and ellisonbg at gmail.com From benjaminrk at gmail.com Fri Oct 3 16:17:04 2014 From: benjaminrk at gmail.com (MinRK) Date: Fri, 3 Oct 2014 13:17:04 -0700 Subject: [IPython-dev] Multi-user IPython Notebook In-Reply-To: References: <1412311899323-5073521.post@n6.nabble.com> <1412317605582-5073526.post@n6.nabble.com> <1412351070200-5073572.post@n6.nabble.com> <1412360330179-5073582.post@n6.nabble.com> Message-ID: The authentication is customizable. I wrote the OAuth example as a test case for this customizability. Anything that takes a username and password should be quite simple. We are still working on what the API should be, but it works right now. It is not planned, however, for JupyterHub to support Windows at all. -MinRK On Fri, Oct 3, 2014 at 1:11 PM, Brian Granger wrote: > No but we are accepting PRs! > > On Fri, Oct 3, 2014 at 1:09 PM, Kenneth Tran wrote: > > Are there any instructions on how to get JupyterHub work on Windows using > > Active Directory or using Online Microsoft accounts? > > > > Fernando mentioned that we can plug-in a custom authenticator (or > something > > like this). I want to know more details. > > > > Thanks. > > -Ken > > > > > > On Fri, Oct 3, 2014 at 11:57 AM, Brian Granger > wrote: > >> > >> (Kyle, working at Rackspace, might not volunteer this information in > >> this context) > >> > >> I highly recommend looking at Rackspace's OnMetal Servers for > >> performance critical things that you want to cost contain. > >> > >> Cheers, > >> > >> Brian > >> > >> On Fri, Oct 3, 2014 at 11:18 AM, ssanderson > >> wrote: > >> > Kyle, > >> > > >> > Currently we're spinning up new containers manually over SSH using > >> > Fabric. > >> > The container we're running is conceptually pretty similar to the > >> > scipyserver container in docker-notebook > >> > . At the moment we're > not > >> > doing any sort of proxying, so each single user server just gets its > own > >> > unique URL and password, which is supplied to its user. > >> > > >> > As far as how many containers we run per host, we're still playing > >> > around > >> > with different instance types and resource limits, so we haven't > settled > >> > on > >> > a specific number. One of the challenges for us is that different > >> > research > >> > activities have radically different performance profiles. For > example, > >> > querying historical financial data is mostly IO-bound, analyses of > large > >> > datasets are often constrained by RAM, and backtesting with in-memory > >> > data > >> > is primarily CPU-bound (in my dream world, we can farm out the > CPU-bound > >> > tasks to pool of worker containers using IPython Parallel, but that's > a > >> > ways > >> > off). That said, I think our current ballpark is to run on the order > >> > of 10 > >> > containers per host. > >> > > >> > > >> > > >> > -- > >> > View this message in context: > >> > > http://python.6.x6.nabble.com/Multi-user-IPython-Notebook-tp5073520p5073582.html > >> > Sent from the IPython - Development mailing list archive at > Nabble.com. > >> > _______________________________________________ > >> > IPython-dev mailing list > >> > IPython-dev at scipy.org > >> > http://mail.scipy.org/mailman/listinfo/ipython-dev > >> > >> > >> > >> -- > >> Brian E. Granger > >> Cal Poly State University, San Luis Obispo > >> @ellisonbg on Twitter and GitHub > >> bgranger at calpoly.edu and ellisonbg at gmail.com > >> _______________________________________________ > >> IPython-dev mailing list > >> IPython-dev at scipy.org > >> http://mail.scipy.org/mailman/listinfo/ipython-dev > > > > > > > > _______________________________________________ > > IPython-dev mailing list > > IPython-dev at scipy.org > > http://mail.scipy.org/mailman/listinfo/ipython-dev > > > > > > -- > Brian E. Granger > Cal Poly State University, San Luis Obispo > @ellisonbg on Twitter and GitHub > bgranger at calpoly.edu and ellisonbg at gmail.com > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rgbkrk at gmail.com Fri Oct 3 16:33:02 2014 From: rgbkrk at gmail.com (Kyle Kelley) Date: Fri, 3 Oct 2014 15:33:02 -0500 Subject: [IPython-dev] Multi-user IPython Notebook In-Reply-To: References: <1412311899323-5073521.post@n6.nabble.com> <1412317605582-5073526.post@n6.nabble.com> <1412351070200-5073572.post@n6.nabble.com> <1412360330179-5073582.post@n6.nabble.com> Message-ID: > It is not planned, however, for JupyterHub to support Windows at all. To clarify, JupyterHub will *only* run (server side) on *nix systems. Clients can reach it from a Windows environment. Getting custom auth with LDAP or Active Directory would work **great** with a large number of corporate environments. It's not currently in the project's plans to build this out, but I think it would be absolutely wonderful to exist with the IPython/Jupyter ecosystem/community. -- Kyle On Fri, Oct 3, 2014 at 3:17 PM, MinRK wrote: > The authentication is customizable. I wrote the OAuth example as a test > case for this customizability. Anything that takes a username and password > should be quite simple. We are still working on what the API should be, but > it works right now. It is not planned, however, for JupyterHub to support > Windows at all. > > -MinRK > > > On Fri, Oct 3, 2014 at 1:11 PM, Brian Granger wrote: > >> No but we are accepting PRs! >> >> On Fri, Oct 3, 2014 at 1:09 PM, Kenneth Tran wrote: >> > Are there any instructions on how to get JupyterHub work on Windows >> using >> > Active Directory or using Online Microsoft accounts? >> > >> > Fernando mentioned that we can plug-in a custom authenticator (or >> something >> > like this). I want to know more details. >> > >> > Thanks. >> > -Ken >> > >> > >> > On Fri, Oct 3, 2014 at 11:57 AM, Brian Granger >> wrote: >> >> >> >> (Kyle, working at Rackspace, might not volunteer this information in >> >> this context) >> >> >> >> I highly recommend looking at Rackspace's OnMetal Servers for >> >> performance critical things that you want to cost contain. >> >> >> >> Cheers, >> >> >> >> Brian >> >> >> >> On Fri, Oct 3, 2014 at 11:18 AM, ssanderson > > >> >> wrote: >> >> > Kyle, >> >> > >> >> > Currently we're spinning up new containers manually over SSH using >> >> > Fabric. >> >> > The container we're running is conceptually pretty similar to the >> >> > scipyserver container in docker-notebook >> >> > . At the moment >> we're not >> >> > doing any sort of proxying, so each single user server just gets its >> own >> >> > unique URL and password, which is supplied to its user. >> >> > >> >> > As far as how many containers we run per host, we're still playing >> >> > around >> >> > with different instance types and resource limits, so we haven't >> settled >> >> > on >> >> > a specific number. One of the challenges for us is that different >> >> > research >> >> > activities have radically different performance profiles. For >> example, >> >> > querying historical financial data is mostly IO-bound, analyses of >> large >> >> > datasets are often constrained by RAM, and backtesting with in-memory >> >> > data >> >> > is primarily CPU-bound (in my dream world, we can farm out the >> CPU-bound >> >> > tasks to pool of worker containers using IPython Parallel, but >> that's a >> >> > ways >> >> > off). That said, I think our current ballpark is to run on the >> order >> >> > of 10 >> >> > containers per host. >> >> > >> >> > >> >> > >> >> > -- >> >> > View this message in context: >> >> > >> http://python.6.x6.nabble.com/Multi-user-IPython-Notebook-tp5073520p5073582.html >> >> > Sent from the IPython - Development mailing list archive at >> Nabble.com. >> >> > _______________________________________________ >> >> > IPython-dev mailing list >> >> > IPython-dev at scipy.org >> >> > http://mail.scipy.org/mailman/listinfo/ipython-dev >> >> >> >> >> >> >> >> -- >> >> Brian E. Granger >> >> Cal Poly State University, San Luis Obispo >> >> @ellisonbg on Twitter and GitHub >> >> bgranger at calpoly.edu and ellisonbg at gmail.com >> >> _______________________________________________ >> >> IPython-dev mailing list >> >> IPython-dev at scipy.org >> >> http://mail.scipy.org/mailman/listinfo/ipython-dev >> > >> > >> > >> > _______________________________________________ >> > IPython-dev mailing list >> > IPython-dev at scipy.org >> > http://mail.scipy.org/mailman/listinfo/ipython-dev >> > >> >> >> >> -- >> Brian E. Granger >> Cal Poly State University, San Luis Obispo >> @ellisonbg on Twitter and GitHub >> bgranger at calpoly.edu and ellisonbg at gmail.com >> _______________________________________________ >> IPython-dev mailing list >> IPython-dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/ipython-dev >> > > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > > -- Kyle Kelley (@rgbkrk ; http://lambdaops.com) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rgbkrk at gmail.com Fri Oct 3 17:15:17 2014 From: rgbkrk at gmail.com (Kyle Kelley) Date: Fri, 3 Oct 2014 16:15:17 -0500 Subject: [IPython-dev] Multi-user IPython Notebook In-Reply-To: References: <1412311899323-5073521.post@n6.nabble.com> <1412317605582-5073526.post@n6.nabble.com> <1412351070200-5073572.post@n6.nabble.com> <1412360330179-5073582.post@n6.nabble.com> Message-ID: Thanks Brian. ;) I'm currently running a setup where I'm subdividing a machine with .5 TB of RAM, where each user gets 512 MB of RAM (~1000 users). Not quite ready to show off beyond a handful of people. Email me directly if you want to check it out. If anyone wants specs or to talk about the environment, feel free to ask. ;) All tools are built agnostic of providers, but I am working cloud configs that optimize for the hardware its being run on, primarily for how /var/lib/docker is mounted. -- Kyle On Fri, Oct 3, 2014 at 1:57 PM, Brian Granger wrote: > (Kyle, working at Rackspace, might not volunteer this information in > this context) > > I highly recommend looking at Rackspace's OnMetal Servers for > performance critical things that you want to cost contain. > > Cheers, > > Brian > > On Fri, Oct 3, 2014 at 11:18 AM, ssanderson > wrote: > > Kyle, > > > > Currently we're spinning up new containers manually over SSH using > Fabric. > > The container we're running is conceptually pretty similar to the > > scipyserver container in docker-notebook > > . At the moment we're not > > doing any sort of proxying, so each single user server just gets its own > > unique URL and password, which is supplied to its user. > > > > As far as how many containers we run per host, we're still playing around > > with different instance types and resource limits, so we haven't settled > on > > a specific number. One of the challenges for us is that different > research > > activities have radically different performance profiles. For example, > > querying historical financial data is mostly IO-bound, analyses of large > > datasets are often constrained by RAM, and backtesting with in-memory > data > > is primarily CPU-bound (in my dream world, we can farm out the CPU-bound > > tasks to pool of worker containers using IPython Parallel, but that's a > ways > > off). That said, I think our current ballpark is to run on the order > of 10 > > containers per host. > > > > > > > > -- > > View this message in context: > http://python.6.x6.nabble.com/Multi-user-IPython-Notebook-tp5073520p5073582.html > > Sent from the IPython - Development mailing list archive at Nabble.com. > > _______________________________________________ > > IPython-dev mailing list > > IPython-dev at scipy.org > > http://mail.scipy.org/mailman/listinfo/ipython-dev > > > > -- > Brian E. Granger > Cal Poly State University, San Luis Obispo > @ellisonbg on Twitter and GitHub > bgranger at calpoly.edu and ellisonbg at gmail.com > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -- Kyle Kelley (@rgbkrk ; http://lambdaops.com) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvraghav93 at gmail.com Mon Oct 6 04:18:39 2014 From: rvraghav93 at gmail.com (Raghav Rv) Date: Mon, 6 Oct 2014 13:48:39 +0530 Subject: [IPython-dev] PEP8 Check For IPython Notebook - A small work around Message-ID: Hi all, I recently modified the pep8 project a little to allow pep8 checks to be run on ipython notebook, inspired by this stack overflow post . [ Though we do have an extension that can check for pep8 on a per-cell basis, this modification can help us run pep8 for a whole ".ipynb" file. ] Using the --format="ipynb" flag we can get the line number offset ( calculated for each individual cell ) and the cell number in the error report, to easily locate errors. This is the pull request : https://github.com/jcrocholl/pep8/pull/337 ( Looks like this won't be merged as the devs of pep8 feel this is out of scope for the project. ) I am just posting it here ( as advised by Matt ), just incase someone finds it useful ... Sample Notebook . Result of running : `python pep8.py --format="ipynb" notebook.ipynb` on the above notebook. ( Output seems to be consistent with that of the pep8 check on the converted ".py" file ) Output Thanks Raghav Rv ( rvraghav93 ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From wstein at gmail.com Mon Oct 6 10:39:18 2014 From: wstein at gmail.com (William Stein) Date: Mon, 6 Oct 2014 07:39:18 -0700 Subject: [IPython-dev] Fwd: [sage-devel] Re: Please review ipython notebook In-Reply-To: <826e9f22-f585-48c4-8f1d-f701bd795327@googlegroups.com> References: <826e9f22-f585-48c4-8f1d-f701bd795327@googlegroups.com> Message-ID: Any chance any IPython devs would have any interest in at least looking at what Volker's doing here? See below. A little effort by somebody could could increase the number of IPython notebook users by thousands... Forwarded conversation Subject: [sage-devel] Please review ipython notebook ------------------------ From: *Volker Braun* Date: Mon, Sep 29, 2014 at 4:48 AM To: sage-devel at googlegroups.com Now would be a good time to review the IPython notebook: http://trac.sagemath.org/ticket/16996 There is clearly more to be done to make the integration perfect, and we could also always wait for the next IPython version. But a bird in the hand is worth two in the bush... -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscribe at googlegroups.com. To post to this group, send email to sage-devel at googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout. ---------- From: *Volker Braun* Date: Mon, Oct 6, 2014 at 7:36 AM To: sage-devel at googlegroups.com anybody? -- William Stein Professor of Mathematics University of Washington http://wstein.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From hughesadam87 at gmail.com Mon Oct 6 14:34:49 2014 From: hughesadam87 at gmail.com (hugadams) Date: Mon, 6 Oct 2014 11:34:49 -0700 (PDT) Subject: [IPython-dev] Interactive, Graphical, Diffable Pstats Viewer for IPython Notebook In-Reply-To: <1412324654616-5073531.post@n6.nabble.com> References: <1412324654616-5073531.post@n6.nabble.com> Message-ID: <1412620489897-5073832.post@n6.nabble.com> Awesome, thanks for posting. The more examples I can find of widget applications, the more I learn! -- View this message in context: http://python.6.x6.nabble.com/Interactive-Graphical-Diffable-Pstats-Viewer-for-IPython-Notebook-tp5073531p5073832.html Sent from the IPython - Development mailing list archive at Nabble.com. From hughesadam87 at gmail.com Mon Oct 6 14:52:30 2014 From: hughesadam87 at gmail.com (hugadams) Date: Mon, 6 Oct 2014 11:52:30 -0700 (PDT) Subject: [IPython-dev] Quick Question about Widgets Message-ID: Hi Nicholas, Sorry to bother you off of the mailing list. We have been working on the example you composed for us in regard to spectroscopy widgets, and it's really working nicely. Thank you for this! We've encounted one hiccup that we thought you might immediately know how to resolve. I posted it on the list recently, but think it's pretty specialized and most users wouldn't have encountered it yet. Our prototype is working, and nicely separates model and view following your original design. http://nbviewer.ipython.org/github/hugadams/pyuvvis/blob/master/pyuvvis/interact/ipynbs/prototypes/widgets_slicing.ipynb So here's the issue. Imagine the user creates a widget with one dataset. We have a couple of widgets that are set based on the extents of the dataset itself. For example, imagine our data is a set of values from 0 to 1000 and we'd like to have a slider widget that runs from 0 to 1000. This works great; however, what if the user wants to switch datasets? Inside the gui, there's a dropdown menu that has DataA, DataB, DataC. If the user switches to DataC, we'd like the view to effectively reset. Imagine that DataC has values that range from 0 to 500: in this case, the float slider which originall ran from 0 to 1000 should redraw and run from 0 to 500. Unfortunately, that means the slider in the view needs to redraw when the model changes. We can't figure out how to execute such a redraw/reset. What's happening is that the slider tries to slice the new DataC values at 0 to 1000 and results in an IndexError. We're looking for a workaround for such a case. Does anything come to mind? Alternatively, if such a use case is impossible (or very difficult) to make work, we can redesign things with the idea of one gui per dataset. This will change our design patterns for all of the gui's to come, so it's really critical for us to obviate this problem as soon as possible. If you have any ideas, they would help us tremendously. And thank you for all of your help with this already. -- View this message in context: http://python.6.x6.nabble.com/Quick-Question-about-Widgets-tp5073833.html Sent from the IPython - Development mailing list archive at Nabble.com. -------------- next part -------------- An HTML attachment was scrubbed... URL: From darcamo at gmail.com Mon Oct 6 16:07:23 2014 From: darcamo at gmail.com (Darlan Cavalcante Moreira) Date: Mon, 06 Oct 2014 17:07:23 -0300 Subject: [IPython-dev] Interactive, Graphical, Diffable Pstats Viewer for IPython Notebook In-Reply-To: <1412620489897-5073832.post@n6.nabble.com> References: <1412324654616-5073531.post@n6.nabble.com> <1412620489897-5073832.post@n6.nabble.com> Message-ID: <87lhot6jyc.fsf@gmail.com> Very nice, indeed. To view the profile files created by cProfile the `runsnakerun`[1] program (available in Ubuntu repos) is also very nice. [1] - http://www.vrplumber.com/programming/runsnakerun/ hughesadam87 at gmail.com writes: > Awesome, thanks for posting. The more examples I can find of widget > applications, the more I learn! > > > > -- > View this message in context: http://python.6.x6.nabble.com/Interactive-Graphical-Diffable-Pstats-Viewer-for-IPython-Notebook-tp5073531p5073832.html > Sent from the IPython - Development mailing list archive at Nabble.com. > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev -- Darlan Cavalcante Moreira darcamo at gmail.com From nick.bollweg at gmail.com Mon Oct 6 16:29:32 2014 From: nick.bollweg at gmail.com (Nicholas Bollweg) Date: Mon, 6 Oct 2014 16:29:32 -0400 Subject: [IPython-dev] Quick Question about Widgets In-Reply-To: References: Message-ID: Sorry; haven't had much time to tinker on widget stuff much outside of directly work-related stuff. The file stuff sounds like it's getting pretty complicated! Maybe you could add another layer outside of the GUI, like a Workspace, and have all of the file-relevant stuff handled there. Then, when you change datasets, you just re-roll the whole GUI related to that specific dataset. However, based on what you have done... first, you could instrument the dataset as a traitlets.Instance: (*caveat: the below things are not tested!* ) class Spectrogram(HTML): > ... > ts = Instance(klass=WhateverTsIs) Then you can use the nice events to react to that being switched to a new instance on a per-linked-widget basis: class Spectroscopy(Box): > ... > def _sampling(self): > ... > start = FloatSlider( > description="Slicing Start Value", > value=self.graph.ts.index[0], > min=self.graph.ts.index.min(), > max=self.graph.ts.index.max() > ) > ... > > def _ts_changed(name, old, new): > start.value = self.graph.ts.index[0] > start.min = self.graph.ts.index.min() > start.max = self.graph.ts.index.max() > > self.graph.on_trait_changed(_ts_changed, "ts") > Now, you'll probably still have the problem of whether you are actually ready to redraw as all those controls re-initialize... the easiest thing would probably be to just add some more error handling to Spectrogram.draw to catch these errors, and let it fail if you are pretty confident that it will work once the inputs are all valid. Or, you could set a time window during which you don't try to redraw. Not sure what would work out best. Maybe this helps? -------------- next part -------------- An HTML attachment was scrubbed... URL: From pi at berkeley.edu Mon Oct 6 17:41:27 2014 From: pi at berkeley.edu (Paul Ivanov) Date: Mon, 6 Oct 2014 14:41:27 -0700 Subject: [IPython-dev] Interactive, Graphical, Diffable Pstats Viewer for IPython Notebook In-Reply-To: <87lhot6jyc.fsf@gmail.com> References: <1412324654616-5073531.post@n6.nabble.com> <1412620489897-5073832.post@n6.nabble.com> <87lhot6jyc.fsf@gmail.com> Message-ID: On Mon, Oct 6, 2014 at 1:07 PM, Darlan Cavalcante Moreira wrote: > > To view the profile files created by cProfile the `runsnakerun` > program (available in Ubuntu repos) is also very nice. For in-browser visualization of cProfiles, see Matt Davis' SnakeViz [1] 1. http://jiffyclub.github.io/snakeviz/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jiffyclub at gmail.com Mon Oct 6 17:46:38 2014 From: jiffyclub at gmail.com (Matt Davis) Date: Mon, 6 Oct 2014 14:46:38 -0700 Subject: [IPython-dev] Interactive, Graphical, Diffable Pstats Viewer for IPython Notebook In-Reply-To: References: <1412324654616-5073531.post@n6.nabble.com> <1412620489897-5073832.post@n6.nabble.com> <87lhot6jyc.fsf@gmail.com> Message-ID: There's also a low-tech IPython magic extension for snakeviz that you can use in-place of %prun: https://github.com/jiffyclub/snakeviz/pull/20#issuecomment-32312354. On Mon, Oct 6, 2014 at 2:41 PM, Paul Ivanov wrote: > > > On Mon, Oct 6, 2014 at 1:07 PM, Darlan Cavalcante Moreira < > darcamo at gmail.com> wrote: >> >> To view the profile files created by cProfile the `runsnakerun` >> program (available in Ubuntu repos) is also very nice. > > > For in-browser visualization of cProfiles, see Matt Davis' SnakeViz [1] > > 1. http://jiffyclub.github.io/snakeviz/ > > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hughesadam87 at gmail.com Mon Oct 6 18:09:11 2014 From: hughesadam87 at gmail.com (Adam Hughes) Date: Mon, 6 Oct 2014 18:09:11 -0400 Subject: [IPython-dev] Quick Question about Widgets In-Reply-To: References: Message-ID: That is a great idea, thanks! I think handling the errors is the best approach. We'll give it a shot. On Mon, Oct 6, 2014 at 4:29 PM, Nicholas Bollweg wrote: > Sorry; haven't had much time to tinker on widget stuff much outside of > directly work-related stuff. > > The file stuff sounds like it's getting pretty complicated! Maybe you > could add another layer outside of the GUI, like a Workspace, and have all > of the file-relevant stuff handled there. Then, when you change datasets, > you just re-roll the whole GUI related to that specific dataset. > > However, based on what you have done... first, you could instrument the > dataset as a traitlets.Instance: (*caveat: the below things are not > tested!*) > > class Spectrogram(HTML): >> ... >> ts = Instance(klass=WhateverTsIs) > > > Then you can use the nice events to react to that being switched to a new > instance on a per-linked-widget basis: > > class Spectroscopy(Box): >> ... >> def _sampling(self): >> ... >> start = FloatSlider( >> description="Slicing Start Value", >> value=self.graph.ts.index[0], >> min=self.graph.ts.index.min(), >> max=self.graph.ts.index.max() >> ) >> ... >> >> def _ts_changed(name, old, new): >> start.value = self.graph.ts.index[0] >> start.min = self.graph.ts.index.min() >> start.max = self.graph.ts.index.max() >> >> self.graph.on_trait_changed(_ts_changed, "ts") >> > > Now, you'll probably still have the problem of whether you are actually > ready to redraw as all those controls re-initialize... the easiest thing > would probably be to just add some more error handling to Spectrogram.draw > to catch these errors, and let it fail if you are pretty confident that it > will work once the inputs are all valid. Or, you could set a time window > during which you don't try to redraw. Not sure what would work out best. > > Maybe this helps? > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Mon Oct 6 18:17:11 2014 From: wes.turner at gmail.com (Wes Turner) Date: Mon, 6 Oct 2014 17:17:11 -0500 Subject: [IPython-dev] Fwd: [sage-devel] Re: Please review ipython notebook In-Reply-To: References: <826e9f22-f585-48c4-8f1d-f701bd795327@googlegroups.com> Message-ID: Is there any chance the SAGE project would consider utilizing a consistent cross-platform packaging strategy? (e.g. http://conda.pydata.org/docs/build.html) > The conda skeleton command can help to make skeleton recipes for common repositories, such as PyPI. [and CPAN] On Mon, Oct 6, 2014 at 9:39 AM, William Stein wrote: > Any chance any IPython devs would have any interest in at least looking at > what Volker's doing here? See below. A little effort by somebody could > could increase the number of IPython notebook users by thousands... > > Forwarded conversation > Subject: [sage-devel] Please review ipython notebook > ------------------------ > > From: Volker Braun > Date: Mon, Sep 29, 2014 at 4:48 AM > To: sage-devel at googlegroups.com > > > Now would be a good time to review the IPython notebook: > > http://trac.sagemath.org/ticket/16996 > > There is clearly more to be done to make the integration perfect, and we > could also always wait for the next IPython version. But a bird in the hand > is worth two in the bush... > > -- > You received this message because you are subscribed to the Google Groups > "sage-devel" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sage-devel+unsubscribe at googlegroups.com. > To post to this group, send email to sage-devel at googlegroups.com. > Visit this group at http://groups.google.com/group/sage-devel. > For more options, visit https://groups.google.com/d/optout. > > ---------- > From: Volker Braun > Date: Mon, Oct 6, 2014 at 7:36 AM > To: sage-devel at googlegroups.com > > > anybody? > > > > > -- > William Stein > Professor of Mathematics > University of Washington > http://wstein.org > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -- Wes Turner https://westurner.github.io/ From takowl at gmail.com Mon Oct 6 18:35:33 2014 From: takowl at gmail.com (Thomas Kluyver) Date: Mon, 6 Oct 2014 15:35:33 -0700 Subject: [IPython-dev] PEP8 Check For IPython Notebook - A small work around In-Reply-To: References: Message-ID: Thanks Raghav! From the conversation on your issue, it looks like it might make sense for your code to live as a flake8 extension. Thomas On 6 October 2014 01:18, Raghav Rv wrote: > Hi all, > > I recently modified the pep8 project a little to allow pep8 checks to be > run on ipython notebook, inspired by this stack overflow post > . > > [ Though we do have an extension that can check for pep8 on a per-cell > basis, this modification can help us run pep8 for a whole ".ipynb" file. ] > > Using the --format="ipynb" flag we can get the line number offset ( > calculated for each individual cell ) and the cell number in the error > report, to easily locate errors. > > This is the pull request : https://github.com/jcrocholl/pep8/pull/337 > > ( Looks like this won't be merged as the devs of pep8 feel this is out of > scope for the project. ) > > I am just posting it here ( as advised by Matt ), just incase someone > finds it useful ... > > Sample Notebook > > . > > Result of running : `python pep8.py --format="ipynb" notebook.ipynb` on > the above notebook. ( Output seems to be consistent with that of the pep8 > check on the converted ".py" file ) > > Output > > Thanks > Raghav Rv ( rvraghav93 ) > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From odewahn at oreilly.com Tue Oct 7 11:25:12 2014 From: odewahn at oreilly.com (Andrew Odewahn) Date: Tue, 7 Oct 2014 11:25:12 -0400 Subject: [IPython-dev] Set default value of Kernel dropdown in 3.0.0-dev Message-ID: Hi. I'm trying to set the default kernel that is used in Notebooks 3.0.0-dev. (i.e., set the value of the kernel_selector_widget dropdown box at the top right corner). Rather than opening in Python 3, I'd like to set it to Python 2. Is there a command line option I can pass for this? Something like: ipython notebook --ip=0.0.0.0 --no-browser --SomeConfig.SomeOption=kernel-python2 Any help appreciated! Andrew -------------- next part -------------- An HTML attachment was scrubbed... URL: From tshawver at quantopian.com Tue Oct 7 12:06:01 2014 From: tshawver at quantopian.com (tshawver) Date: Tue, 7 Oct 2014 09:06:01 -0700 (PDT) Subject: [IPython-dev] Interactive Grid for Sorting and Filtering DataFrames in IPython Notebook Message-ID: <1412697961680-5073936.post@n6.nabble.com> As part of the work on our research environment at Quantopian , I've been building an extension which renders pandas DataFrames as interactive grids in the IPython notebook. The extension uses a Javascript library called SlickGrid to render the grids, and the current state of the project can be found here on GitHub: https://github.com/quantopian/qgrid The extension is also viewable in nbviewer: http://nbviewer.ipython.org/github/quantopian/qgrid/blob/master/qgrid_demo.ipynb The GitHub repository contains some explanation of why I chose to implement the grid as a Python package rather than a standard IPython notebook extension, which might be interesting for other people who are looking to add functionality to the notebook in a similar way. -Tim -- View this message in context: http://python.6.x6.nabble.com/Interactive-Grid-for-Sorting-and-Filtering-DataFrames-in-IPython-Notebook-tp5073936.html Sent from the IPython - Development mailing list archive at Nabble.com. From damianavila at gmail.com Tue Oct 7 12:19:07 2014 From: damianavila at gmail.com (=?UTF-8?Q?Dami=C3=A1n_Avila?=) Date: Tue, 7 Oct 2014 13:19:07 -0300 Subject: [IPython-dev] Set default value of Kernel dropdown in 3.0.0-dev In-Reply-To: References: Message-ID: Something like this has to work: ipython notebook --MappingKernelManager.default_kernel_name='python3' which your kernel, of course... 'python2' instead of 'python3'. 2014-10-07 12:25 GMT-03:00 Andrew Odewahn : > > Hi. I'm trying to set the default kernel that is used in Notebooks > 3.0.0-dev. (i.e., set the value of the kernel_selector_widget dropdown > box at the top right corner). > > Rather than opening in Python 3, I'd like to set it to Python 2. Is there > a command line option I can pass for this? > > Something like: > > ipython notebook --ip=0.0.0.0 --no-browser > --SomeConfig.SomeOption=kernel-python2 > > Any help appreciated! > > Andrew > > > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > > -- *Dami?n* -------------- next part -------------- An HTML attachment was scrubbed... URL: From ocefpaf at gmail.com Tue Oct 7 12:21:09 2014 From: ocefpaf at gmail.com (Filipe) Date: Tue, 07 Oct 2014 13:21:09 -0300 Subject: [IPython-dev] Set default value of Kernel dropdown in 3.0.0-dev In-Reply-To: References: Message-ID: <543412F5.8090901@gmail.com> Hi Andrew, Here is how I did it: https://ocefpaf.github.io/python4oceanographers/blog/2014/09/01/ipython_kernel/ This is probably not the best way to do it, but it works. -Filipe On 10/07/2014 12:25 PM, Andrew Odewahn wrote: > Hi. I'm trying to set the default kernel that is used in Notebooks > 3.0.0-dev. (i.e., set the value of the kernel_selector_widget dropdown box > at the top right corner). > > Rather than opening in Python 3, I'd like to set it to Python 2. Is there a > command line option I can pass for this? > > Something like: > > ipython notebook --ip=0.0.0.0 --no-browser > --SomeConfig.SomeOption=kernel-python2 > > Any help appreciated! > > Andrew From doug.blank at gmail.com Tue Oct 7 13:03:02 2014 From: doug.blank at gmail.com (Doug Blank) Date: Tue, 7 Oct 2014 13:03:02 -0400 Subject: [IPython-dev] Interactive Grid for Sorting and Filtering DataFrames in IPython Notebook In-Reply-To: <1412697961680-5073936.post@n6.nabble.com> References: <1412697961680-5073936.post@n6.nabble.com> Message-ID: On Tue, Oct 7, 2014 at 12:06 PM, tshawver wrote: > As part of the work on our research environment at Quantopian > , I've been building an extension > which renders pandas DataFrames as interactive grids in the IPython > notebook. The extension uses a Javascript library called SlickGrid to > render the grids, and the current state of the project can be found here on > GitHub: https://github.com/quantopian/qgrid > > The extension is also viewable in nbviewer: > > http://nbviewer.ipython.org/github/quantopian/qgrid/blob/master/qgrid_demo.ipynb > > The GitHub repository contains some explanation of why I chose to implement > the grid as a Python package rather than a standard IPython notebook > extension, which might be interesting for other people who are looking to > add functionality to the notebook in a similar way. > > Excellent! Thanks for developing and sharing... this will be very useful, and is the way that data should be viewed. -Doug > -Tim > > > > -- > View this message in context: > http://python.6.x6.nabble.com/Interactive-Grid-for-Sorting-and-Filtering-DataFrames-in-IPython-Notebook-tp5073936.html > Sent from the IPython - Development mailing list archive at Nabble.com. > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Tue Oct 7 14:12:19 2014 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 7 Oct 2014 13:12:19 -0500 Subject: [IPython-dev] Interactive Grid for Sorting and Filtering DataFrames in IPython Notebook In-Reply-To: <1412697961680-5073936.post@n6.nabble.com> References: <1412697961680-5073936.post@n6.nabble.com> Message-ID: Thanks! On Oct 7, 2014 11:06 AM, "tshawver" wrote: > As part of the work on our research environment at Quantopian > , I've been building an extension > which renders pandas DataFrames as interactive grids in the IPython > notebook. The extension uses a Javascript library called SlickGrid to > render the grids, and the current state of the project can be found here on > GitHub: https://github.com/quantopian/qgrid > > The extension is also viewable in nbviewer: > > http://nbviewer.ipython.org/github/quantopian/qgrid/blob/master/qgrid_demo.ipynb > > The GitHub repository contains some explanation of why I chose to implement > the grid as a Python package rather than a standard IPython notebook > extension, which might be interesting for other people who are looking to > add functionality to the notebook in a similar way. > > -Tim > > > > -- > View this message in context: > http://python.6.x6.nabble.com/Interactive-Grid-for-Sorting-and-Filtering-DataFrames-in-IPython-Notebook-tp5073936.html > Sent from the IPython - Development mailing list archive at Nabble.com. > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ssanderson at quantopian.com Tue Oct 7 15:08:27 2014 From: ssanderson at quantopian.com (ssanderson) Date: Tue, 7 Oct 2014 12:08:27 -0700 (PDT) Subject: [IPython-dev] Interactive, Graphical, Diffable Pstats Viewer for IPython Notebook In-Reply-To: <1412324654616-5073531.post@n6.nabble.com> References: <1412324654616-5073531.post@n6.nabble.com> Message-ID: <1412708907832-5073968.post@n6.nabble.com> Thanks for the heads up about snakeviz. Looks like an awesome project. With the files that I was testing here, snakeviz barfed because it took too long to build its visualization, but that may just be because the computer I'm using for testing is made of potato... -- View this message in context: http://python.6.x6.nabble.com/Interactive-Graphical-Diffable-Pstats-Viewer-for-IPython-Notebook-tp5073531p5073968.html Sent from the IPython - Development mailing list archive at Nabble.com. From jiffyclub at gmail.com Tue Oct 7 15:18:12 2014 From: jiffyclub at gmail.com (Matt Davis) Date: Tue, 7 Oct 2014 12:18:12 -0700 Subject: [IPython-dev] Interactive, Graphical, Diffable Pstats Viewer for IPython Notebook In-Reply-To: <1412708907832-5073968.post@n6.nabble.com> References: <1412324654616-5073531.post@n6.nabble.com> <1412708907832-5073968.post@n6.nabble.com> Message-ID: No, SnakeViz doesn't work well with large profiles (profiles with a lot of different function calls). I still find the sortable table useful, but I sometimes try to narrow down the code I'm profiling to get it to visualize. One of these days I'm gonna rewrite SnakeViz... On Tue, Oct 7, 2014 at 12:08 PM, ssanderson wrote: > Thanks for the heads up about snakeviz. Looks like an awesome project. > With > the files that I was testing here, snakeviz barfed because it took too long > to build its visualization, but that may just be because the computer I'm > using for testing is made of potato... > > > > -- > View this message in context: > http://python.6.x6.nabble.com/Interactive-Graphical-Diffable-Pstats-Viewer-for-IPython-Notebook-tp5073531p5073968.html > Sent from the IPython - Development mailing list archive at Nabble.com. > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rgbkrk at gmail.com Tue Oct 7 15:39:26 2014 From: rgbkrk at gmail.com (Kyle Kelley) Date: Tue, 7 Oct 2014 14:39:26 -0500 Subject: [IPython-dev] Set default value of Kernel dropdown in 3.0.0-dev In-Reply-To: References: Message-ID: If you're running from the base ipython/ipython or ipython/scipystack Docker images, it defaults to the last installed version of ipython (culprit is https://github.com/ipython/ipython/blob/master/Dockerfile#L43). There should be a way to set the kernel type as the default though. On Tue, Oct 7, 2014 at 10:25 AM, Andrew Odewahn wrote: > > Hi. I'm trying to set the default kernel that is used in Notebooks > 3.0.0-dev. (i.e., set the value of the kernel_selector_widget dropdown > box at the top right corner). > > Rather than opening in Python 3, I'd like to set it to Python 2. Is there > a command line option I can pass for this? > > Something like: > > ipython notebook --ip=0.0.0.0 --no-browser > --SomeConfig.SomeOption=kernel-python2 > > Any help appreciated! > > Andrew > > > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > > -- Kyle Kelley (@rgbkrk ; http://lambdaops.com) -------------- next part -------------- An HTML attachment was scrubbed... URL: From schueller at phimeca.com Wed Oct 8 03:36:39 2014 From: schueller at phimeca.com (Julien Schueller) Date: Wed, 8 Oct 2014 09:36:39 +0200 (CEST) Subject: [IPython-dev] lookfor magic extension module In-Reply-To: <1325997183.19349339.1412753719729.JavaMail.zimbra@phimeca.com> Message-ID: <1715875050.19350411.1412753799602.JavaMail.zimbra@phimeca.com> Hi list, I just uploaded to pypi the lookfor extension module to search through docstrings: https://github.com/jschueller/lookfor -- J From fperez.net at gmail.com Wed Oct 8 04:14:32 2014 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 8 Oct 2014 01:14:32 -0700 Subject: [IPython-dev] Interactive Grid for Sorting and Filtering DataFrames in IPython Notebook In-Reply-To: <1412697961680-5073936.post@n6.nabble.com> References: <1412697961680-5073936.post@n6.nabble.com> Message-ID: On Tue, Oct 7, 2014 at 9:06 AM, tshawver wrote: > As part of the work on our research environment at Quantopian > , I've been building an extension > which renders pandas DataFrames as interactive grids in the IPython > notebook. The extension uses a Javascript library called SlickGrid to > render the grids, and the current state of the project can be found here on > GitHub: https://github.com/quantopian/qgrid > Awesome! Just a few days ago I was musing on actually spending a bit of time doing this, but I figured someone with far better JS chops than my non-existent ones would likely do a much better job. Thanks a lot!! f -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Wed Oct 8 06:41:30 2014 From: wes.turner at gmail.com (Wes Turner) Date: Wed, 8 Oct 2014 05:41:30 -0500 Subject: [IPython-dev] Interactive Grid for Sorting and Filtering DataFrames in IPython Notebook In-Reply-To: <1412697961680-5073936.post@n6.nabble.com> References: <1412697961680-5073936.post@n6.nabble.com> Message-ID: This would be an excellent addition to the "Pandas Ecosystem" docs page: * http://pandas.pydata.org/pandas-docs/stable/ecosystem.html * https://github.com/pydata/pandas/blob/master/doc/source/ecosystem.rst Thanks again! On Tue, Oct 7, 2014 at 11:06 AM, tshawver wrote: > As part of the work on our research environment at Quantopian > , I've been building an extension > which renders pandas DataFrames as interactive grids in the IPython > notebook. The extension uses a Javascript library called SlickGrid to > render the grids, and the current state of the project can be found here on > GitHub: https://github.com/quantopian/qgrid > > The extension is also viewable in nbviewer: > http://nbviewer.ipython.org/github/quantopian/qgrid/blob/master/qgrid_demo.ipynb > > The GitHub repository contains some explanation of why I chose to implement > the grid as a Python package rather than a standard IPython notebook > extension, which might be interesting for other people who are looking to > add functionality to the notebook in a similar way. > > -Tim > > > > -- > View this message in context: http://python.6.x6.nabble.com/Interactive-Grid-for-Sorting-and-Filtering-DataFrames-in-IPython-Notebook-tp5073936.html > Sent from the IPython - Development mailing list archive at Nabble.com. > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev -- Wes Turner https://westurner.github.io/ From bussonniermatthias at gmail.com Wed Oct 8 06:55:04 2014 From: bussonniermatthias at gmail.com (Matthias BUSSONNIER) Date: Wed, 8 Oct 2014 12:55:04 +0200 Subject: [IPython-dev] lookfor magic extension module In-Reply-To: <1715875050.19350411.1412753799602.JavaMail.zimbra@phimeca.com> References: <1715875050.19350411.1412753799602.JavaMail.zimbra@phimeca.com> Message-ID: <3BF99ADB-19F9-4AE6-9D67-59FCFCA770BB@gmail.com> Le 8 oct. 2014 ? 09:36, Julien Schueller a ?crit : > Hi list, > > I just uploaded to pypi the lookfor extension module to search through docstrings: > https://github.com/jschueller/lookfor Great ! Thanks ! Cf this Pull request for context : https://github.com/ipython/ipython/pull/6641 And it's already available on PyPi ! Awesome ! -- M > > > -- > J > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev From wes.turner at gmail.com Wed Oct 8 07:00:12 2014 From: wes.turner at gmail.com (Wes Turner) Date: Wed, 8 Oct 2014 06:00:12 -0500 Subject: [IPython-dev] Interactive Grid for Sorting and Filtering DataFrames in IPython Notebook In-Reply-To: References: <1412697961680-5073936.post@n6.nabble.com> Message-ID: This PR adds a 'Graphical User Interfaces' heading: https://github.com/pydata/pandas/pull/8507 On Wed, Oct 8, 2014 at 5:41 AM, Wes Turner wrote: > This would be an excellent addition to the "Pandas Ecosystem" docs page: > > * http://pandas.pydata.org/pandas-docs/stable/ecosystem.html > * https://github.com/pydata/pandas/blob/master/doc/source/ecosystem.rst > > Thanks again! > > On Tue, Oct 7, 2014 at 11:06 AM, tshawver wrote: >> As part of the work on our research environment at Quantopian >> , I've been building an extension >> which renders pandas DataFrames as interactive grids in the IPython >> notebook. The extension uses a Javascript library called SlickGrid to >> render the grids, and the current state of the project can be found here on >> GitHub: https://github.com/quantopian/qgrid >> >> The extension is also viewable in nbviewer: >> http://nbviewer.ipython.org/github/quantopian/qgrid/blob/master/qgrid_demo.ipynb >> >> The GitHub repository contains some explanation of why I chose to implement >> the grid as a Python package rather than a standard IPython notebook >> extension, which might be interesting for other people who are looking to >> add functionality to the notebook in a similar way. >> >> -Tim >> >> >> >> -- >> View this message in context: http://python.6.x6.nabble.com/Interactive-Grid-for-Sorting-and-Filtering-DataFrames-in-IPython-Notebook-tp5073936.html >> Sent from the IPython - Development mailing list archive at Nabble.com. >> _______________________________________________ >> IPython-dev mailing list >> IPython-dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/ipython-dev > > > > -- > Wes Turner > https://westurner.github.io/ -- Wes Turner https://westurner.github.io/ From ivan.ogassawara at gmail.com Wed Oct 8 07:29:35 2014 From: ivan.ogassawara at gmail.com (Ivan Ogassawara) Date: Wed, 8 Oct 2014 08:29:35 -0300 Subject: [IPython-dev] Interactive Grid for Sorting and Filtering DataFrames in IPython Notebook In-Reply-To: References: <1412697961680-5073936.post@n6.nabble.com> Message-ID: Excellent job!! 2014-10-08 8:00 GMT-03:00 Wes Turner : > This PR adds a 'Graphical User Interfaces' heading: > https://github.com/pydata/pandas/pull/8507 > > On Wed, Oct 8, 2014 at 5:41 AM, Wes Turner wrote: > > This would be an excellent addition to the "Pandas Ecosystem" docs page: > > > > * http://pandas.pydata.org/pandas-docs/stable/ecosystem.html > > * https://github.com/pydata/pandas/blob/master/doc/source/ecosystem.rst > > > > Thanks again! > > > > On Tue, Oct 7, 2014 at 11:06 AM, tshawver > wrote: > >> As part of the work on our research environment at Quantopian > >> , I've been building an > extension > >> which renders pandas DataFrames as interactive grids in the IPython > >> notebook. The extension uses a Javascript library called SlickGrid to > >> render the grids, and the current state of the project can be found > here on > >> GitHub: https://github.com/quantopian/qgrid > >> > >> The extension is also viewable in nbviewer: > >> > http://nbviewer.ipython.org/github/quantopian/qgrid/blob/master/qgrid_demo.ipynb > >> > >> The GitHub repository contains some explanation of why I chose to > implement > >> the grid as a Python package rather than a standard IPython notebook > >> extension, which might be interesting for other people who are looking > to > >> add functionality to the notebook in a similar way. > >> > >> -Tim > >> > >> > >> > >> -- > >> View this message in context: > http://python.6.x6.nabble.com/Interactive-Grid-for-Sorting-and-Filtering-DataFrames-in-IPython-Notebook-tp5073936.html > >> Sent from the IPython - Development mailing list archive at Nabble.com. > >> _______________________________________________ > >> IPython-dev mailing list > >> IPython-dev at scipy.org > >> http://mail.scipy.org/mailman/listinfo/ipython-dev > > > > > > > > -- > > Wes Turner > > https://westurner.github.io/ > > > > -- > Wes Turner > https://westurner.github.io/ > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tra at popgen.net Wed Oct 8 08:19:03 2014 From: tra at popgen.net (Tiago Antao) Date: Wed, 8 Oct 2014 13:19:03 +0100 Subject: [IPython-dev] Temporarily suspending matplotlib inline backend Message-ID: <20141008131903.481d1b88@grandao> Hi, I have developing some notebooks to teach population genetics, and in order to spice-up some content for students I would like to generate a few animations in run-time (from PCA changes over time - this is faster than it seems). For that I need to generate a few matplotlib figures 'off-line' and then generate an animation with some extra images (i.e., I do not want the figures showing on the notebook: I will collect them as PNGs and generate an MP4 - I do not want want to use matplotlib.animation, BTW). I was wondering if there is a safe way to switch backends on server code (e.g. to Agg and them back to whatever is the default)? I have a look at the ipython source, and I am a bit afraid of potential side effects of a backend change. Thanks, Tiago From glenn.caltech at gmail.com Wed Oct 8 09:15:42 2014 From: glenn.caltech at gmail.com (G Jones) Date: Wed, 8 Oct 2014 09:15:42 -0400 Subject: [IPython-dev] Temporarily suspending matplotlib inline backend In-Reply-To: <20141008131903.481d1b88@grandao> References: <20141008131903.481d1b88@grandao> Message-ID: No need to switch backends, just use this idiom: from matplotlib.figure import Figure from matplotlib.backends.backend_agg import FigureCanvasAgg fig = Figure() ... # use the figure as you like canvas = FigureCanvasAgg(fig) fig.set_canvas(canvas) #not sure this line is strictly necessary fig.savefig('filename') On Wed, Oct 8, 2014 at 8:19 AM, Tiago Antao wrote: > Hi, > > I have developing some notebooks to teach population genetics, and in > order to spice-up some content for students I would like to generate a > few animations in run-time (from PCA changes over time - this is faster > than it seems). For that I need to generate a few matplotlib figures > 'off-line' and then generate an animation with some extra images (i.e., > I do not want the figures showing on the notebook: I will collect them > as PNGs and generate an MP4 - I do not want want to use > matplotlib.animation, BTW). > > I was wondering if there is a safe way to switch backends on server > code (e.g. to Agg and them back to whatever is the default)? I have a > look at the ipython source, and I am a bit afraid of potential side > effects of a backend change. > > Thanks, > Tiago > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tshawver at quantopian.com Wed Oct 8 10:12:01 2014 From: tshawver at quantopian.com (tshawver) Date: Wed, 8 Oct 2014 07:12:01 -0700 (PDT) Subject: [IPython-dev] Interactive Grid for Sorting and Filtering DataFrames in IPython Notebook In-Reply-To: References: <1412697961680-5073936.post@n6.nabble.com> Message-ID: <1412777521056-5074058.post@n6.nabble.com> Awesome, thanks for doing that Wes! -- View this message in context: http://python.6.x6.nabble.com/Interactive-Grid-for-Sorting-and-Filtering-DataFrames-in-IPython-Notebook-tp5073936p5074058.html Sent from the IPython - Development mailing list archive at Nabble.com. From takowl at gmail.com Wed Oct 8 15:10:44 2014 From: takowl at gmail.com (Thomas Kluyver) Date: Wed, 8 Oct 2014 12:10:44 -0700 Subject: [IPython-dev] Nbparameterise: simple forms for notebook input Message-ID: Introducing another of my side projects: Nbparameterise https://github.com/takluyver/nbparameterise The idea is that you prepare a notebook with some 'input' variable definitions in the first cell. These are standard assignments, like `a=200`, of (for now), strings, numbers and booleans. When you run nbparameterise, it will display a simple form where the user can supply new values for these variables. When the user clicks 'run notebook', their values are substituted in, the notebook is run and rendered to HTML. To see a demo of this, clone the repository, run: python3 -m nbparameterise Stock\ display.ipynb and visit http://localhost:8888/ in your browser. Notes: - It builds an HTML form, but I've tried to abstract that from the core, so you could write a local GUI or a terminal interface for it quite easily. - It's written in Python 3, but it will run notebooks in Python 2 if that's what the metadata indicates. - We can integrate backends for other languages - It's fairly rough - come and polish it. We've heard from a couple of companies doing similar things privately; I hope this can form a common base. Thomas P.S. Suggestions for a better name welcome. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ellisonbg at gmail.com Wed Oct 8 15:15:20 2014 From: ellisonbg at gmail.com (Brian Granger) Date: Wed, 8 Oct 2014 12:15:20 -0700 Subject: [IPython-dev] Nbparameterise: simple forms for notebook input In-Reply-To: References: Message-ID: Thanks Thomas! On Wed, Oct 8, 2014 at 12:10 PM, Thomas Kluyver wrote: > Introducing another of my side projects: Nbparameterise > > https://github.com/takluyver/nbparameterise > > The idea is that you prepare a notebook with some 'input' variable > definitions in the first cell. These are standard assignments, like `a=200`, > of (for now), strings, numbers and booleans. When you run nbparameterise, it > will display a simple form where the user can supply new values for these > variables. When the user clicks 'run notebook', their values are substituted > in, the notebook is run and rendered to HTML. > > To see a demo of this, clone the repository, run: > > python3 -m nbparameterise Stock\ display.ipynb > > and visit http://localhost:8888/ in your browser. > > Notes: > - It builds an HTML form, but I've tried to abstract that from the core, so > you could write a local GUI or a terminal interface for it quite easily. > - It's written in Python 3, but it will run notebooks in Python 2 if that's > what the metadata indicates. > - We can integrate backends for other languages > - It's fairly rough - come and polish it. > > We've heard from a couple of companies doing similar things privately; I > hope this can form a common base. > > Thomas > > P.S. Suggestions for a better name welcome. > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -- Brian E. Granger Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub bgranger at calpoly.edu and ellisonbg at gmail.com From hughesadam87 at gmail.com Wed Oct 8 15:17:49 2014 From: hughesadam87 at gmail.com (Adam Hughes) Date: Wed, 8 Oct 2014 15:17:49 -0400 Subject: [IPython-dev] Nbparameterise: simple forms for notebook input In-Reply-To: References: Message-ID: Awesome, we have a great example that fits this architecture. If you want to use it, we can add it to repo On Oct 8, 2014 3:11 PM, "Thomas Kluyver" wrote: > Introducing another of my side projects: Nbparameterise > > https://github.com/takluyver/nbparameterise > > The idea is that you prepare a notebook with some 'input' variable > definitions in the first cell. These are standard assignments, like > `a=200`, of (for now), strings, numbers and booleans. When you run > nbparameterise, it will display a simple form where the user can supply new > values for these variables. When the user clicks 'run notebook', their > values are substituted in, the notebook is run and rendered to HTML. > > To see a demo of this, clone the repository, run: > > python3 -m nbparameterise Stock\ display.ipynb > > and visit http://localhost:8888/ in your browser. > > Notes: > - It builds an HTML form, but I've tried to abstract that from the core, > so you could write a local GUI or a terminal interface for it quite easily. > - It's written in Python 3, but it will run notebooks in Python 2 if > that's what the metadata indicates. > - We can integrate backends for other languages > - It's fairly rough - come and polish it. > > We've heard from a couple of companies doing similar things privately; I > hope this can form a common base. > > Thomas > > P.S. Suggestions for a better name welcome. > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Wed Oct 8 15:21:11 2014 From: takowl at gmail.com (Thomas Kluyver) Date: Wed, 8 Oct 2014 12:21:11 -0700 Subject: [IPython-dev] Nbparameterise: simple forms for notebook input In-Reply-To: References: Message-ID: On 8 October 2014 12:17, Adam Hughes wrote: > Awesome, we have a great example that fits this architecture. If you want > to use it, we can add it to repo > Sure - do you want to make a PR? Thanks, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From hughesadam87 at gmail.com Wed Oct 8 15:23:57 2014 From: hughesadam87 at gmail.com (Adam Hughes) Date: Wed, 8 Oct 2014 15:23:57 -0400 Subject: [IPython-dev] Nbparameterise: simple forms for notebook input In-Reply-To: References: Message-ID: Sure, gimme a few days, out of town On Oct 8, 2014 3:21 PM, "Thomas Kluyver" wrote: > On 8 October 2014 12:17, Adam Hughes wrote: > >> Awesome, we have a great example that fits this architecture. If you >> want to use it, we can add it to repo >> > > Sure - do you want to make a PR? > > Thanks, > Thomas > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bussonniermatthias at gmail.com Wed Oct 8 16:21:31 2014 From: bussonniermatthias at gmail.com (Matthias BUSSONNIER) Date: Wed, 8 Oct 2014 22:21:31 +0200 Subject: [IPython-dev] Nbparameterise: simple forms for notebook input In-Reply-To: References: Message-ID: <3D77BBE7-4FE1-4480-AB8D-55082D7719C9@gmail.com> Great ! I think runipy already have such a feature, but I might have misunderstood : > Passing Arguments > > You can pass arguments to the notebook through environment variables. The use of environment variables is OS- and shell- dependent, but in a typical UNIX-like environment they can be passed on the command line before the program name: > > $ myvar=value runipy MyNotebook.ipynb > > Then in the notebook, to access myvar: > > from os import environ > myvar = environ['myvar'] > > environ is just a dict, so you can use .get() to fall back on a default value: > > from os import environ > myvar = environ.get('myvar', 'default!') from https://github.com/paulgb/runipy Maybe you can share part of the code. -- M Le 8 oct. 2014 ? 21:10, Thomas Kluyver a ?crit : > Introducing another of my side projects: Nbparameterise > > https://github.com/takluyver/nbparameterise > > The idea is that you prepare a notebook with some 'input' variable definitions in the first cell. These are standard assignments, like `a=200`, of (for now), strings, numbers and booleans. When you run nbparameterise, it will display a simple form where the user can supply new values for these variables. When the user clicks 'run notebook', their values are substituted in, the notebook is run and rendered to HTML. > > To see a demo of this, clone the repository, run: > > python3 -m nbparameterise Stock\ display.ipynb > > and visit http://localhost:8888/ in your browser. > > Notes: > - It builds an HTML form, but I've tried to abstract that from the core, so you could write a local GUI or a terminal interface for it quite easily. > - It's written in Python 3, but it will run notebooks in Python 2 if that's what the metadata indicates. > - We can integrate backends for other languages > - It's fairly rough - come and polish it. > > We've heard from a couple of companies doing similar things privately; I hope this can form a common base. > > Thomas > > P.S. Suggestions for a better name welcome. > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Wed Oct 8 16:41:46 2014 From: takowl at gmail.com (Thomas Kluyver) Date: Wed, 8 Oct 2014 13:41:46 -0700 Subject: [IPython-dev] Nbparameterise: simple forms for notebook input In-Reply-To: <3D77BBE7-4FE1-4480-AB8D-55082D7719C9@gmail.com> References: <3D77BBE7-4FE1-4480-AB8D-55082D7719C9@gmail.com> Message-ID: On 8 October 2014 13:21, Matthias BUSSONNIER wrote: > I think runipy already have such a feature, but I might have misunderstood > : > Kind of, but an important part of this is finding what variables can be filled in, and providing a form for them. And the rendered notebooks contain the variable definitions, so you can easily see what the input was. > Maybe you can share part of the code. If I'm understanding it correctly, there shouldn't be any code in runipy to achieve that: all you're doing is setting environment variables before you run a notebook, and then reading them in your code. That will work so long as runipy doesn't clear those environment variables. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From tritemio at gmail.com Wed Oct 8 17:08:27 2014 From: tritemio at gmail.com (Antonino Ingargiola) Date: Wed, 8 Oct 2014 14:08:27 -0700 Subject: [IPython-dev] Nbparameterise: simple forms for notebook input In-Reply-To: References: <3D77BBE7-4FE1-4480-AB8D-55082D7719C9@gmail.com> Message-ID: This is an interesting project, thanks Thomas! I'm using runipy to run a notebook (template notebook) on several data files in batch. I have (what I call) a "master-notebook" that calls the "template notebook" several times setting an environment variable to a different value each time. Each executed notebook is saved to a different file, leaving a trail of the computation.This done for the software of a publication, but I think is quite a common pattern (in terms of needs). What I have done is completely custom (but straightforward). Would be nice if nbparameterise could standardize a way to specify a "list" of values for a variable so the notebook can be executed in batch mode, saving the output for each execution. Saving in notebook format is somewhat diff-safe. What about HTML? I suspect it may be a little more tricky to diff... Thanks, Antonio On Wed, Oct 8, 2014 at 1:41 PM, Thomas Kluyver wrote: > On 8 October 2014 13:21, Matthias BUSSONNIER > wrote: > >> I think runipy already have such a feature, but I might have >> misunderstood : >> > > Kind of, but an important part of this is finding what variables can be > filled in, and providing a form for them. And the rendered notebooks > contain the variable definitions, so you can easily see what the input was. > > > Maybe you can share part of the code. > > If I'm understanding it correctly, there shouldn't be any code in runipy > to achieve that: all you're doing is setting environment variables before > you run a notebook, and then reading them in your code. That will work so > long as runipy doesn't clear those environment variables. > > Thomas > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Wed Oct 8 17:50:37 2014 From: takowl at gmail.com (Thomas Kluyver) Date: Wed, 8 Oct 2014 14:50:37 -0700 Subject: [IPython-dev] Nbparameterise: simple forms for notebook input In-Reply-To: References: <3D77BBE7-4FE1-4480-AB8D-55082D7719C9@gmail.com> Message-ID: On 8 October 2014 14:08, Antonino Ingargiola wrote: > What I have done is completely custom (but straightforward). Would be nice > if nbparameterise could standardize a way to specify a "list" of values for > a variable so the notebook can be executed in batch mode, saving the output > for each execution. Saving in notebook format is somewhat diff-safe. What > about HTML? I suspect it may be a little more tricky to diff... You could certainly use the API to do that. I've added an example: https://github.com/takluyver/nbparameterise/blob/master/batch_eg.py You could wrap that up in a higher level interface if you wanted, but I wouldn't go too high level with it: I can imagine that people might want the list of values to come from lots of different places, e.g. directory listsings, or databases. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From tritemio at gmail.com Wed Oct 8 18:14:48 2014 From: tritemio at gmail.com (Antonino Ingargiola) Date: Wed, 8 Oct 2014 15:14:48 -0700 Subject: [IPython-dev] Nbparameterise: simple forms for notebook input In-Reply-To: References: <3D77BBE7-4FE1-4480-AB8D-55082D7719C9@gmail.com> Message-ID: Interesting. Your example is exactly what I have done with runipy, just cleaner through a proper API. My suggestion however was different. I was thinking of saving the list of parameter values in the "template notebook" so that it will contain all the information about the batch processing. The "runner", in the most basic mode, would just execute the notebook iterating through the specified values. Think about that. If I have 5 notebooks for 5 different analysis, I have to maintain 5 different sets of parameters in 5 different batch-runner functions (or on disk or other location) to be able to batch-execute all of them (and this is what I do right know). If we would have a conventional way of embedding the list of parameters in each notebook, it would make the process much more transparent with all the information in one place (the template notebook). Then I can use the same exact function to batch-execute each of the 5 templates, regardless of which parameters are varied. And for "custom" executions I can always use the full API to override parameters, so no loss of generality. Antonio On Wed, Oct 8, 2014 at 2:50 PM, Thomas Kluyver wrote: > On 8 October 2014 14:08, Antonino Ingargiola wrote: > >> What I have done is completely custom (but straightforward). Would be >> nice if nbparameterise could standardize a way to specify a "list" of >> values for a variable so the notebook can be executed in batch mode, saving >> the output for each execution. Saving in notebook format is somewhat >> diff-safe. What about HTML? I suspect it may be a little more tricky to >> diff... > > > You could certainly use the API to do that. I've added an example: > https://github.com/takluyver/nbparameterise/blob/master/batch_eg.py > > You could wrap that up in a higher level interface if you wanted, but I > wouldn't go too high level with it: I can imagine that people might want > the list of values to come from lots of different places, e.g. directory > listsings, or databases. > > Thomas > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Wed Oct 8 18:24:03 2014 From: takowl at gmail.com (Thomas Kluyver) Date: Wed, 8 Oct 2014 15:24:03 -0700 Subject: [IPython-dev] Nbparameterise: simple forms for notebook input In-Reply-To: References: <3D77BBE7-4FE1-4480-AB8D-55082D7719C9@gmail.com> Message-ID: On 8 October 2014 15:14, Antonino Ingargiola wrote: > My suggestion however was different. I was thinking of saving the list of > parameter values in the "template notebook" so that it will contain all the > information about the batch processing. The "runner", in the most basic > mode, would just execute the notebook iterating through the specified > values. I see what you mean. That's an interesting idea, and I'll keep it in the back of my mind, but it's not something I want to tackle at the moment. It would certainly be possible to store the value ranges in notebook metadata, and generalise the example I gave to use that instead of hardcoded lists. Designing a good UI for writing the notebook metadata might be trickier, but you could do that as an nbextension. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From tritemio at gmail.com Wed Oct 8 19:33:51 2014 From: tritemio at gmail.com (Antonino Ingargiola) Date: Wed, 8 Oct 2014 16:33:51 -0700 Subject: [IPython-dev] Nbparameterise: simple forms for notebook input In-Reply-To: References: <3D77BBE7-4FE1-4480-AB8D-55082D7719C9@gmail.com> Message-ID: But what do you think about using real python lists with a conventional names or with a "magic" comment (or even a cell magic?) from which the values are extracted. This would not require building any custom GUI. The template notebook will have a cell like this: ipyparams_datafile = [...] that is replaced in the executed notebook by a cell like this: datafile = ".." It may be a little hackish but it is extremely simple and would already be a huge improvement over the current situation. Antonio On Wed, Oct 8, 2014 at 3:24 PM, Thomas Kluyver wrote: > On 8 October 2014 15:14, Antonino Ingargiola wrote: > >> My suggestion however was different. I was thinking of saving the list of >> parameter values in the "template notebook" so that it will contain all the >> information about the batch processing. The "runner", in the most basic >> mode, would just execute the notebook iterating through the specified >> values. > > > I see what you mean. That's an interesting idea, and I'll keep it in the > back of my mind, but it's not something I want to tackle at the moment. It > would certainly be possible to store the value ranges in notebook metadata, > and generalise the example I gave to use that instead of hardcoded lists. > Designing a good UI for writing the notebook metadata might be trickier, > but you could do that as an nbextension. > > Thomas > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Wed Oct 8 19:57:47 2014 From: takowl at gmail.com (Thomas Kluyver) Date: Wed, 8 Oct 2014 16:57:47 -0700 Subject: [IPython-dev] Nbparameterise: simple forms for notebook input In-Reply-To: References: <3D77BBE7-4FE1-4480-AB8D-55082D7719C9@gmail.com> Message-ID: On 8 October 2014 16:33, Antonino Ingargiola wrote: > But what do you think about using real python lists with a conventional > names or with a "magic" comment (or even a cell magic?) from which the > values are extracted. This would not require building any custom GUI. > I don't like this abstraction. I spent a while writing different sentences here trying to explain why, but it really just feels inflexible. Part of my concern is that it breaks the model that the notebook can be run as is, without substituting the first cell. Of course, you could get round that with a cell magic, but that doesn't feel quite right either, and it doesn't generalise neatly to other languages. > It may be a little hackish Hackish solutions are great sometimes, but part of the purpose of nbparameterise is to figure out the right ways to do things, and make those easy. So I'd rather we kept thinking about what would be a good abstraction for this, rather than building a hackish one quickly. Of course, it's easy for me to say this, because I'm not the one who wants to use it immediately ;-). Of course, if you want to build this for your own use, go right ahead. Best wishes, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From kikocorreoso at gmail.com Thu Oct 9 17:57:35 2014 From: kikocorreoso at gmail.com (Kiko) Date: Thu, 9 Oct 2014 23:57:35 +0200 Subject: [IPython-dev] Simple kernel not working on a venv Message-ID: Hi all, I am following the recipe found here: http://nbviewer.ipython.org/github/ipython-books/cookbook-code/blob/master/notebooks/chapter01_basic/06_kernel.ipynb to create a custom and simple kernel. The kernel is working if I don't use a venv but inside a venv the notebook is 'busy' and I can't run code. It seems that the qtconsole is working fine with and without the venv. My tests are made on linux mint 64bit, chrome v38 and firefox v32. Kind regards. -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Thu Oct 9 18:06:22 2014 From: takowl at gmail.com (Thomas Kluyver) Date: Thu, 9 Oct 2014 15:06:22 -0700 Subject: [IPython-dev] Simple kernel not working on a venv In-Reply-To: References: Message-ID: On 9 October 2014 14:57, Kiko wrote: > The kernel is working if I don't use a venv but inside a venv the notebook > is 'busy' and I can't run code. My guess would be that it's somehow importing a different version of your kernel, or a different version of IPython, inside the venv. I can't think that anything we do should care whether the kernel runs in a venv or not. If you remove the env and recreate it with fresh installations of IPython and your kernel, do you still see the issue? Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From kikocorreoso at gmail.com Thu Oct 9 18:16:23 2014 From: kikocorreoso at gmail.com (Kiko) Date: Fri, 10 Oct 2014 00:16:23 +0200 Subject: [IPython-dev] Simple kernel not working on a venv In-Reply-To: References: Message-ID: 2014-10-10 0:06 GMT+02:00 Thomas Kluyver : > On 9 October 2014 14:57, Kiko wrote: > >> The kernel is working if I don't use a venv but inside a venv the >> notebook is 'busy' and I can't run code. > > > My guess would be that it's somehow importing a different version of your > kernel, or a different version of IPython, inside the venv. I can't think > that anything we do should care whether the kernel runs in a venv or not. > If you remove the env and recreate it with fresh installations of IPython > and your kernel, do you still see the issue? > Let me check that tomorrow. Thanks, Thomas. > > > Thomas > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ijstokes at alumni.uwaterloo.ca Thu Oct 9 21:58:23 2014 From: ijstokes at alumni.uwaterloo.ca (Ian Stokes-Rees) Date: Thu, 09 Oct 2014 21:58:23 -0400 Subject: [IPython-dev] Nbparameterise: simple forms for notebook input Message-ID: <54373D3F.8020108@alumni.uwaterloo.ca> FWIW, We (at Continuum) have done some work on something similar, loosely tied to conda: https://github.com/conda/conda-launch It supports 3 modes: 1. CLI invocation of a notebook 2. auto-generated web-form of a notebook 3. RESTful API with URL (GET) or POST-encoded arguments It doesn't require any customization of IPython, and the parameterization can either be in the notebook meta-data, or "inline" if the last cell is a "raw" cell with the appropriate JSON. The parameterization meta-data can also include a set of dependencies, in which case a custom conda-environment will be created in which the notebook will be run. The goal is to be able to generate notebook output (HTML, data file, figures, return specific serialized references) with no retained state or persisted processes, basically a notebook-based CGI python script. It runs reasonably well right now, but the input form styling is pants (my CSS skills stink!). I haven't had time to read the thread on nbparameterize, but look forward to exploring it next week. Perhaps the two efforts can be merged. Regards, Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From rgbkrk at gmail.com Thu Oct 9 22:16:57 2014 From: rgbkrk at gmail.com (Kyle Kelley) Date: Thu, 9 Oct 2014 21:16:57 -0500 Subject: [IPython-dev] Docker containers Message-ID: Hey IPythonistas and Jovyans, Our original Docker thread got *lots* of input. It's time I write about where things stand right now, and invite you to take a look at the current images (*ahem* send pull requests). Docker Hub currently builds the most recent copy of github.com/ipython/ipython master on the ipython/ipython image ( https://registry.hub.docker.com/u/ipython/ipython/). When IPython 3 releases, I'd like to make tagged versions of this base image so that you get a stable image and development images ipython/ipython:3.0 - stable 3.0 release ipython/ipython:3.x - current 3.x branch (backported fixes) ipython/ipython:latest - current master Basing off this image are three other images: ipython/scipystack (https://registry.hub.docker.com/u/ipython/scipystack/) * Based off of the ipython/ipython image * All of the scipystack (+ a few more opinionated adds) * Big thank you to Olivier Grisel for a *wonderful* OpenBLAS installation + numpy + scipy config ipython/scipyserver (https://registry.hub.docker.com/u/ipython/scipyserver/) * Based off of scipystack, runs a notebook server ipython/notebook (https://registry.hub.docker.com/u/ipython/notebook/) * Based off of ipython/ipython, runs a notebook server We've seen folks basing images off of ipython/scipystack and I think that's wonderful. We also have an image in development that sets up the Julia and R kernels (to be toggled with the kernel switcher), in https://github.com/jupyter/tmpnb/blob/master/image/Dockerfile. If you have feedback, let us know! -- Kyle Kelley (@rgbkrk ; http://lambdaops.com) -------------- next part -------------- An HTML attachment was scrubbed... URL: From zvoros at gmail.com Fri Oct 10 05:28:08 2014 From: zvoros at gmail.com (=?UTF-8?B?Wm9sdMOhbiBWw7Zyw7Zz?=) Date: Fri, 10 Oct 2014 11:28:08 +0200 Subject: [IPython-dev] notebook validation failed error Message-ID: <5437A6A8.1060707@gmail.com> Hi all, After pulling the latest version from master, I get a "Notebook validation failed" error, whenever I try to open an existing notebook, or save one. This happens even, if I create a new notebook and I try to save it. Is this error somehow related to the change in the notebook schema? In any case, I just wanted to ask if someone else has bumped into the same problem, or perhaps, something is wrong at my end. Cheers, Zolt?n From takowl at gmail.com Fri Oct 10 12:38:50 2014 From: takowl at gmail.com (Thomas Kluyver) Date: Fri, 10 Oct 2014 09:38:50 -0700 Subject: [IPython-dev] notebook validation failed error In-Reply-To: <5437A6A8.1060707@gmail.com> References: <5437A6A8.1060707@gmail.com> Message-ID: Hi Zolt?n, We recently merged some changes to the validation code itself. I'm not sure why you'd get it on brand new notebooks, but if you can send us one of the existing notebooks that fails to validate, we'll look into what the problem is. Thanks, Thomas On 10 October 2014 02:28, Zolt?n V?r?s wrote: > Hi all, > > After pulling the latest version from master, I get a "Notebook > validation failed" error, whenever I try to open an existing notebook, > or save one. This happens even, if I create a new notebook and I try to > save it. Is this error somehow related to the change in the notebook > schema? > > In any case, I just wanted to ask if someone else has bumped into the > same problem, or perhaps, something is wrong at my end. > > Cheers, > Zolt?n > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tritemio at gmail.com Fri Oct 10 13:24:43 2014 From: tritemio at gmail.com (Antonino Ingargiola) Date: Fri, 10 Oct 2014 10:24:43 -0700 Subject: [IPython-dev] Nbparameterise: simple forms for notebook input In-Reply-To: <54373D3F.8020108@alumni.uwaterloo.ca> References: <54373D3F.8020108@alumni.uwaterloo.ca> Message-ID: Hi, I like the idea of putting parameters in raw cells, I believe the all the parameter specifications should stay inside notebook. Would be possible to specify a list so that the notebook is executed for each value in the list? This is what I was discussing in the previous thread. It would make a common workflow very easy and transparent. Antonio On Thu, Oct 9, 2014 at 6:58 PM, Ian Stokes-Rees < ijstokes at alumni.uwaterloo.ca> wrote: > FWIW, We (at Continuum) have done some work on something similar, loosely > tied to conda: > > https://github.com/conda/conda-launch > > It supports 3 modes: > > 1. CLI invocation of a notebook > 2. auto-generated web-form of a notebook > 3. RESTful API with URL (GET) or POST-encoded arguments > > It doesn't require any customization of IPython, and the parameterization > can either be in the notebook meta-data, or "inline" if the last cell is a > "raw" cell with the appropriate JSON. > > The parameterization meta-data can also include a set of dependencies, in > which case a custom conda-environment will be created in which the notebook > will be run. The goal is to be able to generate notebook output (HTML, > data file, figures, return specific serialized references) with no retained > state or persisted processes, basically a notebook-based CGI python script. > > It runs reasonably well right now, but the input form styling is pants (my > CSS skills stink!). > > I haven't had time to read the thread on nbparameterize, but look forward > to exploring it next week. Perhaps the two efforts can be merged. > > Regards, > > Ian > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjaminrk at gmail.com Fri Oct 10 13:43:41 2014 From: benjaminrk at gmail.com (MinRK) Date: Fri, 10 Oct 2014 10:43:41 -0700 Subject: [IPython-dev] notebook validation failed error In-Reply-To: References: <5437A6A8.1060707@gmail.com> Message-ID: The validation might fail if the cached javascript was out of date. I suspect that was the case, especially if the validation failure was on `additionalProperties` of a code cell. -MinRK On Fri, Oct 10, 2014 at 9:38 AM, Thomas Kluyver wrote: > Hi Zolt?n, > > We recently merged some changes to the validation code itself. I'm not > sure why you'd get it on brand new notebooks, but if you can send us one of > the existing notebooks that fails to validate, we'll look into what the > problem is. > > Thanks, > Thomas > > On 10 October 2014 02:28, Zolt?n V?r?s wrote: > >> Hi all, >> >> After pulling the latest version from master, I get a "Notebook >> validation failed" error, whenever I try to open an existing notebook, >> or save one. This happens even, if I create a new notebook and I try to >> save it. Is this error somehow related to the change in the notebook >> schema? >> >> In any case, I just wanted to ask if someone else has bumped into the >> same problem, or perhaps, something is wrong at my end. >> >> Cheers, >> Zolt?n >> _______________________________________________ >> IPython-dev mailing list >> IPython-dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/ipython-dev >> > > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Fri Oct 10 14:02:02 2014 From: takowl at gmail.com (Thomas Kluyver) Date: Fri, 10 Oct 2014 11:02:02 -0700 Subject: [IPython-dev] Nbparameterise: simple forms for notebook input In-Reply-To: <54373D3F.8020108@alumni.uwaterloo.ca> References: <54373D3F.8020108@alumni.uwaterloo.ca> Message-ID: Thanks Ian - conda-launch was one of a few projects that prompted me to work on this. In particular, great though conda is, I don't think creating forms and executing notebooks should be tied to a particular packaging system. Looking through the description of conda-launch, I think what I want to think about most is the different ways of describing the input parameters. Conda-launch uses JSON metadata that is potentially more flexible, but very distinct from the normal workflow of writing a notebook. Nbparameterise currently looks for an initial "definitions cell", which is less extensible, but very natural for the notebook author. Part of me wants to generalise nbparameterise to handle all possible ways of describing inputs, but another part of me wants to hammer out a common way of doing this, so notebook authors don't have to learn different things for different systems. I'll keep thinking about it. Best wishes, Thomas On 9 October 2014 18:58, Ian Stokes-Rees wrote: > FWIW, We (at Continuum) have done some work on something similar, loosely > tied to conda: > > https://github.com/conda/conda-launch > > It supports 3 modes: > > 1. CLI invocation of a notebook > 2. auto-generated web-form of a notebook > 3. RESTful API with URL (GET) or POST-encoded arguments > > It doesn't require any customization of IPython, and the parameterization > can either be in the notebook meta-data, or "inline" if the last cell is a > "raw" cell with the appropriate JSON. > > The parameterization meta-data can also include a set of dependencies, in > which case a custom conda-environment will be created in which the notebook > will be run. The goal is to be able to generate notebook output (HTML, > data file, figures, return specific serialized references) with no retained > state or persisted processes, basically a notebook-based CGI python script. > > It runs reasonably well right now, but the input form styling is pants (my > CSS skills stink!). > > I haven't had time to read the thread on nbparameterize, but look forward > to exploring it next week. Perhaps the two efforts can be merged. > > Regards, > > Ian > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zvoros at gmail.com Fri Oct 10 14:03:58 2014 From: zvoros at gmail.com (=?windows-1252?Q?Zolt=E1n_V=F6r=F6s?=) Date: Fri, 10 Oct 2014 20:03:58 +0200 Subject: [IPython-dev] notebook validation failed error In-Reply-To: References: <5437A6A8.1060707@gmail.com> Message-ID: <54381F8E.1020505@gmail.com> Hi Thomas/Min, > The validation might fail if the cached javascript was out of date. I > suspect that was the case, especially if the validation failure was on > `additionalProperties` of a code cell. > > -MinRK On 10/10/2014 06:38 PM, Thomas Kluyver wrote: > Hi Zolt?n, > > We recently merged some changes to the validation code itself. I'm not > sure why you'd get it on brand new notebooks, but if you can send us > one of the existing notebooks that fails to validate, we'll look into > what the problem is. > I think clearing the cache has solved the problem. Many thanks for the pointer! I am going to disable chaching altogether, since I use chrome only with ipython, anyway. Cheers, Zolt?n From fperez.net at gmail.com Fri Oct 10 18:59:22 2014 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 10 Oct 2014 15:59:22 -0700 Subject: [IPython-dev] notebook validation failed error In-Reply-To: <54381F8E.1020505@gmail.com> References: <5437A6A8.1060707@gmail.com> <54381F8E.1020505@gmail.com> Message-ID: On Fri, Oct 10, 2014 at 11:03 AM, Zolt?n V?r?s wrote: > I think clearing the cache has solved the problem. Many thanks for the > pointer! I am going to disable chaching altogether, since I use chrome > only with ipython, anyway. > To all chrome users, run, don't walk to install this clear cache extension: https://chrome.google.com/webstore/detail/clear-cache/cppjkneekbjaeellbfkmgnhonkkjfpdn?hl=en life saver when dealing with the nb. -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail -------------- next part -------------- An HTML attachment was scrubbed... URL: From damianavila at gmail.com Fri Oct 10 19:06:40 2014 From: damianavila at gmail.com (=?UTF-8?Q?Dami=C3=A1n_Avila?=) Date: Fri, 10 Oct 2014 20:06:40 -0300 Subject: [IPython-dev] notebook validation failed error In-Reply-To: References: <5437A6A8.1060707@gmail.com> <54381F8E.1020505@gmail.com> Message-ID: And it is really working?? my setup when I develop for the notebook is use: * incognito mode * dev tools opened * clear the cache and force the page reload (3rd option of reload in the reload icon when you click with the "secondary/right" button And with all this precautions, sometimes I have close the browser or/and shutdown the server to see the changes... really... I think I will have to try it ;-) 2014-10-10 19:59 GMT-03:00 Fernando Perez : > > On Fri, Oct 10, 2014 at 11:03 AM, Zolt?n V?r?s wrote: > >> I think clearing the cache has solved the problem. Many thanks for the >> pointer! I am going to disable chaching altogether, since I use chrome >> only with ipython, anyway. >> > > To all chrome users, run, don't walk to install this clear cache extension: > > > https://chrome.google.com/webstore/detail/clear-cache/cppjkneekbjaeellbfkmgnhonkkjfpdn?hl=en > > life saver when dealing with the nb. > > > -- > Fernando Perez (@fperez_org; http://fperez.org) > fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) > fernando.perez-at-berkeley: contact me here for any direct mail > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > > -- *Dami?n* -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Fri Oct 10 19:33:49 2014 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 10 Oct 2014 16:33:49 -0700 Subject: [IPython-dev] notebook validation failed error In-Reply-To: References: <5437A6A8.1060707@gmail.com> <54381F8E.1020505@gmail.com> Message-ID: It seems to work for me so far, but maybe I haven't had a problem bad enough :) Cache issues seem to be the new bane of our existence... On Fri, Oct 10, 2014 at 4:06 PM, Dami?n Avila wrote: > And it is really working?? my setup when I develop for the notebook is use: > > * incognito mode > * dev tools opened > * clear the cache and force the page reload (3rd option of reload in the > reload icon when you click with the "secondary/right" button > And with all this precautions, sometimes I have close the browser or/and > shutdown the server to see the changes... really... > > I think I will have to try it ;-) > > > 2014-10-10 19:59 GMT-03:00 Fernando Perez : > >> >> On Fri, Oct 10, 2014 at 11:03 AM, Zolt?n V?r?s wrote: >> >>> I think clearing the cache has solved the problem. Many thanks for the >>> pointer! I am going to disable chaching altogether, since I use chrome >>> only with ipython, anyway. >>> >> >> To all chrome users, run, don't walk to install this clear cache >> extension: >> >> >> https://chrome.google.com/webstore/detail/clear-cache/cppjkneekbjaeellbfkmgnhonkkjfpdn?hl=en >> >> life saver when dealing with the nb. >> >> >> -- >> Fernando Perez (@fperez_org; http://fperez.org) >> fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) >> fernando.perez-at-berkeley: contact me here for any direct mail >> >> _______________________________________________ >> IPython-dev mailing list >> IPython-dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/ipython-dev >> >> > > > -- > *Dami?n* > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > > -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail -------------- next part -------------- An HTML attachment was scrubbed... URL: From wstein at gmail.com Fri Oct 10 22:09:38 2014 From: wstein at gmail.com (William Stein) Date: Fri, 10 Oct 2014 19:09:38 -0700 Subject: [IPython-dev] notebook validation failed error In-Reply-To: References: <5437A6A8.1060707@gmail.com> <54381F8E.1020505@gmail.com> Message-ID: On Oct 10, 2014 4:34 PM, "Fernando Perez" wrote: > > It seems to work for me so far, but maybe I haven't had a problem bad enough :) > > Cache issues seem to be the new bane of our existence... > Add a version number in the client code and server code. Have the client request the latest version periodically. If the client version is too old, notify the user. I do this with SageMathCloud, and Google does it with Google Analytics. The main point is that at least it is clear whether or not one is using the current version of code. For this to work you have to combine everything into one file (eg with uglify.js); anyway whatever else you do, you definitely need a periodic client version check. > On Fri, Oct 10, 2014 at 4:06 PM, Dami?n Avila wrote: >> >> And it is really working?? my setup when I develop for the notebook is use: >> >> * incognito mode >> * dev tools opened >> * clear the cache and force the page reload (3rd option of reload in the reload icon when you click with the "secondary/right" button >> And with all this precautions, sometimes I have close the browser or/and shutdown the server to see the changes... really... >> >> I think I will have to try it ;-) >> >> >> 2014-10-10 19:59 GMT-03:00 Fernando Perez : >>> >>> >>> On Fri, Oct 10, 2014 at 11:03 AM, Zolt?n V?r?s wrote: >>>> >>>> I think clearing the cache has solved the problem. Many thanks for the >>>> pointer! I am going to disable chaching altogether, since I use chrome >>>> only with ipython, anyway. >>> >>> >>> To all chrome users, run, don't walk to install this clear cache extension: >>> >>> https://chrome.google.com/webstore/detail/clear-cache/cppjkneekbjaeellbfkmgnhonkkjfpdn?hl=en >>> >>> life saver when dealing with the nb. >>> >>> >>> -- >>> Fernando Perez (@fperez_org; http://fperez.org) >>> fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) >>> fernando.perez-at-berkeley: contact me here for any direct mail >>> >>> _______________________________________________ >>> IPython-dev mailing list >>> IPython-dev at scipy.org >>> http://mail.scipy.org/mailman/listinfo/ipython-dev >>> >> >> >> >> -- >> Dami?n >> >> _______________________________________________ >> IPython-dev mailing list >> IPython-dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/ipython-dev >> > > > > -- > Fernando Perez (@fperez_org; http://fperez.org) > fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) > fernando.perez-at-berkeley: contact me here for any direct mail > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bussonniermatthias at gmail.com Sat Oct 11 04:57:04 2014 From: bussonniermatthias at gmail.com (Matthias Bussonnier) Date: Sat, 11 Oct 2014 10:57:04 +0200 Subject: [IPython-dev] notebook validation failed error In-Reply-To: References: <5437A6A8.1060707@gmail.com> <54381F8E.1020505@gmail.com> Message-ID: <692C565B-265B-4803-A9C6-0CA90D4D7E2C@gmail.com> Le 11 oct. 2014 ? 04:09, William Stein a ?crit : > > On Oct 10, 2014 4:34 PM, "Fernando Perez" wrote: > > > > It seems to work for me so far, but maybe I haven't had a problem bad enough :) > > > > Cache issues seem to be the new bane of our existence... > > > > Add a version number in the client code and server code. Have the client request the latest version periodically. If the client version is too old, notify the user. I do this with SageMathCloud, and Google does it with Google Analytics. The main point is that at least it is clear whether or not one is using the current version of code. > Which is not very practical when developing which is when theses issues arise most of the time. There are pull requests that should solve that by embedding file hash into the URL and aggressively deactivate caching in the browser on master. -- M > For this to work you have to combine everything into one file (eg with uglify.js); anyway whatever else you do, you definitely need a periodic client version check. > > > On Fri, Oct 10, 2014 at 4:06 PM, Dami?n Avila wrote: > >> > >> And it is really working?? my setup when I develop for the notebook is use: > >> > >> * incognito mode > >> * dev tools opened > >> * clear the cache and force the page reload (3rd option of reload in the reload icon when you click with the "secondary/right" button > >> And with all this precautions, sometimes I have close the browser or/and shutdown the server to see the changes... really... > >> > >> I think I will have to try it ;-) > >> > >> > >> 2014-10-10 19:59 GMT-03:00 Fernando Perez : > >>> > >>> > >>> On Fri, Oct 10, 2014 at 11:03 AM, Zolt?n V?r?s wrote: > >>>> > >>>> I think clearing the cache has solved the problem. Many thanks for the > >>>> pointer! I am going to disable chaching altogether, since I use chrome > >>>> only with ipython, anyway. > >>> > >>> > >>> To all chrome users, run, don't walk to install this clear cache extension: > >>> > >>> https://chrome.google.com/webstore/detail/clear-cache/cppjkneekbjaeellbfkmgnhonkkjfpdn?hl=en > >>> > >>> life saver when dealing with the nb. > >>> > >>> > >>> -- > >>> Fernando Perez (@fperez_org; http://fperez.org) > >>> fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) > >>> fernando.perez-at-berkeley: contact me here for any direct mail > >>> > >>> _______________________________________________ > >>> IPython-dev mailing list > >>> IPython-dev at scipy.org > >>> http://mail.scipy.org/mailman/listinfo/ipython-dev > >>> > >> > >> > >> > >> -- > >> Dami?n > >> > >> _______________________________________________ > >> IPython-dev mailing list > >> IPython-dev at scipy.org > >> http://mail.scipy.org/mailman/listinfo/ipython-dev > >> > > > > > > > > -- > > Fernando Perez (@fperez_org; http://fperez.org) > > fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) > > fernando.perez-at-berkeley: contact me here for any direct mail > > > > _______________________________________________ > > IPython-dev mailing list > > IPython-dev at scipy.org > > http://mail.scipy.org/mailman/listinfo/ipython-dev > > > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From kikocorreoso at gmail.com Sat Oct 11 06:14:26 2014 From: kikocorreoso at gmail.com (Kiko) Date: Sat, 11 Oct 2014 12:14:26 +0200 Subject: [IPython-dev] Simple kernel not working on a venv In-Reply-To: References: Message-ID: 2014-10-10 0:16 GMT+02:00 Kiko : > > > 2014-10-10 0:06 GMT+02:00 Thomas Kluyver : > >> On 9 October 2014 14:57, Kiko wrote: >> >>> The kernel is working if I don't use a venv but inside a venv the >>> notebook is 'busy' and I can't run code. >> >> >> My guess would be that it's somehow importing a different version of your >> kernel, or a different version of IPython, inside the venv. I can't think >> that anything we do should care whether the kernel runs in a venv or not. >> If you remove the env and recreate it with fresh installations of IPython >> and your kernel, do you still see the issue? >> > > Let me check that tomorrow. > > Thanks, Thomas. > Ok, I installed a new linux mint, compiled python 3.4.2 from source, ipython dev version on a venv, I also included kernel.json on IPYTHONDIR/kernels/mykernel/kernel.json. On windows I diddn't include the kernel.json on that folder and now it is working. On windows I didn't include that fiile on the kernels folder and it was working. ?? Maybe it was related with this issue on ubuntu boxes ( http://www.reddit.com/r/Python/comments/2hha8n/python_33_and_34_virtual_environments_pyvenv_are/ ). Thanks. > >> >> >> Thomas >> >> _______________________________________________ >> IPython-dev mailing list >> IPython-dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/ipython-dev >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lucienboillod at gmail.com Sat Oct 11 09:23:00 2014 From: lucienboillod at gmail.com (Lucien Boillod) Date: Sat, 11 Oct 2014 15:23:00 +0200 Subject: [IPython-dev] Wrap Javascript (D3.js) file into IPython Widget Message-ID: Hi all, I currently work with d3.js ans I almost finished (fidd here: http://jsfiddle.net/mv9L7hjq/15/), but now I have to include this part into the project (https://www.lrde.epita.fr/wiki/Vaucanson), so basically we use IPython notebook for the graphical user interface, and what we would like to do is simply wrap my work on D3 into a IPython widget, so that we can display it when we need it. I have made some research about IPython widget using Javascript but all I found is something like that (http://nbviewer.ipython.org/github/ipython/ipython/blob/master/examples/Interactive%20Widgets/Custom%20Widget%20-%20Hello%20World.ipynb) But clearly I don?t want to rewrite the whole javascript file, but just load it and wrap both (HTML/CSS/JS) into a widget. I wonder if there is a way to do that ? Thank you, Cheers, LB -------------- next part -------------- An HTML attachment was scrubbed... URL: From nick.bollweg at gmail.com Sat Oct 11 14:23:28 2014 From: nick.bollweg at gmail.com (Nicholas Bollweg) Date: Sat, 11 Oct 2014 14:23:28 -0400 Subject: [IPython-dev] Wrap Javascript (D3.js) file into IPython Widget In-Reply-To: References: Message-ID: You can stick the whole thing in an iframe, and it will work pretty much as it is right now... the crucial bit of magic will be keeping that link between those two documents. Here is a gist with an untested 20% solution: https://gist.github.com/bollwyvl/f0ba3e16a5a824d53dc8 > Now the widget and the d3 stuff will exist in two separate documents, with two different javascript contexts, but from the view you can touch the frame with view.frame... the other direction is trickier, and if you end up needing to do that, just rewrite the whole thing to be outside of a frame :) - You may want to explore some patterns, like http://bost.ocks.org/mike/chart/, to encapsulate the relationship between data/drawing inside the iframe: right now, if a widget were to update - You'll probably want to explore d3.dispatch, so that you can create custom events (like nodeAdd, nodeRemove, etc) that the widget can react to. This may be a cleaner way than trying to "share" data - be careful of force layout changing your data (nodes, links) in-place, adding x, y, px, py, etc. - when a user changes nodes/links form outside the visualization, you'll probably need to do some things to force... but you'll want to run them at the same time. - look at Backbone.Model.changed and Widget.hold_sync to make sure that this happens atomically... or get ready for a lot of svg errors. All that being said: a well-structured backbone view, which every widget must be, will be more maintainable in the long run. Just having evented get/set on the model, and maybe some custom collections, might be worth the price of admission. Good luck! -------------- next part -------------- An HTML attachment was scrubbed... URL: From claresloggett at gmail.com Sun Oct 12 01:35:34 2014 From: claresloggett at gmail.com (Clare Sloggett) Date: Sun, 12 Oct 2014 16:35:34 +1100 Subject: [IPython-dev] Jupyterhub behind NGINX redirect Message-ID: Hi all, Am I right in thinking this list is also the right place for questions about Jupyterhub? I'm trying to set up Jupyterhub for multiple users, on the same server where we are running several other services. Currently there are NGINX redirects in place to these other services. I'd like to set up http:///ipython/ to redirect to a port on localhost and run Jupyterhub over that port. Is this possible? I've got this working in the past with single-user IPython Notebook. For that, I set config options c.NotebookApp.base_project_url, c.NotebookApp.base_kernel_url, and c.NotebookApp.webapp_settings to be aware of the ipython/ URL prefix. We used an NGINX redirect to forward requests and handle websockets properly, which looked like location /ipython/ { proxy_pass http://127.0.0.1:9510; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; } This worked for the old single-user notebook. But, I'm not clear on the model Jupyterhub is using (and I'm not much of a sysadmin). I read through the command-line options, thought from them that I don't need to strip the /ipython/ from the incoming requests, and have tried using a similar redirect to the above and setting --JupyterHubApp.base_url='ipython/' and also setting --port 9510. This does seem to see the incoming requests but results in amusing requests like 500 GET /ipython/ipython/hub/ipython/ipython/hub/ipython/ipython/hub/ipython/ipython/hub/ipython/ipython/hub/ipython/ipython/hub/ipython/ipython/hub/ipython/ipython/hub/ipython/ipython/hub/ipython/ipython/hub/ipython/ipython/hub/ipython/ipython/hub/ipython/ipython/hub/ipython/ipython/hub/ipython/ipython/hub/...... ... so clearly I haven't understood what's going on. Does anyone have any pointers on how this should work? I haven't read any docs other than the README and the command-line parameter information, so apologies if there's something obvious that I didn't look at. I'm also interested in running Jupyterhub as a daemon, so if that's something that's been done before and there's anything I should know, that would be great. Any help much appreciated! Clare -------------- next part -------------- An HTML attachment was scrubbed... URL: From doug.blank at gmail.com Sun Oct 12 09:44:10 2014 From: doug.blank at gmail.com (Doug Blank) Date: Sun, 12 Oct 2014 09:44:10 -0400 Subject: [IPython-dev] Jupyterhub behind NGINX redirect In-Reply-To: References: Message-ID: On Sun, Oct 12, 2014 at 1:35 AM, Clare Sloggett wrote: > > Hi all, > > Am I right in thinking this list is also the right place for questions about Jupyterhub? > > I'm trying to set up Jupyterhub for multiple users, on the same server where we are running several other services. Currently there are NGINX redirects in place to these other services. I'd like to set up http:///ipython/ to redirect to a port on localhost and run Jupyterhub over that port. Is this possible? > > I've got this working in the past with single-user IPython Notebook. For that, I set config options c.NotebookApp.base_project_url, c.NotebookApp.base_kernel_url, and c.NotebookApp.webapp_settings to be aware of the ipython/ URL prefix. We used an NGINX redirect to forward requests and handle websockets properly, which looked like > > location /ipython/ { > proxy_pass http://127.0.0.1:9510; > proxy_set_header Host $host; > proxy_set_header X-Real-IP $remote_addr; > proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; > proxy_set_header Upgrade $http_upgrade; > proxy_set_header Connection "upgrade"; > } > > This worked for the old single-user notebook. But, I'm not clear on the model Jupyterhub is using (and I'm not much of a sysadmin). I read through the command-line options, thought from them that I don't need to strip the /ipython/ from the incoming requests, and have tried using a similar redirect to the above and setting --JupyterHubApp.base_url='ipython/' and also setting --port 9510. This does seem to see the incoming requests but results in amusing requests like > > 500 GET /ipython/ipython/hub/ipython/ipython/hub/ipython/ipython/hub/ipython/ipython/hub/ipython/ipython/hub/ipython/ipython/hub/ipython/ipython/hub/ipython/ipython/hub/ipython/ipython/hub/ipython/ipython/hub/ipython/ipython/hub/ipython/ipython/hub/ipython/ipython/hub/ipython/ipython/hub/ipython/ipython/hub/...... > > ... so clearly I haven't understood what's going on. Does anyone have any pointers on how this should work? I haven't read any docs other than the README and the command-line parameter information, so apologies if there's something obvious that I didn't look at. > > I'm also interested in running Jupyterhub as a daemon, so if that's something that's been done before and there's anything I should know, that would be great. There is one other document, other than the README.md and the reported issues [1]: https://github.com/jupyter/jupyterhub/wiki/Using-sudo-to-run-the-server-as-non-root Despite the warning at the top of that page, you can make jupyterhub work on some systems (eg, Linux, such as Ubuntu) using sudo, but not as root. I suspect that this would be the recommended setup when jupyterhub is complete. One recent change is the ability to save/load state from a database. The wiki page above hasn't been updated with the note from this issue: https://github.com/jupyter/jupyterhub/issues/57 It might be easier to start without NGINX, and then add it after you have a working jupyterhub system. It would be nice to have a little bash script to make this a "service" that would support "start", "top", and "status"... but I haven't had time. Currently, we're just becoming the non-root sudoer ("rhea" in the docs) and starting the server, something along the lines: jupyterhub --LocalProcessSpawner.set_user=sudo --JupyterHubApp.ip=165.106.10.83 --JupyterHubApp.port=80 --db='sqlite:///:memory:' &>> /var/log/jupyterhub/log & The next step for us is to get it running under https... looks like others have blazed that trail, so it looks possible. If you have success, it would be great to add to the wiki docs... I suspect that many of us that aren't sys admins will be wanting to get this up and running. -Doug [1] - https://github.com/jupyter/jupyterhub > Any help much appreciated! > > Clare > > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lucienboillod at gmail.com Mon Oct 13 03:53:53 2014 From: lucienboillod at gmail.com (Lucien Boillod) Date: Mon, 13 Oct 2014 09:53:53 +0200 Subject: [IPython-dev] Wrap Javascript (D3.js) file into IPython Widget In-Reply-To: References: Message-ID: Really thank you for that ! I?ll explore all the way you gave me, and give you a feedback of my work ! Cheers, Le 11 oct. 2014 ? 20:23, Nicholas Bollweg a ?crit : > You can stick the whole thing in an iframe, and it will work pretty much as it is right now... the crucial bit of magic will be keeping that link between those two documents. Here is a gist with an untested 20% solution: > > https://gist.github.com/bollwyvl/f0ba3e16a5a824d53dc8 > > Now the widget and the d3 stuff will exist in two separate documents, with two different javascript contexts, but from the view you can touch the frame with view.frame... the other direction is trickier, and if you end up needing to do that, just rewrite the whole thing to be outside of a frame :) > You may want to explore some patterns, like http://bost.ocks.org/mike/chart/, to encapsulate the relationship between data/drawing inside the iframe: right now, if a widget were to update > You'll probably want to explore d3.dispatch, so that you can create custom events (like nodeAdd, nodeRemove, etc) that the widget can react to. This may be a cleaner way than trying to "share" data > be careful of force layout changing your data (nodes, links) in-place, adding x, y, px, py, etc. > when a user changes nodes/links form outside the visualization, you'll probably need to do some things to force... but you'll want to run them at the same time. > look at Backbone.Model.changed and Widget.hold_sync to make sure that this happens atomically... or get ready for a lot of svg errors. > All that being said: a well-structured backbone view, which every widget must be, will be more maintainable in the long run. Just having evented get/set on the model, and maybe some custom collections, might be worth the price of admission. > > Good luck! > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From John.Holt at tessella.com Mon Oct 13 07:22:33 2014 From: John.Holt at tessella.com (John.Holt at tessella.com) Date: Mon, 13 Oct 2014 12:22:33 +0100 Subject: [IPython-dev] Jupyter Server Project Startup Message-ID: Hi All, I am just starting a project to deploy the IPython Notebook deployed for multiple user access on a virtual cloud. I am writing in the hope that the community can advise me on any issues I am going to face getting this up and running. I am particularly interested in the following: 1) It looks like Jupyter Hub is a good choice for this system. I am aware that is is still in development but have seen some people using it. How stable is it? 2) Has anyone used IPython parallel within JupyterHub? This is a key requirement of my project. I created a sandbox which runs this and although I can use a pre-setup cluster from my notebook, I can not start a new cluster within the notebook. 3) Any other thoughts, tips, tricks or pitfalls I should avoid? Thank you for your time. Cheers, John This message is commercial in confidence and may be privileged. It is intended for the addressee(s) only. Access to this message by anyone else is unauthorized and strictly prohibited. If you have received this message in error, please inform the sender immediately. Please note that messages sent or received by the Tessella e-mail system may be monitored and stored in an information retrieval system. From doug.blank at gmail.com Mon Oct 13 07:51:33 2014 From: doug.blank at gmail.com (Doug Blank) Date: Mon, 13 Oct 2014 07:51:33 -0400 Subject: [IPython-dev] Jupyter Server Project Startup In-Reply-To: References: Message-ID: On Mon, Oct 13, 2014 at 7:22 AM, wrote: > Hi All, > > I am just starting a project to deploy the IPython Notebook deployed for > multiple user access on a virtual cloud. I am writing in the hope that the > community can advise me on any issues I am going to face getting this up > and running. I am particularly interested in the following: > > 1) It looks like Jupyter Hub is a good choice for this system. I am > aware that is is still in development but have seen some people using it. > How stable is it? > It is fairly stable. I have a class of students using it daily. I don't update the codebase (ipython and jupyterhub) but once every few weeks: * things can still break in development * there have been some needed bug fixes along the way * the ipython group has been very responsive to keeping master working * it may require you to track down issues and report them (time consuming) * if it is a public-facing server, then it requires sys-admin-level support (account management, security updates, etc) > 2) Has anyone used IPython parallel within JupyterHub? This is a key > requirement of my project. I created a sandbox which runs this and although > I can use a pre-setup cluster from my notebook, I can not start a new > cluster within the notebook. > Yes, I am not using that for the class, but we have been developing 3rd-party kernels that utilize the cluster [1]. Everything works as advertised, although we are just running the cluster as regular processes in the system---nothing fancy (no MPI, remote nodes, etc). > 3) Any other thoughts, tips, tricks or pitfalls I should avoid? > I recommend starting simply, and adding additional variations one at a time. As this is new code, there are many combinations that have not been tested, and thus you may find bugs. Hope that helps! -Doug [1] - https://github.com/blink1073/metakernel > > Thank you for your time. > > Cheers, > > John > This message is commercial in confidence and may be privileged. It is > intended for the > addressee(s) only. Access to this message by anyone else is unauthorized > and strictly prohibited. > If you have received this message in error, please inform the sender > immediately. Please note that > messages sent or received by the Tessella e-mail system may be monitored > and stored in an > information retrieval system. > > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lucienboillod at gmail.com Mon Oct 13 11:02:53 2014 From: lucienboillod at gmail.com (Lucien Boillod) Date: Mon, 13 Oct 2014 17:02:53 +0200 Subject: [IPython-dev] Wrap Javascript (D3.js) file into IPython Widget In-Reply-To: References: Message-ID: <115485A7-8090-4669-806A-6B5D3386DD32@gmail.com> When we load the index.html from the javascript file we use: this.$frame = $("