From Fernando.Perez at colorado.edu Fri Jul 1 02:41:46 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Fri, 01 Jul 2005 00:41:46 -0600 Subject: [IPython-dev] Re: questions about the notebook interface In-Reply-To: References: <42C00157.2020808@an.org> Message-ID: <42C4E5AA.90504@colorado.edu> Robert Kern wrote: > Toni Alatalo wrote: > >>Greetings. >> >>i have some questions about the notebook interface (as it's described in >>http://ipython.scipy.org/google_soc/ipnb_google_soc.pdf). must leave >>right now so am trying to be brief, sorry for possible unclarities: > > > I've been thinking a bit about this too, since I was going to volunteer > some work on this if no one got picked for it by Google. Looks like that > won't be necessary. Congratulations! > > >>1. reasoning for true Python over special string / comments >> >>as e.g. rst is nice for writing documents in itself, and can also be >>used to write text, make tables, images etc. in Python files (the >>original purpose), i'd be interested in hearing the main arguments from >>the 'lengthy discussions' (referred in the spec) for deciding on true >>Python instead. am probably bying the result already, but would still >>like to hear the reasoning. > > > I personally don't agree with it. I wasn't a party to those discussions, > so I'm sure most of my complaints have already been addressed, but I'm > going to air them out here anyways. Because I can. :-) And because you should :) Seriously, while I've spent quite a bit of time thinking about this, and John Hunter and I discussed it at length, it's a fairly tricky problem full of potential pitfalls. I _really_ appreciate the feedback, and at this point I consider the sketch I wrote up as only a bare draft, fully open to a complete overhaul. I want to find the right solution to this problem, wherever the idea comes from. So keep the debate coming, by all means. > The foremost use cases in my head are the essentially interactive or > demonstrative ones, not the literate programming ones. I think that > people are going to use notebooks to run interactively or generate > static, pretty documents. They are going to use the exported code to > import or run on a server or whatever other "headless" activity. If you > really want to be able to import from a notebook, you could write an > import hook that will export the bare code and import that. (Disclosure: > I never met a literate programming system that I liked.) Well, neither have I, but Mathematica does have some strong points. One of its major weaknesses, however, is precisely the impossibility of doing major coding from within the notebook interface (the editor sucks rocks). > The markup-via-Python seems to me to be difficult to parse without > running the code. Every time I make a markup change or spelling > correction, my excessively long calculation will run again just to > recreate a plot that I've already embedded in the notebook. I should be > able to selectively run the code when I need to (re)generate output. What I'd thought was that in rendering mode, no code marked as input would ever be run at all. > Name collisions. Or else evil hacks to avoid name collisions. why? We can simply request that the top-level markup import is unmodified: from notebook import markup and the file can then be parsed. Any markup.foo() calls are considered markup, and the rest is code to be executed. As for the problem of people wanting to program their markup, I don't necessarily see that as a bad thing. The code marked as input doesn't need to be re-executed on all rendering calls (thus negating the problem of rendering being potentially an expensive process), but other non-input code can be executed freely. > You can't embed these calls in functions because they won't get called > until the function is run, not when that line of code is executed. And > then they get called every time that function gets called. > > I don't think that any-old-text-editor is a viable target. It just > doesn't provide any of the real notebook capabilities. Targetting it as > an intermediate platform before a GUI shell is available sacrifices the > capabilities that the GUI shell is going to provide. Well, that approach has worked pretty well for Latex. You can edit it with xemacs/vi/notepad/whatever, and yet the LyX guys managed to build a fancy GUI environment on top of it. Furthermore, lyx actually handles the document in its own .lyx format, and exports to latex for the final rendering pass. But if you want, you can embed raw latex into the document via ERT insets. I don't see why a similar model can't work for us: a simple enough format that it's valid python syntax, so it can be edited with normal programming tools. A notebook library to render the files, and a GUI on top which enables interactive use of such documents, using the ipython engine. > Allow me to make a suggestion for an intermediate target: Modify ipython > to make a notebook file (in whatever format you go forward with) as a > kind of log. At minimum, it just writes out the In[NN]'s and Out[NN]'s > in the notebook format. Add a %magic to write out notes to the notebook. > Nothing fancy or formatted, they'd mostly be just reminders that you > would expand and format later, not the final markup. If you're on a > roll, add another %magic for pylab mode that will save the current > figure as a .png and write out an entry to the notebook for including an > image. For a prototype, a curses-based one could be written fairly easily. It wouldn't be portable in the long term, but it might be enough for a lot of users on *nix systems. >>2. usage targets of the library >> >>ReST and LaTEX are pretty clear, but how about possible other types of >>documents, like ones with dynamic / interactive content -- should that >>be addressed by the notebook lib? e.g. Slithy >>(http://isotropic.org/uw/slithy/) provides a pretty nice way to make >>dynamic/interactive diagrams with python (as functions that receive vars >>(user controllable with slides in the player, and/or time for >>animations) and output drawings). what (other) usages should be >>considered? or only 'print targeted' stuff (like the mathreader things >>seem almost like pdf)? > > > For this summer, just assume that anything that wants to do something > funky will create a context in which to do it. Leave how that happens up > to the people writing the GUI. Allow things to fail gracefully when it > doesn't make sense (e.g. Pygame animations in HTML). Yup, fully agreed. >>one other area here is linking, which >>rest/latex/pdf and of course html do support too, and there perhaps >>granularity .. like to have support for so-called 'purple numbers' >>http://www.eekim.com/software/purple/purple.html . > > > Intra-notebook linking gets a big +1 from me. As an offhand thought, > every time In[NN] or Out[NN] is used in an expression, it should get a > link back to the original when the expression is being syntax colored. Yes, a labeling system for cross-references should be part of the document model. In latex, these naturally expand to \label and \ref calls, with similar stuff for reST or HTML. >>4. ipython and non-ipython >> >>i suppose the basic notebook module would be independent from ipython, >>and then some separate glue for it would be made for ipython (in the >>ipython cvs), like has been done with gnuplot - right? or is there >>something ipython specific about the project i'm missing? indeed the >>real-time / interactive aspect is interesting, but does that imply much >>for the API design? and in this case, should be a separate project, >>hosted on some of the sourceforge forks (berlios, gna.org or something?) >>.. would 'notebook' be a good name for the package, and pynotebook for >>the project, or what? or is it more tied to ipython and should start there? Yes, the document-focused project (Toni's) should be mostly ipython-independent code-wise, though we'll carry the development using ipython's resources (website, SVN, lists, etc.). At some point, we'll begin adding support in ipython for it, so Toni may need to put code into ipython as well. In the meantime, Tzanko's project will probably need to cut into ipython much earlier, as there are internal changes needed to separate the engine enough for clean gui embedding. > > Development should probably be done on the scipy.org server. Hopefully > the SVN server will be up soon for ipython. Most of the development will > probably happen outside of the ipython tree itself, I imagine. I think the transition to the new scipy server is mostly done, so we should have an SVN repo available very soon for all this. Cheers, f From Fernando.Perez at colorado.edu Fri Jul 1 02:46:00 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Fri, 01 Jul 2005 00:46:00 -0600 Subject: [FWD] Re: [IPython-dev] questions about the notebook interface In-Reply-To: References: Message-ID: <42C4E6A8.8050605@colorado.edu> [I'm forwarding this manually to the list, please subscribe for future postings as the ipython lists unfortunately are off limits for non-subscribers, due to excessive spam] > Subject: Re: [IPython-dev] questions about the notebook interface > From: Hans Meine > Date: Thu, 30 Jun 2005 11:49:00 +0200 > To: ipython-dev at scipy.net > > Hi! > > On Tuesday 28 June 2005 17:36, Toni Alatalo wrote: > >>>I've thought that XEmacs could be the first environment to target for >>>interactive use, since it already almost has everything we need in its >>>python-mode/ipython combination. In the meantime, Tsanko can work on >>>the more ambitious GUI-oriented project he applied for. >> >>fine with me, as emacs is what i always use for programming otherwise too. >> >>.. looking forward to hearing the GUI ideas too > > > I just want to stress the part of "the notebook interface" that is of greatest > importance to me, since most of these discussion did not seem to be concerned > with it: The interactive programming aspect. > > Actually, I don't care much about the final layouting; the biggest feature of > a "notebook interface" in my eyes would be that I can incrementally write and > debug a program, correcting my mistakes and ending up with a "clean history". > > Right now, ipython is the best environment for my work, but I have to manually > extract "good" code fragments. The most important thing in my eyes would be > a possibility to edit and re-run my code without *appending* to the history, > but just correcting and extending the existing "In [xx]: " paragraphs. That > would be especially handy in combination with loops, too. > > Maybe I should propose developing / collecting proper user stories first, then > you could decide on their priorities and start creating solutions for > existing problems (there may well be people focused on layout, which could > give another user story). > > Some short user stories(*): > ------------------------------ >8 snip 8< ---------------------------------- > I am planning to analyse a multidimensional data structure. I am writing a > loop, collecting data but then realize that I need to initialize a variable > *before* the loop. In ipython, I can only cancel the current edit, type the > initialization and fetch all lines from the history (or retype them) > one-by-one. What I'd like to be able to do is to just press cursor up, > insert a line, and continue further down, then execute the whole loop. > > I successfully analyzed some data derived from images. Now I want to change > some preprocessing parameters and re-run the analysis. Again, I want to > insert / change some code at the beginning and re-execute several blocks of > commands that use the now changed variables. > > It would be useful if some sort of dependency graph could be generated. E.g. > when I change a "cell" (piece of code) which (re-)assigns variables, all code > fragments (cells) that depend on these results (use that variable) should be > marked as "dirty". Automatic execution of these cells should not be the > goal, since that could be unwanted by the user, need a long time, and/or > destroy/overwrite precious results. > > For image processing purposes, I want to be able to have image objects that > result from python expressions shown as embedded images in the notebook. > > Similarly, I'd like to have graphs (e.g. from Gnuplot.py) be embedded in the > notebook. > > A friend of mine wants to be able to write longer Mathematic-like notebooks, > and needs proper namespacing for chapters/sections, so that variable names > can be re-used. > ------------------------------ >8 snip 8< ---------------------------------- > > Actually, the xemacs approach seems to be promising to me, I am looking > forward to further information / results / input. From Fernando.Perez at colorado.edu Fri Jul 1 03:40:40 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Fri, 01 Jul 2005 01:40:40 -0600 Subject: [FWD] Re: [IPython-dev] questions about the notebook interface In-Reply-To: <42C4E6A8.8050605@colorado.edu> References: <42C4E6A8.8050605@colorado.edu> Message-ID: <42C4F378.3000005@colorado.edu> >>From: Hans Meine >>I just want to stress the part of "the notebook interface" that is of greatest >>importance to me, since most of these discussion did not seem to be concerned >>with it: The interactive programming aspect. >> >>Actually, I don't care much about the final layouting; the biggest feature of >>a "notebook interface" in my eyes would be that I can incrementally write and >>debug a program, correcting my mistakes and ending up with a "clean history". >> >>Right now, ipython is the best environment for my work, but I have to manually >>extract "good" code fragments. The most important thing in my eyes would be >>a possibility to edit and re-run my code without *appending* to the history, >>but just correcting and extending the existing "In [xx]: " paragraphs. That >>would be especially handy in combination with loops, too. Indeed, the interactive aspect is pretty dear to my heart (I wouldn't have spent all this time developing ipython if I dindn't care about that :). And I think that Tzanko has a good shot at making headway pretty quickly on this front with ipython+pycrust. A while ago I tried to embed ipython in IDLE and pycrust; while it didn't quite work, it seemed to me like the remaining problems were easy to solve. It needs a bit of time, since the right solution (a quick hack could be done in a few hours) should involve some code reorganization. >> >>Maybe I should propose developing / collecting proper user stories first, then >>you could decide on their priorities and start creating solutions for >>existing problems (there may well be people focused on layout, which could >>give another user story). >> >>Some short user stories(*): >>------------------------------ >8 snip 8< ---------------------------------- >>I am planning to analyse a multidimensional data structure. I am writing a >>loop, collecting data but then realize that I need to initialize a variable >>*before* the loop. In ipython, I can only cancel the current edit, type the >>initialization and fetch all lines from the history (or retype them) >>one-by-one. What I'd like to be able to do is to just press cursor up, >>insert a line, and continue further down, then execute the whole loop. Yup, this is really annoying. For much of this, Xemacs+ipython.el can be a better solution, as you can keep a scratch.py buffer where you do multi-line edits, and run them in the ipython buffer with C-c C-|. >>I successfully analyzed some data derived from images. Now I want to change >>some preprocessing parameters and re-run the analysis. Again, I want to >>insert / change some code at the beginning and re-execute several blocks of >>commands that use the now changed variables. This one is already doable in ipython: just call macro mm 12:30 34 39:50 and then typing mm is like re-executing all those lines. All code is executed clean, so variable changes are recognized: In [1]: print 'x is:',x --------------------------------------------------------------------------- exceptions.NameError Traceback (most recent call last) /usr/local/home/fperez/ NameError: name 'x' is not defined x is: In [2]: x=99 In [3]: macro mm 1 Macro `mm` created. To execute, type its name (without quotes). Macro contents: print 'x is:',x In [4]: mm Out[4]: Executing Macro... x is: 99 In [5]: x='hi' In [6]: mm Out[6]: Executing Macro... x is: hi See macro? for further details. >>It would be useful if some sort of dependency graph could be generated. E.g. >>when I change a "cell" (piece of code) which (re-)assigns variables, all code >>fragments (cells) that depend on these results (use that variable) should be >>marked as "dirty". Automatic execution of these cells should not be the >>goal, since that could be unwanted by the user, need a long time, and/or >>destroy/overwrite precious results. Very ambitious, but an idea worth keeping in mind for the future. >>For image processing purposes, I want to be able to have image objects that >>result from python expressions shown as embedded images in the notebook. >> >>Similarly, I'd like to have graphs (e.g. from Gnuplot.py) be embedded in the >>notebook. My current favorite idea is for the notebook format to have some kind of include directive like: markup.graphic('foo.png',options...) Once a fancy GUI is up and running which can display text and inline graphics, they could be shown on-screen. But at least rendering them to PS/PDF should be easy. Note that the GUI for this is not trivial: I think that only Qt4 (released yesterday) has a rich text widget with inline graphics support. I may be wrong, but I think that none of the text widgets in GTK, Tk or WX have this capability. That's why I'm concentrating on having proper backend support for now, along with a basic GUI which can at least display the text parts and provide the interactive support. This would give us the possibility to: 1. Edit the notebooks 'by hand' with plain editors, even if interactive support isn't there. 2. Have a gui with interactive support that can provide the ability to re-edit the same input group multiple times without re-generating new input cells. If both systems can operate on the same files, one could switch back and forth between the editor and the interactive environment, depending on the task at hand (heavy-duty coding or multiple tests). This might even allow a system like Mozilla's composer (now Nvu), which has both a WYSIWIG tab and a raw html tab for the same document, and one can switch back and forth between them. >>A friend of mine wants to be able to write longer Mathematic-like notebooks, >>and needs proper namespacing for chapters/sections, so that variable names >>can be re-used. This can be done by issuing a 'del' statement at the start, much like in Mathematica you need to call Clear[] on symbols. >>------------------------------ >8 snip 8< ---------------------------------- >> >>Actually, the xemacs approach seems to be promising to me, I am looking >>forward to further information / results / input. Thanks for your input! Best, f From tsanko at gmail.com Fri Jul 1 12:42:03 2005 From: tsanko at gmail.com (Tzanko Matev) Date: Fri, 01 Jul 2005 19:42:03 +0300 Subject: [IPython-dev] Notebook format idea Message-ID: <1120236123.5582.33.camel@tzanko> Hi Toni, I have a question about the notebook format. How do you plan to implement python code? I guess that something like code = Code(""" #some python code """) is not a good idea, because no text editor with python support would recognize the string as python code. Probably something like Code.start() #some python code code = Code.end() would work. When the file is converted to a printable format you first parse it using the parse module and convert all the code segments of the above form to something like the first example, and then you run the result to produce the file you need. Right now I'm using my own very simple notebook format. For now I will keep on using it, because the internal representation of data does not depend much on the file format. Best regards, Tzanko P.S. Sorry for the private email From tsanko at gmail.com Fri Jul 1 14:13:53 2005 From: tsanko at gmail.com (Tzanko Matev) Date: Fri, 01 Jul 2005 21:13:53 +0300 Subject: [IPython-dev] Proposed IPython changes Message-ID: <1120241633.5582.73.camel@tzanko> Hi Fernando, I'm going to start implementing the python code plug-in in the notebook interface. I plan on using the Shell class from PyCrust. For that I need an interpreter class which mimics the behaviour of PyCrust's own. I will try to make the needed modifications to InteractiveShell. At first, however, I plan to convert make_IPython into a number of smaller functions. Then I will start making methods for configuring an InteractiveShell instance, that these functions will use instead of directly messing with the internals. And after that I will implement what is needed for the Shell class of PyCrust. >From new_design.pdf I understand that part of the cleanup is a new class structure which has a better API. My idea is this: make a class Kernel, which will contain all the IPython functionality. Then the class InteractiveShell will simply be a terminal interface to Kernel, and the notebook program will be a graphical interface. Kernel should contain methods for running code, loading configuration, loading logs, etc. I'm not that familiar with the current code base so I don't know how long this rewrite would take. For now I will only implement parts that are needed for the GUI. Please tell me if you like this idea, and what do you think the new class design should be. I want to make the GUI in such a way that the transition to the new design be as easy as possible. Best regards, Tzanko From Fernando.Perez at colorado.edu Fri Jul 1 14:27:27 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Fri, 01 Jul 2005 12:27:27 -0600 Subject: [IPython-dev] Proposed IPython changes In-Reply-To: <1120241633.5582.73.camel@tzanko> References: <1120241633.5582.73.camel@tzanko> Message-ID: <42C58B0F.4040603@colorado.edu> Hi Tzanko, Tzanko Matev wrote: > Hi Fernando, > > I'm going to start implementing the python code plug-in in the notebook > interface. I plan on using the Shell class from PyCrust. For that I need > an interpreter class which mimics the behaviour of PyCrust's own. I will > try to make the needed modifications to InteractiveShell. At first, > however, I plan to convert make_IPython into a number of smaller > functions. Then I will start making methods for configuring an > InteractiveShell instance, that these functions will use instead of > directly messing with the internals. And after that I will implement > what is needed for the Shell class of PyCrust. > >>From new_design.pdf I understand that part of the cleanup is a new class > structure which has a better API. My idea is this: make a class Kernel, > which will contain all the IPython functionality. Then the class > InteractiveShell will simply be a terminal interface to Kernel, and the > notebook program will be a graphical interface. Kernel should contain > methods for running code, loading configuration, loading logs, etc. I'm > not that familiar with the current code base so I don't know how long > this rewrite would take. For now I will only implement parts that are > needed for the GUI. Please tell me if you like this idea, and what do > you think the new class design should be. I want to make the GUI in such > a way that the transition to the new design be as easy as possible. Yes, that's roughly what I have in mind for the long-term structure. Having a kernel which only takes commands over a specified channel and returns output will also allow in the future the possibility to restart the kernel while keeping the user interface active, running a kernel on a remote machine, controlling multiple kernels from a single front-end, etc. Please note that the make_ipython routine is just one monster hack for historical reasons. There is absolutely no clean design reason to have it, it's really nothing more than an initializer for the ipython instance, because when I started writing ipython, I had my own $PYTHONSTARTUP code which I merged with a class-based system (IPP). The quickest way to merge the two codes was to keep the IPP __init__, and then run my $PYTHONSTARTUP script after that, having renamed it to make_ipython. But in reality, the class constructor should be just one. The publicly visible classes live in Shell.py, and currently (because of this make_ipython mess) they hold an internal reference to the ipython instance returned by make_ipython. Ultimately all that should go away. There are two other things which should be cleaned up in this process, so they don't get in the way. These are also historical artifacts from the original IPP code, they are just gross misfeatures in the current code: 1. (easy) - the logging system is a class mixin. It shoulde be a standalone system, and iypthon should simply instantiate it once and hold a reference to this instance at runtime. Making it a mixin just creates a namespace mess. 2. (not so easy) - the magics are also a mixin. The namespace mess is worse here, though I've tried to prepare things for separation since long ago. Now all magics refer back to the shell via a self.shell attribute, rather than relying on self directly. This means that once separated, as long as the standalone magic system is given a valid self.shell pointer to the ipython shell, they should continue to run. Let me know how this all sounds to you. I know it will take a bit of time to familiarize yourself with the tangled hairball that is the ipython engine, but nothing here is particularly hard. Regards, f From rkern at ucsd.edu Fri Jul 1 17:48:45 2005 From: rkern at ucsd.edu (Robert Kern) Date: Fri, 01 Jul 2005 14:48:45 -0700 Subject: [IPython-dev] Re: questions about the notebook interface In-Reply-To: <42C4E5AA.90504@colorado.edu> References: <42C00157.2020808@an.org> <42C4E5AA.90504@colorado.edu> Message-ID: Fernando Perez wrote: > Robert Kern wrote: >> The foremost use cases in my head are the essentially interactive or >> demonstrative ones, not the literate programming ones. I think that >> people are going to use notebooks to run interactively or generate >> static, pretty documents. They are going to use the exported code to >> import or run on a server or whatever other "headless" activity. If >> you really want to be able to import from a notebook, you could write >> an import hook that will export the bare code and import that. >> (Disclosure: I never met a literate programming system that I liked.) > > Well, neither have I, but Mathematica does have some strong points. One of > its major weaknesses, however, is precisely the impossibility of doing > major > coding from within the notebook interface (the editor sucks rocks). This is why I think we need to separate these two cases: 1. Writing modules 2. Interacting with interpreter in a REPL (Read-Eval-Print-Loop, i.e. ipython) The interfaces and formats that are good for one aren't necessarily good for the other. Problem 1 is a task for a literate programming tool. Now, we could certainly write one, and it would be useful, but there are existing, capable LP tools out there. If you want to write another one, we should address why the existing ones are inadequate and how we are going to redo all of the things that they already do well. But a graphical REPL is something different. A literate program is essentially a static document. You're not running it *while* you're editing it. That has consequences which I'm going to get to when I add to Hans's user stories. Preferably in Wiki form if that managed to make it through the server migration. As I said before, I never met a LP system that I liked. Comments and docstrings have always been enough for me. I've also never read a LP "book" that was worth the time. None of them made the code more understandable or the documentation better written. In short, I suggest leaving module-type code alone and focusing on a notebook REPL. >> The markup-via-Python seems to me to be difficult to parse without >> running the code. Every time I make a markup change or spelling >> correction, my excessively long calculation will run again just to >> recreate a plot that I've already embedded in the notebook. I should >> be able to selectively run the code when I need to (re)generate output. > > What I'd thought was that in rendering mode, no code marked as input would > ever be run at all. Yes, you are correct. I had an incorrect picture in my head of how it would work. >> Name collisions. Or else evil hacks to avoid name collisions. > > why? We can simply request that the top-level markup import is unmodified: > > from notebook import markup > > and the file can then be parsed. Any markup.foo() calls are considered > markup, and the rest is code to be executed. That's an arbitrary limitation on the code that can be contained in my program. It's also a restriction that is only visible to the person that is looking at the plain text. In the graphical REPL, the restriction remains, but there's nothing but the ipython's documentation telling you that you should never start a line with "markup." Now, it's a relatively minor restriction and one that I could probably live with, but it increments my ick counter. > As for the problem of > people wanting to program their markup, I don't necessarily see that as > a bad thing. The code marked as input doesn't need to be re-executed on > all rendering calls (thus negating the problem of rendering being > potentially an expensive process), but other non-input code can be > executed freely. That assumes that they're not trying to mix program and markup. def f(x, y): return x + y markup.text("4 + 5 = %s" % f(4, 5)) or markup.text( "| I am | an empty | table |") markup.text( "+------+----------+-------+") for x in range(10): markup.text("| | | |") You just *know* that someone's going to try that. Since the primary motivation for Python-syntax markup is that users already know the syntax, it is confusing to utterly change the semantics. Furthermore, I think that it's going to be difficult or impossible to detect prior to running the separated codes. And then, they'll probably fail with an inscrutable error message. The XML approach treats everything as data. Text is data. Markup is data (okay, it's actually structure, but you get the idea). Code is different data. ipython magic commands are another kind of data. They're kept separate, and the assumptions about how you treat them and the contexts in which they are so treated are kept separate. The presumption for, "Python syntax wherever possible," is understandable, but it's not a good principle in general, I don't think. Where it works, it is usually because it is the application of another principle, "Do the simplest thing that could possibly work." Often Python syntax is the simplest thing that could possibly work because you can just exec it and you have your data. That's why it works for certain config files or Skencil graphics or saved Chaco plots. Unfortunately, that's not the case here. To generate the document or even just the data structure in memory, you have to parse the file and extract the data from an AST which was not designed for this purpose. That is going to take a lot of effort that could profitably be used elsewhere. We're not short of work that needs to get done. It used to be the case that XML was usually far from the simplest thing that would work, too. SAX and DOM and other Java-isms suck in the Python world. You would only touch XML if you had to interoperate with something else that needed XML or you wanted to leverage the various XML standards. Recently, however, the XML tools available to Python have become much more Pythonic. ElementTree (and to a lesser extent, Amara, gnosis.xml, and libxml2) changed the landscape. ElementTree provides not only parsing and output but also a convenient data structure for manipulating the document. That eliminates quote a lot of work that could be spent on more interesting things. >> You can't embed these calls in functions because they won't get called >> until the function is run, not when that line of code is executed. And >> then they get called every time that function gets called. I was wrong on this one, too. def f(x, y): markup.text('Add two numbers together like so:") return x + y could work fine with enough AST-fiddling. >> I don't think that any-old-text-editor is a viable target. It just >> doesn't provide any of the real notebook capabilities. Targetting it >> as an intermediate platform before a GUI shell is available sacrifices >> the capabilities that the GUI shell is going to provide. > > Well, that approach has worked pretty well for Latex. You can edit it > with xemacs/vi/notepad/whatever, and yet the LyX guys managed to build a > fancy GUI environment on top of it. Furthermore, lyx actually handles > the document in its own .lyx format, and exports to latex for the final > rendering pass. But if you want, you can embed raw latex into the > document via ERT insets. > > I don't see why a similar model can't work for us: a simple enough > format that it's valid python syntax, so it can be edited with normal > programming tools. A notebook library to render the files, and a GUI on > top which enables interactive use of such documents, using the ipython > engine. LaTeX does not have a REPL. It's the static document versus, well, REPL distinction again. Python-as-markup and this editing model might work for LP, but it greatly restricts the capabilities of a REPL. >> Allow me to make a suggestion for an intermediate target: Modify >> ipython to make a notebook file (in whatever format you go forward >> with) as a kind of log. At minimum, it just writes out the In[NN]'s >> and Out[NN]'s in the notebook format. Add a %magic to write out notes >> to the notebook. Nothing fancy or formatted, they'd mostly be just >> reminders that you would expand and format later, not the final >> markup. If you're on a roll, add another %magic for pylab mode that >> will save the current figure as a .png and write out an entry to the >> notebook for including an image. > > For a prototype, a curses-based one could be written fairly easily. It > wouldn't be portable in the long term, but it might be enough for a lot > of users on *nix systems. Well, I've got the XML system already working to this state through evil hooks. I'll look into dropping into an SVN repository somewhere accessible. BTW, whenever you'd complain about how ugly ipython's code was and how it was a big mass of tangly code, I'd always think to myself, "Nah, it can't be *that* bad!" It's that bad. ;-) Anyways, I think the next easiest target should probably go straight to wxScintilla or Py using just plain text (or reST) for markup. There's a reason curses is called curses. Or there's always wxMozilla .... -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From hans_meine at gmx.net Sun Jul 3 12:35:19 2005 From: hans_meine at gmx.net (Hans Meine) Date: Sun, 3 Jul 2005 18:35:19 +0200 Subject: [IPython-dev] questions about the notebook interface In-Reply-To: <42C16E88.901@an.org> References: <42C00157.2020808@an.org> <42C055DF.9040505@colorado.edu> <42C16E88.901@an.org> Message-ID: <200507031835.19199.hans_meine@gmx.net> Hi! On Tuesday 28 June 2005 17:36, Toni Alatalo wrote: > > I've thought that XEmacs could be the first environment to target for > > interactive use, since it already almost has everything we need in its > > python-mode/ipython combination. In the meantime, Tsanko can work on > > the more ambitious GUI-oriented project he applied for. > > fine with me, as emacs is what i always use for programming otherwise too. > > .. looking forward to hearing the GUI ideas too I just want to stress the part of "the notebook interface" that is of greatest importance to me, since most of these discussion did not seem to be concerned with it: The interactive programming aspect. Actually, I don't care much about the final layouting; the biggest feature of a "notebook interface" in my eyes would be that I can incrementally write and debug a program, correcting my mistakes and ending up with a "clean history". Right now, ipython is the best environment for my work, but I have to manually extract "good" code fragments. The most important thing in my eyes would be a possibility to edit and re-run my code without *appending* to the history, but just correcting and extending the existing "In [xx]: " paragraphs. That would be especially handy in combination with loops, too. Maybe I should propose developing / collecting proper user stories first, then you could decide on their priorities and start creating solutions for existing problems (there may well be people focused on layout, which could give another user story). Some short user stories(*): ------------------------------ >8 snip 8< ---------------------------------- I am planning to analyse a multidimensional data structure. I am writing a loop, collecting data but then realize that I need to initialize a variable *before* the loop. In ipython, I can only cancel the current edit, type the initialization and fetch all lines from the history (or retype them) one-by-one. What I'd like to be able to do is to just press cursor up, insert a line, and continue further down, then execute the whole loop. I successfully analyzed some data derived from images. Now I want to change some preprocessing parameters and re-run the analysis. Again, I want to insert / change some code at the beginning and re-execute several blocks of commands that use the now changed variables. It would be useful if some sort of dependency graph could be generated. E.g. when I change a "cell" (piece of code) which (re-)assigns variables, all code fragments (cells) that depend on these results (use that variable) should be marked as "dirty". Automatic execution of these cells should not be the goal, since that could be unwanted by the user, need a long time, and/or destroy/overwrite precious results. For image processing purposes, I want to be able to have image objects that result from python expressions shown as embedded images in the notebook. Similarly, I'd like to have graphs (e.g. from Gnuplot.py) be embedded in the notebook. A friend of mine wants to be able to write longer Mathematic-like notebooks, and needs proper namespacing for chapters/sections, so that variable names can be re-used. ------------------------------ >8 snip 8< ---------------------------------- Actually, the xemacs approach seems to be promising to me, I am looking forward to further information / results / input. -- Ciao, / / /--/ / / ANS *: DISCLAIMER: I am no professional software engineer, so my notion of "user stories" could be a little bit simplified, but hopefully still useful. From rkern at ucsd.edu Sun Jul 3 23:59:29 2005 From: rkern at ucsd.edu (Robert Kern) Date: Sun, 03 Jul 2005 20:59:29 -0700 Subject: [IPython-dev] Notebook roadmap Message-ID: Fernando and I hashed out a few things on the phone today. Short version: XML wins for now, even if I had to threaten to kidnap Fernando's cat to do it. Long version: http://www.scipy.org/wikis/featurerequests/NoteBook There's a page there for adding user stories. I've already added the ones Hans has posted and some of my own. If you would like to add your own stories (aka "feature requests"), please make yourself an account and edit the Wiki to your liking. Documentation on the XML format and API will be forthcoming. Hopefully, we will have SVN space by Tuesday or Wednesday. P.S. No I didn't threaten to kidnap his cat. Fluffy is perfectly safe. For now.... -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From antont at an.org Mon Jul 4 07:18:52 2005 From: antont at an.org (Toni Alatalo) Date: Mon, 04 Jul 2005 14:18:52 +0300 Subject: [IPython-dev] Notebook roadmap In-Reply-To: References: Message-ID: <42C91B1C.503@an.org> Robert Kern wrote: > Fernando and I hashed out a few things on the phone today. Great! BTW: Fernando mentioned the meeting possibilities in one early post -- perhaps VOIP using Skype or something would be good at some point, but FYI also that I'm usually hanging on Freenode IRC as antont and we could set a channel there too (#ipython #notebook #scipy or something..) > Short version: XML wins for now, even if I had to threaten to kidnap > Fernando's cat to do it. Long version: > http://www.scipy.org/wikis/featurerequests/NoteBook ok. as the last thing last week i also took a brief look at ElementTree, and actually had decided now to continue the work by looking into it more if there would be no info guiding otherwise, so good to get this confirmation. > There's a page there for adding user stories. I've already added the > ones Hans has posted and some of my own. If you would like to add your > own stories (aka "feature requests"), please make yourself an account > and edit the Wiki to your liking. thanks - actually the main thing i had in mind for the start of this project was to ask for use cases / user stories, 'cause the requirements haven't been too clear at all. but it was good also to do little warm-up coding too to get an idea. was also fun to see that the Scipy tutorial was suggested as the target, 'cause was reading just that last week to further study an example notebook :) .. IIRC the current one is written in LaTEX, i suppose the source is somewhere too. > Documentation on the XML format and API will be forthcoming. > Hopefully, we will have SVN space by Tuesday or Wednesday. ok. i'll study the Docutils DTD and the workings (perhaps internals) of ElementTree before that, so can comment on those then. btw, i joined docutils-dev and have been wondering how they see ElementTree .. as docutils documents are element trees too, perhaps it would make sense even for docutils to use the elementree lib (am not sure yet about what it provides to) .. if it would be written today (probably is not worth a rewrite tho). ~Toni P.S. to Tzanko's question / idea about code in the books -- i have no thoughts on that yet. will try to keep Robert's REPL arguments in mind for that, and the remarks from Hans Meine and others. From hans_meine at gmx.net Mon Jul 4 07:36:44 2005 From: hans_meine at gmx.net (Hans Meine) Date: Mon, 4 Jul 2005 13:36:44 +0200 Subject: [IPython-dev] Re: questions about the notebook interface In-Reply-To: <42C4E5AA.90504@colorado.edu> References: <42C00157.2020808@an.org> <42C4E5AA.90504@colorado.edu> Message-ID: <200507041336.44271.hans_meine@gmx.net> On Friday 01 July 2005 08:41, Fernando Perez wrote: > > Toni Alatalo wrote: > > I don't think that any-old-text-editor is a viable target. It just > > doesn't provide any of the real notebook capabilities. Targetting it as > > an intermediate platform before a GUI shell is available sacrifices the > > capabilities that the GUI shell is going to provide. > > Well, that approach has worked pretty well for Latex. You can edit it with > xemacs/vi/notepad/whatever, and yet the LyX guys managed to build a fancy > GUI environment on top of it. Furthermore, lyx actually handles the > document in its own .lyx format, and exports to latex for the final > rendering pass. But if you want, you can embed raw latex into the document > via ERT insets. Again, I want to stress the importance of the interactive programming aspect of the interface. The LaTeX-way has the disadvantage that LyX basically contains a "LaTeX light" rendering engine, since calling LaTeX is not feasible for interactive feedback. For programming, writing the program in an editor and running it from a shell (or even via a keyboard shortcut from within your IDE) is already possible today, but that's where ipython jumps in as an improvement. I would be interested in a notebook frontend building on that, basically expanding the (already quite powerful) editing possibilities. > I don't see why a similar model can't work for us: a simple enough format > that it's valid python syntax, so it can be edited with normal programming > tools. A notebook library to render the files, and a GUI on top which > enables interactive use of such documents, using the ipython engine. I understand that you're more interested in the rendering part, but I am much more interested in the interactive GUI. Without a powerful, comfortable GUI, I won't be interested in created the documents I would be able to print out. (Admittedly, even then I would seldomly use the printing..) > For a prototype, a curses-based one could be written fairly easily. It > wouldn't be portable in the long term, but it might be enough for a lot of > users on *nix systems. That would be a good start AFAICS. Anybody interested in working on sth. like that? Ciao, / / /--/ / / ANS From hans_meine at gmx.net Mon Jul 4 08:43:17 2005 From: hans_meine at gmx.net (Hans Meine) Date: Mon, 4 Jul 2005 14:43:17 +0200 Subject: [FWD] Re: [IPython-dev] questions about the notebook interface In-Reply-To: <42C4F378.3000005@colorado.edu> References: <42C4E6A8.8050605@colorado.edu> <42C4F378.3000005@colorado.edu> Message-ID: <200507041443.18054.hans_meine@gmx.net> On Friday 01 July 2005 09:40, Fernando Perez wrote: > >>From: Hans Meine BTW: Sorry for reposting my mail, I did not see/read your forward due to time constraints.. > Indeed, the interactive aspect is pretty dear to my heart (I wouldn't have > spent all this time developing ipython if I dindn't care about that :). > And I think that Tzanko has a good shot at making headway pretty quickly on > this front with ipython+pycrust. A while ago I tried to embed ipython in > IDLE and pycrust; while it didn't quite work, it seemed to me like the > remaining problems were easy to solve. Cool. That's what I dreamt of.. ;-) > Yup, this is really annoying. For much of this, Xemacs+ipython.el can be a > better solution, as you can keep a scratch.py buffer where you do > multi-line edits, and run them in the ipython buffer with C-c C-|. I did not find ipython.el that impressive; it surely has advantages (multi-line editing), but also disadvantages (crude shell feeling). > >>I successfully analyzed some data derived from images. Now I want to > >> change some preprocessing parameters and re-run the analysis. Again, I > >> want to insert / change some code at the beginning and re-execute > >> several blocks of commands that use the now changed variables. > > This one is already doable in ipython: just call > > macro mm 12:30 34 39:50 > > and then typing mm is like re-executing all those lines. Ah yes, I repeatedly made mental notes to myself to try that out, but did not use it yet. Indeed, that seems to fill a gap. Thanks for your pointers! > My current favorite idea is for the notebook format to have some kind of > include directive like: > > markup.graphic('foo.png',options...) > > Once a fancy GUI is up and running which can display text and inline > graphics, they could be shown on-screen. But at least rendering them to > PS/PDF should be easy. Note that the GUI for this is not trivial: I think > that only Qt4 (released yesterday) has a rich text widget with inline > graphics support. I may be wrong, but I think that none of the text > widgets in GTK, Tk or WX have this capability. I know Qt3 well and it "supports"(*) it, and I think I saw some graphics-capable richtext in another toolkit-demo, too. Hmm, gtk-demo does not have it (2.4.9). *: Qt4 will have a *much* nicer richtext API though. The 3.3 one is hard to extend anyways (e.g. by subclassing, since quite some stuff is private). > That's why I'm > concentrating on having proper backend support for now, along with a basic > GUI which can at least display the text parts and provide the interactive > support. Good plan. > This would give us the possibility to: > > 1. Edit the notebooks 'by hand' with plain editors, even if interactive > support isn't there. > > 2. Have a gui with interactive support that can provide the ability to > re-edit the same input group multiple times without re-generating new input > cells. > > If both systems can operate on the same files, one could switch back and > forth between the editor and the interactive environment, depending on the > task at hand (heavy-duty coding or multiple tests). I love that idea. > >>A friend of mine wants to be able to write longer Mathematic-like > >> notebooks, and needs proper namespacing for chapters/sections, so that > >> variable names can be re-used. > > This can be done by issuing a 'del' statement at the start, much like in > Mathematica you need to call Clear[] on symbols. Hmm. Does that still allow to re-run code in the previous section that refers to the deleted variables? My impression is that 'del' is not enough, but there need to be separate namespaces, no? Ciao, / / /--/ / / ANS From shlomme at gmx.net Mon Jul 4 14:08:42 2005 From: shlomme at gmx.net (Torsten Marek) Date: Mon, 04 Jul 2005 20:08:42 +0200 Subject: [IPython-dev] Some notebook thoughts Message-ID: <42C97B2A.7090809@gmx.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi all, I read with great pleasure that XML has been taken instead of Python as the markup language for Notebook. It's not that the same goal could not be reached with Python, but XML gives you a whole lot of stuff for free when it comes to parsing. I added some user stories to the wiki and I'd like to elaborate on them more generally: When I thought about the notebook GUI, I doubted that any existing text editor widget might be flexible enough to be used for what I had in mind. There are "normal" text areas (In [xx]s and Out [xx]s), but there are a lot of other widgets (I just call them like that, but a fancy term like notebooklets might be better), for any kind of picture, tables, comments, rich texts, code fragment editors, pretty-printed data structures, other widgets, MathML ... you name it. For each widget type (and In and Out would also be widgets), there is a node-type in the markup file and a handler in the renderer (be it a static one to HTML, LaTeX, PDF, SVG or a GUI interface). The mininum interface would be handlers for In, Out and a handler for stuff from print statements. For a GUI that supports all kinds of widgets, a canvas widget like OGL (from Wx) or QCanvas (from Qt) might be a better choice. However, for a small implementation, Scintilla might be enough, as long as the underlying architecture is flexible enough. Am I making any sense with that or is that too far out in the woods? greetings Torsten - -- Torsten Marek ID: A244C858 -- FP: 1902 0002 5DFC 856B F146 894C 7CC5 451E A244 C858 Keyserver: subkeys.pgp.net -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCyXsqfMVFHqJEyFgRApGAAJ0c+tqadZnFC3L8y5kADAy4oP0+ZACgoW9M QSNz7ibWvRkozh+ac0HIJ5o= =ZeN5 -----END PGP SIGNATURE----- From rkern at ucsd.edu Mon Jul 4 17:19:20 2005 From: rkern at ucsd.edu (Robert Kern) Date: Mon, 04 Jul 2005 14:19:20 -0700 Subject: [IPython-dev] Re: Some notebook thoughts In-Reply-To: <42C97B2A.7090809@gmx.net> References: <42C97B2A.7090809@gmx.net> Message-ID: Torsten Marek wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hi all, > > I read with great pleasure that XML has been taken instead of Python as the > markup language for Notebook. It's not that the same goal could not be reached > with Python, but XML gives you a whole lot of stuff for free when it comes to > parsing. > > I added some user stories to the wiki and I'd like to elaborate on them more > generally: > > When I thought about the notebook GUI, I doubted that any existing text editor > widget might be flexible enough to be used for what I had in mind. There are > "normal" text areas (In [xx]s and Out [xx]s), but there are a lot of other > widgets (I just call them like that, but a fancy term like notebooklets might be > better), for any kind of picture, tables, comments, rich texts, code fragment > editors, pretty-printed data structures, other widgets, MathML ... you name it. > > For each widget type (and In and Out would also be widgets), there is a > node-type in the markup file and a handler in the renderer (be it a static one > to HTML, LaTeX, PDF, SVG or a GUI interface). The mininum interface would be > handlers for In, Out and a handler for stuff from print statements. > > For a GUI that supports all kinds of widgets, a canvas widget like OGL (from Wx) > or QCanvas (from Qt) might be a better choice. However, for a small > implementation, Scintilla might be enough, as long as the underlying > architecture is flexible enough. > > Am I making any sense with that or is that too far out in the woods? A little too far into the woods, I think. :-) I think that we can do everything that we need to do with a decent rich text editor that shows images. Tables, comments, rich texts, code fragment editors, and pretty-printed data structures are all text. Pictures, SVG, embedded LaTeX formulae, MathML, movies, Kiva contexts, etc. can all be shown as static pictures for display. Clicking on them could activate editors or viewers in other windows. It doesn't have to look gorgeous on-screen; it just has to be functional. That said, Scintilla ultimately won't be the widget of choice. It doesn't show images, and it really doesn't do HTML-like rich text well either. It is really only suited for syntax highlighted source code editors. For my personal use, that actually won't be too bad. If I can write reST as my text between In/Out cells, using indentation to control the nesting of those cells in the document, I will really be quite happy. But we, as a whole, really do want to be able to show embedded images and have real rich text. It's the nesting of those cells within rich text that kills the idea of implementing it all on a QCanvas or similar object. We *don't* want to rewrite a rich text editor widget. One idea is to simply have a linear arrangement of cells: Rich Text, In, In, Out, In, Rich Text, In, Out, etc. each cell getting it's own widget layed out in, e.g. a wx.ScrolledWindow. That model fails, however, when (In , In, Out, In) are actually supposed to be three levels down:
Stuff More stuff. The situation might be recoverable, but it's going to be easier on us if we can shove everything into a single rich text editor widget. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From hans_meine at gmx.net Tue Jul 5 07:26:45 2005 From: hans_meine at gmx.net (Hans Meine) Date: Tue, 5 Jul 2005 13:26:45 +0200 Subject: [IPython-dev] Re: Some notebook thoughts In-Reply-To: References: <42C97B2A.7090809@gmx.net> Message-ID: <200507051326.45657.hans_meine@gmx.net> On Monday 04 July 2005 23:19, Robert Kern wrote: > It's the nesting of those cells within rich text that kills the idea of > implementing it all on a QCanvas or similar object. We *don't* want to > rewrite a rich text editor widget. Right. I also think rich-text widgets are more appropriate than general canvas widgets. > One idea is to simply have a linear > arrangement of cells: Rich Text, In, In, Out, In, Rich Text, In, Out, > etc. each cell getting it's own widget layed out in, e.g. a > wx.ScrolledWindow. That model fails, however, when (In , In, Out, In) > are actually supposed to be three levels down: [...] Good point. I want to add another little one; we fiddled around with several setups for a (Qt-based) python shell, and one problem was indentation. The terminal's main author really wants the prompts to be left-aligned, but output to be indented, so that you can immediately recognize the beginnings of paragraphs. Let me sketch it: >>> a = [1,2,3] >>> print a [1, 2, 3] >>> foo Traceback (most recent call last): File "", line 1, in ? NameError: name 'foo' is not defined >>> for i in range(3): ... b = i ... >>> That'd be another user story for a GUI (anybody cares to add it for me?). Actually I don't care much about it, and there will probably be other means to quickly differentiate between the different cells, but anyhow. Oh, and another one: One might want to have the output colored differently depending on whether it came through stdout or stderr; e.g. red for errors/warnings. Ciao, / / /--/ / / ANS From edreamleo at charter.net Tue Jul 5 10:11:39 2005 From: edreamleo at charter.net (Edward K. Ream) Date: Tue, 5 Jul 2005 09:11:39 -0500 Subject: [IPython-dev] Leo, IPython and the summer of code Message-ID: <003d01c5816b$765c2040$6500a8c0@edwardx330hgzb> There are slight inaccuracies in section 2.2 (complexity level) and section 1 (overview) of the description of the interactive notebooks project at http://IPython.scipy.org/google_soc/ipnb_google_soc.pdf Section 2.2 states, "This project is relatively complicated.it will require a fair amount of thinking in terms of overall library architecture." Section 1 states, "The full-blown implementation of a graphical shell supporting this kind of work model is probably too ambitious for a summer project. Simultaneous support for rich-text editing, embedded graphics and syntax-highlighted code is extremely complex, and likely to require far more effort than can be mustered by an individual developer for a short-term project." Actually, this project is already complete. Leo, http://webpages.charter.net/edreamleo/front.html, already _is_ an interactive notebook for Python and IPython. Any remaining work on this project, if done within the framework of Leo, would likely involve tasks of one hour to one day in duration. I can state this with confidence because I am Leo's author. What is Leo you ask? Leo is full featured outlining editor. Leo is Open Software (Python license), written in 100% pure Python using Tkinter. Leo has an extremely active group of users. Many are wildly enthusiastic. Leo is presently one of the top 100 SourceForge sites. Leo is production-quality software; bugs are typically fixed in a few days. More importantly, Leo, right now, implements everything envisioned in the project description: - Leo works well with IPython. See below for details. - Leo has a powerful plugin mechanism. An ipython plugin would be the natural way to do larger summer of code projects in Leo. - Leo outlines are far more capable than typical notebooks. Clones allow multiple views of data to exist within a single outline. - Leo's rst2 plugin already supports rst markup, including embedded graphics. - Leo plugins can define extensions to markup and Leo directives. Such extensions might be useful in larger summer of code projects. - Leo can create various external files (called derived files) from a single outline. Derived files can be edited in any editor, and changes made to derived files in others editors automatically appear in an outline when Leo reopens it. The format of derived files involves 10 years of design and implementation. - Leo shows that a simple markup language called noweb, http://www.eecs.harvard.edu/~nr/noweb/, when used in an outline context, is much more powerful than a more complex markup in a 'flat' context. There is absolutely no need for yet another markup language. - noweb markup is language neutral: it can be used in combination with any other language. - If markup other than noweb were really needed, the ipython plugin could define a syntax colorer for that markup. - Leo supports syntax coloring for LaTeX, cweb and many other languages. In the near future, this colorer will become even more flexible. - Several flavors of htmlize scripts already exist. Writing other custom converters would be dead easy. (For those who understand, just put the script of the converter in an @button node.) - Leo is a superb scripting environment for Python. See http://webpages.charter.net/edreamleo/WhatIsLeo.html In this environment, all manner of customizations become very easy. For example, @button nodes would be the way to do smaller summer of code projects. - As shown below, scripts run from Leo (or from IPython invoked from Leo) have full access to all data in Leo outlines. This crucial element is missing from the project description. - Leo works well with editors such as vim and emacs. - Leo can silently run a script in batch mode. Just execute: leo.py --batch yourScript.py In short, Leo already goes far beyond the capabilities described in the summer of code proposal. The following section describes how IPython presently works with Leo. Using IPython and Leo To embed IPython into Leo simply create a node in a Leo outline whose body text contains the standard embedding code: from IPython.Shell import IPShellEmbed ipshell = IPShellEmbed() ipshell() To run IPython from Leo, do the following: 1. Make sure that Leo is running with a console window visible. 2. Select the node containing the above script and do Leo's Execute Script command. Better yet, you can create a 'script button' in Leo's icon area that will run this script when pressed. (The scripting plugin must be enabled in order to create script buttons.) After executing this script, (either using the Execute Script command or by pressing the 'ipython' script button) Leo continues to run while IPython is running. Way cool. Important: When started from Leo using the script above, the c, g and p variables are defined in IPython. c is Leo's present commander (window), g is the leoGlobals module and p is the present position in the outline. These three variables give you access to all the data in the present Leo outline! This is essentially perfect integration with Leo and IPython, right out of the box. For example, the following script, **when executed by IPython invoked from Leo**, prints all the headlines of the present outline, properly indented: for p in c.allNodes_iter(): print p.level()*' ',p.headString() Conclusions Leo already is a complete interactive notebook for IPython, with all the features described in the proposal. Furthermore, Leo shows how the proposal could be improved :-) Please tell me if I have misunderstood the proposal in any way. Please look at Leo carefully: as far as the proposal is concerned, you can probably accomplish more in one week using Leo than you could in six months without Leo. After using Leo for a bit you will certainly have ideas for improving how Leo and IPython work together. Implementing those ideas would likely take just a few hours in the context of Leo. Use plugins for larger projects; use @button nodes for smaller (instant) projects. I am willing to do whatever it takes to make Leo work better with IPython. I would love for the IPython community to get what Leo is all about. Edward P.S. At the EuroPython conference there was unanimous agreement that Leo needs better documentation for newcomers, and that Leo must be able to simulate emacs and vim key bindings so Leo can be used more easily without a mouse. These are now top priority items and I hope to finish both within a month. Please don't let these defects get in the way of learning Leo. In particular, please do ask questions about Leo either here or at Leo's help forum: http://sourceforge.net/forum/forum.php?forum_id=10227 EKR -------------------------------------------------------------------- Edward K. Ream email: edreamleo at charter.net Leo: Literate Editor with Outlines Leo: http://webpages.charter.net/edreamleo/front.html -------------------------------------------------------------------- From edreamleo at charter.net Tue Jul 5 10:39:53 2005 From: edreamleo at charter.net (Edward K. Ream) Date: Tue, 5 Jul 2005 09:39:53 -0500 Subject: [IPython-dev] Leo and xml Message-ID: <004a01c5816f$68236430$6500a8c0@edwardx330hgzb> I forgot to mention in the long post that .leo files are xml files. The xml contains no domain-specific information--just enough to represent outlines, including clones. 10 years of experience with Leo shows that this general approach works very well. Leo supports so-call user attributes (uA's) in the elements. In practice, this allows for unlimited user-domain information to be added. For sure, Leo outlines are all you are ever going to need to represent any kind of notebook, whatever the data it contains. I am a huge fan of xml, but there is no need for xml outside of the .leo file that represents the overall outline. In Leo's terms, derived files should be free of xml, unless the derived file is itself an xml or html file. This clean split between outlines and the files derived from outlines has taken a huge amount of work to get right. You aren't going to recreate it in a summer :-) I guess my real point is that Leo solved all the conceptual problems in the proposal a long time ago. Every thing just works. Rather than parsing complex markup, conversion scripts handle nodes that may possibly contain Leo directives. Since plugins can define their own directives, this does not contrain the user or the tools in any way. I suspect these remarks may seem a bit oracular at present. All this will become clearer once you start using Leo. Put another way, a lot of design problems are suddenly going to get a lot easier when cast in terms of Leo outlines. Edward -------------------------------------------------------------------- Edward K. Ream email: edreamleo at charter.net Leo: Literate Editor with Outlines Leo: http://webpages.charter.net/edreamleo/front.html -------------------------------------------------------------------- From hans_meine at gmx.net Tue Jul 5 13:55:55 2005 From: hans_meine at gmx.net (Hans Meine) Date: Tue, 5 Jul 2005 19:55:55 +0200 Subject: [IPython-dev] Leo, IPython and the summer of code In-Reply-To: <003d01c5816b$765c2040$6500a8c0@edwardx330hgzb> References: <003d01c5816b$765c2040$6500a8c0@edwardx330hgzb> Message-ID: <200507051955.55220.hans_meine@gmx.net> On Tuesday 05 July 2005 16:11, Edward K. Ream wrote: > There are slight inaccuracies in section 2.2 (complexity level) and section ..this is a nice introduction for what follows.. > [...] In short, Leo already goes far beyond the capabilities described in > the summer of code proposal. [...] My main question is: Why the heck did nobody mention this before?? How come that Leo can be used with IPython but not even the IPython author knows this? How come that we tried to implement a python GUI at our lab in the last months, now I got to know (and love) IPython, and after some weeks somebody tells me that there's even more? I must be dreaming. I *will* investigate what Leo offers. It feels strange to me that there are such large gaps (/missing links) in the world wide "web", that things like IPython and Leo can remain undiscovered. OTOH it's the same with research: Every other day you find that people have already though your thoughts, and most often years before you did. Ciao, / / /--/ / / ANS From rkern at ucsd.edu Tue Jul 5 15:36:04 2005 From: rkern at ucsd.edu (Robert Kern) Date: Tue, 05 Jul 2005 12:36:04 -0700 Subject: [IPython-dev] Re: Leo, IPython and the summer of code In-Reply-To: <003d01c5816b$765c2040$6500a8c0@edwardx330hgzb> References: <003d01c5816b$765c2040$6500a8c0@edwardx330hgzb> Message-ID: Edward K. Ream wrote: > More importantly, Leo, right now, implements everything envisioned in > the project description: > > - Leo works well with IPython. See below for details. It does not interact with IPython in the way we need. The proposal is a bit unclear as to the precise goal of this project. It gives undue emphasis on the file format which gives readers the impression that it is proposing a new literate programming tool. If that were the case, then Leo would satisfy the proposal just fine. However, we are not looking for a literate programming tool. We are looking for a GUI Read-Eval-Print Loop that is embedded in a rich text document. The way that Leo interacts with IPython is to run the REPL in a console divorced from the document. This is not what we want. To see a static example of what we want, please look at this rendered example from my current codebase: http://starship.python.net/~kernr/tut-2.3.5-html.html That is essentially the format that we want to be editing within the GUI. There are two important points here: 1. We want the shell as part of the GUI. 2. We don't want the user to have to do any markup unless he is doing something special. I do not believe that using Leo as a base for this project is justified at this time. Further development to interface with Leo is an option and something to explore later. At this time, I feel that focusing on our core requirements and developing outside of a framework (Leo, or any others like Envisage, Eclipse, Blackadder, and the like) will serve us best in the end. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From hans_meine at gmx.net Tue Jul 5 16:36:14 2005 From: hans_meine at gmx.net (Hans Meine) Date: Tue, 5 Jul 2005 22:36:14 +0200 Subject: [IPython-dev] Leo, IPython and the summer of code In-Reply-To: <200507051955.55220.hans_meine@gmx.net> References: <003d01c5816b$765c2040$6500a8c0@edwardx330hgzb> <200507051955.55220.hans_meine@gmx.net> Message-ID: <200507052236.19061.hans_meine@gmx.net> On Tuesday 05 July 2005 19:55, Hans Meine wrote: > I must be dreaming. I *will* investigate what Leo offers. It feels > strange to me that there are such large gaps (/missing links) in the world > wide "web", that things like IPython and Leo can remain undiscovered. OK, as Robert already pointed out, Leo is not the solution to all our problems. I digged through the first tutorials (awfully slow server AFAICS, and little text with screenies for dummies), this is my summary in case someone wants to pick up where I leave it at for now (maybe you get more ideas for the "ideal" notebook interface): Leo is an "outline editor" (unfortunately using TkInter, which means I had to install TK, TCL and python again). To the upper left, you see the outline, an editable tree structure. You can add, rename and delete nodes as you wish, or even "clone" them; the nodes have reference counts > 2 then and become regular nodes again when you delete all but one instance. Each node can have text associated with it, which is displayed in the editor pane below. This text can be syntax-highlighted code, too. Press Ctrl+Shift+F ("Import to @file") to create a special node which imports a source file (e.g. Python, the tutorial uses Java); leo will create a large subtree with classes and methods from that source. When saving, the tree struture is flattened and saved to the file again. (In addition to the native .leo files.) If you change the structure within Leo (e.g. create sub-nodes that do not correspond to classes or functions), the result will contain special comments that Leo can use to re-create the structure from the file. Much relies on @directives like the @file above or a @language directive that switches on syntax highlighting for all sub-nodes and tells Leo how to handle the sourcecode stuff mentioned above. It's a literate programming tool, and although it looks interesting, it does not seem to be what I hoped it was either. I think Edward was a little unprecise, too. ;-) Interesting (domain) name though.. "edream". :-) -- Ciao, / / .o. /--/ ..o / / ANS ooo -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From Fernando.Perez at colorado.edu Tue Jul 5 17:26:04 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Tue, 05 Jul 2005 15:26:04 -0600 Subject: [IPython-dev] Leo, IPython and the summer of code In-Reply-To: <200507052236.19061.hans_meine@gmx.net> References: <003d01c5816b$765c2040$6500a8c0@edwardx330hgzb> <200507051955.55220.hans_meine@gmx.net> <200507052236.19061.hans_meine@gmx.net> Message-ID: <42CAFAEC.10308@colorado.edu> [ aside: Hans, are you subscribed to ipython-dev? I'm still getting auto-discard notifications from your emails. I want to make sure your setup is Ok, just contact me off-list if you keep having problems ] Hans Meine wrote: > On Tuesday 05 July 2005 19:55, Hans Meine wrote: > >>I must be dreaming. I *will* investigate what Leo offers. It feels >>strange to me that there are such large gaps (/missing links) in the world >>wide "web", that things like IPython and Leo can remain undiscovered. Well, a response to this is simply that there is only a finite amount of time to be spent learning about tools out there :) I was aware of leo's existence, but I thought it was mostly an 'outliner on steroids'. From Ed's email, it seems it also cures AIDS and solves world hunger, so I must have missed some of its capabilities :) Joking aside, Robert points out to the key point, which you confirm: as scientists (these tools are undeniably biased by our own needs and background with tools like Mathematica, though we hope they'll be useful to non-scientists), the interactive read/eval loop is critical. > OK, as Robert already pointed out, Leo is not the solution to all our > problems. I digged through the first tutorials (awfully slow server AFAICS, > and little text with screenies for dummies), this is my summary in case > someone wants to pick up where I leave it at for now (maybe you get more > ideas for the "ideal" notebook interface): > > Leo is an "outline editor" (unfortunately using TkInter, which means I had to > install TK, TCL and python again). To the upper left, you see the outline, > an editable tree structure. You can add, rename and delete nodes as you > wish, or even "clone" them; the nodes have reference counts > 2 then and > become regular nodes again when you delete all but one instance. > > Each node can have text associated with it, which is displayed in the editor > pane below. This text can be syntax-highlighted code, too. > > Press Ctrl+Shift+F ("Import to @file") to create a special node which imports > a source file (e.g. Python, the tutorial uses Java); leo will create a large > subtree with classes and methods from that source. When saving, the tree > struture is flattened and saved to the file again. (In addition to the > native .leo files.) If you change the structure within Leo (e.g. create > sub-nodes that do not correspond to classes or functions), the result will > contain special comments that Leo can use to re-create the structure from the > file. > > Much relies on @directives like the @file above or a @language directive that > switches on syntax highlighting for all sub-nodes and tells Leo how to handle > the sourcecode stuff mentioned above. > > It's a literate programming tool, and although it looks interesting, it does > not seem to be what I hoped it was either. I think Edward was a little > unprecise, too. ;-) Interesting (domain) name though.. "edream". :-) Many thanks for this summary, Hans. I will download Leo and play with it later, as I'm sure it may have a number of interesting things to offer, and we may well steal/collaborate/integrate with it in the future. Cheers, f From Fernando.Perez at colorado.edu Tue Jul 5 17:44:50 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Tue, 05 Jul 2005 15:44:50 -0600 Subject: [IPython-dev] Re: Leo, IPython and the summer of code In-Reply-To: References: <003d01c5816b$765c2040$6500a8c0@edwardx330hgzb> Message-ID: <42CAFF52.2030905@colorado.edu> Robert Kern wrote: > Edward K. Ream wrote: > > >>More importantly, Leo, right now, implements everything envisioned in >>the project description: >> >>- Leo works well with IPython. See below for details. > > > It does not interact with IPython in the way we need. The proposal is a > bit unclear as to the precise goal of this project. It gives undue > emphasis on the file format which gives readers the impression that it > is proposing a new literate programming tool. If that were the case, > then Leo would satisfy the proposal just fine. I take blame for that lack of clarity, and sorry if it confused anyone. It was mostly a poorly-worded attempt at providing a code sketch that would give potential applicants something to read and gauge their own interest in participating. In my defense, part of the confusion comes from the fact that even the basic design had to be hashed out, as it's not completely trivial (at least, it has taken a lot of email and two long telephone conversations with John Hunter --matplotlib-- and Robert Kern, people whom I respect tremendously, so it's not just a 5 minute thing. Either that or John, Robert and I are just dumb beyond salvation :) I jumped on the opportunity to grab the google summer of code students to participate in this, though it's clear that part of the project will involve discussing/understanding the best approach. I think though that by now we've converged fairly well on an that part, and we can start moving on in implementation work. Cheers, f From Fernando.Perez at colorado.edu Tue Jul 5 17:55:01 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Tue, 05 Jul 2005 15:55:01 -0600 Subject: [IPython-dev] Notebook roadmap In-Reply-To: References: Message-ID: <42CB01B5.703@colorado.edu> Robert Kern wrote: > Fernando and I hashed out a few things on the phone today. Short > version: XML wins for now, even if I had to threaten to kidnap > Fernando's cat to do it. Long version: > > http://www.scipy.org/wikis/featurerequests/NoteBook Many thanks for this, Robert. For those who didn't click on the link, Robert has joined in as co-mentor at my pestering. I'm super grateful to him for that, and it will be very useful to have two of us available on this. And besides the threats on Ilona (our lovely Russian blue), Robert's arguments for XML were quite good, esp. in light of how ElementTree serves to placate the gag reflex that all programmers with good taste have when confronted with XML. > There's a page there for adding user stories. I've already added the > ones Hans has posted and some of my own. If you would like to add your > own stories (aka "feature requests"), please make yourself an account > and edit the Wiki to your liking. > > Documentation on the XML format and API will be forthcoming. Hopefully, > we will have SVN space by Tuesday or Wednesday. Yes, the scipy admin is in the process of adding the necessary infrastructure. I can already log in, but can't create users yet. This should be operational within a day or two, and I'll make an SVN repo for this, where Toni and Tzanko can keep their respective codes. IPython itself will also live in SVN soon, so when Tzanko needs to directly patch ipython, he will be able to work there as well. Robert, is your code available anywhere? This could be used as a starter by Toni on the document work, as well as providing Tzanko a mockup target to load. Cheers, f From edreamleo at charter.net Tue Jul 5 18:02:05 2005 From: edreamleo at charter.net (Edward K. Ream) Date: Tue, 5 Jul 2005 17:02:05 -0500 Subject: [IPython-dev] Re: Leo, IPython and the summer of code References: <003d01c5816b$765c2040$6500a8c0@edwardx330hgzb> Message-ID: <003f01c581ad$2ea55680$6500a8c0@edwardx330hgzb> > It does not interact with IPython in the way we need. I implied in my original post that more work would be needed. Suppose you do have some code that you need. Embedding that code in Leo should be straightforward. > However, we are not looking for a literate programming tool. Leo is not _just_ a literate programming tool. The literate programming features are optional, though I suspect that noweb will, in fact, be quite useful for this project. In this discussion what is important is that Leo is a very fine notebook, with many general useful features that have nothing to do with LP. > We are looking for a GUI Read-Eval-Print Loop that is embedded in a rich > text document. What prevents you from putting that loop in Leo? Leo's body pane, that is, the body text of each node, is in fact a rich text document. > 1. We want the shell as part of the GUI. > 2. We don't want the user to have to do any markup unless he is doing > something special. I see no reason why these goals are incompatible with Leo. > I do not believe that using Leo as a base for this project is justified at > this time. Isn't it a bit hasty to dismiss Leo out of hand? Without a code base similar to Leo you are likely to spend years of work on the notebook. Edward -------------------------------------------------------------------- Edward K. Ream email: edreamleo at charter.net Leo: http://webpages.charter.net/edreamleo/front.html -------------------------------------------------------------------- From edreamleo at charter.net Tue Jul 5 18:09:06 2005 From: edreamleo at charter.net (Edward K. Ream) Date: Tue, 5 Jul 2005 17:09:06 -0500 Subject: [IPython-dev] Leo, IPython and the summer of code References: <003d01c5816b$765c2040$6500a8c0@edwardx330hgzb> <200507051955.55220.hans_meine@gmx.net> Message-ID: <006701c581ae$2941adf0$6500a8c0@edwardx330hgzb> > My main question is: Why the heck did nobody mention this before?? How > come > that Leo can be used with IPython but not even the IPython author knows > this? Well, not even the Leo author knew about this until yesterday :-) Edward -------------------------------------------------------------------- Edward K. Ream email: edreamleo at charter.net Leo: http://webpages.charter.net/edreamleo/front.html -------------------------------------------------------------------- From rkern at ucsd.edu Tue Jul 5 18:09:39 2005 From: rkern at ucsd.edu (Robert Kern) Date: Tue, 05 Jul 2005 15:09:39 -0700 Subject: [IPython-dev] Re: Notebook roadmap In-Reply-To: <42CB01B5.703@colorado.edu> References: <42CB01B5.703@colorado.edu> Message-ID: Fernando Perez wrote: > Robert, is your code available anywhere? This could be used as a > starter by Toni on the document work, as well as providing Tzanko a > mockup target to load. The current snaphsot is available here: http://starship.python.net/~kernr/notebook.tar.gz It requires Python 2.4 (can be backported by moving @classmethod down to "foo = classmethod(foo)" and changing some generator expressions to list comprehensions) and ElementTree. It will use cElementTree if available. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From edreamleo at charter.net Tue Jul 5 18:14:26 2005 From: edreamleo at charter.net (Edward K. Ream) Date: Tue, 5 Jul 2005 17:14:26 -0500 Subject: [IPython-dev] Leo, IPython and the summer of code References: <003d01c5816b$765c2040$6500a8c0@edwardx330hgzb><200507051955.55220.hans_meine@gmx.net><200507052236.19061.hans_meine@gmx.net> <42CAFAEC.10308@colorado.edu> Message-ID: <008501c581ae$e83335d0$6500a8c0@edwardx330hgzb> > I will download Leo and play with it later, as I'm sure it may have a > number of interesting things to offer, and we may well > steal/collaborate/integrate with it in the future. Excellent. It appears that I may have offended some by claiming that Leo solves (or can solve) all the proposals requirements. Perhaps I let my excitement that IPython already works well with Leo get in the way of due decorum. I apologize for any hurt feelings. Nevertheless, I think that Leo and IPython have something to offer each other, and I repeat my offer to do whatever it takes to help Leo support IPython. Edward -------------------------------------------------------------------- Edward K. Ream email: edreamleo at charter.net Leo: http://webpages.charter.net/edreamleo/front.html -------------------------------------------------------------------- From rkern at ucsd.edu Tue Jul 5 18:32:01 2005 From: rkern at ucsd.edu (Robert Kern) Date: Tue, 05 Jul 2005 15:32:01 -0700 Subject: [IPython-dev] Re: Leo, IPython and the summer of code In-Reply-To: <003f01c581ad$2ea55680$6500a8c0@edwardx330hgzb> References: <003d01c5816b$765c2040$6500a8c0@edwardx330hgzb> <003f01c581ad$2ea55680$6500a8c0@edwardx330hgzb> Message-ID: Edward K. Ream wrote: >> It does not interact with IPython in the way we need. > > I implied in my original post that more work would be needed. Suppose > you do have some code that you need. Embedding that code in Leo should > be straightforward. You also implied that that work would be a matter of hours or a day. I'm sorry, but that is simply not the case. The bulk of the work, making a REPL that interleaves editable rich text and input/output cells, would still have to be done. Using Leo for the rest of the infrastructure is as likely to hinder as to help. >> However, we are not looking for a literate programming tool. > > Leo is not _just_ a literate programming tool. The literate programming > features are optional, though I suspect that noweb will, in fact, be > quite useful for this project. In this discussion what is important is > that Leo is a very fine notebook, with many general useful features that > have nothing to do with LP. I think we're talking past each other. Leo does not provide a "notebook" in the sense that we are using the term. There are other senses of the term in which Leo does in fanct provide a very fine notebook. >> We are looking for a GUI Read-Eval-Print Loop that is embedded in a >> rich text document. > > What prevents you from putting that loop in Leo? Leo's body pane, that > is, the body text of each node, is in fact a rich text document. Nothing prevents us from putting that loop in Leo. What I am suggesting is that Leo does not currently provide such a thing. It will not be a matter of a day to put it in. >> 1. We want the shell as part of the GUI. >> 2. We don't want the user to have to do any markup unless he is doing >> something special. > > I see no reason why these goals are incompatible with Leo. They aren't, but I don't think that what Leo provides is going to be as much help in achieving these goals as you are claiming. >> I do not believe that using Leo as a base for this project is >> justified at this time. > > Isn't it a bit hasty to dismiss Leo out of hand? Without a code base > similar to Leo you are likely to spend years of work on the notebook. I disagree. The project actually has a fairly limited scope. Another point arguing against using Leo's infrastructure is that we are then tied to having this notebook embedded in Leo. That might be fine if all we wanted was a standalone application, but I hope that the code we develop here can be useful for embedding into other projects as well as Leo. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From rkern at ucsd.edu Tue Jul 5 20:16:38 2005 From: rkern at ucsd.edu (Robert Kern) Date: Tue, 05 Jul 2005 17:16:38 -0700 Subject: [IPython-dev] Re: Notebook roadmap In-Reply-To: <42CB01B5.703@colorado.edu> References: <42CB01B5.703@colorado.edu> Message-ID: Fernando Perez wrote: > Robert, is your code available anywhere? This could be used as a > starter by Toni on the document work, as well as providing Tzanko a > mockup target to load. And some documentation to boot: http://www.scipy.org/wikis/featurerequests/NoteBookXML -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From mtobis at gmail.com Wed Jul 6 12:24:09 2005 From: mtobis at gmail.com (Michael Tobis) Date: Wed, 6 Jul 2005 11:24:09 -0500 Subject: [IPython-dev] notebook composition UI thoughts Message-ID: John Hunter has mentioned the IPython notebook effort to me a couple of times. I've been thinking about the Mathematica user interface for years, and I think it is flawed. I hope that eventually IPython can do better, and I think the steps being plotted out for the short term move in the right direction. A Mathematica notebook serves two primary purposes, which are at odds with each other. The first is as an interactive environment for exploration of mathematical and algorithmic ideas, and the second is as an environment for the development and presentation of formal exposition of the ideas so developed. Consider this Mathematica session, obtained by simply moving the insertion point between the second and third inputs: ==> In[1]:= x = 5 Out[1]= 5 -- In[3] = x * x Out[3]= 36 -- In[2]:= x = 6 Out[2]= 6 <== I save it and get the notebook ==> x = 5 5 -- x * x 36 -- x = 6 6 <== Of course, when I recalculate the cells, the order of execution changes and the second result is the expected 25. There are far more confusing results that are similar in concept but much more confusing in practice. There is also the fact that the notebook inherits context from the entire Mathematica session, including all open notebooks, that leads to similar flavors of confusion. One way to look at the problem with the Mathematica model of notebooks is that it conflates the purposes of authoring and presentation. As a presentation mechanism, the organizing principle is sequential in the file space, but as an interactive execution model, the organizing principle is sequential in time. The quandary is that explorations are developed sequentially in time and read and executed sequentially in text. In practice what I will want if I am developing an interactive educational, um, thing (I'm not sure "notebook" is the right name) is not one but two Python sessions: first an interactive environment not dramatically different from our current iPython, for developing the concepts, and second a live editable interface into which I can paste those bits of text, code and other widgets as I determine that they enhance the presentation. The order in which I edit a document is not always sequential. I need to be able to insert executable statements into the middle of the document. For repeatability, I also want each document to be able to provide all of its own context. For this to make sense I need a new python session (or at least a fresh namespace) invoked every time I make a change in the middle of a document. I need anything after the insertion point that changes to be highlighted and flagged, and I need multiple undos. In the case of a very long document this will cause quite a lot of thrashing around as everything gets recalculated. Perhaps the user interface needs to permit "molecular" rather than atomic changes before enforcing a recalculation. That is, I would insert multiple input lines from my live session into various places in the notebook document before forcing a recalculation of the document. It seems clear to me that some sort of computer mediated presentation mechanism will eventually take over from textbooks in mathematics, engineering and physical science. I think Python deserves to be a big player in this. Mathematica notebooks show us that a great deal of care needs to be put into the authoring interface. The postprocessing model described as the summer project seems to me to support this idea rather than thwart it, so there is fortunately no great urgency in considering these ideas. Basically, my proposed notebook development UI is not one Python session but two, one a conventional iPython session, and the other a rather more ethereal beast that ensures that everything is executed in the order I as author intend and that all context is provided within the notebook and not accidentally in the session. The overlap between these problems and those addressed by Knuth in his original Literate Programming work is, to me, striking. I'm agnostic on the question as to whether the extant Leo code base is the right approach, but the ideas of literate programming are very germane to the ideas of developing scientific "notebooks". Knuth observes that the order in which the reader wants to read the code does not necessarily match the order in which the compiler wants to process it. The flaws in the Mathematica noteboook model show us (among other things) that the order in which an author writes code and documentation is not necessarily the same as either of them. -- mt -- Michael Tobis Geophysical Sciences University of Chicago From rkern at ucsd.edu Wed Jul 6 14:22:08 2005 From: rkern at ucsd.edu (Robert Kern) Date: Wed, 06 Jul 2005 11:22:08 -0700 Subject: [IPython-dev] Re: notebook composition UI thoughts In-Reply-To: References: Message-ID: Michael Tobis wrote: > John Hunter has mentioned the IPython notebook effort to me a couple > of times. I've been thinking about the Mathematica user interface for > years, and I think it is flawed. I hope that eventually IPython can do > better, and I think the steps being plotted out for the short term > move in the right direction. > > A Mathematica notebook serves two primary purposes, which are at odds > with each other. The first is as an interactive environment for > exploration of mathematical and algorithmic ideas, and the second is > as an environment for the development and presentation of formal > exposition of the ideas so developed. > > Consider this Mathematica session, obtained by simply moving the > insertion point between the second and third inputs: > > ==> > In[1]:= x = 5 > > Out[1]= 5 > > -- > > In[3] = x * x > > Out[3]= 36 > > -- > > In[2]:= x = 6 > > Out[2]= 6 > <== > > I save it and get the notebook > > ==> > x = 5 > > 5 > > -- > > x * x > > 36 > > -- > > x = 6 > > 6 > <== > > Of course, when I recalculate the cells, the order of execution > changes and the second result is the expected 25. There are far more > confusing results that are similar in concept but much more confusing > in practice. There is also the fact that the notebook inherits > context from the entire Mathematica session, including all open > notebooks, that leads to similar flavors of confusion. The current document model has two separate sections: a log and a sheet. The log contains all of the text from the In[NN] and Out[NN] prompts and other information. Each element is tagged with the number NN. The sheet contains the rich text that you write and references to those elements in the log. The sheet can contain any, all, or none of the elements in the log in any order and with any number of repetitions. When executed, the notebook will only consider the elements in the log in order. The formatting of the sheet is irrelevant to execution. At the moment, we are focusing on creating an interactive environment, not a code development environment. It is not expected that you will write modules in this environment. For that we leave you to your own editor and/or literate programming tools. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From hans_meine at gmx.net Wed Jul 6 14:40:44 2005 From: hans_meine at gmx.net (Hans Meine) Date: Wed, 6 Jul 2005 20:40:44 +0200 Subject: [IPython-dev] notebook composition UI thoughts In-Reply-To: References: Message-ID: <200507062040.44406.hans_meine@gmx.net> On Wednesday 06 July 2005 18:24, Michael Tobis wrote: > [...] Of course, when I recalculate the cells, the order of execution > changes and the second result is the expected 25. There are far more > confusing results that are similar in concept but much more confusing > in practice. There is also the fact that the notebook inherits > context from the entire Mathematica session, including all open > notebooks, that leads to similar flavors of confusion. Right. That's an important problem to keep in mind. > One way to look at the problem with the Mathematica model of notebooks > is that it conflates the purposes of authoring and presentation. As a > presentation mechanism, the organizing principle is sequential in the > file space, but as an interactive execution model, the organizing > principle is sequential in time. Good point. > In practice what I will want if I am developing an interactive > educational, um, thing (I'm not sure "notebook" is the right name) is > not one but two Python sessions: [...] I think in this part you just make an assumption on the solution, which might not be the right one. I argue that when developing an interactive, educational thing, you do not necessarily want two sessions, but you want a repeatable, understandable log. > [...] For this to make sense I need a new python session (or at least a > fresh namespace) invoked every time I make a change in the > middle of a document. I need anything after the insertion point that > changes to be highlighted and flagged, and I need multiple undos. This goes into the direction I pointed out in one of my user stories: What you'd really like to have is not to flag everything behind the point, but just anything that depends on what you've changed. *Ideally*, you'd want only those parts to be recalculated that actually depend on the changes. > In the case of a very long document this will cause quite a lot of > thrashing around as everything gets recalculated. Ideally, one could just let a dirty flag be passed down a dependency graph, and highlight / gray out dirty cells, which would mean only minimal effort > Perhaps the user > interface needs to permit "molecular" rather than atomic changes > before enforcing a recalculation. That is, I would insert multiple > input lines from my live session into various places in the notebook > document before forcing a recalculation of the document. Right. The dirty cells should be recalculated only on demand/manual activation. > Knuth observes that the order in which the reader wants to read the > code does not necessarily match the order in which the compiler wants > to process it. The flaws in the Mathematica noteboook model show us > (among other things) that the order in which an author writes code and > documentation is not necessarily the same as either of them. OK, then the minimal, nearly trivial first fix for the problem you initially presented would be to store the execution order of cells?! (This corresponds to "the Leo way" BTW, where one could have a sequence of cells that make up a program, and then rearrange clones of all those in a presentation branch of the outline (in a different order).) Note that this does not yet solve the problem that code that initialized variables could have been overwritten in the meantime, although the variables stay in the environment. Those are interesting problems, OTOH it's hard to save the user from shooting him/herself in the foot - he/she will always need to have some kind of idea of what happens behind the UI I guess. Nice greetings, Hans From hans_meine at gmx.net Wed Jul 6 14:54:07 2005 From: hans_meine at gmx.net (Hans Meine) Date: Wed, 6 Jul 2005 20:54:07 +0200 Subject: [IPython-dev] notebook composition UI thoughts In-Reply-To: <200507062040.44406.hans_meine@gmx.net> References: <200507062040.44406.hans_meine@gmx.net> Message-ID: <200507062054.07757.hans_meine@gmx.net> (FYI: At first, I sent my answer to Michael directly by accident, I am quoting his answer. I wish I were able to send mails to this list on the first try... ;-) ) On Wednesday 06 July 2005 19:17, Michael Tobis wrote: > > Ideally, one could just let a dirty flag be passed down a dependency > > graph, and highlight / gray out dirty cells, which would mean only > > minimal effort > > Agreed; this would be nicest. I thought of this (the gray-out > affordance is a nice touch that I missed) and considered it a bit too > much to include in my first message. > > However, the document should vigorously complain about being saved in > such a state. > > It's rather a complicated objective, especially if multiple undo > operations are also supported. My collegue got another quite cool idea; he just asked "ah, you want Python to do that automatically?". Of course that'd be cool, but there's another possibility: By default, each cell should be it's own independent module (I think modules are quite cheap, right?). In order to use stuff from another cell, "from cell_foo import *" could be used. This way, dependencies would be more clear, and one could even use a special syntax / convention to make the extraction of a dependency graph an automatic procedure. There could also be means to make several cells share a module. I think I like that idea very much, because it re-uses existing technology (python modules). One could also use "import foobar" and use fully-qualified references to results from the foobar cell. Additionally, he still wants chapters to be totally independent of each other (maybe separate interpreter instances altogether). > > OK, then the minimal, nearly trivial first fix for the problem you > > initially presented would be to store the execution order of cells?! > > This replaces an inconsistent presentation with one that is > consistently incorrect, which I suppose is an imporvement of a sort. > (giving the "wrong" answer of 36 instead of 25 in the example) It's not even really wrong; the reason for the answer is (more or less) clearly visible in the numbers. > > Those are interesting problems, OTOH it's hard to save the user from > > shooting him/herself in the foot - he/she will always need to have some > > kind of idea of what happens behind the UI I guess. > > We can assume that anyone with an interest in authoring or modifying a > document of this sort is not a fool, but on the other hand the user > interface should not actively endeavor to make a fool of the author > either. That's what I wanted to say. Greetings, Hans From rkern at ucsd.edu Wed Jul 6 15:49:21 2005 From: rkern at ucsd.edu (Robert Kern) Date: Wed, 06 Jul 2005 12:49:21 -0700 Subject: [IPython-dev] Re: notebook composition UI thoughts In-Reply-To: <200507062040.44406.hans_meine@gmx.net> References: <200507062040.44406.hans_meine@gmx.net> Message-ID: Hans Meine wrote: > On Wednesday 06 July 2005 18:24, Michael Tobis wrote: >>[...] For this to make sense I need a new python session (or at least a >>fresh namespace) invoked every time I make a change in the >>middle of a document. I need anything after the insertion point that >>changes to be highlighted and flagged, and I need multiple undos. > > This goes into the direction I pointed out in one of my user stories: What > you'd really like to have is not to flag everything behind the point, but > just anything that depends on what you've changed. *Ideally*, you'd want > only those parts to be recalculated that actually depend on the changes. I think that drastically changing the execution semantics from what ipython currently does is out of scope for this summer's project. We are going to, wherever we can, do the simplest thing that could possibly work. In this case, I think it means the following stories (which happen consecutively): 1. I just made a typo at "In [34]:". I place my cursor back at the typo and correct it. I press Enter to reexecute. The cell is visually replaced by an "In [35]:" cell with the new code. The "Out[34]:" cell has been replaced by "Out[35]:" with the results of executing the new code. 2. I now want to change a value in "In [33]:"'s code and reexecute the cells that follow it, but only if they are still visible in the sheet. Thus, "In [33]:" --> "In [36]:" and "In [35]:" --> "In [37]:", etc. 3. I add an "In [38]:" that references "Out[37]:". I change a value in "In [36]:" and reexecute cells 36--38. "Out[41]:" (was "Out[38]:") doesn't change; I realize I am being silly and actually assign the value I need to a variable and use that instead. 4. Since I find that I am doing this dance quite often, I assign the appropriate cells to a macro that references the variable I keep changing. Now I just reassign that variable and use that macro over and over again. 5. I start a new sheet and specify that I want a fresh interpreter instance. My first input cell is "In [1]:". This is not to say that other execution semantics would not be better suited, but I think that they are much more difficult to implement and really aren't what Toni and Tzanko signed on to do. If you wish to contribute code that does implement these fancier semantics, we will consider using it in addition to the Simplest Way Kern Can Think Of semantics. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From Fernando.Perez at colorado.edu Wed Jul 6 16:34:50 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed, 06 Jul 2005 14:34:50 -0600 Subject: [IPython-dev] Re: notebook composition UI thoughts In-Reply-To: References: <200507062040.44406.hans_meine@gmx.net> Message-ID: <42CC406A.1070205@colorado.edu> Robert Kern wrote: > Hans Meine wrote: > >>On Wednesday 06 July 2005 18:24, Michael Tobis wrote: > > >>>[...] For this to make sense I need a new python session (or at least a >>>fresh namespace) invoked every time I make a change in the >>>middle of a document. I need anything after the insertion point that >>>changes to be highlighted and flagged, and I need multiple undos. >> >>This goes into the direction I pointed out in one of my user stories: What >>you'd really like to have is not to flag everything behind the point, but >>just anything that depends on what you've changed. *Ideally*, you'd want >>only those parts to be recalculated that actually depend on the changes. > > > I think that drastically changing the execution semantics from what > ipython currently does is out of scope for this summer's project. We are > going to, wherever we can, do the simplest thing that could possibly > work. This is mostly a confirmation of Robert's position. Module-per-cell, multiple-interpreters, auto-dependency analysis, etc. are all potentially worthwhile ideas for certain contexts, but certainly not for a first shot. If we can get an interactive environment which enhances the ipython command-line experience with full-screen editing (to address the problem of endless appending to a session), and basic functionality for supporting rich-text documents (even if not rendered directly in real-time in the working window), we'll have a useful system for many of us. This will also be an attainable goal for Toni and Tzanko in the time allotted. I should note that things like automatic dependency analysis are notoriously difficult (I'd actually guess nearly impossible) in python. The language is far too dynamic for that, and this is _exactly_ the same problem which has made all restricted execution systems for python fail (to the point where they've been removed from the standard library). The discussions on python-dev on this issue, both in the context of security and of dynamic type inference for optimization purposes, contain all the gory details. But the basic point is that python has 'eval' and 'exec', both of which are enough to thwart just about any reasonable attempt at dependency analysis. It is true that in mathematica, the top-down flow of the text can be misleading, as the time-flow of execution (only recorded by the prompt numbers) is what really defines the in-memory status of the kernel. There are systems with automatic dependency tracking and recalculation: spreadsheets like excel do this, as well as systems like Mathcad. But those systems have much more limited assignment semantics than python, so they can keep such a dependency tree consistent at all times with reasonable effort. I'd rather not provide such a system if it is going to have random reliability. That's been exactly the position of python-dev on restricted execution: since we can't guarantee it will work, and it can break almost anywhere, we simply won't ship it (until someone can come along with a solution shown to be robust). The extremely dynamic nature of python is what makes it so flexible for many things, but it also carries a price. On the multiple interpreters front, that's more likely in the near term. There's a good possibility that the gui will be built with the ipython 'kernel' living in a separate process, to allow it to be restarted without destroying the gui. If that is done, the basic infrastructure would be in place for supporting multiple interpreters connected to different tabs in a session or something similar. Not necessarily something to be implemented initially, but it the scaffolding would be in place to make it possible. And thanks to all for the feedback! It shows at least that there is a lot of interest in such a system, though everyone's idea of their ideal toy is obviously a bit different :) Cheers, f From hans_meine at gmx.net Wed Jul 6 16:51:37 2005 From: hans_meine at gmx.net (Hans Meine) Date: Wed, 6 Jul 2005 22:51:37 +0200 Subject: [IPython-dev] Re: notebook composition UI thoughts In-Reply-To: <42CC406A.1070205@colorado.edu> References: <42CC406A.1070205@colorado.edu> Message-ID: <200507062251.38149.hans_meine@gmx.net> On Wednesday 06 July 2005 22:34, Fernando Perez wrote: > This is mostly a confirmation of Robert's position. Module-per-cell, > multiple-interpreters, auto-dependency analysis, etc. are all potentially > worthwhile ideas for certain contexts, but certainly not for a first shot. Point taken and agreed. Still, I'd like to have a "future works" section in this discussion and the Wiki. ;-p > If we can get an interactive environment which enhances the ipython > command-line experience with full-screen editing (to address the problem of > endless appending to a session), and basic functionality for supporting > rich-text documents (even if not rendered directly in real-time in the > working window), we'll have a useful system for many of us. Believe me, I am looking forward to it. > I should note that things like automatic dependency analysis are > notoriously difficult (I'd actually guess nearly impossible) in python. > The language is far too dynamic for that, and this is _exactly_ the same > problem which has made all restricted execution systems for python fail (to > the point where they've been removed from the standard library). [...] You're right, it's difficult, but I am not taking the rest of your arguments; I think that our notebook interface is very much different from a restricted execution system, in the sense that it's feasible to just tell the user "don't use import-in-an-eval, but just clearly write it out as an unconditional statement (maybe even at the beginning of the cell), if you want it to work". Some users *want* it to work, and it's no big restriction to use import w/o exec or eval. > It is true that in mathematica, the top-down flow of the text can be > misleading, as the time-flow of execution (only recorded by the prompt > numbers) is what really defines the in-memory status of the kernel. Furthermore, it's about saving and re-executing a freshly opened notebook if I understood that correctly. That is doable with a stored execution order, and important IMO. > There > are systems with automatic dependency tracking and recalculation: > spreadsheets like excel do this, as well as systems like Mathcad. But > those systems have much more limited assignment semantics than python, so > they can keep such a dependency tree consistent at all times with > reasonable effort. Oh, these systems often also contain eval-like things, sometimes even allow (indirect) cyclic references, and often their dep. analysis fails. However, I still find the basic analysis very useful. > I'd rather not provide such a system if it is going to have random > reliability. It's not really "random", right? ;-) Then I wouldn't like to have it either. > That's been exactly the position of python-dev on restricted > execution: since we can't guarantee it will work, and it can break almost > anywhere, we simply won't ship it (until someone can come along with a > solution shown to be robust). This is very much understandable because it's about security, but IMO that's a different world/topic. I probably would not develop critical spaceship-controlling software in any kind of IPython notebook sessions. ;-) > On the multiple interpreters front, that's more likely in the near term. > There's a good possibility that the gui will be built with the ipython > 'kernel' living in a separate process, to allow it to be restarted without > destroying the gui. We did the same in our own "PyTerm", but mainly to make Ctrl-C work. -- Ciao, / / .o. /--/ ..o / / ANS ooo From rkern at ucsd.edu Wed Jul 6 16:59:25 2005 From: rkern at ucsd.edu (Robert Kern) Date: Wed, 06 Jul 2005 13:59:25 -0700 Subject: [IPython-dev] Re: notebook composition UI thoughts In-Reply-To: <200507062251.38149.hans_meine@gmx.net> References: <42CC406A.1070205@colorado.edu> <200507062251.38149.hans_meine@gmx.net> Message-ID: Hans Meine wrote: > On Wednesday 06 July 2005 22:34, Fernando Perez wrote: > >>This is mostly a confirmation of Robert's position. Module-per-cell, >>multiple-interpreters, auto-dependency analysis, etc. are all potentially >>worthwhile ideas for certain contexts, but certainly not for a first shot. > > Point taken and agreed. Still, I'd like to have a "future works" section in > this discussion and the Wiki. ;-p So make one! It's a Wiki. You're allowed to. :-) -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From Fernando.Perez at colorado.edu Wed Jul 6 17:04:15 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed, 06 Jul 2005 15:04:15 -0600 Subject: [IPython-dev] Re: notebook composition UI thoughts In-Reply-To: References: <42CC406A.1070205@colorado.edu> <200507062251.38149.hans_meine@gmx.net> Message-ID: <42CC474F.3010608@colorado.edu> Robert Kern wrote: > Hans Meine wrote: > >>On Wednesday 06 July 2005 22:34, Fernando Perez wrote: >> >> >>>This is mostly a confirmation of Robert's position. Module-per-cell, >>>multiple-interpreters, auto-dependency analysis, etc. are all potentially >>>worthwhile ideas for certain contexts, but certainly not for a first shot. >> >>Point taken and agreed. Still, I'd like to have a "future works" section in >>this discussion and the Wiki. ;-p > > > So make one! It's a Wiki. You're allowed to. :-) Just a note: we're currently using the main scipy wiki for this. I'm working on getting a proper ipython wiki on the new infrastructure being set up by the Enthought folks. We already have our own SVN repo and a bunch of new toys, but this is work in progress, so give it a couple of days... Best, f From edreamleo at charter.net Thu Jul 7 08:25:24 2005 From: edreamleo at charter.net (Edward K. Ream) Date: Thu, 7 Jul 2005 07:25:24 -0500 Subject: [IPython-dev] Why .py files? Message-ID: <001201c582ee$f3a9b560$6500a8c0@edwardx330hgzb> Could someone please explain the fundamental requirement of section 3.1 (files), namely: "A basic requirement of this project will be that the Python notebooks shall be valid Python source files." I think of a notebook as similar to a Mathematica notebook, namely a) the notebook app itself (Mathematica), and b) notebook files (.nb files), for use primarily by the notebook app. In this context, I would suppose the following: - What counts is the user experience in the notebook app. - The natural format for notebook files today would be xml. - Most 'computing' (or scripting) would be done in the notebook app. Why are separate .py files essential? - The notebook app will have filters to produce .pdf, etc. Why not have a filter to produce .py files? Thanks. Edward -------------------------------------------------------------------- Edward K. Ream email: edreamleo at charter.net Leo: http://webpages.charter.net/edreamleo/front.html -------------------------------------------------------------------- From jdhunter at ace.bsd.uchicago.edu Thu Jul 7 09:40:04 2005 From: jdhunter at ace.bsd.uchicago.edu (John Hunter) Date: Thu, 07 Jul 2005 08:40:04 -0500 Subject: [IPython-dev] Why .py files? In-Reply-To: <001201c582ee$f3a9b560$6500a8c0@edwardx330hgzb> ("Edward K. Ream"'s message of "Thu, 7 Jul 2005 07:25:24 -0500") References: <001201c582ee$f3a9b560$6500a8c0@edwardx330hgzb> Message-ID: <87r7eau43v.fsf@peds-pc311.bsd.uchicago.edu> >>>>> "Edward" == Edward K Ream writes: Edward> Could someone please explain the fundamental requirement Edward> of section 3.1 (files), namely: Edward> "A basic requirement of this project will be that the Edward> Python notebooks shall be valid Python source files." Some of the arguments for using *.py files, and the eventual rejection of these arguments in favor of XML, as well as Robert Kern's prototype implementation which uses XML as a storage format, have been discussed exhaustively on this list in the last few weeks http://www.scipy.net/pipermail/ipython-dev/2005-June/thread.html http://www.scipy.net/pipermail/ipython-dev/2005-July/thread.html JDH From edreamleo at charter.net Thu Jul 7 12:58:48 2005 From: edreamleo at charter.net (Edward K. Ream) Date: Thu, 7 Jul 2005 11:58:48 -0500 Subject: [IPython-dev] Why .py files? References: <001201c582ee$f3a9b560$6500a8c0@edwardx330hgzb> <87r7eau43v.fsf@peds-pc311.bsd.uchicago.edu> Message-ID: <000801c58315$25274690$6500a8c0@edwardx330hgzb> > Some of the arguments for using *.py files, and the eventual rejection > of these arguments in favor of XML, as well as Robert Kern's prototype > implementation which uses XML as a storage format, have been discussed > exhaustively on this list in the last few weeks So notebooks are xml files and not .py files? In that case, what, if any, of the original proposal remains valid? Is there a summary of the present proposal? Thanks. Edward -------------------------------------------------------------------- Edward K. Ream email: edreamleo at charter.net Leo: http://webpages.charter.net/edreamleo/front.html -------------------------------------------------------------------- From Fernando.Perez at colorado.edu Thu Jul 7 13:11:36 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Thu, 07 Jul 2005 11:11:36 -0600 Subject: [IPython-dev] Why .py files? In-Reply-To: <000801c58315$25274690$6500a8c0@edwardx330hgzb> References: <001201c582ee$f3a9b560$6500a8c0@edwardx330hgzb> <87r7eau43v.fsf@peds-pc311.bsd.uchicago.edu> <000801c58315$25274690$6500a8c0@edwardx330hgzb> Message-ID: <42CD6248.2060309@colorado.edu> Edward K. Ream wrote: >>Some of the arguments for using *.py files, and the eventual rejection >>of these arguments in favor of XML, as well as Robert Kern's prototype >>implementation which uses XML as a storage format, have been discussed >>exhaustively on this list in the last few weeks > > > So notebooks are xml files and not .py files? In that case, what, if any, > of the original proposal remains valid? Is there a summary of the present > proposal? I haven't (and I'm not going to) write a new formal proposal. The original one was a working draft, and in the discussions John pointed to, we hashed out a new direction. Robert has put up this info here: http://www.scipy.org/wikis/featurerequests/NoteBook There are links to the prototype code he has written using ElementTree there. Toni and Tzanko are finishing up some school-related things, so I expect that we'll begin moving into production by next week. Cheers, f From Fernando.Perez at colorado.edu Thu Jul 7 14:04:40 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Thu, 07 Jul 2005 12:04:40 -0600 Subject: [IPython-dev] Re: notebook composition UI thoughts In-Reply-To: <200507062251.38149.hans_meine@gmx.net> References: <42CC406A.1070205@colorado.edu> <200507062251.38149.hans_meine@gmx.net> Message-ID: <42CD6EB8.4080605@colorado.edu> Hans Meine wrote: > On Wednesday 06 July 2005 22:34, Fernando Perez wrote: > >>This is mostly a confirmation of Robert's position. Module-per-cell, >>multiple-interpreters, auto-dependency analysis, etc. are all potentially >>worthwhile ideas for certain contexts, but certainly not for a first shot. > > > Point taken and agreed. Still, I'd like to have a "future works" section in > this discussion and the Wiki. ;-p As Robert indicated, by all means please post these ideas there. Once we get a proper ipython wiki (the tools are being currently installed) I'll take care of migrating the material over. >>I should note that things like automatic dependency analysis are >>notoriously difficult (I'd actually guess nearly impossible) in python. >>The language is far too dynamic for that, and this is _exactly_ the same >>problem which has made all restricted execution systems for python fail (to >>the point where they've been removed from the standard library). [...] > > > You're right, it's difficult, but I am not taking the rest of your arguments; > I think that our notebook interface is very much different from a restricted > execution system, in the sense that it's feasible to just tell the user > "don't use import-in-an-eval, but just clearly write it out as an > unconditional statement (maybe even at the beginning of the cell), if you > want it to work". Some users *want* it to work, and it's no big restriction > to use import w/o exec or eval. I dislike half-working stuff. For example, ipython has a dreload() command, which tries to do 'deep reloading', to properly update a module chain. This code wasn't written by me, it's tricky and uncommented, and it came as part of one of the original projects I merged into ipython when I started. I have received singing emails of people saying 'dreload() rocks, I'd use ipython if only to have access to it, etc.'. And yet, I hardly publicize it, and I don't let it override the default reload() command (which it used to do in the original LazyPython). The reason is that I've seen it fail in subtle cases, and I've received reports of puzzled users about its behavior. Since I haven't taken the effort to fix it (I tried once, and failed), and in general it can't be 100% fixed (extension modules can't be reloaded as far as I know, regardless of what tricks you play), I keep it in a dark corner. The moment you make a feature reasonably central to a framework, it HAS to be as robust as humanly possible. Because people _will_ find ways of breaking it, either directly or because of third-party code they may be using which does funky things. So I am very, very reluctant to put in a feature such as this, which I know is basically impossible to make really robust. > This is very much understandable because it's about security, but IMO that's a > different world/topic. I probably would not develop critical > spaceship-controlling software in any kind of IPython notebook sessions. ;-) Well, you should talk to Frederic Mantegazza: he uses ipython as the control console for the largest neutron scattering source in Europe (at the Institut Laue-Langevin in Grenoble) :) Cheers, f From Fernando.Perez at colorado.edu Thu Jul 7 14:16:16 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Thu, 07 Jul 2005 12:16:16 -0600 Subject: [IPython-dev] Re: notebook composition UI thoughts In-Reply-To: <42CD6EB8.4080605@colorado.edu> References: <42CC406A.1070205@colorado.edu> <200507062251.38149.hans_meine@gmx.net> <42CD6EB8.4080605@colorado.edu> Message-ID: <42CD7170.3020209@colorado.edu> Fernando Perez wrote: > Well, you should talk to Frederic Mantegazza: he uses ipython as the control > console for the largest neutron scattering source in Europe (at the Institut > Laue-Langevin in Grenoble) :) Oops, hit send a bit early before the urls: http://ipython.scipy.org/misc/PyMAD_description.html http://www.scipy.net/pipermail/ipython-dev/2005-April/000428.html So 24/7 reliability _is_ a goal for ipython :) Best, f From mtobis at gmail.com Fri Jul 8 18:12:32 2005 From: mtobis at gmail.com (Michael Tobis) Date: Fri, 8 Jul 2005 17:12:32 -0500 Subject: [IPython-dev] Re: notebook composition UI thoughts In-Reply-To: References: Message-ID: On 7/6/05, Robert Kern wrote: > The current document model has two separate sections: a log and a sheet. > The log contains all of the text from the In[NN] and Out[NN] prompts and > other information. Each element is tagged with the number NN. The sheet > contains the rich text that you write and references to those elements > in the log. The sheet can contain any, all, or none of the elements in > the log in any order and with any number of repetitions. If I understand you correctly, then the plan does NOT adequately address my concerns with the Mathematica approach, concerns which I believe Hans acknowledged. A second notebook with the same visible inputs in the same order as a first notebook should, in my opinion, be required to yield the same results in both cases. If the sequence of operations is set in a log file normally invisible to the user, then, as in Mathematica, two notebooks can have the same sequence of inputs and different sequences of outputs. This will tend to frustrate and drive away a developer of serious content for the medium. Invisible statements and invisible or at least unemphasized changes in the sequence of operations will cause the notebook to fail to behave in the way that the content developer wants or naturally expects. > When executed, > the notebook will only consider the elements in the log in order. The > formatting of the sheet is irrelevant to execution. If I understand you correctly, then you are replacing Mathematica's inconsistency with a method that is simple, consistent, and wrong. > At the moment, we are focusing on creating an interactive environment, > not a code development environment. I find this distinction not especially helpful. I am using an editor to construct a document to tell a computer what to do. How is this not code development? Sorry to be so strong about this, but I hate to see this unfortunate road traveled again. Michael Tobis From rkern at ucsd.edu Fri Jul 8 19:32:52 2005 From: rkern at ucsd.edu (Robert Kern) Date: Fri, 08 Jul 2005 16:32:52 -0700 Subject: [IPython-dev] Re: notebook composition UI thoughts In-Reply-To: References: Message-ID: Michael Tobis wrote: > On 7/6/05, Robert Kern wrote: > >>The current document model has two separate sections: a log and a sheet. >>The log contains all of the text from the In[NN] and Out[NN] prompts and >>other information. Each element is tagged with the number NN. The sheet >>contains the rich text that you write and references to those elements >>in the log. The sheet can contain any, all, or none of the elements in >>the log in any order and with any number of repetitions. > > If I understand you correctly, then the plan does NOT adequately > address my concerns with the Mathematica approach, concerns which I > believe Hans acknowledged. > > A second notebook with the same visible inputs in the same order as a > first notebook should, in my opinion, be required to yield the same > results in both cases. > > If the sequence of operations is set in a log file normally invisible > to the user, then, as in Mathematica, two notebooks can have the same > sequence of inputs and different sequences of outputs. Every input and output cell is visually marked with its number, just as in current ipython. The placement of the input cells in the document may not correlate with the order they would be reexecuted. The numbers will determine the order of execution. The same sequence of inputs (ordering by the numbers) will give the same outputs. If I rearrange the position of the input cells in the document, the order of execution remains the same. Whether you actually do rearrange the positions, or visually remove cells, is up to you. As you say, "the order in which the reader wants to read the code does not necessarily match the order in which the compiler wants to process it." The order the interpreter wants to process the code is provided by the numbers of the input cells. The order the reader wants to read the code is given by the placement of those cells in the document. There, of course, is the option of manually changing the numbers in order to exercise a finer control over how the document might be reexecuted. Additionally, we can enable more automatic renumberings as well. For example, suppose that I don't want any of the cells I visually removed from the document to show up in the log, and I want the cells renumbered to be contiguous (e.g. instead of skipping from 10 to 14 if I deleted cells 11, 12, and 13). We can provide a command that will clean the log and make the appropriate changes to the cells in the visual document. > This will tend to frustrate and drive away a developer of serious > content for the medium. Invisible statements and invisible or at least > unemphasized changes in the sequence of operations will cause the > notebook to fail to behave in the way that the content developer wants > or naturally expects. They are emphasized by the numbers attached to the cells. We can make an option (via a menu, keyboard shortcut, or toolbar button) that will display the full log. >> When executed, >>the notebook will only consider the elements in the log in order. The >>formatting of the sheet is irrelevant to execution. > > If I understand you correctly, then you are replacing Mathematica's > inconsistency with a method that is simple, consistent, and wrong. Quite possibly. However, it a method that is achievable. If you can contribute code that supports more complicated semantics, we will consider it. I think, however, that I am not explaining myself quite clearly enough. I do believe that the approach I have described does address most of your concerns. >>At the moment, we are focusing on creating an interactive environment, >>not a code development environment. > > I find this distinction not especially helpful. I am using an editor > to construct a document to tell a computer what to do. How is this not > code development? It's the difference between writing a module and typing at the ipython prompt. I, at least, approach these activities very differently. > Sorry to be so strong about this, but I hate to see this unfortunate > road traveled again. > > Michael Tobis -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From mtobis at gmail.com Sun Jul 10 22:42:22 2005 From: mtobis at gmail.com (Michael Tobis) Date: Sun, 10 Jul 2005 21:42:22 -0500 Subject: Fwd: [IPython-dev] Re: notebook composition UI thoughts In-Reply-To: References: Message-ID: On 7/8/05, Robert Kern wrote: > The placement of the input cells in the document may > not correlate with the order they would be reexecuted. The numbers will > determine the order of execution. The same sequence of inputs (ordering > by the numbers) will give the same outputs. If I rearrange the > position of the input cells in the document, the order of execution > remains the same. Understood. This is sufficiently close to Mathematica's approach that it seems to me to fail in the same way to live up to an important potential of the medium, the one which interests me most. That is, as a publication mechanism that can enforce the validity of a formal argument. > There, of course, is the option of manually changing the numbers in > order to exercise a finer control over how the document might be > reexecuted. Additionally, we can enable more automatic renumberings as > well. For example, suppose that I don't want any of the cells I visually > removed from the document to show up in the log, and I want the cells > renumbered to be contiguous (e.g. instead of skipping from 10 to 14 if I > deleted cells 11, 12, and 13). We can provide a command that will clean > the log and make the appropriate changes to the cells in the visual > document. Indeed, this sort of functionality is exactly what I am looking for. I am perfectly OK with it being optional. My purpose in injecting myself into the conversation at this point is to ensure that options of this sort are not foreclosed. > > If I understand you correctly, then you are replacing Mathematica's > > inconsistency with a method that is simple, consistent, and wrong. > > Quite possibly. However, it a method that is achievable. If you can > contribute code that supports more complicated semantics, we will > consider it. I am a bit loathe to commit myself to this but I am indeed inclined to participate in exactly this way. I am not a deep enough Python magician to maintain the dependency trees that Hans is imagining, but there are more pedestrian ways to proceed that are within my capacities. They may lack something in elegance and performance but may be easy to implement and immediately useful in many circumstances. > >>At the moment, we are focusing on creating an interactive environment, > >>not a code development environment. > > > > I find this distinction not especially helpful. I am using an editor > > to construct a document to tell a computer what to do. How is this not > > code development? > > It's the difference between writing a module and typing at the ipython > prompt. I, at least, approach these activities very differently. I think that the boundary between coder and user has become artificially sharp. I suspect this is in part a result of commercial interests that aren't in the best interests of human progress. I suppose that's a topic for another venue, though, and I should take that particular axe and grind it elsewhere. For present purposes, let me just emphasize that some of the best designs are the ones that end up with utility far from their original conceptions. Here's to the prospect of IPython being in that class! regards Michael Tobis From rkern at ucsd.edu Mon Jul 11 00:41:22 2005 From: rkern at ucsd.edu (Robert Kern) Date: Sun, 10 Jul 2005 21:41:22 -0700 Subject: [IPython-dev] Re: Fwd: Re: notebook composition UI thoughts In-Reply-To: References: Message-ID: [Michael originally sent the previous message privately to me and not the list, and I replied privately as well. We seem to be converging on the opinion that the need for more complicated execution semantics (like Hans's dependency graph) can, for the most part, be ameliorated by layering actions on top of the simple, ipython-like execution semantics. Future work on those more complicated semantics may be worth considering after this Summer of Code project has been completed.] Michael Tobis wrote: > On 7/8/05, Robert Kern wrote: > >> The placement of the input cells in the document may >> not correlate with the order they would be reexecuted. The numbers will >> determine the order of execution. The same sequence of inputs (ordering >> by the numbers) will give the same outputs. If I rearrange the >> position of the input cells in the document, the order of execution >> remains the same. > > Understood. This is sufficiently close to Mathematica's approach that > it seems to me to fail in the same way to live up to an important > potential of the medium, the one which interests me most. That is, as > a publication mechanism that can enforce the validity of a formal > argument. I understand your concern, but at the moment, we are not interested in living up to a lofty potential, but to produce useful tools by September 1st. I agree that the semantics I have proposed are not ideal, but they are simple to implement. We hope to build components that can be later reassembled with different semantics as they become implementable. >> There, of course, is the option of manually changing the numbers in >> order to exercise a finer control over how the document might be >> reexecuted. Additionally, we can enable more automatic renumberings as >> well. For example, suppose that I don't want any of the cells I visually >> removed from the document to show up in the log, and I want the cells >> renumbered to be contiguous (e.g. instead of skipping from 10 to 14 if I >> deleted cells 11, 12, and 13). We can provide a command that will clean >> the log and make the appropriate changes to the cells in the visual >> document. > > Indeed, this sort of functionality is exactly what I am looking for. I > am perfectly OK with it being optional. My purpose in injecting myself > into the conversation at this point is to ensure that options of this > sort are not foreclosed. We are not foreclosing the development of anything. We are, however, making choices as to what we will be developing for this Summer of Code project. Especially, we are not foreclosing on things that can be readily built on top of the simple semantics I have proposed. >>> If I understand you correctly, then you are replacing Mathematica's >>> inconsistency with a method that is simple, consistent, and wrong. >> >> Quite possibly. However, it a method that is achievable. If you can >> contribute code that supports more complicated semantics, we will >> consider it. > > I am a bit loathe to commit myself to this but I am indeed inclined to > participate in exactly this way. Then you can understand why I am loathe to commit myself or others to do something that I think is probably impossible. > I am not a deep enough Python magician to maintain the dependency > trees that Hans is imagining, but there are more pedestrian ways to > proceed that are within my capacities. They may lack something in > elegance and performance but may be easy to implement and immediately > useful in many circumstances. It is possible that the specific capabilities that you and Hans are imagining are achievable, but I have my doubts. Working code, of course, relieves all doubts. >>>> At the moment, we are focusing on creating an interactive environment, >>>> not a code development environment. >>> >>> I find this distinction not especially helpful. I am using an editor >>> to construct a document to tell a computer what to do. How is this not >>> code development? >> >> It's the difference between writing a module and typing at the ipython >> prompt. I, at least, approach these activities very differently. > > I think that the boundary between coder and user has become > artificially sharp. I suspect this is in part a result of commercial > interests that aren't in the best interests of human progress. I > suppose that's a topic for another venue, though, and I should take > that particular axe and grind it elsewhere. I am not making a distinction between coder and user. I am mentioning a difference that I see in the kinds of activities that I do. I write reusable modules one way; I write interactive code another way. The mechanisms that I want to support the one activity are different from the mechanisms that I want to support the other. I would also note that literate programming, I think, satisfies the "notebook" niche for the "writing reusable code" activity, and that there is little point in writing a new literate programming tool. Leo, CWEB, et al. do that job just fine. There's no point in trying to reinvent them in one summer. However, I don't think they do the job for interactive work like what we use ipython for, hence we are focusing our summer efforts on creating the best interactive environment. > For present purposes, let me just emphasize that some of the best > designs are the ones that end up with utility far from their original > conceptions. Here's to the prospect of IPython being in that class! > > regards > Michael Tobis -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From antont at an.org Fri Jul 15 08:14:28 2005 From: antont at an.org (Toni Alatalo) Date: Fri, 15 Jul 2005 15:14:28 +0300 Subject: [IPython-dev] Re: Notebook roadmap In-Reply-To: References: <42CB01B5.703@colorado.edu> Message-ID: <42D7A8A4.8090207@an.org> Robert Kern wrote: Hello, > Fernando Perez wrote: > >> Robert, is your code available anywhere? This could be used as a >> starter by Toni on the document work, as well as providing Tzanko a >> mockup target to load. > > And some documentation to boot: > http://www.scipy.org/wikis/featurerequests/NoteBookXML That and the discussions here have been something to digest indeed, but the picture is pretty clear for me now (as in understanding the targets and the Robert's implementation). Where next? Tzanko, can you interface with the 'notabene' API, is there something to discuss about it, or things missing I could start to add? Some random remarks from reading the code: - i also like the target of ReST output, and now we perhaps don't need to write that ourselves, 'cause a xhtml2rest 'renderer' was recenlty posted to docutils-dev. added a comment about that to the wiki page with a link to the code and discussions at http://permalink.gmane.org/gmane.text.docutils.devel/3076 - don't know how important writing notebooks as ReST will be (as sketched in sketch.nbk), but there some docutils directive would be nice for adding the code snippets (cells). .. python: ..even? as XML is not for humans to write.. (and hardly even to read). and of course the ipython log would be also nicer to read in a non-XML format, e.g. like it originally shows in the ipython prompt or something (this is a bit strangely inverse, 'cause rst was originally planned to be written in .py files, but here it's vice versa..?) - the 'ipython monkeypatch' seems to work (except that unlike the example shows a name attribute for the Notebook is required - would it be better to add some default name for notebooks, or keep requiring that and fix the docstring?). am happy that didn't have to write that :o Hm, i feel like am missing all the important issues that do lurk somewhere in the back of the mind, but those are at least some concrete points -- i guess when we get actually going with experimenting with a prorotype etc. they surface. So ideas about what to tackle next would be very welcome. Still this morning I was not totally convinced if Robert's approach was the way to go, but now it seems like that again. Thanks a lot for putting it together! ~Toni From rkern at ucsd.edu Fri Jul 15 10:53:38 2005 From: rkern at ucsd.edu (Robert Kern) Date: Fri, 15 Jul 2005 07:53:38 -0700 Subject: [IPython-dev] Re: Notebook roadmap In-Reply-To: <42D7A8A4.8090207@an.org> References: <42CB01B5.703@colorado.edu> <42D7A8A4.8090207@an.org> Message-ID: Toni Alatalo wrote: > That and the discussions here have been something to digest indeed, but > the picture is pretty clear for me now (as in understanding the targets > and the Robert's implementation). > > Where next? Tzanko, can you interface with the 'notabene' API, is there > something to discuss about it, or things missing I could start to add? > > Some random remarks from reading the code: > > - i also like the target of ReST output, and now we perhaps don't need > to write that ourselves, 'cause a xhtml2rest 'renderer' was recenlty > posted to docutils-dev. added a comment about that to the wiki page with > a link to the code and discussions at > http://permalink.gmane.org/gmane.text.docutils.devel/3076 I didn't really have in mind a reST target. Embedding the tags in text was just the simplest thing I could think of at the time. There are better targets now. > - don't know how important writing notebooks as ReST will be (as > sketched in sketch.nbk), but there some docutils directive would be nice > for adding the code snippets (cells). .. python: ..even? http://bob.pythonmac.org/archives/2003/11/10/making-docutils-do-python-syntax-highlighting-for-html-output/ It still needs a LaTeX handler. > as XML is not > for humans to write.. (and hardly even to read). and of course the > ipython log would be also nicer to read in a non-XML format, e.g. like > it originally shows in the ipython prompt or something (this is a bit > strangely inverse, 'cause rst was originally planned to be written in > .py files, but here it's vice versa..?) Well, outputing just a log to any number of formats has shown itself to be pretty trivial. > - the 'ipython monkeypatch' seems to work (except that unlike the > example shows a name attribute for the Notebook is required - would it > be better to add some default name for notebooks, or keep requiring that > and fix the docstring?). am happy that didn't have to write that :o Tell me about it. Actually, that bit needs to be refactored out to an intermediary object with some other stuff to leave the notebook object pure. Its docstring is the least of the things wrong with it. > Hm, i feel like am missing all the important issues that do lurk > somewhere in the back of the mind, but those are at least some concrete > points -- i guess when we get actually going with experimenting with a > prorotype etc. they surface. I've been fiddling around trying to settle on an XML markup to use. We could try a subset of XHTML, but that is a little less semantic and more presentational than I would prefer. For example, it uses

Title

Something

Lower title

Something else in a sub-section

where Docutils/DocBook uses
Title Something
Lower title Something else in a sub-section
On the plus side, a subset of XHTML is what twisted.lore uses, and twisted.lore seems to be well-designed for plugins for custom tags. It outputs both XHTML and LaTeX. I haven't been so thrilled with prospect of using Docutils as a document generation library. It seems to be very much focused on the input reST-format. Ignoring that bit and generating documents programmatically is apparently not so straightforward. Plugging in custom content is even less so. By far the easiest approach so far has been to use DocBook's XML, transforming the into a (with syntax highlighting tags inside), and getting HTML and LaTeX out via the XSL stylesheets. Using lxml instead of ElementTree (same basic API but adds full XPath and XSLT support), this all happens in Python. The output is very configurable and looks quite nice. Incidentally, Docutils' model seems to be mostly borrowed from DocBook, so if we were to choose an XML subset from Docutils, we'd end up with a subset of DocBook, too. I have a feeling that we'll settle on a DocBook subset even if we handle the document generation ourselves or massage it slightly to pass to some other library and don't bother with the XSL stylesheets. > So ideas about what to tackle next would be very welcome. Still this > morning I was not totally convinced if Robert's approach was the way to > go, but now it seems like that again. Thanks a lot for putting it together! For the document side, one thing we're going to need no matter what is an object(s) to hold style information. I made a stupid stab at it in my SVN sandbox. On the GUI side, we should try to get to the point of rendering something as simple as this: I am a paragraph. ... I am another paragraph. ... Oh yes, it's me again. On the ipython side, we need clean ways to do all kinds of things. I'll get around to writing them down later. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From Fernando.Perez at colorado.edu Fri Jul 15 19:27:42 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Fri, 15 Jul 2005 17:27:42 -0600 Subject: [IPython-dev] [FWD] An interesting take on the Notebook Problem In-Reply-To: References: Message-ID: <42D8466E.1010509@colorado.edu> [Chris, note that ipytho-dev discards non-subscriber posts (too much spam). I've manually forwarded your message ] > ------------------------------------------------------------------------ > > Subject: An interesting take on the Notebook Problem > From: Christopher Hart > Date: Fri, 15 Jul 2005 13:05:02 -0400 > To: ipython-dev at scipy.net > > To: ipython-dev at scipy.net > > > Hi, > > I've been lurking and casually reading the notebook discussion during > the past month or so. I'm quite excited about the progress that seems > to be going on, and the ideas that are being flushed out. > > In the bioinformatics community the problem of recording ones analysis > for both later inspection and sharing is a large problem. This paper > takes, although not precisely focused on the notion of an interactive > notebook, might still be an interesting read to anyone involved in > creating a notebook. I think it highlights nicely some of the > problems and the desires of at least one active community that would > benefit greatly from a ipython notebook. Personally, the particular > implementation they provide doesn't appeal to me - but that's largely > because I prefer (i)python ;) Aside from the methods section - little > actually knowledge of bioinformatics is a prereq for reading. > > Just thought I'd share.... > > Robert Gentleman (2005) "Reproducible Research: A Bioinformatics Case > Study", Statistical Applications in Genetics and Molecular Biology: > Vol. 4: No. 1, Article 2. > http://www.bepress.com/sagmb/vol4/iss1/art2 > > """ > Reproducible Research: A Bioinformatics Case Study > > Robert Gentleman, Harvard University > > ABSTRACT: > While scientific research and the methodologies involved have gone > through substantial technological evolution the technology involved in > the publication of the results of these endeavors has remained > relatively stagnant. Publication is largely done in the same manner > today as it was fifty years ago. Many journals have adopted electronic > formats, however, their orientation and style is little different from > a printed document. The documents tend to be static and take little > advantage of computational resources that might be available. Recent > work, Gentleman and Temple Lang (2003), suggests a methodology and > basic infrastructure that can be used to publish documents in a > substantially different way. Their approach is suitable for the > publication of papers whose message relies on computation. Stated > quite simply, Gentleman and Temple Lang (2003) propose a paradigm > where documents are mixtures of code and text. Such documents may be > self-contained or they may be a component of a compendium which > provides the infrastructure needed to provide access to data and > supporting software. These documents, or compendiums, can be processed > in a number of different ways. One transformation will be to replace > the code with its output -- thereby providing the familiar, but > limited, static document.

In this paper we apply these concepts > to a seminal paper in bioinformatics, namely The Molecular > Classification of Cancer, Golub et al (1999). The authors of that > paper have generously provided data and other information that have > allowed us to largely reproduce their results. Rather than reproduce > this paper exactly we demonstrate that such a reproduction is possible > and instead concentrate on demonstrating the usefulness of the > compendium concept itself. > """ > > --chris From Fernando.Perez at colorado.edu Sun Jul 17 23:13:39 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Sun, 17 Jul 2005 21:13:39 -0600 Subject: [IPython-dev] IPython moved over to SVN... Message-ID: <42DB1E63.8090803@colorado.edu> Hi all, this is just a heads up for those of you who follow ipython's CVS tree: it is no more (well, it remains up, but it's not being updated anymore). We finally have subversion support for ipython, so those of you who are interested should do a fresh svn checkout: svn co http://ipython.scipy.org/svn/ipython/ipython/trunk ipython That's the trunk checkout; if we start doing very disruptive work we'll open a branch for that and will announce it accordingly. One of the benefits of the new SVN setup is the flexibility to have new users and host other work right there. In particular, Robert Kern, Toni, Tzanko and I are all working on the Summer projects at the same repo. Anyone interested can either check out the respective trunks here: svn co http://ipython.scipy.org/svn/ipython/nbdoc/trunk nbdoc svn co http://ipython.scipy.org/svn/ipython/nbshell/trunk nbshell svn co http://ipython.scipy.org/svn/ipython/google-rkern/trunk google-rkern Or, if you are interested in the whole thing, you can just pull the entire repo in one call: svn co http://ipython.scipy.org/svn/ipython Note that this brings the whole repo, with all branches and tags, not just the trunks for each project. All new work, including recent bugfixes, is now taking place in the SVN tree. I'd like to thank Joe Cooper, the Enthought admin, for his unending patience with all my little questions as I was transitioning to the new environment. The new setup is extremely nice, with a lot more flexibility and control than we had before, and I should also thank Enthought for their generous hosting with such nice, speedy resources. Regards, f From tsanko at gmail.com Mon Jul 18 06:18:43 2005 From: tsanko at gmail.com (Tzanko Matev) Date: Mon, 18 Jul 2005 13:18:43 +0300 Subject: [IPython-dev] Changes to Notebook Format Message-ID: <1121681923.3210.18.camel@tzanko> Hi all, I would like to suggest a few changes to the ipython-log part of the notebook format. I think that the following structure might be better: def foo(x, y): return x + y foo(3, 4) 7 The input tags can have types "normal" and "special" which correspond to the tags and of the current format. The output tags can have types "normal", "stdout", "figure", etc. One input tag can have several corresponding output tags with different types. I think that the resulting ElementTree object would be more useful that way. In order to parse a file of the current format I need to create its corresponding ElementTree object and then I must make my own data representation from that object, because searching for tags with XPath is slow and with the current format there is no easier way to retrieve the text I need. With the proposed format I don't really need another internal data rpresentation, which would make my code cleaner. Also with the new format we won't really need the id argument which will make the format less error-prone. Tzanko From rkern at ucsd.edu Mon Jul 18 07:17:37 2005 From: rkern at ucsd.edu (Robert Kern) Date: Mon, 18 Jul 2005 04:17:37 -0700 Subject: [IPython-dev] Re: Changes to Notebook Format In-Reply-To: <1121681923.3210.18.camel@tzanko> References: <1121681923.3210.18.camel@tzanko> Message-ID: Tzanko Matev wrote: > Hi all, > > I would like to suggest a few changes to the ipython-log part of the > notebook format. I think that the following structure might be better: > > > > As a sidenote: conventionally, the "id" attribute on any tag must be unique document-wide, so we'll stick with "number" here. > > def foo(x, y): > return x + y > > > > > > foo(3, 4) > > > 7 > > > > > > The input tags can have types "normal" and "special" which correspond to > the tags and of the current format. The output > tags can have types "normal", "stdout", "figure", etc. One input tag can > have several corresponding output tags with different types. Every "special" input will also have a "normal" input that keeps the transformed-into-valid-Python command, too. > I think > that the resulting ElementTree object would be more useful that way. In > order to parse a file of the current format I need to create its > corresponding ElementTree object and then I must make my own data > representation from that object, because searching for tags with XPath > is slow and with the current format there is no easier way to retrieve > the text I need. With the proposed format I don't really need another > internal data rpresentation, which would make my code cleaner. > > Also with the new format we won't really need the id argument which will > make the format less error-prone. I think that it's entirely possible that the log might have holes in it. "Remove unused log entries" and "Renumber entries to restore contiguity" are two different operations. One might want to perform the former but not the latter if one references In[NN] or Out[NN]. But that may be something we can worry about later. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From rkern at ucsd.edu Mon Jul 18 09:20:50 2005 From: rkern at ucsd.edu (Robert Kern) Date: Mon, 18 Jul 2005 06:20:50 -0700 Subject: [IPython-dev] Re: Changes to Notebook Format In-Reply-To: <1121681923.3210.18.camel@tzanko> References: <1121681923.3210.18.camel@tzanko> Message-ID: Tzanko Matev wrote: > Hi all, > > I would like to suggest a few changes to the ipython-log part of the > notebook format. I think that the following structure might be better: > > > > > > def foo(x, y): > return x + y > > > > > > foo(3, 4) > > > 7 > > > > > > The input tags can have types "normal" and "special" which correspond to > the tags and of the current format. The output > tags can have types "normal", "stdout", "figure", etc. In trying to convert some code to this scheme, I'm finding that it's rather cumbersome. Since the number is attached to the ancestor of and , I can't just get all normal inputs or all special inputs. Nor can I easily tell when a an particular entry also has a special input. For example, when I want to export the plain code, I used to use a single XPath expression to get all normal inputs and another XPath expression to all special inputs, which I stick in a dictionary keyed by the number. Then I go through the normal inputs and replace them with the special inputs if they are in the dictionary. With the above scheme, I iterate over the elements (I changed to ), check for the presence of a special input with an XPath expression, use it if found or if not, then I get the normal input with an XPath expression. That's not ideal. Can you describe the operations that you need to do? How often do you have to access this data such that XPath is observably too slow? I think it's probable that this rearrangement may *increase* the number of XPath expressions you need to evaluate. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From Fernando.Perez at colorado.edu Mon Jul 18 12:09:01 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon, 18 Jul 2005 10:09:01 -0600 Subject: [FWD] Re: [IPython-dev] Re: Changes to Notebook Format Message-ID: <42DBD41D.9030809@colorado.edu> [Toni, I've added you to the whitelist, but if you use multiple email aliases, remember that the ipython lists discard non-subscriber posts] -------------- next part -------------- An embedded message was scrubbed... From: Toni Alatalo Subject: Re: [IPython-dev] Re: Changes to Notebook Format Date: Mon, 18 Jul 2005 14:36:32 +0300 (EEST) Size: 3483 URL: From rkern at ucsd.edu Mon Jul 18 13:02:47 2005 From: rkern at ucsd.edu (Robert Kern) Date: Mon, 18 Jul 2005 10:02:47 -0700 Subject: [IPython-dev] Re: [FWD] Re: Re: Changes to Notebook Format In-Reply-To: <42DBD41D.9030809@colorado.edu> References: <42DBD41D.9030809@colorado.edu> Message-ID: Fernando Perez wrote: > [Toni, I've added you to the whitelist, but if you use multiple email > aliases, remember that the ipython lists discard non-subscriber posts] > > ------------------------------------------------------------------------ > > Subject: > Re: [IPython-dev] Re: Changes to Notebook Format > From: > Toni Alatalo > Date: > Mon, 18 Jul 2005 14:36:32 +0300 (EEST) > To: > ipython-dev at scipy.net > > To: > ipython-dev at scipy.net > > On Mon, 18 Jul 2005, Robert Kern wrote: > >>> >>> >> >>As a sidenote: conventionally, the "id" attribute on any tag must be >>unique document-wide, so we'll stick with "number" here. > > ok. > >>>The input tags can have types "normal" and "special" which correspond to >>>the tags and of the current format. The output >>>tags can have types "normal", "stdout", "figure", etc. One input tag can >>>have several corresponding output tags with different types. >> >>Every "special" input will also have a "normal" input that keeps the >>transformed-into-valid-Python command, too. > > hm didn't get all of this yet but ok i guess. When one does a "special" input (a %magic, a !system command, etc.), IPython will transform it to valid Python, either to a method invocation on the interpreter object or to a comment. I think we need to store both because the transformation may be dependent on what has previously been executed. >>>that the resulting ElementTree object would be more useful that way. In >>>order to parse a file of the current format I need to create its >>>corresponding ElementTree object and then I must make my own data >>>representation from that object, because searching for tags with XPath >>>is slow and with the current format there is no easier way to retrieve >>>the text I need. With the proposed format I don't really need another >>>internal data rpresentation, which would make my code cleaner. > > i definitely agree that you should not have to do any parsing / > structuring of the data for the GUI work -- it's my task to keep the > underlying lib in such a shape that it facilitates your work (of course we > don't have to keep this separation artificially strict, but that's the > idea in principle, right?) > > so please voice out all the demands for the API - like methods you'd like > to have for the notebook objects etc. > >>are two different operations. One might want to perform the former but >>not the latter if one references In[NN] or Out[NN]. >>But that may be something we can worry about later. > > ok > >>Robert Kern > > btw Robert, did you implement / plan to implement those changes already, > or would it be for me to do? a chance to get it going with your codebase, > perhaps.. http://ipython.scipy.org/svn/ipython/google-rkern/branches/cell I resolved my issues by making a Cell object that takes the element and assigns the contents of its children to Python attributes. I've left the figures out of the cells because they have a somewhat different structure. We can change that later when we start doing images in the GUI and think that it's necessary. Also, check out the sample output in the test/ directory. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From tsanko at gmail.com Tue Jul 19 05:01:34 2005 From: tsanko at gmail.com (Tzanko Matev) Date: Tue, 19 Jul 2005 12:01:34 +0300 Subject: [IPython-dev] Python version Message-ID: <1121763694.4785.5.camel@tzanko> Hi all, What Python version should we require for the notebook packages to run? Right now I am using Python 2.3 but I think that my code should work on Python 2.2 as I don't use any new features. Robert Kern's code uses some Python 2.4 features like class decorators, for example. So which is the lowest version we should support? Best, tzanko From Fernando.Perez at colorado.edu Tue Jul 19 12:54:31 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Tue, 19 Jul 2005 10:54:31 -0600 Subject: [IPython-dev] Python version In-Reply-To: <1121763694.4785.5.camel@tzanko> References: <1121763694.4785.5.camel@tzanko> Message-ID: <42DD3047.7030501@colorado.edu> Tzanko Matev wrote: > Hi all, > > What Python version should we require for the notebook packages to run? > Right now I am using Python 2.3 but I think that my code should work on > Python 2.2 as I don't use any new features. Robert Kern's code uses some > Python 2.4 features like class decorators, for example. So which is the > lowest version we should support? I try to keep ipython 2.2-compatible, though to be honest I haven't actually _checked_ with 2.2 in a long time. My default python runtime is 2.3, and there are still a lot of 2.3 installations out there. I think it would be nice to keep 2.3 compatibility, but if it becomes a real burden, it's OK to go for 2.4 (I have it installed as well). By the time this code becomes really stable and widely usable, 2.4 will have been out for long enough that it will be reasonable to use it as a baseline. In summary, if @decorators and such really are helping the readability and clarity of the code, keep 2.4. I'll adapt. Cheers, f From rkern at ucsd.edu Tue Jul 19 13:15:38 2005 From: rkern at ucsd.edu (Robert Kern) Date: Tue, 19 Jul 2005 10:15:38 -0700 Subject: [IPython-dev] Re: Python version In-Reply-To: <42DD3047.7030501@colorado.edu> References: <1121763694.4785.5.camel@tzanko> <42DD3047.7030501@colorado.edu> Message-ID: Fernando Perez wrote: > In summary, if @decorators and such really are helping the readability > and clarity of the code, keep 2.4. I'll adapt. 2.4 features I use: * @classmethod: easily replaced * builtin set(): easily replaced * generator expressions: [easily replaced], but oh, how I'll miss them * keyed sorts and sorted(): less easily replaced -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From shlomme at gmx.net Tue Jul 19 13:25:11 2005 From: shlomme at gmx.net (Torsten Marek) Date: Tue, 19 Jul 2005 19:25:11 +0200 Subject: [IPython-dev] Re: Python version In-Reply-To: References: <1121763694.4785.5.camel@tzanko> <42DD3047.7030501@colorado.edu> Message-ID: <42DD3777.6070305@gmx.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Robert Kern schrieb: > Fernando Perez wrote: > >> In summary, if @decorators and such really are helping the readability >> and clarity of the code, keep 2.4. I'll adapt. > > > 2.4 features I use: > > * @classmethod: easily replaced > * builtin set(): easily replaced > * generator expressions: [easily replaced], but oh, how I'll miss them > * keyed sorts and sorted(): less easily replaced > I think I there are "backports" of sorted() to Python 2.3, but I just can't find them on google right now. greetings Torsten - -- Torsten Marek ID: A244C858 -- FP: 1902 0002 5DFC 856B F146 894C 7CC5 451E A244 C858 Keyserver: subkeys.pgp.net -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFC3TdlfMVFHqJEyFgRAlo2AJ9IabMm15NDADtAmtFbTl6i7IvPdwCgm68A nsAfzgaQrCVEufoErTwXWxs= =Qij6 -----END PGP SIGNATURE----- From Fernando.Perez at colorado.edu Tue Jul 19 13:28:24 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Tue, 19 Jul 2005 11:28:24 -0600 Subject: [IPython-dev] Re: Python version In-Reply-To: References: <1121763694.4785.5.camel@tzanko> <42DD3047.7030501@colorado.edu> Message-ID: <42DD3838.2000409@colorado.edu> Robert Kern wrote: > Fernando Perez wrote: > > >>In summary, if @decorators and such really are helping the readability >>and clarity of the code, keep 2.4. I'll adapt. > > > 2.4 features I use: > > * @classmethod: easily replaced > * builtin set(): easily replaced > * generator expressions: [easily replaced], but oh, how I'll miss them > * keyed sorts and sorted(): less easily replaced Nah, keep 2.4 then. I'll survive. Let's keep the mainline ipython 2.2-safe, though. That one is in real use out there, so I don't want to break things for existing users. I've stuffed all 2.2 compatibility symbols in genutils.py, so put in there anything you need which may be a backport from 2.3/4, if you touch ipython itself (this goes especially to Tzanko). cheers, f From antont at an.org Fri Jul 22 10:05:18 2005 From: antont at an.org (Toni Alatalo) Date: Fri, 22 Jul 2005 17:05:18 +0300 Subject: [IPython-dev] [FWD] An interesting take on the Notebook Problem In-Reply-To: <42D8466E.1010509@colorado.edu> References: <42D8466E.1010509@colorado.edu> Message-ID: <42E0FD1E.6000807@an.org> Fernando Perez wrote: > [Chris, note that ipytho-dev discards non-subscriber posts (too much > spam). I've manually thanks for posting this - was quite a good read, and something that makes the notebook project even more interesting >> takes, although not precisely focused on the notion of an interactive >> notebook, might still be an interesting read to anyone involved in >> (..) >> Robert Gentleman (2005) "Reproducible Research: A Bioinformatics Case >> Study", Statistical Applications in Genetics and Molecular Biology: >> Vol. 4: No. 1, Article 2. >> http://www.bepress.com/sagmb/vol4/iss1/art2 > in fact, the question i was left pondering, regarding the work on python notebooks and the discussions about literate programming vs. the 'REPL' (Read-Eval-Print-Loop, i.e. ipython) modes of working, was how the interactive use fits the picture there. in that paper, they define the so-called Compendiums 'navigable documents', referring to the reader being able to 'explore and reproduce' the content. is such exploration of e.g. a dataset or a model a case of 'REPL', when e.g. changing variables or perhaps defining functions to explore te data/model in different cases? i would think so. that way such compendiums and ipython would be a perfect match, something that literate programming (which if of course one of the underlying basic ideas, as Gentleman also mentions) perhaps does not facilitate in itself? (although checking out Leo etc. is still in my to-do list so can't really tell). installed R the other day to get a feel of that environment, seemed like a rich world and the interactive use was pretty nice (though i still think it is & will be better with Python), but unfortunately could not get the GolubRR package to work (yet). seems to be some version issue: perhaps that GolubRR is packaged for R < 2.0, while the current (in e.g. Debian) is > 2.0. a question about it had been posted to the Bioconductor mailing list, but i could not find any answer (nor haven't gotten one from the poster of that mail) - https://stat.ethz.ch/pipermail/bioconductor/2004-November/006906.html . Besides the SciPy tutorial, perhaps that seminal paper Golub and the R compendium should also be among the 'test cases' for Python notebooks (hopefully the dataset is not too tricky to convert to whatever the format in our system would be). oh and one technical issue relating reproducibility (although that is probably not among the top questions for us yet): as Gentleman notes, some research methods use randomness, and to be able to reproduce the exact same computations, and a heavy way is to include the random table used in the compendium. quick Googling now didn't tell me if the Python random generator is guaranteed to give the same results with the same seed and in what conditions (i know from working on procedural modelling that it at least works on the same computer..), but that can be looked at later. ah and one simple issue: how in our format do we include values of variables in a text? Gentleman shows on p. 11 how a result from a computation is included in a sentence using the Sexpr command in their system. in Python it would of course be, supposing that 'genes' would be the list of selected genes, simply: ".. filtering process selected %d genes" % len(genes) OR "selected " + len(genes) + " genes" OR "selected", len(genes), "genes" -- if we somehow support the form that the print statement handles , but how in the XML? (i guess it's trivial but just don't know yet). there may be other issues in the paper that would be important to note, but those are what i was left thinking. ~Toni From antont at an.org Fri Jul 22 10:05:33 2005 From: antont at an.org (Toni Alatalo) Date: Fri, 22 Jul 2005 17:05:33 +0300 Subject: [IPython-dev] Re: Changes to Notebook Format In-Reply-To: References: <1121681923.3210.18.camel@tzanko> Message-ID: <42E0FD2D.2080607@an.org> Robert Kern wrote: > Tzanko Matev wrote: > >> >> >> def foo(x, y): >> return x + y >> >> > > In trying to convert some code to this scheme, I'm finding that it's > rather cumbersome. Since the number is attached to the ancestor of > and , I can't just get all normal inputs or all > special inputs. Nor can I easily tell when a an particular entry also > has a special input. For example, hm. > when I want to export the plain code, I used to use a single XPath > expression to get all normal inputs and another XPath expression to > all special inputs, which I stick in a dictionary keyed by the number. > Then I go through the normal inputs and replace them with the special > inputs if they are in the dictionary. With the above scheme, I iterate > over the elements (I changed to ), check for > the presence of a special input with an XPath expression, use it if > found or if not, then I get the normal input with an XPath expression. > That's not ideal. thanks for writing up this info on needed operations, btw. > Can you describe the operations that you need to do? How often do you > have to access this data such that XPath is observably too slow? I > think it's probable that this rearrangement may *increase* the number > of XPath expressions you need to evaluate. any info on how this is turning out? i suppose this is not a case of "using xml to solve a problem? now you have two problems.." like the saying goes. yet i wonder, would this be different for you if we were talking about Python? (dunno if the question really warrants an answer) as already told in a private message for the mentors, i'm off-line next week - back to work on this after that, looking forward to looking into these issues then. ~Toni BTW, if i may suggest a style issue: the nbshell code seems to include 'getters' that would be naturally properties in current Python (they came already in 2.2 so no version prob.), e.g. these in PythonPluginFactory: def GetString(self): """ Returns the type string of the plugin. This is used when a notebook file is loaded. See notebookformat.txt for more info""" return "python" def GetType(self): """ Returns the way data should be passed to the plugin. Currently supported types are "raw" and "encoded". See notebookformat.txt for more info""" return "raw" #Probably only the python code plugin should be raw those are currently unnecessary, but could be just .string and .type (or perhaps .type and .input_type?). then later if they need to wrapped to some method just make them properties ( http://www.python.org/2.2/descrintro.html#property ). in one other project i've tried using properties even for something really dynamic, in fact for an exactly similar thing as this in ipnDocument.py: def IsModified(self): """returns if the file has been modified since last save""" #TODO: Right now there is no point in doing anything useful #here. When things are done fix it return True there am using a property called 'changed', which has such a getter that it resets the change flag in the object. so this 'IsModified()' might also be better just as a property called 'modified', but that is already a bit more controversial issue (as ppl are perhaps used to such dynamic things being exposed as methods?) From jdhunter at ace.bsd.uchicago.edu Fri Jul 22 10:10:00 2005 From: jdhunter at ace.bsd.uchicago.edu (John Hunter) Date: Fri, 22 Jul 2005 09:10:00 -0500 Subject: [IPython-dev] Re: Python version In-Reply-To: <42DD3838.2000409@colorado.edu> (Fernando Perez's message of "Tue, 19 Jul 2005 11:28:24 -0600") References: <1121763694.4785.5.camel@tzanko> <42DD3047.7030501@colorado.edu> <42DD3838.2000409@colorado.edu> Message-ID: <87wtnjszif.fsf@peds-pc311.bsd.uchicago.edu> >>>>> "Fernando" == Fernando Perez writes: Fernando> Nah, keep 2.4 then. I'll survive. When starting new projects, my approach is to use the latest of everything, even CVS if you want. Since it will likely be months before anything is usable, might as well go with the latest-greatest, since you'll be stuck with it for some time. JDH From Fernando.Perez at colorado.edu Fri Jul 22 14:42:42 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Fri, 22 Jul 2005 12:42:42 -0600 Subject: [IPython-dev] Re: Changes to Notebook Format In-Reply-To: <42E0FD2D.2080607@an.org> References: <1121681923.3210.18.camel@tzanko> <42E0FD2D.2080607@an.org> Message-ID: <42E13E22.3030207@colorado.edu> Toni Alatalo wrote: > BTW, if i may suggest a style issue: the nbshell code seems to include > 'getters' that would be naturally properties in current Python (they > came already in 2.2 so no version prob.), e.g. these in PythonPluginFactory: > > def GetString(self): > """ Returns the type string of the plugin. This is used when a > notebook > file is loaded. See notebookformat.txt for more info""" > return "python" > > def GetType(self): > """ Returns the way data should be passed to the plugin. Currently > supported types are "raw" and "encoded". See notebookformat.txt for > more info""" > return "raw" #Probably only the python code plugin should be raw > > those are currently unnecessary, but could be just .string and .type (or > perhaps .type and .input_type?). then later if they need to wrapped to > some method just make them properties ( > http://www.python.org/2.2/descrintro.html#property ). +1 on properties. I _hate_ code littered with getX/setX everywhere. This is not Java. I've been convinced to use XML, but that's about as far as you'll get me to go ;) In fact, make X a regular attribute unless it really requires extra validation, synchronization, or work to read/write. But none of this, please: def getX(self): return _X If that's all we need, then X can just be read as foo.X, set as foo.X='bar', and we're done. As Guido likes to say, python is a language for adults :) If in the future things evolve to the point where X needs fancier access controls, it can be turned into a property, and user code (which still respects the validation constraints) will be unaffected. Cheers, f From tsanko at gmail.com Sat Jul 23 14:44:31 2005 From: tsanko at gmail.com (Tzanko Matev) Date: Sat, 23 Jul 2005 21:44:31 +0300 Subject: [IPython-dev] Re: Changes to Notebook Format In-Reply-To: <42E0FD2D.2080607@an.org> References: <1121681923.3210.18.camel@tzanko> <42E0FD2D.2080607@an.org> Message-ID: <1122144272.4157.55.camel@tzanko> > Robert Kern wrote: > > > Tzanko Matev wrote: > > > >> > >> > >> def foo(x, y): > >> return x + y > >> > >> > > > > In trying to convert some code to this scheme, I'm finding that it's > > rather cumbersome. Since the number is attached to the ancestor of > > and , I can't just get all normal inputs or all > > special inputs. Nor can I easily tell when a an particular entry also > > has a special input. For example, > > hm. > > > when I want to export the plain code, I used to use a single XPath > > expression to get all normal inputs and another XPath expression to > > all special inputs, which I stick in a dictionary keyed by the number. > > Then I go through the normal inputs and replace them with the special > > inputs if they are in the dictionary. With the above scheme, I iterate > > over the elements (I changed to ), check for > > the presence of a special input with an XPath expression, use it if > > found or if not, then I get the normal input with an XPath expression. > > That's not ideal. > > thanks for writing up this info on needed operations, btw. > > > Can you describe the operations that you need to do? How often do you > > have to access this data such that XPath is observably too slow? I > > think it's probable that this rearrangement may *increase* the number > > of XPath expressions you need to evaluate. > > any info on how this is turning out? > I attach my response to Robert Kern, and his response. I must have missed to cc: my letter to the mailing list for which I'm sorry. Basicly the idea is that often I will need to find all the outputs that a given input generates and that was slow with the first implementation because I had to use XPath to do it. > BTW, if i may suggest a style issue: the nbshell code seems to include > 'getters' that would be naturally properties in current Python (they > came already in 2.2 so no version prob.), e.g. these in PythonPluginFactory: > > def GetString(self): > """ Returns the type string of the plugin. This is used when a > notebook > file is loaded. See notebookformat.txt for more info""" > return "python" > > def GetType(self): > """ Returns the way data should be passed to the plugin. Currently > supported types are "raw" and "encoded". See notebookformat.txt for > more info""" > return "raw" #Probably only the python code plugin should be raw > > those are currently unnecessary, but could be just .string and .type (or > perhaps .type and .input_type?). then later if they need to wrapped to > some method just make them properties ( > http://www.python.org/2.2/descrintro.html#property ). > > in one other project i've tried using properties even for something > really dynamic, in fact for an exactly similar thing as this in > ipnDocument.py: > > def IsModified(self): > """returns if the file has been modified since last save""" > #TODO: Right now there is no point in doing anything useful > #here. When things are done fix it > return True > > there am using a property called 'changed', which has such a getter that > it resets the change flag in the object. so this 'IsModified()' might > also be better just as a property called 'modified', but that is already > a bit more controversial issue (as ppl are perhaps used to such dynamic > things being exposed as methods?) > Point taken. I've written a lot of C++ code so I have learned some bad habits from it :) I changed some of the get/set methods to properties or variables. A lot of my code was written before it was decided to go with xml so a lot of unnecessary stuff needs to be cleaned up. Right now, I will leave it as it is, because there are more important things to do. Best, tzanko -------------- next part -------------- An embedded message was scrubbed... From: Tzanko Matev Subject: Re: [IPython-dev] Re: Changes to Notebook Format Date: Mon, 18 Jul 2005 23:22:16 +0300 Size: 3583 URL: -------------- next part -------------- An embedded message was scrubbed... From: Robert Kern Subject: Re: [IPython-dev] Re: Changes to Notebook Format Date: Mon, 18 Jul 2005 13:54:07 -0700 Size: 6023 URL: From tsanko at gmail.com Tue Jul 26 06:40:21 2005 From: tsanko at gmail.com (Tzanko Matev) Date: Tue, 26 Jul 2005 13:40:21 +0300 Subject: [IPython-dev] Question about IPython behaviour Message-ID: <1122374421.4682.13.camel@tzanko> Hi all, The prefilter() method turns strings consisting only of whitespace to empty strings. This means that in IPython I cannot write the following code: In[1]: if 1==1: ... ___print 1 ... ___ ... ___print2 ... (here I have replaced spaces with underscore). prefilter() will turn the third line into an empty string and the interpreter will decide that the block ends there. This differs from the behaviour of the standard interpreter where the code would run as expected. So is this a bug or a feature? I think that empty lines should be allowed in the GUI. Best, tzanko From tsanko at gmail.com Tue Jul 26 08:50:44 2005 From: tsanko at gmail.com (Tzanko Matev) Date: Tue, 26 Jul 2005 15:50:44 +0300 Subject: [IPython-dev] Re: Changes to Notebook Format In-Reply-To: <1121681923.3210.18.camel@tzanko> References: <1121681923.3210.18.camel@tzanko> Message-ID: <1122382245.4682.53.camel@tzanko> Hi all, I found other problems with the current format. It is entirely possible that a single input has several outputs, and prints to stdout and/or stderr several times. Take for examlpe the following code. In [1]: if True: ...: 1 ...: print 'foo' ...: 2 ...: print 'bar' ...: Out[1]: 1 foo Out[1]: 2 bar To fix that we should allow several elements of types , and to be included in a cell. Also we want to keep the order in which they were produced. I can see two ways to do this. One way is to add an 'order' attribute to all the , and elements in the cell. Also in the element all the , and elements should have this attribute. Then we won't need separate tags in for all types of output, because the 'number' and 'order' attributes will be enough to decide which output element should be displayed at the given position. Another way is to use the order in which the elements in the cell are written. We can state that the first one or two elements must be of type and . The elements after them will be of types , , ,

, etc. and the order in which they were produced by the input corresponds to the order in which they are written in the cell. Then in we can get rid of all the different tags and replace them with tags like which will point to the second element in the first cell in the log. Right now, I'm not really sure which one is better. The second way will probably be faster to run, because I will be able to work with a cell like a list and not use the find() method or XPath to retrieve elements. What do you think? Best, tzanko From rkern at ucsd.edu Tue Jul 26 13:41:44 2005 From: rkern at ucsd.edu (Robert Kern) Date: Tue, 26 Jul 2005 10:41:44 -0700 Subject: [IPython-dev] Re: Question about IPython behaviour In-Reply-To: <1122374421.4682.13.camel@tzanko> References: <1122374421.4682.13.camel@tzanko> Message-ID: Tzanko Matev wrote: > Hi all, > > The prefilter() method turns strings consisting only of whitespace to > empty strings. This means that in IPython I cannot write the following > code: > > In[1]: if 1==1: > ... ___print 1 > ... ___ > ... ___print2 > ... > > (here I have replaced spaces with underscore). prefilter() will turn the > third line into an empty string and the interpreter will decide that the > block ends there. This differs from the behaviour of the standard > interpreter where the code would run as expected. So is this a bug or a > feature? I think that empty lines should be allowed in the GUI. So code to a different interface like the one in notabene.executor. The GUI decides when the input is complete, sends it whole to the interpreter, and gets a response back. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From rkern at ucsd.edu Tue Jul 26 13:55:22 2005 From: rkern at ucsd.edu (Robert Kern) Date: Tue, 26 Jul 2005 10:55:22 -0700 Subject: [IPython-dev] Re: Changes to Notebook Format In-Reply-To: <1122382245.4682.53.camel@tzanko> References: <1121681923.3210.18.camel@tzanko> <1122382245.4682.53.camel@tzanko> Message-ID: Tzanko Matev wrote: > Hi all, > > I found other problems with the current format. It is entirely possible > that a single input has several outputs, and prints to stdout and/or > stderr several times. Take for examlpe the following code. > > In [1]: if True: > ...: 1 > ...: print 'foo' > ...: 2 > ...: print 'bar' > ...: > Out[1]: 1 > foo > Out[1]: 2 > bar I've never run into a case where I actually want to see multiple Out[1]'s. It's usually in the middle of a for loop and I've forgotten that a function returns an object that I don't really need to see (e.g. pylab.plot()). Since only the last one actually goes into the output cache (that is, is accessible for later use via Out[1]), I'm not convinced that printing multiple Out[1]'s is useful. If I really want to see something in the middle of a loop, I'll print it. As for having stdout cells interleaved with other types of output cells, I think that's going to be pretty hard. You'd have to collect the output from each statement separately. I'm of the opinion that you should just collect one each of stdout and stderr. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From Fernando.Perez at colorado.edu Wed Jul 27 01:44:40 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Tue, 26 Jul 2005 23:44:40 -0600 Subject: [IPython-dev] Re: Changes to Notebook Format In-Reply-To: References: <1121681923.3210.18.camel@tzanko> <1122382245.4682.53.camel@tzanko> Message-ID: <42E71F48.6020205@colorado.edu> Robert Kern wrote: > Tzanko Matev wrote: > >>Hi all, >> >>I found other problems with the current format. It is entirely possible >>that a single input has several outputs, and prints to stdout and/or >>stderr several times. Take for examlpe the following code. >> >>In [1]: if True: >> ...: 1 >> ...: print 'foo' >> ...: 2 >> ...: print 'bar' >> ...: >>Out[1]: 1 >>foo >>Out[1]: 2 >>bar > > > I've never run into a case where I actually want to see multiple > Out[1]'s. It's usually in the middle of a for loop and I've forgotten > that a function returns an object that I don't really need to see (e.g. > pylab.plot()). Since only the last one actually goes into the output > cache (that is, is accessible for later use via Out[1]), I'm not > convinced that printing multiple Out[1]'s is useful. If I really want to > see something in the middle of a loop, I'll print it. Here's an even worse example: In [3]: for i in range(3): ...: print 'passing...' ...: i ...: passing... Out[3]: 0 passing... Out[3]: 1 passing... Out[3]: 2 The multiple Out[] behavior is really mostly an implementation accident, not a design feature. I'd be willing to consider it a bug, in the sense that naked expressions in compound statements should not really be treated as return valuues. I think the example above should simply print 'passing...' three times, and NOT generate an Out[] cell at all. > As for having stdout cells interleaved with other types of output cells, > I think that's going to be pretty hard. You'd have to collect the output > from each statement separately. I'm of the opinion that you should just > collect one each of stdout and stderr. Yup. Cheers, f From Fernando.Perez at colorado.edu Wed Jul 27 01:48:37 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Tue, 26 Jul 2005 23:48:37 -0600 Subject: [IPython-dev] Re: Question about IPython behaviour In-Reply-To: References: <1122374421.4682.13.camel@tzanko> Message-ID: <42E72035.20309@colorado.edu> Robert Kern wrote: > Tzanko Matev wrote: > >>Hi all, >> >>The prefilter() method turns strings consisting only of whitespace to >>empty strings. This means that in IPython I cannot write the following >>code: >> >>In[1]: if 1==1: >> ... ___print 1 >> ... ___ >> ... ___print2 >> ... >> >>(here I have replaced spaces with underscore). prefilter() will turn the >>third line into an empty string and the interpreter will decide that the >>block ends there. This differs from the behaviour of the standard >>interpreter where the code would run as expected. So is this a bug or a >>feature? I think that empty lines should be allowed in the GUI. > > > So code to a different interface like the one in notabene.executor. The > GUI decides when the input is complete, sends it whole to the > interpreter, and gets a response back. I second Robert's suggestion. The reason for ipython's behavior is partly to mimic the default '>>>' interpreter, partly because it's tricky to do otherwise in a line-oriented environment. The default python shell simply uses the convention 'a single empty line signals the end of input'. IPython inherits from code.InteractiveConsole, which maintains this behavior. While not ideal, I think that's not terrible for a simple line-oriented system which doesn't really offer any serious multiline editing. But now that we are migrating to a GUI, we should most certainly offer a way to have blank lines. Cheers, f From rkern at ucsd.edu Wed Jul 27 02:04:20 2005 From: rkern at ucsd.edu (Robert Kern) Date: Tue, 26 Jul 2005 23:04:20 -0700 Subject: [IPython-dev] Re: Question about IPython behaviour In-Reply-To: <42E72035.20309@colorado.edu> References: <1122374421.4682.13.camel@tzanko> <42E72035.20309@colorado.edu> Message-ID: Fernando Perez wrote: > While not ideal, I think that's not terrible for a simple line-oriented > system which doesn't really offer any serious multiline editing. But > now that we are migrating to a GUI, we should most certainly offer a way > to have blank lines. We should take a look at the design of pyrepl_. It has real multi-line editing and probably a design more suited to our endeavors here. We should stea^H^H^H^Hbe inspired by how it solves these problems. .. _pyrepl: http://codespeak.net/pyrepl/ -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From Fernando.Perez at colorado.edu Wed Jul 27 02:21:10 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed, 27 Jul 2005 00:21:10 -0600 Subject: [IPython-dev] Re: Question about IPython behaviour In-Reply-To: References: <1122374421.4682.13.camel@tzanko> <42E72035.20309@colorado.edu> Message-ID: <42E727D6.2020603@colorado.edu> Robert Kern wrote: > Fernando Perez wrote: > > >>While not ideal, I think that's not terrible for a simple line-oriented >>system which doesn't really offer any serious multiline editing. But >>now that we are migrating to a GUI, we should most certainly offer a way >>to have blank lines. > > > We should take a look at the design of pyrepl_. It has real multi-line > editing and probably a design more suited to our endeavors here. We > should stea^H^H^H^Hbe inspired by how it solves these problems. > > .. _pyrepl: http://codespeak.net/pyrepl/ I talked to M. Hudson (pyrepl's author) ages ago about this: pyrepl wasn't suitable for ipython because it's curses-based. That's how they get multi-line capabilities. It is certainly a possibility to build a curses-enabled version of ipython which could run on top of pyrepl. But I think the decision to keep ipython line-oriented was a good one: curses is *nix only, so had I gone with pyrepl early on, we would have lost all the win32 users. And I know that many people in the win32 world do make heavy use of ipython, so I don't want to break things for them. An _optional_ pyrepl-enhanced ipython would be great, and we certainly can look into how they tackle a number of issues. I just wanted to fill you in a bit on the history behind ipython/pyrepl. Cheers, f From rkern at ucsd.edu Wed Jul 27 02:49:23 2005 From: rkern at ucsd.edu (Robert Kern) Date: Tue, 26 Jul 2005 23:49:23 -0700 Subject: [IPython-dev] Re: Question about IPython behaviour In-Reply-To: <42E727D6.2020603@colorado.edu> References: <1122374421.4682.13.camel@tzanko> <42E72035.20309@colorado.edu> <42E727D6.2020603@colorado.edu> Message-ID: Fernando Perez wrote: > I talked to M. Hudson (pyrepl's author) ages ago about this: pyrepl > wasn't suitable for ipython because it's curses-based. That's how they > get multi-line capabilities. It is certainly a possibility to build a > curses-enabled version of ipython which could run on top of pyrepl. I'm not so interested in that part of pyrepl. I haven't delved through the code enough, but it does do multiline editing pretty much as we want to do in the GUI. To enable that, the component that executes the inputs probably doesn't use the code.InteractiveConsole line-based API. *That's* the part we should steal (if indeed, it does exist and I'm not just thinking wishfully). The curses bit is fluff that is obviated by going GUI. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From Fernando.Perez at colorado.edu Wed Jul 27 12:50:55 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed, 27 Jul 2005 10:50:55 -0600 Subject: [IPython-dev] Re: Question about IPython behaviour In-Reply-To: References: <1122374421.4682.13.camel@tzanko> <42E72035.20309@colorado.edu> <42E727D6.2020603@colorado.edu> Message-ID: <42E7BB6F.7090500@colorado.edu> Robert Kern wrote: > Fernando Perez wrote: > > >>I talked to M. Hudson (pyrepl's author) ages ago about this: pyrepl >>wasn't suitable for ipython because it's curses-based. That's how they >>get multi-line capabilities. It is certainly a possibility to build a >>curses-enabled version of ipython which could run on top of pyrepl. > > > I'm not so interested in that part of pyrepl. I haven't delved through > the code enough, but it does do multiline editing pretty much as we want > to do in the GUI. To enable that, the component that executes the inputs > probably doesn't use the code.InteractiveConsole line-based API. > *That's* the part we should steal (if indeed, it does exist and I'm not > just thinking wishfully). The curses bit is fluff that is obviated by > going GUI. I was just trying to fill the history in vs ipython/pyrepl. We should certainly see if they have that part built in a reasonably curses-independent way which we can reuse/steal. Cheers, f From antont at an.org Fri Jul 29 10:05:16 2005 From: antont at an.org (Toni Alatalo) Date: Fri, 29 Jul 2005 17:05:16 +0300 Subject: [IPython-dev] small interim info In-Reply-To: References: Message-ID: <42EA379C.4030900@an.org> Toni Alatalo wrote: (got married last weekend :) and had a little holiday after that, discussed related things in private with mentors, but bringing the discussion to public now that got back to work - first just to tell a little where am heading) >read the messages to get myself back on the map, but am not all clear yet >what to tackle next. probably can get going by trying out Tzanko's app and >studying it again to see how it uses Robert's backend, and finding out >what is missing/should be improved there. > > did that today - most of the time unfortunately went into fighting with getting a right version of wxwidgets & wxpython working for the right python version (had to modify the 2.6 in debian experimental to compile for 2.4, as default is still 2.3, and then had a strange prob. with imports but all those are finally solved now) studied how nbshell ipnDocument uses notabene.notebook, and know from before most of how notabene 'writers' work. unfortunately couldn't experiment with using the 'live in the GUI ' notabene.notebook object interactively from the interpreter, i wonder if there's a way? i mean to do something like this: antont at vesuri:~/development/ipython/nbshell/trunk/nbshell$ python2.4 Python 2.4.1 (#2, May 5 2005, 11:32:06) [GCC 3.3.5 (Debian 1:3.3.5-12)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import Main >>> Main.start() (type things in the editor) and then: Main.app.notebook.my_experimental_notebook_method() ? currently if i e.g. quit the wx app, the prompt returns ok, but the interpreter namespace is in a pretty strange state: >>> Main Traceback (most recent call last): File "", line 1, in ? NameError: name 'Main' is not defined >>> import Main Traceback (most recent call last): File "", line 1, in ? ImportError: __import__ not found that is (again) pretty irrelevant, but would be just a nice way to try new things with the document objects without adding buttons to the GUI, but still being able to edit the actual docs in the GUI. anyhow testing the notebook api without the GUI interactively of course works normally, and saving test files from the GUI seems to work, so no showstoppers there. >but if there are suggestions / TODOs for what i should focus on, starting >latest on Monday, please do tell. > > unless adviced otherwise, will continue with the the document output (e.g. latex) components. >~Toni > > same. From tsanko at gmail.com Sun Jul 31 07:06:21 2005 From: tsanko at gmail.com (Tzanko Matev) Date: Sun, 31 Jul 2005 14:06:21 +0300 Subject: [IPython-dev] NBShell update Message-ID: <1122807981.4724.34.camel@tzanko> Hi all, The Notebook GUI can already be usable. Currently you can mainly use it as a simple python shell. It lacks history, autocompletion, etc. Its only advantage is better multiline support. You can edit all the lines of your last input until it is complete. Also I have added a variable __app in the namespace which allows you to play with the internals of NBShell. The svn repository is http://ipython.scipy.org/svn/ipython/nbshell/trunk The instalation instructions are in the README file. I would appreciate it if you played with it and tell me what bugs you find and what features you would like me to add. Best, tzanko