From hans_meine at gmx.net  Wed Jun  1 05:43:30 2005
From: hans_meine at gmx.net (Hans Meine)
Date: Wed, 1 Jun 2005 11:43:30 +0200
Subject: [IPython-dev] IPython and Qt?!
In-Reply-To: <429CB52A.3060807@colorado.edu>
References: <200505311322.54388.hans_meine@gmx.net>
	<200505311450.15353.hans_meine@gmx.net> <429CB52A.3060807@colorado.edu>
Message-ID: <200506011143.31000.hans_meine@gmx.net>

On Tuesday 31 May 2005 21:04, Fernando Perez wrote:
> >>I patched IPython/Shell.py to run a QApplication in
> >>a separate thread, and my first experiments were quite successful.
> [...]
>
> I love it when I get to use my little time machine :)

Quite effective, that one.

> Did you have a look at last night's (my time) announcement of   ipython
> 0.6.14?
Er.. _your time_ ;-)  BTW: I suscribed just before posting, but of course I 
could have had a look in the archives only to find that there would have been 
no announcement because you travelled back in time just after my mail, but 
then I could have caught the difference in the archives later.. ;-))

> The big new feature there is Qt support... 
Great.

> Note that the Qt support in ipython wasn't written by me, but by a group
> of friends who are extremely experienced with Qt.  I think they found
> out, after trying something like your patch, that it wasn't quite
> enough.
I myself was very suprised that my short patch worked so well in my tests, 
but...

> In certain cases, things could still block.
...intuitively expected sth. like this.

> If you read the  
> current Shell.py, you'll see that they play some special tricks with the
> timer to avoid deadlocks.
Will have a look. :-)

> Have a look at .14, and let me know if it doesn't do what you need...
That's what I do ATM (trying .14).

Thanks for the great work.

BTW: In our python shell, filename completion works (only) within strings.  
So, when typing
  loadFile(42, "foo/bar/b<tab>
I will get the filename parameter completed.

I guess when I ask whether you planned sth. like this, too, you'll tell my 
that you released (and announced) .15 just yesterday. ;-)

Have a nice day,
   Hans



From Fernando.Perez at colorado.edu  Wed Jun  1 13:53:52 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Wed, 01 Jun 2005 11:53:52 -0600
Subject: [IPython-dev] IPython and Qt?!
In-Reply-To: <200506011143.31000.hans_meine@gmx.net>
References: <200505311322.54388.hans_meine@gmx.net>
	<200505311450.15353.hans_meine@gmx.net> <429CB52A.3060807@colorado.edu>
	<200506011143.31000.hans_meine@gmx.net>
Message-ID: <429DF630.4050903@colorado.edu>

Hans Meine wrote:
> On Tuesday 31 May 2005 21:04, Fernando Perez wrote:

> BTW: In our python shell, filename completion works (only) within strings.  
> So, when typing
>   loadFile(42, "foo/bar/b<tab>
> I will get the filename parameter completed.
> 
> I guess when I ask whether you planned sth. like this, too, you'll tell my 
> that you released (and announced) .15 just yesterday. ;-)

Something like...

In [1]: loadfile(42,'/etc/hos
/etc/host.conf    /etc/hosts        /etc/hosts.allow  /etc/hosts.deny

In [1]: loadfile(42,"/etc/ho
/etc/host.conf    /etc/hosts.allow  /etc/hotplug      /etc/howl
/etc/hosts        /etc/hosts.deny   /etc/hotplug.d

:)

But no, I can't pull the time-machine on you for this one. 
Tab-completion of filenames within strings had been broken for ages, and 
I just had a quick look, while waiting for the cvs->svn transition.  It 
turned out not to be so hard to fix.  It may not work in all cases 
(filename completion, when trying to account for embedded spaces and 
platform-dependent problems, is a nasty problem full of special cases). 
  But my quick fixes seem to do the job for most cases.

Since CVS will now live for bugfixes, with all new development in the 
soon-to-be svn repo, I've just committed this to CVS.  Feel free to test 
from there, but note that you'll need to change your ipythonrc file to 
read (look for this string):

readline_remove_delims -/~

so that the '" characters are treated as delimiters and tab-completion 
works correctly.

Cheers,

f



From Fernando.Perez at colorado.edu  Thu Jun  2 16:26:28 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu, 02 Jun 2005 14:26:28 -0600
Subject: [IPython-dev] [ANN] IPython 0.6.15 - minor bugfix
Message-ID: <429F6B74.5050505@colorado.edu>

Hi all,

I've just put out a minor bugfix release, 0.6.15.  It really only fixes
one thing, tab-completion inside strings:

In [1]: foo('/etc/hos <<<--- TAB HERE
/etc/host.conf    /etc/hosts        /etc/hosts.allow  /etc/hosts.deny


In [1]: foo("/etc/ini <<<--- TAB HERE
/etc/init.d        /etc/initlog.conf  /etc/inittab

but I wanted it out for a class I'm teaching next week.  If anyone is
interested in this functionality, you'll need to fix
~/.ipython/ipythonrc, so the readline delimiters configuration reads:

readline_remove_delims -/~

(or at least, make sure neither ' nor " are in there).

As usual, downloads at

http://ipython.scipy.org/dist

Cheers,

f



From Fernando.Perez at colorado.edu  Fri Jun  3 21:00:36 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Fri, 03 Jun 2005 19:00:36 -0600
Subject: [IPython-dev] Google's summer of code and IPython
Message-ID: <42A0FD34.3030906@colorado.edu>

Hi all,

if any of you is interested in participating in Google's summer of code 
helping ipython, see more details here:

http://ipython.scipy.org/google_soc

Cheers,

f



From mantegazza at ill.fr  Mon Jun  6 09:55:52 2005
From: mantegazza at ill.fr (=?iso-8859-15?q?Fr=E9d=E9ric_Mantegazza?=)
Date: Mon, 6 Jun 2005 15:55:52 +0200
Subject: [IPython-dev] custom help() function
Message-ID: <200506061555.52284.mantegazza@ill.fr>

Hello Fernando,

Is there a way to set a custom hook on the help() function, as it can be 
done with the '?' syntax ?

I'm still playing with my remote Pyro objects ;o)

Thank's,

-- 
   Fr?d?ric



From Fernando.Perez at colorado.edu  Mon Jun  6 10:19:34 2005
From: Fernando.Perez at colorado.edu (Fernando.Perez at colorado.edu)
Date: Mon, 06 Jun 2005 08:19:34 -0600
Subject: [IPython-dev] custom help() function
In-Reply-To: <200506061555.52284.mantegazza@ill.fr>
References: <200506061555.52284.mantegazza@ill.fr>
Message-ID: <1118067574.42a45b7622ecf@webmail.colorado.edu>

Quoting Fr?d?ric Mantegazza <mantegazza at ill.fr>:

> Hello Fernando,
>
> Is there a way to set a custom hook on the help() function, as it can be
> done with the '?' syntax ?

Well, you could just write your own help() which checks for special objects and
does something funky with them, and otherwise delegates to the native help
(which lives in pydoc, by the way).  I don't think pydoc.help() has
customization capabilities, but I'm not 100% sure.

Best,

f



From shlomme at gmx.net  Mon Jun  6 15:44:05 2005
From: shlomme at gmx.net (Torsten Marek)
Date: Mon, 06 Jun 2005 21:44:05 +0200
Subject: [IPython-dev] Notebook interface for IPython
Message-ID: <42A4A785.4030602@gmx.net>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello Fernando,

I'd like to work the the notebook interface for IPython (if there's not yet been
anybody else).

I've read your project description document, and I have some remarks, for
everything else I need some quiet hours:

@GUI: I'd really like to see IPython having a GUI interface at all. If it were
only code, using Scintilla (like PyCrust) will be the best choice, but I think
embedding pictures and other content will require changes in the codebase of
Scintilla, and therefore in wx. I'm not sure how likely it is that the changes
are included into the actual codebase, and forking makes it very uncomfortable
for the users.
For Gtk, there seems to be a simple text widget only, and using GtkHTML might
not be the best idea. A canvas widget exists, but is GNOME only.
For wx, there is a canvas widget and several text widgets. I do not know if any
of them support pictures out of the box.
Anyhow, creating a (possibly limited) prototype should be possible

@File format:
What about magic functions? If they are used, the file is not valid Python any
more, unless they are handled separately from normal Python code (which should
be feasible).
What if pictures are inserted? Should the be encoded and put into the Python
file? I think a zip archive with a Python file + pictures would be the better
choice there.


Moreover, I would be both interested in making the notebook interface better and
helping out with the development of IPython in general, as my time allows.


Regarding the application, I assume that you are the mentor and PSF is the
mentoring organization?


greetings

Torsten
- --
Torsten Marek <shlomme at gmx.net>
ID: A244C858 -- FP: 1902 0002 5DFC 856B F146  894C 7CC5 451E A244 C858
Keyserver: subkeys.pgp.net

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFCpKb8fMVFHqJEyFgRAsP/AKCVKg2EzpwQSePMHchyH0tw0mEhpACgqA8q
AT5BrASS/EKaasNw7yXaXYI=
=ue0m
-----END PGP SIGNATURE-----



From shlomme at gmx.net  Mon Jun  6 15:41:49 2005
From: shlomme at gmx.net (Torsten Marek)
Date: Mon, 06 Jun 2005 21:41:49 +0200
Subject: [IPython-dev] Notebook interface for IPython
Message-ID: <42A4A6FD.1090105@gmx.net>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello Fernando,

I'd like to work the the notebook interface for IPython (if there's not yet been
anybody else).

I've read your project description document, and I have some remarks, for
everything else I need some quiet hours:

@GUI: I'd really like to see IPython having a GUI interface at all. If it were
only code, using Scintilla (like PyCrust) will be the best choice, but I think
embedding pictures and other content will require changes in the codebase of
Scintilla, and therefore in wx. I'm not sure how likely it is that the changes
are included into the actual codebase, and forking makes it very uncomfortable
for the users.
For Gtk, there seems to be a simple text widget only, and using GtkHTML might
not be the best idea. A canvas widget exists, but is GNOME only.
For wx, there is a canvas widget and several text widgets. I do not know if any
of them support pictures out of the box.
Anyhow, creating a (possibly limited) prototype should be possible

@File format:
What about magic functions? If they are used, the file is not valid Python any
more, unless they are handled separately from normal Python code (which should
be feasible).
What if pictures are inserted? Should the be encoded and put into the Python
file? I think a zip archive with a Python file + pictures would be the better
choice there.


Moreover, I would be both interested in making the notebook interface better and
helping out with the development of IPython in general, as my time allows.


Regarding the application, I assume that you are the mentor and PSF is the
mentoring organization?


greetings

Torsten
- --
Torsten Marek <shlomme at gmx.net>
ID: A244C858 -- FP: 1902 0002 5DFC 856B F146  894C 7CC5 451E A244 C858
Keyserver: subkeys.pgp.net

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFCpKb8fMVFHqJEyFgRAsP/AKCVKg2EzpwQSePMHchyH0tw0mEhpACgqA8q
AT5BrASS/EKaasNw7yXaXYI=
=ue0m
-----END PGP SIGNATURE-----



From Fernando.Perez at colorado.edu  Mon Jun  6 16:03:24 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Mon, 06 Jun 2005 14:03:24 -0600
Subject: [IPython-dev] Notebook interface for IPython
In-Reply-To: <42A4A785.4030602@gmx.net>
References: <42A4A785.4030602@gmx.net>
Message-ID: <42A4AC0C.4090405@colorado.edu>

Torsten Marek wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Hello Fernando,
> 
> I'd like to work the the notebook interface for IPython (if there's not yet been
> anybody else).

Great.  A few others have expressed interest, but that shouldn't stop 
you from applying.  I'm not the one making the decisions, so as far as 
I'm concerned, the more the merrier :)

> I've read your project description document, and I have some remarks, for
> everything else I need some quiet hours:

[...]

just a note: a few hours ago I updated the description significantly, in 
  good part to address the GUI issues.  Please read the new one and 
let's re-discuss things based on that.

> @File format:
> What about magic functions? If they are used, the file is not valid Python any
> more, unless they are handled separately from normal Python code (which should
> be feasible).

magics are called as ipmagic('foo') already, so the magic calls are 
already valid python code.  You can try logstart foo.py in ipython and 
then look at the log file (foo.py) to see what ipython generates internally.

> What if pictures are inserted? Should the be encoded and put into the Python
> file? I think a zip archive with a Python file + pictures would be the better
> choice there.

The new PDF gives some examples of this.  Obviously, all is open to 
discussion at this point, and a good part of the project will actually 
involve thinking about these problems.

> Moreover, I would be both interested in making the notebook interface better and
> helping out with the development of IPython in general, as my time allows.

Great, welcome aboard!

> Regarding the application, I assume that you are the mentor and PSF is the
> mentoring organization?

Yup.  Use the PSF radio button on the application, and mention in the 
description that I'll be mentoring students.  Don't forget to link to 
the project PDF, which is the key document for reviewers.

Best,

f



From mantegazza at ill.fr  Tue Jun  7 02:12:13 2005
From: mantegazza at ill.fr (=?iso-8859-15?q?Fr=E9d=E9ric_Mantegazza?=)
Date: Tue, 7 Jun 2005 08:12:13 +0200
Subject: [IPython-dev] custom help() function
In-Reply-To: <1118067574.42a45b7622ecf@webmail.colorado.edu>
References: <200506061555.52284.mantegazza@ill.fr>
	<1118067574.42a45b7622ecf@webmail.colorado.edu>
Message-ID: <200506070812.13466.mantegazza@ill.fr>

Le Lundi 6 Juin 2005 16:19, Fernando.Perez at colorado.edu a ?crit :

> Well, you could just write your own help() which checks for special
> objects and does something funky with them, and otherwise delegates to
> the native help (which lives in pydoc, by the way).  I don't think
> pydoc.help() has customization capabilities, but I'm not 100% sure.

Ok, I'll try to customize pydoc.help() first. If it is not possible, I'm 
make a magic help function.

Thank's,

-- 
   Fr?d?ric



From Fernando.Perez at colorado.edu  Tue Jun 21 13:58:47 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Tue, 21 Jun 2005 11:58:47 -0600
Subject: [IPython-dev] ipython servers will be down
Message-ID: <42B85557.3050102@colorado.edu>

Hi all,

the scipy.org website is undergoing an upgrade, which will result in 
some temporary disruptions.  Below is the email from the scipy admin. 
Once this is completed, we'll have SVN for ipython :)

After letting things settle for a few days, let me know if any problems 
persist.

best,

f

=======================================
Hi all,

I've been threatening a server migration of SciPy.org for a couple of
weeks now, and tonight's the night on which at least some services will
move (whether they like it or not).

I will be shutting down CVS access in a few minutes, as the cvs2svn
conversion takes a while to process and I will definitely move SciPy to
Subversion tonight.  Throughout the evening, other services will
probably disappear for a bit, hopefully to reappear on the new server
soon after.

I'm certain there will be issues, as we're moving to much newer versions
of all software, removing a bunch of old users, switching from CVS to
Subversion, and a couple dozen other changes that don't come to mind
right now.  Please let me know if you see any such problems in the
coming days.

I'll notify the list when things are back to sanity as best as I can figure.

Thanks!



From Fernando.Perez at colorado.edu  Tue Jun 21 14:06:20 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Tue, 21 Jun 2005 12:06:20 -0600
Subject: [IPython-dev] Re: Auto-discard notification
In-Reply-To: <mailman.4893.1119290253.917.ipython-dev@scipy.net>
References: <mailman.4893.1119290253.917.ipython-dev@scipy.net>
Message-ID: <42B8571C.7050405@colorado.edu>

Hi,

note that your mail bounced, since the ipython lists require 
subscription first.  Too much spam otherwise, sorry.

ipython-dev-bounces at scipy.net wrote:

> Subject:  Implanting IPython with PyCrust
> From:  axelyamel at prtc.net
> Date: Mon, 20 Jun 2005 15:42:55 -0400
> To: ipython-dev at scipy.net
> 
> To: ipython-dev at scipy.net
> 
> 
> Hi,
> 
>     I being working on the implementation of IPython with Pycrust. When I try to
> import the embed file that I made to run the IPython, it give me an error.
> 
> 
>      >>> import embed.py
>      WARNING: Stream <cin> looks suspicious: it lacks a 'fileno' attribute.
>      WARNING: Stream <cin> looks suspicious: it lacks a 'mode' attribute.
>      WARNING: Stream <cout> looks suspicious: it lacks a 'fileno' attribute.
>      WARNING: Stream <cout> looks suspicious: it lacks a 'mode' attribute.
>      WARNING: Stream <cerr> looks suspicious: it lacks a 'fileno' attribute.
>      WARNING: Stream <cerr> looks suspicious: it lacks a 'mode' attribute.
>      Traceback (most recent call last):
>       File "<input>", line 1, in ?
>       File "embed.py", line 18, in ?
>         ipshell = IPShellEmbed([argv])
>       NameError: name 'argv' is not defined
> 
> 
>  The embed file have the following code:
> 
> 
>      from IPython.Shell import IPShellEmbed
>      ipshell = IPShellEmbed([argv])
>      ipshell()
> 
>      print 'Main program finished. Bye!'
> 
>  If someone know what is happening please let me know, thanks.
> 
> 
> ----------------------------------------------------------------
> This message was sent using http://webmail.coqui.net

IPython can't currently be easily embedded into any GUI shell.  The 
needed work is not too hard, but it hasn't been done yet.  So unless you 
can contribute patches in that direction, it won't work for you in its 
current state.

Regards,

f



From mantegazza at ill.fr  Mon Jun 27 02:44:57 2005
From: mantegazza at ill.fr (=?iso-8859-15?q?Fr=E9d=E9ric_Mantegazza?=)
Date: Mon, 27 Jun 2005 08:44:57 +0200
Subject: [IPython-dev] Redirecting ipython in and out
Message-ID: <200506270844.57215.mantegazza@ill.fr>

Hello Fernando,

I would like to know if it is possible to redirect ipython stdin and 
stdout ? Here is what I would be able to do:

You remember that our application embedds a ipython instance as a user 
input/output console. As it is a client connected to a server, we can 
launch several ipython instances. When a command is sent, the result is 
printed by a Pyro mecanism called Event Server on all connected clients. 
This is fine because they can see what is executed by others.

But I would like to go further: I would also like to see what other clients 
*send* to the server. I mean that when a client enter the command:

In [15]: A1.drive(10)

after the return key is pressed all clients should print :

In [xx]: A1.drive(10)

But the command should be executed only on the ipython instance where user 
entered it. Others should act as passive displays.

Do you think this is possible ? Is there a hook to do this, or do I have to 
hack a little bit the code ?

Thank's,

PS : I'm running ipython 0.6.15.

-- 
   Fr?d?ric



From antont at an.org  Mon Jun 27 09:38:31 2005
From: antont at an.org (Toni Alatalo)
Date: Mon, 27 Jun 2005 16:38:31 +0300
Subject: [IPython-dev] questions about the notebook interface
Message-ID: <42C00157.2020808@an.org>

Greetings.

i have some questions about the notebook interface (as it's described in 
http://ipython.scipy.org/google_soc/ipnb_google_soc.pdf). must leave 
right now so am trying to be brief, sorry for possible unclarities:

1. reasoning for true Python over special string / comments

as e.g. rst is nice for writing documents in itself, and can also be 
used to write text, make tables, images etc. in Python files (the 
original purpose), i'd be interested in hearing the main arguments from 
the 'lengthy discussions' (referred in the spec) for deciding on true 
Python instead. am probably bying the result already, but would still 
like to hear the reasoning.

2. usage targets of the library

ReST and LaTEX are pretty clear, but how about possible other types of 
documents, like ones with dynamic / interactive content -- should that 
be addressed by the notebook lib? e.g. Slithy 
(http://isotropic.org/uw/slithy/) provides a pretty nice way to make 
dynamic/interactive diagrams with python (as functions that receive vars 
(user controllable with slides in the player, and/or time for 
animations) and output drawings). what (other) usages should be 
considered? or only 'print targeted' stuff (like the mathreader things 
seem almost like pdf)? one other area here is linking, which 
rest/latex/pdf and of course html do support too, and there perhaps 
granularity .. like to have support for so-called 'purple numbers' 
http://www.eekim.com/software/purple/purple.html .
 
3. markup

is something like Stan from twisted.nevow considered nice? 
http://nevow.com/ , http://nevowexamples.adytum.us/sources/hellostan.py

4. ipython and non-ipython

i suppose the basic notebook module would be independent from ipython, 
and then some separate glue for it would be made for ipython (in the 
ipython cvs), like has been done with gnuplot - right? or is there 
something ipython specific about the project i'm missing? indeed the 
real-time / interactive aspect is interesting, but does that imply much 
for the API design? and in this case, should be a separate project, 
hosted on some of the sourceforge forks (berlios, gna.org or something?) 
.. would 'notebook' be a good name for the package, and pynotebook for 
the project, or what? or is it more tied to ipython and should start there?

i guess those were the main things on top of my mind. and yes for those 
who are wondering -- i received a confirmation message from Google last 
saturday and was .. and am really happy about it :) .. in case someone's 
curious my application was http://an.org/programming/ipynote.html

yours,
~Toni




From rkern at ucsd.edu  Mon Jun 27 22:38:13 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Mon, 27 Jun 2005 19:38:13 -0700
Subject: [IPython-dev] Re: questions about the notebook interface
In-Reply-To: <42C00157.2020808@an.org>
References: <42C00157.2020808@an.org>
Message-ID: <d9qcpe$7as$1@sea.gmane.org>

Toni Alatalo wrote:
> Greetings.
> 
> i have some questions about the notebook interface (as it's described in 
> http://ipython.scipy.org/google_soc/ipnb_google_soc.pdf). must leave 
> right now so am trying to be brief, sorry for possible unclarities:

I've been thinking a bit about this too, since I was going to volunteer 
some work on this if no one got picked for it by Google. Looks like that 
won't be necessary. Congratulations!

> 1. reasoning for true Python over special string / comments
> 
> as e.g. rst is nice for writing documents in itself, and can also be 
> used to write text, make tables, images etc. in Python files (the 
> original purpose), i'd be interested in hearing the main arguments from 
> the 'lengthy discussions' (referred in the spec) for deciding on true 
> Python instead. am probably bying the result already, but would still 
> like to hear the reasoning.

I personally don't agree with it. I wasn't a party to those discussions, 
so I'm sure most of my complaints have already been addressed, but I'm 
going to air them out here anyways. Because I can. :-)

The foremost use cases in my head are the essentially interactive or 
demonstrative ones, not the literate programming ones. I think that 
people are going to use notebooks to run interactively or generate 
static, pretty documents. They are going to use the exported code to 
import or run on a server or whatever other "headless" activity. If you 
really want to be able to import from a notebook, you could write an 
import hook that will export the bare code and import that. (Disclosure: 
I never met a literate programming system that I liked.)

The markup-via-Python seems to me to be difficult to parse without 
running the code. Every time I make a markup change or spelling 
correction, my excessively long calculation will run again just to 
recreate a plot that I've already embedded in the notebook. I should be 
able to selectively run the code when I need to (re)generate output.

Name collisions. Or else evil hacks to avoid name collisions.

You can't embed these calls in functions because they won't get called 
until the function is run, not when that line of code is executed. And 
then they get called every time that function gets called.

I don't think that any-old-text-editor is a viable target. It just 
doesn't provide any of the real notebook capabilities. Targetting it as 
an intermediate platform before a GUI shell is available sacrifices the 
capabilities that the GUI shell is going to provide.

Allow me to make a suggestion for an intermediate target: Modify ipython 
to make a notebook file (in whatever format you go forward with) as a 
kind of log. At minimum, it just writes out the In[NN]'s and Out[NN]'s 
in the notebook format. Add a %magic to write out notes to the notebook. 
Nothing fancy or formatted, they'd mostly be just reminders that you 
would expand and format later, not the final markup. If you're on a 
roll, add another %magic for pylab mode that will save the current 
figure as a .png and write out an entry to the notebook for including an 
image.

If the format is mildly human editable (I'm thinking something on the 
level of basic HTML as my standard for human editability here), then a 
basic text editor could be used to expand those reminders, add markup, 
add titles, and that whole jazz. Now that I think about it, you will 
want a script that can read a notebook file, rerun all of the In[NN]'s 
to (re)generate their Out[NN]'s.

The next step would be to write a wxPython (or other) editor that knows 
the structure of the notebook file and can do real WYSIWY{G,M} rich text 
editing and syntax coloration and image embedding, but no running of code.

The final step would be to implement the shell parts. Every step along 
the way is a smooth, evolutionary transition.

And now that I've convinced you that all I'm off my rocker, I'll also 
add that XML might actually be a reasonable choice for the meta-format 
of the notebook file. But since that choice assumes that you've agreed 
with everything else I've said, I'll let my explanation wait until 
people give me feedback on that.

> 2. usage targets of the library
> 
> ReST and LaTEX are pretty clear, but how about possible other types of 
> documents, like ones with dynamic / interactive content -- should that 
> be addressed by the notebook lib? e.g. Slithy 
> (http://isotropic.org/uw/slithy/) provides a pretty nice way to make 
> dynamic/interactive diagrams with python (as functions that receive vars 
> (user controllable with slides in the player, and/or time for 
> animations) and output drawings). what (other) usages should be 
> considered? or only 'print targeted' stuff (like the mathreader things 
> seem almost like pdf)?

For this summer, just assume that anything that wants to do something 
funky will create a context in which to do it. Leave how that happens up 
to the people writing the GUI. Allow things to fail gracefully when it 
doesn't make sense (e.g. Pygame animations in HTML).

> one other area here is linking, which 
> rest/latex/pdf and of course html do support too, and there perhaps 
> granularity .. like to have support for so-called 'purple numbers' 
> http://www.eekim.com/software/purple/purple.html .

Intra-notebook linking gets a big +1 from me. As an offhand thought, 
every time In[NN] or Out[NN] is used in an expression, it should get a 
link back to the original when the expression is being syntax colored.

> 3. markup
> 
> is something like Stan from twisted.nevow considered nice? 
> http://nevow.com/ , http://nevowexamples.adytum.us/sources/hellostan.py

Yes. Or any of the other umpteen syntax colorers.

> 4. ipython and non-ipython
> 
> i suppose the basic notebook module would be independent from ipython, 
> and then some separate glue for it would be made for ipython (in the 
> ipython cvs), like has been done with gnuplot - right? or is there 
> something ipython specific about the project i'm missing? indeed the 
> real-time / interactive aspect is interesting, but does that imply much 
> for the API design? and in this case, should be a separate project, 
> hosted on some of the sourceforge forks (berlios, gna.org or something?) 
> .. would 'notebook' be a good name for the package, and pynotebook for 
> the project, or what? or is it more tied to ipython and should start there?

Development should probably be done on the scipy.org server. Hopefully 
the SVN server will be up soon for ipython. Most of the development will 
probably happen outside of the ipython tree itself, I imagine.

I recommend against pynotebook. It's redundant and duplicative.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From jdhunter at ace.bsd.uchicago.edu  Mon Jun 27 23:18:53 2005
From: jdhunter at ace.bsd.uchicago.edu (John Hunter)
Date: Mon, 27 Jun 2005 22:18:53 -0500
Subject: [IPython-dev] Re: questions about the notebook interface
In-Reply-To: <d9qcpe$7as$1@sea.gmane.org> (Robert Kern's message of "Mon, 27
	Jun 2005 19:38:13 -0700")
References: <42C00157.2020808@an.org> <d9qcpe$7as$1@sea.gmane.org>
Message-ID: <87aclbrwwy.fsf@peds-pc311.bsd.uchicago.edu>

>>>>> "Robert" == Robert Kern <rkern at ucsd.edu> writes:

    Robert> And now that I've convinced you that all I'm off my
    Robert> rocker, I'll also add that XML might actually be a
    Robert> reasonable choice for the meta-format of the notebook
    Robert> file. But since that choice assumes that you've agreed

What are the advantages of XML over python as a document format?
Typically people point to extensibility, parsability and widespread
utilization.  Python is clearly extensible and widely utilized; with
access to the interpreter's internal AST representation you also have
built-in access to the parse tree if you need it.  Th idea is that
everyone writing python pretty much knows python and already has a
decent python editor and so can readily edit the notebook in their
favorite editor; the same is not true for python coders and an XML
format.  If pure python can offer most or all of what XML does as a
markup language, what do we gain by using XML?  "Say no to toy
languages" and all that.  [ Truth in advertising: I've never used the
python parser module so I can't really speak to how suitable it is in
this context ]

That is one of my arguments in favor of the "python-as-format" option.
There are a couple of others, but lest I embarrass myself by
disagreeing with Robert, I think it best if I collect my thoughts for
a while.

JDH



From rkern at ucsd.edu  Tue Jun 28 00:24:02 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Mon, 27 Jun 2005 21:24:02 -0700
Subject: [IPython-dev] Re: questions about the notebook interface
In-Reply-To: <87aclbrwwy.fsf@peds-pc311.bsd.uchicago.edu>
References: <42C00157.2020808@an.org> <d9qcpe$7as$1@sea.gmane.org>
	<87aclbrwwy.fsf@peds-pc311.bsd.uchicago.edu>
Message-ID: <d9qivp$ifo$1@sea.gmane.org>

John Hunter wrote:
>>>>>>"Robert" == Robert Kern <rkern at ucsd.edu> writes:
> 
>     Robert> And now that I've convinced you that all I'm off my
>     Robert> rocker, I'll also add that XML might actually be a
>     Robert> reasonable choice for the meta-format of the notebook
>     Robert> file. But since that choice assumes that you've agreed
> 
> What are the advantages of XML over python as a document format?
> Typically people point to extensibility, parsability and widespread
> utilization.  Python is clearly extensible and widely utilized; with
> access to the interpreter's internal AST representation you also have
> built-in access to the parse tree if you need it.  Th idea is that
> everyone writing python pretty much knows python and already has a
> decent python editor and so can readily edit the notebook in their
> favorite editor; the same is not true for python coders and an XML
> format.  If pure python can offer most or all of what XML does as a
> markup language, what do we gain by using XML?  "Say no to toy
> languages" and all that.  [ Truth in advertising: I've never used the
> python parser module so I can't really speak to how suitable it is in
> this context ]

Precisely. The AST has been tuned for working with code, not documents, 
and isn't very friendly even for that.

In [327]: compiler.parse("""from notebook.markup import rest
    .....: rest.title('This is a Python Notebook')
    .....: rest.heading(1,'A first-level heading')
    .....: def f(x, y):
    .....:     print y-x
    .....: """)
Out[327]: Module(None, Stmt([From('notebook.markup', [('rest', None)]), 
Discard(CallFunc(Getattr(Name('rest'), 'title'), [Const('This is a 
Python Notebook')], None, None)), Discard(CallFunc(Getattr(Name('rest'), 
'heading'), [Const(1), Const('A first-level heading')], None, None)), 
Function(None, 'f', ['x', 'y'], [], 0, None, 
Stmt([Printnl([Sub((Name('y'), Name('x')))], None)]))]))

There's compile.walk() to visit each node of the AST, but that's not a 
tool to process anything; it's a tool to build a tool. Fredrik Lundh's 
ElementTree really simplifies XML processing and very nearly makes XML 
Pythonic. In my experience it has raised XML from "Ew! Avoid at all 
costs!" to "It actually beats anything else I can come up with in as 
short a time." And with cElementTree, it's even going to be faster than 
anything else I'd write. ElementTree is really an in-memory 
representation of a document. It just happens to read from and write to 
disk as XML. If it makes you feel better, we can call it "ElementTree 
format" instead of XML.  :-)

A problem that you are going to come across is differentiating between 
Python code and notebook markup. On the one hand, you're going to have 
to deal with namespace collisions in the Python code. You're either 
going to have to enforce an arbitrary restriction on names, or you're 
going to have to choose unwieldly names for the notebook markup objects.

On the other hand, when people are writing Python, they are going to 
code Python. With Python markup, people are going to do things like 
string interpolation and computed values because they can. And while 
that might be a nice ability to have (although I'd have to be convinced 
why it's necessary above and beyond the recording of Out[NN]'s or 
explicit outputs like the %magics I mentioned), it eliminates the 
ability to process the notebook via passive parsing, which I think is 
crucial. You'd have to execute it.

I consider using a plain-old-editor to be a temporary transition stage 
in the project and a quick-and-dirty-need-to-change-something-quickly 
even when the project matures. I think that people have enough 
experience with HTML in this day and age to do these temporary/simple 
tasks. I don't think that we should cater to this stage of development 
at the expense of other qualities that should be in the final product. 
But I may be underestimating the significance of the plain-old-editor 
method. That said, I think the other technical issues are dispositive.

Python markup doesn't quite solve the experience problem. People know 
Python, but the notebook markup isn't actually Python. It's a subset of 
Python syntax being interleaved with real Python in semantically strange 
ways. Once they get past that, they still need to learn the details of 
the markup pseudo-API. Given that they're forgoing the shiny GUI for a 
plain-old-editor, I think it's likely they know enough HTML to do what 
they need to do inside an XML file.

I think that a set of well-crafted example files would go a long way 
towards enabling people to hand-edit the notebook files for their needs.

I don't think you can "Say no to toy languages" here. You need to build 
a toy language regardless. I think the tools for building toy languages 
heavily favor XML and not Python.

Oh yeah, links.

> That is one of my arguments in favor of the "python-as-format" option.
> There are a couple of others, but lest I embarrass myself by
> disagreeing with Robert, I think it best if I collect my thoughts for
> a while.

Don't mistake my long-windedness for knowledgability. I'm just throwing 
stuff out here.  :-)

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From antont at an.org  Tue Jun 28 11:36:40 2005
From: antont at an.org (Toni Alatalo)
Date: Tue, 28 Jun 2005 18:36:40 +0300
Subject: [IPython-dev] questions about the notebook interface
In-Reply-To: <42C055DF.9040505@colorado.edu>
References: <42C00157.2020808@an.org> <42C055DF.9040505@colorado.edu>
Message-ID: <42C16E88.901@an.org>

Fernando Perez wrote:

(btw just noticed that the message i'm replying to did not go to the 
list, at least i can't see it in the web archives - i suppose that was 
in error?)

hi again and thank you all of you for the comments - quite a lot to 
digest, and i actually had some other things to do today too .. am 
throwing some more stuff in still based on the notes from the morning, 
hopefully they open up some perspectives. oh and the answers were quite 
helpful, many things are clearer now.

>> as e.g. rst is nice for writing documents in itself, and can also be 
>> used to write text, make tables, images etc. in Python files (the 
>> original purpose), i'd be interested in hearing the main 
>
> There was also the possibility of leaving the top-level syntax as 
> Python, but embedding the markup in specially-tagged comments or 
> triple-quoted strings. Eventually, after discussing this approach at 
> length with John Hunter (matplotlib), I was convinced that using the 
> Python language itself would be the cleanest approach.  If we start 
> embedding markup in strings, we'll need to  write our own parsing, 
> processing, etc.  It's just cleaner to simply expose that as true 
> python objects and be done 

hm, except if we just used rst, 'cause docutils can already parse that 
in .py files - right?

> with it.  Python is a very powerful programming language, why not use 
> it to represent the structure of the document?

right.
(dismissing now the point that the code must be executed to have the doc 
;) ((btw i guess it's well possible to cache objects, like the results 
of complex calculations))

a couple of things about that:

> abstract representation of the structure of a document, in terms of 
> calls to things like
> markup.title('my doc title')
> markup.heading('first heading')
> markup.text('some regular text...')

will we have an own abstraction, notebook.title, notebook.heading etc., 
like docutils/rst has, and then different writers for pdf, html etc. 
(again like docutils has)? or is it better to use some of the existing 
'document objects models', like the one that rst, latex, html or 
something uses? (like your example with rest.title and latex.title would 
indicate). and if using rest and/or latex directly like that, are they 
mutually exclusive, i.e. does it mean that one notebook can contain only 
one kind of markup? 'cause doing rest.title() and latex.section() in the 
same doc would be quite strange..

hm perhaps your following text already answered that, but i'm not yet 
sure how.

> If the user simply sticks to plain text everywhere, the documents can 
> render more or less across backends.  But users are free to pass 
> things like
> markup.text('and now let us define $f(x)=x+1$ ...')
> which will obviously not render as math in anything but a latex-based 
> backend.
> Something like this gives a generic object abstraction, along with a 
> mechanism to access the low-level capabilities of a specific backend.  

(...)

>> 3. markup
>> http://nevowexamples.adytum.us/sources/hellostan.py
>
> I don't see how in this approach one can have top-level code woven 
> inside the page template, but perhaps it's possible.  I'd encourage 
> you to play with it a bit, and let me know how it feels for our target 
> problem.

ok i might explore it - but here is something perhaps interesting in a 
different direction, Python notebooks the (meta)class way .. perhaps 
this is also a comment to how writing Python can be more powerful than 
writing xml..? :

---begin---

from notebook import Section, Image, Reference #like there was the own 
abstraction
from notebook.constants import *

class introduction(Section): #a singleton class, i.e. a section called 
"introduction"
    + "This is the first paragraph in the introduction section."

    + "Although this may seem quite peculiar, it's still pure 
Python!(?)" #2nd paragraph

    + "This way to use metaclasses for compact descriptions of content 
items as singletons, featuring 'operator overloading' etc., is taken 
directly from mr. de Ridder's Blendit, see" + 
Reference("http://www.blender3d.org/cms/Theo_de_Ridder.459.0.html") #end 
of 3rd paragraph so far

background = Section() #the same without (meta)class trickery
background += "Now this is just another section, created in more typical 
Python fashion."
background += Image("smiley")

class forfun(Image): #and back to classes (blendit uses metaclasses for 
these somehow)
    src = "drawing.svg"
    align = CENTER #a notebook.constant
    size = 0.50

introduction += forfun

---end---

that's of course totally non-tested, not too much thought over, and not 
commented by anyone before..

> Yes, the markup project should be mostly independent of ipython.  Much 
> like matplotlib, which can be used non-interactively on its own, or 
> interactively with ipython.  This means that initially, Toni's project 
> can live on its own, while Tsanko will need to work much more closely 
> with me on 

ok.

i'll start on this machine here, and commit things to the repository 
once it's set up.

> I've thought that XEmacs could be the first environment to target for 
> interactive use, since it already almost has everything we need in its 
> python-mode/ipython combination.  In the meantime, Tsanko can work on 
> the more ambitious GUI-oriented project he applied for.

fine with me, as emacs is what i always use for programming otherwise too.

.. looking forward to hearing the GUI ideas too

> Fernando

~Toni



From Fernando.Perez at colorado.edu  Tue Jun 28 12:12:48 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Tue, 28 Jun 2005 10:12:48 -0600
Subject: [IPython-dev] Redirecting ipython in and out
In-Reply-To: <200506270844.57215.mantegazza@ill.fr>
References: <200506270844.57215.mantegazza@ill.fr>
Message-ID: <42C17700.8020107@colorado.edu>

Fr?d?ric Mantegazza wrote:
> Hello Fernando,
> 
> I would like to know if it is possible to redirect ipython stdin and 
> stdout ? Here is what I would be able to do:
> 
> You remember that our application embedds a ipython instance as a user 
> input/output console. As it is a client connected to a server, we can 
> launch several ipython instances. When a command is sent, the result is 
> printed by a Pyro mecanism called Event Server on all connected clients. 
> This is fine because they can see what is executed by others.
> 
> But I would like to go further: I would also like to see what other clients 
> *send* to the server. I mean that when a client enter the command:
> 
> In [15]: A1.drive(10)
> 
> after the return key is pressed all clients should print :
> 
> In [xx]: A1.drive(10)
> 
> But the command should be executed only on the ipython instance where user 
> entered it. Others should act as passive displays.
> 
> Do you think this is possible ? Is there a hook to do this, or do I have to 
> hack a little bit the code ?

No, unfortunately there's no simple way to do this at the moment.  While I've 
tried to clean up stdin/out assumptions in ipython, this is one of the major 
stumbling blocks preventing it from being easily embeddable in a GUI shell. 
It still makes too many assumptions about stdin/out being the default ones, 
connected to a TTY.

But there may be light at the end of the tunnel: since I can never find enough 
time for deep ipython work, I went the google route and got lucky.  Two 
students will be working this summer on the ipython/notebooks project, and one 
of them specifically proposed to work on the GUI aspects.  This will 
inevitably require finishing up the stdin/out abstraction work, so if this 
effort is successful, we'll have fairly significant improvements by summer's 
end.

Let's wait and see how it all unfolds...

Best,

f



From mantegazza at ill.fr  Wed Jun 29 02:20:16 2005
From: mantegazza at ill.fr (=?iso-8859-15?q?Fr=E9d=E9ric_Mantegazza?=)
Date: Wed, 29 Jun 2005 08:20:16 +0200
Subject: [IPython-dev] Redirecting ipython in and out
In-Reply-To: <42C17700.8020107@colorado.edu>
References: <200506270844.57215.mantegazza@ill.fr>
	<42C17700.8020107@colorado.edu>
Message-ID: <200506290820.16493.mantegazza@ill.fr>

Le Mardi 28 Juin 2005 18:12, Fernando Perez a ?crit :

> But there may be light at the end of the tunnel: since I can never find
> enough time for deep ipython work, I went the google route and got lucky.
>  Two students will be working this summer on the ipython/notebooks
> project, and one of them specifically proposed to work on the GUI
> aspects.  This will inevitably require finishing up the stdin/out
> abstraction work, so if this effort is successful, we'll have fairly
> significant improvements by summer's end.
>
> Let's wait and see how it all unfolds...

Good news ! Hope this guys are on this list...

-- 
   Fr?d?ric



From antont at an.org  Wed Jun 29 11:52:02 2005
From: antont at an.org (Toni Alatalo)
Date: Wed, 29 Jun 2005 18:52:02 +0300
Subject: [IPython-dev] questions about the notebook interface
In-Reply-To: <42C16E88.901@an.org>
References: <42C00157.2020808@an.org> <42C055DF.9040505@colorado.edu>
	<42C16E88.901@an.org>
Message-ID: <42C2C3A2.8040303@an.org>

Toni Alatalo wrote:

this is not about ipython anymore, so i guess should move the discussion 
somewhere else .. an own list for the notebook thing perhaps (or how 
close is this work to docutils, in fact?)? but i hope you don't mind 
this so little (off-topic) update follows:

> will we have an own abstraction, notebook.title, notebook.heading 
> etc., like docutils/rst has, and 

(...)

> background = Section() #the same without (meta)class trickery
> background += "Now this is just another section, created in more 
> typical Python fashion."

spent most of the day digging around docutils, and quite like the things 
that saw. incidentally, it already has an interface that is almost 
exactly like that (i did read parts of docutils earlier too for other 
reasons, but don't think saw those areas).

managed finally to pull together a very crude first working version of a 
python file that builds a document (a docutils document object) and 
outputs it, now as html but could as well use the latex etc. writers 
(some xml seemed to be the default string representation). feel free to 
see http://an.org/programming/notebook/rest.py and if don't care to run 
it the result is http://an.org/programming/notebook/result.html (there 
is no stylesheet so the paragraph does not render quite correctly).

will (perhaps tomorrow) continue from there to have a nice syntax, 
notebook.rest wrapping those docutils details .. perhaps in the way that 
was drafted in that message yesterday -- suggestions and any comments 
welcome.

> ~Toni

same.



From rkern at ucsd.edu  Wed Jun 29 12:07:24 2005
From: rkern at ucsd.edu (Robert Kern)
Date: Wed, 29 Jun 2005 09:07:24 -0700
Subject: [IPython-dev] Re: questions about the notebook interface
In-Reply-To: <42C2C3A2.8040303@an.org>
References: <42C00157.2020808@an.org> <42C055DF.9040505@colorado.edu>
	<42C16E88.901@an.org> <42C2C3A2.8040303@an.org>
Message-ID: <d9ugi5$404$1@sea.gmane.org>

Toni Alatalo wrote:
> Toni Alatalo wrote:
> 
> this is not about ipython anymore, so i guess should move the discussion 
> somewhere else .. an own list for the notebook thing perhaps (or how 
> close is this work to docutils, in fact?)? but i hope you don't mind 
> this so little (off-topic) update follows:

This is the right place. It's not like we're flooded with traffic.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter



From Fernando.Perez at colorado.edu  Wed Jun 29 12:29:29 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Wed, 29 Jun 2005 10:29:29 -0600
Subject: [IPython-dev] Re: questions about the notebook interface
In-Reply-To: <d9ugi5$404$1@sea.gmane.org>
References: <42C00157.2020808@an.org> <42C055DF.9040505@colorado.edu>
	<42C16E88.901@an.org> <42C2C3A2.8040303@an.org>
	<d9ugi5$404$1@sea.gmane.org>
Message-ID: <42C2CC69.5070308@colorado.edu>

Robert Kern wrote:
> Toni Alatalo wrote:
> 
>>Toni Alatalo wrote:
>>
>>this is not about ipython anymore, so i guess should move the discussion 
>>somewhere else .. an own list for the notebook thing perhaps (or how 
>>close is this work to docutils, in fact?)? but i hope you don't mind 
>>this so little (off-topic) update follows:
> 
> 
> This is the right place. It's not like we're flooded with traffic.

Thanks Robert, I was about to say the same thing.  And this project will weave 
in and out of core ipython concerns, so this is the proper place, don't worry.

Cheers,

f



From antont at an.org  Thu Jun 30 07:10:48 2005
From: antont at an.org (Toni Alatalo)
Date: Thu, 30 Jun 2005 14:10:48 +0300
Subject: [IPython-dev] Re: questions about the notebook interface
In-Reply-To: <42C2CC69.5070308@colorado.edu>
References: <42C00157.2020808@an.org> <42C055DF.9040505@colorado.edu>
	<42C16E88.901@an.org>
	<42C2C3A2.8040303@an.org>	<d9ugi5$404$1@sea.gmane.org>
	<42C2CC69.5070308@colorado.edu>
Message-ID: <42C3D338.6030505@an.org>

Fernando Perez wrote:

> Thanks Robert, I was about to say the same thing.  And this project 
> will weave in and out of core ipython concerns, so this is the proper 
> place, don't worry.

ok great.

there's an update now,
http://an.org/programming/notebook/test.py is already much nicer, uses
http://an.org/programming/notebook/__init__.py

.. one bug remaining, will have to think about it.

> f

~Toni