From barrywark at gmail.com Tue Jul 1 01:11:58 2008 From: barrywark at gmail.com (Barry Wark) Date: Mon, 30 Jun 2008 22:11:58 -0700 Subject: [IPython-dev] frontend plans In-Reply-To: References: <6ce0ac130806251559s7350cc6csd1ce6ab65b7510f6@mail.gmail.com> <6ce0ac130806261603m24868dedh26cd9243ff389b20@mail.gmail.com> <6ce0ac130806262031l191304d9xbd49100ab09def8@mail.gmail.com> Message-ID: On Mon, Jun 30, 2008 at 5:37 PM, Fernando Perez wrote: > Hey folks, > > I'm sorry that I failed to respond during this thread, but as it > happened I was at a conference with only micro-breaks for 'easy' > emails, but not to digest all of this :) > > Having said that, here are a few thoughts on what has been said so far: > > - Gael: as Brian asked, what are your design parameters regarding GUI > blocking? If you put the exec calls in a thread (like ipython > -Xthread does) you have some hope of interface responsiveness, if you > put them in a process you get full GUI responsiveness at the cost of > other complexities. Are you OK with a non-responsive GUI while exec() > is busy? > > - Zope interfaces: what do they exactly bring us? Nose uses > interfaces in a non-enforcing way by making them pure python classes > that are meant to document behavior but *not* to be subclassed. > Perhaps we could have something similar for the pure python version > and then a ZI version for the rest of the twisted layer: > > class BasicInterface(object): > def foo(self,x,y): > """does z with x and y""" > raise NotImplementedError() > > class RealInterface(BasicInterface,zope.interfaces.whatever): > pass > > Code NOT using twisted would inherit from the first type of functions, > and all twisted-based code would inherit from the second. What about something like the following:: try: raise ImportError() from zope.interface import Interface, Attribute, implements, classProvides except ImportError: def Attribute(name,doc): pass def implements(interface): pass def classProvides(interface): pass class IExample(Interface): Attribute('name', 'Attribute name __doc__') def example_method(): """example_method""" pass class Implementor(object): """Implements the IExample interface.""" implements(IExample) def example_method(self): """example_method""" print self The only thing you don't get is being able to check that Implementor implements IExample if zope.interface isn't present. > > - basic observation: exec() is fundamentally a blocking primitive. At > the end of the day, you have to wait for that particular chunk of code > to finish, and that's the python interpreter itself you're waiting > for. For this reason, it makes sense that the lowest level > abstractions we have should be blocking, and asynchronous interfaces > are wrapped around those for systems that are 'removed' from this > execution core (by being out of process, in another computer, etc). > Yes, I know that exec() could be calling threaded code, but that > simply means that exec(x) can finish quickly and some of the results > will be ready later. But the overall operation of completing the > execution of 'x' is still a blocking one. > > - Because of the above, I'm not crazy about the whole "synchronous > deferred" use. In my mind, the logical containment chain is: > > (0 - python VM - fundamentally synchronous object) < (1 - Ipython > layer that manages this) < (2 - Asynchronous wrappers for systems that > will work out of process, over the network, etc). I completely agree. The previous discussion convinced me this is the right way to go. > > > - Tab completion: this is just one case of the more generic case of > how to represent out-of-process information. We have to assume that > in general, only the core has true access to the real in-memory > objects of the user's namespace. Clients may request information > about this for display purposes, and some communication of reduced > data may happen with such clients, but we can't expect to copy the > user's namespace across the wire in a general sense. So yes, tab > completion and similar introspection will always happen by clients > requesting the operation on the core, and the core sending back > something reasonable back (a list of strings, for example). Agreed. I think the statement "clients may request information about this for display purposes" is a somewhat complicated idea that will require some more work. > > > In summary: it seems from what Barry said in the end, as well as > Gael, that we're all happy with the notion of a core, blocking system > that ultimately is just a bells-and-whistles version of python's "exec > code in namespace" statement. It lets you manage that namespace, > introspect it, it offers extensions, history, etc, but ultimately it's > just wrapping that one single statement. Because that statement > *fundamentally* blocks, this system blocks. You can embed it in a GUI > or use it in a terminal, but it still blocks. > > Beyond that, it makes sense to wrap this in a Twisted layer that turns > the result of exec() into a deferred. This makes complete sense when > the process doing exec() is different than your own (gui, network, > etc) and you actively want to move on with your life, while having a > notification mechanism for handling results/errors arising from the > exec call. At this point, the Twisted callback/errback system seems > well tailored for this. > > And I certainly want to ensure that Gael finds the code that's in > there sufficient for him to use in his WX project. It seems to me > that's the case, but have we left any question unanswered on that > front? There aren't any immediate questions that I'm aware of. -Barry > > Cheers, > > f > From gael.varoquaux at normalesup.org Tue Jul 1 01:27:59 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 1 Jul 2008 07:27:59 +0200 Subject: [IPython-dev] frontend plans In-Reply-To: References: <6ce0ac130806251559s7350cc6csd1ce6ab65b7510f6@mail.gmail.com> <6ce0ac130806261603m24868dedh26cd9243ff389b20@mail.gmail.com> <6ce0ac130806262031l191304d9xbd49100ab09def8@mail.gmail.com> Message-ID: <20080701052759.GA4509@phare.normalesup.org> On Mon, Jun 30, 2008 at 05:37:44PM -0700, Fernando Perez wrote: > - Gael: as Brian asked, what are your design parameters regarding GUI > blocking? If you put the exec calls in a thread (like ipython > -Xthread does) you have some hope of interface responsiveness, if you > put them in a process you get full GUI responsiveness at the cost of > other complexities. Are you OK with a non-responsive GUI while exec() > is busy? I thought I had laid this out clearly: yes I am OK with that. > - Zope interfaces: what do they exactly bring us? Nose uses > interfaces in a non-enforcing way by making them pure python classes > that are meant to document behavior but *not* to be subclassed. > Perhaps we could have something similar for the pure python version > and then a ZI version for the rest of the twisted layer: I am fine with Zope interfaces, we can imbed the files. > In summary: it seems from what Barry said in the end, as well as > Gael, that we're all happy with the notion of a core, blocking system > that ultimately is just a bells-and-whistles version of python's "exec > code in namespace" statement. It lets you manage that namespace, > introspect it, it offers extensions, history, etc, but ultimately it's > just wrapping that one single statement. Because that statement > *fundamentally* blocks, this system blocks. You can embed it in a GUI > or use it in a terminal, but it still blocks. > Beyond that, it makes sense to wrap this in a Twisted layer that turns > the result of exec() into a deferred. This makes complete sense when > the process doing exec() is different than your own (gui, network, > etc) and you actively want to move on with your life, while having a > notification mechanism for handling results/errors arising from the > exec call. At this point, the Twisted callback/errback system seems > well tailored for this. > And I certainly want to ensure that Gael finds the code that's in > there sufficient for him to use in his WX project. It seems to me > that's the case, but have we left any question unanswered on that > front? I am fine with all you said up here. Your summary of the situation is good, and I am happy with the way things turn out. I'll be on Mayavi sprint this week, so pretty much off line for this discussin, sorry, but I like the way everything sounds. Thanks guys, Ga?l From fperez.net at gmail.com Tue Jul 1 03:12:29 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 1 Jul 2008 00:12:29 -0700 Subject: [IPython-dev] frontend plans In-Reply-To: <20080701052759.GA4509@phare.normalesup.org> References: <6ce0ac130806251559s7350cc6csd1ce6ab65b7510f6@mail.gmail.com> <6ce0ac130806261603m24868dedh26cd9243ff389b20@mail.gmail.com> <6ce0ac130806262031l191304d9xbd49100ab09def8@mail.gmail.com> <20080701052759.GA4509@phare.normalesup.org> Message-ID: On Mon, Jun 30, 2008 at 10:27 PM, Gael Varoquaux wrote: > On Mon, Jun 30, 2008 at 05:37:44PM -0700, Fernando Perez wrote: >> - Gael: as Brian asked, what are your design parameters regarding GUI >> blocking? If you put the exec calls in a thread (like ipython >> -Xthread does) you have some hope of interface responsiveness, if you >> put them in a process you get full GUI responsiveness at the cost of >> other complexities. Are you OK with a non-responsive GUI while exec() >> is busy? > > I thought I had laid this out clearly: yes I am OK with that. Thanks. I was pretty much sure that's what you meant, but with all the back and forth, there was a tiny bit of confusion left in my brain, thanks for the clarification. >> - Zope interfaces: what do they exactly bring us? Nose uses >> interfaces in a non-enforcing way by making them pure python classes >> that are meant to document behavior but *not* to be subclassed. >> Perhaps we could have something similar for the pure python version >> and then a ZI version for the rest of the twisted layer: > > I am fine with Zope interfaces, we can imbed the files. As Brian pointed out, they bring in a C dependency we'd rather avoid if we can (and if it can be avoided with minimal cost). Barry outlined the solution with more detail than my flaky sketch, and I don't see any problem with that approach, so I think we can use that. > I am fine with all you said up here. Your summary of the situation is > good, and I am happy with the way things turn out. I'll be on Mayavi > sprint this week, so pretty much off line for this discussin, sorry, but > I like the way everything sounds. > > Thanks guys, Well, thanks to you for having the patience to sort all of this! Good luck with the Mayavi week, let us know how it all goes! In the meantime, we've got plenty to stay busy here :) Cheers, f From jorgen.stenarson at bostream.nu Tue Jul 1 16:26:13 2008 From: jorgen.stenarson at bostream.nu (=?ISO-8859-1?Q?J=F6rgen_Stenarson?=) Date: Tue, 01 Jul 2008 22:26:13 +0200 Subject: [IPython-dev] pyreadline refactor progess Message-ID: <486A92E5.5010701@bostream.nu> Hi, the refactoring of pyreadline is progressing nicely. I have now moved all stuff that depends on the console out of the emacs mode. The vi mode still uses some console stuff and should not be used in non-console mode. There is a very simplistic tk_gui.py example in the examples directory. The translation of tk key events to the event type used in pyreadline is not complete. Dead keys, keys using alt gr, as well as many non-alphanumeric keys are not translated. The example also has a very simple python prompt. There also needs to be work on getting the cursor visible. The code is available, using bzr, from: lp:~jorgen-stenarson/pyreadline/pyreadline-refactor If you want to try it out note that an install will install a readline.py which may interfere with your regular readline. /J?rgen From ondrej at certik.cz Wed Jul 2 09:38:37 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 2 Jul 2008 15:38:37 +0200 Subject: [IPython-dev] how to filter launchpad bugs Message-ID: <85b5c3130807020638j3e73008fkd0435aed52d13de3@mail.gmail.com> Hi, is there a way to filter launchpad bugs? They are filling up my inbox, but I would rather put them to the ipython folder or something. They usually look like this: [Bug 244854] [NEW] ipython.el doesn't work with optparse so I wonder how to filter them. Some mailinglists have a header like this: mailing list: ipython-dev.scipy.org or at least some footer, but these bug reports have nothing. Any ideas? Ondrej From fperez.net at gmail.com Wed Jul 2 13:21:16 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 2 Jul 2008 10:21:16 -0700 Subject: [IPython-dev] how to filter launchpad bugs In-Reply-To: <85b5c3130807020638j3e73008fkd0435aed52d13de3@mail.gmail.com> References: <85b5c3130807020638j3e73008fkd0435aed52d13de3@mail.gmail.com> Message-ID: On Wed, Jul 2, 2008 at 6:38 AM, Ondrej Certik wrote: > Hi, > > is there a way to filter launchpad bugs? They are filling up my inbox, > but I would rather put them to the ipython folder or something. They > usually look like this: > > [Bug 244854] [NEW] ipython.el doesn't work with optparse > > so I wonder how to filter them. Some mailinglists have a header like this: > > mailing list: ipython-dev.scipy.org > > or at least some footer, but these bug reports have nothing. Any ideas? The best I could do was to make a filter (in gmail) for "has the words bugs.launchpad.net ipython", because they don't mark the headers with list ids or anything sensible. But that's working reasonably well for me so far (though it could easily produce a false positive, obviously). Cheers, f From ondrej at certik.cz Wed Jul 2 13:30:01 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 2 Jul 2008 19:30:01 +0200 Subject: [IPython-dev] how to filter launchpad bugs In-Reply-To: References: <85b5c3130807020638j3e73008fkd0435aed52d13de3@mail.gmail.com> Message-ID: <85b5c3130807021030q4d0d2e14h5f63fa94d1997670@mail.gmail.com> On Wed, Jul 2, 2008 at 7:21 PM, Fernando Perez wrote: > On Wed, Jul 2, 2008 at 6:38 AM, Ondrej Certik wrote: >> Hi, >> >> is there a way to filter launchpad bugs? They are filling up my inbox, >> but I would rather put them to the ipython folder or something. They >> usually look like this: >> >> [Bug 244854] [NEW] ipython.el doesn't work with optparse >> >> so I wonder how to filter them. Some mailinglists have a header like this: >> >> mailing list: ipython-dev.scipy.org >> >> or at least some footer, but these bug reports have nothing. Any ideas? > > The best I could do was to make a filter (in gmail) for "has the words > bugs.launchpad.net ipython", because they don't mark the headers with > list ids or anything sensible. But that's working reasonably well for > me so far (though it could easily produce a false positive, > obviously). Yes, for example this email. :) But I marked all of them with ipython tag, so it doesn't matter. Thanks for the tip. Ondrej From fperez.net at gmail.com Wed Jul 2 13:44:40 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 2 Jul 2008 10:44:40 -0700 Subject: [IPython-dev] how to filter launchpad bugs In-Reply-To: <85b5c3130807021030q4d0d2e14h5f63fa94d1997670@mail.gmail.com> References: <85b5c3130807020638j3e73008fkd0435aed52d13de3@mail.gmail.com> <85b5c3130807021030q4d0d2e14h5f63fa94d1997670@mail.gmail.com> Message-ID: On Wed, Jul 2, 2008 at 10:30 AM, Ondrej Certik wrote: > Yes, for example this email. :) But I marked all of them with ipython GMailRuntimeError: maximum recursion depth exceeded in filter. ;) cheers, f From fperez.net at gmail.com Wed Jul 2 16:55:47 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 2 Jul 2008 13:55:47 -0700 Subject: [IPython-dev] Tentative release plans... Message-ID: Hi all, how does a quick release next week sound for everyone? Rationale: we all seem to agree that making more frequent releases to checkpoint development is good for the project, and recently we've had a large merge of all the previous ip1 code and docs, as well as all the frontend scaffolding. I'd like to push for a release next week because we're almost done with the two things that in my mind were necessary: - process control, so it's easier to start/stop engines/controllers from within a python standalone script. Min is hard at work on this. - testing: many of our recent headaches are due to our lack of testing for the core code in ip0, and with the upcoming frondend refactoring not having this fixed would simply be catastrophic. I've thus 'enjoyed' for the last few days a knee-deep excursion into the intertwined jungle of unittest/doctest/nose and I'm almost done (will be by the weekend) with a nose plugin that recognizes ipython docstrings, standalone ipython sessions and extension code with docstrings. This will give us a way to quickly turn interactive sessions into doctests so anyone who's about to refactor some big piece of code can simply first paste a working session and then go hog-wild with the refactoring, safely knowing that any breakage will be automatically caught by the tests. The suggested timing is also due to the fact that Brian and I will both be together for a few days at a conference in San Diego, so we can find lunch/evening times to work and push the testing and logistics of a release. How does this sound to everyone? Keep in mind that we DON'T need to have everything in place! We're trying to make releases reasonably frequently, so as long as we clearly mark what's ready and what's not yet, it's OK. I just don't want to catch anyone in mid-stride with any specific changes. Cheers, f From barrywark at gmail.com Wed Jul 2 17:35:56 2008 From: barrywark at gmail.com (Barry Wark) Date: Wed, 2 Jul 2008 14:35:56 -0700 Subject: [IPython-dev] Tentative release plans... In-Reply-To: References: Message-ID: +1 I'm really looking forward to the process control improvements for engines/controllers. On Wed, Jul 2, 2008 at 1:55 PM, Fernando Perez wrote: > Hi all, > > how does a quick release next week sound for everyone? Rationale: we > all seem to agree that making more frequent releases to checkpoint > development is good for the project, and recently we've had a large > merge of all the previous ip1 code and docs, as well as all the > frontend scaffolding. I'd like to push for a release next week > because we're almost done with the two things that in my mind were > necessary: > > - process control, so it's easier to start/stop engines/controllers > from within a python standalone script. Min is hard at work on this. > > - testing: many of our recent headaches are due to our lack of testing > for the core code in ip0, and with the upcoming frondend refactoring > not having this fixed would simply be catastrophic. I've thus > 'enjoyed' for the last few days a knee-deep excursion into the > intertwined jungle of unittest/doctest/nose and I'm almost done (will > be by the weekend) with a nose plugin that recognizes ipython > docstrings, standalone ipython sessions and extension code with > docstrings. This will give us a way to quickly turn interactive > sessions into doctests so anyone who's about to refactor some big > piece of code can simply first paste a working session and then go > hog-wild with the refactoring, safely knowing that any breakage will > be automatically caught by the tests. > > The suggested timing is also due to the fact that Brian and I will > both be together for a few days at a conference in San Diego, so we > can find lunch/evening times to work and push the testing and > logistics of a release. > > How does this sound to everyone? > > Keep in mind that we DON'T need to have everything in place! We're > trying to make releases reasonably frequently, so as long as we > clearly mark what's ready and what's not yet, it's OK. I just don't > want to catch anyone in mid-stride with any specific changes. > > Cheers, > > f > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev > From ellisonbg.net at gmail.com Wed Jul 2 18:25:59 2008 From: ellisonbg.net at gmail.com (Brian Granger) Date: Wed, 2 Jul 2008 16:25:59 -0600 Subject: [IPython-dev] Tentative release plans... In-Reply-To: References: Message-ID: <6ce0ac130807021525j68c2b54ar1fcb3d9a79c3e2c9@mail.gmail.com> I am all for this. There are a few other things that must be done for this release: 1. The docs need to be updated to reflect the merge. The most important things are the install docs and some of the parallel computing stuff. 2. Some bugs with setup.py and friends. The data files are broken right now and last night I found some bugs with .txt files in the testing dirs not being included in the package_data 3. Dependencies. We now have lots of code that has optional dependencies (like Twisted, zope.interface, pyobjc, etc.). We need to really make sure these are truly optional and don't cause problems for users who don't have these things installed. Sounds like I should file some tickets :) Cheers, Brian On Wed, Jul 2, 2008 at 2:55 PM, Fernando Perez wrote: > Hi all, > > how does a quick release next week sound for everyone? Rationale: we > all seem to agree that making more frequent releases to checkpoint > development is good for the project, and recently we've had a large > merge of all the previous ip1 code and docs, as well as all the > frontend scaffolding. I'd like to push for a release next week > because we're almost done with the two things that in my mind were > necessary: > > - process control, so it's easier to start/stop engines/controllers > from within a python standalone script. Min is hard at work on this. > > - testing: many of our recent headaches are due to our lack of testing > for the core code in ip0, and with the upcoming frondend refactoring > not having this fixed would simply be catastrophic. I've thus > 'enjoyed' for the last few days a knee-deep excursion into the > intertwined jungle of unittest/doctest/nose and I'm almost done (will > be by the weekend) with a nose plugin that recognizes ipython > docstrings, standalone ipython sessions and extension code with > docstrings. This will give us a way to quickly turn interactive > sessions into doctests so anyone who's about to refactor some big > piece of code can simply first paste a working session and then go > hog-wild with the refactoring, safely knowing that any breakage will > be automatically caught by the tests. > > The suggested timing is also due to the fact that Brian and I will > both be together for a few days at a conference in San Diego, so we > can find lunch/evening times to work and push the testing and > logistics of a release. > > How does this sound to everyone? > > Keep in mind that we DON'T need to have everything in place! We're > trying to make releases reasonably frequently, so as long as we > clearly mark what's ready and what's not yet, it's OK. I just don't > want to catch anyone in mid-stride with any specific changes. > > Cheers, > > f > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev > From fperez.net at gmail.com Wed Jul 2 18:28:40 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 2 Jul 2008 15:28:40 -0700 Subject: [IPython-dev] Tentative release plans... In-Reply-To: <6ce0ac130807021525j68c2b54ar1fcb3d9a79c3e2c9@mail.gmail.com> References: <6ce0ac130807021525j68c2b54ar1fcb3d9a79c3e2c9@mail.gmail.com> Message-ID: On Wed, Jul 2, 2008 at 3:25 PM, Brian Granger wrote: > I am all for this. There are a few other things that must be done for > this release: > > 1. The docs need to be updated to reflect the merge. The most > important things are the install docs and some of the parallel > computing stuff. > > 2. Some bugs with setup.py and friends. The data files are broken > right now and last night I found some bugs with .txt files in the > testing dirs not being included in the package_data > > 3. Dependencies. We now have lots of code that has optional > dependencies (like Twisted, zope.interface, pyobjc, etc.). We need to > really make sure these are truly optional and don't cause problems for > users who don't have these things installed. > > Sounds like I should file some tickets :) I'm up to my neck in the testing stuff (but I just a *second* ago got a key piece to work, so happy here :), so if anyone else can help with some of this, that would be great. If not, I guess Brian and I know what we'll be doing Sunday evening :) Cheers, f From anand.prabhakar.patil at gmail.com Thu Jul 3 04:35:48 2008 From: anand.prabhakar.patil at gmail.com (Anand Patil) Date: Thu, 3 Jul 2008 09:35:48 +0100 Subject: [IPython-dev] Tentative release plans... In-Reply-To: References: Message-ID: <934A35FD-E0FE-4D1E-A10B-5488BDCC4447@gmail.com> On 2 Jul 2008, at 21:55, Fernando Perez wrote: > recently we've had a large > merge of all the previous ip1 code and docs, as well as all the > frontend scaffolding. Does this mean the ipython1 Bazaar branch will be no more? Cheers, Anand From ellisonbg.net at gmail.com Thu Jul 3 10:23:37 2008 From: ellisonbg.net at gmail.com (Brian Granger) Date: Thu, 3 Jul 2008 08:23:37 -0600 Subject: [IPython-dev] Tentative release plans... In-Reply-To: <934A35FD-E0FE-4D1E-A10B-5488BDCC4447@gmail.com> References: <934A35FD-E0FE-4D1E-A10B-5488BDCC4447@gmail.com> Message-ID: <6ce0ac130807030723q2becc5e6n2bd905de9e8d63a7@mail.gmail.com> On Thu, Jul 3, 2008 at 2:35 AM, Anand Patil wrote: > On 2 Jul 2008, at 21:55, Fernando Perez wrote: > >> recently we've had a large >> merge of all the previous ip1 code and docs, as well as all the >> frontend scaffolding. > > Does this mean the ipython1 Bazaar branch will be no more? Yep, all of the stuff in ipython1 has been merged into IPython trunk. We will probably leave ipython1-dev branch around for a while to be safe, but people should not be using ipython1-dev anymore. Cheers, Brian > Cheers, > Anand > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev > From hans_meine at gmx.net Thu Jul 3 10:38:37 2008 From: hans_meine at gmx.net (Hans Meine) Date: Thu, 3 Jul 2008 16:38:37 +0200 Subject: [IPython-dev] frontend plans In-Reply-To: References: <6ce0ac130806262031l191304d9xbd49100ab09def8@mail.gmail.com> Message-ID: <200807031638.38346.hans_meine@gmx.net> Am Dienstag, 01. Juli 2008 02:37:44 schrieb Fernando Perez: > - Tab completion: this is just one case of the more generic case of > how to represent out-of-process information. We have to assume that > in general, only the core has true access to the real in-memory > objects of the user's namespace. Right. In our PyQt-based pyterm, tab-completion checks whether the backend (TCP server running the python commands) is busy and does nothing then. If you think about it, it's not too different from a regular (ipython / operating system) shell, and it has not bothered me a lot at all. One may think about possible out-of-process completions for some corner cases later (e.g. for filenames, it may be nontricky to implement something that's useful at least in many cases). > Clients may request information > about this for display purposes, and some communication of reduced > data may happen with such clients, but we can't expect to copy the > user's namespace across the wire in a general sense. Definitely not. I am particularly happy that the idea of having an optional IPC/nonblocking layer between the front- and backend has been discussed; I have myself refactored our pyterm in the past in order to separate the communication between the two (and make it optional) such that I can run the backend in-process (all in one thread). As IIRC Gael wrote, this can be a common, desired mode of operation in GUI programs; at least you can add scripting to your GUI very easily then, without thinking about threading and/or synchronization (which may be overkill e.g. for your own mini-GUI for scientific visualization). Have a nice day, Hans From ellisonbg.net at gmail.com Thu Jul 3 11:49:01 2008 From: ellisonbg.net at gmail.com (Brian Granger) Date: Thu, 3 Jul 2008 09:49:01 -0600 Subject: [IPython-dev] Tentative release plans... In-Reply-To: References: <6ce0ac130807021525j68c2b54ar1fcb3d9a79c3e2c9@mail.gmail.com> Message-ID: <6ce0ac130807030849h60a99665mf0cf95a48e704ada@mail.gmail.com> > I'm up to my neck in the testing stuff (but I just a *second* ago got > a key piece to work, so happy here :), so if anyone else can help with > some of this, that would be great. If not, I guess Brian and I know > what we'll be doing Sunday evening :) I didn't mean for this to sound like more things that _you_ should be doing. It was more of a reminder list to myself. I would much rather have you continue on the testing stuff. I can start to work on this other stuff later next week. Cheers, Brian > Cheers, > > f > From fperez.net at gmail.com Thu Jul 3 12:09:10 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 3 Jul 2008 09:09:10 -0700 Subject: [IPython-dev] Tentative release plans... In-Reply-To: <6ce0ac130807030849h60a99665mf0cf95a48e704ada@mail.gmail.com> References: <6ce0ac130807021525j68c2b54ar1fcb3d9a79c3e2c9@mail.gmail.com> <6ce0ac130807030849h60a99665mf0cf95a48e704ada@mail.gmail.com> Message-ID: On Thu, Jul 3, 2008 at 8:49 AM, Brian Granger wrote: >> I'm up to my neck in the testing stuff (but I just a *second* ago got >> a key piece to work, so happy here :), so if anyone else can help with >> some of this, that would be great. If not, I guess Brian and I know >> what we'll be doing Sunday evening :) > > I didn't mean for this to sound like more things that _you_ should be > doing. It was more of a reminder list to myself. I would much rather > have you continue on the testing stuff. I can start to work on this > other stuff later next week. No worries, I didn't take it that way :) I did get most of the pieces working right, so I'll package things up and push it upstream probably in a day or so. There are a few more things I want to make sure work, but I think we're in pretty good shape now. It was surprisingly hard to get this code working even though it's a fairly small amount, but I ended up chasing/fixing bugs in all of Cython, doctest and nose. Having to work over bugs in 3 separate codebases that aren't your own while trying to integrate them felt like a root canal... In any case, I'm reasonably happy with the outcome, even if it's not perfect. I'll clear my remaining todo items on that code and put it in for others to review/comment. Cheers, f From ellisonbg.net at gmail.com Thu Jul 3 12:22:16 2008 From: ellisonbg.net at gmail.com (Brian Granger) Date: Thu, 3 Jul 2008 10:22:16 -0600 Subject: [IPython-dev] Tentative release plans... In-Reply-To: References: <6ce0ac130807021525j68c2b54ar1fcb3d9a79c3e2c9@mail.gmail.com> <6ce0ac130807030849h60a99665mf0cf95a48e704ada@mail.gmail.com> Message-ID: <6ce0ac130807030922w41444441ka4789da678ef4179@mail.gmail.com> > I did get most of the pieces working right, so I'll package things up > and push it upstream probably in a day or so. There are a few more > things I want to make sure work, but I think we're in pretty good > shape now. It was surprisingly hard to get this code working even > though it's a fairly small amount, but I ended up chasing/fixing bugs > in all of Cython, doctest and nose. Having to work over bugs in 3 > separate codebases that aren't your own while trying to integrate them > felt like a root canal... Well, this will be a huge step forward in helping us to get testing into IPython code. Look forward to seeing it. > In any case, I'm reasonably happy with the outcome, even if it's not > perfect. I'll clear my remaining todo items on that code and put it > in for others to review/comment. Great. Brian > Cheers, > > f > From barrywark at gmail.com Thu Jul 3 14:18:21 2008 From: barrywark at gmail.com (Barry Wark) Date: Thu, 3 Jul 2008 11:18:21 -0700 Subject: [IPython-dev] Tentative release plans... In-Reply-To: References: <6ce0ac130807021525j68c2b54ar1fcb3d9a79c3e2c9@mail.gmail.com> <6ce0ac130807030849h60a99665mf0cf95a48e704ada@mail.gmail.com> Message-ID: On Thu, Jul 3, 2008 at 9:09 AM, Fernando Perez wrote: > On Thu, Jul 3, 2008 at 8:49 AM, Brian Granger wrote: >>> I'm up to my neck in the testing stuff (but I just a *second* ago got >>> a key piece to work, so happy here :), so if anyone else can help with >>> some of this, that would be great. If not, I guess Brian and I know >>> what we'll be doing Sunday evening :) >> >> I didn't mean for this to sound like more things that _you_ should be >> doing. It was more of a reminder list to myself. I would much rather >> have you continue on the testing stuff. I can start to work on this >> other stuff later next week. > > No worries, I didn't take it that way :) > > I did get most of the pieces working right, so I'll package things up > and push it upstream probably in a day or so. There are a few more > things I want to make sure work, but I think we're in pretty good > shape now. It was surprisingly hard to get this code working even > though it's a fairly small amount, but I ended up chasing/fixing bugs > in all of Cython, doctest and nose. Having to work over bugs in 3 > separate codebases that aren't your own while trying to integrate them > felt like a root canal... > > In any case, I'm reasonably happy with the outcome, even if it's not > perfect. I'll clear my remaining todo items on that code and put it > in for others to review/comment. Thanks for taking on this not-so-fun task! It's obviously an important step in moving forward with confidence and we'll all benefit from your misery :) > > Cheers, > > f > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev > From stefan at sun.ac.za Fri Jul 4 03:55:53 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Fri, 4 Jul 2008 09:55:53 +0200 Subject: [IPython-dev] how to filter launchpad bugs In-Reply-To: <85b5c3130807020638j3e73008fkd0435aed52d13de3@mail.gmail.com> References: <85b5c3130807020638j3e73008fkd0435aed52d13de3@mail.gmail.com> Message-ID: <9457e7c80807040055u518b0c5cyed5e3e15d5e6a052@mail.gmail.com> 2008/7/2 Ondrej Certik : > is there a way to filter launchpad bugs? They are filling up my inbox, > but I would rather put them to the ipython folder or something. They > usually look like this: Use the following header: X-Launchpad-Message-Rationale: Subscriber (IPython) @ipython You can add a Google Mail filter by adding the part after the colon in the "Must contain" field. Cheers St?fan From vivainio at gmail.com Fri Jul 4 11:57:11 2008 From: vivainio at gmail.com (Ville M. Vainio) Date: Fri, 4 Jul 2008 18:57:11 +0300 Subject: [IPython-dev] Getting rid of "print" in ipython Message-ID: <46cb515a0807040857w6252c7a4l71977155b2a668ed@mail.gmail.com> IPython source code is riddled with direct calls to print statement. Obviously, we should get rid of them. What's the exact plan for this? For the time being, we could get rid of them by introducing a global function for this (that initially just does normal "print", and add proper output handling later. My most urgent interest in this lies in the current inability to easily capture magic command output. All magic command output should be accessible in output history as string list, which we could easily add if printing (inside magics) happened in controlled fashion. My current suggestion: add global callable "pr" in ipapi, that is connected to _ip.pr for the currently active IPApi object. Then, all extensions could do "from IPython.ipapi import pr" and print to that object to their heart's content. -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From fperez.net at gmail.com Fri Jul 4 15:35:24 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 4 Jul 2008 12:35:24 -0700 Subject: [IPython-dev] Getting rid of "print" in ipython In-Reply-To: <46cb515a0807040857w6252c7a4l71977155b2a668ed@mail.gmail.com> References: <46cb515a0807040857w6252c7a4l71977155b2a668ed@mail.gmail.com> Message-ID: Howdy, On Fri, Jul 4, 2008 at 8:57 AM, Ville M. Vainio wrote: > IPython source code is riddled with direct calls to print statement. > Obviously, we should get rid of them. Yup. > > What's the exact plan for this? For the time being, we could get rid > of them by introducing a global function for this (that initially just > does normal "print", and add proper output handling later. > > My most urgent interest in this lies in the current inability to > easily capture magic command output. All magic command output should > be accessible in output history as string list, which we could easily > add if printing (inside magics) happened in controlled fashion. > > My current suggestion: add global callable "pr" in ipapi, that is > connected to _ip.pr for the currently active IPApi object. Then, all > extensions could do "from IPython.ipapi import pr" and print to that > object to their heart's content. We already have a mechanism in place, we just don't use it consistenly: self.write/write_err. Those redirect to the Term object that can then be provided by a GUI, or replaced to log output, etc. Rather than a module global called pr, let's simply expose the write/write_err methods in ipapi and have users explicitly access the instance. I don't want more module-level globals, and especially not in ipapi. Something like self.write = self.IP.write # similar for _err should do. Cheers, f From fperez.net at gmail.com Fri Jul 4 15:40:59 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 4 Jul 2008 12:40:59 -0700 Subject: [IPython-dev] pyreadline refactor progess In-Reply-To: <486A92E5.5010701@bostream.nu> References: <486A92E5.5010701@bostream.nu> Message-ID: Hey Jorgen, On Tue, Jul 1, 2008 at 1:26 PM, J?rgen Stenarson wrote: > Hi, > > the refactoring of pyreadline is progressing nicely. I have now moved > all stuff that depends on the console out of the emacs mode. The vi mode > still uses some console stuff and should not be used in non-console mode. What are your thoughts on this being workable as a full replacement for python's readline in text terminals? There are platforms where readline continues to be a problem (until recently OSX, and John Hunter tells me his Solaris boxes never work quite right, for example). It would be nice if ipython could use its own self-contained readline everywhere, with a uniform API for text consoles or GUIs. But I don't know how to go about the work of trapping keyboard events in a text terminal (precisely what GNU readline does), and I don't feel like learning :) Cheers, f From jorgen.stenarson at bostream.nu Fri Jul 4 16:52:05 2008 From: jorgen.stenarson at bostream.nu (=?ISO-8859-1?Q?J=F6rgen_Stenarson?=) Date: Fri, 04 Jul 2008 22:52:05 +0200 Subject: [IPython-dev] pyreadline refactor progess In-Reply-To: References: <486A92E5.5010701@bostream.nu> Message-ID: <486E8D75.6020002@bostream.nu> Fernando Perez skrev: > Hey Jorgen, > > What are your thoughts on this being workable as a full replacement > for python's readline in text terminals? There are platforms where > readline continues to be a problem (until recently OSX, and John > Hunter tells me his Solaris boxes never work quite right, for > example). It would be nice if ipython could use its own > self-contained readline everywhere, with a uniform API for text > consoles or GUIs. But I don't know how to go about the work of > trapping keyboard events in a text terminal (precisely what GNU > readline does), and I don't feel like learning :) > Fernando, I think it should be possible, but I'm not at a position to do the terminal backend development myself (I'm only on windows currently). There is an ironpython backend in there but I haven't tried it for a long time so I'm not sure how well (if at all) it works now, there were some workarounds I had to do because the .NET console class does not implement the full windows console functionality. As a longer term goal of the refactoring I should isolate the methods that are necessary for pyreadline to work from those that are included but not really necessary. This should make it easier to develop new backends by calrifying which methods are necessary for its functionality. Perhaps it is possible to use the curses library on platforms where that is available but I have no idea how well that transmits the keystrokes and their interpretation for different locales etc. /J?rgen From Alexander_Brown at uml.edu Fri Jul 4 18:13:55 2008 From: Alexander_Brown at uml.edu (Alex Brown) Date: Fri, 04 Jul 2008 18:13:55 -0400 Subject: [IPython-dev] ipython-0.9.0.bzr.r1016-py2.5.win32-setup.exe?? [was: Re: Win32 IPython1?] In-Reply-To: References: <48457029.8020401@uml.edu> <48459D9F.9020900@uml.edu> <4864549D.2010407@uml.edu> <48659443.4050109@uml.edu> <4865AB18.4040000@uml.edu> Message-ID: <486EA0A3.9020307@uml.edu> Following your recommendations I'm working with your (merged) trunk in WinXP. However I am having continuing difficulty getting ipython-0.9.0.bzr.r1016-py2.5 to work with either Python 2.4 or 2.5 (both Enthought distributions). It may be that I'm not familiar enough with the setup and installer tools, or with Windows touchup tricks. Is there any chance of getting an exe installer, perhaps ipython-0.9.0.bzr.r1016-py2.5.win32-setup.exe? Alex Brown http://gis.uml.edu/abrown2 http://ipython.scipy.org/moin/IpythonOnWindows > Running IPython on Win32 > > IPython works well with Windows, even as a replacement for that clunky > old cmd.exe command prompt. There are some considerations, though. ... > > ... Sometimes when installing a new version of IPython, you need to > remove old IPython files to get a clean slate. > > To delete all traces of old IPython installation(s), delete the > following files (adjust for your python installation): > > * c:\Python25\Scripts\ipython* > * C:\Python25\Lib\site-packages\ipython* (also the directories). > > Delete start menu shortcuts: > > * > > Start menu -> All programs -> right click on IPython, choose > delete. > > Install new IPython, launch it, run %upgrade. Unfortunately this does not catch the version numbers in the "ipcontroller", "ipengine", and "ipcluster" scripts. Are there other scripts which have embedded version numbers? http://ipython.scipy.org/dist/ > > > Index of /dist > > Icon Name Last modified Size Description > ------------------------------------------------------------------------ > [DIR] Parent Directory - > ... > > [ ] ipython-0.8.2.win32-setup.exe 29-Nov-2007 13:14 1.7M Barry Wark wrote: > On Sat, Jun 28, 2008 at 11:38 AM, Fernando Perez wrote: > >> Hi Alex, >> >> again, please send these messages on-list, so they get properly >> archived and others may also help you if I'm not available (I'm at a >> conference right now). >> >> On Fri, Jun 27, 2008 at 8:08 PM, Alex Brown wrote: >> >> >>> I could not find a way to uninstall a module either with setup.py or >>> easy_install >>> (http://mail.python.org/pipermail/python-list/2004-December/294789.html) - >>> >> easy_install and distutils have no automated uninstallation, unfortunately. >> > > I believe you can do easy_install -m to effectively uninstall an app > that was installed using setuptools (not one that was installed via > distutils). The -m (or --multi-version) flag removes the package from > the setuptools .pth file, but doesn't remove it from the > site-directory. This way, anyone that wants to use it has to > explicitly require it from setuptools but everything else will not > know it's there. This is one way to have multiple versions of > setuptools-installed packages available or to "uninstall" a package > without breaking any other packages that depend on that package. > > > >>> I've simply moved ipython1* out of the tree, OK? I can't find anything >>> other than comment and docstring references to ipython1 after removal. >>> >> That is OK. >> >> Cheers, >> >> f >> _______________________________________________ >> IPython-dev mailing list >> IPython-dev at scipy.org >> http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: blank.gif Type: image/gif Size: 148 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: back.gif Type: image/gif Size: 216 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: binary.gif Type: image/gif Size: 246 bytes Desc: not available URL: From fperez.net at gmail.com Fri Jul 4 18:36:32 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 4 Jul 2008 15:36:32 -0700 Subject: [IPython-dev] Doc guidelines Message-ID: Hi all, I'll be committing soon the testing stuff, and because of this I've been looking again at the docs directories. We now have merged docs from various sources (hand-written reST from ipython1, converted reST from the old lyx sources, by various people, etc) and I realized that we have a mix of lots of styles. Fortunately reST is fairly flexible so we don't need to go on a major cleanup expedition, but I think that moving forward we should try to also keep a consistent formatting style in reST for the same reasons that consistent style in source code is a good idea when working in a team. Rather than coming up with any ad-hoc rules of our own, we might as well use what everyone else related to us is using. The sphinx project itself has some information on the basic use of reST with sphinx here: http://sphinx.pocoo.org/contents.html and I'd suggest that we also follow the advice of what matplotlib is doing: http://matplotlib.sourceforge.net/doc/html/devel/documenting_mpl.html MPL has recently gone on a *major* documentation effort, and we might as well share the tools they've added on top of basic sphinx. We obviously don't need the math rendering, but things like ipython-specific highlighting might come in handy :) Finally, a few things I noticed that I'd like to propose as standard practice, that aren't mentioned above. They basically boil down to re-using PEP-8 conventions in reST when applicable: - No hard tabs, 4 spaces for indenting code blocks and the like. While having a mix of tabs and spaces won't cause bugs like it can in python so easily, it's still likely to produce in the long run messy-to-edit sources since different people will do different things. It seems best to just stick to one solution consistent with the rest, else we'll be forever dealing with mixups due to how everyone had their editor set on any given day. - Lines to 78 chars or so. There are lots of unwrapped paragraphs, these tend to wrap differently depending on the editor used, so it's probably best to stick to regular line-broken paragraphs as for python source. - Any code block meant to represent a command line should have '$' markers. I think I'll be able to build support for all reST in the testing stuff, and this might allow us to mix commands meant for the shell with python sources without confusing the testing code nor making the parsing code a complete nightmare. (the sources are already pretty good in doing this, though I've seen a few spots where we've missed it). I can't think of anything else for now. But if you all agree on this, we'll put it in the guidelines and we can update things as we go. Cheers, f From gael.varoquaux at normalesup.org Sun Jul 6 17:07:31 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 6 Jul 2008 23:07:31 +0200 Subject: [IPython-dev] Schedule for the SciPy08 conferencez Message-ID: <20080706210731.GC25810@phare.normalesup.org> We have received a large number of excellent contributions for papers for the SciPy 2008 conference. The program committee has had to make a difficult selection and we are happy to bring to you a preliminary schedule: Thursday ========= **8:00** Registration/Breakfast **8:55** Welcome (Travis Vaught) **9:10** Keynote (Alex Martelli) **10:00** State of SciPy (Travis Vaught, Jarrod Millman) **10:40** -- Break -- **11:00** Sympy - Python library for symbolic mathematics: introduction and applications (Ond?ej ?ertik) **11:40** Interval arithmetic: Python implementation and applications (Stefano Taschini) **12:00** Experiences Using Scipy for Computer Vision Research (Damian Eads) **12:20** -- Lunch -- **1:40** The new NumPy documentation framework (St?fan Van der Walt) **2:00** Matplotlib solves the riddle of the sphinx (Michael Droettboom) **2:40** The SciPy documentation project (Joe Harrington) **3:00** -- Break -- **3:40** Sage: creating a viable free Python-based open source alternatice to Magma, Maple, Mathematica and Matlab (William Stein) **4:20** Open space for lightning talks Friday ======== **8:30** Breakfast **9:00** Pysynphot: A Python Re-Implementation of a Legacy App in Astronomy (Perry Greenfield) **9:40** How the Large Synoptic Survey Telescope (LSST) is using Python (Robert Lupton) **10:00** Real-time Astronomical Time-series Classification and Broadcast Pipeline (Dan Starr) **10:20** Analysis and Visualization of Multi-Scale Astrophysical Simulations using Python and NumPy (Matthew Turk) **10:40** -- Break -- **11:00** Exploring network structure, dynamics, and function using NetworkX (Aric Hagberg) **11:40** Mayavi: Making 3D data visualization reusable (Prabhu Ramachandran, Ga?l Varoquaux) **12:00** Finite Element Modeling of Contact and Impact Problems Using Python (Ryan Krauss) **12:20** -- Lunch -- **2:00** PyCircuitScape: A Tool for Landscape Ecology (Viral Shah) **2:20** Summarizing Complexity in High Dimensional Spaces (Karl Young) **2:40** UFuncs: A generic function mechanism in Python (Travis Oliphant) **3:20** -- Break -- **3:40** NumPy Optimization: Manual tuning and automated approaches (Evan Patterson) **4:00** Converting Python functions to dynamically-compiled C (Ilan Schnell) **4:20** unPython: Converting Python numerical programs into C (Rahul Garg) **4:40** Implementing the Grammar of Graphics for Python (Robert Kern) **5:00** Ask the experts session. A more detailled booklet including the abstract text will be available soon. We are looking forward to seeing you in Caltech, Ga?l Varoquaux, on behalf of the program committee. -- SciPy2008 conference. Program committee Anne Archibald, McGill University Matthew Brett Perry Greenfield, Space Telescope Science Institute Charles Harris Ryan Krauss, Southern Illinois University Ga?l Varoquaux St?fan van der Walt, University of Stellenbosch From fperez.net at gmail.com Sun Jul 6 21:34:38 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Sun, 6 Jul 2008 18:34:38 -0700 Subject: [IPython-dev] Wiki edit policy change Message-ID: Hi all, for a very long time, Ryan James has tirelessly been cleaning up spam on the wiki, but that's neither fun nor fair nor a wise use of resources (his time in this case). We recently realized, with Jarrod Millman's help, what the problem was: all the recommended techniques for disabling Moin account creation (at least the publicly listed ones) fail hopelessly, because from any existing logged in user, it is trivial to trick Moin into auto-generating new user accounts that it recognizes. So instead of the ineffective account creation block, I've changed things to openly allow new accounts, but without write privileges. Only users listed here: http://ipython.scipy.org/moin/WritersGroup can edit pages. I went through all the logs I could find to add everyone I could find that seemed like a valid user, please just let us know and we'll add you if I missed you. Please note that anyone in http://ipython.scipy.org/moin/EditorsGroup can edit the WritersGroup page and add new names, so it should be trivial to get added if you want to contribute, just ask here on the list or email an editor (or me) and we'll add you right away. Hopefully this will now work, since ACLs in Moin seem to be more robust than their Gruy?re cheese security model for user creation. And many, many thanks again to Ryan for an endless stream of spam-cleaning edits!!! Cheers, f From fperez.net at gmail.com Sun Jul 6 22:10:36 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Sun, 6 Jul 2008 19:10:36 -0700 Subject: [IPython-dev] Testing cleanup and support committed Message-ID: Hi all, I've just added http://bazaar.launchpad.net/~ipython/ipython/trunk/revision/1038 the nose support for ipython doctests and doctests in extension code. I also cleaned up a bunch of things from old testing stuff that we had. It's NOT finished, there's one piece of the puzzle that I still want to have done, which is the ability to run regular reST files with doctests AND code blocks, where the code blocks are executed and considered to succeed as long as they don't raise, but the doctests are still executed as such. This last piece would let us run all our examples in the new documentation, even if they have significant bits of code, as part of the test suite. The plugin directory has a little makefile for temporary testing, it's just a handy way to keep a note of the command you need to run while debugging. Eventually the (unfinished) iptest script that's there will provide this functionality, and it will not be necessary to actually install the plugin. But the nice thing is that now it is trivial to have entire ipython sessions copy/pasted from a terminal into either a docstring or a reST file as a codeblock, and it's automatically a test. Even though I'm now busy for a few days at a a conference (with the python talk yet to be written :) I wanted this out so others could play with it. It is NOT meant to cleanly test all of ipython yet, but we'll get there soon. And once we're happy with the functionality, I hope we'll all use this to properly butress any refactoring BEFORE breaking anything with good tests. For years the main (terminal) ipython had suffered tremendously from the lack of solid testing. I hope this is a first step to remedy that problem. Cheers, f From fperez.net at gmail.com Mon Jul 7 14:10:43 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 7 Jul 2008 11:10:43 -0700 Subject: [IPython-dev] [IPython-user] Wiki edit policy change In-Reply-To: <18546.21598.436480.380370@montanaro-dyndns-org.local> References: <18546.21598.436480.380370@montanaro-dyndns-org.local> Message-ID: On Mon, Jul 7, 2008 at 10:37 AM, wrote: > > Fernando> Please note that anyone in > > Fernando> http://ipython.scipy.org/moin/EditorsGroup > > Fernando> can edit the WritersGroup page and add new names, so it should > Fernando> be trivial to get added if you want to contribute, just ask > Fernando> here on the list or email an editor (or me) and we'll add you > Fernando> right away. > > FWIW, that page appears to be unreadable, even when logged in, so it won't > work as a way to discover an editor to contact for write privileges. (No, > I'm not asking for write privileges. Just checking the setup.) Thanks, fixed. It turns out ACLs in Moin are super-picky about whitespace: #acl EditorsGroup:read, write, revert, admin All:read turned out to be (silently) invalid, it should be: #acl EditorsGroup:read,write,revert,admin All:read (no spaces after the commas). Cheers, f From stefan at sun.ac.za Tue Jul 8 05:31:33 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 8 Jul 2008 11:31:33 +0200 Subject: [IPython-dev] Using pyreadline by default Message-ID: <9457e7c80807080231i5260403cvd65b13cd61b897c3@mail.gmail.com> Hi, As I mentioned earlier, I am not fond of the pager in readline, and would like to migrate towards pyreadline, in which I can implement my own. It looks like the choice between pyreadline and readline is hard-coded on a platform-by-platform basis in rlineimpl.py. It would be useful if we had a configuration variable which allowed the user to make that choice. If you are amenable to this idea, I'll write a patch. Regards St?fan From fperez.net at gmail.com Tue Jul 8 05:39:55 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 8 Jul 2008 02:39:55 -0700 Subject: [IPython-dev] Using pyreadline by default In-Reply-To: <9457e7c80807080231i5260403cvd65b13cd61b897c3@mail.gmail.com> References: <9457e7c80807080231i5260403cvd65b13cd61b897c3@mail.gmail.com> Message-ID: On Tue, Jul 8, 2008 at 2:31 AM, St?fan van der Walt wrote: > Hi, > > As I mentioned earlier, I am not fond of the pager in readline, and > would like to migrate towards pyreadline, in which I can implement my > own. > > It looks like the choice between pyreadline and readline is hard-coded > on a platform-by-platform basis in rlineimpl.py. It would be useful > if we had a configuration variable which allowed the user to make that > choice. If you are amenable to this idea, I'll write a patch. As Jorgen mentioned, the problem is that pyreadline currently only implements the full line-handling functionality you'd want under win32 (by accessing the console APIs). Under osx/linux, you'd be stuck without any actual line handling support, unless I'm not understanding what you are trying to do. By the way, isn't it easier for what you want to just define your own $PAGER environment variable? You can even do it for ipython only by making the change in your config file. We'd certainly love to see pyreadline grow further, as we've been recently discussing. But I have the impression that what you're talking about is a lot of work, not just a small patch (and likely to require extensive testing on multiple platforms/terminals). Or else I totally misunderstood what you're talking about. Cheers, f From stefan at sun.ac.za Tue Jul 8 06:32:37 2008 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 8 Jul 2008 12:32:37 +0200 Subject: [IPython-dev] Using pyreadline by default In-Reply-To: References: <9457e7c80807080231i5260403cvd65b13cd61b897c3@mail.gmail.com> Message-ID: <9457e7c80807080332s6c090d9btf5005d71c5b61d6f@mail.gmail.com> 2008/7/8 Fernando Perez : > As Jorgen mentioned, the problem is that pyreadline currently only > implements the full line-handling functionality you'd want under win32 > (by accessing the console APIs). Under osx/linux, you'd be stuck > without any actual line handling support, unless I'm not understanding > what you are trying to do. Oops, I missed that part. > By the way, isn't it easier for what you want to just define your own > $PAGER environment variable? You can even do it for ipython only by > making the change in your config file. Unfortunately, that doesn't help: the pager is not used for tab completion. Regards St?fan From dsdale24 at gmail.com Tue Jul 8 10:20:25 2008 From: dsdale24 at gmail.com (Darren Dale) Date: Tue, 8 Jul 2008 10:20:25 -0400 Subject: [IPython-dev] question about decision to use twisted Message-ID: <200807081020.27330.dsdale24@gmail.com> I have a data acquisition and analysis project that currently uses Parallel Python for distributed processing and PyQt4's for some threading and event handling. I am concidering the processing module scheduled for inclusion in the python standard library starting with version 2.6, and also I am trying to get my head around twisted. I would like to know what the ipython developers concerns were at the time the decision was made to use twisted in ipython1, was there some discussion on the ipython mailing lists? Any advice or comments would be greatly appreciated. Darren From jorgen.stenarson at bostream.nu Tue Jul 8 13:14:41 2008 From: jorgen.stenarson at bostream.nu (=?ISO-8859-1?Q?J=F6rgen_Stenarson?=) Date: Tue, 08 Jul 2008 19:14:41 +0200 Subject: [IPython-dev] Using pyreadline by default In-Reply-To: <9457e7c80807080332s6c090d9btf5005d71c5b61d6f@mail.gmail.com> References: <9457e7c80807080231i5260403cvd65b13cd61b897c3@mail.gmail.com> <9457e7c80807080332s6c090d9btf5005d71c5b61d6f@mail.gmail.com> Message-ID: <4873A081.7090708@bostream.nu> St?fan van der Walt skrev: > 2008/7/8 Fernando Perez : >> As Jorgen mentioned, the problem is that pyreadline currently only >> implements the full line-handling functionality you'd want under win32 >> (by accessing the console APIs). Under osx/linux, you'd be stuck >> without any actual line handling support, unless I'm not understanding >> what you are trying to do. > > Oops, I missed that part. > >> By the way, isn't it easier for what you want to just define your own >> $PAGER environment variable? You can even do it for ipython only by >> making the change in your config file. > > Unfortunately, that doesn't help: the pager is not used for tab completion. > You are right right, now in pyreadline there is a _display_completions method which just dumps all the matches to the screen. Which means there is no check of the $pager variable. I'm not even sure it is a good idea if we want to use the code for guis. The problem with the current approach is that it is at the moment hardcoded to use the console. Which means nothing will be shown if you try to use it in a gui. There are at least one more function (clear_screen) that talks directly to the console and thus does not work in a gui. Ideally the completions as well as the request for a clear_screen should be propagated to the calling function that implements the drawing. I have not decided yet how this is done best. /J?rgen From ellisonbg.net at gmail.com Tue Jul 8 15:57:39 2008 From: ellisonbg.net at gmail.com (Brian Granger) Date: Tue, 8 Jul 2008 12:57:39 -0700 Subject: [IPython-dev] question about decision to use twisted In-Reply-To: <200807081020.27330.dsdale24@gmail.com> References: <200807081020.27330.dsdale24@gmail.com> Message-ID: <6ce0ac130807081257w17730d32y57c85f10eef48610@mail.gmail.com> > I have a data acquisition and analysis project that currently uses Parallel > Python for distributed processing and PyQt4's for some threading and event > handling. I am concidering the processing module scheduled for inclusion in > the python standard library starting with version 2.6, and also I am trying > to get my head around twisted. Twisted is a much lower level solution than parallel python or processing. Are you thinking about using Twisted directly or just using it through IPython (ipython1 is not in trunk IPython). Unless you really want the treading API that processing provides, I think IPython is by far the best solution [I fully acknowledge my bias here] for high-level parallelism. > I would like to know what the ipython developers concerns were at the time the > decision was made to use twisted in ipython1, was there some discussion on > the ipython mailing lists? Any advice or comments would be greatly > appreciated. I don't think we really discussed this on the mailing list, it was probably private discussion between Fernando and myself. We went with Twisted as: 1) we didn't want to reinvent Twisted 2) we really need what Twisted provides, so the temptation to reinvent it was great. 3) Twisted has the right abstractions that force you to write correct an robust networking code. Of course you can do this without Twisted, but you are going to work much harder and probably get it wrong in subtle ways that are hard to figure out. We have been extremely happy about going with Twisted - it is our secret sauce. Are there any specific questions you have about Twisted? Cheers, Brian > Darren > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev > From robert.kern at gmail.com Tue Jul 8 16:21:17 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 08 Jul 2008 15:21:17 -0500 Subject: [IPython-dev] question about decision to use twisted In-Reply-To: <6ce0ac130807081257w17730d32y57c85f10eef48610@mail.gmail.com> References: <200807081020.27330.dsdale24@gmail.com> <6ce0ac130807081257w17730d32y57c85f10eef48610@mail.gmail.com> Message-ID: Brian Granger wrote: >> I have a data acquisition and analysis project that currently uses Parallel >> Python for distributed processing and PyQt4's for some threading and event >> handling. I am concidering the processing module scheduled for inclusion in >> the python standard library starting with version 2.6, and also I am trying >> to get my head around twisted. > > Twisted is a much lower level solution than parallel python or > processing. Are you thinking about using Twisted directly or just > using it through IPython (ipython1 is not in trunk IPython). s/not/now/ -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From barrywark at gmail.com Tue Jul 8 16:23:33 2008 From: barrywark at gmail.com (Barry Wark) Date: Tue, 8 Jul 2008 13:23:33 -0700 Subject: [IPython-dev] question about decision to use twisted In-Reply-To: <6ce0ac130807081257w17730d32y57c85f10eef48610@mail.gmail.com> References: <200807081020.27330.dsdale24@gmail.com> <6ce0ac130807081257w17730d32y57c85f10eef48610@mail.gmail.com> Message-ID: On Tue, Jul 8, 2008 at 12:57 PM, Brian Granger wrote: >> I have a data acquisition and analysis project that currently uses Parallel >> Python for distributed processing and PyQt4's for some threading and event >> handling. I am concidering the processing module scheduled for inclusion in >> the python standard library starting with version 2.6, and also I am trying >> to get my head around twisted. > > Twisted is a much lower level solution than parallel python or > processing. Are you thinking about using Twisted directly or just > using it through IPython (ipython1 is not in trunk IPython). Unless > you really want the treading API that processing provides, I think > IPython is by far the best solution [I fully acknowledge my bias here] > for high-level parallelism. Just wanted to clarify Brian's comment... ipython1 is _now_ in trunk IPython. Brian's done a ton of work to make this happen and I just didn't want a typo to get in they way ;) > >> I would like to know what the ipython developers concerns were at the time the >> decision was made to use twisted in ipython1, was there some discussion on >> the ipython mailing lists? Any advice or comments would be greatly >> appreciated. > > I don't think we really discussed this on the mailing list, it was > probably private discussion between Fernando and myself. We went with > Twisted as: > > 1) we didn't want to reinvent Twisted > > 2) we really need what Twisted provides, so the temptation to reinvent > it was great. > > 3) Twisted has the right abstractions that force you to write correct > an robust networking code. Of course you can do this without Twisted, > but you are going to work much harder and probably get it wrong in > subtle ways that are hard to figure out. > > We have been extremely happy about going with Twisted - it is our secret sauce. > > Are there any specific questions you have about Twisted? > > Cheers, > > Brian > >> Darren >> _______________________________________________ >> IPython-dev mailing list >> IPython-dev at scipy.org >> http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev >> > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev > From gael.varoquaux at normalesup.org Tue Jul 8 16:35:52 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 8 Jul 2008 22:35:52 +0200 Subject: [IPython-dev] question about decision to use twisted In-Reply-To: References: <200807081020.27330.dsdale24@gmail.com> <6ce0ac130807081257w17730d32y57c85f10eef48610@mail.gmail.com> Message-ID: <20080708203552.GB12533@phare.normalesup.org> On Tue, Jul 08, 2008 at 01:23:33PM -0700, Barry Wark wrote: > Just wanted to clarify Brian's comment... ipython1 is _now_ in trunk > IPython. Brian's done a ton of work to make this happen and I just > didn't want a typo to get in they way ;) Yes, this is a huge deal. Thanks a lot to Brian, Fernando, and everybody else who helped make this happen. Ga?l From dsdale24 at gmail.com Tue Jul 8 16:59:45 2008 From: dsdale24 at gmail.com (Darren Dale) Date: Tue, 8 Jul 2008 16:59:45 -0400 Subject: [IPython-dev] question about decision to use twisted In-Reply-To: <6ce0ac130807081257w17730d32y57c85f10eef48610@mail.gmail.com> References: <200807081020.27330.dsdale24@gmail.com> <6ce0ac130807081257w17730d32y57c85f10eef48610@mail.gmail.com> Message-ID: <200807081659.46721.dsdale24@gmail.com> Hi Brian, On Tuesday 08 July 2008 03:57:39 pm Brian Granger wrote: > > I have a data acquisition and analysis project that currently uses > > Parallel Python for distributed processing and PyQt4's for some threading > > and event handling. I am concidering the processing module scheduled for > > inclusion in the python standard library starting with version 2.6, and > > also I am trying to get my head around twisted. > > Twisted is a much lower level solution than parallel python or > processing. Are you thinking about using Twisted directly or just > using it through IPython (ipython1 is now in trunk IPython). (typo corrected) I was thinking of using twisted directly, but I guess I really need to dig into ipython1. I don't have a sense of how I would use ipython1 in a non-interactive way, is there some discussion in the docs or some examples of scripting with ipython1? Is there a callback mechanism in ipython1 to respond to non-blocking events? > Unless > you really want the treading API that processing provides, I think > IPython is by far the best solution [I fully acknowledge my bias here] > for high-level parallelism. > > > I would like to know what the ipython developers concerns were at the > > time the decision was made to use twisted in ipython1, was there some > > discussion on the ipython mailing lists? Any advice or comments would be > > greatly appreciated. > > I don't think we really discussed this on the mailing list, it was > probably private discussion between Fernando and myself. We went with > Twisted as: > > 1) we didn't want to reinvent Twisted > > 2) we really need what Twisted provides, so the temptation to reinvent > it was great. > > 3) Twisted has the right abstractions that force you to write correct > an robust networking code. Of course you can do this without Twisted, > but you are going to work much harder and probably get it wrong in > subtle ways that are hard to figure out. > > We have been extremely happy about going with Twisted - it is our secret > sauce. > > Are there any specific questions you have about Twisted? After reading some positive things about twisted, I went looking for a couple negative reviews. I've read that twisted is a bit of a moving target, and the twisted jargon is pretty unfamiliar to me. So finally I was hoping to temper all this information with the informed opinion from a group I know and respect. This is just the kind of feedback I was hoping to get. Thank you, Darren From robert.kern at gmail.com Tue Jul 8 18:02:05 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 08 Jul 2008 17:02:05 -0500 Subject: [IPython-dev] question about decision to use twisted In-Reply-To: <200807081659.46721.dsdale24@gmail.com> References: <200807081020.27330.dsdale24@gmail.com> <6ce0ac130807081257w17730d32y57c85f10eef48610@mail.gmail.com> <200807081659.46721.dsdale24@gmail.com> Message-ID: Darren Dale wrote: > After reading some positive things about twisted, I went looking for a couple > negative reviews. I've read that twisted is a bit of a moving target, There are parts that are a moving target, but the team is fairly rigorous about labeling those parts as unstable. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ellisonbg.net at gmail.com Tue Jul 8 19:36:33 2008 From: ellisonbg.net at gmail.com (Brian Granger) Date: Tue, 8 Jul 2008 16:36:33 -0700 Subject: [IPython-dev] question about decision to use twisted In-Reply-To: References: <200807081020.27330.dsdale24@gmail.com> <6ce0ac130807081257w17730d32y57c85f10eef48610@mail.gmail.com> Message-ID: <6ce0ac130807081636h5e2bc676xd344f1b0aed972e2@mail.gmail.com> > Just wanted to clarify Brian's comment... ipython1 is _now_ in trunk > IPython. Brian's done a ton of work to make this happen and I just > didn't want a typo to get in they way ;) Thanks Barry for catching this. Makes me think of Homer: I am so smart, SMRT. I am so smart, SMRT. From ellisonbg.net at gmail.com Tue Jul 8 19:55:06 2008 From: ellisonbg.net at gmail.com (Brian Granger) Date: Tue, 8 Jul 2008 16:55:06 -0700 Subject: [IPython-dev] question about decision to use twisted In-Reply-To: <200807081659.46721.dsdale24@gmail.com> References: <200807081020.27330.dsdale24@gmail.com> <6ce0ac130807081257w17730d32y57c85f10eef48610@mail.gmail.com> <200807081659.46721.dsdale24@gmail.com> Message-ID: <6ce0ac130807081655g72f528cehe329e3e8d15709b3@mail.gmail.com> > I was thinking of using twisted directly, but I guess I really need to dig > into ipython1. I don't have a sense of how I would use ipython1 in a > non-interactive way, is there some discussion in the docs or some examples of > scripting with ipython1? Is there a callback mechanism in ipython1 to respond > to non-blocking events? This is one aspect of our marketing that has been bad. We always talk about "interactive parallel computing", but IPython (because ipython1 has been merged into IPython, I will just talk about IPython from here on out) fully supports non-interactive usage. We have a full set of non-interactive examples that can be found here: Warning: these examples have not been merged from ipython1->IPython. Fernando and I are about to do that this week. As far as callbacks, we have two interfaces for everything: 1. A fully asynchronous version that uses twisted and all method return deferreds. This doesn't work interactively, but it what you want to use if you need to respond to non-blocking events. 2. A regular blocking interface that can be used interactively or not. This interface doesn't really block - it just lets you poll for asynchronous results. But, we fully supports both approaches and option 1 will fully plug into a regular Twisted based app. >> Unless >> you really want the treading API that processing provides, I think >> IPython is by far the best solution [I fully acknowledge my bias here] >> for high-level parallelism. >> >> > I would like to know what the ipython developers concerns were at the >> > time the decision was made to use twisted in ipython1, was there some >> > discussion on the ipython mailing lists? Any advice or comments would be >> > greatly appreciated. >> >> I don't think we really discussed this on the mailing list, it was >> probably private discussion between Fernando and myself. We went with >> Twisted as: >> >> 1) we didn't want to reinvent Twisted >> >> 2) we really need what Twisted provides, so the temptation to reinvent >> it was great. >> >> 3) Twisted has the right abstractions that force you to write correct >> an robust networking code. Of course you can do this without Twisted, >> but you are going to work much harder and probably get it wrong in >> subtle ways that are hard to figure out. >> >> We have been extremely happy about going with Twisted - it is our secret >> sauce. >> >> Are there any specific questions you have about Twisted? > > After reading some positive things about twisted, I went looking for a couple > negative reviews. I've read that twisted is a bit of a moving target, and the > twisted jargon is pretty unfamiliar to me. So finally I was hoping to temper > all this information with the informed opinion from a group I know and > respect. This is just the kind of feedback I was hoping to get. The Twisted dev team is borderline OCD about their code. Everything is tested or else removed, they do code review and have a great policy about API changes. In summary, while there are parts that are unstable, most of Twisted is rock solid and really robust. There are not many other open source projects that I would recommend as highly as Twisted in this respect. I am not a Twisted dev. Some people complain about the documentation, but I have never really found it to be a problem. With that said, it does take a while to really understand the model. But it is time well spent. Cheers, Brian > Thank you, > Darren > From edreamleo at gmail.com Wed Jul 9 06:58:34 2008 From: edreamleo at gmail.com (Edward K. Ream) Date: Wed, 9 Jul 2008 05:58:34 -0500 Subject: [IPython-dev] Getting rid of "print" in ipython In-Reply-To: References: <46cb515a0807040857w6252c7a4l71977155b2a668ed@mail.gmail.com> Message-ID: On Fri, Jul 4, 2008 at 2:35 PM, Fernando Perez wrote: > Howdy, > > On Fri, Jul 4, 2008 at 8:57 AM, Ville M. Vainio > wrote: > > IPython source code is riddled with direct calls to print statement. > > Obviously, we should get rid of them. > > Yup. > > > What's the exact plan for this? For the time being, we could get rid > > of them by introducing a global function for this (that initially just > > does normal "print", and add proper output handling later. > Does anyone know of a way to define the global function in a straightforward way? There appears to be no way to use 'print' in that function that will work on both Python 2.x and Python 3.x. My working assumption is that in order to preserve a common code base the function will have to use something like sys.stdout. Edward -------------------------------------------------------------------- Edward K. Ream email: edreamleo at gmail.com Leo: http://webpages.charter.net/edreamleo/front.html -------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From dsdale24 at gmail.com Wed Jul 9 09:26:08 2008 From: dsdale24 at gmail.com (Darren Dale) Date: Wed, 9 Jul 2008 09:26:08 -0400 Subject: [IPython-dev] furls Message-ID: <200807090926.10183.dsdale24@gmail.com> I am getting into ipython's parallel execution this morning, but I've run into a problem with missing furl files. The docs say: > As a user, you don?t need to know anything about the details of these > FURLs, other than that when the controller starts, it saves a set of FURLs > to files named something.furl. The default location of these files is your > ~./ipython directory. but when I run examples/kernel/helloworld.py, I get: ValueError: not a furl or a file containing a furl: /home/darren/.ipython/ipcontroller-tc.furl Probably it is something simple that broke during the transition from the ipython1 branch, and devs who already have furl files wouldnt have run into it. Darren From hans_meine at gmx.net Wed Jul 9 10:59:34 2008 From: hans_meine at gmx.net (Hans Meine) Date: Wed, 9 Jul 2008 16:59:34 +0200 Subject: [IPython-dev] Getting rid of "print" in ipython In-Reply-To: References: <46cb515a0807040857w6252c7a4l71977155b2a668ed@mail.gmail.com> <200807091337.04454.hans_meine@gmx.net> Message-ID: <200807091659.35233.hans_meine@gmx.net> Am Mittwoch, 09. Juli 2008 14:20:40 schrieb Edward K. Ream: > > [in reply to my private question what's wrong with sys.stdout.write] > Good question. There is nothing, per se, wrong with sys.std.write, but it > is not equivalent to print. In Python 3.x, the print *function* has > several keyword arguments, and the semantics of those arguments must be > handled by, say, IPython's pr function. What are those arguments? Where would they be needed in IPython? So far, the usual string interpolation techniques in connection with sys.stdout.write() have always been sufficient for me. Asking the Python folks for a transition method could make sense (although I am sure that the name sys.print_function will *not* be chosen ;-p ), but obviously, that is not relevant for IPython. Greetings, Hans From ellisonbg.net at gmail.com Wed Jul 9 12:45:28 2008 From: ellisonbg.net at gmail.com (Brian Granger) Date: Wed, 9 Jul 2008 09:45:28 -0700 Subject: [IPython-dev] furls In-Reply-To: <200807090926.10183.dsdale24@gmail.com> References: <200807090926.10183.dsdale24@gmail.com> Message-ID: <6ce0ac130807090945t23f1fc0fq9cded513a6f4c3d3@mail.gmail.com> Here is a bit more about how this works. The FURL stuff is new and we are working on the docs for it. As you can see, you have caught us right _before_ a stable release ;-) How are you staring the controller and engines? When the controller is started (either through ipcontroller or ipcluster, which calls ipcontroller on localhost), it should write the furl files to the ~./ipython dir on _that_ machine. If the engines or clint is running on a different machine, you will need to move the furl files to the ~./ipython dir on the machine that the client or engine is running on. The error you are getting makes it look like the controller is running on a different machine that the client and that the furl file for the client has not been moved to the ~./ipython dir of the clients host. Just to make sure everything is working, can you try to stat the controller and engines on localhost using: ipcluster -n 4 And then try to run helloworld.py on that same host? If that doesn't work, something else is going on and we can help you figure it out. The other thing to look at is the log files in ~./ipython/log. These should show if there are other odd things going on. Cheers, Brian On Wed, Jul 9, 2008 at 6:26 AM, Darren Dale wrote: > I am getting into ipython's parallel execution this morning, but I've run into > a problem with missing furl files. The docs say: > >> As a user, you don't need to know anything about the details of these >> FURLs, other than that when the controller starts, it saves a set of FURLs >> to files named something.furl. The default location of these files is your >> ~./ipython directory. > > but when I run examples/kernel/helloworld.py, I get: > > ValueError: not a furl or a file containing a > furl: /home/darren/.ipython/ipcontroller-tc.furl > > Probably it is something simple that broke during the transition from the > ipython1 branch, and devs who already have furl files wouldnt have run into > it. > > Darren > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev > From hans_meine at gmx.net Wed Jul 9 13:00:51 2008 From: hans_meine at gmx.net (Hans Meine) Date: Wed, 9 Jul 2008 19:00:51 +0200 Subject: [IPython-dev] Getting rid of "print" in ipython In-Reply-To: References: <46cb515a0807040857w6252c7a4l71977155b2a668ed@mail.gmail.com> <200807091659.35233.hans_meine@gmx.net> Message-ID: <200807091900.58132.hans_meine@gmx.net> On Mittwoch 09 Juli 2008, Edward K. Ream wrote: > On Wed, Jul 9, 2008 at 9:59 AM, Hans Meine wrote: > > What are those arguments? > > http://www.python.org/dev/peps/pep-3105/ > > def print(*args, sep=' ', end='\n', file=None) OK, so this is not really surprising, but I don't see which of the arguments could be needed in IPython. Actually, I like string interpolation even more than passing separate args, and one could even discuss the \n (e.g. whether to append it automatically or be explicit about it), but file is certainly not needed (it's up to the frontend IMO). So I would really propose to just use a minimum wrapper around sys.stdout/err.write, which is exactly what iplib already offers. :-) -- Ciao, / / .o. /--/ ..o / / ANS ooo -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part. URL: From dsdale24 at gmail.com Wed Jul 9 13:37:36 2008 From: dsdale24 at gmail.com (Darren Dale) Date: Wed, 9 Jul 2008 13:37:36 -0400 Subject: [IPython-dev] furls In-Reply-To: <6ce0ac130807090945t23f1fc0fq9cded513a6f4c3d3@mail.gmail.com> References: <200807090926.10183.dsdale24@gmail.com> <6ce0ac130807090945t23f1fc0fq9cded513a6f4c3d3@mail.gmail.com> Message-ID: <200807091337.37727.dsdale24@gmail.com> On Wednesday 09 July 2008 12:45:28 pm you wrote: > Here is a bit more about how this works. The FURL stuff is new and we > are working on the docs for it. As you can see, you have caught us > right _before_ a stable release ;-) > > How are you staring the controller and engines? I was just running the examples in the doc/examples/kernel directory, like: python helloworld.py > When the controller is started (either through ipcontroller or > ipcluster, which calls ipcontroller on localhost), it should write the > furl files to the ~./ipython dir on _that_ machine. If the engines or > clint is running on a different machine, you will need to move the > furl files to the ~./ipython dir on the machine that the client or > engine is running on. > > The error you are getting makes it look like the controller is running > on a different machine that the client and that the furl file for the > client has not been moved to the ~./ipython dir of the clients host. > > Just to make sure everything is working, can you try to stat the > controller and engines on localhost using: > > ipcluster -n 4 > > And then try to run helloworld.py on that same host? Ok. I didn't realize I had to run ipcluster first. Sorry for the trouble. Maybe a readme in examples/kernel would help provide a gentle introduction to new users. Darren From fperez.net at gmail.com Wed Jul 9 17:44:12 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 9 Jul 2008 14:44:12 -0700 Subject: [IPython-dev] Getting rid of "print" in ipython In-Reply-To: <200807091900.58132.hans_meine@gmx.net> References: <46cb515a0807040857w6252c7a4l71977155b2a668ed@mail.gmail.com> <200807091659.35233.hans_meine@gmx.net> <200807091900.58132.hans_meine@gmx.net> Message-ID: On Wed, Jul 9, 2008 at 10:00 AM, Hans Meine wrote: > On Mittwoch 09 Juli 2008, Edward K. Ream wrote: >> On Wed, Jul 9, 2008 at 9:59 AM, Hans Meine wrote: >> > What are those arguments? >> >> http://www.python.org/dev/peps/pep-3105/ >> >> def print(*args, sep=' ', end='\n', file=None) > > OK, so this is not really surprising, but I don't see which of the arguments > could be needed in IPython. Actually, I like string interpolation even more > than passing separate args, and one could even discuss the \n (e.g. whether > to append it automatically or be explicit about it), but file is certainly > not needed (it's up to the frontend IMO). > > So I would really propose to just use a minimum wrapper around > sys.stdout/err.write, which is exactly what iplib already offers. :-) Fortunately the current interface is almost a subset of the above, except for the file option and the fact that we don't append the \n. So if we decide to extend to a 3k-compatible print function, no code working today would break. We might want to add at least the 'end' option soon to keep the newline-handing interface future-proof. I still do like separate out/err functions rather than a keyword in this case. Cheers, f From gael.varoquaux at normalesup.org Thu Jul 10 23:48:49 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Fri, 11 Jul 2008 05:48:49 +0200 Subject: [IPython-dev] Tentative release plans... In-Reply-To: References: Message-ID: <20080711034849.GC10447@phare.normalesup.org> Hi guys, Regarding the release, Eric Jones just mentionned to me that he wanted to start teaching ipython1 for parallelisme in his classes. He bases his teaching on EPD (Enthought Python Distribution) and there should be a new release of EPD soon (early August). He would be very happy to include a new version of ipython, while it is still warm :). Ga?l From ellisonbg.net at gmail.com Fri Jul 11 13:24:19 2008 From: ellisonbg.net at gmail.com (Brian Granger) Date: Fri, 11 Jul 2008 10:24:19 -0700 Subject: [IPython-dev] Tentative release plans... In-Reply-To: <20080711034849.GC10447@phare.normalesup.org> References: <20080711034849.GC10447@phare.normalesup.org> Message-ID: <6ce0ac130807111024m1186abbat79bce090290834a0@mail.gmail.com> Hi, We would like to see if we can get a release out by a few days before August 1st. This should allow us to get this into the next EPD. I have just updated the tickets on launchpad and have marked the released blockers that I know of as critical. Can everyone look through the other tickets and mark as critical anything that must be done before the release? Also, if people know of un-ticketed items that need to be done as well, please file critical tickets soon as well. Also, this release will be 0.9. Cheers, Brian On Thu, Jul 10, 2008 at 8:48 PM, Gael Varoquaux wrote: > Hi guys, > > Regarding the release, Eric Jones just mentionned to me that he wanted to > start teaching ipython1 for parallelisme in his classes. He bases his > teaching on EPD (Enthought Python Distribution) and there should be a new > release of EPD soon (early August). He would be very happy to include a > new version of ipython, while it is still warm :). > > Ga?l > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev > From gael.varoquaux at normalesup.org Sat Jul 12 20:28:59 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 13 Jul 2008 02:28:59 +0200 Subject: [IPython-dev] Unused imports Message-ID: <20080713002859.GB5193@phare.normalesup.org> I am currently exploring the IPython codebase (I am looking at how to extract the line processing of IPython0 to port it in IPython1). We all know that the IPython codebase is a web of entangled object hard to break down. One thing that strikes me as a low hanging fruit to improve the situation, though, is cleaning up the imports. Almost every file I have looked at has a large amount of unused imports. If you use a powerful IDE like eclipse, you can configure it to show the unused imports, but if you like emacs or vim like me you can use pyflakes to check for these. Inspired by the mighty Robert Kern, I have binded one of my keys in vim to running the file in pyflakes. This is easy and allows to spot the unused imports in a split second. Ga?l From fperez.net at gmail.com Sun Jul 13 01:31:07 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 12 Jul 2008 22:31:07 -0700 Subject: [IPython-dev] Unused imports In-Reply-To: <20080713002859.GB5193@phare.normalesup.org> References: <20080713002859.GB5193@phare.normalesup.org> Message-ID: Hey, On Sat, Jul 12, 2008 at 5:28 PM, Gael Varoquaux wrote: > I am currently exploring the IPython codebase (I am looking at how to > extract the line processing of IPython0 to port it in IPython1). > > We all know that the IPython codebase is a web of entangled object hard > to break down. One thing that strikes me as a low hanging fruit to > improve the situation, though, is cleaning up the imports. Almost every > file I have looked at has a large amount of unused imports. Historical cruft left over after something gets split into two files, old imports stay behind. > If you use a powerful IDE like eclipse, you can configure it to show the > unused imports, but if you like emacs or vim like me you can use pyflakes > to check for these. Inspired by the mighty Robert Kern, I have binded one > of my keys in vim to running the file in pyflakes. This is easy and > allows to spot the unused imports in a split second. Go for it, by all means! Cheers, f From gael.varoquaux at normalesup.org Sun Jul 13 01:35:37 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 13 Jul 2008 07:35:37 +0200 Subject: [IPython-dev] Unused imports In-Reply-To: References: <20080713002859.GB5193@phare.normalesup.org> Message-ID: <20080713053537.GE5193@phare.normalesup.org> On Sat, Jul 12, 2008 at 10:31:07PM -0700, Fernando Perez wrote: > Go for it, by all means! Will do. Right now I am working in a branch, so the changes won't propagate to trunk immediately, but I have started cleaning up the output of pyflakes. See you, Ga?l From gael.varoquaux at normalesup.org Sun Jul 13 22:04:30 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 14 Jul 2008 04:04:30 +0200 Subject: [IPython-dev] Traceback trap in Interpreter Message-ID: <20080714020430.GF22362@phare.normalesup.org> Hi, I must be very dumb. I don't get how I am supposed to use the traceback trap in interpreter? Right now when I send to the interpreter something that generates a traceback, I get an exception, and the interpreter does not finish its execution procedure. As a result I don't get the traceback trapped in the return message. Maybe I am doing something stupid. My test code for this is: from IPython.kernel.core.interpreter import Interpreter i = Interpreter() i.execute('print i') Any clues? Cheers, Ga?l From mhansen at gmail.com Mon Jul 14 15:30:11 2008 From: mhansen at gmail.com (Mike Hansen) Date: Mon, 14 Jul 2008 14:30:11 -0500 Subject: [IPython-dev] IPython issues in Sage Message-ID: tHello, I have two issues that I'd like to report to see if someone knows what's going wrong since after looking at them for awhile, I'm stuck. 1) When Sage upgraded to IPython 0.8.2, the following trace command broke: import pdb import preprocessor def trace(code, preparse=True): code = preparser.preparse(code) return pdb.runeval(code) Before, it would have all the the global variables in the namespace that code was being run in. After the swtich to 0.8.2, none of the globals where present. I tried explicitly including ip.user_ns in the environment, but only sort of worked: see http://trac.sagemath.org/sage_trac/ticket/3345 2) I tried updating Sage to use IPython 0.8.4, and code in sage.misc.interpreter which refers to "_ip" now gives NameError: global name '_ip' is not defined Was this variable magically inserted into the namespace before? Anyways, if I add import IPython.ipapi _ip = IPython.ipapi.get() then I don't get the error about "_ip", but Sage quits without running the mainloop after starting up. This seems similar to what was happening here: http://trac.sagemath.org/sage_trac/ticket/1264 . I tried clearing out all old IPython modules from site-packages and then install 0.8.4 cleanly, but that didn't seem to help. Thanks, Mike From robert.kern at gmail.com Mon Jul 14 15:58:24 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 14 Jul 2008 14:58:24 -0500 Subject: [IPython-dev] Traceback trap in Interpreter In-Reply-To: <20080714020430.GF22362@phare.normalesup.org> References: <20080714020430.GF22362@phare.normalesup.org> Message-ID: Gael Varoquaux wrote: > Hi, > > I must be very dumb. I don't get how I am supposed to use the traceback > trap in interpreter? Right now when I send to the interpreter something > that generates a traceback, I get an exception, and the interpreter does > not finish its execution procedure. As a result I don't get the traceback > trapped in the return message. Maybe I am doing something stupid. My test > code for this is: > > from IPython.kernel.core.interpreter import Interpreter > i = Interpreter() > i.execute('print i') i.execute('print i', raiseException=False) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From gael.varoquaux at normalesup.org Mon Jul 14 19:38:44 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 15 Jul 2008 01:38:44 +0200 Subject: [IPython-dev] Traceback trap in Interpreter In-Reply-To: References: <20080714020430.GF22362@phare.normalesup.org> Message-ID: <20080714233844.GA14606@phare.normalesup.org> On Mon, Jul 14, 2008 at 02:58:24PM -0500, Robert Kern wrote: > i.execute('print i', raiseException=False) Hey Robert, thanks a lot. I can't believe I managed to miss this. Ga?l From vivainio at gmail.com Tue Jul 15 03:40:15 2008 From: vivainio at gmail.com (Ville M. Vainio) Date: Tue, 15 Jul 2008 10:40:15 +0300 Subject: [IPython-dev] IPython issues in Sage In-Reply-To: References: Message-ID: <46cb515a0807150040k44cdf434he63ab3cce9d1e940@mail.gmail.com> On Mon, Jul 14, 2008 at 10:30 PM, Mike Hansen wrote: Just a quick answer: > 2) I tried updating Sage to use IPython 0.8.4, and code in > sage.misc.interpreter which refers to "_ip" now gives > > NameError: global name '_ip' is not defined > > Was this variable magically inserted into the namespace before? Yes, and still is. IPython and depends on it to run at all. Perhaps you are looking in a wrong namespace dict? -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From mhansen at gmail.com Tue Jul 15 03:48:49 2008 From: mhansen at gmail.com (Mike Hansen) Date: Tue, 15 Jul 2008 02:48:49 -0500 Subject: [IPython-dev] IPython issues in Sage In-Reply-To: <46cb515a0807150040k44cdf434he63ab3cce9d1e940@mail.gmail.com> References: <46cb515a0807150040k44cdf434he63ab3cce9d1e940@mail.gmail.com> Message-ID: > Yes, and still is. IPython and depends on it to run at all. Perhaps > you are looking in a wrong namespace dict? I'm not sure what you mean by "looking in a wrong namespace dict". There was code that used the _ip variable that was working fine with IPython 0.8.2, but when I upgraded to 0.8.4, it gives me a NameError. --Mike From vivainio at gmail.com Tue Jul 15 04:21:26 2008 From: vivainio at gmail.com (Ville M. Vainio) Date: Tue, 15 Jul 2008 11:21:26 +0300 Subject: [IPython-dev] IPython issues in Sage In-Reply-To: References: <46cb515a0807150040k44cdf434he63ab3cce9d1e940@mail.gmail.com> Message-ID: <46cb515a0807150121r6e4f21fah1c27cda7112677f7@mail.gmail.com> On Tue, Jul 15, 2008 at 10:48 AM, Mike Hansen wrote: >> Yes, and still is. IPython and depends on it to run at all. Perhaps >> you are looking in a wrong namespace dict? > > I'm not sure what you mean by "looking in a wrong namespace dict". > There was code that used the _ip variable that was working fine with > IPython 0.8.2, but when I upgraded to 0.8.4, it gives me a NameError. Yeah, but perhaps it didn't look for it in the right place? It certainly is in the IPython user namespace. However, it's not in __builtin__ namespace anymore (it caused various problems when used from there, can't remember the exact bugs that were fixed by that). You need to acquire the _ip object by using IPython.ipapi.get() if you are not operating within ipython user namespace. -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From mhansen at gmail.com Tue Jul 15 04:26:10 2008 From: mhansen at gmail.com (Mike Hansen) Date: Tue, 15 Jul 2008 03:26:10 -0500 Subject: [IPython-dev] IPython issues in Sage In-Reply-To: <46cb515a0807150121r6e4f21fah1c27cda7112677f7@mail.gmail.com> References: <46cb515a0807150040k44cdf434he63ab3cce9d1e940@mail.gmail.com> <46cb515a0807150121r6e4f21fah1c27cda7112677f7@mail.gmail.com> Message-ID: > Yeah, but perhaps it didn't look for it in the right place? It > certainly is in the IPython user namespace. However, it's not in > __builtin__ namespace anymore (it caused various problems when used > from there, can't remember the exact bugs that were fixed by that). > You need to acquire the _ip object by using IPython.ipapi.get() if you > are not operating within ipython user namespace. In my original message, I mentioned that I tried that and it eliminated the NameError, but then I get the issue where the main loop is never run. Which is similar to what is happening here: http://trac.sagemath.org/sage_trac/ticket/1264 . I wasn't able to figure out what Fernando did the last time to get rid of the problem. --Mike From vivainio at gmail.com Tue Jul 15 15:37:50 2008 From: vivainio at gmail.com (Ville M. Vainio) Date: Tue, 15 Jul 2008 22:37:50 +0300 Subject: [IPython-dev] IPython issues in Sage In-Reply-To: References: <46cb515a0807150040k44cdf434he63ab3cce9d1e940@mail.gmail.com> <46cb515a0807150121r6e4f21fah1c27cda7112677f7@mail.gmail.com> Message-ID: <46cb515a0807151237ufed29bcl3ca4229ec2c03155@mail.gmail.com> On Tue, Jul 15, 2008 at 11:26 AM, Mike Hansen wrote: > In my original message, I mentioned that I tried that and it > eliminated the NameError, but then I get the issue where the main loop > is never run. Which is similar to what is happening here: These problems are probably unrelated. > http://trac.sagemath.org/sage_trac/ticket/1264 . I wasn't able to > figure out what Fernando did the last time to get rid of the problem. That report seems to indicate that it happen because sage launches ipython so that it enters "batch" mode and exits ipython. See the end of "post_config_initialization" in iplib.py. Your problem probably is that you are launching ipython the wrong way, so that "rc.interact" option is not set. The old behaviour was broken for the documented functionality of command line args -i and -c (possibly the functionality was broken exactly for the benefit of sage). How do you launch IPython from sage, exactly? Is sys.argv clean? Does it have -c without -i? Also, have you considered making a "sage profile" (ipy_profile_sage.py) that would launch ipython first and only bring in sage later on? -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From vivainio at gmail.com Tue Jul 15 15:57:36 2008 From: vivainio at gmail.com (Ville M. Vainio) Date: Tue, 15 Jul 2008 22:57:36 +0300 Subject: [IPython-dev] Unused imports In-Reply-To: <20080713002859.GB5193@phare.normalesup.org> References: <20080713002859.GB5193@phare.normalesup.org> Message-ID: <46cb515a0807151257x68740df5x21c63b8f0f04fb7b@mail.gmail.com> On Sun, Jul 13, 2008 at 3:28 AM, Gael Varoquaux wrote: > We all know that the IPython codebase is a web of entangled object hard > to break down. One thing that strikes me as a low hanging fruit to > improve the situation, though, is cleaning up the imports. Almost every > file I have looked at has a large amount of unused imports. Other low-hanging fruit in related vein is splitting up genutils to smaller parts. Most of the circular dependency problems I've had were because of genutils. When working with input history, be careful to observe that raw history and "native history" stay in sync. I would be in favor of ditching the native history completely, since it would appear to be quite unnecessary apart from debugging why certain statements do not work (i.e. it's more interesting to developers than end users). -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From strawman at astraw.com Tue Jul 15 20:01:03 2008 From: strawman at astraw.com (Andrew Straw) Date: Tue, 15 Jul 2008 17:01:03 -0700 Subject: [IPython-dev] pushing python class definition to engines Message-ID: <487D3A3F.4000304@astraw.com> After seeing Brian Granger's great demo of the parallel abilities in IPython last week at the SIAM conference, I'm playing with ipython trunk which I just pulled out of launchpad (for this reason, I'm posting on the -dev list). Anyhow, if I start a client in the usual way: from IPython.kernel import client mec = client.MultiEngineClient() I can easily push Python integers: a=5 mec['a'] = a print mec.gather('a') But pushing classes doesn't work: class MyObj(object): def __init__(self,x): print 'PID',os.getpid() self.x=x mec['MyObj'] = MyObj print mec.gather('MyObj') That gives the following error. How can I push a class definition to my engines? Traceback (most recent call last): File "flydra/test_ipython.py", line 14, in mec.push_function(dict(MyObj= MyObj,)) File "/home/astraw/PY_ipython/lib/python2.5/site-packages/IPython/kernel/multiengineclient.py", line 606, in push_function return self._blockFromThread(self.smultiengine.push_function, namespace, targets=targets, block=block) File "/home/astraw/PY_ipython/lib/python2.5/site-packages/IPython/kernel/multiengineclient.py", line 456, in _blockFromThread result = blockingCallFromThread(function, *args, **kwargs) File "/home/astraw/PY_ipython/lib/python2.5/site-packages/IPython/kernel/twistedutil.py", line 69, in blockingCallFromThread return twisted.internet.threads.blockingCallFromThread(reactor, f, *a, **kw) File "/home/astraw/PY_ipython/lib/python2.5/site-packages/Twisted-8.1.0-py2.5-linux-x86_64.egg/twisted/internet/threads.py", line 83, in blockingCallFromThread result.raiseException() File "/home/astraw/PY_ipython/lib/python2.5/site-packages/Twisted-8.1.0-py2.5-linux-x86_64.egg/twisted/python/failure.py", line 319, in raiseException raise self.type, self.value, self.tb AttributeError: 'module' object has no attribute 'MyObj' From robert.kern at gmail.com Tue Jul 15 20:16:39 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 15 Jul 2008 19:16:39 -0500 Subject: [IPython-dev] pushing python class definition to engines In-Reply-To: <487D3A3F.4000304@astraw.com> References: <487D3A3F.4000304@astraw.com> Message-ID: Andrew Straw wrote: > After seeing Brian Granger's great demo of the parallel abilities in > IPython last week at the SIAM conference, I'm playing with ipython trunk > which I just pulled out of launchpad (for this reason, I'm posting on > the -dev list). > > Anyhow, if I start a client in the usual way: > > from IPython.kernel import client > mec = client.MultiEngineClient() > > I can easily push Python integers: > > a=5 > mec['a'] = a > print mec.gather('a') > > But pushing classes doesn't work: > > class MyObj(object): > def __init__(self,x): > print 'PID',os.getpid() > self.x=x > > mec['MyObj'] = MyObj > print mec.gather('MyObj') > > That gives the following error. How can I push a class definition to my > engines? It's really tricky since class objects normally don't really serialize. Pickles just contain a reference to the fully-qualified name of the class. In this case, it's __main__.MyObj. Since there is no MyObj in __main__ in your remote namespace, you get the error you see. You will have to push actual code to your remote kernels. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From strawman at astraw.com Tue Jul 15 20:30:35 2008 From: strawman at astraw.com (Andrew Straw) Date: Tue, 15 Jul 2008 17:30:35 -0700 Subject: [IPython-dev] pushing python class definition to engines In-Reply-To: References: <487D3A3F.4000304@astraw.com> Message-ID: <487D412B.9040709@astraw.com> Robert Kern wrote: > Andrew Straw wrote: > >> After seeing Brian Granger's great demo of the parallel abilities in >> IPython last week at the SIAM conference, I'm playing with ipython trunk >> which I just pulled out of launchpad (for this reason, I'm posting on >> the -dev list). >> >> Anyhow, if I start a client in the usual way: >> >> from IPython.kernel import client >> mec = client.MultiEngineClient() >> >> I can easily push Python integers: >> >> a=5 >> mec['a'] = a >> print mec.gather('a') >> >> But pushing classes doesn't work: >> >> class MyObj(object): >> def __init__(self,x): >> print 'PID',os.getpid() >> self.x=x >> >> mec['MyObj'] = MyObj >> print mec.gather('MyObj') >> >> That gives the following error. How can I push a class definition to my >> engines? >> > > It's really tricky since class objects normally don't really serialize. Pickles > just contain a reference to the fully-qualified name of the class. In this case, > it's __main__.MyObj. Since there is no MyObj in __main__ in your remote > namespace, you get the error you see. You will have to push actual code to your > remote kernels. > OK. Thanks for the info. How do I do that/what do you mean? (Do I simply make a string with my class definition and push that across?) -Andrew From robert.kern at gmail.com Tue Jul 15 20:39:05 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 15 Jul 2008 19:39:05 -0500 Subject: [IPython-dev] pushing python class definition to engines In-Reply-To: <487D412B.9040709@astraw.com> References: <487D3A3F.4000304@astraw.com> <487D412B.9040709@astraw.com> Message-ID: Andrew Straw wrote: > Robert Kern wrote: >> Andrew Straw wrote: >> >>> After seeing Brian Granger's great demo of the parallel abilities in >>> IPython last week at the SIAM conference, I'm playing with ipython trunk >>> which I just pulled out of launchpad (for this reason, I'm posting on >>> the -dev list). >>> >>> Anyhow, if I start a client in the usual way: >>> >>> from IPython.kernel import client >>> mec = client.MultiEngineClient() >>> >>> I can easily push Python integers: >>> >>> a=5 >>> mec['a'] = a >>> print mec.gather('a') >>> >>> But pushing classes doesn't work: >>> >>> class MyObj(object): >>> def __init__(self,x): >>> print 'PID',os.getpid() >>> self.x=x >>> >>> mec['MyObj'] = MyObj >>> print mec.gather('MyObj') >>> >>> That gives the following error. How can I push a class definition to my >>> engines? >>> >> It's really tricky since class objects normally don't really serialize. Pickles >> just contain a reference to the fully-qualified name of the class. In this case, >> it's __main__.MyObj. Since there is no MyObj in __main__ in your remote >> namespace, you get the error you see. You will have to push actual code to your >> remote kernels. >> > OK. Thanks for the info. How do I do that/what do you mean? (Do I simply > make a string with my class definition and push that across?) mec.execute('class MyObj(object):\n pass') for example. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ellisonbg.net at gmail.com Tue Jul 15 22:36:45 2008 From: ellisonbg.net at gmail.com (Brian Granger) Date: Tue, 15 Jul 2008 20:36:45 -0600 Subject: [IPython-dev] pushing python class definition to engines In-Reply-To: <487D3A3F.4000304@astraw.com> References: <487D3A3F.4000304@astraw.com> Message-ID: <6ce0ac130807151936o2b8fe47do7ee7900d8df62e4f@mail.gmail.com> Andrew, As Robert mentioned, this is really a limitation in how Python serializes classes and instances. I have dealt with this in a couple of ways: 1. Don't use classes :) More, seriously though, functions, even those defined interactively can be pushed and pulled using push_function and pull_function. There are only a few limitations, like no closures. 2. Define your classes in a standalone module that can be imported by the engines, controller and client. Then, you can push and pull instances just fine using push and pull. This is mostly how I work - I develop classes in serial first, put them in a module and then simply import. Ironically, functions work better for this than classes. Until Python has a different way of serializing classes and instances, we are stuck. Cheers, Brian On Tue, Jul 15, 2008 at 6:01 PM, Andrew Straw wrote: > After seeing Brian Granger's great demo of the parallel abilities in > IPython last week at the SIAM conference, I'm playing with ipython trunk > which I just pulled out of launchpad (for this reason, I'm posting on > the -dev list). > > Anyhow, if I start a client in the usual way: > > from IPython.kernel import client > mec = client.MultiEngineClient() > > I can easily push Python integers: > > a=5 > mec['a'] = a > print mec.gather('a') > > But pushing classes doesn't work: > > class MyObj(object): > def __init__(self,x): > print 'PID',os.getpid() > self.x=x > > mec['MyObj'] = MyObj > print mec.gather('MyObj') > > That gives the following error. How can I push a class definition to my > engines? > > Traceback (most recent call last): > File "flydra/test_ipython.py", line 14, in > mec.push_function(dict(MyObj= MyObj,)) > File > "/home/astraw/PY_ipython/lib/python2.5/site-packages/IPython/kernel/multiengineclient.py", > line 606, in push_function > return self._blockFromThread(self.smultiengine.push_function, > namespace, targets=targets, block=block) > File > "/home/astraw/PY_ipython/lib/python2.5/site-packages/IPython/kernel/multiengineclient.py", > line 456, in _blockFromThread > result = blockingCallFromThread(function, *args, **kwargs) > File > "/home/astraw/PY_ipython/lib/python2.5/site-packages/IPython/kernel/twistedutil.py", > line 69, in blockingCallFromThread > return twisted.internet.threads.blockingCallFromThread(reactor, f, > *a, **kw) > File > "/home/astraw/PY_ipython/lib/python2.5/site-packages/Twisted-8.1.0-py2.5-linux-x86_64.egg/twisted/internet/threads.py", > line 83, in blockingCallFromThread > result.raiseException() > File > "/home/astraw/PY_ipython/lib/python2.5/site-packages/Twisted-8.1.0-py2.5-linux-x86_64.egg/twisted/python/failure.py", > line 319, in raiseException > raise self.type, self.value, self.tb > AttributeError: 'module' object has no attribute 'MyObj' > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev > From fperez.net at gmail.com Wed Jul 16 02:23:54 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 15 Jul 2008 23:23:54 -0700 Subject: [IPython-dev] Unused imports In-Reply-To: <46cb515a0807151257x68740df5x21c63b8f0f04fb7b@mail.gmail.com> References: <20080713002859.GB5193@phare.normalesup.org> <46cb515a0807151257x68740df5x21c63b8f0f04fb7b@mail.gmail.com> Message-ID: On Tue, Jul 15, 2008 at 12:57 PM, Ville M. Vainio wrote: > When working with input history, be careful to observe that raw > history and "native history" stay in sync. I would be in favor of > ditching the native history completely, since it would appear to be > quite unnecessary apart from debugging why certain statements do not > work (i.e. it's more interesting to developers than end users). I seem to recall Sage needing it, but I could be wrong. It's been a while, but let's not pull it out without confirmation that they don't need it. Cheers, f From fperez.net at gmail.com Wed Jul 16 02:31:26 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 15 Jul 2008 23:31:26 -0700 Subject: [IPython-dev] IPython issues in Sage In-Reply-To: References: <46cb515a0807150040k44cdf434he63ab3cce9d1e940@mail.gmail.com> Message-ID: On Tue, Jul 15, 2008 at 12:48 AM, Mike Hansen wrote: >> Yes, and still is. IPython and depends on it to run at all. Perhaps >> you are looking in a wrong namespace dict? > > I'm not sure what you mean by "looking in a wrong namespace dict". > There was code that used the _ip variable that was working fine with > IPython 0.8.2, but when I upgraded to 0.8.4, it gives me a NameError. Ville's explanation is correct, I'll just give you a tiny bit of detail to clarify things further: ipython executes all user code in a special dict called user_ns. It *used* to also populate the builtin namespace with a few things, but we're trying to move away as much as possible from this, because having system-wide globals causes all kinds of other problems. Sorry if this one bit you, since it was an API change, but it was for the best. Let us know if Ville's other suggestions don't fully solve your problem. Cheers, f From mhansen at gmail.com Wed Jul 16 02:38:09 2008 From: mhansen at gmail.com (Mike Hansen) Date: Wed, 16 Jul 2008 01:38:09 -0500 Subject: [IPython-dev] IPython issues in Sage In-Reply-To: References: <46cb515a0807150040k44cdf434he63ab3cce9d1e940@mail.gmail.com> Message-ID: > Ville's explanation is correct, I'll just give you a tiny bit of > detail to clarify things further: ipython executes all user code in a > special dict called user_ns. It *used* to also populate the builtin > namespace with a few things, but we're trying to move away as much as > possible from this, because having system-wide globals causes all > kinds of other problems. > > Sorry if this one bit you, since it was an API change, but it was for the best. > > Let us know if Ville's other suggestions don't fully solve your problem. Yep, that worked. Thanks to both of you for the explanation. --Mike From vivainio at gmail.com Wed Jul 16 15:09:06 2008 From: vivainio at gmail.com (Ville M. Vainio) Date: Wed, 16 Jul 2008 22:09:06 +0300 Subject: [IPython-dev] IPython issues in Sage In-Reply-To: References: <46cb515a0807150040k44cdf434he63ab3cce9d1e940@mail.gmail.com> Message-ID: <46cb515a0807161209y2a89d071q3762e8b68f8f1965@mail.gmail.com> On Wed, Jul 16, 2008 at 9:38 AM, Mike Hansen wrote: >> Ville's explanation is correct, I'll just give you a tiny bit of >> detail to clarify things further: ipython executes all user code in a >> special dict called user_ns. It *used* to also populate the builtin >> namespace with a few things, but we're trying to move away as much as >> possible from this, because having system-wide globals causes all >> kinds of other problems. >> >> Sorry if this one bit you, since it was an API change, but it was for the best. >> >> Let us know if Ville's other suggestions don't fully solve your problem. > > Yep, that worked. Thanks to both of you for the explanation. Please explain the answer you found here as well, for the benefit of search engines and others who could bump on the same problem. -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From gael.varoquaux at normalesup.org Sun Jul 20 12:46:15 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 20 Jul 2008 18:46:15 +0200 Subject: [IPython-dev] ipython1 and synchronous printing of stdout Message-ID: <20080720164615.GB23971@phare.normalesup.org> Hi, I am making progress on my graphical fronend for ipython1, but I have hit an interesting small difficulty: currently the way the interreter is set, it traps all output, and returns it at the end of the execution. This is quite bad if a users is (in a synchronous, local workflow) running a long job, and printing information on the advancment of his job, say for instance: for i in range(10): print i sleep(1) I can off course monkey patch the output_trap with my own write method, but this seems a bit ugly. What do the ipython1 developpers think the right way of doing this (and maybe modifying output_trap) is? Cheers, Ga?l From gael.varoquaux at normalesup.org Sun Jul 20 15:27:02 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 20 Jul 2008 21:27:02 +0200 Subject: [IPython-dev] ipython1 and synchronous printing of stdout In-Reply-To: <20080720164615.GB23971@phare.normalesup.org> References: <20080720164615.GB23971@phare.normalesup.org> Message-ID: <20080720192702.GH23971@phare.normalesup.org> > I can off course monkey patch the output_trap with my own write method, > but this seems a bit ugly. What do the ipython1 developpers think the > right way of doing this (and maybe modifying output_trap) is? I went ahead experimenting, and created a subclass of output_trap gear towards printing via a callback, as can be seen on http://bazaar.launchpad.net/~gael-varoquaux/ipython/ipython-sync-frontend/annotate/1041?file_id=sync_output_trap.py-20080720174256-b1fz1ebbxl2ka4t9-1 I hope this approach is OK. By passing this this output_trap to the interpreter I get the desired result, and I sync this approach is reusable for any synchronous frontend, such as the one into which ipython0 will, hopefully, morph one day. Ga?l From ellisonbg.net at gmail.com Sun Jul 20 15:50:01 2008 From: ellisonbg.net at gmail.com (Brian Granger) Date: Sun, 20 Jul 2008 13:50:01 -0600 Subject: [IPython-dev] ipython1 and synchronous printing of stdout In-Reply-To: <20080720192702.GH23971@phare.normalesup.org> References: <20080720164615.GB23971@phare.normalesup.org> <20080720192702.GH23971@phare.normalesup.org> Message-ID: <6ce0ac130807201250m26fb1a5flcbc9bdb62844cda8@mail.gmail.com> I don't have any time to look at this now, but we should have an interface in the interpreter that allow the registration of a callback that gets called or these things. Cheers, Brian On Sun, Jul 20, 2008 at 1:27 PM, Gael Varoquaux wrote: >> I can off course monkey patch the output_trap with my own write method, >> but this seems a bit ugly. What do the ipython1 developpers think the >> right way of doing this (and maybe modifying output_trap) is? > > I went ahead experimenting, and created a subclass of output_trap gear > towards printing via a callback, as can be seen on > http://bazaar.launchpad.net/~gael-varoquaux/ipython/ipython-sync-frontend/annotate/1041?file_id=sync_output_trap.py-20080720174256-b1fz1ebbxl2ka4t9-1 > > I hope this approach is OK. By passing this this output_trap to the > interpreter I get the desired result, and I sync this approach is > reusable for any synchronous frontend, such as the one into which > ipython0 will, hopefully, morph one day. > > Ga?l > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev > From laurent.dufrechou at gmail.com Sun Jul 20 15:56:23 2008 From: laurent.dufrechou at gmail.com (Laurent Dufrechou) Date: Sun, 20 Jul 2008 21:56:23 +0200 Subject: [IPython-dev] ipython1 and synchronous printing of stdout In-Reply-To: <20080720192702.GH23971@phare.normalesup.org> References: <20080720164615.GB23971@phare.normalesup.org> <20080720192702.GH23971@phare.normalesup.org> Message-ID: <7ced318f0807201256i7fb5826ah21455c5faf9a92b@mail.gmail.com> Hi gael, I've checked out your branch and done `python setup.py build_ext --inplace` Launching your app: laurent at Pep:~/ipython-sync-frontend/IPython$ python ./frontend/wx/wx_frontend.py Traceback (most recent call last): File "./frontend/wx/wx_frontend.py", line 30, in from IPython.frontend.prefilterfrontend import PrefilterFrontEnd ImportError: No module named frontend.prefilterfrontend Am I missing something? laurent at Pep:~/ipython-sync-frontend/IPython/frontend$ ls asyncfrontendbase.py cocoa frontendbase.py __init__.py linefrontendbase.py tests wx Laurent 2008/7/20 Gael Varoquaux : >> I can off course monkey patch the output_trap with my own write method, >> but this seems a bit ugly. What do the ipython1 developpers think the >> right way of doing this (and maybe modifying output_trap) is? > > I went ahead experimenting, and created a subclass of output_trap gear > towards printing via a callback, as can be seen on > http://bazaar.launchpad.net/~gael-varoquaux/ipython/ipython-sync-frontend/annotate/1041?file_id=sync_output_trap.py-20080720174256-b1fz1ebbxl2ka4t9-1 > > I hope this approach is OK. By passing this this output_trap to the > interpreter I get the desired result, and I sync this approach is > reusable for any synchronous frontend, such as the one into which > ipython0 will, hopefully, morph one day. > > Ga?l > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev > From gael.varoquaux at normalesup.org Sun Jul 20 16:03:50 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 20 Jul 2008 22:03:50 +0200 Subject: [IPython-dev] ipython1 and synchronous printing of stdout In-Reply-To: <6ce0ac130807201250m26fb1a5flcbc9bdb62844cda8@mail.gmail.com> References: <20080720164615.GB23971@phare.normalesup.org> <20080720192702.GH23971@phare.normalesup.org> <6ce0ac130807201250m26fb1a5flcbc9bdb62844cda8@mail.gmail.com> Message-ID: <20080720200350.GA31836@phare.normalesup.org> On Sun, Jul 20, 2008 at 01:50:01PM -0600, Brian Granger wrote: > I don't have any time to look at this now, but we should have an > interface in the interpreter that allow the registration of a callback > that gets called or these things. OK. Well, there is no hurry, but when you have time it would be nice to agree on an interface. For now my branch is not merge into trunk. Ga?l From gael.varoquaux at normalesup.org Sun Jul 20 16:07:12 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 20 Jul 2008 22:07:12 +0200 Subject: [IPython-dev] ipython1 and synchronous printing of stdout In-Reply-To: <7ced318f0807201256i7fb5826ah21455c5faf9a92b@mail.gmail.com> References: <20080720164615.GB23971@phare.normalesup.org> <20080720192702.GH23971@phare.normalesup.org> <7ced318f0807201256i7fb5826ah21455c5faf9a92b@mail.gmail.com> Message-ID: <20080720200712.GB31836@phare.normalesup.org> On Sun, Jul 20, 2008 at 09:56:23PM +0200, Laurent Dufrechou wrote: > Hi gael, > I've checked out your branch and done `python setup.py build_ext --inplace` > Launching your app: > laurent at Pep:~/ipython-sync-frontend/IPython$ python > ./frontend/wx/wx_frontend.py > Traceback (most recent call last): > File "./frontend/wx/wx_frontend.py", line 30, in > from IPython.frontend.prefilterfrontend import PrefilterFrontEnd > ImportError: No module named frontend.prefilterfrontend Hey Laurent, Thanks a lot for taking time to check this out. I had forgetten to add a file in bzr. I have just corrected that and pushed a new version. The prefilterfrontend.py file is the file where are concentred all the really hacky tricks I have to play to get ipython1 working with magics, tab completion, and all that. This is what Fernando and I are going to use as a guide to drive our refactoring of ipython0 and ipython1 to get the magics and co cleanely in ipython1. Ga?l From gael.varoquaux at normalesup.org Sun Jul 20 16:14:14 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 20 Jul 2008 22:14:14 +0200 Subject: [IPython-dev] ipython1 and synchronous printing of stdout In-Reply-To: <7ced318f0807201256i7fb5826ah21455c5faf9a92b@mail.gmail.com> References: <20080720164615.GB23971@phare.normalesup.org> <20080720192702.GH23971@phare.normalesup.org> <7ced318f0807201256i7fb5826ah21455c5faf9a92b@mail.gmail.com> Message-ID: <20080720201414.GC31836@phare.normalesup.org> On Sun, Jul 20, 2008 at 09:56:23PM +0200, Laurent Dufrechou wrote: > I've checked out your branch and done `python setup.py build_ext --inplace` > Launching your app: Oh, by the way, Laurent, be careful, the frontend is currently not at all windows friendly. When executing external commands that might ask for terminal input there is a severe gotcha that I haven't managed to work around (namely: how to you redirect stdin, stout, and stderr simultanous to a process). As a result I falled back on starting an xterm. obvious this works only on Linux. I am on conference right now, and do not have windows or Macs to get a proper solution. I'll work on that when I get back in Austin. Cheers, Ga?l From laurent.dufrechou at gmail.com Sun Jul 20 16:16:52 2008 From: laurent.dufrechou at gmail.com (Laurent Dufrechou) Date: Sun, 20 Jul 2008 22:16:52 +0200 Subject: [IPython-dev] ipython1 and synchronous printing of stdout In-Reply-To: <20080720201414.GC31836@phare.normalesup.org> References: <20080720164615.GB23971@phare.normalesup.org> <20080720192702.GH23971@phare.normalesup.org> <7ced318f0807201256i7fb5826ah21455c5faf9a92b@mail.gmail.com> <20080720201414.GC31836@phare.normalesup.org> Message-ID: <7ced318f0807201316v6faf9d47j9237bfb9252455f4@mail.gmail.com> No problem, I'm under linux rigth now ;) 2008/7/20 Gael Varoquaux : > On Sun, Jul 20, 2008 at 09:56:23PM +0200, Laurent Dufrechou wrote: >> I've checked out your br anch and done `python setup.py build_ext --inplace` >> Launching your app: > > Oh, by the way, Laurent, be careful, the frontend is currently not at all > windows friendly. When executing external commands that might ask for > terminal input there is a severe gotcha that I haven't managed to work > around (namely: how to you redirect stdin, stout, and stderr simultanous > to a process). As a result I falled back on starting an xterm. obvious > this works only on Linux. I am on conference right now, and do not have > windows or Macs to get a proper solution. I'll work on that when I get > back in Austin. > > Cheers, > > Ga?l > From gael.varoquaux at normalesup.org Sun Jul 20 16:54:29 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 20 Jul 2008 22:54:29 +0200 Subject: [IPython-dev] Nicer display of exceptions with ipython1 Message-ID: <20080720205429.GD31836@phare.normalesup.org> Currently, with the ipython1 interpreter, there are two ways of dealing with exception: either you let the interpreter deal with them, and you get the traceback in the return message, or you ask the interpreter to raise them, and you deal with them yourself. I have been doing the second, in my frontend, because I want my frontend to be able to display more information about the traceback that there is in the message. The problem with this is that I have to catch the exception outside of the interpreter, thus the callstack contains irrelevant information, for instance this is the kind of session I might have: In [1]: print i --------------------------------------------------------------------------- NameError Traceback (most recent call last) /home/varoquau/dev/ipython/ipython-frontend/IPython/kernel/core/interpreter.pyc in execute_block(self, code) 373 outflag = 1 # start by assuming error, success will reset it 374 try: --> 375 exec code in self.user_ns 376 outflag = 00 377 except SystemExit :/home/varoquau/dev/ipython/ipython-frontend/IPython/frontend/wx/ () in ----> 1 2 3 4 5 NameError It seems to me that an elegant way to resolve this problem would be to provide a callback to the interpreter to deal with traceback. I am looking at the traceback_trap and trying to figure out if what I want fits in this model or not, and I am not too sure it does. I think I want something more flexible. Should I go ahead and implement a callback somewhere? And if so, where? In the traceback trap? Cheers, Ga?l From ellisonbg.net at gmail.com Sun Jul 20 17:52:05 2008 From: ellisonbg.net at gmail.com (Brian Granger) Date: Sun, 20 Jul 2008 15:52:05 -0600 Subject: [IPython-dev] Nicer display of exceptions with ipython1 In-Reply-To: <20080720205429.GD31836@phare.normalesup.org> References: <20080720205429.GD31836@phare.normalesup.org> Message-ID: <6ce0ac130807201452u7fdff84agbd409ed14f326951@mail.gmail.com> Yes, this is a generic problem that we have in IPython.kernel - all of the user's exceptions get polluted with information from IPython itself. > It seems to me that an elegant way to resolve this problem would be to > provide a callback to the interpreter to deal with traceback. I am > looking at the traceback_trap and trying to figure out if what I want > fits in this model or not, and I am not too sure it does. I think I want > something more flexible. In this case, I am not sure that a callback will really do what we want. One reason we have the option to allow the interpreter actually raise the exception is that is how Python handles errors. If we move things to a callback, we sort of get stuck with our own non-exception based error handling. That would be very non-ideal. I am not sure if it is possible, but I think the best solution is if the traceback_trap can be smart and dynamically modify the traceback to remove the unwanted information. This is also needed even if the interpreter is not raising the exception. In that case, the interpreter returns a dict that has the traceback as a string - with the unwanted info as well. Conclusion: we need to figure out a way of remove the unwanted info in the first place. Also, isn't this what the callback would have to do anyway? If so, that logic should just be in the traceback_trap itself. Cheers, Brian > Should I go ahead and implement a callback somewhere? And if so, where? > In the traceback trap? > Cheers, > > Ga?l > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev > From gael.varoquaux at normalesup.org Sun Jul 20 18:13:53 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 21 Jul 2008 00:13:53 +0200 Subject: [IPython-dev] Nicer display of exceptions with ipython1 In-Reply-To: <6ce0ac130807201452u7fdff84agbd409ed14f326951@mail.gmail.com> References: <20080720205429.GD31836@phare.normalesup.org> <6ce0ac130807201452u7fdff84agbd409ed14f326951@mail.gmail.com> Message-ID: <20080720221353.GH31836@phare.normalesup.org> On Sun, Jul 20, 2008 at 03:52:05PM -0600, Brian Granger wrote: > In this case, I am not sure that a callback will really do what we > want. One reason we have the option to allow the interpreter actually > raise the exception is that is how Python handles errors. If we move > things to a callback, we sort of get stuck with our own non-exception > based error handling. That would be very non-ideal. Well, I think this needs to be thought of carefully, and we probably need the option to reraise the exception anyhow. > I am not sure if it is possible, but I think the best solution is if > the traceback_trap can be smart and dynamically modify the traceback > to remove the unwanted information. This is also needed even if the > interpreter is not raising the exception. In that case, the > interpreter returns a dict that has the traceback as a string - with > the unwanted info as well. Conclusion: we need to figure out a way > of remove the unwanted info in the first place. Hum, I thought that if you catched the exception in the very function where the exec call lies, like it is done in ipython0 (iplib.py, line 2083), this give you a cleaner traceback with no postprocessing required. However, there must be an additional trick that ipython0 plays. It seems that the code is executed in a virtual module. As a result the traceback is very clean: In [7]: print i --------------------------------------------------------------------------- NameError Traceback (most recent call last) /home/varoquau/ in () NameError: name 'i' is not defined We need to figure out how this is done. I'll ask Fernando tomorrow. > Also, isn't this what the callback would have to do anyway? If so, > that logic should just be in the traceback_trap itself. Well, the callback could do more. In an IDE, for instance, you could have an option to open the relevant file in an editor, or open a debugger, or a stack inspector. This is why I am thinking of a callback. I can hardly see how to account for all these possibilities without either resort to some obfuscated code, or using a callback. Maybe I am missing the obvious. Cheers, Ga?l From ellisonbg.net at gmail.com Sun Jul 20 19:47:35 2008 From: ellisonbg.net at gmail.com (Brian Granger) Date: Sun, 20 Jul 2008 17:47:35 -0600 Subject: [IPython-dev] Nicer display of exceptions with ipython1 In-Reply-To: <20080720221353.GH31836@phare.normalesup.org> References: <20080720205429.GD31836@phare.normalesup.org> <6ce0ac130807201452u7fdff84agbd409ed14f326951@mail.gmail.com> <20080720221353.GH31836@phare.normalesup.org> Message-ID: <6ce0ac130807201647r77018438x1fa0839e4a55e245@mail.gmail.com> >> I am not sure if it is possible, but I think the best solution is if >> the traceback_trap can be smart and dynamically modify the traceback >> to remove the unwanted information. This is also needed even if the >> interpreter is not raising the exception. In that case, the >> interpreter returns a dict that has the traceback as a string - with >> the unwanted info as well. Conclusion: we need to figure out a way >> of remove the unwanted info in the first place. > > Hum, I thought that if you catched the exception in the very function > where the exec call lies, like it is done in ipython0 (iplib.py, line > 2083), this give you a cleaner traceback with no postprocessing required. > However, there must be an additional trick that ipython0 plays. It seems > that the code is executed in a virtual module. As a result the traceback > is very clean: > > In [7]: print i > --------------------------------------------------------------------------- > NameError Traceback (most recent call > last) > > /home/varoquau/ in () > > NameError: name 'i' is not defined > > We need to figure out how this is done. I'll ask Fernando tomorrow. Sounds good, I think he knows that code much better than I do. >> Also, isn't this what the callback would have to do anyway? If so, >> that logic should just be in the traceback_trap itself. > > Well, the callback could do more. In an IDE, for instance, you could have > an option to open the relevant file in an editor, or open a debugger, or > a stack inspector. This is why I am thinking of a callback. I can hardly > see how to account for all these possibilities without either resort to > some obfuscated code, or using a callback. Maybe I am missing the > obvious. Ah, yes, now I see why something like a callback would be nice. But I don't think we should do that only instead of figuring out how to get a nicer traceback. We should probably do both. Brian > Cheers, > > Ga?l > From gael.varoquaux at normalesup.org Sun Jul 20 19:49:43 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 21 Jul 2008 01:49:43 +0200 Subject: [IPython-dev] Nicer display of exceptions with ipython1 In-Reply-To: <6ce0ac130807201647r77018438x1fa0839e4a55e245@mail.gmail.com> References: <20080720205429.GD31836@phare.normalesup.org> <6ce0ac130807201452u7fdff84agbd409ed14f326951@mail.gmail.com> <20080720221353.GH31836@phare.normalesup.org> <6ce0ac130807201647r77018438x1fa0839e4a55e245@mail.gmail.com> Message-ID: <20080720234943.GJ31836@phare.normalesup.org> On Sun, Jul 20, 2008 at 05:47:35PM -0600, Brian Granger wrote: > Ah, yes, now I see why something like a callback would be nice. But I > don't think we should do that only instead of figuring out how to get > a nicer traceback. We should probably do both. Agreed. I'll work on that. Ga?l From gael.varoquaux at normalesup.org Mon Jul 21 00:00:11 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 21 Jul 2008 06:00:11 +0200 Subject: [IPython-dev] History in ipython1 Message-ID: <20080721040011.GO31836@phare.normalesup.org> I am about to implement saving history between session for my frontend. I cannot reuse the existing code, as it heavily relies on readline. I had a look at the pyreadline code, and I don't think I am going to ruese it. First of all, I would have to extract it from pylreadline, as pyreadline cannot be installed on a non-windows OS (not only because of its dependencies, but also because it overrides readline when it is installed), second there is a lot of stuff I don't need in there. Before doing this, I want to clean up the history code. I have the impression that the current History objects where derived from the history implementation I had sketched almost a year ago. In this implementation the history cursor was 0 to indicate the last line, and went up as you would go up the history. In the current version, althought the docstrings haven't been modified, it is the other way around. I prefer the way in which last=0, the reason been that with the exception, and the various code path that can lead to an execution or a new prompt, or the loading of history, it is easy to loose track of the number of past executions. Of course, we can always fallback on our feet by checking the length of the history, and resetting the cursor to that, but it seems simpler to count from the back. How to people feel about that? In addition, I would like to add history loading in the history object, throught a seperate method. No controverse? Ga?l From barrywark at gmail.com Mon Jul 21 00:36:43 2008 From: barrywark at gmail.com (Barry Wark) Date: Sun, 20 Jul 2008 21:36:43 -0700 Subject: [IPython-dev] ipython1 and synchronous printing of stdout In-Reply-To: <6ce0ac130807201250m26fb1a5flcbc9bdb62844cda8@mail.gmail.com> References: <20080720164615.GB23971@phare.normalesup.org> <20080720192702.GH23971@phare.normalesup.org> <6ce0ac130807201250m26fb1a5flcbc9bdb62844cda8@mail.gmail.com> Message-ID: On Sun, Jul 20, 2008 at 12:50 PM, Brian Granger wrote: > I don't have any time to look at this now, but we should have an > interface in the interpreter that allow the registration of a callback > that gets called or these things. > > Cheers, > > Brian There are, I think, several "events" for which a frontend might want a callback. Opening files, stdout/stderr output, and modifications to the user_ns are some that come immediately to mind. Opening files might be of interest if the frontend wants to notify the user that there is file output to be retrieved from the engine, stdout/stderr for the resason Gael outlines, and modifications to the user_ns to allow the frontend to keep an up-to-date view of the user_ns (ala Matlab's Workspace browser) without querying the the entire engine user_ns after every executed block. Apple's NSDistributedNotificationCenter API (http://developer.apple.com/documentation/Cocoa/Reference/Foundation/Classes/NSDistributedNotificationCenter_Class/Reference/Reference.html) seems like a good pattern for this task (I'm sure there are equivalent APIs in other frameworks). Basically, the frontend can register for notifications by name. The interpreter then fires notifications for all events and they are passed on, by the notification center, to all registered observers. Just my 2c to start the discussion. Barry > > On Sun, Jul 20, 2008 at 1:27 PM, Gael Varoquaux > wrote: >>> I can off course monkey patch the output_trap with my own write method, >>> but this seems a bit ugly. What do the ipython1 developpers think the >>> right way of doing this (and maybe modifying output_trap) is? >> >> I went ahead experimenting, and created a subclass of output_trap gear >> towards printing via a callback, as can be seen on >> http://bazaar.launchpad.net/~gael-varoquaux/ipython/ipython-sync-frontend/annotate/1041?file_id=sync_output_trap.py-20080720174256-b1fz1ebbxl2ka4t9-1 >> >> I hope this approach is OK. By passing this this output_trap to the >> interpreter I get the desired result, and I sync this approach is >> reusable for any synchronous frontend, such as the one into which >> ipython0 will, hopefully, morph one day. >> >> Ga?l >> _______________________________________________ >> IPython-dev mailing list >> IPython-dev at scipy.org >> http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev >> > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev > From gael.varoquaux at normalesup.org Mon Jul 21 00:50:21 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 21 Jul 2008 06:50:21 +0200 Subject: [IPython-dev] ipython1 and synchronous printing of stdout In-Reply-To: References: <20080720164615.GB23971@phare.normalesup.org> <20080720192702.GH23971@phare.normalesup.org> <6ce0ac130807201250m26fb1a5flcbc9bdb62844cda8@mail.gmail.com> Message-ID: <20080721045021.GP31836@phare.normalesup.org> On Sun, Jul 20, 2008 at 09:36:43PM -0700, Barry Wark wrote: > There are, I think, several "events" for which a frontend might want a > callback. Opening files, Hum, in what sens? What would trigger this event? > stdout/stderr output, Sure, this is why I came up with SyncOutputTrap, that allows me to pass a callback to write. I add a wx.Yield() in this calback and my UI can refresh each time you write something to stdout/stdin. > and modifications to the user_ns are some that come immediately to > mind. Opening files might be of interest if the frontend wants to > notify the user that there is file output to be retrieved from the > engine, stdout/stderr for the resason Gael outlines, and modifications > to the user_ns to allow the frontend to keep an up-to-date view of the > user_ns (ala Matlab's Workspace browser) without querying the the > entire engine user_ns after every executed block. Hum, I am not too excited about firing events when the namespace gets updated. This could slow things down a lot. I general I am not too excited about having events fired all over the place. This can lead to tedious code with getters and setters and a udge performance loss. I am much more interested in using objects with a standard interface (such as a file, for instance) and then, if you want getters and setters, or the possibility to add callabcks, you do it in these objects. > Apple's NSDistributedNotificationCenter API > (http://developer.apple.com/documentation/Cocoa/Reference/Foundation/Classes/NSDistributedNotificationCenter_Class/Reference/Reference.html) > seems like a good pattern for this task (I'm sure there are equivalent > APIs in other frameworks). Basically, the frontend can register for > notifications by name. The interpreter then fires notifications for all > events and they are passed on, by the notification center, to all > registered observers. Well, I don't really like this kind of code. I find it clunky, and you end up have notification registrations all over the place (reminds me of wx). I actually prefer a lot the traits model, or the VTK model, AFAIK, where objects can be there own observers. As a result you do not have notifications blending in objects where they do not belong. For the core frontends, obviously, we do not want to add a dependence to traits, but I must imagine I cannot imagine going much further without it. To take your example of updating the namespace view, all you have to do is to pass a traited dict, instead of a dict, to the interpreter as a namespace. Then you can have your namespace update automatically (AFAIK Robert is actually working on a branch where he does that). If we start adding callbacks whereever we need them, I am worried we'll end up with callbacks everywhere. I think we need to add callback only where they are absolutely necessary. My two cents, Ga?l From vivainio at gmail.com Mon Jul 21 11:32:30 2008 From: vivainio at gmail.com (Ville M. Vainio) Date: Mon, 21 Jul 2008 18:32:30 +0300 Subject: [IPython-dev] History in ipython1 In-Reply-To: <20080721040011.GO31836@phare.normalesup.org> References: <20080721040011.GO31836@phare.normalesup.org> Message-ID: <46cb515a0807210832g3322a398kddba126ba7ecaa2a@mail.gmail.com> On Mon, Jul 21, 2008 at 7:00 AM, Gael Varoquaux wrote: > number of past executions. Of course, we can always fallback on our feet > by checking the length of the history, and resetting the cursor to that, > but it seems simpler to count from the back. How to people feel about > that? It would seem simpler to have 0 as the first element, mostly to ensure that item number 3 will always be the same item (i.e. stable numbering). -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From gael.varoquaux at normalesup.org Mon Jul 21 23:47:31 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 22 Jul 2008 05:47:31 +0200 Subject: [IPython-dev] History in ipython1 In-Reply-To: <46cb515a0807210832g3322a398kddba126ba7ecaa2a@mail.gmail.com> References: <20080721040011.GO31836@phare.normalesup.org> <46cb515a0807210832g3322a398kddba126ba7ecaa2a@mail.gmail.com> Message-ID: <20080722034731.GF5258@phare.normalesup.org> On Mon, Jul 21, 2008 at 06:32:30PM +0300, Ville M. Vainio wrote: > On Mon, Jul 21, 2008 at 7:00 AM, Gael Varoquaux > wrote: > > number of past executions. Of course, we can always fallback on our feet > > by checking the length of the history, and resetting the cursor to that, > > but it seems simpler to count from the back. How to people feel about > > that? > It would seem simpler to have 0 as the first element, mostly to ensure > that item number 3 will always be the same item (i.e. stable > numbering). Hum, as you load a history, you prepend things to the beginning of the list, so item number 3 will stop being item number 3. I think of history as "item 3 lines ago". Ga?l From fperez.net at gmail.com Tue Jul 22 04:20:43 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 22 Jul 2008 01:20:43 -0700 Subject: [IPython-dev] Nicer display of exceptions with ipython1 In-Reply-To: <6ce0ac130807201647r77018438x1fa0839e4a55e245@mail.gmail.com> References: <20080720205429.GD31836@phare.normalesup.org> <6ce0ac130807201452u7fdff84agbd409ed14f326951@mail.gmail.com> <20080720221353.GH31836@phare.normalesup.org> <6ce0ac130807201647r77018438x1fa0839e4a55e245@mail.gmail.com> Message-ID: On Sun, Jul 20, 2008 at 4:47 PM, Brian Granger wrote: >> We need to figure out how this is done. I'll ask Fernando tomorrow. > > Sounds good, I think he knows that code much better than I do. Done. There's a parameter in the traceback handlers for the number of stack frames to skip. >>> Also, isn't this what the callback would have to do anyway? If so, >>> that logic should just be in the traceback_trap itself. >> >> Well, the callback could do more. In an IDE, for instance, you could have >> an option to open the relevant file in an editor, or open a debugger, or >> a stack inspector. This is why I am thinking of a callback. I can hardly >> see how to account for all these possibilities without either resort to >> some obfuscated code, or using a callback. Maybe I am missing the >> obvious. > > Ah, yes, now I see why something like a callback would be nice. But I > don't think we should do that only instead of figuring out how to get > a nicer traceback. We should probably do both. Extending the exception handling API to optionally take a callback is certainly possible, and I can see valid uses for it. Cheers, f From barrywark at gmail.com Tue Jul 22 10:36:14 2008 From: barrywark at gmail.com (Barry Wark) Date: Tue, 22 Jul 2008 09:36:14 -0500 Subject: [IPython-dev] History in ipython1 In-Reply-To: <20080722034731.GF5258@phare.normalesup.org> References: <20080721040011.GO31836@phare.normalesup.org> <46cb515a0807210832g3322a398kddba126ba7ecaa2a@mail.gmail.com> <20080722034731.GF5258@phare.normalesup.org> Message-ID: On Mon, Jul 21, 2008 at 10:47 PM, Gael Varoquaux wrote: > On Mon, Jul 21, 2008 at 06:32:30PM +0300, Ville M. Vainio wrote: >> On Mon, Jul 21, 2008 at 7:00 AM, Gael Varoquaux >> wrote: > >> > number of past executions. Of course, we can always fallback on our feet >> > by checking the length of the history, and resetting the cursor to that, >> > but it seems simpler to count from the back. How to people feel about >> > that? > >> It would seem simpler to have 0 as the first element, mostly to ensure >> that item number 3 will always be the same item (i.e. stable >> numbering). > > Hum, as you load a history, you prepend things to the beginning of the > list, so item number 3 will stop being item number 3. I think of history > as "item 3 lines ago". It seems this is a frontend-specific issue. A notebook interface would want to think of history counting forwards in time from 0 whereas a terminal interface might want to think of history going backwards in time from 0. > > Ga?l > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev > From ellisonbg.net at gmail.com Tue Jul 22 12:00:32 2008 From: ellisonbg.net at gmail.com (Brian Granger) Date: Tue, 22 Jul 2008 10:00:32 -0600 Subject: [IPython-dev] History in ipython1 In-Reply-To: References: <20080721040011.GO31836@phare.normalesup.org> <46cb515a0807210832g3322a398kddba126ba7ecaa2a@mail.gmail.com> <20080722034731.GF5258@phare.normalesup.org> Message-ID: <6ce0ac130807220900o4f87f8day20bb9b7571d06ea0@mail.gmail.com> Also, I think the history of history is important: In [1]: a = 10 In [2]: b = 20 In [3]: history 1: a = 10 2: b = 20 3: _ip.magic("history ") Any departure from this will probably confuse developers and users. Brian On Tue, Jul 22, 2008 at 8:36 AM, Barry Wark wrote: > On Mon, Jul 21, 2008 at 10:47 PM, Gael Varoquaux > wrote: >> On Mon, Jul 21, 2008 at 06:32:30PM +0300, Ville M. Vainio wrote: >>> On Mon, Jul 21, 2008 at 7:00 AM, Gael Varoquaux >>> wrote: >> >>> > number of past executions. Of course, we can always fallback on our feet >>> > by checking the length of the history, and resetting the cursor to that, >>> > but it seems simpler to count from the back. How to people feel about >>> > that? >> >>> It would seem simpler to have 0 as the first element, mostly to ensure >>> that item number 3 will always be the same item (i.e. stable >>> numbering). >> >> Hum, as you load a history, you prepend things to the beginning of the >> list, so item number 3 will stop being item number 3. I think of history >> as "item 3 lines ago". > > It seems this is a frontend-specific issue. A notebook interface would > want to think of history counting forwards in time from 0 whereas a > terminal interface might want to think of history going backwards in > time from 0. > >> >> Ga?l >> _______________________________________________ >> IPython-dev mailing list >> IPython-dev at scipy.org >> http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev >> > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev > From gael.varoquaux at normalesup.org Tue Jul 22 12:50:43 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 22 Jul 2008 18:50:43 +0200 Subject: [IPython-dev] History in ipython1 In-Reply-To: <6ce0ac130807220900o4f87f8day20bb9b7571d06ea0@mail.gmail.com> References: <20080721040011.GO31836@phare.normalesup.org> <46cb515a0807210832g3322a398kddba126ba7ecaa2a@mail.gmail.com> <20080722034731.GF5258@phare.normalesup.org> <6ce0ac130807220900o4f87f8day20bb9b7571d06ea0@mail.gmail.com> Message-ID: <20080722165043.GA9152@phare.normalesup.org> On Tue, Jul 22, 2008 at 10:00:32AM -0600, Brian Granger wrote: > In [3]: history > 1: a = 10 > 2: b = 20 > 3: _ip.magic("history ") > Any departure from this will probably confuse developers and users. OK, so lets stick to the way it is currenlty implemented. I just wanted to have this discussion rather early than late. Ga?l From barrywark at gmail.com Tue Jul 22 14:05:00 2008 From: barrywark at gmail.com (Barry Wark) Date: Tue, 22 Jul 2008 13:05:00 -0500 Subject: [IPython-dev] ipython1 and synchronous printing of stdout In-Reply-To: <20080721045021.GP31836@phare.normalesup.org> References: <20080720164615.GB23971@phare.normalesup.org> <20080720192702.GH23971@phare.normalesup.org> <6ce0ac130807201250m26fb1a5flcbc9bdb62844cda8@mail.gmail.com> <20080721045021.GP31836@phare.normalesup.org> Message-ID: As always, I think working code is better than good theory, so please read the following with that in mind. I think that the opinion you express below may put us on the road to an architectural mistake in the redesign of ipython. My understanding of the historical context is that the current ipython0 interpreter is direclty tied to the frontend in a way that makes new frontends or stand-alone interpreters difficult. The solution, as embodied by the the ipython1 codebase and the evolving IPython.frontend package is to establish a separation between interpreter and frontend to prevent the type of dependency that exists in the current code base. Now, back to the issue at hand. In developing frontends using the new architecture, you have noticed that there are a lot of things that the new system is missing. In general, these fall into the category of the frontend wanting to monitor and/or modify execution in the interpreter at a more fine-grained level than entire code blocks (the only option currently). There is a standard pattern for handling this type of notification, the Observer pattern, and this type of fine-grained modification, the Delegate pattern. Propper implementation of either produces minimal overhead (and it's worth remembering that by writing IPython in python instead of a lower level language, we've all already stated our preference for abstraction over raw efficiency) [1]. The alternative, that I believe you are proposing below and in the current thread on adding a callback to the exception handling code in the interpreter, is to add callback hooks to the interpreter so that one (or more?) callbacks can be attached to specific events in the interpreter's execution flow. In fact, I think these are not really different solutions, just different implementations of a similar idea. You propose handling observer notification from within the shell, whereas I propose handling it in an intermediary entity (the notification center). The callback approach is more explicit -- the interpreter must explicitly expose a callback hook and the frontend must explicitly register *with the interpreter* for that callback -- but less general. Unless we create a callback registry etc. in the shell, essentially duplicating the notification center idea, then when we want to add a new notification (e.g. for exceptions as in the currently running thread), we will need to duplicate the callback calling code in the shell rather than just marking that event for notification by the notification center. The observer/delegate combination allows the interpreter to remain agnostic wrt the interface of the frontend and vice-versa (and huge win, in my opinion). By adding an intermediary -- the notification center and/or delegate dispatcher -- we avoid the issue of frontends and interpreters being closely tied by contract and make it easier to provide the same interface directly (between frontend and shell as in your Wx frontend) or locally or remotely via a twisted service (as between and engine and the frontend in my Cocoa frontend). I know that you're on a deadline and need to write code, but I would urge a little bit of restraint and planning before you commit to the callback-only approach. I think avoiding this type of dependency between shell and frontend will benefit us in the long run. Barry [1] For example, firing a notification when no observers are registered for that notification should not incur significant overhead beyond a couple of method calls, and perhaps a dictionary lookup, even in a naive implementation. On Sun, Jul 20, 2008 at 9:50 PM, Gael Varoquaux wrote: > On Sun, Jul 20, 2008 at 09:36:43PM -0700, Barry Wark wrote: >> There are, I think, several "events" for which a frontend might want a >> callback. Opening files, > > Hum, in what sens? What would trigger this event? > >> stdout/stderr output, > > Sure, this is why I came up with SyncOutputTrap, that allows me to pass a > callback to write. I add a wx.Yield() in this calback and my UI can > refresh each time you write something to stdout/stdin. > >> and modifications to the user_ns are some that come immediately to >> mind. Opening files might be of interest if the frontend wants to >> notify the user that there is file output to be retrieved from the >> engine, stdout/stderr for the resason Gael outlines, and modifications >> to the user_ns to allow the frontend to keep an up-to-date view of the >> user_ns (ala Matlab's Workspace browser) without querying the the >> entire engine user_ns after every executed block. > > Hum, I am not too excited about firing events when the namespace gets > updated. This could slow things down a lot. I general I am not too > excited about having events fired all over the place. This can lead to > tedious code with getters and setters and a udge performance loss. I am > much more interested in using objects with a standard interface (such as > a file, for instance) and then, if you want getters and setters, or > the possibility to add callabcks, you do it in these objects. > >> Apple's NSDistributedNotificationCenter API >> (http://developer.apple.com/documentation/Cocoa/Reference/Foundation/Classes/NSDistributedNotificationCenter_Class/Reference/Reference.html) >> seems like a good pattern for this task (I'm sure there are equivalent >> APIs in other frameworks). Basically, the frontend can register for >> notifications by name. The interpreter then fires notifications for all >> events and they are passed on, by the notification center, to all >> registered observers. > > Well, I don't really like this kind of code. I find it clunky, and you > end up have notification registrations all over the place (reminds me of > wx). I actually prefer a lot the traits model, or the VTK model, AFAIK, > where objects can be there own observers. As a result you do not have > notifications blending in objects where they do not belong. For the core > frontends, obviously, we do not want to add a dependence to traits, but I > must imagine I cannot imagine going much further without it. To take your > example of updating the namespace view, all you have to do is to pass a > traited dict, instead of a dict, to the interpreter as a namespace. Then > you can have your namespace update automatically (AFAIK Robert is > actually working on a branch where he does that). > > If we start adding callbacks whereever we need them, I am worried we'll > end up with callbacks everywhere. I think we need to add callback only > where they are absolutely necessary. > My two cents, > > Ga?l > From fperez.net at gmail.com Tue Jul 22 14:24:26 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 22 Jul 2008 11:24:26 -0700 Subject: [IPython-dev] History in ipython1 In-Reply-To: <6ce0ac130807220900o4f87f8day20bb9b7571d06ea0@mail.gmail.com> References: <20080721040011.GO31836@phare.normalesup.org> <46cb515a0807210832g3322a398kddba126ba7ecaa2a@mail.gmail.com> <20080722034731.GF5258@phare.normalesup.org> <6ce0ac130807220900o4f87f8day20bb9b7571d06ea0@mail.gmail.com> Message-ID: On Tue, Jul 22, 2008 at 9:00 AM, Brian Granger wrote: > Also, I think the history of history is important: > > In [1]: a = 10 > > In [2]: b = 20 > > In [3]: history > 1: a = 10 > 2: b = 20 > 3: _ip.magic("history ") > > Any departure from this will probably confuse developers and users. Yes, esp. because there are things like %macro and %edit that use those numbers, as well as the fact that the In[] list-like object is also fully accessible and can be used at run-time, and it relies on the same indexing mode: In [1]: x = 1 In [2]: x+=1 In [3]: print x 2 In [4]: exec In[2:4] 3 In [6]: exec In[2:4] 4 In [7]: exec In[2:4] 5 Cheers, f From ellisonbg.net at gmail.com Tue Jul 22 16:15:59 2008 From: ellisonbg.net at gmail.com (Brian Granger) Date: Tue, 22 Jul 2008 14:15:59 -0600 Subject: [IPython-dev] ipython1 and synchronous printing of stdout In-Reply-To: <20080721045021.GP31836@phare.normalesup.org> References: <20080720164615.GB23971@phare.normalesup.org> <20080720192702.GH23971@phare.normalesup.org> <6ce0ac130807201250m26fb1a5flcbc9bdb62844cda8@mail.gmail.com> <20080721045021.GP31836@phare.normalesup.org> Message-ID: <6ce0ac130807221315p386a629avb87d6cd5feba57f4@mail.gmail.com> > Well, I don't really like this kind of code. I find it clunky, and you > end up have notification registrations all over the place (reminds me of > wx). I actually prefer a lot the traits model, or the VTK model, AFAIK, > where objects can be there own observers. As a result you do not have > notifications blending in objects where they do not belong. For the core > frontends, obviously, we do not want to add a dependence to traits, but I > must imagine I cannot imagine going much further without it. To take your > example of updating the namespace view, all you have to do is to pass a > traited dict, instead of a dict, to the interpreter as a namespace. Then > you can have your namespace update automatically (AFAIK Robert is > actually working on a branch where he does that). I understand the benefits of traits, but we have already made a firm decision to keep Traits (or any other compiled code for that matter) out of the core of IPython. IPython has to be able to run (at least in principal) on non CPython implementation of Python. Obviously, a GUI frontend can use traits, wx, qt, etc., and the kernel relies on Twisted which has compiled code. But I it almost sounds like you are proposing using traits in the interpreter class itself. I am all for clever uses of Traits like Robert Kern has been thinking about. But, we do need a more general approach that doesn't rely on Traits. > If we start adding callbacks whereever we need them, I am worried we'll > end up with callbacks everywhere. I think we need to add callback only > where they are absolutely necessary. I tend to agree with this. A bunch of random disconnected callbacks makes an interface very disjoint. See my upcoming reply to Barry to the rest of my thoughts... Cheers, Brian > My two cents, > > Ga?l > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev > From ellisonbg.net at gmail.com Tue Jul 22 16:25:45 2008 From: ellisonbg.net at gmail.com (Brian Granger) Date: Tue, 22 Jul 2008 14:25:45 -0600 Subject: [IPython-dev] ipython1 and synchronous printing of stdout In-Reply-To: References: <20080720164615.GB23971@phare.normalesup.org> <20080720192702.GH23971@phare.normalesup.org> <6ce0ac130807201250m26fb1a5flcbc9bdb62844cda8@mail.gmail.com> <20080721045021.GP31836@phare.normalesup.org> Message-ID: <6ce0ac130807221325y19ffd8dbtafc27d1c6ca5f4ac@mail.gmail.com> > As always, I think working code is better than good theory, so please > read the following with that in mind. I think that the opinion you > express below may put us on the road to an architectural mistake in > the redesign of ipython. My understanding of the historical context is > that the current ipython0 interpreter is direclty tied to the frontend > in a way that makes new frontends or stand-alone interpreters > difficult. The solution, as embodied by the the ipython1 codebase and > the evolving IPython.frontend package is to establish a separation > between interpreter and frontend to prevent the type of dependency > that exists in the current code base. One thing that is absolutely critical is loose coupling. All the parts of IPython need to depend only very weakly on each other, and interact through clean, standardized interfaces (I don't mean this word in the script zope.interface sense though). The big problem with the traditional IPython codebase is that everything is welded together. The question is what types of design patterns will make this possible. >From this thread, I see a few options: 1. Traits. While I really like the Traits model and I think it is highly appropriate for GUI programming, I think it would lead to too tightly coupled code. Also, we can't use Traits in the core of ipython because it has compiled code. 2. Plain callbacks. At least a couple of us seem to think that an interface that has lots of callbacks becomes very cumbersome. 3. The Observer/Delegate patterns as are typical is Cocoa. I know that Gael is not fund of this, but I do think this pattern is much better than plain callbacks, as it give a nice formal structure to everything. Also, I agree with Barry that this pattern is pretty much what you get when you try to do plain callbacks properly. Thus, my own conclusion is that the Observer/Delegate pattern is probably the best option for us and will lead to the loose coupling that we need. Barry, what would it take to put together an example of this pattern? Will we need some amount of infrastructure to make this possible? Cheers, Brian > Now, back to the issue at hand. In developing frontends using the new > architecture, you have noticed that there are a lot of things that the > new system is missing. In general, these fall into the category of the > frontend wanting to monitor and/or modify execution in the interpreter > at a more fine-grained level than entire code blocks (the only option > currently). There is a standard pattern for handling this type of > notification, the Observer pattern, and this type of fine-grained > modification, the Delegate pattern. Propper implementation of either > produces minimal overhead (and it's worth remembering that by writing > IPython in python instead of a lower level language, we've all already > stated our preference for abstraction over raw efficiency) [1]. The > alternative, that I believe you are proposing below and in the current > thread on adding a callback to the exception handling code in the > interpreter, is to add callback hooks to the interpreter so that one > (or more?) callbacks can be attached to specific events in the > interpreter's execution flow. > > In fact, I think these are not really different solutions, just > different implementations of a similar idea. You propose handling > observer notification from within the shell, whereas I propose > handling it in an intermediary entity (the notification center). The > callback approach is more explicit -- the interpreter must explicitly > expose a callback hook and the frontend must explicitly register *with > the interpreter* for that callback -- but less general. Unless we > create a callback registry etc. in the shell, essentially duplicating > the notification center idea, then when we want to add a new > notification (e.g. for exceptions as in the currently running thread), > we will need to duplicate the callback calling code in the shell > rather than just marking that event for notification by the > notification center. The observer/delegate combination allows the > interpreter to remain agnostic wrt the interface of the frontend and > vice-versa (and huge win, in my opinion). By adding an intermediary -- > the notification center and/or delegate dispatcher -- we avoid the > issue of frontends and interpreters being closely tied by contract and > make it easier to provide the same interface directly (between > frontend and shell as in your Wx frontend) or locally or remotely via > a twisted service (as between and engine and the frontend in my Cocoa > frontend). > > I know that you're on a deadline and need to write code, but I would > urge a little bit of restraint and planning before you commit to the > callback-only approach. I think avoiding this type of dependency > between shell and frontend will benefit us in the long run. > > Barry > > [1] For example, firing a notification when no observers are > registered for that notification should not incur significant overhead > beyond a couple of method calls, and perhaps a dictionary lookup, even > in a naive implementation. > > On Sun, Jul 20, 2008 at 9:50 PM, Gael Varoquaux > wrote: >> On Sun, Jul 20, 2008 at 09:36:43PM -0700, Barry Wark wrote: >>> There are, I think, several "events" for which a frontend might want a >>> callback. Opening files, >> >> Hum, in what sens? What would trigger this event? >> >>> stdout/stderr output, >> >> Sure, this is why I came up with SyncOutputTrap, that allows me to pass a >> callback to write. I add a wx.Yield() in this calback and my UI can >> refresh each time you write something to stdout/stdin. >> >>> and modifications to the user_ns are some that come immediately to >>> mind. Opening files might be of interest if the frontend wants to >>> notify the user that there is file output to be retrieved from the >>> engine, stdout/stderr for the resason Gael outlines, and modifications >>> to the user_ns to allow the frontend to keep an up-to-date view of the >>> user_ns (ala Matlab's Workspace browser) without querying the the >>> entire engine user_ns after every executed block. >> >> Hum, I am not too excited about firing events when the namespace gets >> updated. This could slow things down a lot. I general I am not too >> excited about having events fired all over the place. This can lead to >> tedious code with getters and setters and a udge performance loss. I am >> much more interested in using objects with a standard interface (such as >> a file, for instance) and then, if you want getters and setters, or >> the possibility to add callabcks, you do it in these objects. >> >>> Apple's NSDistributedNotificationCenter API >>> (http://developer.apple.com/documentation/Cocoa/Reference/Foundation/Classes/NSDistributedNotificationCenter_Class/Reference/Reference.html) >>> seems like a good pattern for this task (I'm sure there are equivalent >>> APIs in other frameworks). Basically, the frontend can register for >>> notifications by name. The interpreter then fires notifications for all >>> events and they are passed on, by the notification center, to all >>> registered observers. >> >> Well, I don't really like this kind of code. I find it clunky, and you >> end up have notification registrations all over the place (reminds me of >> wx). I actually prefer a lot the traits model, or the VTK model, AFAIK, >> where objects can be there own observers. As a result you do not have >> notifications blending in objects where they do not belong. For the core >> frontends, obviously, we do not want to add a dependence to traits, but I >> must imagine I cannot imagine going much further without it. To take your >> example of updating the namespace view, all you have to do is to pass a >> traited dict, instead of a dict, to the interpreter as a namespace. Then >> you can have your namespace update automatically (AFAIK Robert is >> actually working on a branch where he does that). >> >> If we start adding callbacks whereever we need them, I am worried we'll >> end up with callbacks everywhere. I think we need to add callback only >> where they are absolutely necessary. >> My two cents, >> >> Ga?l >> > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev > From barrywark at gmail.com Wed Jul 23 12:17:46 2008 From: barrywark at gmail.com (Barry Wark) Date: Wed, 23 Jul 2008 11:17:46 -0500 Subject: [IPython-dev] ipython1 and synchronous printing of stdout In-Reply-To: <6ce0ac130807221325y19ffd8dbtafc27d1c6ca5f4ac@mail.gmail.com> References: <20080720164615.GB23971@phare.normalesup.org> <20080720192702.GH23971@phare.normalesup.org> <6ce0ac130807201250m26fb1a5flcbc9bdb62844cda8@mail.gmail.com> <20080721045021.GP31836@phare.normalesup.org> <6ce0ac130807221325y19ffd8dbtafc27d1c6ca5f4ac@mail.gmail.com> Message-ID: <8C8A5DF0-F0A1-4294-90F0-BA2F4CE277BB@gmail.com> On Jul 22, 2008, at 3:25 PM, "Brian Granger" wrote: >> As always, I think working code is better than good theory, so please >> read the following with that in mind. I think that the opinion you >> express below may put us on the road to an architectural mistake in >> the redesign of ipython. My understanding of the historical context >> is >> that the current ipython0 interpreter is direclty tied to the >> frontend >> in a way that makes new frontends or stand-alone interpreters >> difficult. The solution, as embodied by the the ipython1 codebase and >> the evolving IPython.frontend package is to establish a separation >> between interpreter and frontend to prevent the type of dependency >> that exists in the current code base. > > One thing that is absolutely critical is loose coupling. All the > parts of IPython need to depend only very weakly on each other, and > interact through clean, standardized interfaces (I don't mean this > word in the script zope.interface sense though). The big problem with > the traditional IPython codebase is that everything is welded > together. The question is what types of design patterns will make > this possible. > > From this thread, I see a few options: > > 1. Traits. While I really like the Traits model and I think it is > highly appropriate for GUI programming, I think it would lead to too > tightly coupled code. Also, we can't use Traits in the core of > ipython because it has compiled code. > > 2. Plain callbacks. At least a couple of us seem to think that an > interface that has lots of callbacks becomes very cumbersome. > > 3. The Observer/Delegate patterns as are typical is Cocoa. I know > that Gael is not fund of this, but I do think this pattern is much > better than plain callbacks, as it give a nice formal structure to > everything. Also, I agree with Barry that this pattern is pretty much > what you get when you try to do plain callbacks properly. > > Thus, my own conclusion is that the Observer/Delegate pattern is > probably the best option for us and will lead to the loose coupling > that we need. > > Barry, what would it take to put together an example of this pattern? > Will we need some amount of infrastructure to make this possible? I'd be happy to write up an example and a draft implementation. I'm going to be mostly offline, traveling, until the weekend. I'm most familiar with the Cocoa implementation so I'd like to do a bit of research of alternatives before I write the spec. Therefore next week is the realistic timeline. Off the top of my head, I think a synchronous notification center implementation in I.k.core and an addition to the engineservice API along with an asynchronous notification center impl in I.kernel would cover the use cases. I will write up a functional/technical spec and solicit feedback before I make any decisions though. Besides the notificatipn center implementation(s), I don't think any more infrastructure is required beyond modifying the interpreter to actually fire the appropriate notifications (support for obvious callbacks such as Gael's use cases should be easy; the notifying worskspace ala Robert's work will obviously require more effort). As you pointed out, this is really just carefully implemented callbacks with a loose coupling. I'm glad we're dealing with this issue early. I think getting it right will pay big dividends in the future. Cheers, Barry > > > Cheers, > > Brian > >> Now, back to the issue at hand. In developing frontends using the new >> architecture, you have noticed that there are a lot of things that >> the >> new system is missing. In general, these fall into the category of >> the >> frontend wanting to monitor and/or modify execution in the >> interpreter >> at a more fine-grained level than entire code blocks (the only option >> currently). There is a standard pattern for handling this type of >> notification, the Observer pattern, and this type of fine-grained >> modification, the Delegate pattern. Propper implementation of either >> produces minimal overhead (and it's worth remembering that by writing >> IPython in python instead of a lower level language, we've all >> already >> stated our preference for abstraction over raw efficiency) [1]. The >> alternative, that I believe you are proposing below and in the >> current >> thread on adding a callback to the exception handling code in the >> interpreter, is to add callback hooks to the interpreter so that one >> (or more?) callbacks can be attached to specific events in the >> interpreter's execution flow. >> >> In fact, I think these are not really different solutions, just >> different implementations of a similar idea. You propose handling >> observer notification from within the shell, whereas I propose >> handling it in an intermediary entity (the notification center). The >> callback approach is more explicit -- the interpreter must explicitly >> expose a callback hook and the frontend must explicitly register >> *with >> the interpreter* for that callback -- but less general. Unless we >> create a callback registry etc. in the shell, essentially duplicating >> the notification center idea, then when we want to add a new >> notification (e.g. for exceptions as in the currently running >> thread), >> we will need to duplicate the callback calling code in the shell >> rather than just marking that event for notification by the >> notification center. The observer/delegate combination allows the >> interpreter to remain agnostic wrt the interface of the frontend and >> vice-versa (and huge win, in my opinion). By adding an intermediary >> -- >> the notification center and/or delegate dispatcher -- we avoid the >> issue of frontends and interpreters being closely tied by contract >> and >> make it easier to provide the same interface directly (between >> frontend and shell as in your Wx frontend) or locally or remotely via >> a twisted service (as between and engine and the frontend in my Cocoa >> frontend). >> >> I know that you're on a deadline and need to write code, but I would >> urge a little bit of restraint and planning before you commit to the >> callback-only approach. I think avoiding this type of dependency >> between shell and frontend will benefit us in the long run. >> >> Barry >> >> [1] For example, firing a notification when no observers are >> registered for that notification should not incur significant >> overhead >> beyond a couple of method calls, and perhaps a dictionary lookup, >> even >> in a naive implementation. >> >> On Sun, Jul 20, 2008 at 9:50 PM, Gael Varoquaux >> wrote: >>> On Sun, Jul 20, 2008 at 09:36:43PM -0700, Barry Wark wrote: >>>> There are, I think, several "events" for which a frontend might >>>> want a >>>> callback. Opening files, >>> >>> Hum, in what sens? What would trigger this event? >>> >>>> stdout/stderr output, >>> >>> Sure, this is why I came up with SyncOutputTrap, that allows me to >>> pass a >>> callback to write. I add a wx.Yield() in this calback and my UI can >>> refresh each time you write something to stdout/stdin. >>> >>>> and modifications to the user_ns are some that come immediately to >>>> mind. Opening files might be of interest if the frontend wants to >>>> notify the user that there is file output to be retrieved from the >>>> engine, stdout/stderr for the resason Gael outlines, and >>>> modifications >>>> to the user_ns to allow the frontend to keep an up-to-date view >>>> of the >>>> user_ns (ala Matlab's Workspace browser) without querying the the >>>> entire engine user_ns after every executed block. >>> >>> Hum, I am not too excited about firing events when the namespace >>> gets >>> updated. This could slow things down a lot. I general I am not too >>> excited about having events fired all over the place. This can >>> lead to >>> tedious code with getters and setters and a udge performance loss. >>> I am >>> much more interested in using objects with a standard interface >>> (such as >>> a file, for instance) and then, if you want getters and setters, or >>> the possibility to add callabcks, you do it in these objects. >>> >>>> Apple's NSDistributedNotificationCenter API >>>> (http://developer.apple.com/documentation/Cocoa/Reference/Foundation/Classes/NSDistributedNotificationCenter_Class/Reference/Reference.html >>>> ) >>>> seems like a good pattern for this task (I'm sure there are >>>> equivalent >>>> APIs in other frameworks). Basically, the frontend can register for >>>> notifications by name. The interpreter then fires notifications >>>> for all >>>> events and they are passed on, by the notification center, to all >>>> registered observers. >>> >>> Well, I don't really like this kind of code. I find it clunky, and >>> you >>> end up have notification registrations all over the place (reminds >>> me of >>> wx). I actually prefer a lot the traits model, or the VTK model, >>> AFAIK, >>> where objects can be there own observers. As a result you do not >>> have >>> notifications blending in objects where they do not belong. For >>> the core >>> frontends, obviously, we do not want to add a dependence to >>> traits, but I >>> must imagine I cannot imagine going much further without it. To >>> take your >>> example of updating the namespace view, all you have to do is to >>> pass a >>> traited dict, instead of a dict, to the interpreter as a >>> namespace. Then >>> you can have your namespace update automatically (AFAIK Robert is >>> actually working on a branch where he does that). >>> >>> If we start adding callbacks whereever we need them, I am worried >>> we'll >>> end up with callbacks everywhere. I think we need to add callback >>> only >>> where they are absolutely necessary. >>> My two cents, >>> >>> Ga?l >>> >> _______________________________________________ >> IPython-dev mailing list >> IPython-dev at scipy.org >> http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev >> From vivainio at gmail.com Wed Jul 23 12:35:38 2008 From: vivainio at gmail.com (Ville M. Vainio) Date: Wed, 23 Jul 2008 19:35:38 +0300 Subject: [IPython-dev] History in ipython1 In-Reply-To: <20080722165043.GA9152@phare.normalesup.org> References: <20080721040011.GO31836@phare.normalesup.org> <46cb515a0807210832g3322a398kddba126ba7ecaa2a@mail.gmail.com> <20080722034731.GF5258@phare.normalesup.org> <6ce0ac130807220900o4f87f8day20bb9b7571d06ea0@mail.gmail.com> <20080722165043.GA9152@phare.normalesup.org> Message-ID: <46cb515a0807230935k4d6a2ca0r76052ac8b3458ad0@mail.gmail.com> On Tue, Jul 22, 2008 at 7:50 PM, Gael Varoquaux wrote: >> Any departure from this will probably confuse developers and users. > > OK, so lets stick to the way it is currenlty implemented. I just wanted > to have this discussion rather early than late. Still, there is nothing stopping you from using indices like -1, -3 to start counting from the end. -- Ville M. Vainio - vivainio.googlepages.com blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio' From ondrej at certik.cz Wed Jul 23 16:11:04 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 23 Jul 2008 22:11:04 +0200 Subject: [IPython-dev] let ipython return parent's class method docstring automatically Message-ID: <85b5c3130807231311g4d1382cr58831b124f2f25f6@mail.gmail.com> Hi, we are with Mateusz Paprocki at my place doing some hacking and we'd like to know what is the best way to handle the following situation in sympy: In [1]: e = x*y In [2]: e.nseries? Type: instancemethod Base Class: String Form: Namespace: Interactive File: /home/ondra/sympy/sympy/core/mul.py Definition: e.nseries(self, x, x0, n) Docstring: So the Mul.nseries() is missing a docstring, but: In [4]: print e.__class__.__base__.nseries.__doc__ Calculates a generalized series expansion. The difference between oseries and nseries is that nseries calculates "n" terms in the innermost expressions and then builds up the final series just by "cross-mutliplying" everything out. Advantage -- it's fast, because we don't have to determine how many terms we need to calculate in advance. Disadvantage -- you may endup with less terms than you may have expected, but the O(x**n) term appended will always be correct, so the result is correct, but maybe shorter. The partent class, in this case Basic, has a nice docstring. The thing is, that the Basic class has the docstring, but the child classes don't (obviously, because it'd be the same). What is the best way to handle this? We can see two possibilities: 1) patch ipython to return parent's (or parent's parent's) docstring. I checked that and the patch would be just a few lines of code in OInspect.py 2) have just Basic.nseries() method and use _eval_nseries() in all subclasses. That way, if the user types e.nseries(), the Basic.nseries() with a nice docstring will be invoked. So I'd like to ask the ipython guys if 1) is the way to go, or rather 2)? Or is there also some other way to fix this. Ondrej & Mateusz From fperez.net at gmail.com Wed Jul 23 17:59:57 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 23 Jul 2008 14:59:57 -0700 Subject: [IPython-dev] let ipython return parent's class method docstring automatically In-Reply-To: <85b5c3130807231311g4d1382cr58831b124f2f25f6@mail.gmail.com> References: <85b5c3130807231311g4d1382cr58831b124f2f25f6@mail.gmail.com> Message-ID: On Wed, Jul 23, 2008 at 1:11 PM, Ondrej Certik wrote: > The partent class, in this case Basic, has a nice docstring. The thing > is, that the Basic class has the docstring, but the child classes > don't (obviously, because it'd be the same). > What is the best way to handle this? > > We can see two possibilities: > > 1) patch ipython to return parent's (or parent's parent's) docstring. > I checked that and the patch would be just a few lines of code in > OInspect.py I think this is a fairly generic problem, that of children who override parent methods but don't rewrite the docstrings. So I'd be happy with ipython making the user's life easier out of the box, though I think in this case we should say something like: Docstring : [extracted from parent class foo.bar.baz] blah... so users are aware that they're reading the parent's docstring, just in case there are small inconsistencies the developer forgot to document. How does this sound? Cheers, f From ondrej at certik.cz Wed Jul 23 18:25:28 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Thu, 24 Jul 2008 00:25:28 +0200 Subject: [IPython-dev] let ipython return parent's class method docstring automatically In-Reply-To: References: <85b5c3130807231311g4d1382cr58831b124f2f25f6@mail.gmail.com> Message-ID: <85b5c3130807231525w3064eef9xa0f53777fc67a479@mail.gmail.com> On Wed, Jul 23, 2008 at 11:59 PM, Fernando Perez wrote: > On Wed, Jul 23, 2008 at 1:11 PM, Ondrej Certik wrote: > >> The partent class, in this case Basic, has a nice docstring. The thing >> is, that the Basic class has the docstring, but the child classes >> don't (obviously, because it'd be the same). >> What is the best way to handle this? >> >> We can see two possibilities: >> >> 1) patch ipython to return parent's (or parent's parent's) docstring. >> I checked that and the patch would be just a few lines of code in >> OInspect.py > > I think this is a fairly generic problem, that of children who > override parent methods but don't rewrite the docstrings. So I'd be > happy with ipython making the user's life easier out of the box, > though I think in this case we should say something like: > > Docstring : [extracted from parent class foo.bar.baz] > blah... > > so users are aware that they're reading the parent's docstring, just > in case there are small inconsistencies the developer forgot to > document. > > How does this sound? I'll send a patch. Ondrej From fperez.net at gmail.com Wed Jul 23 23:22:53 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 23 Jul 2008 20:22:53 -0700 Subject: [IPython-dev] ipython1 and synchronous printing of stdout In-Reply-To: <6ce0ac130807221325y19ffd8dbtafc27d1c6ca5f4ac@mail.gmail.com> References: <20080720164615.GB23971@phare.normalesup.org> <20080720192702.GH23971@phare.normalesup.org> <6ce0ac130807201250m26fb1a5flcbc9bdb62844cda8@mail.gmail.com> <20080721045021.GP31836@phare.normalesup.org> <6ce0ac130807221325y19ffd8dbtafc27d1c6ca5f4ac@mail.gmail.com> Message-ID: Hi all, I think we all agree on the fact that we want the combination of loose coupling, ease of code reuse across actual frontend implementations, and a reasonable implementation in terms of performance. The latter is actually important: while it's true that by using python we've gone for flexibility over speed, we still need to be careful. It's not that hard to write python code that runs slow enough to be a problem in practice (Traits 1.0 was implemented in pure python, for example, and it had serious performance issues). On Tue, Jul 22, 2008 at 1:25 PM, Brian Granger wrote: Now, on to what to do next: > >From this thread, I see a few options: > > 1. Traits. While I really like the Traits model and I think it is > highly appropriate for GUI programming, I think it would lead to too > tightly coupled code. Also, we can't use Traits in the core of > ipython because it has compiled code. > > 2. Plain callbacks. At least a couple of us seem to think that an > interface that has lots of callbacks becomes very cumbersome. > > 3. The Observer/Delegate pNSDistributedNotificationCenteratterns as are typical is Cocoa. I know > that Gael is not fund of this, but I do think this pattern is much > better than plain callbacks, as it give a nice formal structure to > everything. Also, I agree with Barry that this pattern is pretty much > what you get when you try to do plain callbacks properly. I think there's a fourth, that Gael mentioned in passing and which we've been discussing here face to face for a while: using objects whose basic behavior defines the core API, but which can be given at construction time instead of being hardcoded in our various __init__ methods. This allows subclasses (or even non-subclassing application writers who use the code as a library to build their own tools) to cleanly provide only the level of behavior modification they need. I've just gone through the exercise of building the nose plugin for ipython testing, and I precisely ran into the problem that in many places, I was forced to copy nose code simply to override a specific internal object in one place because it hadn't been parametrized. But if the nose plugins had more of their components be parametric, it would have been much simpler to write my code, since all I was doing was replacing these hard-coded objects with subclasses of the same that customized one small specific behavior. This seems to provide a simple solution to our design question: all we need to do is to clarify which objects are our core publicly modifiable components in the API, and then users can tweak only the parts they need. Said objects can themselves provide observer/delegation behavior if they so desire, but they can do it in the most appropriate way for a given problem. For example, someone writing a Traits app (who's already committed to using Traits for their own reasons) can simply stick in there traited versions of the same things, and with essentially zero extra code they get the Traits event model everyhwere. In a Cocoa environment, these objects could be lightly wrapped versions of the originals that register (via a modified __getattr__ for example) the necessary observation/update calls into the NSDistributedNotificationCenter that Cocoa provides, if that's the best way to do it in such an environment. And someone who just needs a callback here and there can simply stick their callback-enhanced object where they want. I could even see this being very handy for occasional debugging, by quickly activating tracing versions of certain objects as needed. One thing that I like about this approach is that it doesn't necessarily close the door on a *future* implementation of an ipython-wide notification center, if we end up finding in practice that it's really needed as the code grows. But right now, it seems to me that this simpler approach solves all of our problems, with the advantage of introducing exactly *zero* (not small, but truly zero) overhead for the normal use that has no customizations. Does this sound reasonable to others, or did I totally miss this particular boat? I really want us all to be happy with the solution we find to this so that we move forward reasonably convinced of it being viable. Cheers, f From ellisonbg.net at gmail.com Thu Jul 24 01:22:58 2008 From: ellisonbg.net at gmail.com (Brian Granger) Date: Wed, 23 Jul 2008 23:22:58 -0600 Subject: [IPython-dev] ipython1 and synchronous printing of stdout In-Reply-To: References: <20080720164615.GB23971@phare.normalesup.org> <20080720192702.GH23971@phare.normalesup.org> <6ce0ac130807201250m26fb1a5flcbc9bdb62844cda8@mail.gmail.com> <20080721045021.GP31836@phare.normalesup.org> <6ce0ac130807221325y19ffd8dbtafc27d1c6ca5f4ac@mail.gmail.com> Message-ID: <6ce0ac130807232222j295701denfa660731fc729f07@mail.gmail.com> > I think there's a fourth, that Gael mentioned in passing and which > we've been discussing here face to face for a while: using objects > whose basic behavior defines the core API, but which can be given at > construction time instead of being hardcoded in our various __init__ > methods. This allows subclasses (or even non-subclassing application > writers who use the code as a library to build their own tools) to > cleanly provide only the level of behavior modification they need. Yes, this is a design pattern that I very much like and that is well suited for certain things in ipython. We actually use this in a few places in the ipython1 code: 1. An IPython engine takes an interpreter class as a parameter in its init method. Thus a user can pass the engine a custom interpreter subclass to get custom behavior. 2. We don't do it yet, but the interpreter could take a Traited dict like object to use as the users namespace. We definitely should do more of this. But, I think this design pattern addresses a slightly different need than the callback/observer/notification stuff. Here is why: The real benefit of the observer/notification model is that everything can be completely transient. For example, you might have a notifier that observes the user's namespace. But you might only want it to operate when the user opens a specific window. Another example is when an observer is hooked up to an unreliable network connection. So I guess I would ad another design constraint that I have until now kept in the back of my mind: we need loose coupling that is also dynamic and transient. And for these situations, I still think the observer pattern is the best solution. > This seems to provide a simple solution to our design question: all we > need to do is to clarify which objects are our core publicly > modifiable components in the API, and then users can tweak only the > parts they need. Said objects can themselves provide > observer/delegation behavior if they so desire, but they can do it in > the most appropriate way for a given problem. For example, someone > writing a Traits app (who's already committed to using Traits for > their own reasons) can simply stick in there traited versions of the > same things, and with essentially zero extra code they get the Traits > event model everyhwere. I worry that such interfaces _can_ at times be a bit too implicit and thus hide subtle behaviors and introduce code coupling that isn't obvious. One example of a subtle behavior is the following. Traits is based on a model that says this "an interface consists of a set of attributes that you simply get and set as atributes" But, the second you go to propagate an interface over a network connection, you discover that it is really difficult. Interfaces that "network friendly" tend to be i) be methods/functions that ii) have arguments with very basic types. We do have some examples in IPython.kernel where we propagate interfaces that are attribute based, but a good amount of extra work and care is required. Thus, while I think the idea of passing a Traited dict to the interpreter to observe the user's namespace is a beautiful idea, the reality is much more painful: there is no straightforward way of propagating that interface over a network connection. With a standardize observer/notifier pattern that is based on methods, it is very straightforward to: * Propagate observation/notification events over a network * Unregister such events when network connections fail * Integrate such events with Twisted. * Integrate with other event loops like wx, qt, cocoa One thing that I hadn't realized is that the observer/notifier pattern has come up for Min and I in IPython.kernel a number of times, but we have never taken the time to really abstract it properly. .....a few minutes later... I just reread this email and I think I am talking in circles and a bit tired. But, I do think we need to have a solution that is "network friendly". Cheers, Brian PS - starting tomorrow, I will be offline for about a week. > In a Cocoa environment, these objects could be lightly wrapped > versions of the originals that register (via a modified __getattr__ > for example) the necessary observation/update calls into the > NSDistributedNotificationCenter that Cocoa provides, if that's the > best way to do it in such an environment. > > And someone who just needs a callback here and there can simply stick > their callback-enhanced object where they want. I could even see this > being very handy for occasional debugging, by quickly activating > tracing versions of certain objects as needed. > > One thing that I like about this approach is that it doesn't > necessarily close the door on a *future* implementation of an > ipython-wide notification center, if we end up finding in practice > that it's really needed as the code grows. But right now, it seems to > me that this simpler approach solves all of our problems, with the > advantage of introducing exactly *zero* (not small, but truly zero) > overhead for the normal use that has no customizations. > > Does this sound reasonable to others, or did I totally miss this > particular boat? I really want us all to be happy with the solution > we find to this so that we move forward reasonably convinced of it > being viable. > > Cheers, > > f > From hans_meine at gmx.net Thu Jul 24 04:44:45 2008 From: hans_meine at gmx.net (Hans Meine) Date: Thu, 24 Jul 2008 10:44:45 +0200 Subject: [IPython-dev] ipython1 and synchronous printing of stdout In-Reply-To: <6ce0ac130807221325y19ffd8dbtafc27d1c6ca5f4ac@mail.gmail.com> References: <20080720164615.GB23971@phare.normalesup.org> <6ce0ac130807221325y19ffd8dbtafc27d1c6ca5f4ac@mail.gmail.com> Message-ID: <200807241044.46716.hans_meine@gmx.net> Am Dienstag, 22. Juli 2008 22:25:45 schrieb Brian Granger: > 3. The Observer/Delegate patterns as are typical is Cocoa. I know > that Gael is not fund of this, but I do think this pattern is much > better than plain callbacks, as it give a nice formal structure to > everything. Also, I agree with Barry that this pattern is pretty much > what you get when you try to do plain callbacks properly. > > Thus, my own conclusion is that the Observer/Delegate pattern is > probably the best option for us and will lead to the loose coupling > that we need. > > Barry, what would it take to put together an example of this pattern? > Will we need some amount of infrastructure to make this possible? My experience is that a callback mechanism is easily implemented in Python as a (standard python) list of callables that are called in turn. Optionally, you may allow to specify priorities when adding callbacks to get the proper order, but that looks like overkill here. This relies on Python's automatic member function binding and duck typing (i.e. you can pass objects with a __call__ method, normal functions, or bound member functions) and works very well. AFAICS, there is no big difference between supporting callbacks and the Observer pattern; if you provide an "addOutputCallback" method that internally appends its argument to a list of outputCallbacks, an Observer would simply call that method and pass self (if it has a specific __call__ method) or e.g. self._watchOutput. The latter IMHO makes it so powerful because an Observer may easily watch several different Subjects without being forced to follow a certain interface (i.e. method names) for that. Just my 2 cents, Hans PS: Forgot to press send; I actually wrote this about 24 hours ago. Looks nearly off-topic to me now, but anyhow.. From barrywark at gmail.com Thu Jul 24 16:23:53 2008 From: barrywark at gmail.com (Barry Wark) Date: Thu, 24 Jul 2008 15:23:53 -0500 Subject: [IPython-dev] ipython1 and synchronous printing of stdout In-Reply-To: References: <20080720164615.GB23971@phare.normalesup.org> <20080720192702.GH23971@phare.normalesup.org> <6ce0ac130807201250m26fb1a5flcbc9bdb62844cda8@mail.gmail.com> <20080721045021.GP31836@phare.normalesup.org> <6ce0ac130807221325y19ffd8dbtafc27d1c6ca5f4ac@mail.gmail.com> <6ce0ac130807232222j295701denfa660731fc729f07@mail.gmail.com> Message-ID: I fear that I may have muddied the waters in talking about notification and delegation all together. They really solve two different problems. I'll try to lay that out below, but the short version is that notification/observation seems to be the most appropriate when the flow of information is unidirectional (e.g. a frontend wants to know that something was printed to stdout), whereas delegation is more appropriate when the flow is bidirectional (e.g. the Shell wants to give a delegate the opportunity to modify its internal behavior). I think that Brian and I are really talking about the former, while Gael and Fernando are really talking mostly about the later. I'll try to make the case below that delegation is not the right solution for the observer problem, but that it may also be very useful. I will be truly offline until Sunday, so I'll say my piece here, and let the discussion run its course. On Thu, Jul 24, 2008 at 12:22 AM, Brian Granger wrote: >> I think there's a fourth, that Gael mentioned in passing and which >> we've been discussing here face to face for a while: using objects >> whose basic behavior defines the core API, but which can be given at >> construction time instead of being hardcoded in our various __init__ >> methods. This allows subclasses (or even non-subclassing application >> writers who use the code as a library to build their own tools) to >> cleanly provide only the level of behavior modification they need. > > Yes, this is a design pattern that I very much like and that is well > suited for certain things in ipython. We actually use this in a few > places in the ipython1 code: > > 1. An IPython engine takes an interpreter class as a parameter in its > init method. Thus a user can pass the engine a custom interpreter > subclass to get custom behavior. > > 2. We don't do it yet, but the interpreter could take a Traited dict > like object to use as the users namespace. I agree that parameterizing many of the iteractions is useful (especially for testing). When we want to be able to modify the Interperter's behavior without subclassing Interpreter, this is probably the right approach. However, if we believe that the intersection of notifications that the set of delegates (e.g. the frontends) may want to recieve is non-empty, the observer pattern may be more appropriate *for event notification*... As Brian points out below, delegation can lead to some unexpected dependencies. Let me give one other example: suppose we use subclasses of parameterized objects to add behavior to the interpreter. In this scenario, the Interpreter constructor gets a parameter to which it delegates responsibility for handling event notification (by callback, observer pattern, etc.). Suppose Gael implements this object to use callbacks for the events he needs in his Wx frontend. I also implement a subclass to provide an observer-pattern based implementation of event handling. Fernando wants to avoid the overhead of events/callbacks and so passes None as this paramteer (or provides a subclass that has no-op methods) [1]. Now imagine that we want to add new functionality to the Interpreter class that changes when or how the Interpreter will call this parameterized object (such as calling the delegate's wrote_to_stdout method when the Interpreter writes to stdout). Even if the change does not directly affect the use-case for which Gael and I designed our subclasses, all our code needs to be reviewed to see if it is affected by this change in the Interpreter's internal implementation (in the example case, all of the possible delegates need to be checked to make sure they didn't use wrote_to_stdout as a method name for a different purpose *and* the common base class needs to be updated to respond to the wrote_to_stdout method with a no-op). In other words, because there is a direct contractual relationship between the Interpreter and these subclasses, dependencies are likely. In the observer pattern, the Interpreter could fire a new event, but it would be ignored by any observer that didnt' explicitly register for that event type. No changes needed outside of the Interpreter class. A real world example, I believe, is the matplotlib backend architecture. The matplotlib library uses a plugable backend module to render plots. Each backend thus needs to be maintained in parallel, a difficult and growing problem for the Matplotlib team. However, as Brian points out, delegation is *very* useful and is probably the best way to handle the user_ns use case. > > We definitely should do more of this. But, I think this design > pattern addresses a slightly different need than the > callback/observer/notification stuff. Here is why: > > The real benefit of the observer/notification model is that everything > can be completely transient. For example, you might have a notifier > that observes the user's namespace. But you might only want it to > operate when the user opens a specific window. Another example is > when an observer is hooked up to an unreliable network connection. So > I guess I would ad another design constraint that I have until now > kept in the back of my mind: we need loose coupling that is also > dynamic and transient. And for these situations, I still think the > observer pattern is the best solution. Just to reiterate and paraphrase Brian, the observer pattern lets the Interpreter not know *anything* about the frontend (even whether it exists or not) that is observing its behavior and it lets the frontend not know *anything* about the Interpreter's implementation except that it will fire notifications for the events defined in its interface. So, when a new event is added to the Interpreter, no observer code needs to be modified (it doesn't even need to know that the new notification is fired). If the Interpreter's implementation is changes so that, e.g. notification of writing to stdout happens immediately after writing or after some short delay, none of the observing code needs to be reviewed as long as the original Interpreter API didn't specify the exact time. In other words, using an intermediary notification center enforces a loose coupling; the only dependency between Interpreter and frontend is the API that defines the event/notificaiton itself. Using the observer pattern makes adding new event types to the shell easy too. For example, we could write function decorators to fire events before/after the function. So, assuming all stdout goes to a print_stdout() method in shell or some such, then print_stdout(...): blah... becomes @notify_after(STDOUT_EVENT_TYPE) print_stdout(...): blah... In other words, no code in the function that fires the notification needs to be modified if the notification can be fired before or after the entire methods completes. We benefit from code reuse in the notify method and don't have to rewrite similar functionality for each individual callback use (this was Brian's point that a properly implemented callback system is, essentially, a notification center). In the case of Fernando's use-case above, notify_after() can call through directly to the print_stdout function. The only performance penalty is one additional method call. Is that acceptable? > >> This seems to provide a simple solution to our design question: all we >> need to do is to clarify which objects are our core publicly >> modifiable components in the API, and then users can tweak only the >> parts they need. Said objects can themselves provide >> observer/delegation behavior if they so desire, but they can do it in >> the most appropriate way for a given problem. For example, someone >> writing a Traits app (who's already committed to using Traits for >> their own reasons) can simply stick in there traited versions of the >> same things, and with essentially zero extra code they get the Traits >> event model everyhwere. > > I worry that such interfaces _can_ at times be a bit too implicit and > thus hide subtle behaviors and introduce code coupling that isn't > obvious. One example of a subtle behavior is the following. Traits > is based on a model that says this "an interface consists of a set of > attributes that you simply get and set as atributes" But, the second > you go to propagate an interface over a network connection, you > discover that it is really difficult. Interfaces that "network > friendly" tend to be i) be methods/functions that ii) have arguments > with very basic types. > > We do have some examples in IPython.kernel where we propagate > interfaces that are attribute based, but a good amount of extra work > and care is required. > > Thus, while I think the idea of passing a Traited dict to the > interpreter to observe the user's namespace is a beautiful idea, the > reality is much more painful: there is no straightforward way of > propagating that interface over a network connection. > > With a standardize observer/notifier pattern that is based on methods, > it is very straightforward to: > > * Propagate observation/notification events over a network > * Unregister such events when network connections fail > * Integrate such events with Twisted. > * Integrate with other event loops like wx, qt, cocoa > > One thing that I hadn't realized is that the observer/notifier pattern > has come up for Min and I in IPython.kernel a number of times, but we > have never taken the time to really abstract it properly. > > .....a few minutes later... > > I just reread this email and I think I am talking in circles and a bit > tired. But, I do think we need to have a solution that is "network > friendly". > > Cheers, > > Brian > > PS - starting tomorrow, I will be offline for about a week. > > >> In a Cocoa environment, these objects could be lightly wrapped >> versions of the originals that register (via a modified __getattr__ >> for example) the necessary observation/update calls into the >> NSDistributedNotificationCenter that Cocoa provides, if that's the >> best way to do it in such an environment. >> >> And someone who just needs a callback here and there can simply stick >> their callback-enhanced object where they want. I could even see this >> being very handy for occasional debugging, by quickly activating >> tracing versions of certain objects as needed. >> >> One thing that I like about this approach is that it doesn't >> necessarily close the door on a *future* implementation of an >> ipython-wide notification center, if we end up finding in practice >> that it's really needed as the code grows. But right now, it seems to >> me that this simpler approach solves all of our problems, with the >> advantage of introducing exactly *zero* (not small, but truly zero) >> overhead for the normal use that has no customizations. [1] Re: performance I believe that Fernando's "no-op" parameter must incur *some* overhead; either the Interpreter must test for None, or the Interpreter must call a method in Fernando's object which does nothing. In other words, there's no free lunch. I can't think of a way to add the ability to notify observers and/or modify the behavior of Interpreter via delegates with truely zero overhead while avoiding the parallel implementation problem that I describe above. >> >> Does this sound reasonable to others, or did I totally miss this >> particular boat? I really want us all to be happy with the solution >> we find to this so that we move forward reasonably convinced of it >> being viable. >> >> Cheers, >> >> f >> > From fperez.net at gmail.com Thu Jul 24 22:45:38 2008 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 24 Jul 2008 19:45:38 -0700 Subject: [IPython-dev] ipython1 and synchronous printing of stdout In-Reply-To: References: <20080720164615.GB23971@phare.normalesup.org> <6ce0ac130807201250m26fb1a5flcbc9bdb62844cda8@mail.gmail.com> <20080721045021.GP31836@phare.normalesup.org> <6ce0ac130807221325y19ffd8dbtafc27d1c6ca5f4ac@mail.gmail.com> <6ce0ac130807232222j295701denfa660731fc729f07@mail.gmail.com> Message-ID: I realize that Barry and Brian will be going offline, and I'll be traveling tomorrow back home as well, so this is mostly for the conversation to continue later, but while the ideas are in my head. I greatly appreciate that everyone is trying to wrap their heads around a rather abstract and difficult question, but I do worry a bit that it's getting too far for me to make a solid judgment on which road will really work better. We all agree on the basics we want (loose coupling, acceptable performance, etc), but we're trying to figure out a flexible architecture that lets us implement things we need: - across gui toolkits - out of process (which more or less implies also in a network-independent way, since something that can be communicated out of process can be sent over the network, given that we're not using shared memory tricks). The question is how to get from here to there with the least amount of pain and keeping us all one happy family :) I'd really prefer it if we didn't end up with a fork just because the Wx/traits frontend gets bogged down in a heavy and over-abstracted notification API, so I hope we can figure out a viable solution for the lot. On Thu, Jul 24, 2008 at 1:23 PM, Barry Wark wrote: > As Brian points out below, delegation can lead to some unexpected > dependencies. Let me give one other example: suppose we use subclasses > of parameterized objects to add behavior to the interpreter. [...snip careful description of inheritance issues...] Note that the problems you point out here are simply the generic issues that exist with subclassing with Python: any time you use a subclass in python, you have to worry about the parent class in the future growing new attribute names that may collide with your own. That's actually a flaw of the Python object model that is rarely discussed, and which languages like C++ in fact do not have. But this particular issue is a general OO/Python one, that has nothing to do with a notification pattern or event handling. [Brian] >> The real benefit of the observer/notification model is that everything >> can be completely transient. For example, you might have a notifier >> that observes the user's namespace. But you might only want it to >> operate when the user opens a specific window. Another example is >> when an observer is hooked up to an unreliable network connection. So >> I guess I would ad another design constraint that I have until now >> kept in the back of my mind: we need loose coupling that is also >> dynamic and transient. And for these situations, I still think the >> observer pattern is the best solution. > > Just to reiterate and paraphrase Brian, the observer pattern lets the > Interpreter not know *anything* about the frontend (even whether it > exists or not) that is observing its behavior and it lets the frontend > not know *anything* about the Interpreter's implementation except that > it will fire notifications for the events defined in its interface. > So, when a new event is added to the Interpreter, no observer code > needs to be modified (it doesn't even need to know that the new > notification is fired). If the Interpreter's implementation is changes > so that, e.g. notification of writing to stdout happens immediately > after writing or after some short delay, none of the observing code > needs to be reviewed as long as the original Interpreter API didn't > specify the exact time. In other words, using an intermediary > notification center enforces a loose coupling; the only dependency > between Interpreter and frontend is the API that defines the > event/notificaiton itself. The part that worries me is building YAEHA (Yet Another Event Handling API)... Matplotlib did exactly that, and while it does allow you to wire events using pure matplotlib code, it adds real, non-trivial complexity to the code and makes it harder to fit into a specific event handling API 'from the front'. All GUI systems have their own event models, so does Twisted, and I'm really not convinced that IPython should grow its own (next to matplotlib's, and Traits', and WX's, ...) On the other hand, I think this is a discussion that is beginning to get far enough into design abstractions that it's escaping my (admittedly very limited) ability to gauge fruitfully the real world outcomes. So here's a concrete proposal: Gael is already hard at work (and far along) on a WX frontend that Enthought is funding for use in Envisage apps. Since this is real, working code, let's see where it goes. In the meantime, Barry can play with the design for a lightweight notification center idea, and actual implementation will likely be very enlightening. I've got my hands more than full trying to get at least a first cut of all the tests that appeared (once I completed the testing plugin, all sorts of little things got picked up as tests that now need to actually work). Since we're trying to push a release before scipy, this is really more than we can do all at the same time. With Gael's and Barry's implementation on hand, I think it will be easier to see if the notification center can be made lightweight enough for adapting it to Traits/Twisted/Cocoa/whatever without undue pain, or if an object-by-object approach ends up being better. I can easily see it being both: object parametrization for certain kinds of architectural customization with a little message center for lightweight notifications. But honestly, I'm finding it hard to see far enough into the fog with pure 'whiteboard' thinking. We always brag about how nice it is to use python to actually implement ideas rather than only think about them in the abstract, I think in this case this may be the wisest approach. > [1] Re: performance > I believe that Fernando's "no-op" parameter must incur *some* > overhead; either the Interpreter must test for None, or the > Interpreter must call a method in Fernando's object which does > nothing. In other words, there's no free lunch. I can't think of a way > to add the ability to notify observers and/or modify the behavior of > Interpreter via delegates with truely zero overhead while avoiding the > parallel implementation problem that I describe above. No, the None check is only done at isntantiation time of the main objects. Afterwards, it's the responsibility of the subclasses to produce/manage events for specific behavior. The default object will for example have a .write() method that simply prints, while an event handling subclass would have its .write() be Traited so that a GUI could listen in, or a Twisted one would handle it over the network. So it really is zero-overhead, simply because in the default, there is no event production/consumption at all. So from a performance perspective, that approach really produces exactly zero cost. The question that I think will be best answered with a bit of prototyping, is whether the architectural costs become greater than those of a message center. Gael just emphasized that in the code he's doing, if it turns out the message center approach works well, there's little cost in adding it later on. So it's really not an either/or issue, just one of getting functioning prototypes working quickly so we can judge based on the actual code rather than pure abstractions. Does this sound like a reasonable plan? Cheers, f From barrywark at gmail.com Sun Jul 27 22:02:08 2008 From: barrywark at gmail.com (Barry Wark) Date: Sun, 27 Jul 2008 22:02:08 -0400 Subject: [IPython-dev] ipython1 and synchronous printing of stdout In-Reply-To: References: <20080720164615.GB23971@phare.normalesup.org> <20080721045021.GP31836@phare.normalesup.org> <6ce0ac130807221325y19ffd8dbtafc27d1c6ca5f4ac@mail.gmail.com> <6ce0ac130807232222j295701denfa660731fc729f07@mail.gmail.com> Message-ID: Taking a bit of my own medicine?that working code is better than awesome theory?I've checked in the ipython-notification branch and proposed it for merging into trunk. It includes a functional spec in /blueprints (it's a bit tounge-in-cheek; I hope no one is offended) as well as an implementation of the NotificationCenter I've been discussin in IPython.kernel.core.notification. Hopefully code will answer more questions than the "fog" we've created so far. I've added a couple of comments below, but I'd guess a quick look at the code is more valuable. B On Thu, Jul 24, 2008 at 10:45 PM, Fernando Perez wrote: > I realize that Barry and Brian will be going offline, and I'll be > traveling tomorrow back home as well, so this is mostly for the > conversation to continue later, but while the ideas are in my head. > > I greatly appreciate that everyone is trying to wrap their heads > around a rather abstract and difficult question, but I do worry a bit > that it's getting too far for me to make a solid judgment on which > road will really work better. We all agree on the basics we want > (loose coupling, acceptable performance, etc), but we're trying to > figure out a flexible architecture that lets us implement things we > need: > > - across gui toolkits > > - out of process (which more or less implies also in a > network-independent way, since something that can be communicated out > of process can be sent over the network, given that we're not using > shared memory tricks). > > The question is how to get from here to there with the least amount of > pain and keeping us all one happy family :) I'd really prefer it if > we didn't end up with a fork just because the Wx/traits frontend gets > bogged down in a heavy and over-abstracted notification API, so I hope > we can figure out a viable solution for the lot. I hope you'll see that this is a pretty light-weight implementation... > > > On Thu, Jul 24, 2008 at 1:23 PM, Barry Wark wrote: > >> As Brian points out below, delegation can lead to some unexpected >> dependencies. Let me give one other example: suppose we use subclasses >> of parameterized objects to add behavior to the interpreter. > [...snip careful description of inheritance issues...] > > Note that the problems you point out here are simply the generic > issues that exist with subclassing with Python: any time you use a > subclass in python, you have to worry about the parent class in the > future growing new attribute names that may collide with your own. > That's actually a flaw of the Python object model that is rarely > discussed, and which languages like C++ in fact do not have. > > But this particular issue is a general OO/Python one, that has nothing > to do with a notification pattern or event handling. I wasn't trying to suggest that this is an observer pattern-specific problem, just that this particular problem is likely to crop up in the notification/callback context if we're not careful. Probably not a great reason to go with one solution over the other, however. > > [Brian] > >>> The real benefit of the observer/notification model is that everything >>> can be completely transient. For example, you might have a notifier >>> that observes the user's namespace. But you might only want it to >>> operate when the user opens a specific window. Another example is >>> when an observer is hooked up to an unreliable network connection. So >>> I guess I would ad another design constraint that I have until now >>> kept in the back of my mind: we need loose coupling that is also >>> dynamic and transient. And for these situations, I still think the >>> observer pattern is the best solution. >> >> Just to reiterate and paraphrase Brian, the observer pattern lets the >> Interpreter not know *anything* about the frontend (even whether it >> exists or not) that is observing its behavior and it lets the frontend >> not know *anything* about the Interpreter's implementation except that >> it will fire notifications for the events defined in its interface. >> So, when a new event is added to the Interpreter, no observer code >> needs to be modified (it doesn't even need to know that the new >> notification is fired). If the Interpreter's implementation is changes >> so that, e.g. notification of writing to stdout happens immediately >> after writing or after some short delay, none of the observing code >> needs to be reviewed as long as the original Interpreter API didn't >> specify the exact time. In other words, using an intermediary >> notification center enforces a loose coupling; the only dependency >> between Interpreter and frontend is the API that defines the >> event/notificaiton itself. > > The part that worries me is building YAEHA (Yet Another Event Handling > API)... Matplotlib did exactly that, and while it does allow you to > wire events using pure matplotlib code, it adds real, non-trivial > complexity to the code and makes it harder to fit into a specific > event handling API 'from the front'. All GUI systems have their own > event models, so does Twisted, and I'm really not convinced that > IPython should grow its own (next to matplotlib's, and Traits', and > WX's, ...) I appologize if I gave the impression I was trying to create YAEHA. This is about non-frontend events. There shouldn't be much, if any overlap between GUI events and the notifications we're talking about. It really is just a clean implementation of callbacks, nothing more, hopefully not too much less :) > > On the other hand, I think this is a discussion that is beginning to > get far enough into design abstractions that it's escaping my > (admittedly very limited) ability to gauge fruitfully the real world > outcomes. So here's a concrete proposal: Gael is already hard at > work (and far along) on a WX frontend that Enthought is funding for > use in Envisage apps. Since this is real, working code, let's see > where it goes. In the meantime, Barry can play with the design for a > lightweight notification center idea, and actual implementation will > likely be very enlightening. I've got my hands more than full trying > to get at least a first cut of all the tests that appeared (once I > completed the testing plugin, all sorts of little things got picked up > as tests that now need to actually work). Since we're trying to push > a release before scipy, this is really more than we can do all at the > same time. > > With Gael's and Barry's implementation on hand, I think it will be > easier to see if the notification center can be made lightweight > enough for adapting it to Traits/Twisted/Cocoa/whatever without undue > pain, or if an object-by-object approach ends up being better. I can > easily see it being both: object parametrization for certain kinds of > architectural customization with a little message center for > lightweight notifications. As always, the best solution is likely the middle road between two good options. > > But honestly, I'm finding it hard to see far enough into the fog with > pure 'whiteboard' thinking. We always brag about how nice it is to > use python to actually implement ideas rather than only think about > them in the abstract, I think in this case this may be the wisest > approach. > > >> [1] Re: performance >> I believe that Fernando's "no-op" parameter must incur *some* >> overhead; either the Interpreter must test for None, or the >> Interpreter must call a method in Fernando's object which does >> nothing. In other words, there's no free lunch. I can't think of a way >> to add the ability to notify observers and/or modify the behavior of >> Interpreter via delegates with truely zero overhead while avoiding the >> parallel implementation problem that I describe above. > > No, the None check is only done at isntantiation time of the main > objects. Afterwards, it's the responsibility of the subclasses to > produce/manage events for specific behavior. The default object will > for example have a .write() method that simply prints, while an event > handling subclass would have its .write() be Traited so that a GUI > could listen in, or a Twisted one would handle it over the network. > So it really is zero-overhead, simply because in the default, there is > no event production/consumption at all. Ah, I see. You're proposing moving _all_ the event generating code into the parametrized objects. Now I get it. I've written into the spec O(1) performance of my implementation, but it currently depends on the (undocumented) complexity of the builtin set type. We'll have to see if performance is acceptable. > > So from a performance perspective, that approach really produces > exactly zero cost. The question that I think will be best answered > with a bit of prototyping, is whether the architectural costs become > greater than those of a message center. And I suppose whether the added number of objects and method calls ends up being the same cost as the notification center implementation. > > Gael just emphasized that in the code he's doing, if it turns out the > message center approach works well, there's little cost in adding it > later on. So it's really not an either/or issue, just one of getting > functioning prototypes working quickly so we can judge based on the > actual code rather than pure abstractions. > > Does this sound like a reasonable plan? Absolutely. I look forward to hearing you (and others') opinions on the ipython-notification branch. Cheers, Barry From edreamleo at gmail.com Mon Jul 28 11:27:40 2008 From: edreamleo at gmail.com (Edward K. Ream) Date: Mon, 28 Jul 2008 10:27:40 -0500 Subject: [IPython-dev] Thoughts re Leo and Ipython notebooks. Message-ID: I've just posted some thoughts re Leo as a compound document: http://groups.google.com/group/leo-editor/browse_thread/thread/759d20f81119e5ca Among other things, the post discusses adding spreadsheet-like features to Leo in a fairly easy way. These are just ideas, intended to encourage further invention. My intuition tells me that IPython folk might find these ideas useful or provocative: - Leo could be considered an IPython notebook. - Generalized spreadsheets might be useful in recording scientific experiments. - Scientific experiments might want to generate notebook/spreadsheet entries using scripts. - The post shows how to declare the "type" of scripts/nodes and show how to embed scripts of arbitrary types into notebook entries. Imo, we have only just scratched the surface of what can be done with scripts and notebooks. Edward -------------------------------------------------------------------- Edward K. Ream email: edreamleo at gmail.com Leo: http://webpages.charter.net/edreamleo/front.html -------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Wed Jul 30 20:55:28 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 31 Jul 2008 02:55:28 +0200 Subject: [IPython-dev] ipython1 and synchronous printing of stdout In-Reply-To: References: <20080721045021.GP31836@phare.normalesup.org> <6ce0ac130807221325y19ffd8dbtafc27d1c6ca5f4ac@mail.gmail.com> <6ce0ac130807232222j295701denfa660731fc729f07@mail.gmail.com> Message-ID: <20080731005528.GA13565@phare.normalesup.org> Hi, Sorry for the silence, I have been working my ass off on a release of Mayavi. Now I can go back to working on ipython. On Sun, Jul 27, 2008 at 10:02:08PM -0400, Barry Wark wrote: > Taking a bit of my own medicine?that working code is better than > awesome theory?I've checked in the ipython-notification branch and > proposed it for merging into trunk. It includes a functional spec in > /blueprints (it's a bit tounge-in-cheek; I hope no one is offended) as > well as an implementation of the NotificationCenter I've been > discussin in IPython.kernel.core.notification. I had a look at that. I seems to me like a (good) textbook implementation of the patterns we are discussing. I still fail to see the point of having this in Ipython, at least at the state we are in right now. My point is that there are many implementation of such frameworks. The one you have developed is a simple one, but when you start using it a lot, you will need to add features, like delegations, or ordering the callbacks. Also, something that is very important is how does this blend in the code people write? How do you make it easy for people to write notifications without thinking too much about it, and registering handlers (hint decorators are cool). But once again, this looks a lot to me like traits 1. I am not trying to sell traits here, I don't think IPyhton should use traits, I am just flashing a warning: this smells like weel reinvention, and on top of that developing a framework, which is something you want to limit as much as possible: we already have too many frameworks. I have the feeling every single person coming to develop an app using Ipython will already have his notification framework (traits, cocoa, does Qt have one?). So now he will to write adapting code to use your framework. Now there is value in having an iptyhon-specific notification layer, and it is not having uniform code across backends; matplotlib tried that, and it doesn't bring much value, amongst other things because the backend are anyhow riddled with toolkit-specific code. The value is that we could think of network-transparent notifications. That is much harder (obviously, the simple problems don't need solutions), and I would like to know if twisted doesn't provide this kind of service. I guess my point is not that we shouldn't do this, but that we should do this when the need arises, and not now, for no benefit. Doing this too early is a good way of getting it wrong. In addition this will just slow our progress towards getting frontends and we absolutely do not need this to get the simple frontends. I am not going to spend anytime on this, and spending too much time on this is a good way of not having any useable code for a long time, I believe. Sorry, not only am I not paid to worry about distant future, or network-transparency problems, I am also having difficulties being interested in something that I cannot specify properly because it doesn't answer a problem I have. I am not in the business of making a framework, I am in the business of making a widget. The whole notification framework stuff, for me, will be done in traits, because this is what we use all over our apps, because we have a lot of code that works well with it, and because is it a mature and powerful library. I develop my apps using traits, and I want an ipython widget to plug in my apps. If the objects used internally in the ipython widget are standard object like dicts or lists, I can use traits without any problem. If there are part of a notification framework that I have to adapt to, I probably have to double the size of my codebase. Let us see where we really need this framework. Maybe it can also be developed in a subclass of the standard objects we use in ipython, say a different frontend base class? I hope I am not being to rough or criticizing too much your software engineering efforts. I am just trying to be practical and figure out how to ship software that is useful to users (and yes this includes open-source software) in a reasonable amount of time. And the basis of agile methods is to get it working, ship it to your users, see the limitations in your implementation, refactor, ship it again, always focusing on what benefits to your users, rather than on the architecture or the software engineering, which is a tool, not a goal. As for your code, if you want to merge it in, it is fine by me, but I am afraid it won't be used for a while, and when we get to use it, we might want to change it because we have more insight on our problems. Thanks for bringing up on interesting discussion, Ga?l