From ronena at gmail.com Sun Mar 1 05:10:34 2009 From: ronena at gmail.com (Ronen Abravanel) Date: Sun, 1 Mar 2009 12:10:34 +0200 Subject: [IPython-dev] Ipython plugin for Editra In-Reply-To: <49a9e60b.07eb300a.6f2e.010c@mx.google.com> References: <2498a980902281352q48df3928v1b6bb05662f31ef3@mail.gmail.com> <49a9e60b.07eb300a.6f2e.010c@mx.google.com> Message-ID: <2498a980903010210t23201eafx749eef70368f6281@mail.gmail.com> Hi Laurent, My hack attached. It is mostly defined at "ShellIntegrator" class at the end of the file. The work flow I want to achieve is as following: * in Ipython - "%edit myfile.py" or "edit 1:5" etc. Open editing buffer within the current windows, and returns without running the file. * Selectin "Run" from the tool menu in Editra run the buffer, as it was sopused to run after closing the editor. By the way, the plugin have some series stabilty issues. calling '%run filename.py' for none trevial python file is cusing editra crash with: python2.5: ../../src/xcb_lock.c:77: _XGetXCBBuffer: Assertion `((int) ((xcb_req) - (dpy->request)) >= 0)' failed. Aborted Thanks, Ronen On Sun, Mar 1, 2009 at 3:33 AM, Laurent Dufr?chou < laurent.dufrechou at gmail.com> wrote: > Hi Ronen, > > > > Happy to see you?ve tried to hack the editra plugin ;) > > > > IPyShell/__init__.py contains only the interface with editra. > > I do not cleanly understand the step your are trying to do. You?ve used ? > %ed? than it launched editra new source panel? And closing it executed your > code, but stdin/out goes outside ipython shell? > > Is that what you tried? > > > > About the files to modify. __init__.py in editra plugn is the definition of > the plugin interface and how you interact with editra. > > Ipython dir is a pure copy that I was used to upload from time to time. I > have to take a look at some updates that cody did. (some compatibility issue > with python 2.6 and ipython tree) > > The fact is that the update you?re currently trying to di will surely need > some update into /gui/wx I think L. But well we?ll see. > > Depending on what you?ll want to do we will perhaps expose some methods so > plugin will be able to configure the things you need. (ex being able to > define an editor) > > (I think we can use ipython config file but I?ve never used them since then > so?) > > > > I will try to upload updated sources for python tree tomorrow, so you?ll > have a bleeding edge wx integration. I?ve corrected some bugs since then. > > Sure we will find a way to workaround the stdin/out issue you saw. > > > > Send me your current hack, I?ll take a look at your issue. > > > > Cheers, > > Laurent > > > > *De :* ipython-dev-bounces at scipy.org [mailto:ipython-dev-bounces at scipy.org] > *De la part de* Ronen Abravanel > *Envoy? :* samedi 28 f?vrier 2009 22:53 > *? :* ipython-dev at scipy.org > *Objet :* [IPython-dev] Ipython plugin for Editra > > > > Editra, an extensible programmer's editor written in wxPython, contain a > cool IPython plugin. This mail is a follow up for a discussion with > Laurent , The author of the plugin, in his blog, continue here according > to his request. The Orginal discassion can be found in > http://ipython0.wordpress.com/2008/04/25/editra-ipython-plugin-under-work/or at the end of this mail. > > > Using papi to set\redefine IPython's magic word partly do the job, The > other part is to execute some arbitrary python code inside The ipython > shell. Invoking papis "ex" method (Using papi from *ipython_panel.IP._IP.getapi() > *) let the Code run in IPython's context, but the output is printed to > Editra's stdout (The console in wich I invoked editra's from) and not in the > IPython in the shelf. My attempts to set sys.stdout to an handler from, let > say, _term.cout or others, cused IPython to stack. futher more, the Exeption > throws while I call ip.ex, is cought within Editra and withn IPython. > > Further more - I wish to change, If I can, Only IPyShell/__init__.py and > not the stuff in "IPython" dir, as it seems as a "pure" copy of IPython's > orginal source (is it?) > > > Thanks, > Ronen. > > > -------------------------------------- > The orginal discassion: > > 1. Ronen Says: > February 21, 2009 at 8:18 pm > > That?s a one cool Plugin. > > What will make it ideal, is better integration with Editra?s Editor: > Make ipython?s %edit Open the edit-buffer in Editra?s current Windows, and > Enable poeple to run The code they edit in Editra inside the open IPython > windows? > Any hint where should I try to hack in order to achieve those? > > Thanks, > Ronen. > > 1. Laurent Says: > February 22, 2009 at 10:45 am > > Hello, > You can have access to the plugin repos: > > # Non-members may check out a read-only working copy anonymously over HTTP. > svn checkout http://editra-plugins.googlecode.com/svn/trunk/editra-plugins-read-only > > The plugin name is ipyshell, in it?s directory you?ve got the whole ipython > source + the plugin source. > > It need a little refresh for sure [image: :)]. > > If you want to try to hack it take a look @Ipython/gui/wx the whole code is > in ipython_view and ipshellnonblocking. > > ipython_view -> gui interface > ipshellnonblocking -> the interface with ipython > > You can add: > ip = IPython.ipapi.get() > def myEdit(self, arg): > print ?do what you want? > ip.expose_magic(?edit?, myEdit) > > to WxNonBlockingIPShell inside ipython_view.py > > Please note that if ipython is ever installed on your system you?ll have to > make a eggsetup.py develop to make sure to use your new local copy. > > We can discuss about all of this in ipython-dev mailing list [image: :)] > > Cheers, > Laurent > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: __init__.py Type: text/x-python Size: 9005 bytes Desc: not available URL: From laurent.dufrechou at gmail.com Sun Mar 1 07:35:44 2009 From: laurent.dufrechou at gmail.com (=?UTF-8?Q?Laurent_Dufr=C3=A9chou?=) Date: Sun, 1 Mar 2009 13:35:44 +0100 Subject: [IPython-dev] Ipython plugin for Editra In-Reply-To: <2498a980903010210t23201eafx749eef70368f6281@mail.gmail.com> References: <2498a980902281352q48df3928v1b6bb05662f31ef3@mail.gmail.com> <49a9e60b.07eb300a.6f2e.010c@mx.google.com> <2498a980903010210t23201eafx749eef70368f6281@mail.gmail.com> Message-ID: <49aa8130.0508d00a.21e6.ffff8ea1@mx.google.com> Hey quite interesting ! I?ve seen you get directly access to ipython directly inside the plugin, quite good! To execute lines of code, it is currently done like this in ipython_view.py: User push enter -> #------------------------ Key Handler ------------------------------------ def keyPress(self, event): if event.KeyCode == wx.WXK_RETURN: if self.cur_state == 'IDLE': #we change the state ot the state machine self.setCurrentState('DO_EXECUTE_LINE') self.stateDoExecuteLine() return def stateDoExecuteLine(self): lines=self.text_ctrl.getCurrentLine() self.text_ctrl.write('\n') lines_to_execute = lines.replace('\t',' '*4) lines_to_execute = lines_to_execute.replace('\r','') self.IP.doExecute(lines_to_execute.encode(ENCODING)) self.updateHistoryTracker(lines) self.setCurrentState('WAIT_END_OF_EXECUTION') So the hackish way to do you what you?re trying to do is: (self was self._ipanel for your plugin) Set self._ipanel.setCurrentState('DO_EXECUTE_LINE'), this is necessary because there is s state machine, and depending what the gui is doing key event and behavior change. This is usefull when displaying help for example where display and keys are handled differently. Then do: nb = self._mw.GetNotebook() current = nb.GetCurrentCtrl() Lines_to_execute = current.GetSelectedText() lines_to_execute = lines.replace('\t',' '*4) lines_to_execute = lines_to_execute.replace('\r','') self._ipanel.IP.doExecute(lines_to_execute.encode(ENCODING)) self._ipanel.updateHistoryTracker(lines) self._ipanel.setCurrentState('WAIT_END_OF_EXECUTION') the best way should be to have a method to allow execution of an external string buffer. If the method that I tell you works, I will change the method to something like def stateDoExecuteLine(self, string). Will resintall my plugin dev env of editra. Hope I can have some time today, but not sure? I forgot I got something urgent to do before ? J Laurent De : Ronen Abravanel [mailto:ronena at gmail.com] Envoy? : dimanche 1 mars 2009 11:11 ? : Laurent Dufr?chou Cc : ipython-dev at scipy.org Objet : Re: [IPython-dev] Ipython plugin for Editra Hi Laurent, My hack attached. It is mostly defined at "ShellIntegrator" class at the end of the file. The work flow I want to achieve is as following: * in Ipython - "%edit myfile.py" or "edit 1:5" etc. Open editing buffer within the current windows, and returns without running the file. * Selectin "Run" from the tool menu in Editra run the buffer, as it was sopused to run after closing the editor. By the way, the plugin have some series stabilty issues. calling '%run filename.py' for none trevial python file is cusing editra crash with: python2.5: ../../src/xcb_lock.c:77: _XGetXCBBuffer: Assertion `((int) ((xcb_req) - (dpy->request)) >= 0)' failed. Aborted Thanks, Ronen On Sun, Mar 1, 2009 at 3:33 AM, Laurent Dufr?chou wrote: Hi Ronen, Happy to see you?ve tried to hack the editra plugin ;) IPyShell/__init__.py contains only the interface with editra. I do not cleanly understand the step your are trying to do. You?ve used ? %ed? than it launched editra new source panel? And closing it executed your code, but stdin/out goes outside ipython shell? Is that what you tried? About the files to modify. __init__.py in editra plugn is the definition of the plugin interface and how you interact with editra. Ipython dir is a pure copy that I was used to upload from time to time. I have to take a look at some updates that cody did. (some compatibility issue with python 2.6 and ipython tree) The fact is that the update you?re currently trying to di will surely need some update into /gui/wx I think L. But well we?ll see. Depending on what you?ll want to do we will perhaps expose some methods so plugin will be able to configure the things you need. (ex being able to define an editor) (I think we can use ipython config file but I?ve never used them since then so?) I will try to upload updated sources for python tree tomorrow, so you?ll have a bleeding edge wx integration. I?ve corrected some bugs since then. Sure we will find a way to workaround the stdin/out issue you saw. Send me your current hack, I?ll take a look at your issue. Cheers, Laurent De : ipython-dev-bounces at scipy.org [mailto:ipython-dev-bounces at scipy.org] De la part de Ronen Abravanel Envoy? : samedi 28 f?vrier 2009 22:53 ? : ipython-dev at scipy.org Objet : [IPython-dev] Ipython plugin for Editra Editra, an extensible programmer's editor written in wxPython, contain a cool IPython plugin. This mail is a follow up for a discussion with Laurent , The author of the plugin, in his blog, continue here according to his request. The Orginal discassion can be found in http://ipython0.wordpress.com/2008/04/25/editra-ipython-plugin-under-work/ or at the end of this mail. Using papi to set\redefine IPython's magic word partly do the job, The other part is to execute some arbitrary python code inside The ipython shell. Invoking papis "ex" method (Using papi from ipython_panel.IP._IP.getapi() ) let the Code run in IPython's context, but the output is printed to Editra's stdout (The console in wich I invoked editra's from) and not in the IPython in the shelf. My attempts to set sys.stdout to an handler from, let say, _term.cout or others, cused IPython to stack. futher more, the Exeption throws while I call ip.ex, is cought within Editra and withn IPython. Further more - I wish to change, If I can, Only IPyShell/__init__.py and not the stuff in "IPython" dir, as it seems as a "pure" copy of IPython's orginal source (is it?) Thanks, Ronen. -------------------------------------- The orginal discassion: 1. Ronen Says: February 21, 2009 at 8:18 pm That?s a one cool Plugin. What will make it ideal, is better integration with Editra?s Editor: Make ipython?s %edit Open the edit-buffer in Editra?s current Windows, and Enable poeple to run The code they edit in Editra inside the open IPython windows? Any hint where should I try to hack in order to achieve those? Thanks, Ronen. 2. Erreur ! Nom du fichier non sp?cifi?.Laurent Says: February 22, 2009 at 10:45 am Hello, You can have access to the plugin repos: # Non-members may check out a read-only working copy anonymously over HTTP. svn checkout http://editra-plugins.googlecode.com/svn/trunk/ editra-plugins-read-only The plugin name is ipyshell, in it?s directory you?ve got the whole ipython source + the plugin source. It need a little refresh for sure Erreur ! Nom du fichier non sp?cifi?.. If you want to try to hack it take a look @Ipython/gui/wx the whole code is in ipython_view and ipshellnonblocking. ipython_view -> gui interface ipshellnonblocking -> the interface with ipython You can add: ip = IPython.ipapi.get() def myEdit(self, arg): print ?do what you want? ip.expose_magic(?edit?, myEdit) to WxNonBlockingIPShell inside ipython_view.py Please note that if ipython is ever installed on your system you?ll have to make a eggsetup.py develop to make sure to use your new local copy. We can discuss about all of this in ipython-dev mailing list Erreur ! Nom du fichier non sp?cifi?. Cheers, Laurent -------------- next part -------------- An HTML attachment was scrubbed... URL: From laurent.dufrechou at gmail.com Sun Mar 1 08:22:22 2009 From: laurent.dufrechou at gmail.com (=?UTF-8?Q?Laurent_Dufr=C3=A9chou?=) Date: Sun, 1 Mar 2009 14:22:22 +0100 Subject: [IPython-dev] Ipython plugin for Editra In-Reply-To: <2498a980903010210t23201eafx749eef70368f6281@mail.gmail.com> References: <2498a980902281352q48df3928v1b6bb05662f31ef3@mail.gmail.com> <49a9e60b.07eb300a.6f2e.010c@mx.google.com> <2498a980903010210t23201eafx749eef70368f6281@mail.gmail.com> Message-ID: <49aa8c1e.2315300a.0c24.3da8@mx.google.com> By the way, I send you the last update of the ipython shell. You can copy it into Ipython/gui/wx It works better, for example to test if you load the good one, just type ?raw_input()? it will not freeze and will display a box(had a problem before). I have added also an ?execute in thread checkbox? to allow disabling threading is you?re not interested with ctrl+C and want matplotlib for example.(matplotlib is not compatible with threading currently) I have to check how %run works, because all lines must be executed via self.IP.doExecute(lines_to_execute.encode(ENCODING)) Because it take care of protecting stdin/stdout (that is the main issue with using ipython0 core). So perhaps we will have to take care of some protection before calling run magic. Laurent De : Ronen Abravanel [mailto:ronena at gmail.com] Envoy? : dimanche 1 mars 2009 11:11 ? : Laurent Dufr?chou Cc : ipython-dev at scipy.org Objet : Re: [IPython-dev] Ipython plugin for Editra Hi Laurent, My hack attached. It is mostly defined at "ShellIntegrator" class at the end of the file. The work flow I want to achieve is as following: * in Ipython - "%edit myfile.py" or "edit 1:5" etc. Open editing buffer within the current windows, and returns without running the file. * Selectin "Run" from the tool menu in Editra run the buffer, as it was sopused to run after closing the editor. By the way, the plugin have some series stabilty issues. calling '%run filename.py' for none trevial python file is cusing editra crash with: python2.5: ../../src/xcb_lock.c:77: _XGetXCBBuffer: Assertion `((int) ((xcb_req) - (dpy->request)) >= 0)' failed. Aborted Thanks, Ronen On Sun, Mar 1, 2009 at 3:33 AM, Laurent Dufr?chou wrote: Hi Ronen, Happy to see you?ve tried to hack the editra plugin ;) IPyShell/__init__.py contains only the interface with editra. I do not cleanly understand the step your are trying to do. You?ve used ? %ed? than it launched editra new source panel? And closing it executed your code, but stdin/out goes outside ipython shell? Is that what you tried? About the files to modify. __init__.py in editra plugn is the definition of the plugin interface and how you interact with editra. Ipython dir is a pure copy that I was used to upload from time to time. I have to take a look at some updates that cody did. (some compatibility issue with python 2.6 and ipython tree) The fact is that the update you?re currently trying to di will surely need some update into /gui/wx I think L. But well we?ll see. Depending on what you?ll want to do we will perhaps expose some methods so plugin will be able to configure the things you need. (ex being able to define an editor) (I think we can use ipython config file but I?ve never used them since then so?) I will try to upload updated sources for python tree tomorrow, so you?ll have a bleeding edge wx integration. I?ve corrected some bugs since then. Sure we will find a way to workaround the stdin/out issue you saw. Send me your current hack, I?ll take a look at your issue. Cheers, Laurent De : ipython-dev-bounces at scipy.org [mailto:ipython-dev-bounces at scipy.org] De la part de Ronen Abravanel Envoy? : samedi 28 f?vrier 2009 22:53 ? : ipython-dev at scipy.org Objet : [IPython-dev] Ipython plugin for Editra Editra, an extensible programmer's editor written in wxPython, contain a cool IPython plugin. This mail is a follow up for a discussion with Laurent , The author of the plugin, in his blog, continue here according to his request. The Orginal discassion can be found in http://ipython0.wordpress.com/2008/04/25/editra-ipython-plugin-under-work/ or at the end of this mail. Using papi to set\redefine IPython's magic word partly do the job, The other part is to execute some arbitrary python code inside The ipython shell. Invoking papis "ex" method (Using papi from ipython_panel.IP._IP.getapi() ) let the Code run in IPython's context, but the output is printed to Editra's stdout (The console in wich I invoked editra's from) and not in the IPython in the shelf. My attempts to set sys.stdout to an handler from, let say, _term.cout or others, cused IPython to stack. futher more, the Exeption throws while I call ip.ex, is cought within Editra and withn IPython. Further more - I wish to change, If I can, Only IPyShell/__init__.py and not the stuff in "IPython" dir, as it seems as a "pure" copy of IPython's orginal source (is it?) Thanks, Ronen. -------------------------------------- The orginal discassion: 1. Ronen Says: February 21, 2009 at 8:18 pm That?s a one cool Plugin. What will make it ideal, is better integration with Editra?s Editor: Make ipython?s %edit Open the edit-buffer in Editra?s current Windows, and Enable poeple to run The code they edit in Editra inside the open IPython windows? Any hint where should I try to hack in order to achieve those? Thanks, Ronen. 2. Erreur ! Nom du fichier non sp?cifi?.Laurent Says: February 22, 2009 at 10:45 am Hello, You can have access to the plugin repos: # Non-members may check out a read-only working copy anonymously over HTTP. svn checkout http://editra-plugins.googlecode.com/svn/trunk/ editra-plugins-read-only The plugin name is ipyshell, in it?s directory you?ve got the whole ipython source + the plugin source. It need a little refresh for sure Erreur ! Nom du fichier non sp?cifi?.. If you want to try to hack it take a look @Ipython/gui/wx the whole code is in ipython_view and ipshellnonblocking. ipython_view -> gui interface ipshellnonblocking -> the interface with ipython You can add: ip = IPython.ipapi.get() def myEdit(self, arg): print ?do what you want? ip.expose_magic(?edit?, myEdit) to WxNonBlockingIPShell inside ipython_view.py Please note that if ipython is ever installed on your system you?ll have to make a eggsetup.py develop to make sure to use your new local copy. We can discuss about all of this in ipython-dev mailing list Erreur ! Nom du fichier non sp?cifi?. Cheers, Laurent -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: wx.zip Type: application/octet-stream Size: 22536 bytes Desc: not available URL: From dsdale24 at gmail.com Mon Mar 2 16:18:32 2009 From: dsdale24 at gmail.com (Darren Dale) Date: Mon, 2 Mar 2009 16:18:32 -0500 Subject: [IPython-dev] Looking towards 0.10... In-Reply-To: References: Message-ID: On Tue, Feb 10, 2009 at 8:32 PM, Fernando Perez wrote: > > > there's been a bunch of recent work done on various branches, and we > should probably look into putting an 0.10 out in a few weeks, once we > can merge everything in. [...] > Pending review, these two look easy: > > - > https://code.launchpad.net/~villemvainio/ipython/ip-sets-warn/+merge/2456 Any idea when this particular branch will be merged into the trunk? Thanks, Darren -------------- next part -------------- An HTML attachment was scrubbed... URL: From allen.fowler at yahoo.com Tue Mar 3 10:44:49 2009 From: allen.fowler at yahoo.com (Allen Fowler) Date: Tue, 3 Mar 2009 07:44:49 -0800 (PST) Subject: [IPython-dev] quick way to reset & run a program In-Reply-To: References: <105574.85628.qm@web45604.mail.sp1.yahoo.com> Message-ID: <589910.86719.qm@web45607.mail.sp1.yahoo.com> > > Basically I'm looking for a quicker way to execute: "%reset, confirm [yes], > > %run". ( With optionally clearing the cached output history. ) > > Not out of the box, but writing a little magic to do this is and > installing it in your home dir is pretty easy. Unfortunatley you > can't quite just make your magic do 'reset -y, run' because %reset > asks unconditionally. But %reset is small enough (see magic_reset in > Magic.py) that you can simply copy its contents into your own personal > %rrun and then have that call %run. Let us know if you have any > problems with this and need further help. > Is there a "user/profile specific" Magic.py file to edit, or do I have to edit the system-wide version? From allen.fowler at yahoo.com Tue Mar 3 10:47:27 2009 From: allen.fowler at yahoo.com (Allen Fowler) Date: Tue, 3 Mar 2009 07:47:27 -0800 (PST) Subject: [IPython-dev] quick way to reset & run a program In-Reply-To: <589910.86719.qm@web45607.mail.sp1.yahoo.com> References: <105574.85628.qm@web45604.mail.sp1.yahoo.com> <589910.86719.qm@web45607.mail.sp1.yahoo.com> Message-ID: <510884.43201.qm@web45606.mail.sp1.yahoo.com> > > > Basically I'm looking for a quicker way to execute: "%reset, confirm [yes], > > > %run". ( With optionally clearing the cached output history. ) > > > > Not out of the box, but writing a little magic to do this is and > > installing it in your home dir is pretty easy. Unfortunatley you > > can't quite just make your magic do 'reset -y, run' because %reset > > asks unconditionally. But %reset is small enough (see magic_reset in > > Magic.py) that you can simply copy its contents into your own personal > > %rrun and then have that call %run. Let us know if you have any > > problems with this and need further help. > > > > > Is there a "user/profile specific" Magic.py file to edit, or do I have to edit > the system-wide version? > Let me rephrase that. :) How do I make a user-specific Magic function definition? From fperez.net at gmail.com Wed Mar 4 08:36:11 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 4 Mar 2009 05:36:11 -0800 Subject: [IPython-dev] Looking towards 0.10... In-Reply-To: References: Message-ID: On Mon, Mar 2, 2009 at 1:18 PM, Darren Dale wrote: > Any idea when this particular branch will be merged into the trunk? If I can get my talk for this conference finished quickly, over the next few days I'll try to catch up on reviews. There's also a lingering bug on Jorgen's branch that I see but he doesn't, that I'd like a clean solution on before merge. I'll post the slides for the talk when finished, we're having a little python-fest at a SIAM meeting with Brian, Ondrej, John Hunter and others: http://meetings.siam.org/sess/dsp_programsess.cfm?SESSIONCODE=8044 http://meetings.siam.org/sess/dsp_programsess.cfm?SESSIONCODE=8045 http://meetings.siam.org/sess/dsp_programsess.cfm?SESSIONCODE=8046 Cheers, f From fperez.net at gmail.com Wed Mar 4 08:39:17 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 4 Mar 2009 05:39:17 -0800 Subject: [IPython-dev] quick way to reset & run a progra In-Reply-To: <510884.43201.qm@web45606.mail.sp1.yahoo.com> References: <105574.85628.qm@web45604.mail.sp1.yahoo.com> <589910.86719.qm@web45607.mail.sp1.yahoo.com> <510884.43201.qm@web45606.mail.sp1.yahoo.com> Message-ID: On Tue, Mar 3, 2009 at 7:47 AM, Allen Fowler wrote: > Let me rephrase that. :) > > > How do I make a user-specific Magic function definition? Fair question :) http://ipython.scipy.org/doc/rel-0.9.1/html/interactive/extension_api.html HTH, f From hans_meine at gmx.net Thu Mar 5 07:18:08 2009 From: hans_meine at gmx.net (Hans Meine) Date: Thu, 5 Mar 2009 13:18:08 +0100 Subject: [IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook In-Reply-To: <20090208205354.GU14469@phare.normalesup.org> References: <6ce0ac130902070800s16e7365bp9bde1277f06b973f@mail.gmail.com> <6ce0ac130902081215i2304c6e4y92a06f7eba7cffbe@mail.gmail.com> <20090208205354.GU14469@phare.normalesup.org> Message-ID: <200903051318.09710.hans_meine@gmx.net> On Sunday 08 February 2009 21:53:54 Gael Varoquaux wrote: > I have given a lot of thought to this, and I believe that to avoid > freezing, there is no other option than multiple processes. I just wanted to chime in again and mention that this is also the approach we (the VIGRA developers, i.e. mainly Ullrich K?the and me) took with our PyQt- based interactive python terminal ('pyterm', not officially released, although some versions should be floating around the WWW). I am not very fond of its actual code and state, but the basic design of separate front- and backend has paid out AFAICS. Ulli had actually designed a very simple socket protocol between the two processes, with commands for communicating e.g. I/O and completions. I think that that should better be factored out, i.e. one should have proxy objects that behave like their counterparts in the opposite processes, such that it becomes possible (or at least less work) to directly wire a frontend and backend in the same process, too. (In the past, I needed this for easy embedding of the terminal widget in a Qt program of mine, with direct access to the embedding program's state.) One big advantage of the separation was *proper* Ctrl-C handling. The main disadvantage was that API was needed to access the frontend's UI from the backend; Ulli introduced this to be able to add menu entries from within the backend. HTH, Hans From rocky at gnu.org Tue Mar 10 11:18:50 2009 From: rocky at gnu.org (Rocky Bernstein) Date: Tue, 10 Mar 2009 11:18:50 -0400 Subject: [IPython-dev] ipython as a debugger command-line interface? Message-ID: <6cd6de210903100818j6a1ea2c8n14e93e5455245fb5@mail.gmail.com> Recently, I've been rewriting the stock Python debugger (or more precisely the pydb fork of that) from the ground up using some ideas and experience gained in working ruby-debug . One comment sometimes heard with respect to debuggers is "Gee, I don't need a debugger because I write modular code and so I just go into a Python or the cooler IPython shell and enter from that what I need." I find this remark a little odd (the not needing a debugger part, not the coolness of the IPython part), because I don't see a good debugger as that much different from one of these shells. True, the automatic string evaluation in pdb and pydb of commands that aren't seen as debugger commands is a little lame compared to IPython, but to that end in the various debuggers I've tried to make more and more like a Python shell. (I write "Python" here rather than "IPython" because Python's an easier target to hit :-). So you can write things like: (Pydbgr) x = 5 (Pydbgr) list x+1 # Admittedly this is contrived But perhaps I've not been going far enough. Although in pydb and pydbgr there are commands to go into Python or IPython, but maybe more should be done here. In ruby-debug right now there's a settingto *automatically* enter irb initially; the same could be done in pydbgrto enter python or ipython. So maybe what should be done is to add add debugger commands "step", "list", "continue" and so on into these invocations of these shells. Thoughts on whether this would be a desirable? Or how easy or difficult would it be to do? Finally, I should note that the rewrite allows for multiple command interfaces. So adding a python or ipython shell interface is not exclusive of say a gdb-style or DBGp interfaces. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jorgen.stenarson at bostream.nu Tue Mar 10 13:45:16 2009 From: jorgen.stenarson at bostream.nu (=?ISO-8859-1?Q?J=F6rgen_Stenarson?=) Date: Tue, 10 Mar 2009 18:45:16 +0100 Subject: [IPython-dev] Looking towards 0.10... In-Reply-To: References: Message-ID: <49B6A72C.8050500@bostream.nu> Fernando Perez skrev: > Hi all, > > there's been a bunch of recent work done on various branches, and we > should probably look into putting an 0.10 out in a few weeks, once we > can merge everything in. I have my own branch to clean up before I > post it up for review, but even if things aren't perfect, I think we > should make a new version available. > > Right off the top of my head I know we have: > > - Laurent's recent branch, Gael finished the review and it's ready to merge. > - Brian's recent win32 fixes, I reviewed it and with a few fixes it can go. > - Jorgen's had some lingering test failures, but once those get > resolved it can go, he's addressed all other concerns. The only failure left is one I can't reproduce myself because it works on my ubuntu partition. I also think bug https://bugs.launchpad.net/ipython/+bug/315392 is fairly serious because it crashes ipython. I'm not sure of the best way to fix it though. Is it a pyreadline bug to return unicode strings here or is this something ipython should deal with. /J?rgen From vivainio at gmail.com Tue Mar 10 13:49:25 2009 From: vivainio at gmail.com (Ville M. Vainio) Date: Tue, 10 Mar 2009 19:49:25 +0200 Subject: [IPython-dev] ipython as a debugger command-line interface? In-Reply-To: <6cd6de210903100818j6a1ea2c8n14e93e5455245fb5@mail.gmail.com> References: <6cd6de210903100818j6a1ea2c8n14e93e5455245fb5@mail.gmail.com> Message-ID: <46cb515a0903101049i1ae2ec85r72663617b20a6f4@mail.gmail.com> On Tue, Mar 10, 2009 at 5:18 PM, Rocky Bernstein wrote: > done here. In ruby-debug right now there's a setting to automatically enter > irb initially; the same could be done in pydbgr to enter python or ipython. > So maybe what should be done is to add add debugger commands "step", "list", > "continue" and so on into these invocations of these shells. > > Thoughts on whether this would be a desirable? Or how easy or difficult > would it be to do? One interesting avenue would be making ipython a frontend for rpdb2 - one such frontend is winpdb, and I don't see why other couldn't be ipython. -- Ville M. Vainio http://tinyurl.com/vainio From fperez.net at gmail.com Tue Mar 10 13:56:03 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 10 Mar 2009 10:56:03 -0700 Subject: [IPython-dev] Looking towards 0.10... In-Reply-To: <49B6A72C.8050500@bostream.nu> References: <49B6A72C.8050500@bostream.nu> Message-ID: On Tue, Mar 10, 2009 at 10:45 AM, J?rgen Stenarson wrote: > The only failure left is one I can't reproduce myself because it works on my > ubuntu partition. I just finished a bunch of things leftover from a conference trip, and I'm working on it right now. I'll report back soon... Cheers, f From fperez.net at gmail.com Tue Mar 10 15:06:04 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 10 Mar 2009 12:06:04 -0700 Subject: [IPython-dev] Looking towards 0.10... In-Reply-To: References: <49B6A72C.8050500@bostream.nu> Message-ID: Hi Jorgen, On Tue, Mar 10, 2009 at 10:56 AM, Fernando Perez wrote: > On Tue, Mar 10, 2009 at 10:45 AM, J?rgen Stenarson > wrote: > >> The only failure left is one I can't reproduce myself because it works on my >> ubuntu partition. > > I just finished a bunch of things leftover from a conference trip, and > I'm working on it right now. ?I'll report back soon... OK, I understand better what's going on. The test should indeed fail, depending on where you run it from. The setup function in the test suite is def setup(): """Setup testenvironment for the module: - Adds dummy home dir tree """ try: os.makedirs("home_test_dir/_ipython") except WindowsError: pass #Or should we complain that the test directory already exists?? but the failing test sets homedir below the current path. So depending on where you are positioned in the path when you run the test suite, this can indeed fail. I'm fixing up the whole test suite now, because there are some other problems, I'll post soon. But you shold be able to verify the failure by running iptest -vvs from some other location in the filesystem than the ipython test directory... Cheers, f From rocky at gnu.org Tue Mar 10 15:23:08 2009 From: rocky at gnu.org (Rocky Bernstein) Date: Tue, 10 Mar 2009 15:23:08 -0400 Subject: [IPython-dev] ipython as a debugger command-line interface? In-Reply-To: <46cb515a0903101049i1ae2ec85r72663617b20a6f4@mail.gmail.com> References: <6cd6de210903100818j6a1ea2c8n14e93e5455245fb5@mail.gmail.com> <46cb515a0903101049i1ae2ec85r72663617b20a6f4@mail.gmail.com> Message-ID: <6cd6de210903101223r3c182026jac966234fd87006e@mail.gmail.com> This is slightly different that what I was suggesting, but I subscribe to the Perl notion that "There's more than one way to do it". If one wants to use ipython as a front end (and I don't see a reason why not), then the way I'd approach this is to make ipython understand some sort of common debugging protocol. Otherwise you have M x N combinations of M front-ends and N back ends, versus M+N. (But I realize since this list may only care about about ipython, it is N vs. 1.) Gdb uses the protocol called GDB/MI, but outside of gdb I don't think any other debugger has adopted it. ActiveState uses DBGp, and not just for their Python IDE and debugger but also I believe for Tcl, Ruby and Python. And I'm given to believe that there are other Python debuggers that use DBGp as well. So right now that's what I'm contemplating in pydbgr . On Tue, Mar 10, 2009 at 1:49 PM, Ville M. Vainio wrote: > On Tue, Mar 10, 2009 at 5:18 PM, Rocky Bernstein wrote: > > > done here. In ruby-debug right now there's a setting to automatically > enter > > irb initially; the same could be done in pydbgr to enter python or > ipython. > > So maybe what should be done is to add add debugger commands "step", > "list", > > "continue" and so on into these invocations of these shells. > > > > Thoughts on whether this would be a desirable? Or how easy or difficult > > would it be to do? > > One interesting avenue would be making ipython a frontend for rpdb2 - > one such frontend is winpdb, and I don't see why other couldn't be > ipython. > > -- > Ville M. Vainio > http://tinyurl.com/vainio > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Tue Mar 10 15:32:27 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 10 Mar 2009 12:32:27 -0700 Subject: [IPython-dev] Looking towards 0.10... In-Reply-To: References: <49B6A72C.8050500@bostream.nu> Message-ID: On Tue, Mar 10, 2009 at 12:06 PM, Fernando Perez wrote: >>> The only failure left is one I can't reproduce myself because it works on my >>> ubuntu partition. OK, code uploaded with comments on the branch review system. I think we're close on this one... Moving on :) f From vivainio at gmail.com Tue Mar 10 16:26:52 2009 From: vivainio at gmail.com (Ville M. Vainio) Date: Tue, 10 Mar 2009 22:26:52 +0200 Subject: [IPython-dev] ipython as a debugger command-line interface? In-Reply-To: <6cd6de210903101223r3c182026jac966234fd87006e@mail.gmail.com> References: <6cd6de210903100818j6a1ea2c8n14e93e5455245fb5@mail.gmail.com> <46cb515a0903101049i1ae2ec85r72663617b20a6f4@mail.gmail.com> <6cd6de210903101223r3c182026jac966234fd87006e@mail.gmail.com> Message-ID: <46cb515a0903101326q2b3a40cje7ec7908a5d68537@mail.gmail.com> On Tue, Mar 10, 2009 at 9:23 PM, Rocky Bernstein wrote: > Gdb uses the protocol called GDB/MI, but outside of gdb I don't think any > other debugger has adopted it. ActiveState uses DBGp, and not just for their > Python IDE and debugger but also I believe for Tcl, Ruby and Python. And I'm > given to believe that there are other Python debuggers that use DBGp as > well. So right now that's what I'm contemplating in pydbgr. So is pydbgr meant to sit on server side of DBGp, client side, or both? DBGp does look like an exhaustive protocol; and admittedly more documented than rpdb2 protocol. Can you reuse an existing DBGp server module? -- Ville M. Vainio http://tinyurl.com/vainio From rocky at gnu.org Tue Mar 10 17:28:08 2009 From: rocky at gnu.org (Rocky Bernstein) Date: Tue, 10 Mar 2009 17:28:08 -0400 Subject: [IPython-dev] ipython as a debugger command-line interface? In-Reply-To: <46cb515a0903101326q2b3a40cje7ec7908a5d68537@mail.gmail.com> References: <6cd6de210903100818j6a1ea2c8n14e93e5455245fb5@mail.gmail.com> <46cb515a0903101049i1ae2ec85r72663617b20a6f4@mail.gmail.com> <6cd6de210903101223r3c182026jac966234fd87006e@mail.gmail.com> <46cb515a0903101326q2b3a40cje7ec7908a5d68537@mail.gmail.com> Message-ID: <6cd6de210903101428l213f5699wfe1035ebee7eb4d0@mail.gmail.com> On Tue, Mar 10, 2009 at 4:26 PM, Ville M. Vainio wrote: > On Tue, Mar 10, 2009 at 9:23 PM, Rocky Bernstein wrote: > > > Gdb uses the protocol called GDB/MI, but outside of gdb I don't think any > > other debugger has adopted it. ActiveState uses DBGp, and not just for > their > > Python IDE and debugger but also I believe for Tcl, Ruby and Python. And > I'm > > given to believe that there are other Python debuggers that use DBGp as > > well. So right now that's what I'm contemplating in pydbgr. > > So is pydbgr meant to sit on server side of DBGp, client side, or both? Both. One could use the pydbgr client side as a "back-end" like dddor gnu emacs do for pydb . But most of the emphasis is and will on the debugging support side, followed by protocol support. > > DBGp does look like an exhaustive protocol; and admittedly more > documented than rpdb2 protocol. Can you reuse an existing DBGp server > module? Shane Caraveo in private email wrote (I hope this is okay to cite): > > While our UI code is proprietary and only part of Komodo IDE, several of > our dbgp implementations are open source, and available from > http://svn.openkomodo.com/openkomodo/browse/ > It would be nice to have skeletal DBGp client/server implementations for testing. As best as I can tell though this doesn't exist yet or I am not aware of it. But we've strayed from the original thought which was adding a set of domain-specific (or debugger-specific) commands to ipython to use that as a debugger command-line + ipython. Here's an additional thought along that vein. The way pydbgr gets its commands for the command-line command processor is to import all files in a directory, introspect and look for classes that end in "Command" and instantiate each of those. (Now that I write this, better might be to check that they are instances of some Command superclass) . Given this, it's possible to add a feature where users can write their own customized debugging commands and put them in a directory and when you start the debugger point to the directory and have the debugger add those additional debugger commands. I throw this idea out as an alternative to ipython way to add "magic" commands. I'm not that familiar with ipython magic commands, but probably that's the way one would add debugger-specific commands in ipython. > > > -- > Ville M. Vainio > http://tinyurl.com/vainio > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Wed Mar 11 03:08:39 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 11 Mar 2009 00:08:39 -0700 Subject: [IPython-dev] Looking towards 0.10... In-Reply-To: References: <49B6A72C.8050500@bostream.nu> Message-ID: Hi all, I realize that I haven't finished reviewing branches yet, but I'd really appreciate some eyes on https://code.launchpad.net/~fdo.perez/ipython/trunk-dev There's a LOT of code in there, mostly work to update our testing and doc machinery. While not all of it is as clean as I'd like it to be, I think it's important that we get it in (feel free to pitch in with improvements), because I've focused on things that should help the rest of the code get better. We can now run more tests than before, I've updated doc notices about sphinx and added a bunch of sphinx-specific tools, and now we can get full PDF API docs: the full manual is now a 453 page PDF that includes the entire API. Because all of this infrastructure isn't really functionality but rather support for the rest of our code to be better tested and documented, and even for us to more easily know what we have in the first place, I'd really appreciate some eyes on it. This is much of the type of work that I should have done years ago, but oh well... I'll try to continue reviewing branches as soon as I can, too. I have Ville's and Robert's in mind right now, am I missing something? Cheers, f From fperez.net at gmail.com Wed Mar 11 03:12:03 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 11 Mar 2009 00:12:03 -0700 Subject: [IPython-dev] ipython as a debugger command-line interface? In-Reply-To: <6cd6de210903101428l213f5699wfe1035ebee7eb4d0@mail.gmail.com> References: <6cd6de210903100818j6a1ea2c8n14e93e5455245fb5@mail.gmail.com> <46cb515a0903101049i1ae2ec85r72663617b20a6f4@mail.gmail.com> <6cd6de210903101223r3c182026jac966234fd87006e@mail.gmail.com> <46cb515a0903101326q2b3a40cje7ec7908a5d68537@mail.gmail.com> <6cd6de210903101428l213f5699wfe1035ebee7eb4d0@mail.gmail.com> Message-ID: On Tue, Mar 10, 2009 at 2:28 PM, Rocky Bernstein wrote: > I throw this idea out as an alternative to ipython way to add "magic" > commands. I'm not that familiar with ipython magic commands, but probably > that's the way one would add debugger-specific commands in ipython. Very much so. In fact, I prefer how IPython works than how debuggers typically do it: ipython can have automatic calling of commands like ls but if you have a conflict, it has a way to disambiguate with %ls The advantage of that is that you always get the full python language as a first class citizen, without having to jump through the contortions you need to in pdb, where the commands and the language are always stepping on each other's toes. So it would be very easy to simply have a group of magics that load together and implement all the debugger commands in an ipython-based one, but you'd retain 100% of the python language without any name clashes (like 'normal' ipython does). Best, f From vivainio at gmail.com Wed Mar 11 03:35:48 2009 From: vivainio at gmail.com (Ville M. Vainio) Date: Wed, 11 Mar 2009 09:35:48 +0200 Subject: [IPython-dev] ipython as a debugger command-line interface? In-Reply-To: <6cd6de210903101428l213f5699wfe1035ebee7eb4d0@mail.gmail.com> References: <6cd6de210903100818j6a1ea2c8n14e93e5455245fb5@mail.gmail.com> <46cb515a0903101049i1ae2ec85r72663617b20a6f4@mail.gmail.com> <6cd6de210903101223r3c182026jac966234fd87006e@mail.gmail.com> <46cb515a0903101326q2b3a40cje7ec7908a5d68537@mail.gmail.com> <6cd6de210903101428l213f5699wfe1035ebee7eb4d0@mail.gmail.com> Message-ID: <46cb515a0903110035n3036e437hbed97ebbf66b0fac@mail.gmail.com> On Tue, Mar 10, 2009 at 11:28 PM, Rocky Bernstein wrote: > Here's an additional thought along that vein. The way pydbgr gets its > commands for the command-line command processor is to import all files in a > directory, introspect and look for classes that end in "Command" and > instantiate each of those. (Now that I write this, better might be to check > that they are instances of some Command superclass) .? Given this, it's > possible to add a feature where users can write their own customized > debugging commands and put them in a directory and when you start the > debugger point to the directory and have the debugger add those additional > debugger commands. > > I throw this idea out as an alternative to ipython way to add "magic" > commands. I'm not that familiar with ipython magic commands, but probably > that's the way one would add debugger-specific commands in ipython. You can add magics programmatically through _ip.expose_magic("mymagic", callable). So these schemes are not mutually exclusive. -- Ville M. Vainio http://tinyurl.com/vainio From rocky at gnu.org Wed Mar 11 05:35:05 2009 From: rocky at gnu.org (Rocky Bernstein) Date: Wed, 11 Mar 2009 05:35:05 -0400 Subject: [IPython-dev] ipython as a debugger command-line interface? In-Reply-To: <46cb515a0903110035n3036e437hbed97ebbf66b0fac@mail.gmail.com> References: <6cd6de210903100818j6a1ea2c8n14e93e5455245fb5@mail.gmail.com> <46cb515a0903101049i1ae2ec85r72663617b20a6f4@mail.gmail.com> <6cd6de210903101223r3c182026jac966234fd87006e@mail.gmail.com> <46cb515a0903101326q2b3a40cje7ec7908a5d68537@mail.gmail.com> <6cd6de210903101428l213f5699wfe1035ebee7eb4d0@mail.gmail.com> <46cb515a0903110035n3036e437hbed97ebbf66b0fac@mail.gmail.com> Message-ID: <6cd6de210903110235q13c9da46m1a1e3eda88708497@mail.gmail.com> Ok - thanks for the information. I'll use that when I get around to it. On Wed, Mar 11, 2009 at 3:35 AM, Ville M. Vainio wrote: > On Tue, Mar 10, 2009 at 11:28 PM, Rocky Bernstein wrote: > > > Here's an additional thought along that vein. The way pydbgr gets its > > commands for the command-line command processor is to import all files in > a > > directory, introspect and look for classes that end in "Command" and > > instantiate each of those. (Now that I write this, better might be to > check > > that they are instances of some Command superclass) . Given this, it's > > possible to add a feature where users can write their own customized > > debugging commands and put them in a directory and when you start the > > debugger point to the directory and have the debugger add those > additional > > debugger commands. > > > > I throw this idea out as an alternative to ipython way to add "magic" > > commands. I'm not that familiar with ipython magic commands, but probably > > that's the way one would add debugger-specific commands in ipython. > > You can add magics programmatically through > _ip.expose_magic("mymagic", callable). So these schemes are not > mutually exclusive. > > -- > Ville M. Vainio > http://tinyurl.com/vainio > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Mar 11 14:57:06 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 11 Mar 2009 13:57:06 -0500 Subject: [IPython-dev] Looking towards 0.10... In-Reply-To: References: <49B6A72C.8050500@bostream.nu> Message-ID: On 2009-03-11 02:08, Fernando Perez wrote: > I'll try to continue reviewing branches as soon as I can, too. I have > Ville's and Robert's in mind right now, am I missing something? I have two, the pretty-printing extension and the %paste magic. I don't know if you were thinking of just one of them or both. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From fperez.net at gmail.com Wed Mar 11 15:01:14 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 11 Mar 2009 12:01:14 -0700 Subject: [IPython-dev] Looking towards 0.10... In-Reply-To: References: <49B6A72C.8050500@bostream.nu> Message-ID: On Wed, Mar 11, 2009 at 11:57 AM, Robert Kern wrote: > On 2009-03-11 02:08, Fernando Perez wrote: > >> I'll try to continue reviewing branches as soon as I can, too. ?I have >> Ville's and Robert's in mind right now, am I missing something? > > I have two, the pretty-printing extension and the %paste magic. I don't know if > you were thinking of just one of them or both. Both. I'll try to go over them tomorrow, as well as Ville's. Hopefully that big of a multi-branch merge won't get too messy :) Cheers, f From fperez.net at gmail.com Sat Mar 14 20:27:00 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 14 Mar 2009 17:27:00 -0700 Subject: [IPython-dev] Extensions: decisions on loading them and API (thoughts from R. Kern's branch) Message-ID: Hi all, I was reviewing Robert's branch https://code.launchpad.net/~robert-kern/ipython/pretty which reminded me of revisiting our Extensions setup. That space has, over time, gotten somewhat disorganized: maqroll[Extensions]> ls *py astyle.py ipy_foo.py ipy_signals.py clearcmd.py ipy_fsops.py ipy_stock_completers.py envpersist.py ipy_gnuglobal.py ipy_synchronize_with.py ext_rescapture.py ipy_greedycompleter.py ipy_system_conf.py ibrowse.py ipy_jot.py ipy_traits_completer.py igrid.py ipy_kitcfg.py ipy_vimserver.py __init__.py ipy_legacy.py ipy_which.py InterpreterExec.py ipy_lookfor.py ipy_winpdb.py InterpreterPasteInput.py ipy_p4.py ipy_workdir.py ipipe.py ipy_profile_doctest.py jobctrl.py ipy_app_completers.py ipy_profile_none.py ledit.py ipy_autoreload.py ipy_profile_numpy.py numeric_formats.py ipy_bzr.py ipy_profile_scipy.py PhysicalQInput.py ipy_completers.py ipy_profile_sh.py PhysicalQInteractive.py ipy_constants.py ipy_profile_zope.py pickleshare.py* ipy_defaults.py ipy_pydb.py pspersistence.py ipy_editors.py ipy_rehashdir.py scitedirector.py ipy_exportdb.py ipy_render.py win32clip.py ipy_extutil.py ipy_server.py maqroll[Extensions]> The two key things we need to do, probably right after the 0.10 release: 1. Stop putting Extensions into sys.path. This is just bad practice, as it pollutes people's import path with our stuff. Users who want an extension can just write from IPython.Extensions import foo instead of import foo (or even ipy_foo) it's not that much more work, and much, much cleaner. 2. Have all extensions require an explicit entry point to be activated. Something like from IPython.Extensions import foo foo.activate() # this could take parameters, if the extension file provides more than one thing. I'm open to suggestions on this point, API ideas, etc. But I have no doubt that the current practice of extensions automatically registering themselves with top-level code is a bad idea. It means that their mere import has side effects, which goes counter to most standard python practice and can cause problems. For example, things like sphinx that import all modules can be badly messed up if each extension starts having global side effects. Ideally, extensions could also be unloaded, so perhaps the api should read instead foo.load() foo.unload() for those that support it. This would make the job of testing them much, much easier. Thoughts on the API, etc? If we can agree on this, then at least Robert's current one could already be fixed for this mode of use before it sees public life. From fperez.net at gmail.com Sat Mar 14 22:06:23 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 14 Mar 2009 19:06:23 -0700 Subject: [IPython-dev] Looking towards 0.10... In-Reply-To: References: <49B6A72C.8050500@bostream.nu> Message-ID: On Wed, Mar 11, 2009 at 12:01 PM, Fernando Perez wrote: > On Wed, Mar 11, 2009 at 11:57 AM, Robert Kern wrote: >> On 2009-03-11 02:08, Fernando Perez wrote: >> >>> I'll try to continue reviewing branches as soon as I can, too. ?I have >>> Ville's and Robert's in mind right now, am I missing something? >> >> I have two, the pretty-printing extension and the %paste magic. I don't know if >> you were thinking of just one of them or both. > > Both. ?I'll try to go over them tomorrow, as well as Ville's. > Hopefully that big of a multi-branch merge won't get too messy :) OK, I have to head out now, but things are looking better: https://code.launchpad.net/ipython/+activereviews Only three unreviewed branches. As I mentioned, in mine there's a massive amount of code, including much that improves the testing and documentation building machinery so that others can do better testing of their own changes. So I'd be very grateful for another pair of eyes on that before merging. I need to run now, but tomorrow I'll try to look at what's left there. A review from Gael on Laurent's other one (I already merged one) would be great, since that's a branch off yours. We're closing in for 0.10. cheers, f From fperez.net at gmail.com Sat Mar 14 22:08:19 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 14 Mar 2009 19:08:19 -0700 Subject: [IPython-dev] Looking towards 0.10... In-Reply-To: References: <49B6A72C.8050500@bostream.nu> Message-ID: On Sat, Mar 14, 2009 at 7:06 PM, Fernando Perez wrote: > We're closing in for 0.10. BTW, since there's been some merging, those of you with branches for review, please *remember to sync with trunk again* so we know the patches apply cleanly (I just had to resolve a few small conflicts myself). That will make the review process easier. Cheers, f From robert.kern at gmail.com Sat Mar 14 22:30:15 2009 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 14 Mar 2009 21:30:15 -0500 Subject: [IPython-dev] Looking towards 0.10... In-Reply-To: References: <49B6A72C.8050500@bostream.nu> Message-ID: <3d375d730903141930w355b3feaif6cdf018bc58b886@mail.gmail.com> On Sat, Mar 14, 2009 at 21:08, Fernando Perez wrote: > On Sat, Mar 14, 2009 at 7:06 PM, Fernando Perez wrote: > >> We're closing in for 0.10. > > BTW, since there's been some merging, those of you with branches for > review, please *remember to sync with trunk again* so we know the > patches apply cleanly (I just had to resolve a few small conflicts > myself). lp:ipython or your trunk-dev? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From fperez.net at gmail.com Sun Mar 15 02:15:04 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 14 Mar 2009 23:15:04 -0700 Subject: [IPython-dev] Looking towards 0.10... In-Reply-To: <3d375d730903141930w355b3feaif6cdf018bc58b886@mail.gmail.com> References: <3d375d730903141930w355b3feaif6cdf018bc58b886@mail.gmail.com> Message-ID: On Sat, Mar 14, 2009 at 7:30 PM, Robert Kern wrote: > lp:ipython or your trunk-dev? My trunk-dev, sorry I didn't see this before leaving... f From vivainio at gmail.com Sun Mar 15 04:57:52 2009 From: vivainio at gmail.com (Ville M. Vainio) Date: Sun, 15 Mar 2009 10:57:52 +0200 Subject: [IPython-dev] Extensions: decisions on loading them and API (thoughts from R. Kern's branch) In-Reply-To: References: Message-ID: <46cb515a0903150157n2ceeac75wc4824eb752b644e2@mail.gmail.com> On Sun, Mar 15, 2009 at 2:27 AM, Fernando Perez wrote: > The two key things we need to do, probably right after the 0.10 release: > > 1. Stop putting Extensions into sys.path. ?This is just bad practice, > as it pollutes people's import path with our stuff. ?Users who want an -1. This just breaks stuff without any real benefits (provided that there are no collisions - something taken care of by ipy_ prefix). > 2. Have all extensions require an explicit entry point to be > activated. ?Something like To recap, this is already done (as can be seen by looking at "_ip.load??"). -- Ville M. Vainio http://tinyurl.com/vainio From gael.varoquaux at normalesup.org Sun Mar 15 05:24:30 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 15 Mar 2009 10:24:30 +0100 Subject: [IPython-dev] Looking towards 0.10... In-Reply-To: References: <49B6A72C.8050500@bostream.nu> Message-ID: <20090315092430.GA24801@phare.normalesup.org> On Sat, Mar 14, 2009 at 07:06:23PM -0700, Fernando Perez wrote: > We're closing in for 0.10. Could I have a week (say a week and a half ;->). I would like to merge back in IPython some modifications we did to the frontend in the ETS code base. Problem is, now that my paper deadline is past, I am spending the week end on a Mayavi release :). Ga?l From gael.varoquaux at normalesup.org Sun Mar 15 05:27:14 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 15 Mar 2009 10:27:14 +0100 Subject: [IPython-dev] Extensions: decisions on loading them and API (thoughts from R. Kern's branch) In-Reply-To: <46cb515a0903150157n2ceeac75wc4824eb752b644e2@mail.gmail.com> References: <46cb515a0903150157n2ceeac75wc4824eb752b644e2@mail.gmail.com> Message-ID: <20090315092714.GB24801@phare.normalesup.org> On Sun, Mar 15, 2009 at 10:57:52AM +0200, Ville M. Vainio wrote: > On Sun, Mar 15, 2009 at 2:27 AM, Fernando Perez wrote: > > The two key things we need to do, probably right after the 0.10 release: > > 1. Stop putting Extensions into sys.path. ?This is just bad practice, > > as it pollutes people's import path with our stuff. ?Users who want an > -1. This just breaks stuff without any real benefits (provided that > there are no collisions - something taken care of by ipy_ prefix). Monkey patching sys.path is ugly and fragile. it makes code impossible to run without going throught the one given entry point, and most often break Python import priority. I am +1 on the proposition. Ga?l From fperez.net at gmail.com Sun Mar 15 16:51:48 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Sun, 15 Mar 2009 13:51:48 -0700 Subject: [IPython-dev] Extensions: decisions on loading them and API (thoughts from R. Kern's branch) In-Reply-To: <20090315092714.GB24801@phare.normalesup.org> References: <46cb515a0903150157n2ceeac75wc4824eb752b644e2@mail.gmail.com> <20090315092714.GB24801@phare.normalesup.org> Message-ID: On Sun, Mar 15, 2009 at 2:27 AM, Gael Varoquaux wrote: > On Sun, Mar 15, 2009 at 10:57:52AM +0200, Ville M. Vainio wrote: >> On Sun, Mar 15, 2009 at 2:27 AM, Fernando Perez wrote: > >> > The two key things we need to do, probably right after the 0.10 release: > >> > 1. Stop putting Extensions into sys.path. ?This is just bad practice, >> > as it pollutes people's import path with our stuff. ?Users who want an > >> -1. This just breaks stuff without any real benefits (provided that >> there are no collisions - something taken care of by ipy_ prefix). > > Monkey patching sys.path is ugly and fragile. it makes code impossible to > run without going throught the one given entry point, and most often > break Python import priority. I am +1 on the proposition. Yup. Unfortunately the mistake we made was to allow the decision in the first place, because yes, now this will be an API breakage we'll have to announce loudly. But the whole ipy_* naming convention is ugly (and there are a bunch of things in there with non-ipy_ names already), and my point is precisely Gael's: applications should *never* mess with sys.path under the user. Setuptools does that: arguably they can't avoid it to support namespace packages, but it still causes no end of grief under certain circumstances. In our case, the only justification is to save users a few characters when loading ipython extensions so they can do it implicitly. That's breaking the 'explicit is better than implicit' mantra for no justifiable reason. I realize you (Ville) like this, but I'm afraid it was a mistake that we must correct now rather than later. There's a good chance this summer we may get some funding to support Brian doing some serious refactoring of the core, and now is the time to get all our APIs in order. Honestly, I allowed some of these changes that I felt were ill-advised when I thought that the ipython1 work would simply replace ip0, and that we would get all new APIs. I simply didn't pay close enough attention to what was going into ip0, thinking it would be an end-of-the road code, which turned out not to be true. It was my mistake, 100%, I admit that. But I'd rather face up to my mistake now, fix things and endure a minor API breakage, than live with these problematic decisions for the long haul now that we are all using that codebase for the core and that all the code developed in ip1 is building on top of it as its foundation. BTW, I'm not proposing this to change for 0.10, no. It's for the next release, and we may actually put in a deprecation warning Cheers, f From fperez.net at gmail.com Sun Mar 15 16:56:50 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Sun, 15 Mar 2009 13:56:50 -0700 Subject: [IPython-dev] Looking towards 0.10... In-Reply-To: <20090315092430.GA24801@phare.normalesup.org> References: <49B6A72C.8050500@bostream.nu> <20090315092430.GA24801@phare.normalesup.org> Message-ID: On Sun, Mar 15, 2009 at 2:24 AM, Gael Varoquaux wrote: > On Sat, Mar 14, 2009 at 07:06:23PM -0700, Fernando Perez wrote: >> We're closing in for 0.10. > > Could I have a week (say a week and a half ;->). I would like to merge > back in IPython some modifications we did to the frontend in the ETS code > base. Problem is, now that my paper deadline is past, I am spending the > week end on a Mayavi release :). I imagine that reviewing all my changes will take some time, so go for it (as quick as you can). My branch is big and nobody has looked at it, so I'm sure it will take a few iterations on it. There's also Jorgen's that's almost ready to merge, and Laurent's that needs review. So I'm sure we'll need a few days to finish the review/merge work we have on the table. Here's the table, by the way, so we can all pick a fruit and start munching on it: https://code.launchpad.net/ipython/+activereviews Cheers f From gael.varoquaux at normalesup.org Sun Mar 15 17:02:25 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 15 Mar 2009 22:02:25 +0100 Subject: [IPython-dev] Extensions: decisions on loading them and API (thoughts from R. Kern's branch) In-Reply-To: References: <46cb515a0903150157n2ceeac75wc4824eb752b644e2@mail.gmail.com> <20090315092714.GB24801@phare.normalesup.org> Message-ID: <20090315210225.GA8854@phare.normalesup.org> On Sun, Mar 15, 2009 at 01:51:48PM -0700, Fernando Perez wrote: > There's a good chance this summer we may get some funding to support > Brian doing some serious refactoring of the core, and now is the time > to get all our APIs in order. Freeking awesome ! The core really needs a rewrite, but as it is a lot of work, everybody has been avoiding it, so having Brian leading the effort with some funding would be really great. Ga?l From fperez.net at gmail.com Mon Mar 16 00:42:46 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Sun, 15 Mar 2009 21:42:46 -0700 Subject: [IPython-dev] Has IPython been useful to you? Please let me know... Message-ID: Hi all, [ apologies for the semi-spam, I'll keep this brief and expect all replies off-list ] IPython is a project that many of you on this list are likely to use in your daily work, either directly or indirectly (if you've embedded it or used it as a component of some other system). I would simply like to ask you, if IPython has been significantly useful for a project you use, lead, develop, etc., to let me know. For legal/professional reasons, I need to gather information about who has found IPython to be of value. I started IPython as a toy 'afternoon hack' in late 2001, and today it continues to grow, as the nicely summarized Ohloh stats show: https://www.ohloh.net/p/ipython (obviously, this is now the result of the work of many, not just myself, as is true of any healthy open source project as it grows). But I have never systematically tracked its impact, and now I need to do so. So, if you have used IPython and it has made a significant contribution to your project, work, research, company, whatever, I'd be very grateful if you let me know. A short paragraph on what this benefit has been is all I ask. Once I gather any information I get, I would contact directly some of the responders to ask for your authorization before quoting you. I should stress that any information you give me will only go in a documentation packet in support of my legal/residency process here in the USA (think of it as an oversized, obnoxiously detailed CV that goes beyond just publications and regular academic information). To keep traffic off this list, please send your replies directly to me, either at this address or my regular work one: Fernando.Perez at berkeley.edu In advance, many thanks to anyone willing to reply. I've never asked for anything in return for working on IPython and the ecosystem of scientific Python tools, but this is actually very important, so any information you can provide me will be very useful. Best regards, Fernando Perez. From stefan at sun.ac.za Mon Mar 16 02:46:42 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 16 Mar 2009 08:46:42 +0200 Subject: [IPython-dev] Fwd: Pyglet python/ipython interactive terminal In-Reply-To: <6fb1833c-d9cd-4f7f-9ae9-395a1ca2eb28@w34g2000yqm.googlegroups.com> References: <6fb1833c-d9cd-4f7f-9ae9-395a1ca2eb28@w34g2000yqm.googlegroups.com> Message-ID: <9457e7c80903152346h62cd672bubf11d1b52e651e58@mail.gmail.com> Another way to embed IPython! ---------- Forwarded message ---------- From: Nicolas Rougier Date: 2009/3/15 Subject: Pyglet python/ipython interactive terminal To: pyglet-users Hi all, I've coded a terminal that implements a python (or ipython) shell within a pyglet window and handles python stdin/stderr/stdout redirection and system wide stdout/stderr redirection (using a pipe), provides history based on the GNU readline package and automatic completion. All ansi color codes are interpreted correctly using a dedicated pyglet Document subclass. Screenshot and code are available here: http://www.loria.fr/~rougier/pyglet-terminal.html Nicolas --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "pyglet-users" group. To post to this group, send email to pyglet-users at googlegroups.com To unsubscribe from this group, send email to pyglet-users+unsubscribe at googlegroups.com For more options, visit this group at http://groups.google.com/group/pyglet-users?hl=en -~----------~----~----~----~------~----~------~--~--- From fperez.net at gmail.com Mon Mar 16 03:07:01 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 16 Mar 2009 00:07:01 -0700 Subject: [IPython-dev] Fwd: Pyglet python/ipython interactive terminal In-Reply-To: <9457e7c80903152346h62cd672bubf11d1b52e651e58@mail.gmail.com> References: <6fb1833c-d9cd-4f7f-9ae9-395a1ca2eb28@w34g2000yqm.googlegroups.com> <9457e7c80903152346h62cd672bubf11d1b52e651e58@mail.gmail.com> Message-ID: 2009/3/15 St?fan van der Walt : > Another way to embed IPython! > > ---------- Forwarded message ---------- > From: Nicolas Rougier > Date: 2009/3/15 > Subject: Pyglet python/ipython interactive terminal > To: pyglet-users Cool! Duly added: http://ipython.scipy.org/moin/About/Projects_Using_IPython Thanks for the link! f From jorgen.stenarson at bostream.nu Mon Mar 16 14:16:12 2009 From: jorgen.stenarson at bostream.nu (=?ISO-8859-1?Q?J=F6rgen_Stenarson?=) Date: Mon, 16 Mar 2009 19:16:12 +0100 Subject: [IPython-dev] Looking towards 0.10... In-Reply-To: References: <49B6A72C.8050500@bostream.nu> <20090315092430.GA24801@phare.normalesup.org> Message-ID: <49BE976C.1060909@bostream.nu> Fernando Perez skrev: > On Sun, Mar 15, 2009 at 2:24 AM, Gael Varoquaux > wrote: >> On Sat, Mar 14, 2009 at 07:06:23PM -0700, Fernando Perez wrote: >>> We're closing in for 0.10. >> Could I have a week (say a week and a half ;->). I would like to merge >> back in IPython some modifications we did to the frontend in the ETS code >> base. Problem is, now that my paper deadline is past, I am spending the >> week end on a Mayavi release :). > > I imagine that reviewing all my changes will take some time, so go for > it (as quick as you can). My branch is big and nobody has looked at > it, so I'm sure it will take a few iterations on it. > > There's also Jorgen's that's almost ready to merge, and Laurent's that > needs review. So I'm sure we'll need a few days to finish the > review/merge work we have on the table. Here's the table, by the way, > so we can all pick a fruit and start munching on it: I have just merged from trunk and removed the debug code that was left over from your fix of the tests. Hopefully it's good to go now. When doing this I tried to run iptest one level above IPython/ and got a cryptic crash that claims it is not possible to import Release. I guess this is because iptest is not meant to be run from there but a better error message would be good to have. /J?rgen C:\python\external\ipython\IPython>cd .. C:\python\external\ipython>dir Volymen i enhet C har ingen etikett. Volymens serienummer ?r 100A-559F Inneh?ll i katalogen C:\python\external\ipython 2009-03-16 18:55 . 2009-03-16 18:55 .. 2008-09-22 18:25 10 .checkeol 2008-09-22 18:25 161 .hgignore 2008-09-22 18:25 3 469 COPYING.txt 2008-09-22 18:25 debian 2008-09-22 18:25 docs 2008-09-22 18:25 627 ipykit.py 2009-03-16 19:01 IPython 2008-09-22 18:25 420 ipython.py 2008-10-20 18:29 224 magic_run_test.py 2009-03-16 18:55 522 MANIFEST.in 2008-09-22 18:25 281 README.txt 2008-10-20 18:38 21 runscript.py 2009-03-16 18:55 sandbox 2008-09-22 18:25 scripts 2008-09-22 18:25 6 833 setup.py 2008-09-22 18:25 9 633 setupbase.py 2008-09-24 22:49 6 771 setupbase.pyc 2008-09-22 18:25 341 setupegg.py 2008-09-22 18:25 3 277 setupexe.py 2008-09-24 22:49 setupext 2008-09-22 18:25 468 setup_bdist_egg.py 2008-09-24 18:22 test 2008-09-22 18:25 tools 15 fil(er) 33 058 byte 10 katalog(er) 356 193 054 720 byte ledigt C:\python\external\ipython>python scripts/iptest E ====================================================================== ERROR: Failure: ImportError (cannot import name Release) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python25\lib\site-packages\nose-0.11.0.dev_r0-py2.5.egg\nose\loader.py", line 364, in loadTestsFromName addr.filename, addr.module) File "C:\Python25\lib\site-packages\nose-0.11.0.dev_r0-py2.5.egg\nose\importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "C:\Python25\lib\site-packages\nose-0.11.0.dev_r0-py2.5.egg\nose\importer.py", line 82, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "c:\python\external\ipython\IPython\__init__.py", line 63, in from IPython import Release # do it explicitly so pydoc can see it - pydoc bug ImportError: cannot import name Release ---------------------------------------------------------------------- Ran 1 test in 0.000s FAILED (errors=1) C:\python\external\ipython>cd IPython C:\python\external\ipython\IPython>python ../scripts/iptest SS...........SSSSSSSSSSS.S...S................ ---------------------------------------------------------------------- Ran 46 tests in 0.703s OK (SKIP=15) C:\python\external\ipython\IPython> From fperez.net at gmail.com Mon Mar 16 14:55:37 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 16 Mar 2009 11:55:37 -0700 Subject: [IPython-dev] Looking towards 0.10... In-Reply-To: <49BE976C.1060909@bostream.nu> References: <20090315092430.GA24801@phare.normalesup.org> <49BE976C.1060909@bostream.nu> Message-ID: Hi Jorgen, On Mon, Mar 16, 2009 at 11:16 AM, J?rgen Stenarson wrote: > I have just merged from trunk and removed the debug code that was left over > from your fix of the tests. Hopefully it's good to go now. > > When doing this I tried to run iptest one level above IPython/ and got a > cryptic crash that claims it is not possible to import Release. I guess this > is because iptest is not meant to be run from there but a better error > message would be good to have. > C:\python\external\ipython>python scripts/iptest > E > ====================================================================== > ERROR: Failure: ImportError (cannot import name Release) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File > "C:\Python25\lib\site-packages\nose-0.11.0.dev_r0-py2.5.egg\nose\loader.py", > line 364, in loadTestsFromName > ? ?addr.filename, addr.module) > ?File > "C:\Python25\lib\site-packages\nose-0.11.0.dev_r0-py2.5.egg\nose\importer.py", > line 39, in importFromPath > ? ?return self.importFromDir(dir_path, fqname) > ?File > "C:\Python25\lib\site-packages\nose-0.11.0.dev_r0-py2.5.egg\nose\importer.py", > line 82, in importFromDir > ? ?mod = load_module(part_fqname, fh, filename, desc) > ?File "c:\python\external\ipython\IPython\__init__.py", line 63, in > ? ?from IPython import Release # do it explicitly so pydoc can see it - > pydoc bug > ImportError: cannot import name Release > > ---------------------------------------------------------------------- > Ran 1 test in 0.000s > > FAILED (errors=1) > > C:\python\external\ipython>cd IPython > > C:\python\external\ipython\IPython>python ../scripts/iptest > SS...........SSSSSSSSSSS.S...S................ > ---------------------------------------------------------------------- > Ran 46 tests in 0.703s > > OK (SKIP=15) Mmh, I don't see that error here: maqroll[ipython]> python scripts/iptest -vvs Failure: SkipTest (This test requires zope.interface, Twisted, Foolscap and PyObjC) ... SKIP: This test requires zope.interface, Twisted, Foolscap and PyObjC test_blockID_added_to_failure (IPython.frontend.tests.test_asyncfrontendbase.TestAsyncFrontendBase) ... ok [...] Testcase to see if we can call get_security_dir without Exceptions. ... ok test_popkey_1, Basic usage test of popkey ... ok test_popkey_2, Test to see that popkey of non occuring keys ... ok test_popkey_3, Tests to see that popkey calls returns the correct value ... ok IPython.tests.test_magic.test_rehashx ... ok ---------------------------------------------------------------------- Ran 431 tests in 42.090s OK (SKIP=9) So I'm going to go ahead and merge your branch in a minute. Later, once all the new testing machinery is in place, we can more easily shake out if there are platform- or installation-specific issues with being able to load the whole test suite. Thanks for the work and being so accomodating with the review process! Cheers, f From barrywark at gmail.com Mon Mar 16 17:01:36 2009 From: barrywark at gmail.com (Barry Wark) Date: Mon, 16 Mar 2009 14:01:36 -0700 Subject: [IPython-dev] Extensions: decisions on loading them and API (thoughts from R. Kern's branch) In-Reply-To: References: Message-ID: On Sat, Mar 14, 2009 at 5:27 PM, Fernando Perez wrote: > Hi all, > > I was reviewing Robert's branch > > https://code.launchpad.net/~robert-kern/ipython/pretty > > which reminded me of revisiting our Extensions setup. That space has, > over time, gotten somewhat disorganized: > > maqroll[Extensions]> ls *py > astyle.py ? ? ? ? ? ? ? ? ipy_foo.py ? ? ? ? ? ? ?ipy_signals.py > clearcmd.py ? ? ? ? ? ? ? ipy_fsops.py ? ? ? ? ? ?ipy_stock_completers.py > envpersist.py ? ? ? ? ? ? ipy_gnuglobal.py ? ? ? ?ipy_synchronize_with.py > ext_rescapture.py ? ? ? ? ipy_greedycompleter.py ?ipy_system_conf.py > ibrowse.py ? ? ? ? ? ? ? ?ipy_jot.py ? ? ? ? ? ? ?ipy_traits_completer.py > igrid.py ? ? ? ? ? ? ? ? ?ipy_kitcfg.py ? ? ? ? ? ipy_vimserver.py > __init__.py ? ? ? ? ? ? ? ipy_legacy.py ? ? ? ? ? ipy_which.py > InterpreterExec.py ? ? ? ?ipy_lookfor.py ? ? ? ? ?ipy_winpdb.py > InterpreterPasteInput.py ?ipy_p4.py ? ? ? ? ? ? ? ipy_workdir.py > ipipe.py ? ? ? ? ? ? ? ? ?ipy_profile_doctest.py ?jobctrl.py > ipy_app_completers.py ? ? ipy_profile_none.py ? ? ledit.py > ipy_autoreload.py ? ? ? ? ipy_profile_numpy.py ? ?numeric_formats.py > ipy_bzr.py ? ? ? ? ? ? ? ?ipy_profile_scipy.py ? ?PhysicalQInput.py > ipy_completers.py ? ? ? ? ipy_profile_sh.py ? ? ? PhysicalQInteractive.py > ipy_constants.py ? ? ? ? ?ipy_profile_zope.py ? ? pickleshare.py* > ipy_defaults.py ? ? ? ? ? ipy_pydb.py ? ? ? ? ? ? pspersistence.py > ipy_editors.py ? ? ? ? ? ?ipy_rehashdir.py ? ? ? ?scitedirector.py > ipy_exportdb.py ? ? ? ? ? ipy_render.py ? ? ? ? ? win32clip.py > ipy_extutil.py ? ? ? ? ? ?ipy_server.py > maqroll[Extensions]> > > The two key things we need to do, probably right after the 0.10 release: > > 1. Stop putting Extensions into sys.path. ?This is just bad practice, > as it pollutes people's import path with our stuff. ?Users who want an > extension can just write > > from IPython.Extensions import foo > > instead of > > import foo (or even ipy_foo) > > it's not that much more work, and much, much cleaner. > > 2. Have all extensions require an explicit entry point to be > activated. ?Something like > > from IPython.Extensions import foo > foo.activate() ?# this could take parameters, if the extension file > provides more than one thing. > > I'm open to suggestions on this point, API ideas, etc. ?But I have no > doubt that the current practice of extensions automatically > registering themselves with top-level code is a bad idea. ?It means > that their mere import has side effects, which goes counter to most > standard python practice and can cause problems. ?For example, things > like sphinx that import all modules can be badly messed up if each > extension starts having global side effects. > > Ideally, extensions could also be unloaded, so perhaps the api should > read instead > > foo.load() > foo.unload() > > for those that support it. ?This would make the job of testing them > much, much easier. > > Thoughts on the API, etc? ?If we can agree on this, then at least > Robert's current one could already be fixed for this mode of use > before it sees public life. +1 on making extension loading and importing more explicit. I think load()/unload() are good, as it would also allow more dynamic loading/unloading of extensions depending on context (ala TextMate's bundle contexts) in the future. We could ease the transition, perhaps, by adding a startup flag that, if set, will monkey-patch sys.path as is done now. Again, forcing this behavior to be explicitly requested is a good idea. More broadly, extensions might be viewed as a plugin system. Is there any will towards incorporating one of the many python plugin systems instead of rolling our own? I don't know the core very well, so there could easily be good reasons against such an approach. -Barry > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > From vivainio at gmail.com Mon Mar 16 17:33:43 2009 From: vivainio at gmail.com (Ville M. Vainio) Date: Mon, 16 Mar 2009 23:33:43 +0200 Subject: [IPython-dev] Extensions: decisions on loading them and API (thoughts from R. Kern's branch) In-Reply-To: References: Message-ID: <46cb515a0903161433n1017de2dn513a852866f0ceac@mail.gmail.com> On Mon, Mar 16, 2009 at 11:01 PM, Barry Wark wrote: > +1 on making extension loading and importing more explicit. I think > bundle contexts) in the future. We could ease the transition, perhaps, > by adding a startup flag that, if set, will monkey-patch sys.path as > is done now. Again, forcing this behavior to be explicitly requested > is a good idea. sys.path manipulation is just a convenience mechanism to make all the extensions appear in the same namespace (global modules, starting with ipy_). So, if you have plugins installed by 5 different packages, you could see all of them by entering "import ipy_", without digging through package source trees for them. However, I'm done with defending the scheme. If it has to go down, it has to go down, and I can personally add the feature back by having it in my ipy_user_conf.py.. > More broadly, extensions might be viewed as a plugin system. Is there > any will towards incorporating one of the many python plugin systems > instead of rolling our own? I don't know the core very well, so there > could easily be good reasons against such an approach. The existing plugin systems seem a bit overengineered for ipythons purpose. What we need are addition of unload and unset_hook. I don't know how much people really wont to unregisters plugin, but it might come in handy down the road. -- Ville M. Vainio http://tinyurl.com/vainio From fperez.net at gmail.com Mon Mar 16 19:10:08 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 16 Mar 2009 16:10:08 -0700 Subject: [IPython-dev] Looking towards 0.10... In-Reply-To: References: <20090315092430.GA24801@phare.normalesup.org> <49BE976C.1060909@bostream.nu> Message-ID: On Mon, Mar 16, 2009 at 11:55 AM, Fernando Perez wrote: > So I'm going to go ahead and merge your branch in a minute. ?Later, > once all the new testing machinery is in place, we can more easily > shake out if there are platform- or installation-specific issues with > being able to load the whole test suite. Sorry for the delay, I got sidetacked. I just merged your branch and pushed upstream to the public trunk, so your branch appears now as merged. Thanks for all the work! Could you please go over any tickets that were related to this branch and mark them as 'fix committed'? Cheers, f From fperez.net at gmail.com Mon Mar 16 20:25:22 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 16 Mar 2009 17:25:22 -0700 Subject: [IPython-dev] Overzealous introspection In-Reply-To: References: <46cb515a0901230631n6d641880xb849fba14091b107@mail.gmail.com> Message-ID: Hi all, sorry I hadn't noticed this old thread... On Tue, Jan 27, 2009 at 4:54 AM, Shashwath T.R. wrote: > Happens without declaring the function too: > > In [2]: x=""" > ? ...: Eh? > Object `Eh` not found. > ? ...: """ > > And now, the string is actually empty... > > In [3]: x > Out[3]: '\n\n' > > In [4]: print x > > > > In [5]: > > Definitely a bug, I think? Yup: https://bugs.launchpad.net/ipython/+bug/343992 At least now we'll track it... Sorry again for the long delay. Post 0.10, I'd like to do some serious bug squashing, esp. now that we have an easy way to test interactive ipython input and we can ensure that once fixed, it stays fixed :) Cheers, f From ellisonbg.net at gmail.com Mon Mar 16 21:47:09 2009 From: ellisonbg.net at gmail.com (Brian Granger) Date: Mon, 16 Mar 2009 18:47:09 -0700 Subject: [IPython-dev] Extensions: decisions on loading them and API (thoughts from R. Kern's branch) In-Reply-To: <6ce0ac130903161652v6647f235u56d039e90990f169@mail.gmail.com> References: <46cb515a0903150157n2ceeac75wc4824eb752b644e2@mail.gmail.com> <20090315092714.GB24801@phare.normalesup.org> <6ce0ac130903161652v6647f235u56d039e90990f169@mail.gmail.com> Message-ID: <6ce0ac130903161847k58c9663esbad5432940a91da5@mail.gmail.com> >?There's a good chance this > summer we may get some funding to support Brian doing some serious > refactoring of the core, and now is the time to get all our APIs in > order. > >?But I'd rather face up to my mistake > now, fix things and endure a minor API breakage, than live with these > problematic decisions for the long haul now that we are all using that > codebase for the core and that all the code developed in ip1 is > building on top of it as its foundation. As Fernando has eluded to, it looks like I may have some funding to work on a refactor of the ipython core this summer. ?I am very excited to finally get a chance to wrestle the beast into submission. If I can make a broad generalization, currently, the fantastic part of IPython is its features, not its developer API. ?If everything goes as planned this summer my goal will be to 1) maintain or improve IPython's feature set while 2) fixing the underlying API and design. Another way of saying this is that the user's experience will be the same or better, but a developer's experience will be *completely different*. ?This will unavoidably mean API breakage. ?Where appropriate deprecation warnings will be used, but in many cases, the API will change so significantly that even deprecations warnings won't be possible. ?I will also try to do things in place and incrementally. I will consult with the list extensively about design decisions and hopefully as things move forward, others will be interested in helping out. ?After 0.10 is released, I will post to the list some more details about my near future plans. Cheers, Brian From ondrej at certik.cz Mon Mar 16 23:18:52 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 16 Mar 2009 20:18:52 -0700 Subject: [IPython-dev] fixing warnings in python2.6 Message-ID: <85b5c3130903162018r72c451c6j2113f110de0a7cda@mail.gmail.com> Hi, the attached patch fixes the annoying warning in python2.6: $ ipython /var/lib/python-support/python2.6/IPython/Magic.py:38: DeprecationWarning: the sets module is deprecated from sets import Set Python 2.6.1+ (r261:67515, Mar 2 2009, 13:11:28) Type "copyright", "credits" or "license" for more information. IPython 0.9.1 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object'. ?object also works, ?? prints more. In [1]: I created the patch with: $ bzr send -o python2.6.patch Using saved parent location "http://bazaar.launchpad.net/%7Eipython-dev/ipython/trunk/" to determine what changes to submit. Bundling 1 revision(s). is that the correct way? How can I execute tests? I tried to find such an info at: http://ipython.scipy.org/moin/Developer_Zone but haven't find it. If I try nosetests I get: $ nosetests Traceback (most recent call last): File "/usr/bin/nosetests", line 8, in load_entry_point('nose==0.10.3', 'console_scripts', 'nosetests')() File "/var/lib/python-support/python2.6/nose/core.py", line 219, in __init__ argv=argv, testRunner=testRunner, testLoader=testLoader) File "/usr/lib/python2.6/unittest.py", line 817, in __init__ self.runTests() File "/var/lib/python-support/python2.6/nose/core.py", line 298, in runTests result = self.testRunner.run(self.test) File "/var/lib/python-support/python2.6/nose/core.py", line 62, in run test(result) File "/var/lib/python-support/python2.6/nose/suite.py", line 132, in __call__ return self.run(*arg, **kw) File "/var/lib/python-support/python2.6/nose/suite.py", line 162, in run test(orig) File "/usr/lib/python2.6/unittest.py", line 464, in __call__ return self.run(*args, **kwds) File "/usr/lib/python2.6/unittest.py", line 457, in run for test in self._tests: File "/var/lib/python-support/python2.6/nose/suite.py", line 76, in _get_tests for test in self.test_generator: File "/var/lib/python-support/python2.6/nose/loader.py", line 165, in loadTestsFromDir entry_path, discovered=True) File "/var/lib/python-support/python2.6/nose/loader.py", line 377, in loadTestsFromName module, discovered=discovered) File "/var/lib/python-support/python2.6/nose/loader.py", line 301, in loadTestsFromModule tests.extend(self.loadTestsFromDir(path)) File "/var/lib/python-support/python2.6/nose/loader.py", line 165, in loadTestsFromDir entry_path, discovered=True) File "/var/lib/python-support/python2.6/nose/loader.py", line 377, in loadTestsFromName module, discovered=discovered) File "/var/lib/python-support/python2.6/nose/loader.py", line 306, in loadTestsFromModule return self.suiteClass(ContextList(tests, context=module)) File "/var/lib/python-support/python2.6/nose/suite.py", line 332, in __call__ return self.makeSuite(tests, context) File "/var/lib/python-support/python2.6/nose/suite.py", line 377, in makeSuite self.context.setdefault(suite, []).append(context) TypeError: unhashable type: 'ContextSuite' If I try py.test I get: $ py.test [... lots of errors ] E from IPython import Release > ImportError: cannot import name Release [/home/ondrej/repos/ipython/IPython/ipstruct.py:13] ________________________________________________________________________________ ============= tests finished: 3 passed, 31 failed in 0.14 seconds ============== so I think that's not the way. :) Anyway, so I didn't test my patch yet --- but I will once you point me to the right direction. Thanks, Ondrej -------------- next part -------------- A non-text attachment was scrubbed... Name: python2.6.patch Type: text/x-diff Size: 2525 bytes Desc: not available URL: From fperez.net at gmail.com Mon Mar 16 23:21:04 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 16 Mar 2009 20:21:04 -0700 Subject: [IPython-dev] fixing warnings in python2.6 In-Reply-To: <85b5c3130903162018r72c451c6j2113f110de0a7cda@mail.gmail.com> References: <85b5c3130903162018r72c451c6j2113f110de0a7cda@mail.gmail.com> Message-ID: 2009/3/16 Ondrej Certik : > Hi, > > the attached patch fixes the annoying warning in python2.6: > > $ ipython > /var/lib/python-support/python2.6/IPython/Magic.py:38: > DeprecationWarning: the sets module is deprecated > ?from sets import Set thanks, man, but it's already fixed in my branch: https://code.launchpad.net/~fdo.perez/ipython/trunk-dev/+merge/4450 I'm addressing Brian's review, and should be able to merge by tomorrow, I think. Sorry :) f From ondrej at certik.cz Mon Mar 16 23:25:39 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 16 Mar 2009 20:25:39 -0700 Subject: [IPython-dev] fixing warnings in python2.6 In-Reply-To: References: <85b5c3130903162018r72c451c6j2113f110de0a7cda@mail.gmail.com> Message-ID: <85b5c3130903162025g7271d8duf5afc9c3191dc37c@mail.gmail.com> On Mon, Mar 16, 2009 at 8:21 PM, Fernando Perez wrote: > 2009/3/16 Ondrej Certik : >> Hi, >> >> the attached patch fixes the annoying warning in python2.6: >> >> $ ipython >> /var/lib/python-support/python2.6/IPython/Magic.py:38: >> DeprecationWarning: the sets module is deprecated >> ?from sets import Set > > thanks, man, but it's already fixed in my branch: > > https://code.launchpad.net/~fdo.perez/ipython/trunk-dev/+merge/4450 > > I'm addressing Brian's review, and should be able to merge by tomorrow, I think. Ok, I think I can wait till tomorrow. I was only following the main branch. How do I run the tests though if I wanted to fix something else? Ondrej From fperez.net at gmail.com Mon Mar 16 23:27:08 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 16 Mar 2009 20:27:08 -0700 Subject: [IPython-dev] fixing warnings in python2.6 In-Reply-To: <85b5c3130903162018r72c451c6j2113f110de0a7cda@mail.gmail.com> References: <85b5c3130903162018r72c451c6j2113f110de0a7cda@mail.gmail.com> Message-ID: 2009/3/16 Ondrej Certik : > I created the patch with: > > $ bzr send -o python2.6.patch > Using saved parent location > "http://bazaar.launchpad.net/%7Eipython-dev/ipython/trunk/" to > determine what changes to submit. > Bundling 1 revision(s) > > > is that the correct way? That works, you can also email a plain old diff-based patch from 'bzr diff', or make your own branch on launchpad, if you expect it to be an ongoing development with feedback. > How can I execute tests? I tried to find such an info at: > > http://ipython.scipy.org/moin/Developer_Zone Sorry, we should have a link to the dev part of the manual up there. These days we're trying to keep as much as we can in the sphinx doc tree (the one you started! :) http://ipython.scipy.org/doc/manual/html/development/development.html#testing-system Right now on my branch (note that for me, iptest is an alias to 'iptest -vvs'): maqroll[tmp]> iptest *** Pasting of code with ">>>" or "..." has been enabled. c referrers: [] x is: 1 Doctest: IPython.Extensions.InterpreterPasteInput ... ok Doctest: IPython.Extensions.astyle.Style.__init__ ... ok [ lots of output] ---------------------------------------------------------------------- Ran 528 tests in 42.611s OK (SKIP=11) Cheers, f From fperez.net at gmail.com Mon Mar 16 23:28:56 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 16 Mar 2009 20:28:56 -0700 Subject: [IPython-dev] fixing warnings in python2.6 In-Reply-To: References: <85b5c3130903162018r72c451c6j2113f110de0a7cda@mail.gmail.com> Message-ID: On Mon, Mar 16, 2009 at 8:27 PM, Fernando Perez wrote: > http://ipython.scipy.org/doc/manual/html/development/development.html#testing-system > > Right now on my branch (note that for me, iptest is an alias to 'iptest -vvs'): > > maqroll[tmp]> iptest I should add that iptest is in ipython/scripts. I simply symlink it from ~/usr/bin, so that I don't have to mess with installs and can develop nicely in place, but you'll get it installed if you do a real (or setuptools, or virtualenv) installation. Cheers, f From ondrej at certik.cz Mon Mar 16 23:33:28 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 16 Mar 2009 20:33:28 -0700 Subject: [IPython-dev] fixing warnings in python2.6 In-Reply-To: References: <85b5c3130903162018r72c451c6j2113f110de0a7cda@mail.gmail.com> Message-ID: <85b5c3130903162033k52b4c97bq246bb11cbaf14032@mail.gmail.com> On Mon, Mar 16, 2009 at 8:27 PM, Fernando Perez wrote: > 2009/3/16 Ondrej Certik : > >> I created the patch with: >> >> $ bzr send -o python2.6.patch >> Using saved parent location >> "http://bazaar.launchpad.net/%7Eipython-dev/ipython/trunk/" to >> determine what changes to submit. >> Bundling 1 revision(s) >> >> >> is that the correct way? > > That works, you can also email a plain old diff-based patch from 'bzr > diff', or make your own branch on launchpad, if you expect it to be an > ongoing development with feedback. > >> How can I execute tests? I tried to find such an info at: >> >> http://ipython.scipy.org/moin/Developer_Zone > > Sorry, we should have a link to the dev part of the manual up there. > These days we're trying to keep as much as we can in the sphinx doc > tree (the one you started! :) But that was a year ago. :) > http://ipython.scipy.org/doc/manual/html/development/development.html#testing-system > > Right now on my branch (note that for me, iptest is an alias to 'iptest -vvs'): > > maqroll[tmp]> iptest > *** Pasting of code with ">>>" or "..." has been enabled. > c referrers: [] > x is: 1 > Doctest: IPython.Extensions.InterpreterPasteInput ... ok > Doctest: IPython.Extensions.astyle.Style.__init__ ... ok > > [ lots of output] > > ---------------------------------------------------------------------- > Ran 528 tests in 42.611s > > OK (SKIP=11) I am getting: $ python2.5 scripts/iptest S...S.....................................................................................................................................................................................................................................................................................................................................................................................................................S..S..SS..SSS........ ---------------------------------------------------------------------- Ran 431 tests in 23.585s OK (SKIP=9) Which is ok, but this is a disaster: $ scripts/iptest Traceback (most recent call last): File "scripts/iptest", line 8, in iptest.main() File "/home/ondrej/repos/ipython/IPython/testing/iptest.py", line 53, in main TestProgram(argv=argv,plugins=plugins) File "/var/lib/python-support/python2.6/nose/core.py", line 219, in __init__ argv=argv, testRunner=testRunner, testLoader=testLoader) File "/usr/lib/python2.6/unittest.py", line 816, in __init__ self.parseArgs(argv) File "/var/lib/python-support/python2.6/nose/core.py", line 270, in parseArgs self.createTests() File "/var/lib/python-support/python2.6/nose/core.py", line 284, in createTests self.test = self.testLoader.loadTestsFromNames(self.testNames) File "/var/lib/python-support/python2.6/nose/loader.py", line 422, in loadTestsFromNames return unittest.TestLoader.loadTestsFromNames(self, names, module) File "/usr/lib/python2.6/unittest.py", line 613, in loadTestsFromNames suites = [self.loadTestsFromName(name, module) for name in names] File "/var/lib/python-support/python2.6/nose/loader.py", line 377, in loadTestsFromName module, discovered=discovered) File "/var/lib/python-support/python2.6/nose/loader.py", line 301, in loadTestsFromModule tests.extend(self.loadTestsFromDir(path)) File "/var/lib/python-support/python2.6/nose/loader.py", line 165, in loadTestsFromDir entry_path, discovered=True) File "/var/lib/python-support/python2.6/nose/loader.py", line 377, in loadTestsFromName module, discovered=discovered) File "/var/lib/python-support/python2.6/nose/loader.py", line 306, in loadTestsFromModule return self.suiteClass(ContextList(tests, context=module)) File "/var/lib/python-support/python2.6/nose/suite.py", line 332, in __call__ return self.makeSuite(tests, context) File "/var/lib/python-support/python2.6/nose/suite.py", line 377, in makeSuite self.context.setdefault(suite, []).append(context) TypeError: unhashable type: 'ContextSuite' So I guess I should first install nosetests that work with python2.6. Ondrej From ondrej at certik.cz Mon Mar 16 23:34:59 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 16 Mar 2009 20:34:59 -0700 Subject: [IPython-dev] fixing warnings in python2.6 In-Reply-To: References: <85b5c3130903162018r72c451c6j2113f110de0a7cda@mail.gmail.com> Message-ID: <85b5c3130903162034r72533d3fq5028e2392c1915fa@mail.gmail.com> On Mon, Mar 16, 2009 at 8:28 PM, Fernando Perez wrote: > On Mon, Mar 16, 2009 at 8:27 PM, Fernando Perez wrote: > >> http://ipython.scipy.org/doc/manual/html/development/development.html#testing-system >> >> Right now on my branch (note that for me, iptest is an alias to 'iptest -vvs'): >> >> maqroll[tmp]> iptest > > I should add that iptest is in ipython/scripts. ?I simply symlink it > from ~/usr/bin, so that I don't have to mess with installs and can > develop nicely in place, but you'll get it installed if you do a real > (or setuptools, or virtualenv) installation. Or just $ scripts/iptest as I did in my previous email? Ondrej From fperez.net at gmail.com Mon Mar 16 23:37:42 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 16 Mar 2009 20:37:42 -0700 Subject: [IPython-dev] fixing warnings in python2.6 In-Reply-To: <85b5c3130903162033k52b4c97bq246bb11cbaf14032@mail.gmail.com> References: <85b5c3130903162018r72c451c6j2113f110de0a7cda@mail.gmail.com> <85b5c3130903162033k52b4c97bq246bb11cbaf14032@mail.gmail.com> Message-ID: On Mon, Mar 16, 2009 at 8:33 PM, Ondrej Certik wrote: > I am getting: > > $ python2.5 scripts/iptest > S...S.....................................................................................................................................................................................................................................................................................................................................................................................................................S..S..SS..SSS........ > ---------------------------------------------------------------------- > Ran 431 tests in 23.585s > > OK (SKIP=9) That's about right, I've written tons of tests and made the testing machinery more robust, so it picks up more doctests as well. But that number sounds right for the current trunk. > Which is ok, but this is a disaster: > > $ scripts/iptest [...] > So I guess I should first install nosetests that work with python2.6. Ouch. You're in uncharted waters, man: I haven't tried to set up a full 2.6 environment yet, to actually run the test suite there. I've lightly played with ipython in 2.6 and the sets thing is gone, but I don't have a working nose (and twisted & friends) set up for 2.6 yet. Let us know how it goes! f From ondrej at certik.cz Tue Mar 17 00:00:46 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 16 Mar 2009 21:00:46 -0700 Subject: [IPython-dev] fixing warnings in python2.6 In-Reply-To: References: <85b5c3130903162018r72c451c6j2113f110de0a7cda@mail.gmail.com> <85b5c3130903162033k52b4c97bq246bb11cbaf14032@mail.gmail.com> Message-ID: <85b5c3130903162100l5736ff4et21f0a9c9ad49f29c@mail.gmail.com> > That's about right, I've written tons of tests and made the testing > machinery more robust, so it picks up more doctests as well. ?But that > number sounds right for the current trunk. With sympy we thought it'd be nice to have some kind of hash or something, so that one is sure that it is picking up all the tests that it should. We were burned by a bug in py.test that it didn't find some tests and noone discovered it for couple months, so you can imagine how I pleased I was to fix the code that got broken because it was not tested just because the tests were not executed. :) > >> Which is ok, but this is a disaster: >> >> $ scripts/iptest > [...] >> So I guess I should first install nosetests that work with python2.6. > > Ouch. ?You're in uncharted waters, man: I haven't tried to set up a > full 2.6 environment yet, to actually run the test suite there. ?I've > lightly played with ipython in 2.6 and the sets thing is gone, but I > don't have a working nose (and twisted & friends) set up for 2.6 yet. > > Let us know how it goes! I wrote my own test runner for sympy exactly because I wanted something simple, that works everywhere, I am now implementing parallel runs. Does nosetest support it? According to this: http://code.google.com/p/python-nose/issues/detail?id=93 it doesn't. My runner is compatible with py.test and nosetests, if you only use standard function tests. It can also pickup doctests and it works with python2.6. It's available at in sympy/utilities/runtests.py: http://git.sympy.org/?p=sympy.git;a=blob;f=sympy/utilities/runtests.py;h=e0034d945455305463a4572e90a8d04cd688d386;hb=HEAD and it has the nice green [OK] that I always wanted, if you remember. :) I hope to polish it, so that it can be used in any project, by just executing it in the project dir. Ondrej From fperez.net at gmail.com Tue Mar 17 00:16:45 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 16 Mar 2009 21:16:45 -0700 Subject: [IPython-dev] fixing warnings in python2.6 In-Reply-To: <85b5c3130903162100l5736ff4et21f0a9c9ad49f29c@mail.gmail.com> References: <85b5c3130903162018r72c451c6j2113f110de0a7cda@mail.gmail.com> <85b5c3130903162033k52b4c97bq246bb11cbaf14032@mail.gmail.com> <85b5c3130903162100l5736ff4et21f0a9c9ad49f29c@mail.gmail.com> Message-ID: Hey, On Mon, Mar 16, 2009 at 9:00 PM, Ondrej Certik wrote: >> That's about right, I've written tons of tests and made the testing >> machinery more robust, so it picks up more doctests as well. ?But that >> number sounds right for the current trunk. > > With sympy we thought it'd be nice to have some kind of hash or > something, so that one is sure that it is picking up all the tests > that it should. Yes, that would be very good to have indeed. > We were burned by a bug in py.test that it didn't find some tests and > noone discovered it for couple months, so you can imagine how I > pleased I was to fix the code that got broken because it was not > tested just because the tests were not executed. :) > I wrote my own test runner for sympy exactly because I wanted > something simple, that works everywhere, I am now implementing > parallel runs. Does nosetest support it? > > According to this: > > http://code.google.com/p/python-nose/issues/detail?id=93 > > it doesn't. My runner is compatible with py.test and nosetests, if you > only use standard function tests. It can also pickup doctests and it > works with python2.6. It's available at in > sympy/utilities/runtests.py: > > http://git.sympy.org/?p=sympy.git;a=blob;f=sympy/utilities/runtests.py;h=e0034d945455305463a4572e90a8d04cd688d386;hb=HEAD > > and it has the nice green [OK] that I always wanted, if you remember. > :) I hope to polish it, so that it can be used in any project, by just > executing it in the project dir. Nice! I'd be happy to play with it, if it can: - handle doctests in extension modules - handle doctests written with ipython prompts (or be taught how to via a plugin, like I had to do for nose) - easily be told how to run just one file, or even just one function inside one file: maqroll[~]> iptest IPython.tests.test_magic ---------------------------------------------------------------------- Ran 10 tests in 0.348s OK (SKIP=3) Deleting object: second_pass maqroll[~]> iptest IPython.tests.test_magic:test_obj_del ---------------------------------------------------------------------- Ran 1 test in 0.204s OK - How to ignore random output in doctests, either in just one entry or for the whole doctest, but still executing the code - Stop at the first failure - Load pdb interactively on failures, so I can debug things right there. Does it have those things? Nose ain't perfect, but it's pretty darn useful even without green 'ok' printouts ;) Cheers, f From ondrej at certik.cz Tue Mar 17 00:28:13 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 16 Mar 2009 21:28:13 -0700 Subject: [IPython-dev] fixing warnings in python2.6 In-Reply-To: References: <85b5c3130903162018r72c451c6j2113f110de0a7cda@mail.gmail.com> <85b5c3130903162033k52b4c97bq246bb11cbaf14032@mail.gmail.com> <85b5c3130903162100l5736ff4et21f0a9c9ad49f29c@mail.gmail.com> Message-ID: <85b5c3130903162128q3126073bh6f1caaaa3633855b@mail.gmail.com> > Nice! ?I'd be happy to play with it, if it can: > > - handle doctests in extension modules What exactly do you mean here? Like doctests in cython or C modules? > - handle doctests written with ipython prompts (or be taught how to > via a plugin, like I had to do for nose) Good point, I created an issue for it: http://code.google.com/p/sympy/issues/detail?id=1328 > > - easily be told how to run just one file, or even just one function > inside one file: it uses the py.test like syntax, e.g. ./test IPython/tests/test_magic.py ./test IPython/tests/test_magic.py -k test_obj_del > - How to ignore random output in doctests, either in just one entry or > for the whole doctest, but still executing the code Nice idea, added to the issue above. > > - Stop at the first failure Nice idea, added to the issue. > > - Load pdb interactively on failures, so I can debug things right there. Yep, it uses post mortem debugging if --pdb is specified, if this is what you mean. > > > Does it have those things? ?Nose ain't perfect, but it's pretty darn > useful even without green 'ok' printouts ;) [ok] is very important for me. :) Ondrej From fperez.net at gmail.com Tue Mar 17 01:16:06 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 16 Mar 2009 22:16:06 -0700 Subject: [IPython-dev] fixing warnings in python2.6 In-Reply-To: <85b5c3130903162128q3126073bh6f1caaaa3633855b@mail.gmail.com> References: <85b5c3130903162018r72c451c6j2113f110de0a7cda@mail.gmail.com> <85b5c3130903162033k52b4c97bq246bb11cbaf14032@mail.gmail.com> <85b5c3130903162100l5736ff4et21f0a9c9ad49f29c@mail.gmail.com> <85b5c3130903162128q3126073bh6f1caaaa3633855b@mail.gmail.com> Message-ID: On Mon, Mar 16, 2009 at 9:28 PM, Ondrej Certik wrote: >> - handle doctests in extension modules > > What exactly do you mean here? Like doctests in cython or C modules? Yup. Nose didn't, I had to fix that myself in ipdoctest.py. > >> - handle doctests written with ipython prompts (or be taught how to >> via a plugin, like I had to do for nose) > > Good point, I created an issue for it: > > http://code.google.com/p/sympy/issues/detail?id=1328 Great. Feel free to pilfer ipdoctest.py for anything of use. >> - easily be told how to run just one file, or even just one function >> inside one file: > > it uses the py.test like syntax, e.g. > > ./test IPython/tests/test_magic.py > ./test IPython/tests/test_magic.py -k test_obj_del Great. >> - How to ignore random output in doctests, either in just one entry or >> for the whole doctest, but still executing the code > > Nice idea, added to the issue above. Same as above, ipdoctest has (easy) code for that, grab it if you need it. >> Does it have those things? ?Nose ain't perfect, but it's pretty darn >> useful even without green 'ok' printouts ;) > > [ok] is very important for me. :) Oh, I like it too :) Keep us posted on your testing code. I have no special attachment to nose, and if yours does the same things and it's lighter to use, I'll be more than glad to use it! Cheers, f From ondrej at certik.cz Tue Mar 17 04:36:30 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Tue, 17 Mar 2009 01:36:30 -0700 Subject: [IPython-dev] running tests using ipython parallel Message-ID: <85b5c3130903170136i4cae74b4peb54f5cd50ef6f04@mail.gmail.com> Hi, I am playing with ipython parallel and it behaves much more robustly than the multiprocessing module from python2.6. In the multiprocessing module if I get an exception, the process basically gets stuck, ctrl-c doesn't help and I need to kill it using the "kill" command. In ipython it works pretty well. I have couple questions though: * which approach do you think would be the best to implement parallel testing? Basically you have a test suite (nosetest and py.test compatible) and currently in the sequential mode I just call "execfile", get all functions from the file and execute them. In parallel I do something along these lines: from IPython.kernel import client mec = client.MultiEngineClient() mec.reset() print "running %d jobs" % self._jobs ids = mec.get_ids() mec.execute("filename = '%s'" % filename) mec.execute("gl = {'__file__': filename}") mec.execute("execfile(filename, gl)") i = 0 for f in funcs: if i >= len(ids): i = 0 #mec.push({"f": f}) mec.push(dict(f=f)) #print mec.execute("gl['%s']" % f.__name__, targets=[ids[i]]) print mec.execute("f", targets=[ids[i]]) i += 1 funcs list contains all the test functions that I can then execute using f(). Unfortunately this approach gives me an error at the line "mec.push(dict(f=f))": File "/home/ondrej/repos/sympy/sympy/utilities/runtests.py", line 214, in test_file mec.push(dict(f=f)) File "/var/lib/python-support/python2.6/IPython/kernel/multiengineclient.py", line 552, in push targets=targets, block=block) File "/var/lib/python-support/python2.6/IPython/kernel/multiengineclient.py", line 441, in _blockFromThread result = blockingCallFromThread(function, *args, **kwargs) File "/var/lib/python-support/python2.6/IPython/kernel/twistedutil.py", line 69, in blockingCallFromThread return twisted.internet.threads.blockingCallFromThread(reactor, f, *a, **kw) File "/usr/lib/python2.6/dist-packages/twisted/internet/threads.py", line 114, in blockingCallFromThread result.raiseException() File "/usr/lib/python2.6/dist-packages/twisted/python/failure.py", line 326, in raiseException raise self.type, self.value, self.tb TypeError: expected string or Unicode object, NoneType found So I thought, ok, let's not push things in and execute everything at the engines. * what things could be safely pushed to engines? I know it can push some functions, but I didn't manage to get it actually working for the actual test functions (it works nice for simple functions from the tutorial). So the only option that seems to me that it should work is that I first implement a function that executes for "n"th test case from the test suite and this function will not be transfered to engines. The only thing that will be pushed is one string ("function name") and then couple integers to specify all the parameters. Ondrej From vivainio at gmail.com Tue Mar 17 12:05:52 2009 From: vivainio at gmail.com (Ville M. Vainio) Date: Tue, 17 Mar 2009 18:05:52 +0200 Subject: [IPython-dev] Extensions: decisions on loading them and API (thoughts from R. Kern's branch) In-Reply-To: <6ce0ac130903161847k58c9663esbad5432940a91da5@mail.gmail.com> References: <46cb515a0903150157n2ceeac75wc4824eb752b644e2@mail.gmail.com> <20090315092714.GB24801@phare.normalesup.org> <6ce0ac130903161652v6647f235u56d039e90990f169@mail.gmail.com> <6ce0ac130903161847k58c9663esbad5432940a91da5@mail.gmail.com> Message-ID: <46cb515a0903170905x7b40a67evd4f9942ea5ad8da3@mail.gmail.com> On Tue, Mar 17, 2009 at 3:47 AM, Brian Granger wrote: > As Fernando has eluded to, it looks like I may have some funding to > work on a refactor of the ipython core this summer. ?I am very excited > to finally get a chance to wrestle the beast into submission. That's great news! Refactoring it does need some 8hour/day work, a privilege it certainly hasn't ever enjoyed... > Another way of saying this is that the user's experience will be the > same or better, but a developer's experience will be *completely > different*. ?This will unavoidably mean API breakage. ?Where > appropriate deprecation warnings will be used, but in many cases, the > API will change so significantly that even deprecations warnings won't > be possible. ?I will also try to do things in place and incrementally. It's a mixed blessing that the "public" api has been quite shallow. It's bad, because users have needed to go for InteractiveShell directly, but it's also good in that breaking the compatibility can be done without annoying too many ;-). Most of the compatibility with IPApi can be retained quite easily, even with a radically different underlying implementation. Take expose_magic, for example, or custom completers. > I will consult with the list extensively about design decisions and > hopefully as things move forward, others will be interested in helping > out. ?After 0.10 is released, I will post to the list some more > details about my near future plans. Obvious targest that really need some care are completer.py, OutputTrap.py, genutils.py. Something to get started anyway ;-). -- Ville M. Vainio http://tinyurl.com/vainio From jorgen.stenarson at bostream.nu Tue Mar 17 13:16:24 2009 From: jorgen.stenarson at bostream.nu (=?ISO-8859-1?Q?J=F6rgen_Stenarson?=) Date: Tue, 17 Mar 2009 18:16:24 +0100 Subject: [IPython-dev] Looking towards 0.10... In-Reply-To: References: <20090315092430.GA24801@phare.normalesup.org> <49BE976C.1060909@bostream.nu> Message-ID: <49BFDAE8.1060909@bostream.nu> Fernando Perez skrev: > On Mon, Mar 16, 2009 at 11:55 AM, Fernando Perez wrote: > >> So I'm going to go ahead and merge your branch in a minute. Later, >> once all the new testing machinery is in place, we can more easily >> shake out if there are platform- or installation-specific issues with >> being able to load the whole test suite. > > Sorry for the delay, I got sidetacked. > > I just merged your branch and pushed upstream to the public trunk, so > your branch appears now as merged. > > Thanks for all the work! > > Could you please go over any tickets that were related to this branch > and mark them as 'fix committed'? > Done, I could only find one bug (#274067) that was relevant. /J?rgen From ellisonbg.net at gmail.com Tue Mar 17 13:23:06 2009 From: ellisonbg.net at gmail.com (Brian Granger) Date: Tue, 17 Mar 2009 10:23:06 -0700 Subject: [IPython-dev] Extensions: decisions on loading them and API (thoughts from R. Kern's branch) In-Reply-To: <46cb515a0903170905x7b40a67evd4f9942ea5ad8da3@mail.gmail.com> References: <46cb515a0903150157n2ceeac75wc4824eb752b644e2@mail.gmail.com> <20090315092714.GB24801@phare.normalesup.org> <6ce0ac130903161652v6647f235u56d039e90990f169@mail.gmail.com> <6ce0ac130903161847k58c9663esbad5432940a91da5@mail.gmail.com> <46cb515a0903170905x7b40a67evd4f9942ea5ad8da3@mail.gmail.com> Message-ID: <6ce0ac130903171023w5a7df4bcwdd2d351d3517ad53@mail.gmail.com> >> As Fernando has eluded to, it looks like I may have some funding to >> work on a refactor of the ipython core this summer. ?I am very excited >> to finally get a chance to wrestle the beast into submission. > > That's great news! Refactoring it does need some 8hour/day work, a > privilege it certainly hasn't ever enjoyed... Yep, it definitely will be a full time job for a while. >> Another way of saying this is that the user's experience will be the >> same or better, but a developer's experience will be *completely >> different*. ?This will unavoidably mean API breakage. ?Where >> appropriate deprecation warnings will be used, but in many cases, the >> API will change so significantly that even deprecations warnings won't >> be possible. ?I will also try to do things in place and incrementally. > > It's a mixed blessing that the "public" api has been quite shallow. > It's bad, because users have needed to go for InteractiveShell > directly, but it's also good in that breaking the compatibility can be > done without annoying too many ;-). Yes, at some level, we don't have a ton of people who use the dev API. > Most of the compatibility with IPApi can be retained quite easily, > even with a radically different underlying implementation. Take > expose_magic, for example, or custom completers. Yes, where appropriate, I will try to maintain the API. No reason to change the parts that are not broken. >> I will consult with the list extensively about design decisions and >> hopefully as things move forward, others will be interested in helping >> out. ?After 0.10 is released, I will post to the list some more >> details about my near future plans. > > Obvious targest that really need some care are completer.py, > OutputTrap.py, genutils.py. Something to get started anyway ;-). Plenty to get started with. More details to come! Brian > -- > Ville M. Vainio > http://tinyurl.com/vainio > From fperez.net at gmail.com Tue Mar 17 14:39:49 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 17 Mar 2009 11:39:49 -0700 Subject: [IPython-dev] Looking towards 0.10... In-Reply-To: <49BFDAE8.1060909@bostream.nu> References: <20090315092430.GA24801@phare.normalesup.org> <49BE976C.1060909@bostream.nu> <49BFDAE8.1060909@bostream.nu> Message-ID: On Tue, Mar 17, 2009 at 10:16 AM, J?rgen Stenarson > I could only find one bug (#274067) that was relevant. Great, thanks! Cheers, f From jorgen.stenarson at bostream.nu Tue Mar 17 15:24:00 2009 From: jorgen.stenarson at bostream.nu (=?ISO-8859-1?Q?J=F6rgen_Stenarson?=) Date: Tue, 17 Mar 2009 20:24:00 +0100 Subject: [IPython-dev] New branch for more genutils tests Message-ID: <49BFF8D0.3000103@bostream.nu> Hi, I have started a new branch to work on some more tests for genutils. lp:~jorgen-stenarson/ipython/ipython-genutils-tests This is not intended for 0.10 /J?rgen From ellisonbg.net at gmail.com Tue Mar 17 15:55:12 2009 From: ellisonbg.net at gmail.com (Brian Granger) Date: Tue, 17 Mar 2009 12:55:12 -0700 Subject: [IPython-dev] running tests using ipython parallel In-Reply-To: <6ce0ac130903170807q1c920bdbs572733ca94d7cb6c@mail.gmail.com> References: <85b5c3130903170136i4cae74b4peb54f5cd50ef6f04@mail.gmail.com> <6ce0ac130903170807q1c920bdbs572733ca94d7cb6c@mail.gmail.com> Message-ID: <6ce0ac130903171255y381a0298k4745a199ebdc744e@mail.gmail.com> Ondrej, > I am playing with ipython parallel and it behaves much more robustly > than the multiprocessing module from python2.6. In the multiprocessing > module if I get an exception, the process basically gets stuck, ctrl-c > doesn't help and I need to kill it using the "kill" command. Yep, we have worked *very* hard to make sure that exceptions at least get propagated back to the client. ?If they don't, it is a bug and please let us know. > In ipython it works pretty well. I have couple questions though: > > * which approach do you think would be the best to implement parallel > testing? Basically you have a test suite (nosetest and py.test > compatible) and currently in the sequential mode I just call > "execfile", get all functions from the file and execute them. In > parallel I do something along these lines: > > ? ? ? ?from IPython.kernel import client > ? ? ? ?mec = client.MultiEngineClient() > ? ? ? ?mec.reset() > ? ? ? ?print "running %d jobs" % self._jobs > ? ? ? ?ids = mec.get_ids() > ? ? ? ?mec.execute("filename = '%s'" % filename) > ? ? ? ?mec.execute("gl = {'__file__': filename}") > ? ? ? ?mec.execute("execfile(filename, gl)") > ? ? ? ?i = 0 > ? ? ? ?for f in funcs: > ? ? ? ? ? ?if i >= len(ids): i = 0 > ? ? ? ? ? ?#mec.push({"f": f}) > ? ? ? ? ? ?mec.push(dict(f=f)) > ? ? ? ? ? ?#print mec.execute("gl['%s']" % f.__name__, targets=[ids[i]]) > ? ? ? ? ? ?print mec.execute("f", targets=[ids[i]]) > ? ? ? ? ? ?i += 1 > > funcs list contains all the test functions that I can then execute > using f(). Unfortunately this approach gives me an error at the line > "mec.push(dict(f=f))": Is f here an actual function? ?If so, you need to use the method "push_function" instead of push. ?The reason for this is that functions can't be pickled out of the box. ?The push_function method has extra logic that makes it work. Other tips: * If the function you are calling don't take very long, the latency in your current approach will really get you. ?A much better way would be to define a function that could test everything in a packages hierarchy below a certain point. ?That way you could have an engine test an entire subpackage. ?Then the latency will matter less. * The loop you are writing is basically just doing what the map method does: mec.map(lambda x: x**2, range(10)) It works just like python's map, but is parallel. ?The only difference is that map takes either a function or a string that can is exec'd. * If you want to get rid of the code in strings "feature", just define functions, push the function using push_function and then call the function using execute. ?We probably should also implement something like this: mec.call(f, args, **kwargs) If this would be useful to you, could you file a ticket for this? ?In the ticket, could be mention that we should use a cache to make sure that functions are only pushed one time? > ?File "/home/ondrej/repos/sympy/sympy/utilities/runtests.py", line > 214, in test_file > ? ?mec.push(dict(f=f)) > ?File "/var/lib/python-support/python2.6/IPython/kernel/multiengineclient.py", > line 552, in push > ? ?targets=targets, block=block) > ?File "/var/lib/python-support/python2.6/IPython/kernel/multiengineclient.py", > line 441, in _blockFromThread > ? ?result = blockingCallFromThread(function, *args, **kwargs) > ?File "/var/lib/python-support/python2.6/IPython/kernel/twistedutil.py", > line 69, in blockingCallFromThread > ? ?return twisted.internet.threads.blockingCallFromThread(reactor, f, *a, **kw) > ?File "/usr/lib/python2.6/dist-packages/twisted/internet/threads.py", > line 114, in blockingCallFromThread > ? ?result.raiseException() > ?File "/usr/lib/python2.6/dist-packages/twisted/python/failure.py", > line 326, in raiseException > ? ?raise self.type, self.value, self.tb > TypeError: expected string or Unicode object, NoneType found > > > So I thought, ok, let's not push things in and execute everything at > the engines. > > * what things could be safely pushed to engines? I know it can push > some functions, but I didn't manage to get it actually working for the > actual test functions (it works nice for simple functions from the > tutorial). So the only option that seems to me that it should work is > that I first implement a function that executes for "n"th test case > from the test suite and this function will not be transfered to > engines. The only thing that will be pushed is one string ("function > name") and then couple integers to specify all the parameters. Other than functions, anything that can be pickled can be pushed. ?The only big limitation is that classes need to be importable to be pushed. ?Thus, classes that are defined interactively can't be pushed. Functions should be pushed using push_function. But, from the speed perspective, you want to push as little as possible, so just pushing strings is not a bad idea. ?Why don't you get something working first though and then we can figure out the performance issues. One final note: ?we are aware of some performance issues in the current parallel ipython. ?These issues mainly affect latency (small amounts of work done in parallel) and pushing very large objects. ?We are working on these things. Cheers, Brian > > > Ondrej > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > From laurent.dufrechou at gmail.com Tue Mar 17 16:08:01 2009 From: laurent.dufrechou at gmail.com (=?us-ascii?Q?Laurent_Dufrechou?=) Date: Tue, 17 Mar 2009 21:08:01 +0100 Subject: [IPython-dev] Should we fix this bug before 0.10? Message-ID: <49c00326.04ff300a.04e6.435e@mx.google.com> Hi all, Reviewing bugs, i've seen this one that has been shipped with a potential patch. Do one specialist see any bad things in the patch? https://bugs.launchpad.net/bugs/339642 Laurent -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Tue Mar 17 16:24:17 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 17 Mar 2009 13:24:17 -0700 Subject: [IPython-dev] Should we fix this bug before 0.10? In-Reply-To: <49c00326.04ff300a.04e6.435e@mx.google.com> References: <49c00326.04ff300a.04e6.435e@mx.google.com> Message-ID: 2009/3/17 Laurent Dufrechou : > Hi all, > > Reviewing bugs, i?ve seen this one that has been shipped with a potential > patch. > > Do one specialist see any bad things in the patch? > > > > https://bugs.launchpad.net/bugs/339642 For those not subscribed to the bug, here's what I said there: That's indeed a bug, but the patch is removing a line that was put in there explicitly for some reason. So what I'd like to have, before committing this, is a set of tests in a file named test_unicode.py, that encapsulates all of the recent unicode work. Unfortunately a lot of these unicode fixes have been made in a completely ad-hoc manner, as people report problems, but we don't have a centralized list of cases to check against. His may be a reasonable fix, for all I know, but I'm afraid that if we apply it we'll get back 10 old bugs again. I don't know, maybe not, but there's simply no way to be sure. I'm one of the most ignorant of our bunch in unicode issues, blissfully living in the stupidity of the ascii world. It would be great if one of us who knows more about this stuff could at least write a set of simple unicode tests that catch many of the recently reported encoding problems. Jorgen, Ville, any chance you guys could take this up at some point? You know about it a lot more than I do... From laurent.dufrechou at gmail.com Tue Mar 17 16:27:56 2009 From: laurent.dufrechou at gmail.com (=?UTF-8?Q?Laurent_Dufr=C3=A9chou?=) Date: Tue, 17 Mar 2009 21:27:56 +0100 Subject: [IPython-dev] Ipython plugin for Editra In-Reply-To: <2498a980903010210t23201eafx749eef70368f6281@mail.gmail.com> References: <2498a980902281352q48df3928v1b6bb05662f31ef3@mail.gmail.com> <49a9e60b.07eb300a.6f2e.010c@mx.google.com> <2498a980903010210t23201eafx749eef70368f6281@mail.gmail.com> Message-ID: <49c007d1.06e9300a.54fc.19db@mx.google.com> Hi ronen, As Ipython 0.10 will be out probably next week, I?ll wait to update ipython inside editra. Then we will be able to update the plugin with the new ipython code and your plugin update. Laurent De : Ronen Abravanel [mailto:ronena at gmail.com] Envoy? : dimanche 1 mars 2009 11:11 ? : Laurent Dufr?chou Cc : ipython-dev at scipy.org Objet : Re: [IPython-dev] Ipython plugin for Editra Hi Laurent, My hack attached. It is mostly defined at "ShellIntegrator" class at the end of the file. The work flow I want to achieve is as following: * in Ipython - "%edit myfile.py" or "edit 1:5" etc. Open editing buffer within the current windows, and returns without running the file. * Selectin "Run" from the tool menu in Editra run the buffer, as it was sopused to run after closing the editor. By the way, the plugin have some series stabilty issues. calling '%run filename.py' for none trevial python file is cusing editra crash with: python2.5: ../../src/xcb_lock.c:77: _XGetXCBBuffer: Assertion `((int) ((xcb_req) - (dpy->request)) >= 0)' failed. Aborted Thanks, Ronen On Sun, Mar 1, 2009 at 3:33 AM, Laurent Dufr?chou wrote: Hi Ronen, Happy to see you?ve tried to hack the editra plugin ;) IPyShell/__init__.py contains only the interface with editra. I do not cleanly understand the step your are trying to do. You?ve used ? %ed? than it launched editra new source panel? And closing it executed your code, but stdin/out goes outside ipython shell? Is that what you tried? About the files to modify. __init__.py in editra plugn is the definition of the plugin interface and how you interact with editra. Ipython dir is a pure copy that I was used to upload from time to time. I have to take a look at some updates that cody did. (some compatibility issue with python 2.6 and ipython tree) The fact is that the update you?re currently trying to di will surely need some update into /gui/wx I think L. But well we?ll see. Depending on what you?ll want to do we will perhaps expose some methods so plugin will be able to configure the things you need. (ex being able to define an editor) (I think we can use ipython config file but I?ve never used them since then so?) I will try to upload updated sources for python tree tomorrow, so you?ll have a bleeding edge wx integration. I?ve corrected some bugs since then. Sure we will find a way to workaround the stdin/out issue you saw. Send me your current hack, I?ll take a look at your issue. Cheers, Laurent De : ipython-dev-bounces at scipy.org [mailto:ipython-dev-bounces at scipy.org] De la part de Ronen Abravanel Envoy? : samedi 28 f?vrier 2009 22:53 ? : ipython-dev at scipy.org Objet : [IPython-dev] Ipython plugin for Editra Editra, an extensible programmer's editor written in wxPython, contain a cool IPython plugin. This mail is a follow up for a discussion with Laurent , The author of the plugin, in his blog, continue here according to his request. The Orginal discassion can be found in http://ipython0.wordpress.com/2008/04/25/editra-ipython-plugin-under-work/ or at the end of this mail. Using papi to set\redefine IPython's magic word partly do the job, The other part is to execute some arbitrary python code inside The ipython shell. Invoking papis "ex" method (Using papi from ipython_panel.IP._IP.getapi() ) let the Code run in IPython's context, but the output is printed to Editra's stdout (The console in wich I invoked editra's from) and not in the IPython in the shelf. My attempts to set sys.stdout to an handler from, let say, _term.cout or others, cused IPython to stack. futher more, the Exeption throws while I call ip.ex, is cought within Editra and withn IPython. Further more - I wish to change, If I can, Only IPyShell/__init__.py and not the stuff in "IPython" dir, as it seems as a "pure" copy of IPython's orginal source (is it?) Thanks, Ronen. -------------------------------------- The orginal discassion: 1. Ronen Says: February 21, 2009 at 8:18 pm That?s a one cool Plugin. What will make it ideal, is better integration with Editra?s Editor: Make ipython?s %edit Open the edit-buffer in Editra?s current Windows, and Enable poeple to run The code they edit in Editra inside the open IPython windows? Any hint where should I try to hack in order to achieve those? Thanks, Ronen. 2. Erreur ! Nom du fichier non sp?cifi?.Laurent Says: February 22, 2009 at 10:45 am Hello, You can have access to the plugin repos: # Non-members may check out a read-only working copy anonymously over HTTP. svn checkout http://editra-plugins.googlecode.com/svn/trunk/ editra-plugins-read-only The plugin name is ipyshell, in it?s directory you?ve got the whole ipython source + the plugin source. It need a little refresh for sure Erreur ! Nom du fichier non sp?cifi?.. If you want to try to hack it take a look @Ipython/gui/wx the whole code is in ipython_view and ipshellnonblocking. ipython_view -> gui interface ipshellnonblocking -> the interface with ipython You can add: ip = IPython.ipapi.get() def myEdit(self, arg): print ?do what you want? ip.expose_magic(?edit?, myEdit) to WxNonBlockingIPShell inside ipython_view.py Please note that if ipython is ever installed on your system you?ll have to make a eggsetup.py develop to make sure to use your new local copy. We can discuss about all of this in ipython-dev mailing list Erreur ! Nom du fichier non sp?cifi?. Cheers, Laurent -------------- next part -------------- An HTML attachment was scrubbed... URL: From jorgen.stenarson at bostream.nu Tue Mar 17 16:32:18 2009 From: jorgen.stenarson at bostream.nu (=?windows-1252?Q?J=F6rgen_Stenarson?=) Date: Tue, 17 Mar 2009 21:32:18 +0100 Subject: [IPython-dev] Should we fix this bug before 0.10? In-Reply-To: References: <49c00326.04ff300a.04e6.435e@mx.google.com> Message-ID: <49C008D2.6010900@bostream.nu> Fernando Perez skrev: > > For those not subscribed to the bug, here's what I said there: > This is the comment I just made on that bug: The proposed patch does not work for me on win32 with or without pyreadline sys.stdin.encoding == "cp1252" Standard python: c:\python>python Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> "???" '\xe5\xe4\xf6' >>> u"???" u'\xe5\xe4\xf6' >>> IPython from trunk: c:\python>ipython Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] Type "copyright", "credits" or "license" for more information. IPython 0.9.1 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object'. ?object also works, ?? prints more. In [1]: "???" Out[1]: '\xe5\xe4\xf6' In [2]: u"???" Out[2]: u'\xe5\xe4\xf6' In [3]: Do you really want to exit ([y]/n)? IPython with proposed change: c:\python>ipython Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] Type "copyright", "credits" or "license" for more information. IPython 0.9.1 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object'. ?object also works, ?? prints more. In [1]: "???" Out[1]: '\xc3\xa5\xc3\xa4\xc3\xb6' In [2]: u"???" Out[2]: u'\xe5\xe4\xf6' In [3]: Do you really want to exit ([y]/n)? From laurent.dufrechou at gmail.com Tue Mar 17 16:32:55 2009 From: laurent.dufrechou at gmail.com (=?iso-8859-1?Q?Laurent_Dufr=E9chou?=) Date: Tue, 17 Mar 2009 21:32:55 +0100 Subject: [IPython-dev] Should we fix this bug before 0.10? In-Reply-To: References: <49c00326.04ff300a.04e6.435e@mx.google.com> Message-ID: <49c008fc.2115300a.31f9.fffff264@mx.google.com> Just in case you haven't seen this too: https://bugs.launchpad.net/ipython/+bug/290677 It is exactly the same issue. If we'll be able to find a clean way to solve it we close two bug report ;) (and open 10 new ;_; ?) > -----Message d'origine----- > De?: Fernando Perez [mailto:fperez.net at gmail.com] > Envoy??: mardi 17 mars 2009 21:24 > ??: Laurent Dufrechou > Cc?: IPython Development list > Objet?: Re: [IPython-dev] Should we fix this bug before 0.10? > > 2009/3/17 Laurent Dufrechou : > > Hi all, > > > > Reviewing bugs, i?ve seen this one that has been shipped with a > potential > > patch. > > > > Do one specialist see any bad things in the patch? > > > > > > > > https://bugs.launchpad.net/bugs/339642 > > For those not subscribed to the bug, here's what I said there: > > > That's indeed a bug, but the patch is removing a line that was put in > there explicitly for some reason. So what I'd like to have, before > committing this, is a set of tests in a file named test_unicode.py, > that encapsulates all of the recent unicode work. > > Unfortunately a lot of these unicode fixes have been made in a > completely ad-hoc manner, as people report problems, but we don't have > a centralized list of cases to check against. His may be a reasonable > fix, for all I know, but I'm afraid that if we apply it we'll get back > 10 old bugs again. I don't know, maybe not, but there's simply no way > to be sure. > > I'm one of the most ignorant of our bunch in unicode issues, > blissfully living in the stupidity of the ascii world. It would be > great if one of us who knows more about this stuff could at least > write a set of simple unicode tests that catch many of the recently > reported encoding problems. Jorgen, Ville, any chance you guys could > take this up at some point? You know about it a lot more than I do... From ondrej at certik.cz Tue Mar 17 17:05:53 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Tue, 17 Mar 2009 14:05:53 -0700 Subject: [IPython-dev] running tests using ipython parallel In-Reply-To: <6ce0ac130903171255y381a0298k4745a199ebdc744e@mail.gmail.com> References: <85b5c3130903170136i4cae74b4peb54f5cd50ef6f04@mail.gmail.com> <6ce0ac130903170807q1c920bdbs572733ca94d7cb6c@mail.gmail.com> <6ce0ac130903171255y381a0298k4745a199ebdc744e@mail.gmail.com> Message-ID: <85b5c3130903171405k120837ewcbbd736a69d3832e@mail.gmail.com> On Tue, Mar 17, 2009 at 8:07 AM, Brian Granger wrote: >> I am playing with ipython parallel and it behaves much more robustly >> than the multiprocessing module from python2.6. In the multiprocessing >> module if I get an exception, the process basically gets stuck, ctrl-c >> doesn't help and I need to kill it using the "kill" command. > > Yep, we have worked *very* hard to make sure that exceptions at least > get propagated back to the client. If they don't, it is a bug and > please let us know. > >> In ipython it works pretty well. I have couple questions though: >> >> * which approach do you think would be the best to implement parallel >> testing? Basically you have a test suite (nosetest and py.test >> compatible) and currently in the sequential mode I just call >> "execfile", get all functions from the file and execute them. In >> parallel I do something along these lines: >> >> from IPython.kernel import client >> mec = client.MultiEngineClient() >> mec.reset() >> print "running %d jobs" % self._jobs >> ids = mec.get_ids() >> mec.execute("filename = '%s'" % filename) >> mec.execute("gl = {'__file__': filename}") >> mec.execute("execfile(filename, gl)") >> i = 0 >> for f in funcs: >> if i >= len(ids): i = 0 >> #mec.push({"f": f}) >> mec.push(dict(f=f)) >> #print mec.execute("gl['%s']" % f.__name__, targets=[ids[i]]) >> print mec.execute("f", targets=[ids[i]]) >> i += 1 >> >> funcs list contains all the test functions that I can then execute >> using f(). Unfortunately this approach gives me an error at the line >> "mec.push(dict(f=f))": > > Is f here an actual function? If so, you need to use the method > "push_function" instead of push. The reason for this is that > functions can't be pickled out of the box. The push_function method > has extra logic that makes it work. Oops, you are right. Now I got a bit further. Simple functions now work, but once some test function is decorated, it fails: Traceback (most recent call last): File "bin/test", line 36, in "jobs": options.jobs}) File "/home/ondrej/repos/sympy/sympy/utilities/runtests.py", line 71, in test return t.test() File "/home/ondrej/repos/sympy/sympy/utilities/runtests.py", line 135, in test self.test_file(f) File "/home/ondrej/repos/sympy/sympy/utilities/runtests.py", line 214, in test_file mec.push_function(dict(f=f)) File "/home/ondrej/lib/lib/python/IPython/kernel/multiengineclient.py", line 591, in push_function return self._blockFromThread(self.smultiengine.push_function, namespace, targets=targets, block=block) File "/home/ondrej/lib/lib/python/IPython/kernel/multiengineclient.py", line 441, in _blockFromThread result = blockingCallFromThread(function, *args, **kwargs) File "/home/ondrej/lib/lib/python/IPython/kernel/twistedutil.py", line 69, in blockingCallFromThread return twisted.internet.threads.blockingCallFromThread(reactor, f, *a, **kw) File "/usr/lib/python2.6/dist-packages/twisted/internet/threads.py", line 114, in blockingCallFromThread result.raiseException() File "/usr/lib/python2.6/dist-packages/twisted/python/failure.py", line 326, in raiseException raise self.type, self.value, self.tb ValueError: Sorry, cannot pickle code objects with closures So it seems to me, that the only way to go forward is not to push any code, because this will not work in general. So I think what I need is to implement a simple function in sympy tests, that will parse the test file and return me a list of tests (like a string or something) and this will happen *on the engine*. And I will just tell it -- hey, execute the test 37 and tell me the result. > > Other tips: > > * If the function you are calling don't take very long, the latency in > your current approach will really get you. A much better way would be > to define a function that could test everything in a packages > hierarchy below a certain point. That way you could have an engine > test an entire subpackage. Then the latency will matter less. At this moment I don't care about latency, but as it turns out, I don't have any other option, see above. > > * The loop you are writing is basically just doing what the map method does: > > mec.map(lambda x: x**2, range(10)) > > It works just like python's map, but is parallel. The only difference > is that map takes either a function or a string that can is exec'd. Yes, in fact I wrote it using python map and then tried to use multiprocessing map and it failed if the function uses a decorator. So I think pushing any nontrivial code is a bad idea. > > * If you want to get rid of the code in strings "feature", just define > functions, push the function using push_function and then call the > function using execute. We probably should also implement something > like this: > > mec.call(f, args, **kwargs) > > If this would be useful to you, could you file a ticket for this? In > the ticket, could be mention that we should use a cache to make sure > that functions are only pushed one time? If I find it useful, I'll post the ticket. So far I am not sure due to the problems above. > > > > >> File "/home/ondrej/repos/sympy/sympy/utilities/runtests.py", line >> 214, in test_file >> mec.push(dict(f=f)) >> File "/var/lib/python-support/python2.6/IPython/kernel/multiengineclient.py", >> line 552, in push >> targets=targets, block=block) >> File "/var/lib/python-support/python2.6/IPython/kernel/multiengineclient.py", >> line 441, in _blockFromThread >> result = blockingCallFromThread(function, *args, **kwargs) >> File "/var/lib/python-support/python2.6/IPython/kernel/twistedutil.py", >> line 69, in blockingCallFromThread >> return twisted.internet.threads.blockingCallFromThread(reactor, f, *a, **kw) >> File "/usr/lib/python2.6/dist-packages/twisted/internet/threads.py", >> line 114, in blockingCallFromThread >> result.raiseException() >> File "/usr/lib/python2.6/dist-packages/twisted/python/failure.py", >> line 326, in raiseException >> raise self.type, self.value, self.tb >> TypeError: expected string or Unicode object, NoneType found >> >> >> So I thought, ok, let's not push things in and execute everything at >> the engines. >> >> * what things could be safely pushed to engines? I know it can push >> some functions, but I didn't manage to get it actually working for the >> actual test functions (it works nice for simple functions from the >> tutorial). So the only option that seems to me that it should work is >> that I first implement a function that executes for "n"th test case >> from the test suite and this function will not be transfered to >> engines. The only thing that will be pushed is one string ("function >> name") and then couple integers to specify all the parameters. > > Other than functions, anything that can be pickled can be pushed. The > only big limitation is that classes need to be importable to be > pushed. Thus, classes that are defined interactively can't be pushed. > > Functions should be pushed using push_function. > > But, from the speed perspective, you want to push as little as > possible, so just pushing strings is not a bad idea. Why don't you > get something working first though and then we can figure out the > performance issues. Exactly, that's my approach. I need to go the "do everything on engines" route anyway though. > > One final note: we are aware of some performance issues in the > current parallel ipython. These issues mainly affect latency (small > amounts of work done in parallel) and pushing very large objects. We > are working on these things. Sure, that's not my problem at the moment. :) Ondrej From fperez.net at gmail.com Tue Mar 17 17:18:25 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 17 Mar 2009 14:18:25 -0700 Subject: [IPython-dev] Looking towards 0.10... In-Reply-To: References: <20090315092430.GA24801@phare.normalesup.org> <49BE976C.1060909@bostream.nu> <49BFDAE8.1060909@bostream.nu> Message-ID: Hi all, status update on 0.10: after review of my trunk and updates post-review (hey, code review works! ;), I've just merged it, and it's a *massive* diff. So I'd like to encourage those with branches marked for review: https://code.launchpad.net/ipython/+activereviews to first make sure you merge with trunk again. That will ensure there are no conflicts left (you may have to fix a few first in your branch), and it also gives you better support for more easily writing tests. I tried to add a fair amount of info on the testing support, and I just triggered manually a doc rebuild so the branch docs are easily available online for all of you: http://ipython.scipy.org/doc/manual/ So once all the pending branches get the trunk merged, we can try to wrap up the remaining reviews and close off 0.10! Cheers, f From ellisonbg.net at gmail.com Tue Mar 17 17:20:09 2009 From: ellisonbg.net at gmail.com (Brian Granger) Date: Tue, 17 Mar 2009 14:20:09 -0700 Subject: [IPython-dev] Looking towards 0.10... In-Reply-To: References: <20090315092430.GA24801@phare.normalesup.org> <49BE976C.1060909@bostream.nu> <49BFDAE8.1060909@bostream.nu> Message-ID: <6ce0ac130903171420xffb81a9iefa75984343321ea@mail.gmail.com> Great, thanks for all the hard work, especially all the great code reviews from everyone! From fperez.net at gmail.com Tue Mar 17 17:22:42 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 17 Mar 2009 14:22:42 -0700 Subject: [IPython-dev] Looking towards 0.10... In-Reply-To: <6ce0ac130903171420xffb81a9iefa75984343321ea@mail.gmail.com> References: <20090315092430.GA24801@phare.normalesup.org> <49BE976C.1060909@bostream.nu> <49BFDAE8.1060909@bostream.nu> <6ce0ac130903171420xffb81a9iefa75984343321ea@mail.gmail.com> Message-ID: On Tue, Mar 17, 2009 at 2:20 PM, Brian Granger wrote: > Great, thanks for all the hard work, especially all the great code > reviews from everyone! BTW, have a look at the docs now: http://ipython.scipy.org/doc/manual/ The pdf is 488 pages :) f From fperez.net at gmail.com Tue Mar 17 18:00:26 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 17 Mar 2009 15:00:26 -0700 Subject: [IPython-dev] Should we fix this bug before 0.10? In-Reply-To: <49c008fc.2115300a.31f9.fffff264@mx.google.com> References: <49c00326.04ff300a.04e6.435e@mx.google.com> <49c008fc.2115300a.31f9.fffff264@mx.google.com> Message-ID: On Tue, Mar 17, 2009 at 1:32 PM, Laurent Dufr?chou wrote: > Just in case you haven't seen this too: > https://bugs.launchpad.net/ipython/+bug/290677 > > It is exactly the same issue. If we'll be able to find a clean way to solve > it we close two bug report ;) > (and open 10 new ;_; ?) If it's really a dupe, mark it as such. Cheers, f From fperez.net at gmail.com Tue Mar 17 18:06:01 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 17 Mar 2009 15:06:01 -0700 Subject: [IPython-dev] Ipython plugin for Editra In-Reply-To: <49c007d1.06e9300a.54fc.19db@mx.google.com> References: <2498a980902281352q48df3928v1b6bb05662f31ef3@mail.gmail.com> <49a9e60b.07eb300a.6f2e.010c@mx.google.com> <2498a980903010210t23201eafx749eef70368f6281@mail.gmail.com> <49c007d1.06e9300a.54fc.19db@mx.google.com> Message-ID: 2009/3/17 Laurent Dufr?chou : > Hi ronen, > > As Ipython 0.10 will be out probably next week, I?ll wait to update ipython > inside editra. Just as a side note: could you guys please stick to plain text email on this list? The orgy of fonts and color in this thread is giving me seizures. The 90's are over, the tag has been mostly eradicated from the face of the Earth :) Thanks much. Cheers, f From fperez.net at gmail.com Tue Mar 17 18:11:34 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 17 Mar 2009 15:11:34 -0700 Subject: [IPython-dev] Looking towards 0.10... In-Reply-To: References: <49BFDAE8.1060909@bostream.nu> <6ce0ac130903171420xffb81a9iefa75984343321ea@mail.gmail.com> <6ce0ac130903171506k59eb5733i69b8133ee3e0d926@mail.gmail.com> Message-ID: [ sent off-list by accident] On Tue, Mar 17, 2009 at 3:10 PM, Fernando Perez wrote: > On Tue, Mar 17, 2009 at 3:06 PM, Brian Granger wrote: >>> BTW, have a look at the docs now: >>> >>> http://ipython.scipy.org/doc/manual/ >>> >>> The pdf is 488 pages :) >> >> Is this because of the API docs? > > Yup, mostly. ?And I switched ?the build to happen on my box and push > up instead of on the scipy host. ?This allows things like the Wx api > docs to build as well (and it's *much* faster to boot). > > Also, I just updated the docs page, so that now the doc links are: > > http://ipython.scipy.org/doc/stable/ > http://ipython.scipy.org/doc/nightly/ > http://ipython.scipy.org/doc/rel-0.9/ > http://ipython.scipy.org/doc/rel-0.9.1/ > etc... > > 'stable' is a symlink I'll update on each release to point to the > latest one, but this gives us a way to access any set of manuals from > all the builds. ?It would be nice to add back the old ones, I might do > that at some point but not now. > > Cheers, > > f > From fperez.net at gmail.com Tue Mar 17 18:12:37 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 17 Mar 2009 15:12:37 -0700 Subject: [IPython-dev] Looking towards 0.10... In-Reply-To: <6ce0ac130903171508s304ec844i4c962b359682d81a@mail.gmail.com> References: <49BFDAE8.1060909@bostream.nu> <6ce0ac130903171420xffb81a9iefa75984343321ea@mail.gmail.com> <6ce0ac130903171506k59eb5733i69b8133ee3e0d926@mail.gmail.com> <6ce0ac130903171508s304ec844i4c962b359682d81a@mail.gmail.com> Message-ID: On Tue, Mar 17, 2009 at 3:08 PM, Brian Granger wrote: > On Tue, Mar 17, 2009 at 3:06 PM, Brian Granger wrote: >>> BTW, have a look at the docs now: >>> >>> http://ipython.scipy.org/doc/manual/ >>> >>> The pdf is 488 pages :) >> > > Is this because of the API docs? > > Answer to self: ?yes, wow! ?The API docs look great! ?One question: > The APi docs have a huge footprint on the main TOC page of the HTML. > Is there a way of making that smaller and maybe only list the > top-level modules and packages there? I don't know... I left it that way for now, we'll need to dig into sphinx to figure out if that's possible or not. It would be nice, I just don't know. Cheers, f ps - I assume you meant this on-list, I put it back there. From jorgen.stenarson at bostream.nu Tue Mar 17 18:42:15 2009 From: jorgen.stenarson at bostream.nu (=?ISO-8859-1?Q?J=F6rgen_Stenarson?=) Date: Tue, 17 Mar 2009 23:42:15 +0100 Subject: [IPython-dev] Looking towards 0.10... In-Reply-To: References: <20090315092430.GA24801@phare.normalesup.org> <49BE976C.1060909@bostream.nu> <49BFDAE8.1060909@bostream.nu> Message-ID: <49C02747.4000406@bostream.nu> Fernando Perez skrev: > Hi all, > > status update on 0.10: after review of my trunk and updates > post-review (hey, code review works! ;), I've just merged it, and it's > a *massive* diff. So I'd like to encourage those with branches marked > for review: > Hi, unfortunately I get massive errors on the test suite now FAILED (SKIP=18, errors=116, failures=30) on win32. I just wanted to report it, I will look more closely tomorrow on what the problem may be. /J?rgen From fperez.net at gmail.com Tue Mar 17 18:43:35 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 17 Mar 2009 15:43:35 -0700 Subject: [IPython-dev] Looking towards 0.10... In-Reply-To: <49C02747.4000406@bostream.nu> References: <20090315092430.GA24801@phare.normalesup.org> <49BE976C.1060909@bostream.nu> <49BFDAE8.1060909@bostream.nu> <49C02747.4000406@bostream.nu> Message-ID: On Tue, Mar 17, 2009 at 3:42 PM, J?rgen Stenarson wrote: > Fernando Perez skrev: >> >> Hi all, >> >> status update on 0.10: ?after review of my trunk and updates >> post-review (hey, code review works! ;), I've just merged it, and it's >> a *massive* diff. ?So I'd like to encourage those with branches marked >> for review: >> > Hi, > > unfortunately I get massive errors on the test suite now > FAILED (SKIP=18, errors=116, failures=30) on win32. I just wanted to report > it, I will look more closely tomorrow on what the problem may be. Ouch. Please post a log even just now so I can have a look, it might be obvious to me right away. I don't have any windows box to test on. f From fperez.net at gmail.com Tue Mar 17 18:59:43 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 17 Mar 2009 15:59:43 -0700 Subject: [IPython-dev] New branch for more genutils tests In-Reply-To: <49BFF8D0.3000103@bostream.nu> References: <49BFF8D0.3000103@bostream.nu> Message-ID: On Tue, Mar 17, 2009 at 12:24 PM, J?rgen Stenarson wrote: > Hi, > > I have started a new branch to work on some more tests for genutils. > > lp:~jorgen-stenarson/ipython/ipython-genutils-tests > > This is not intended for 0.10 Great! I might pitch in on that one... f From ellisonbg.net at gmail.com Tue Mar 17 19:14:35 2009 From: ellisonbg.net at gmail.com (Brian Granger) Date: Tue, 17 Mar 2009 16:14:35 -0700 Subject: [IPython-dev] New branch for more genutils tests In-Reply-To: <49BFF8D0.3000103@bostream.nu> References: <49BFF8D0.3000103@bostream.nu> Message-ID: <6ce0ac130903171614n78e38c77gcb32ee9f5003511c@mail.gmail.com> One thing to note is that right after 0.10 is released, I will be doing a massive reorganization of the top level modules into appropriate subpackage. Thus, genutils will probably end up being moved. You can go ahead and work in this branch, but I wanted to let you know that merging it back into the post 0.10 trunk could be a bit painful because of the changes. My hope is that we could freeze the trunk for a very short time while I make all of these changes so that things don't become too much of a mess. I will post something to the list about my plans soon. Cheers, Brian On Tue, Mar 17, 2009 at 12:24 PM, J?rgen Stenarson wrote: > Hi, > > I have started a new branch to work on some more tests for genutils. > > lp:~jorgen-stenarson/ipython/ipython-genutils-tests > > This is not intended for 0.10 > > > /J?rgen > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > From jorgen.stenarson at bostream.nu Wed Mar 18 13:20:04 2009 From: jorgen.stenarson at bostream.nu (=?ISO-8859-1?Q?J=F6rgen_Stenarson?=) Date: Wed, 18 Mar 2009 18:20:04 +0100 Subject: [IPython-dev] New branch for more genutils tests In-Reply-To: <6ce0ac130903171614n78e38c77gcb32ee9f5003511c@mail.gmail.com> References: <49BFF8D0.3000103@bostream.nu> <6ce0ac130903171614n78e38c77gcb32ee9f5003511c@mail.gmail.com> Message-ID: <49C12D44.50200@bostream.nu> Brian Granger skrev: > One thing to note is that right after 0.10 is released, I will be > doing a massive reorganization of the top level modules into > appropriate subpackage. Thus, genutils will probably end up being > moved. You can go ahead and work in this branch, but I wanted to let > you know that merging it back into the post 0.10 trunk could be a bit > painful because of the changes. My hope is that we could freeze the > trunk for a very short time while I make all of these changes so that > things don't become too much of a mess. > > I will post something to the list about my plans soon. > or we could review/merge the tests we have done so far when you are about to start your work. /J?rgen From fperez.net at gmail.com Wed Mar 18 13:54:50 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 18 Mar 2009 10:54:50 -0700 Subject: [IPython-dev] Looking towards 0.10... In-Reply-To: <49C12CA0.8020207@bostream.nu> References: <49BE976C.1060909@bostream.nu> <49BFDAE8.1060909@bostream.nu> <49C02747.4000406@bostream.nu> <49C12CA0.8020207@bostream.nu> Message-ID: Hey, On Wed, Mar 18, 2009 at 10:17 AM, J?rgen Stenarson wrote: > Here is the log. > > A quick look and it seems there are a lot of ImportErrors Thanks for that log. Since we say that ipython depends only on the stdlib for its core functionality, then the test suite should run with only that as well. There are a lot of tests there that use third-party functionality, we'll need to mark them as optional. Brian already did a lot of that for the twisted ones, it's just a matter of following up with the same idea on the others. Thanks for the log, it contains all the info we need to fix this. I'll set up a 'naked' virtualenv that should show me most of these errors. Cheers, f From jorgen.stenarson at bostream.nu Wed Mar 18 15:49:28 2009 From: jorgen.stenarson at bostream.nu (=?ISO-8859-1?Q?J=F6rgen_Stenarson?=) Date: Wed, 18 Mar 2009 20:49:28 +0100 Subject: [IPython-dev] New branch for more genutils tests In-Reply-To: References: <49BFF8D0.3000103@bostream.nu> Message-ID: <49C15048.2080101@bostream.nu> Fernando Perez skrev: > On Tue, Mar 17, 2009 at 12:24 PM, J?rgen Stenarson > wrote: >> Hi, >> >> I have started a new branch to work on some more tests for genutils. >> >> lp:~jorgen-stenarson/ipython/ipython-genutils-tests >> >> This is not intended for 0.10 > > Great! I might pitch in on that one... > > f > I'll be gone on vacation for about a week, I will not work on the branch during that time. /J?rgen From ellisonbg.net at gmail.com Wed Mar 18 16:17:48 2009 From: ellisonbg.net at gmail.com (Brian Granger) Date: Wed, 18 Mar 2009 13:17:48 -0700 Subject: [IPython-dev] New branch for more genutils tests In-Reply-To: <49C12D44.50200@bostream.nu> References: <49BFF8D0.3000103@bostream.nu> <6ce0ac130903171614n78e38c77gcb32ee9f5003511c@mail.gmail.com> <49C12D44.50200@bostream.nu> Message-ID: <6ce0ac130903181317o775c9a7w670604508d87830f@mail.gmail.com> OK, let's just see how the timing of all of this goes. Cheers, Brian On Wed, Mar 18, 2009 at 10:20 AM, J?rgen Stenarson wrote: > Brian Granger skrev: >> >> One thing to note is that right after 0.10 is released, I will be >> doing a massive reorganization of the top level modules into >> appropriate subpackage. ?Thus, genutils will probably end up being >> moved. ?You can go ahead and work in this branch, but I wanted to let >> you know that merging it back into the post 0.10 trunk could be a bit >> painful because of the changes. ?My hope is that we could freeze the >> trunk for a very short time while I make all of these changes so that >> things don't become too much of a mess. >> >> I will post something to the list about my plans soon. >> > > or we could review/merge the tests we have done so far when you are about to > start your work. > > /J?rgen > From ellisonbg.net at gmail.com Wed Mar 18 16:24:41 2009 From: ellisonbg.net at gmail.com (Brian Granger) Date: Wed, 18 Mar 2009 13:24:41 -0700 Subject: [IPython-dev] Test failures and errors in trunk Message-ID: <6ce0ac130903181324j9cb1352veae1fe9f2a8b85cd@mail.gmail.com> Hi, I pulled the latest trunk this morning and ran the test suite. I got a huge mess of errors and failures. My pre-merge branch had none of this, so the recent merges have introduced many new bugs. Some of these bugs look to be very subtle and related to the deep mysteries of Twisted. Soo... * Can everyone go and run the test suite WITH ALL OF IPYTHON's DEPENDENCIES INSTALLED!!! If you don't have some of the optional dependencies installed, many tests will be skipped even though they would fail if run. * If one of your branches was merged, please go back and make sure the test suite passes in the merged version of your branch. * Please find and fix the bugs that your branch/merge introduced. I am hoping everyone can track their bugs down. If not, we will need to start backing out the merges to help narrow down where the subtle twisted related bugs have crept in. Here is the current result of running the test suite: brian-grangers-macbook-pro:~ bgranger$ iptest .................E.............S......................................../Users/bgranger/Library/Python/2.5/site-packages/Twisted-8.2.0pre1-py2.5-macosx-10.5-i386.egg/twisted/trial/unittest.py:1350: DeprecationWarning: Reactor already running! This behavior is deprecated since Twisted 8.0 reactor.run() ...........................................................................................................................................................................................ETraceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/threading.py", line 460, in __bootstrap self.run() File "/Users/bgranger/Documents/Computation/IPython/branches/ipython/IPython/kernel/twistedutil.py", line 42, in run File "/Users/bgranger/Library/Python/2.5/site-packages/Twisted-8.2.0pre1-py2.5-macosx-10.5-i386.egg/twisted/internet/base.py", line 1128, in run self.mainLoop() File "/Users/bgranger/Library/Python/2.5/site-packages/Twisted-8.2.0pre1-py2.5-macosx-10.5-i386.egg/twisted/internet/base.py", line 1137, in mainLoop self.runUntilCurrent() --- --- File "/Users/bgranger/Library/Python/2.5/site-packages/Twisted-8.2.0pre1-py2.5-macosx-10.5-i386.egg/twisted/internet/base.py", line 729, in runUntilCurrent f(*a, **kw) File "/Users/bgranger/Library/Python/2.5/site-packages/Twisted-8.2.0pre1-py2.5-macosx-10.5-i386.egg/twisted/internet/defer.py", line 269, in errback self._startRunCallbacks(fail) File "/Users/bgranger/Library/Python/2.5/site-packages/Twisted-8.2.0pre1-py2.5-macosx-10.5-i386.egg/twisted/internet/defer.py", line 298, in _startRunCallbacks raise AlreadyCalledError twisted.internet.defer.AlreadyCalledError: ............................................................................................................................................................................................E...........F.........F.......S....S......SS..SSS..............SSS.... ====================================================================== ERROR: Failure: ImportError (No module named Numeric) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/bgranger/Library/Python/2.5/site-packages/nose-0.10.4-py2.5.egg/nose/loader.py", line 364, in loadTestsFromName addr.filename, addr.module) File "/Users/bgranger/Library/Python/2.5/site-packages/nose-0.10.4-py2.5.egg/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/Users/bgranger/Library/Python/2.5/site-packages/nose-0.10.4-py2.5.egg/nose/importer.py", line 84, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/Users/bgranger/Documents/Computation/IPython/Code/ipython/IPython/Extensions/numeric_formats.py", line 6, in from Numeric import ArrayType ImportError: No module named Numeric ====================================================================== ERROR: testExecuteFailuresEngineService_0 ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/bgranger/Library/Python/2.5/site-packages/Twisted-8.2.0pre1-py2.5-macosx-10.5-i386.egg/twisted/internet/base.py", line 729, in runUntilCurrent f(*a, **kw) File "/Users/bgranger/Library/Python/2.5/site-packages/Twisted-8.2.0pre1-py2.5-macosx-10.5-i386.egg/twisted/internet/defer.py", line 269, in errback self._startRunCallbacks(fail) File "/Users/bgranger/Library/Python/2.5/site-packages/Twisted-8.2.0pre1-py2.5-macosx-10.5-i386.egg/twisted/internet/defer.py", line 298, in _startRunCallbacks raise AlreadyCalledError AlreadyCalledError ====================================================================== ERROR: Failure: ImportError (No module named Numeric) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/bgranger/Library/Python/2.5/site-packages/nose-0.10.4-py2.5.egg/nose/loader.py", line 364, in loadTestsFromName addr.filename, addr.module) File "/Users/bgranger/Library/Python/2.5/site-packages/nose-0.10.4-py2.5.egg/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/Users/bgranger/Library/Python/2.5/site-packages/nose-0.10.4-py2.5.egg/nose/importer.py", line 84, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/Users/bgranger/Documents/Computation/IPython/Code/ipython/IPython/numutils.py", line 31, in import Numeric ImportError: No module named Numeric ====================================================================== FAIL: Doctest: IPython.testing.plugin.dtexample.ipfunc ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/bgranger/Documents/Computation/IPython/Code/ipython/IPython/testing/plugin/ipdoctest.py", line 382, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for IPython.testing.plugin.dtexample.ipfunc File "/Users/bgranger/Documents/Computation/IPython/Code/ipython/IPython/testing/plugin/dtexample.py", line 27, in ipfunc ---------------------------------------------------------------------- File "/Users/bgranger/Documents/Computation/IPython/Code/ipython/IPython/testing/plugin/dtexample.py", line 44, in IPython.testing.plugin.dtexample.ipfunc Failed example: _ip.system("echo hello") Exception raised: Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/doctest.py", line 1212, in __run compileflags, 1) in test.globs File "", line 1, in _ip.system("echo hello") File "/Users/bgranger/Documents/Computation/IPython/Code/ipython/IPython/testing/plugin/ipdoctest.py", line 137, in xsys sys.stdout.write(commands.getoutput(cmd)) File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/commands.py", line 44, in getoutput return getstatusoutput(cmd)[1] File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/commands.py", line 54, in getstatusoutput text = pipe.read() IOError: [Errno 4] Interrupted system call ---------------------------------------------------------------------- File "/Users/bgranger/Documents/Computation/IPython/Code/ipython/IPython/testing/plugin/dtexample.py", line 47, in IPython.testing.plugin.dtexample.ipfunc Failed example: _ip.system("echo hello > /tmp/foo") Exception raised: Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/doctest.py", line 1212, in __run compileflags, 1) in test.globs File "", line 1, in _ip.system("echo hello > /tmp/foo") File "/Users/bgranger/Documents/Computation/IPython/Code/ipython/IPython/testing/plugin/ipdoctest.py", line 137, in xsys sys.stdout.write(commands.getoutput(cmd)) File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/commands.py", line 44, in getoutput return getstatusoutput(cmd)[1] File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/commands.py", line 54, in getstatusoutput text = pipe.read() IOError: [Errno 4] Interrupted system call ---------------------------------------------------------------------- File "/Users/bgranger/Documents/Computation/IPython/Code/ipython/IPython/testing/plugin/dtexample.py", line 52, in IPython.testing.plugin.dtexample.ipfunc Failed example: _ip.system("rm -f /tmp/foo") Exception raised: Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/doctest.py", line 1212, in __run compileflags, 1) in test.globs File "", line 1, in _ip.system("rm -f /tmp/foo") File "/Users/bgranger/Documents/Computation/IPython/Code/ipython/IPython/testing/plugin/ipdoctest.py", line 137, in xsys sys.stdout.write(commands.getoutput(cmd)) File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/commands.py", line 44, in getoutput return getstatusoutput(cmd)[1] File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/commands.py", line 54, in getstatusoutput text = pipe.read() IOError: [Errno 4] Interrupted system call >> raise self.failureException(self.format_failure(.getvalue())) ====================================================================== FAIL: Doctest: test_exampleip.txt ---------------------------------------------------------------------- Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/doctest.py", line 2112, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for test_exampleip.txt File "/Users/bgranger/Documents/Computation/IPython/Code/ipython/IPython/testing/plugin/test_exampleip.txt", line 0 ---------------------------------------------------------------------- File "/Users/bgranger/Documents/Computation/IPython/Code/ipython/IPython/testing/plugin/test_exampleip.txt", line 29, in test_exampleip.txt Failed example: _ip.system("echo $a") Exception raised: Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/doctest.py", line 1212, in __run compileflags, 1) in test.globs File "", line 1, in _ip.system("echo $a") File "/Users/bgranger/Documents/Computation/IPython/Code/ipython/IPython/testing/plugin/ipdoctest.py", line 137, in xsys sys.stdout.write(commands.getoutput(cmd)) File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/commands.py", line 44, in getoutput return getstatusoutput(cmd)[1] File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/commands.py", line 54, in getstatusoutput text = pipe.read() IOError: [Errno 4] Interrupted system call >> raise self.failureException(self.format_failure(.getvalue())) ---------------------------------------------------------------------- Ran 515 tests in 57.113s FAILED (SKIP=11, errors=3, failures=2) Deleting object: second_pass object A deleted deleting object... From fperez.net at gmail.com Wed Mar 18 18:11:13 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 18 Mar 2009 15:11:13 -0700 Subject: [IPython-dev] Test failures and errors in trunk In-Reply-To: <6ce0ac130903181324j9cb1352veae1fe9f2a8b85cd@mail.gmail.com> References: <6ce0ac130903181324j9cb1352veae1fe9f2a8b85cd@mail.gmail.com> Message-ID: Howdy, On Wed, Mar 18, 2009 at 1:24 PM, Brian Granger wrote: > Hi, > > I pulled the latest trunk this morning and ran the test suite. ?I got > a huge mess of errors and failures. ?My pre-merge branch had none of > this, so the recent merges have introduced many new bugs. > > Some of these bugs look to be very subtle and related to the deep > mysteries of Twisted. ............................................................................................................................................................................................E...........F.........F.......S....S......SS..SSS..............SSS.... > ====================================================================== > ERROR: Failure: ImportError (No module named Numeric) > ---------------------------------------------------------------------- This one should be protected so it doesn't run without numeric, like Jorgen said earlier. I didn't realize I'd left a Numeric-dependent test in there. My Python install has just about everything, so I didn't see these. > ====================================================================== > ERROR: testExecuteFailuresEngineService_0 > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/Users/bgranger/Library/Python/2.5/site-packages/Twisted-8.2.0pre1-py2.5-macosx-10.5-i386.egg/twisted/internet/base.py", > line 729, in runUntilCurrent > ? ?f(*a, **kw) > ?File "/Users/bgranger/Library/Python/2.5/site-packages/Twisted-8.2.0pre1-py2.5-macosx-10.5-i386.egg/twisted/internet/defer.py", > line 269, in errback > ? ?self._startRunCallbacks(fail) > ?File "/Users/bgranger/Library/Python/2.5/site-packages/Twisted-8.2.0pre1-py2.5-macosx-10.5-i386.egg/twisted/internet/defer.py", > line 298, in _startRunCallbacks > ? ?raise AlreadyCalledError > AlreadyCalledError This one, I've seen occasionally, and I remember we've talked about it several times. It's completely non-deterministic, and you said that you'd also seen it some times. I'd love to know how to track it, but I have no clue: it's obviously deep in Twisted. This isn't just a Heisenbug, it's a Twisted Heisenbug :) > ====================================================================== > FAIL: Doctest: IPython.testing.plugin.dtexample.ipfunc > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/Users/bgranger/Documents/Computation/IPython/Code/ipython/IPython/testing/plugin/ipdoctest.py", > line 382, in runTest > ? ?raise self.failureException(self.format_failure(new.getvalue())) > AssertionError: Failed doctest test for IPython.testing.plugin.dtexample.ipfunc > ?File "/Users/bgranger/Documents/Computation/IPython/Code/ipython/IPython/testing/plugin/dtexample.py", > line 27, in ipfunc > > ---------------------------------------------------------------------- > File "/Users/bgranger/Documents/Computation/IPython/Code/ipython/IPython/testing/plugin/dtexample.py", > line 44, in IPython.testing.plugin.dtexample.ipfunc > Failed example: > ? ?_ip.system("echo hello") > Exception raised: > ? ?Traceback (most recent call last): > ? ? ?File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/doctest.py", > line 1212, in __run > ? ? ? ?compileflags, 1) in test.globs > ? ? ?File "", > line 1, in > ? ? ? ?_ip.system("echo hello") > ? ? ?File "/Users/bgranger/Documents/Computation/IPython/Code/ipython/IPython/testing/plugin/ipdoctest.py", > line 137, in xsys > ? ? ? ?sys.stdout.write(commands.getoutput(cmd)) > ? ? ?File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/commands.py", > line 44, in getoutput > ? ? ? ?return getstatusoutput(cmd)[1] > ? ? ?File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/commands.py", > line 54, in getstatusoutput > ? ? ? ?text = pipe.read() > ? ?IOError: [Errno 4] Interrupted system call Now, these stump me: you see several of these, which I've never ever seen. I wonder if the commands.py module behaves differently on different OSes. It's important that we can run these tests reliably, because those are the ones that let us do full 'real ipython' testing. > Ran 515 tests in 57.113s > FAILED (SKIP=11, errors=3, failures=2) In all, you get three errors, so it's not *that bad*, considering we added over 100 new tests. Those are: 1. The twisted heisenbug. No idea, this one has been haunting us for well over a year (I remember talking to you about it back when I was still in Boulder). It's only that now I see it more often (I'd say now I see it ~80% of the time, before it was more like ~40%). 2. The Numeric dependency. Easy to fix. In fact, I need to clean many more of those, for people who don't have MPL or other packages (c.f. conversation earlier with Jorgen). 3. The IOError on OSX. I have no clue on that one, but I'll think about it. It's extremely strange. On my box, I get: Ran 515 tests in 43.525s FAILED (SKIP=11, errors=1) The error is #1 above, I don't get 2 and 3 (2 because I have Numeric, 3 I don't know). I'm actually much more worried about the fact that today, John Hunter discovered that my lovely fix to https://bugs.launchpad.net/ipython/+bug/269966 actually can cause problems with MPL (open figure objects hold references to the old module objects that get zeroed out). I'm actually starting to wonder if this isn't a bug with Python itself, in how it clears module objects ignoring reference counts (or that's what appears to be happening, at least). Relax :) That's why I wanted all that testing support code merged in: because by enabling many more tests to run, it's likely to show us these problems. Now we can actually target them for fixing :) Cheers, f From ellisonbg.net at gmail.com Wed Mar 18 19:35:49 2009 From: ellisonbg.net at gmail.com (Brian Granger) Date: Wed, 18 Mar 2009 16:35:49 -0700 Subject: [IPython-dev] Test failures and errors in trunk In-Reply-To: References: <6ce0ac130903181324j9cb1352veae1fe9f2a8b85cd@mail.gmail.com> Message-ID: <6ce0ac130903181635l75db524bme9c0baf8f4d39ab9@mail.gmail.com> ====================================================================== >> ERROR: testExecuteFailuresEngineService_0 >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> ?File "/Users/bgranger/Library/Python/2.5/site-packages/Twisted-8.2.0pre1-py2.5-macosx-10.5-i386.egg/twisted/internet/base.py", >> line 729, in runUntilCurrent >> ? ?f(*a, **kw) >> ?File "/Users/bgranger/Library/Python/2.5/site-packages/Twisted-8.2.0pre1-py2.5-macosx-10.5-i386.egg/twisted/internet/defer.py", >> line 269, in errback >> ? ?self._startRunCallbacks(fail) >> ?File "/Users/bgranger/Library/Python/2.5/site-packages/Twisted-8.2.0pre1-py2.5-macosx-10.5-i386.egg/twisted/internet/defer.py", >> line 298, in _startRunCallbacks >> ? ?raise AlreadyCalledError >> AlreadyCalledError > > This one, I've seen occasionally, and I remember we've talked about it > several times. ?It's completely non-deterministic, and you said that > you'd also seen it some times. ?I'd love to know how to track it, but > I have no clue: it's obviously deep in Twisted. > This isn't just a Heisenbug, it's a Twisted Heisenbug :) Yes, we have seen this one before. But a while back Barry and I figured out where it was coming from and we completely fixed it (so we thought). These are super difficult to track down. If anyone see odd Twisted bugs, please don't disregard them. Usually, finding these bugs means doing a line by line audit of all new code. The problem is that all the usual debugging tricks don't work (print statements, debuggers) because of the non-deterministic nature of them. But, I do know what causes them. I am going to back and find the last working revision of trunk that lacks these errors and then start to move forward to see when the problem appears. ====================================================================== >> FAIL: Doctest: IPython.testing.plugin.dtexample.ipfunc >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> ?File "/Users/bgranger/Documents/Computation/IPython/Code/ipython/IPython/testing/plugin/ipdoctest.py", >> line 382, in runTest >> ? ?raise self.failureException(self.format_failure(new.getvalue())) >> AssertionError: Failed doctest test for IPython.testing.plugin.dtexample.ipfunc >> ?File "/Users/bgranger/Documents/Computation/IPython/Code/ipython/IPython/testing/plugin/dtexample.py", >> line 27, in ipfunc >> >> ---------------------------------------------------------------------- >> File "/Users/bgranger/Documents/Computation/IPython/Code/ipython/IPython/testing/plugin/dtexample.py", >> line 44, in IPython.testing.plugin.dtexample.ipfunc >> Failed example: >> ? ?_ip.system("echo hello") >> Exception raised: >> ? ?Traceback (most recent call last): >> ? ? ?File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/doctest.py", >> line 1212, in __run >> ? ? ? ?compileflags, 1) in test.globs >> ? ? ?File "", >> line 1, in >> ? ? ? ?_ip.system("echo hello") >> ? ? ?File "/Users/bgranger/Documents/Computation/IPython/Code/ipython/IPython/testing/plugin/ipdoctest.py", >> line 137, in xsys >> ? ? ? ?sys.stdout.write(commands.getoutput(cmd)) >> ? ? ?File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/commands.py", >> line 44, in getoutput >> ? ? ? ?return getstatusoutput(cmd)[1] >> ? ? ?File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/commands.py", >> line 54, in getstatusoutput >> ? ? ? ?text = pipe.read() >> ? ?IOError: [Errno 4] Interrupted system call > > Now, these stump me: you see several of these, which I've never ever > seen. ?I wonder if the commands.py module behaves differently on > different OSes. > It's important that we can run these tests reliably, because those are > the ones that let us do full 'real ipython' testing. I agree that this is important. But this does sound subtle. One possibility is that this is a side effect of the above Twisted bug. But again, I think I will have to simple rewind to the last revision that was clean, and then star to move forward through the merges. This is just messy when there have been so many big merges. >> Ran 515 tests in 57.113s >> FAILED (SKIP=11, errors=3, failures=2) > > In all, you get three errors, so it's not *that bad*, considering we > added over 100 new tests. ?Those are: True. And like you say, the bright side is that we have a test suite that finds bugs!!!! > 1. The twisted heisenbug. ?No idea, this one has been haunting us for > well over a year (I remember talking to you about it back when I was > still in Boulder). ?It's only that now I see it more often (I'd say > now I see it ~80% of the time, before it was more like ~40%). > > 2. The Numeric dependency. ?Easy to fix. ?In fact, I need to clean > many more of those, for people who don't have MPL or other packages > (c.f. conversation earlier with Jorgen). Yes, this is trivial, but it is really important that we regularly run the test suite on Python installs that don't have any of the dependencies installed. Much of the extra effort that arose in doing the 0.9 release was catching these types of things. I almost think we need an API that allows modules or package to declare what they depend on, so we can write tests to make sure our tests work (if that makes sense). > 3. The IOError on OSX. ?I have no clue on that one, but I'll think > about it. ?It's extremely strange. The first and third of these are in the "painful" category. I guess at some level it is my fault though for not running the test suite after each merge is done. Oh well. > I'm actually much more worried about the fact that today, John Hunter > discovered that my lovely fix to > > ?https://bugs.launchpad.net/ipython/+bug/269966 > > actually can cause problems with MPL (open figure objects hold > references to the old module objects that get zeroed out). ?I'm > actually starting to wonder if this isn't a bug with Python itself, in > how it clears module objects ignoring reference counts (or that's what > appears to be happening, at least). Sounds subtle. You would never guess that a project like IPython would have to deal with such subtle things. It is pretty amazing. > Relax :) ?That's why I wanted all that testing support code merged in: > because by enabling many more tests to run, it's likely to show us > these problems. ?Now we can actually target them for fixing :) Yep, I definitely agree with this. Sorry about my test failure grumpiness :) Brian > Cheers, > > f > From fperez.net at gmail.com Wed Mar 18 21:04:16 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 18 Mar 2009 18:04:16 -0700 Subject: [IPython-dev] Test failures and errors in trunk In-Reply-To: <6ce0ac130903181635l75db524bme9c0baf8f4d39ab9@mail.gmail.com> References: <6ce0ac130903181324j9cb1352veae1fe9f2a8b85cd@mail.gmail.com> <6ce0ac130903181635l75db524bme9c0baf8f4d39ab9@mail.gmail.com> Message-ID: On Wed, Mar 18, 2009 at 4:35 PM, Brian Granger wrote: > Yep, I definitely agree with this. ?Sorry about my test failure grumpiness :) No worries :) Let's not beat ourselves though; instead we can learn useful lessons to get better: - big merges==bad. That was totally my fault on this particular occasion, I left that branch of mine grow too big for too long. Big apology. - When reviewing non-trivial branches from someone else: we must always branch it locally, switch into it and run it before approving. I did manage to do that with Jorgen's py2exe-fix one, and I got errors he wasn't seeing. I made some fixes and republished it for him to pick up, and eventually we were able to clean things up outside of trunk before it got merged. That even lets you test the merge into trunk locally to see how it would go (in a throw-away copy of trunk you only use locally), before 'the public' sees it. For example, if Jorgen had gotten my branch (even without merging, just running it as-is), he would have seen a ton of those errors that would have alerted me to my mistake of forgetting about dependencies. For this kind of work, one thing that may be useful, if a particular branch turns out to require collaborative work, is to change it from ~person to ~ipython-dev. That way more than one person can directly commit into it, svn-style, before it goes into the real trunk. We've done that with nipy a few times to speed up the collaboration between two or three people in shaping up a branch before merging it into trunk. So the good thing is that we're refining our process, learning, and these problems do look fixable (though a few are indeed subtle). I'll continue to fix things as much as I can over the next few days, promised. Cheers, f From Tfetherston at aol.com Wed Mar 18 22:33:00 2009 From: Tfetherston at aol.com (Tfetherston at aol.com) Date: Wed, 18 Mar 2009 22:33:00 EDT Subject: [IPython-dev] Looking towards 0.10... Message-ID: Hi, Haven't been keeping up with the list lately so I just noticed that you are looking to put out a 0.10. My branch only has changes to demo.py and it does three things: 1. Cause demo generated ansi escapes to be interpreted by pyreadline so they work on windows machines instead of appearing as strange characters in the window shell. 2. Tests for platform and uses cls for windows, clear for others so the ClearMixin's work on all jplatforms. 3. Reworks Demo's __init__ so it can take file and file-like objects in addition to just filenames to create Demo instances from. I did the latter to support creating/storing demos from stringio objects kept in a Leo Workbook, with an eye to adding direct demo illustration of the "Dive into Python" web book. I also created a file called demoExercizer.py that makes it convenient to see these changes in action. This is not meant to be part of the distribution. A full branch is probably over kill for two files, but I'm new to all this. Tested this on windows XP and Ubuntu. I like to propose these changes for review and inclusion in 0.10 Tom Fetherston **************Feeling the pinch at the grocery store? Make meals for Under $10. (http://food.aol.com/frugal-feasts?ncid=emlcntusfood00000002) -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Wed Mar 18 22:40:06 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 18 Mar 2009 19:40:06 -0700 Subject: [IPython-dev] Looking towards 0.10... In-Reply-To: References: Message-ID: Hi Tom, 2009/3/18 : > Hi, > > Haven't been keeping up with the list lately so I just noticed that you are > looking to put out a 0.10.? My branch only has changes to demo.py and > it?does three things: > > 1. Cause demo generated ansi escapes to be interpreted by pyreadline so they > work on windows machines instead of appearing as strange characters in the > window shell. > 2. Tests for platform and uses cls for windows, clear for others so the > ClearMixin's work on all jplatforms. > 3. Reworks Demo's __init__ so it can take file and file-like objects in > addition to just filenames to create Demo instances from. > > I did the latter to support creating/storing demos from stringio objects > kept in a Leo Workbook, with an eye to adding direct demo illustration of > the "Dive into Python" web book. > > I also created a file called demoExercizer.py that makes it convenient to > see these changes in action.? This is not meant to be part of the > distribution. That sounds all great! Thanks for the contribution. > A full branch is probably over kill for two files, but I'm new to all this. Not at all, launchpad makes branches be the most natural 'unit' for review, testing, commenting and evolution of a contribution. You're fine. > > Tested this on windows XP and Ubuntu. > > I like to propose these changes for review and inclusion in 0.10 Great. Please go ahead and mark it as requested for merge on LP, and copy your above message into it so we have a summary of what you've done, your intent, etc. We'll review it on LP. As you may have noticed, we first have a bit of a mess to clean up after the giant merge from yesterday, so it may take us a few days :) Best f From fperez.net at gmail.com Wed Mar 18 22:43:01 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 18 Mar 2009 19:43:01 -0700 Subject: [IPython-dev] Test failures and errors in trunk In-Reply-To: <6ce0ac130903181920n27cd8aaase53e6e8c20adba6b@mail.gmail.com> References: <6ce0ac130903181324j9cb1352veae1fe9f2a8b85cd@mail.gmail.com> <6ce0ac130903181635l75db524bme9c0baf8f4d39ab9@mail.gmail.com> <6ce0ac130903181920n27cd8aaase53e6e8c20adba6b@mail.gmail.com> Message-ID: On Wed, Mar 18, 2009 at 7:20 PM, Brian Granger wrote: > I will reply more later. ?I have narrowed the problem to something in > your branch between revision 1153 and 1157. ?I should be able to find > it in a later tonight. Wow, that would be fantastic. I didn't think I was touching anything twisted-related, and since I'd seen this problem before, I thought it was the same thing. I'll know better from now on never to ignore a twisted problem, even if it's a bit of a Heisenbug. In the meantime, I'm working up some utilities to ease the virtualenv workflow, based on some of the scripts that are out there from Doug Hellman and Chris Burns. This will let us do tests in 'naked python' virtualenvs more easily, so we can really have a test suite that at least runs gracefully (skips but no errors) on nothing but python(stdlib)+nose+ipython. Stay tuned. f From fperez.net at gmail.com Wed Mar 18 22:44:52 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 18 Mar 2009 19:44:52 -0700 Subject: [IPython-dev] Test failures and errors in trunk In-Reply-To: <6ce0ac130903181659q76ee42c7ka2e301a011ffdaa@mail.gmail.com> References: <6ce0ac130903181324j9cb1352veae1fe9f2a8b85cd@mail.gmail.com> <6ce0ac130903181659q76ee42c7ka2e301a011ffdaa@mail.gmail.com> Message-ID: On Wed, Mar 18, 2009 at 4:59 PM, Brian Granger wrote: > OK, I found out what merge introduced these problems. ?It was the big > merge from your trunk Fernando. ?To make sure that this is really the > source of the problem, Fernando, can you check out r1158 of trunk and > run the test suite. ?I am hoping that you will find no errors. ?If > this is what you find, then I will start to go through the revisions > in your branch to see where these things came in exactly. > > One off thing. ?On my machine, I get 518 tests rather than the 151 > that you find. ?Is is possible that on my machine I am finding extra > tests that aren't really tests and thus fail? ?What version of nose > are you running? I'd imagine those are the Cocoa ones. I'm using nose 0.10.4. It would be nice to rework the test reporter so it lists (when in -v mode) the skipped tests at the end. It's always useful to see that printed. Cheers, f From ondrej at certik.cz Wed Mar 18 22:52:27 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 18 Mar 2009 19:52:27 -0700 Subject: [IPython-dev] parallel: BaseException.message has been deprecated Message-ID: <85b5c3130903181952x225de0b5we09b5ceed38e8536@mail.gmail.com> Hi, if I run this code: try: print mec.execute("get_tests()") except: t, v, tr = sys.exc_info() print t print v print tr I keep getting this traceback: /home/ondrej/lib/lib/python/IPython/kernel/error.py:130: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 s = str(self.message) Traceback (most recent call last): File "t.py", line 27, in print v File "/home/ondrej/lib/lib/python/IPython/kernel/error.py", line 133, in __str__ s = s + '\n' + engine_str + str(et.__name__) + ': ' + str(ev) AttributeError: 'str' object has no attribute '__name__' So this patch fixes the deprecation warning: === modified file 'IPython/kernel/error.py' --- IPython/kernel/error.py 2008-06-06 19:41:55 +0000 +++ IPython/kernel/error.py 2009-03-19 02:38:41 +0000 @@ -107,7 +107,7 @@ class CompositeError(KernelError): def __init__(self, message, elist): Exception.__init__(self, *(message, elist)) - self.message = message + self.msg = message self.elist = elist def _get_engine_str(self, ev): @@ -127,7 +127,7 @@ return tb def __str__(self): - s = str(self.message) + s = str(self.msg) for et, ev, etb in self.elist: engine_str = self._get_engine_str(ev) s = s + '\n' + engine_str + str(et.__name__) + ': ' + str(ev) Now I only get this traceback: import struct, sets, time Traceback (most recent call last): File "t.py", line 27, in print v File "/home/ondrej/lib/lib/python/IPython/kernel/error.py", line 133, in __str__ s = s + '\n' + engine_str + str(et.__name__) + ': ' + str(ev) AttributeError: 'str' object has no attribute '__name__' a quick debugging shows: ipdb> p et 'exceptions.NameError' ipdb> p type(et) and str doesn't have any .__name__. So a fix could be along these lines: - s = s + '\n' + engine_str + str(et.__name__) + ': ' + str(ev) + s = s + '\n' + engine_str + et + ': ' + str(ev) Now my script above shows the actual error: one or more exceptions from call to method: execute [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined So I tried to get the line which goes wrong, by following the docs: http://ipython.scipy.org/doc/nightly/html/parallel/parallel_multiengine.html#parallel-exceptions by using .raise_exception() try: print mec.execute("get_tests()") except: t, v, tr = sys.exc_info() print t print v print tr print dir(v) v.raise_exception() but I only got: Traceback (most recent call last): File "t.py", line 30, in v.raise_exception() File "/home/ondrej/lib/lib/python/IPython/kernel/error.py", line 157, in raise_exception raise et, ev, etb TypeError: exceptions must be classes or instances, not str So I'll try to debug it by some other methods to see what went wrong. Ondrej From ondrej at certik.cz Wed Mar 18 23:11:52 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 18 Mar 2009 20:11:52 -0700 Subject: [IPython-dev] parallel: BaseException.message has been deprecated In-Reply-To: <85b5c3130903181952x225de0b5we09b5ceed38e8536@mail.gmail.com> References: <85b5c3130903181952x225de0b5we09b5ceed38e8536@mail.gmail.com> Message-ID: <85b5c3130903182011q72f1f06er15d20f85c6c26f08@mail.gmail.com> > one or more exceptions from call to method: execute > [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined > [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined > [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined > [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined > [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined > [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined > [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined > [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined This seems to be some problem with ipython, because if I push a function that works, e.g.: def get_tests(): pass #from sympy.utilities.runtests import PyTestReporter, SymPyTests #r = PyTestReporter(verbose, tb, colors) #t = SymPyTests(r, kw, post_mortem) #t.add_paths(["sympy/core"]) #tests = t.collect_tests_in_file(t._tests[0]) #return tests from IPython.kernel import client mec = client.MultiEngineClient() mec.reset() ids = mec.get_ids() mec.push_function({"get_tests": get_tests}) try: print mec.execute("get_tests()") except: t, v, tr = sys.exc_info() print t print v print tr print dir(v) v.raise_exception() Then it prints: [0] In [1]: get_tests() [1] In [1]: get_tests() [2] In [1]: get_tests() [3] In [1]: get_tests() [4] In [1]: get_tests() [5] In [1]: get_tests() [6] In [1]: get_tests() [7] In [1]: get_tests() But if I push the following function: def get_tests(): pass from sympy.utilities.runtests import PyTestReporter, SymPyTests Then it prints the error from my previous email. The problem is however that sympy is not in my sys.path so it should raise a regular import error exception. To prove my theory, here is a simple script, that fails for me: ---------- import sys def fails(): e = 1/0 from IPython.kernel import client mec = client.MultiEngineClient() mec.reset() ids = mec.get_ids() mec.push_function({"f": fails}) try: print mec.execute("get_tests()") except: t, v, tr = sys.exc_info() print v v.raise_exception() ----------- I am using python2.6, since this is what is default in ubuntu now. If you have time, I'll provide you with ssh access to my computer so that you can debug this yourself. I originally wanted to go for a beer tonight, but this makes me reconsider. :) I need to get the parallel testing working soon, since I won't have a time for this later. Ondrej From ellisonbg.net at gmail.com Wed Mar 18 23:24:22 2009 From: ellisonbg.net at gmail.com (Brian Granger) Date: Wed, 18 Mar 2009 20:24:22 -0700 Subject: [IPython-dev] parallel: BaseException.message has been deprecated In-Reply-To: <85b5c3130903181952x225de0b5we09b5ceed38e8536@mail.gmail.com> References: <85b5c3130903181952x225de0b5we09b5ceed38e8536@mail.gmail.com> Message-ID: <6ce0ac130903182024l19e2a287m72dee427a4d7f7ad@mail.gmail.com> I will have a look at this tonight. Are you on IRC? Brian On Wed, Mar 18, 2009 at 7:52 PM, Ondrej Certik wrote: > Hi, > > if I run this code: > > try: > ? ?print mec.execute("get_tests()") > except: > ? ?t, v, tr = sys.exc_info() > ? ?print t > ? ?print v > ? ?print tr > > I keep getting this traceback: > > /home/ondrej/lib/lib/python/IPython/kernel/error.py:130: > DeprecationWarning: BaseException.message has been deprecated as of > Python 2.6 > ?s = str(self.message) > Traceback (most recent call last): > ?File "t.py", line 27, in > ? ?print v > ?File "/home/ondrej/lib/lib/python/IPython/kernel/error.py", line > 133, in __str__ > ? ?s = s + '\n' + engine_str + str(et.__name__) + ': ' + str(ev) > AttributeError: 'str' object has no attribute '__name__' > > > So this patch fixes the deprecation warning: > > === modified file 'IPython/kernel/error.py' > --- IPython/kernel/error.py ? ? 2008-06-06 19:41:55 +0000 > +++ IPython/kernel/error.py ? ? 2009-03-19 02:38:41 +0000 > @@ -107,7 +107,7 @@ > ?class CompositeError(KernelError): > ? ? def __init__(self, message, elist): > ? ? ? ? Exception.__init__(self, *(message, elist)) > - ? ? ? ?self.message = message > + ? ? ? ?self.msg = message > ? ? ? ? self.elist = elist > > ? ? def _get_engine_str(self, ev): > @@ -127,7 +127,7 @@ > ? ? ? ? ? ? return tb > > ? ? def __str__(self): > - ? ? ? ?s = str(self.message) > + ? ? ? ?s = str(self.msg) > ? ? ? ? for et, ev, etb in self.elist: > ? ? ? ? ? ? engine_str = self._get_engine_str(ev) > ? ? ? ? ? ? s = s + '\n' + engine_str + str(et.__name__) + ': ' + str(ev) > > > > > Now I only get this traceback: > > ?import struct, sets, time > > Traceback (most recent call last): > ?File "t.py", line 27, in > ? ?print v > ?File "/home/ondrej/lib/lib/python/IPython/kernel/error.py", line > 133, in __str__ > ? ?s = s + '\n' + engine_str + str(et.__name__) + ': ' + str(ev) > AttributeError: 'str' object has no attribute '__name__' > > a quick debugging shows: > > ipdb> p et > 'exceptions.NameError' > ipdb> p type(et) > > > and str doesn't have any .__name__. So a fix could be along these lines: > > - ? ? ? ? ? ?s = s + '\n' + engine_str + str(et.__name__) + ': ' + str(ev) > + ? ? ? ? ? ?s = s + '\n' + engine_str + et + ': ' + str(ev) > > > Now my script above shows the actual error: > > one or more exceptions from call to method: execute > [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined > [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined > [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined > [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined > [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined > [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined > [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined > [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined > > > So I tried to get the line which goes wrong, by following the docs: > > http://ipython.scipy.org/doc/nightly/html/parallel/parallel_multiengine.html#parallel-exceptions > > by using .raise_exception() > > try: > ? ?print mec.execute("get_tests()") > except: > ? ?t, v, tr = sys.exc_info() > ? ?print t > ? ?print v > ? ?print tr > ? ?print dir(v) > ? ?v.raise_exception() > > but I only got: > > Traceback (most recent call last): > ?File "t.py", line 30, in > ? ?v.raise_exception() > ?File "/home/ondrej/lib/lib/python/IPython/kernel/error.py", line > 157, in raise_exception > ? ?raise et, ev, etb > TypeError: exceptions must be classes or instances, not str > > > So I'll try to debug it by some other methods to see what went wrong. > > Ondrej > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > From ondrej at certik.cz Wed Mar 18 23:25:51 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 18 Mar 2009 20:25:51 -0700 Subject: [IPython-dev] parallel: BaseException.message has been deprecated In-Reply-To: <6ce0ac130903182024l19e2a287m72dee427a4d7f7ad@mail.gmail.com> References: <85b5c3130903181952x225de0b5we09b5ceed38e8536@mail.gmail.com> <6ce0ac130903182024l19e2a287m72dee427a4d7f7ad@mail.gmail.com> Message-ID: <85b5c3130903182025s203fc06br9c356a09a79feba8@mail.gmail.com> On Wed, Mar 18, 2009 at 8:24 PM, Brian Granger wrote: > I will have a look at this tonight. ?Are you on IRC? Yep, on #sympy and #scipy and #ipython. Ondrej From ondrej at certik.cz Wed Mar 18 23:40:42 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 18 Mar 2009 20:40:42 -0700 Subject: [IPython-dev] parallel: BaseException.message has been deprecated In-Reply-To: <85b5c3130903181952x225de0b5we09b5ceed38e8536@mail.gmail.com> References: <85b5c3130903181952x225de0b5we09b5ceed38e8536@mail.gmail.com> Message-ID: <85b5c3130903182040m32b2d503x1051bfa820657fbd@mail.gmail.com> > but I only got: > > Traceback (most recent call last): > ?File "t.py", line 30, in > ? ?v.raise_exception() > ?File "/home/ondrej/lib/lib/python/IPython/kernel/error.py", line > 157, in raise_exception > ? ?raise et, ev, etb > TypeError: exceptions must be classes or instances, not str So this problem is because python2.6 doesn't allow string exceptions. For some reason, et, ev and etb are just strings. So I am now using this patch: === modified file 'IPython/kernel/error.py' --- IPython/kernel/error.py 2008-06-06 19:41:55 +0000 +++ IPython/kernel/error.py 2009-03-19 03:23:44 +0000 @@ -107,7 +107,7 @@ class CompositeError(KernelError): def __init__(self, message, elist): Exception.__init__(self, *(message, elist)) - self.message = message + self.msg = message self.elist = elist def _get_engine_str(self, ev): @@ -127,10 +127,10 @@ return tb def __str__(self): - s = str(self.message) + s = str(self.msg) for et, ev, etb in self.elist: engine_str = self._get_engine_str(ev) - s = s + '\n' + engine_str + str(et.__name__) + ': ' + str(ev) + s = s + '\n' + engine_str + et + ': ' + str(ev) return s def print_tracebacks(self, excid=None): @@ -154,7 +154,8 @@ except: raise IndexError("an exception with index %i does not exist"%excid) else: - raise et, ev, etb + msg = "%s | %s | %s" % (et, ev, etb) + raise Exception(msg) def collect_exceptions(rlist, method): elist = [] And now if I run the following script: -------------- import sys def fails(): e = 1/0 from IPython.kernel import client mec = client.MultiEngineClient() mec.reset() ids = mec.get_ids() mec.push_function({"f": fails}) try: print mec.execute("get_tests()") except: t, v, tr = sys.exc_info() print v v.raise_exception() ---------- I get: /usr/lib/python2.6/dist-packages/twisted/python/filepath.py:12: DeprecationWarning: the sha module is deprecated; use the hashlib module instead import sha /usr/lib/python2.6/dist-packages/foolscap/banana.py:2: DeprecationWarning: the sets module is deprecated import struct, sets, time one or more exceptions from call to method: execute [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined [Engine Exception]exceptions.NameError: global name '__IPYTHON__' is not defined Traceback (most recent call last): File "q.py", line 16, in v.raise_exception() File "/home/ondrej/lib/lib/python/IPython/kernel/error.py", line 158, in raise_exception raise Exception(msg) Exception: exceptions.NameError | global name '__IPYTHON__' is not defined | None So this is some progress -- basically some code tries to use __IPYTHON__, which is not defined. Anyway, it seems it is all caused by using python2.6. The problem is that even if I run $ python2.5 ~/lib/bin/ipcluster local -n 8 2009-03-18 20:32:53-0700 [-] Log opened. 2009-03-18 20:32:53-0700 [-] Process ['ipcontroller', '--logfile=/home/ondrej/.ipython/log/ipcontroller'] has started with pid=17407 2009-03-18 20:32:53-0700 [-] '/usr/lib/python2.6/dist-packages/twisted/python/filepath.py:12: DeprecationWarning: the sha module is deprecated; use the hashlib module instead\n import sha\n' [...] It still seems it still runs python2.6, inferring from the deprecation warnings. :( So I decided to do small steps. So I reverted all my local changed and tried to get the testsuite run. It seems that it fails at many places: $ scripts/iptest /var/lib/python-support/python2.6/nose/plugins/manager.py:386: UserWarning: Module nose was already imported from /var/lib/python-support/python2.6/nose/__init__.py, but /var/lib/python-support/python2.6 is being added to sys.path import pkg_resources /usr/lib/python2.6/dist-packages/twisted/python/filepath.py:12: DeprecationWarning: the sha module is deprecated; use the hashlib module instead import sha >print(x) /usr/lib/python2.6/dist-packages/foolscap/banana.py:2: DeprecationWarning: the sets module is deprecated import struct, sets, time >print('hello') >print('hello') >print(repr(_)) .............................S......................................../usr/lib/python2.6/dist-packages/twisted/trial/unittest.py:1350: DeprecationWarning: Reactor already running! This behavior is deprecated since Twisted 8.0 reactor.run() .....................................................................................................................ERROR: An unexpected error occurred while tokenizing input The following traceback may be corrupted or invalid The error message is: ('EOF in multi-line statement', (43, 0)) ..................................................................ERROR: An unexpected error occurred while tokenizing input The following traceback may be corrupted or invalid The error message is: ('EOF in multi-line statement', (43, 0)) ....Traceback (most recent call last): File "/usr/lib/python2.6/threading.py", line 522, in __bootstrap_inner self.run() File "/home/ondrej/repos/ipython/IPython/kernel/twistedutil.py", line 42, in run reactor.run(installSignalHandlers=0) File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 1128, in run self.mainLoop() File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 1137, in mainLoop self.runUntilCurrent() --- --- File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 729, in runUntilCurrent f(*a, **kw) File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 269, in errback self._startRunCallbacks(fail) File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 298, in _startRunCallbacks raise AlreadyCalledError twisted.internet.defer.AlreadyCalledError: ETraceback (most recent call last): File "/usr/lib/python2.6/threading.py", line 522, in __bootstrap_inner self.run() File "/home/ondrej/repos/ipython/IPython/kernel/twistedutil.py", line 42, in run reactor.run(installSignalHandlers=0) File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 1128, in run self.mainLoop() File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 1137, in mainLoop self.runUntilCurrent() --- --- File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 729, in runUntilCurrent f(*a, **kw) File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 269, in errback self._startRunCallbacks(fail) File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 298, in _startRunCallbacks raise AlreadyCalledError twisted.internet.defer.AlreadyCalledError: ETraceback (most recent call last): File "/usr/lib/python2.6/dist-packages/twisted/trial/unittest.py", line 1195, in runThunk self._wait(d) File "/usr/lib/python2.6/dist-packages/twisted/trial/unittest.py", line 1350, in _wait reactor.run() File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 1128, in run self.mainLoop() File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 1137, in mainLoop self.runUntilCurrent() --- --- File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 729, in runUntilCurrent f(*a, **kw) File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 269, in errback self._startRunCallbacks(fail) File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 298, in _startRunCallbacks raise AlreadyCalledError twisted.internet.defer.AlreadyCalledError: E.....................................................ERROR: An unexpected error occurred while tokenizing input The following traceback may be corrupted or invalid The error message is: ('EOF in multi-line statement', (43, 0)) ......./home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ./home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ./home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ./home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ./home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ./home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ./home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ..../home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message .........../home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ../home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ............../home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ................./home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ...../home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ........../home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ../home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ........................./home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message .............../home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ......./home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ../home/ondrej/repos/ipython/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ...................................S..S.......SS..SSS.........sh: ipython: not found F....SSS.... ====================================================================== ERROR: testExecuteFailuresEngineService_0 ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 729, in runUntilCurrent f(*a, **kw) File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 269, in errback self._startRunCallbacks(fail) File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 298, in _startRunCallbacks raise AlreadyCalledError AlreadyCalledError ====================================================================== ERROR: testExecuteFailuresEngineService_1 ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 729, in runUntilCurrent f(*a, **kw) File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 269, in errback self._startRunCallbacks(fail) File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 298, in _startRunCallbacks raise AlreadyCalledError AlreadyCalledError ====================================================================== ERROR: testExecuteFailuresEngineService_2 ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 729, in runUntilCurrent f(*a, **kw) File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 269, in errback self._startRunCallbacks(fail) File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 298, in _startRunCallbacks raise AlreadyCalledError AlreadyCalledError ====================================================================== FAIL: Test that object's __del__ methods are called on exit. ---------------------------------------------------------------------- Traceback (most recent call last): File "/var/lib/python-support/python2.6/nose/case.py", line 182, in runTest self.test(*self.arg) File "/home/ondrej/repos/ipython/IPython/tests/test_magic.py", line 88, in test_obj_del nt.assert_equals(out,'object A deleted') AssertionError: '' != 'object A deleted' >> raise self.failureException, \ (None or '%r != %r' % ('', 'object A deleted')) ---------------------------------------------------------------------- Ran 511 tests in 50.749s FAILED (SKIP=11, errors=3, failures=1) Deleting object: second_pass object A deleted deleting object... So I am now looking at how to fix it, so that it runs. Then I can see what I can do about my problem. Ondrej From ellisonbg.net at gmail.com Wed Mar 18 23:41:59 2009 From: ellisonbg.net at gmail.com (Brian Granger) Date: Wed, 18 Mar 2009 20:41:59 -0700 Subject: [IPython-dev] Test failures and errors in trunk In-Reply-To: References: <6ce0ac130903181324j9cb1352veae1fe9f2a8b85cd@mail.gmail.com> <6ce0ac130903181635l75db524bme9c0baf8f4d39ab9@mail.gmail.com> <6ce0ac130903181920n27cd8aaase53e6e8c20adba6b@mail.gmail.com> Message-ID: <6ce0ac130903182041w58684ad7ufe817a6844a453b2@mail.gmail.com> OK, I found the commit that introduced the error. It first appears in this revision: http://bazaar.launchpad.net/~fdo.perez/ipython/trunk-dev/revision/1155 The previous revision passes all tests just fine. You can get the problematic revision by doing: bzr branch lp:~fdo.perez/ipython/trunk-dev -r 1155 It is worth checking that the previous revision is OK on your system as well. But, this one could be a challenge to figure out. In this revision, you were only touching non-twisted parts of the test suite - basically getting the ipdoctest stuff working right. Hopefully between the two of us we can figure this out. I know the Twisted stuff, you know the ip testing stuff. I am going to switch gears to help Ondrej with some things. If you end up looking at this and have questions on the Twisted side of things let me know. > Wow, that would be fantastic. ?I didn't think I was touching anything > twisted-related, and since I'd seen this problem before, I thought it > was the same thing. Looks pretty similar, but I think it is somewhat different. I have never seen the double call to reactor.run() before. But > I'll know better from now on never to ignore a twisted problem, even > if it's a bit of a Heisenbug. > > In the meantime, I'm working up some utilities to ease the virtualenv > workflow, based on some of the scripts that are out there from Doug > Hellman and Chris Burns. ?This will let us do tests in 'naked python' > virtualenvs more easily, so we can really have a test suite that at > least runs gracefully (skips but no errors) on nothing but > python(stdlib)+nose+ipython. Yes, this seems like it is pretty important. I ended up doing that when we did the last release and it really helped. Cheers, Brian > Stay tuned. > > f > From ondrej at certik.cz Wed Mar 18 23:44:40 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 18 Mar 2009 20:44:40 -0700 Subject: [IPython-dev] parallel: BaseException.message has been deprecated In-Reply-To: <85b5c3130903182040m32b2d503x1051bfa820657fbd@mail.gmail.com> References: <85b5c3130903181952x225de0b5we09b5ceed38e8536@mail.gmail.com> <85b5c3130903182040m32b2d503x1051bfa820657fbd@mail.gmail.com> Message-ID: <85b5c3130903182044x73018816i21703d58861239b0@mail.gmail.com> > Anyway, it seems it is all caused by using python2.6. The problem is > that even if I run I wished it was that easy. The problem is maybe in a broken twisted installation in ubuntu, because the tests fail even with python2.5: $ python2.5 scripts/iptest /var/lib/python-support/python2.5/nose/plugins/manager.py:386: UserWarning: Module nose was already imported from /var/lib/python-support/python2.5/nose/__init__.py, but /var/lib/python-support/python2.5 is being added to sys.path import pkg_resources .............................S......................................../usr/lib/python2.5/site-packages/twisted/trial/unittest.py:1350: DeprecationWarning: Reactor already running! This behavior is deprecated since Twisted 8.0 reactor.run() ...........................................................................................................................................................................................Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/twisted/trial/unittest.py", line 1195, in runThunk self._wait(d) File "/usr/lib/python2.5/site-packages/twisted/trial/unittest.py", line 1350, in _wait reactor.run() File "/usr/lib/python2.5/site-packages/twisted/internet/base.py", line 1128, in run self.mainLoop() File "/usr/lib/python2.5/site-packages/twisted/internet/base.py", line 1137, in mainLoop self.runUntilCurrent() --- --- File "/usr/lib/python2.5/site-packages/twisted/internet/base.py", line 729, in runUntilCurrent f(*a, **kw) File "/usr/lib/python2.5/site-packages/twisted/internet/defer.py", line 269, in errback self._startRunCallbacks(fail) File "/usr/lib/python2.5/site-packages/twisted/internet/defer.py", line 298, in _startRunCallbacks raise AlreadyCalledError twisted.internet.defer.AlreadyCalledError: E....................................................................................................................................................................ERROR: An unexpected error occurred while tokenizing input The following traceback may be corrupted or invalid The error message is: ('EOF in multi-line statement', (288, 0)) ERROR: An unexpected error occurred while tokenizing input The following traceback may be corrupted or invalid The error message is: ('EOF in multi-line statement', (211, 0)) ERROR: An unexpected error occurred while tokenizing input The following traceback may be corrupted or invalid The error message is: ('EOF in multi-line statement', (182, 0)) ......................ERROR: An unexpected error occurred while tokenizing input The following traceback may be corrupted or invalid The error message is: ('EOF in multi-line statement', (288, 0)) ERROR: An unexpected error occurred while tokenizing input The following traceback may be corrupted or invalid The error message is: ('EOF in multi-line statement', (211, 0)) ERROR: An unexpected error occurred while tokenizing input The following traceback may be corrupted or invalid The error message is: ('EOF in multi-line statement', (182, 0)) ...............................S..S.......SS..SSS.........sh: ipython: not found F....SSS.... ====================================================================== ERROR: testExecuteFailuresEngineService_0 ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/twisted/internet/base.py", line 729, in runUntilCurrent f(*a, **kw) File "/usr/lib/python2.5/site-packages/twisted/internet/defer.py", line 269, in errback self._startRunCallbacks(fail) File "/usr/lib/python2.5/site-packages/twisted/internet/defer.py", line 298, in _startRunCallbacks raise AlreadyCalledError AlreadyCalledError ====================================================================== FAIL: Test that object's __del__ methods are called on exit. ---------------------------------------------------------------------- Traceback (most recent call last): File "/var/lib/python-support/python2.5/nose/case.py", line 182, in runTest self.test(*self.arg) File "/home/ondrej/repos/ipython/IPython/tests/test_magic.py", line 88, in test_obj_del nt.assert_equals(out,'object A deleted') AssertionError: '' != 'object A deleted' >> raise self.failureException, \ (None or '%r != %r' % ('', 'object A deleted')) ---------------------------------------------------------------------- Ran 511 tests in 51.917s FAILED (SKIP=11, errors=1, failures=1) Deleting object: second_pass object A deleted deleting object... So my plan for tonight is: * fix tests in python2.5 * fix tests in python2.6 * fix my problem Ondrej From ellisonbg.net at gmail.com Wed Mar 18 23:52:52 2009 From: ellisonbg.net at gmail.com (Brian Granger) Date: Wed, 18 Mar 2009 20:52:52 -0700 Subject: [IPython-dev] Test failures and errors in trunk In-Reply-To: <6ce0ac130903182041w58684ad7ufe817a6844a453b2@mail.gmail.com> References: <6ce0ac130903181324j9cb1352veae1fe9f2a8b85cd@mail.gmail.com> <6ce0ac130903181635l75db524bme9c0baf8f4d39ab9@mail.gmail.com> <6ce0ac130903181920n27cd8aaase53e6e8c20adba6b@mail.gmail.com> <6ce0ac130903182041w58684ad7ufe817a6844a453b2@mail.gmail.com> Message-ID: <6ce0ac130903182052u47e62940xf9b3d5324a1b0241@mail.gmail.com> Fernando, Ondrej has just tried iptest on python 2.5 and gets the following semi-interesting result: http://paste.debian.net/30957/ If you get a chance, at least have a look at the message, as they might show us what is going on with this issue. Brian On Wed, Mar 18, 2009 at 8:41 PM, Brian Granger wrote: > OK, I found the commit that introduced the error. ?It first appears in > this revision: > > http://bazaar.launchpad.net/~fdo.perez/ipython/trunk-dev/revision/1155 > > The previous revision passes all tests just fine. ?You can get the > problematic revision by doing: > > bzr branch lp:~fdo.perez/ipython/trunk-dev -r 1155 > > It is worth checking that the previous revision is OK on your system as well. > > But, this one could be a challenge to figure out. ?In this revision, > you were only touching non-twisted parts of the test suite - basically > getting the ipdoctest stuff working right. ?Hopefully between the two > of us we can figure this out. ?I know the Twisted stuff, you know the > ip testing stuff. > > I am going to switch gears to help Ondrej with some things. ?If you > end up looking at this and have questions on the Twisted side of > things let me know. > >> Wow, that would be fantastic. ?I didn't think I was touching anything >> twisted-related, and since I'd seen this problem before, I thought it >> was the same thing. > > Looks pretty similar, but I think it is somewhat different. ?I have > never seen the double call to reactor.run() before. > > But > >> I'll know better from now on never to ignore a twisted problem, even >> if it's a bit of a Heisenbug. >> >> In the meantime, I'm working up some utilities to ease the virtualenv >> workflow, based on some of the scripts that are out there from Doug >> Hellman and Chris Burns. ?This will let us do tests in 'naked python' >> virtualenvs more easily, so we can really have a test suite that at >> least runs gracefully (skips but no errors) on nothing but >> python(stdlib)+nose+ipython. > > Yes, this seems like it is pretty important. ?I ended up doing that > when we did the last release and it really helped. > > Cheers, > > Brian > >> Stay tuned. >> >> f >> > From ondrej at certik.cz Thu Mar 19 01:21:55 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 18 Mar 2009 22:21:55 -0700 Subject: [IPython-dev] Test failures and errors in trunk In-Reply-To: <6ce0ac130903182052u47e62940xf9b3d5324a1b0241@mail.gmail.com> References: <6ce0ac130903181324j9cb1352veae1fe9f2a8b85cd@mail.gmail.com> <6ce0ac130903181635l75db524bme9c0baf8f4d39ab9@mail.gmail.com> <6ce0ac130903181920n27cd8aaase53e6e8c20adba6b@mail.gmail.com> <6ce0ac130903182041w58684ad7ufe817a6844a453b2@mail.gmail.com> <6ce0ac130903182052u47e62940xf9b3d5324a1b0241@mail.gmail.com> Message-ID: <85b5c3130903182221v92f293cs81d9032c7abd3738@mail.gmail.com> On Wed, Mar 18, 2009 at 8:52 PM, Brian Granger wrote: > Fernando, > > Ondrej has just tried iptest on python 2.5 and gets the following > semi-interesting result: > > http://paste.debian.net/30957/ > > If you get a chance, at least have a look at the message, as they > might show us what is going on with this issue. Ok, so if I use the revision: bzr branch lp:ipython -r 1158 ipython-r1158 Then all tests pass even with python2.6 and in the trunk itself: $ scripts/iptest /home/ondrej/repos/ipython/ipython-r1158/IPython/Magic.py:38: DeprecationWarning: the sets module is deprecated from sets import Set /var/lib/python-support/python2.6/nose/plugins/manager.py:386: UserWarning: Module nose was already imported from /var/lib/python-support/python2.6/nose/__init__.py, but /var/lib/python-support/python2.6 is being added to sys.path import pkg_resources /usr/lib/python2.6/dist-packages/twisted/python/filepath.py:12: DeprecationWarning: the sha module is deprecated; use the hashlib module instead import sha S...S....................................................................................................................................................................................................................................................................................../home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ./home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ./home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ./home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ./home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ./home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ./home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ..../home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message .........../home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ../home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ............../home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ................./home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ...../home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ........../home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ../home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ........................./home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message .............../home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ......./home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message ../home/ondrej/repos/ipython/ipython-r1158/IPython/kernel/error.py:110: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message .......S..S..SS..SSS........ ---------------------------------------------------------------------- Ran 431 tests in 22.982s OK (SKIP=9) The later revisions are broken for me. Ondrej From ondrej at certik.cz Thu Mar 19 03:52:15 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Thu, 19 Mar 2009 00:52:15 -0700 Subject: [IPython-dev] parallel testing of sympy with ipython Message-ID: <85b5c3130903190052y5f7435d0gf00da2a7daee23dc@mail.gmail.com> Hi, Brian helped me fix my problems with ipython and it then was super easy to create parallel testing for sympy. Currently in my branch here: http://github.com/certik/sympy/tree/test2 e.g. start the cluster: $ ipcluster local -n 8 Install sympy: $ git clone git://github.com/certik/sympy.git $ cd sympy $ git checkout -b test2 origin/test2 $ python setup.py install --home=~/lib (make sure ~/lib/lib/python is in your PYTHONPATH) test in parallel: $ python t.py number of tests: 1376 distributing jobs collecting results processor: 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . exceptions ________________________________________________________________________________ File "/home/ondrej/lib/lib/python/sympy/mpmath/tests/test_basic_ops.py", line 115, in test_complex_misc assert mpc(2+1e-15j).ae(2) File "/home/ondrej/lib/lib/python/sympy/mpmath/tests/test_basic_ops.py", line 115, in test_complex_misc assert mpc(2+1e-15j).ae(2) processor: 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . exceptions processor: 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . exceptions processor: 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F . . . . . exceptions ________________________________________________________________________________ File "/home/ondrej/lib/lib/python/sympy/utilities/tests/test_lambdify.py", line 97, in test_mpmath_lambda assert -prec < f(mpmath.mpf("0.2")) - sin02 < prec File "/home/ondrej/lib/lib/python/sympy/utilities/tests/test_lambdify.py", line 97, in test_mpmath_lambda assert -prec < f(mpmath.mpf("0.2")) - sin02 < prec processor: 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . exceptions processor: 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . exceptions processor: 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . exceptions processor: 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . exceptions All tests pass, except 4 mpmath tests, but I suspect it could be some bug in mpmath. Compare the timing: Sequential: $ time bin/test sympy [...] real 1m4.295s user 1m3.872s sys 0m0.332s Parallel: $ time python t.py [..] real 0m10.206s user 0m0.308s sys 0m0.064s And this doesn't do any load balancing, so in the late phase (last 2s) it is basically just computing on one processor only. I am looking forward when I run this on our UNR cluster, I guess all sympy tests could be done in just couple seconds. Now some thoughts about load balancing: It starts here: http://github.com/certik/sympy/blob/55de47b6f0a7bee01249fc24c03e5567695c4569/t.py#L50 I create a tasks dictionary, which contains processors ids as keys and sympy tests numbers as values (those tests should be executed on that processor), where one test is just one testfunction in a file (e.g. there is almost 1400 tests). Then I distribute the tests on the processors here using nonblocking mec.execute: http://github.com/certik/sympy/blob/55de47b6f0a7bee01249fc24c03e5567695c4569/t.py#L64 And finally I collect the results: http://github.com/certik/sympy/blob/55de47b6f0a7bee01249fc24c03e5567695c4569/t.py#L69 and report the results to the user. This should of course be integrated into our testing framework, so that it looks exactly the same. The goal should be that one would just use bin/test or bin/test -j8 but otherwise the output doesn't change, only it will be 8x faster. Ondrej From ondrej at certik.cz Thu Mar 19 03:56:44 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Thu, 19 Mar 2009 00:56:44 -0700 Subject: [IPython-dev] parallel testing of sympy with ipython In-Reply-To: <85b5c3130903190052y5f7435d0gf00da2a7daee23dc@mail.gmail.com> References: <85b5c3130903190052y5f7435d0gf00da2a7daee23dc@mail.gmail.com> Message-ID: <85b5c3130903190056m7b16148fg1f64c880171719c7@mail.gmail.com> > Now some thoughts about load balancing: [...] > And finally I collect the results: > > http://github.com/certik/sympy/blob/55de47b6f0a7bee01249fc24c03e5567695c4569/t.py#L69 > > and report the results to the user. This should of course be I forgot to ask the question: how could this be made more efficient using some native ipython parrallel tools? Ondrej From ondrej at certik.cz Thu Mar 19 04:09:36 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Thu, 19 Mar 2009 01:09:36 -0700 Subject: [IPython-dev] parallel testing of sympy with ipython In-Reply-To: <85b5c3130903190056m7b16148fg1f64c880171719c7@mail.gmail.com> References: <85b5c3130903190052y5f7435d0gf00da2a7daee23dc@mail.gmail.com> <85b5c3130903190056m7b16148fg1f64c880171719c7@mail.gmail.com> Message-ID: <85b5c3130903190109r391ad701u7fcaadbe6aea298@mail.gmail.com> On Thu, Mar 19, 2009 at 12:56 AM, Ondrej Certik wrote: [...] > I forgot to ask the question: > > how could this be made more efficient using some native ipython parrallel tools? And last question: Currently all the engines must be able to import sympy and I noticed it is not even enough to install sympy after each change, but the ipcluster must be restarted as well, otherwise the old version of sympy stays in the memory and it fails. What is the best way to transfer the library (sympy) and tests to the engines automatically to make the edit/run/debug cycle easier? And what about a big cluster --- wouldn't it be handy to just tell ipython: take this library and make it available on the engines, without me requiring to install it manually and then restarting the ipcluster? Ondrej From fperez.net at gmail.com Thu Mar 19 04:17:54 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 19 Mar 2009 01:17:54 -0700 Subject: [IPython-dev] Test failures and errors in trunk In-Reply-To: <6ce0ac130903182041w58684ad7ufe817a6844a453b2@mail.gmail.com> References: <6ce0ac130903181324j9cb1352veae1fe9f2a8b85cd@mail.gmail.com> <6ce0ac130903181635l75db524bme9c0baf8f4d39ab9@mail.gmail.com> <6ce0ac130903181920n27cd8aaase53e6e8c20adba6b@mail.gmail.com> <6ce0ac130903182041w58684ad7ufe817a6844a453b2@mail.gmail.com> Message-ID: Howdy, On Wed, Mar 18, 2009 at 8:41 PM, Brian Granger wrote: > OK, I found the commit that introduced the error. ?It first appears in > this revision: > > http://bazaar.launchpad.net/~fdo.perez/ipython/trunk-dev/revision/1155 > > The previous revision passes all tests just fine. ?You can get the > problematic revision by doing: > > bzr branch lp:~fdo.perez/ipython/trunk-dev -r 1155 > > It is worth checking that the previous revision is OK on your system as well. Awesome, many thanks for bisecting this! I just now finished setting up all the tools to be able to robustly create virtualenvs that do not see anything system-wide, yet run on the normal python. There's actually quite a few moving parts to all of it, so I think I'm going to document this thoroughly in our dev guide. I ended up basically combining these two approaches: http://www.doughellmann.com/articles/CompletelyDifferent-2008-05-virtualenvwrapper/index.html http://neuroimaging.scipy.org/site/doc/manual/html/devel/tools/virtualenv-tutor.html along with a custom sitecustomize.py, to carefully control sys.path and friends for the virtualenv in a subshell. This approach really isolates the virtualenv in a much more reliable way than what virtualenv's normal tools do by default. And now I can easily reproduce Jorgen's problem: maqroll[~]> workon ipython *** Starting new shell for virtualenv *** (ipython)maqroll[~]> iptest [...] ---------------------------------------------------------------------- Ran 186 tests in 1.051s FAILED (SKIP=21, errors=48, failures=1) Deleting object: second_pass object A deleted deleting object... with all those import errors he was seeing. The bad news is that I'm exhausted and behind on a few work deadlines, so I won't be able to dive into that nasty twisted bug nor the one John reported this morning quite now. Sorry about that. I think in order of priorities, we should: 1. Clean up the test suite so that in an empty environment, it runs to completion (though obviously skipping things that may test twisted, numpy or anything else not part of the stdlib). I think this is the first priority, so that everyone can actually run the test suite, even if the Twisted bug is present. 2. Attack the Twisted Heisenbug. 3. The memory leak/lost references bug (and clarify whether it could be a core Python bug). Since I may be awol for a few days, if anyone needs to know how to set up the 'naked' virtualenv, let me know and I can post a quick draft. Cheers, f From ondrej at certik.cz Thu Mar 19 05:21:53 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Thu, 19 Mar 2009 02:21:53 -0700 Subject: [IPython-dev] Test failures and errors in trunk In-Reply-To: References: <6ce0ac130903181324j9cb1352veae1fe9f2a8b85cd@mail.gmail.com> <6ce0ac130903181635l75db524bme9c0baf8f4d39ab9@mail.gmail.com> <6ce0ac130903181920n27cd8aaase53e6e8c20adba6b@mail.gmail.com> <6ce0ac130903182041w58684ad7ufe817a6844a453b2@mail.gmail.com> Message-ID: <85b5c3130903190221i5b8fea78u806a95a91c99973e@mail.gmail.com> On Thu, Mar 19, 2009 at 1:17 AM, Fernando Perez wrote: > Howdy, > > This approach really isolates the virtualenv in a much more reliable > way than what virtualenv's normal tools do by default. ?And now I can > easily reproduce Jorgen's problem: Great job Fernando! I am also behind couple deadlines as usual, but ipython parallel works and tests in parallel just fine, so I am very happy I managed to get the job done. Many thanks to Brian. I decided to have a beer for that, even though it's 2:20pm. Ondrej From ellisonbg.net at gmail.com Sat Mar 21 00:26:05 2009 From: ellisonbg.net at gmail.com (Brian Granger) Date: Fri, 20 Mar 2009 21:26:05 -0700 Subject: [IPython-dev] bzr criss-cross merge?!?! Message-ID: <6ce0ac130903202126o6a534960p3c91274055967663@mail.gmail.com> Just when I was feeling like I have the hang of working with bzr, I encountered this... I am trying to merge the latest lp:ipython into my own branch. When I did the merge I got: brian-grangers-macbook-pro:bg-trunk-dev bgranger$ bzr merge ../ipython Warning: criss-cross merge encountered. See bzr help criss-cross. I read more about this and have some vague idea of what it means. But I don't recall doing anything funny and I have no idea how to go about finding the source of this. Does anyone know if I can just ignore this, resolve my conflicts and carry on? Any insight from the bzr masters among us would be greatly appreciated. Cheers, Brian From ellisonbg.net at gmail.com Sat Mar 21 00:35:03 2009 From: ellisonbg.net at gmail.com (Brian Granger) Date: Fri, 20 Mar 2009 21:35:03 -0700 Subject: [IPython-dev] Update on status of trunk (things to do for 0.10) Message-ID: <6ce0ac130903202135t67ebfbdbidc5169fa6db54df0@mail.gmail.com> I wanted to let everyone know that I have temporarily disabled the ipdoctest nose plugin in lp:ipython (trunk). The loading of this plugin is causing the odd Twisted errors/failures we have been seeing in iptest. I don't know the code in ipdoctest.py at all, so Fernando and I will probably have to go at this together in a few days. With this minor change though, the test suite passes completely if the appropriate dependencies are installed. Here is a slightly modified version of Fernando's recent list of where we stand with trunk currently: 1. Clean up the test suite so that in an empty environment, it runs to completion (though obviously skipping things that may test twisted, numpy or anything else not part of the stdlib). 2. Debug the ipdoctest plugin and its unhappiness with some aspect of Twisted. 3. The memory leak/lost references bug (and clarify whether it could be a core Python bug). My only question (this is probably for Fernando) about (1) is this. I can go in and clean up by hand the import errors. But, do we want to come up with a more formal method of handling these types of errors in our test suite. My only hesitancy is about automating this is that we might end up skipping tests that are actually failing. Thoughts? Cheers, Brian From ellisonbg.net at gmail.com Sat Mar 21 00:48:47 2009 From: ellisonbg.net at gmail.com (Brian Granger) Date: Fri, 20 Mar 2009 21:48:47 -0700 Subject: [IPython-dev] Creation of .ipython directory Message-ID: <6ce0ac130903202148g3b7e16co3f618a5b7c58376b@mail.gmail.com> Hi, Previously, the only IPython command line program that used the .ipython directory was "ipython" itself. Now, however, other command line programs (ipcluster, ipcontroller, ipengine) also use the .ipython directory. A user just found a bug where they ran "ipcluster" *before* "ipython" was ever run on their system. Thus, "ipcluster" couldn't find the .ipython directory and an exception was raised. So, I have a question: where in the IPython code base is the code that looks to see if the .ipython directory exists and then creates it if not? Should this same code be called by all IPython processes that might need the .ipython directory? Cheers, Brian From ondrej at certik.cz Sat Mar 21 23:17:50 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Sat, 21 Mar 2009 20:17:50 -0700 Subject: [IPython-dev] Update on status of trunk (things to do for 0.10) In-Reply-To: <6ce0ac130903202135t67ebfbdbidc5169fa6db54df0@mail.gmail.com> References: <6ce0ac130903202135t67ebfbdbidc5169fa6db54df0@mail.gmail.com> Message-ID: <85b5c3130903212017v32c9a18bgdb8edd5547b2141f@mail.gmail.com> On Fri, Mar 20, 2009 at 9:35 PM, Brian Granger wrote: > I wanted to let everyone know that I have temporarily disabled the > ipdoctest nose plugin in lp:ipython (trunk). ?The loading of this All tests pass in python2.6 in Ubuntu Jaunty. Thanks, Ondrej From ondrej at certik.cz Sun Mar 22 00:10:28 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Sat, 21 Mar 2009 21:10:28 -0700 Subject: [IPython-dev] running ipython inside SPD Message-ID: <85b5c3130903212110i4b47b8c2xd55a24fa584a1d22@mail.gmail.com> Hi, I started to work on the Source Python Distribution and here is howto: http://code.google.com/p/qsnake/wiki/SPD I installed it on our cluster and everything seems to work, but ipcluster is showing me this exception: $ ./spd -sh $ ipcluster local -n 2 2009-03-21 21:07:27-0700 [-] Log opened. 2009-03-21 21:07:27-0700 [-] Process ['ipcontroller', '--logfile=/home/ocertik/.ipython/log/ipcontroller'] has started with pid=31949 2009-03-21 21:07:27-0700 [-] 'Traceback (most recent call last):\n' 2009-03-21 21:07:27-0700 [-] ' File "/home/ocertik/spd/spd-0.3/local/bin/ipcontroller", line 18, in \n' 2009-03-21 21:07:27-0700 [-] ' ' 2009-03-21 21:07:27-0700 [-] 'from IPython.kernel.scripts import ipcontroller\n' 2009-03-21 21:07:27-0700 [-] ' File "/home/ocertik/spd/spd-0.3/local/lib/python2.5/site-packages/IPython/kernel/scripts/ipcontroller.py", line 40, in \n from IPython.kernel.config import config_manager as kernel_config_manager\n' 2009-03-21 21:07:27-0700 [-] ' File "/home/ocertik/spd/spd-0.3/local/lib/python2.5/site-packages/IPython/kernel/config/__init__.py", line 26, in \n security_dir = get_security_dir()\n' 2009-03-21 21:07:27-0700 [-] ' File "/home/ocertik/spd/spd-0.3/local/lib/python2.5/site-packages/IPython/genutils.py", line 1006, in get_security_dir\n os.mkdir(security_dir, 0700)\nOSError: [Errno 2] No such file or directory: \'/home/ocertik/.ipython/security\'\n' 2009-03-21 21:07:27-0700 [-] Process ['ipcontroller', '--logfile=/home/ocertik/.ipython/log/ipcontroller'] has stopped with {'status': 256, 'signal': None, 'exit_code': 1} 2009-03-21 21:07:28-0700 [-] Process ['ipengine', '--logfile=/home/ocertik/.ipython/log/ipengine31949-'] has started with pid=31950 2009-03-21 21:07:28-0700 [-] Process ['ipengine', '--logfile=/home/ocertik/.ipython/log/ipengine31949-'] has started with pid=31951 2009-03-21 21:07:28-0700 [-] Engines started with pids: [31950, 31951] 2009-03-21 21:07:28-0700 [-] 'Traceback (most recent call last):\n' 2009-03-21 21:07:28-0700 [-] ' File "/home/ocertik/spd/spd-0.3/local/bin/ipengine", line 18, in \n from IPython.kernel.scripts import ipengine\n File "/home/ocertik/spd/spd-0.3/local/lib/python2.5/site-packages/IPython/kernel/scripts/ipengine.py", line 36, in \n from IPython.kernel.config import config_manager as kernel_config_manager\n' 2009-03-21 21:07:28-0700 [-] ' File "/home/ocertik/spd/spd-0.3/local/lib/python2.5/site-packages/IPython/kernel/config/__init__.py", line 26, in \n security_dir = get_security_dir()\n File "/home/ocertik/spd/spd-0.3/local/lib/python2.5/site-packages/IPython/genutils.py", line 1006, in get_security_dir\n' 2009-03-21 21:07:28-0700 [-] " os.mkdir(security_dir, 0700)\nOSError: [Errno 2] No such file or directory: '/home/ocertik/.ipython/security'\n" 2009-03-21 21:07:28-0700 [-] Process ['ipengine', '--logfile=/home/ocertik/.ipython/log/ipengine31949-'] has stopped with {'status': 256, 'signal': None, 'exit_code': 1} 2009-03-21 21:07:28-0700 [-] 'Traceback (most recent call last):\n' 2009-03-21 21:07:28-0700 [-] ' File "/home/ocertik/spd/spd-0.3/local/bin/ipengine", line 18, in \n from IPython.kernel.scripts import ipengine\n File "/home/ocertik/spd/spd-0.3/local/lib/python2.5/site-packages/IPython/kernel/scripts/ipengine.py", line 36, in \n' 2009-03-21 21:07:28-0700 [-] ' from IPython.kernel.config import config_manager as kernel_config_manager\n File "/home/ocertik/spd/spd-0.3/local/lib/python2.5/site-packages/IPython/kernel/config/__init__.py", line 26, in \n security_dir = get_security_dir()\n' 2009-03-21 21:07:28-0700 [-] ' File "/home/ocertik/spd/spd-0.3/local/lib/python2.5/site-packages/IPython/genutils.py", line 1006, in get_security_dir\n' 2009-03-21 21:07:28-0700 [-] " os.mkdir(security_dir, 0700)\nOSError: [Errno 2] No such file or directory: '/home/ocertik/.ipython/security'\n" 2009-03-21 21:07:28-0700 [-] Process ['ipengine', '--logfile=/home/ocertik/.ipython/log/ipengine31949-'] has stopped with {'status': 256, 'signal': None, 'exit_code': 1} I am investigating it now. Is this a bug? Ondrej From ondrej at certik.cz Sun Mar 22 00:12:06 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Sat, 21 Mar 2009 21:12:06 -0700 Subject: [IPython-dev] running ipython inside SPD In-Reply-To: <85b5c3130903212110i4b47b8c2xd55a24fa584a1d22@mail.gmail.com> References: <85b5c3130903212110i4b47b8c2xd55a24fa584a1d22@mail.gmail.com> Message-ID: <85b5c3130903212112p52ce240dm72b3f5a40511a5c1@mail.gmail.com> > I am investigating it now. Is this a bug? And a fix: mkdir ~/.ipython shouldn't ipcluster do this automatically? Ondrej From ondrej at certik.cz Sun Mar 22 00:24:48 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Sat, 21 Mar 2009 21:24:48 -0700 Subject: [IPython-dev] running ipython inside SPD In-Reply-To: <85b5c3130903212112p52ce240dm72b3f5a40511a5c1@mail.gmail.com> References: <85b5c3130903212110i4b47b8c2xd55a24fa584a1d22@mail.gmail.com> <85b5c3130903212112p52ce240dm72b3f5a40511a5c1@mail.gmail.com> Message-ID: <85b5c3130903212124x37829aa2ye8ea924894341578@mail.gmail.com> On Sat, Mar 21, 2009 at 9:12 PM, Ondrej Certik wrote: >> I am investigating it now. Is this a bug? > > And a fix: > > mkdir ~/.ipython > > shouldn't ipcluster do this automatically? Another problem: ipcluster uses ~/.ipython/security by default (when run from ./spd -sh), but the ipython in SPD uses ~/.sage/ipython/security by default. So I get: In [2]: mec = client.MultiEngineClient() [...] ValueError: not a furl or a file containing a furl: /home/ocertik/.sage/ipython/security/ipcontroller-mec.furl Again, a quick fix: $ ln -s ~/.ipython/security/ ~/.sage/ipython/ Ondrej From ellisonbg.net at gmail.com Sun Mar 22 00:42:59 2009 From: ellisonbg.net at gmail.com (Brian Granger) Date: Sat, 21 Mar 2009 21:42:59 -0700 Subject: [IPython-dev] running ipython inside SPD In-Reply-To: <85b5c3130903212112p52ce240dm72b3f5a40511a5c1@mail.gmail.com> References: <85b5c3130903212110i4b47b8c2xd55a24fa584a1d22@mail.gmail.com> <85b5c3130903212112p52ce240dm72b3f5a40511a5c1@mail.gmail.com> Message-ID: <6ce0ac130903212142r45a6bd4bm5fb82749819c391c@mail.gmail.com> Yes, this is a known bug that we are working on fixing for the 0.10 release. Brian On Sat, Mar 21, 2009 at 9:12 PM, Ondrej Certik wrote: >> I am investigating it now. Is this a bug? > > And a fix: > > mkdir ~/.ipython > > shouldn't ipcluster do this automatically? > > Ondrej > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > From ellisonbg.net at gmail.com Sun Mar 22 00:45:03 2009 From: ellisonbg.net at gmail.com (Brian Granger) Date: Sat, 21 Mar 2009 21:45:03 -0700 Subject: [IPython-dev] running ipython inside SPD In-Reply-To: <85b5c3130903212124x37829aa2ye8ea924894341578@mail.gmail.com> References: <85b5c3130903212110i4b47b8c2xd55a24fa584a1d22@mail.gmail.com> <85b5c3130903212112p52ce240dm72b3f5a40511a5c1@mail.gmail.com> <85b5c3130903212124x37829aa2ye8ea924894341578@mail.gmail.com> Message-ID: <6ce0ac130903212145o25a3d610l9ac32af7f6db45de@mail.gmail.com> For this, you will have to set the IPYTHONDIR environment variable to point to you non ~/.ipython ipython directory. Let me know if this doesn't work. Brian On Sat, Mar 21, 2009 at 9:24 PM, Ondrej Certik wrote: > On Sat, Mar 21, 2009 at 9:12 PM, Ondrej Certik wrote: >>> I am investigating it now. Is this a bug? >> >> And a fix: >> >> mkdir ~/.ipython >> >> shouldn't ipcluster do this automatically? > > Another problem: ipcluster uses ~/.ipython/security by default (when > run from ./spd -sh), but the ipython in SPD uses > ~/.sage/ipython/security by default. So I get: > > In [2]: mec = client.MultiEngineClient() > [...] > ValueError: not a furl or a file containing a furl: > /home/ocertik/.sage/ipython/security/ipcontroller-mec.furl > > Again, a quick fix: > > $ ln -s ~/.ipython/security/ ~/.sage/ipython/ > > Ondrej > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > From ondrej at certik.cz Sun Mar 22 01:38:29 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Sat, 21 Mar 2009 22:38:29 -0700 Subject: [IPython-dev] ipython on the sun grid engine Message-ID: <85b5c3130903212238m5ec92a2cj4e1676a9b7fe6784@mail.gmail.com> Hi, so I made it run. 1. install SPD using this howto: http://code.google.com/p/qsnake/wiki/SPD 2. create ex9.sge (adjust the path to spd): $ cat ex9.sge #/bin/bash #$ -cwd #$ -pe Common.pe 40 NUM_WORKER=$(($NSLOTS - 1)) MASTER=$(head -n 1 $TMP/machines) WORKERS=$(tail -n $NUM_WORKER $TMP/machines) echo "My Master nodes:" echo $MASTER echo "My Worker nodes:" echo $WORKERS { ssh $MASTER '~/spd/spd-0.3/spd -python ~/spd/spd-0.3/local/bin/ipcontroller' & sleep 3 for x in $WORKERS; do ssh $x '~/spd/spd-0.3/spd -python ~/spd/spd-0.3/local/bin/ipengine' & done } > runlog.txt wait 3. submit the job: $ qsub ex9.sge 4. connect to the ipcontroller: $ ./spd ---------------------------------------------------------------------- | Source Python Distribution, Version 0.3, Release Date: 2009-03-21 | | Type notebook() for the GUI, and license() for information. | ---------------------------------------------------------------------- In [1]: from IPython.kernel import client In [2]: mec = client.MultiEngineClient() In [3]: mec.get_ids() Out[3]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37] In [8]: print mec.get_ids() [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37] In [9]: mec.activate() In [10]: mec.block=True In [11]: %px import numpy Parallel execution on engines: all Out[11]: [0] In [1]: import numpy [1] In [1]: import numpy [2] In [1]: import numpy [3] In [1]: import numpy [4] In [1]: import numpy [5] In [1]: import numpy [6] In [1]: import numpy [7] In [1]: import numpy [8] In [1]: import numpy [9] In [1]: import numpy [10] In [1]: import numpy [11] In [1]: import numpy [12] In [1]: import numpy [13] In [1]: import numpy [14] In [1]: import numpy [15] In [1]: import numpy [16] In [1]: import numpy [17] In [1]: import numpy [18] In [1]: import numpy [19] In [1]: import numpy [20] In [1]: import numpy [21] In [1]: import numpy [22] In [1]: import numpy [23] In [1]: import numpy [24] In [1]: import numpy [25] In [1]: import numpy [26] In [1]: import numpy [27] In [1]: import numpy [28] In [1]: import numpy [29] In [1]: import numpy [30] In [1]: import numpy [31] In [1]: import numpy [32] In [1]: import numpy [33] In [1]: import numpy [34] In [1]: import numpy [35] In [1]: import numpy [36] In [1]: import numpy [37] In [1]: import numpy In [12]: %px a = numpy.random.rand(2,2) Parallel execution on engines: all Out[12]: [0] In [2]: a = numpy.random.rand(2,2) [1] In [2]: a = numpy.random.rand(2,2) [2] In [2]: a = numpy.random.rand(2,2) [3] In [2]: a = numpy.random.rand(2,2) [4] In [2]: a = numpy.random.rand(2,2) [5] In [2]: a = numpy.random.rand(2,2) [6] In [2]: a = numpy.random.rand(2,2) [7] In [2]: a = numpy.random.rand(2,2) [8] In [2]: a = numpy.random.rand(2,2) [9] In [2]: a = numpy.random.rand(2,2) [10] In [2]: a = numpy.random.rand(2,2) [11] In [2]: a = numpy.random.rand(2,2) [12] In [2]: a = numpy.random.rand(2,2) [13] In [2]: a = numpy.random.rand(2,2) [14] In [2]: a = numpy.random.rand(2,2) [15] In [2]: a = numpy.random.rand(2,2) [16] In [2]: a = numpy.random.rand(2,2) [17] In [2]: a = numpy.random.rand(2,2) [18] In [2]: a = numpy.random.rand(2,2) [19] In [2]: a = numpy.random.rand(2,2) [20] In [2]: a = numpy.random.rand(2,2) [21] In [2]: a = numpy.random.rand(2,2) [22] In [2]: a = numpy.random.rand(2,2) [23] In [2]: a = numpy.random.rand(2,2) [24] In [2]: a = numpy.random.rand(2,2) [25] In [2]: a = numpy.random.rand(2,2) [26] In [2]: a = numpy.random.rand(2,2) [27] In [2]: a = numpy.random.rand(2,2) [28] In [2]: a = numpy.random.rand(2,2) [29] In [2]: a = numpy.random.rand(2,2) [30] In [2]: a = numpy.random.rand(2,2) [31] In [2]: a = numpy.random.rand(2,2) [32] In [2]: a = numpy.random.rand(2,2) [33] In [2]: a = numpy.random.rand(2,2) [34] In [2]: a = numpy.random.rand(2,2) [35] In [2]: a = numpy.random.rand(2,2) [36] In [2]: a = numpy.random.rand(2,2) [37] In [2]: a = numpy.random.rand(2,2) In [13]: %px print numpy.linalg.eigvals(a) Parallel execution on engines: all Out[13]: [0] In [3]: print numpy.linalg.eigvals(a) [0] Out[3]: [ 0.04392391 0.3245973 ] [1] In [3]: print numpy.linalg.eigvals(a) [1] Out[3]: [ 1.55163721 0.20437739] [2] In [3]: print numpy.linalg.eigvals(a) [2] Out[3]: [ 1.20045213 0.04814805] [3] In [3]: print numpy.linalg.eigvals(a) [3] Out[3]: [ 0.32003643 0.57111291] [4] In [3]: print numpy.linalg.eigvals(a) [4] Out[3]: [-0.19009279 0.99128112] [5] In [3]: print numpy.linalg.eigvals(a) [5] Out[3]: [ 1.03829332 0.36075953] [6] In [3]: print numpy.linalg.eigvals(a) [6] Out[3]: [-0.1517235 1.67516513] [7] In [3]: print numpy.linalg.eigvals(a) [7] Out[3]: [ 1.39960728 -0.28465163] [8] In [3]: print numpy.linalg.eigvals(a) [8] Out[3]: [ 1.17044538 0.36920544] [9] In [3]: print numpy.linalg.eigvals(a) [9] Out[3]: [ 1.29444925 0.03147179] [10] In [3]: print numpy.linalg.eigvals(a) [10] Out[3]: [ 1.06269256 0.00629928] [11] In [3]: print numpy.linalg.eigvals(a) [11] Out[3]: [ 0.1934025 0.04709656] [12] In [3]: print numpy.linalg.eigvals(a) [12] Out[3]: [-0.23157366 0.79206212] [13] In [3]: print numpy.linalg.eigvals(a) [13] Out[3]: [ 0.87736399 0.36763171] [14] In [3]: print numpy.linalg.eigvals(a) [14] Out[3]: [ 0.75825399 0.42571366] [15] In [3]: print numpy.linalg.eigvals(a) [15] Out[3]: [ 0.83819298 -0.33422046] [16] In [3]: print numpy.linalg.eigvals(a) [16] Out[3]: [ 0.05119763 1.25161191] [17] In [3]: print numpy.linalg.eigvals(a) [17] Out[3]: [ 0.6356765 -0.18668048] [18] In [3]: print numpy.linalg.eigvals(a) [18] Out[3]: [ 0.05417072 0.64346778] [19] In [3]: print numpy.linalg.eigvals(a) [19] Out[3]: [ 0.94357665 0.08495554] [20] In [3]: print numpy.linalg.eigvals(a) [20] Out[3]: [-0.20071214 1.44548662] [21] In [3]: print numpy.linalg.eigvals(a) [21] Out[3]: [ 1.20371805 0.17458575] [22] In [3]: print numpy.linalg.eigvals(a) [22] Out[3]: [ 1.47189454 -0.25060313] [23] In [3]: print numpy.linalg.eigvals(a) [23] Out[3]: [ 0.62675299 0.00833881] [24] In [3]: print numpy.linalg.eigvals(a) [24] Out[3]: [-0.61704795 0.86785185] [25] In [3]: print numpy.linalg.eigvals(a) [25] Out[3]: [ 0.51454737 1.09251881] [26] In [3]: print numpy.linalg.eigvals(a) [26] Out[3]: [ 1.0025157 0.37926302] [27] In [3]: print numpy.linalg.eigvals(a) [27] Out[3]: [ 0.07562955 0.69529292] [28] In [3]: print numpy.linalg.eigvals(a) [28] Out[3]: [ 0.70206288 0.09943419] [29] In [3]: print numpy.linalg.eigvals(a) [29] Out[3]: [-0.41140114 0.76592533] [30] In [3]: print numpy.linalg.eigvals(a) [30] Out[3]: [-0.41483813 0.66156836] [31] In [3]: print numpy.linalg.eigvals(a) [31] Out[3]: [ 0.06939694 0.82218779] [32] In [3]: print numpy.linalg.eigvals(a) [32] Out[3]: [-0.27583508 0.96545541] [33] In [3]: print numpy.linalg.eigvals(a) [33] Out[3]: [-0.07448614 0.95500699] [34] In [3]: print numpy.linalg.eigvals(a) [34] Out[3]: [ 0.57380518 1.01918719] [35] In [3]: print numpy.linalg.eigvals(a) [35] Out[3]: [ 0.90397042 0.06531227] [36] In [3]: print numpy.linalg.eigvals(a) [36] Out[3]: [ 0.89331523 0.58087954] [37] In [3]: print numpy.linalg.eigvals(a) [37] Out[3]: [ 0.69227186 1.25013097] Any idea, why the output [3] looks so ugly? Is it some pretty printing relict from Sage? the ssh command actually failed to connect to 1 node, as seen from the ex9.sge.e105489: Permission denied, please try again. Permission denied, please try again. Permission denied (publickey,gssapi-with-mic,password). so only 38 engines were registered (e.g. 1 node is for ipcontroller, 38 for engines and 1 failed = 40 nodes). This rocks! Ondrej From ellisonbg.net at gmail.com Sun Mar 22 02:06:54 2009 From: ellisonbg.net at gmail.com (Brian Granger) Date: Sat, 21 Mar 2009 23:06:54 -0700 Subject: [IPython-dev] ipython on the sun grid engine In-Reply-To: <85b5c3130903212238m5ec92a2cj4e1676a9b7fe6784@mail.gmail.com> References: <85b5c3130903212238m5ec92a2cj4e1676a9b7fe6784@mail.gmail.com> Message-ID: <6ce0ac130903212306j404df7c0pdca41fc8d7ab4cb9@mail.gmail.com> Ondrej, Very nice. Can you just run the controller on the head node (that you use to submit the job)? It is odd that one of the ssh trys failed. Usually we start the controller on the head node and then run the engines using the queue. But on some systems, this is bad I think. As for the ugly output [3], it is definitely a pretty print thing, but I don't know where it is coming from. It could be from sage (maybe they have a custom ipython config file). But, this look great. Were you able to connect to the running cluster using a client on your laptop, or was the client on the head node? Great! Brian On Sat, Mar 21, 2009 at 10:38 PM, Ondrej Certik wrote: > Hi, > > so I made it run. > > 1. install SPD using this howto: > > http://code.google.com/p/qsnake/wiki/SPD > > 2. create ex9.sge (adjust the path to spd): > > $ cat ex9.sge > #/bin/bash > #$ -cwd > #$ -pe Common.pe 40 > NUM_WORKER=$(($NSLOTS - 1)) > MASTER=$(head -n 1 $TMP/machines) > WORKERS=$(tail -n $NUM_WORKER $TMP/machines) > echo "My Master nodes:" > echo $MASTER > echo "My Worker nodes:" > echo $WORKERS > { > ssh $MASTER '~/spd/spd-0.3/spd -python ~/spd/spd-0.3/local/bin/ipcontroller' & > sleep 3 > for x in $WORKERS; do > ? ?ssh $x '~/spd/spd-0.3/spd -python ~/spd/spd-0.3/local/bin/ipengine' & > done > } > runlog.txt > wait > > > 3. submit the job: > > $ qsub ex9.sge > > 4. connect to the ipcontroller: > > $ ./spd > ---------------------------------------------------------------------- > | Source Python Distribution, Version 0.3, Release Date: 2009-03-21 ?| > | Type notebook() for the GUI, and license() for information. ? ? ? ?| > ---------------------------------------------------------------------- > In [1]: from IPython.kernel import client > In [2]: mec = client.MultiEngineClient() > In [3]: mec.get_ids() > Out[3]: > [0, > ?1, > ?2, > ?3, > ?4, > ?5, > ?6, > ?7, > ?8, > ?9, > ?10, > ?11, > ?12, > ?13, > ?14, > ?15, > ?16, > ?17, > ?18, > ?19, > ?20, > ?21, > ?22, > ?23, > ?24, > ?25, > ?26, > ?27, > ?28, > ?29, > ?30, > ?31, > ?32, > ?33, > ?34, > ?35, > ?36, > ?37] > In [8]: print mec.get_ids() > [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, > 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, > 37] > In [9]: mec.activate() > In [10]: mec.block=True > In [11]: %px import numpy > Parallel execution on engines: all > Out[11]: > > [0] In [1]: import numpy > [1] In [1]: import numpy > [2] In [1]: import numpy > [3] In [1]: import numpy > [4] In [1]: import numpy > [5] In [1]: import numpy > [6] In [1]: import numpy > [7] In [1]: import numpy > [8] In [1]: import numpy > [9] In [1]: import numpy > [10] In [1]: import numpy > [11] In [1]: import numpy > [12] In [1]: import numpy > [13] In [1]: import numpy > [14] In [1]: import numpy > [15] In [1]: import numpy > [16] In [1]: import numpy > [17] In [1]: import numpy > [18] In [1]: import numpy > [19] In [1]: import numpy > [20] In [1]: import numpy > [21] In [1]: import numpy > [22] In [1]: import numpy > [23] In [1]: import numpy > [24] In [1]: import numpy > [25] In [1]: import numpy > [26] In [1]: import numpy > [27] In [1]: import numpy > [28] In [1]: import numpy > [29] In [1]: import numpy > [30] In [1]: import numpy > [31] In [1]: import numpy > [32] In [1]: import numpy > [33] In [1]: import numpy > [34] In [1]: import numpy > [35] In [1]: import numpy > [36] In [1]: import numpy > [37] In [1]: import numpy > > In [12]: %px a = numpy.random.rand(2,2) > Parallel execution on engines: all > Out[12]: > > [0] In [2]: a = numpy.random.rand(2,2) > [1] In [2]: a = numpy.random.rand(2,2) > [2] In [2]: a = numpy.random.rand(2,2) > [3] In [2]: a = numpy.random.rand(2,2) > [4] In [2]: a = numpy.random.rand(2,2) > [5] In [2]: a = numpy.random.rand(2,2) > [6] In [2]: a = numpy.random.rand(2,2) > [7] In [2]: a = numpy.random.rand(2,2) > [8] In [2]: a = numpy.random.rand(2,2) > [9] In [2]: a = numpy.random.rand(2,2) > [10] In [2]: a = numpy.random.rand(2,2) > [11] In [2]: a = numpy.random.rand(2,2) > [12] In [2]: a = numpy.random.rand(2,2) > [13] In [2]: a = numpy.random.rand(2,2) > [14] In [2]: a = numpy.random.rand(2,2) > [15] In [2]: a = numpy.random.rand(2,2) > [16] In [2]: a = numpy.random.rand(2,2) > [17] In [2]: a = numpy.random.rand(2,2) > [18] In [2]: a = numpy.random.rand(2,2) > [19] In [2]: a = numpy.random.rand(2,2) > [20] In [2]: a = numpy.random.rand(2,2) > [21] In [2]: a = numpy.random.rand(2,2) > [22] In [2]: a = numpy.random.rand(2,2) > [23] In [2]: a = numpy.random.rand(2,2) > [24] In [2]: a = numpy.random.rand(2,2) > [25] In [2]: a = numpy.random.rand(2,2) > [26] In [2]: a = numpy.random.rand(2,2) > [27] In [2]: a = numpy.random.rand(2,2) > [28] In [2]: a = numpy.random.rand(2,2) > [29] In [2]: a = numpy.random.rand(2,2) > [30] In [2]: a = numpy.random.rand(2,2) > [31] In [2]: a = numpy.random.rand(2,2) > [32] In [2]: a = numpy.random.rand(2,2) > [33] In [2]: a = numpy.random.rand(2,2) > [34] In [2]: a = numpy.random.rand(2,2) > [35] In [2]: a = numpy.random.rand(2,2) > [36] In [2]: a = numpy.random.rand(2,2) > [37] In [2]: a = numpy.random.rand(2,2) > > In [13]: %px print numpy.linalg.eigvals(a) > Parallel execution on engines: all > Out[13]: > > [0] In [3]: print numpy.linalg.eigvals(a) > [0] Out[3]: [ 0.04392391 ?0.3245973 ] > > [1] In [3]: print numpy.linalg.eigvals(a) > [1] Out[3]: [ 1.55163721 ?0.20437739] > > [2] In [3]: print numpy.linalg.eigvals(a) > [2] Out[3]: [ 1.20045213 ?0.04814805] > > [3] In [3]: print numpy.linalg.eigvals(a) > [3] Out[3]: [ 0.32003643 ?0.57111291] > > [4] In [3]: print numpy.linalg.eigvals(a) > [4] Out[3]: [-0.19009279 ?0.99128112] > > [5] In [3]: print numpy.linalg.eigvals(a) > [5] Out[3]: [ 1.03829332 ?0.36075953] > > [6] In [3]: print numpy.linalg.eigvals(a) > [6] Out[3]: [-0.1517235 ? 1.67516513] > > [7] In [3]: print numpy.linalg.eigvals(a) > [7] Out[3]: [ 1.39960728 -0.28465163] > > [8] In [3]: print numpy.linalg.eigvals(a) > [8] Out[3]: [ 1.17044538 ?0.36920544] > > [9] In [3]: print numpy.linalg.eigvals(a) > [9] Out[3]: [ 1.29444925 ?0.03147179] > > [10] In [3]: print numpy.linalg.eigvals(a) > [10] Out[3]: [ 1.06269256 ?0.00629928] > > [11] In [3]: print numpy.linalg.eigvals(a) > [11] Out[3]: [ 0.1934025 ? 0.04709656] > > [12] In [3]: print numpy.linalg.eigvals(a) > [12] Out[3]: [-0.23157366 ?0.79206212] > > [13] In [3]: print numpy.linalg.eigvals(a) > [13] Out[3]: [ 0.87736399 ?0.36763171] > > [14] In [3]: print numpy.linalg.eigvals(a) > [14] Out[3]: [ 0.75825399 ?0.42571366] > > [15] In [3]: print numpy.linalg.eigvals(a) > [15] Out[3]: [ 0.83819298 -0.33422046] > > [16] In [3]: print numpy.linalg.eigvals(a) > [16] Out[3]: [ 0.05119763 ?1.25161191] > > [17] In [3]: print numpy.linalg.eigvals(a) > [17] Out[3]: [ 0.6356765 ?-0.18668048] > > [18] In [3]: print numpy.linalg.eigvals(a) > [18] Out[3]: [ 0.05417072 ?0.64346778] > > [19] In [3]: print numpy.linalg.eigvals(a) > [19] Out[3]: [ 0.94357665 ?0.08495554] > > [20] In [3]: print numpy.linalg.eigvals(a) > [20] Out[3]: [-0.20071214 ?1.44548662] > > [21] In [3]: print numpy.linalg.eigvals(a) > [21] Out[3]: [ 1.20371805 ?0.17458575] > > [22] In [3]: print numpy.linalg.eigvals(a) > [22] Out[3]: [ 1.47189454 -0.25060313] > > [23] In [3]: print numpy.linalg.eigvals(a) > [23] Out[3]: [ 0.62675299 ?0.00833881] > > [24] In [3]: print numpy.linalg.eigvals(a) > [24] Out[3]: [-0.61704795 ?0.86785185] > > [25] In [3]: print numpy.linalg.eigvals(a) > [25] Out[3]: [ 0.51454737 ?1.09251881] > > [26] In [3]: print numpy.linalg.eigvals(a) > [26] Out[3]: [ 1.0025157 ? 0.37926302] > > [27] In [3]: print numpy.linalg.eigvals(a) > [27] Out[3]: [ 0.07562955 ?0.69529292] > > [28] In [3]: print numpy.linalg.eigvals(a) > [28] Out[3]: [ 0.70206288 ?0.09943419] > > [29] In [3]: print numpy.linalg.eigvals(a) > [29] Out[3]: [-0.41140114 ?0.76592533] > > [30] In [3]: print numpy.linalg.eigvals(a) > [30] Out[3]: [-0.41483813 ?0.66156836] > > [31] In [3]: print numpy.linalg.eigvals(a) > [31] Out[3]: [ 0.06939694 ?0.82218779] > > [32] In [3]: print numpy.linalg.eigvals(a) > [32] Out[3]: [-0.27583508 ?0.96545541] > > [33] In [3]: print numpy.linalg.eigvals(a) > [33] Out[3]: [-0.07448614 ?0.95500699] > > [34] In [3]: print numpy.linalg.eigvals(a) > [34] Out[3]: [ 0.57380518 ?1.01918719] > > [35] In [3]: print numpy.linalg.eigvals(a) > [35] Out[3]: [ 0.90397042 ?0.06531227] > > [36] In [3]: print numpy.linalg.eigvals(a) > [36] Out[3]: [ 0.89331523 ?0.58087954] > > [37] In [3]: print numpy.linalg.eigvals(a) > [37] Out[3]: [ 0.69227186 ?1.25013097] > > > > > > > Any idea, why the output [3] looks so ugly? Is it some pretty printing > relict from Sage? > > the ssh command actually failed to connect to 1 node, as seen from the > ex9.sge.e105489: > > Permission denied, please try again. > Permission denied, please try again. > Permission denied (publickey,gssapi-with-mic,password). > > so only 38 engines were registered (e.g. 1 node is for ipcontroller, > 38 for engines and 1 failed = 40 nodes). > > This rocks! > > Ondrej > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > From ondrej at certik.cz Sun Mar 22 02:33:14 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Sat, 21 Mar 2009 23:33:14 -0700 Subject: [IPython-dev] ipython on the sun grid engine In-Reply-To: <6ce0ac130903212306j404df7c0pdca41fc8d7ab4cb9@mail.gmail.com> References: <85b5c3130903212238m5ec92a2cj4e1676a9b7fe6784@mail.gmail.com> <6ce0ac130903212306j404df7c0pdca41fc8d7ab4cb9@mail.gmail.com> Message-ID: <85b5c3130903212333s5f39751y443ebe1f5af3b0aa@mail.gmail.com> On Sat, Mar 21, 2009 at 11:06 PM, Brian Granger wrote: > Ondrej, > > Very nice. ?Can you just run the controller on the head node (that you > use to submit the job)? It doesn't work, the log is full of messages like: 2009-03-21 23:31:44-0700 [-] Main loop terminated. 2009-03-21 23:31:45-0700 [-] 'EngineConnector: engine registration failed:' 2009-03-21 23:31:45-0700 [-] Unhandled Error Traceback (most recent call last): Failure: twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused. > It is odd that one of the ssh trys failed. It didn't fail for the second time when I tried it. > Usually we start the controller on the head node and then run the > engines using the queue. ?But on some systems, this is bad I think. Why is it bad (besides that it doesn't work for me)? > > As for the ugly output [3], it is definitely a pretty print thing, but > I don't know where it is coming from. ?It could be from sage (maybe > they have a custom ipython config file). Right. > > But, this look great. ?Were you able to connect to the running cluster > using a client on your laptop, or was the client on the head node? The client was on the head node. When I tried to connect from my workstation, I did: scp hnode:.ipython/security/ipcontroller-mec.furl .ipython/security/ and then from ipython: In [1]: from IPython.kernel import client In [2]: mec = client.MultiEngineClient() [...] /usr/lib/python2.6/dist-packages/twisted/python/failure.pyc in raiseException(self) 324 information if available. 325 """ --> 326 raise self.type, self.value, self.tb 327 328 ConnectionRefusedError: Connection was refused by other side: 111: Connection refused. Ondrej From ondrej at certik.cz Sun Mar 22 04:19:53 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Sun, 22 Mar 2009 01:19:53 -0700 Subject: [IPython-dev] ipcluster pbs gives no attribute 'job_id' Message-ID: <85b5c3130903220119n1d13e53dv7cf96320583aade8@mail.gmail.com> Hi, I am trying to figure out how the pbs works, so I started on my laptop: $ ipcluster pbs -n 2 --pbs-script=pbs.template 2009-03-22 01:08:50-0700 [-] Log opened. 2009-03-22 01:08:50-0700 [-] Process ['ipcontroller', '--logfile=/home/ondra/.ipython/log/ipcontroller'] has started with pid=5404 and nothing more happened. So I wanted to stop with with CTRL-C, but I got: ^C2009-03-22 01:09:36-0700 [-] Stopping pbs cluster 2009-03-22 01:09:36-0700 [-] Unexpected error in main loop. 2009-03-22 01:09:36-0700 [-] Unhandled Error Traceback (most recent call last): File "/home/ondra/lib/bin/ipcluster", line 21, in ipcluster.main() File "/home/ondra/lib/lib/python/IPython/kernel/scripts/ipcluster.py", line 724, in main reactor.run() File "/usr/lib/python2.5/site-packages/twisted/internet/base.py", line 1128, in run self.mainLoop() --- --- File "/usr/lib/python2.5/site-packages/twisted/internet/base.py", line 1140, in mainLoop self.doIteration(t) File "/usr/lib/python2.5/site-packages/twisted/internet/selectreactor.py", line 104, in doSelect [], timeout) File "/home/ondra/lib/lib/python/IPython/kernel/scripts/ipcluster.py", line 572, in shutdown d = pbs_set.kill() File "/home/ondra/lib/lib/python/IPython/kernel/scripts/ipcluster.py", line 315, in kill [self.job_id],env=os.environ) exceptions.AttributeError: 'PBSEngineSet' object has no attribute 'job_id' and it keeps running. So it's unstoppable. :) I had to kill it with kill. Ondrej From gael.varoquaux at normalesup.org Sun Mar 22 10:23:02 2009 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 22 Mar 2009 15:23:02 +0100 Subject: [IPython-dev] bzr criss-cross merge?!?! Message-ID: <20090322142302.GB30682@phare.normalesup.org> On Fri, Mar 20, 2009 at 09:26:05PM -0700, Brian Granger wrote: > I read more about this and have some vague idea of what it means. But > I don't recall doing anything funny and I have no idea how to go about > finding the source of this. Does anyone know if I can just ignore > this, resolve my conflicts and carry on? Any insight from the bzr > masters among us would be greatly appreciated. AFAIK, criss-cross merge can be created by doing the same modifications to 2 branches, and trying to merge them. bzr is clever-enough to figure this out. It is safe to go on working. Ga?l From ellisonbg.net at gmail.com Mon Mar 23 00:43:13 2009 From: ellisonbg.net at gmail.com (Brian Granger) Date: Sun, 22 Mar 2009 21:43:13 -0700 Subject: [IPython-dev] ipcluster pbs gives no attribute 'job_id' In-Reply-To: <85b5c3130903220119n1d13e53dv7cf96320583aade8@mail.gmail.com> References: <85b5c3130903220119n1d13e53dv7cf96320583aade8@mail.gmail.com> Message-ID: <6ce0ac130903222143m24edb82kb0d0799dd0f3b827@mail.gmail.com> Have you tried to submit a regular job using pbs without ipcluster. I just want to know that your pbs is setup correctly. The self.job_id attribute is set in reponse to the PBS job being submitted. Thus, if it hangs and the job_id has not been set when you try to kil it, that means that something went wrong with the submission. Because of this, I would first make sure that you can submit a basic job using PBS. Then take that PBS script and use it to make a template for ipcluster. Cheers, Brian On Sun, Mar 22, 2009 at 1:19 AM, Ondrej Certik wrote: > Hi, > > I am trying to figure out how the pbs works, so I started on my laptop: > > $ ipcluster pbs -n 2 --pbs-script=pbs.template > 2009-03-22 01:08:50-0700 [-] Log opened. > 2009-03-22 01:08:50-0700 [-] Process ['ipcontroller', > '--logfile=/home/ondra/.ipython/log/ipcontroller'] has started with > pid=5404 > > > > and nothing more happened. So I wanted to stop with with CTRL-C, but I got: > > ^C2009-03-22 01:09:36-0700 [-] Stopping pbs cluster > 2009-03-22 01:09:36-0700 [-] Unexpected error in main loop. > 2009-03-22 01:09:36-0700 [-] Unhandled Error > ? ? ? ?Traceback (most recent call last): > ? ? ? ? ?File "/home/ondra/lib/bin/ipcluster", line 21, in > ? ? ? ? ? ?ipcluster.main() > ? ? ? ? ?File "/home/ondra/lib/lib/python/IPython/kernel/scripts/ipcluster.py", > line 724, in main > ? ? ? ? ? ?reactor.run() > ? ? ? ? ?File "/usr/lib/python2.5/site-packages/twisted/internet/base.py", > line 1128, in run > ? ? ? ? ? ?self.mainLoop() > ? ? ? ?--- --- > ? ? ? ? ?File "/usr/lib/python2.5/site-packages/twisted/internet/base.py", > line 1140, in mainLoop > ? ? ? ? ? ?self.doIteration(t) > ? ? ? ? ?File "/usr/lib/python2.5/site-packages/twisted/internet/selectreactor.py", > line 104, in doSelect > ? ? ? ? ? ?[], timeout) > ? ? ? ? ?File "/home/ondra/lib/lib/python/IPython/kernel/scripts/ipcluster.py", > line 572, in shutdown > ? ? ? ? ? ?d = pbs_set.kill() > ? ? ? ? ?File "/home/ondra/lib/lib/python/IPython/kernel/scripts/ipcluster.py", > line 315, in kill > ? ? ? ? ? ?[self.job_id],env=os.environ) > ? ? ? ?exceptions.AttributeError: 'PBSEngineSet' object has no attribute 'job_id' > > > and it keeps running. So it's unstoppable. :) I had to kill it with kill. > > Ondrej > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > From tobis at ig.utexas.edu Mon Mar 23 15:36:27 2009 From: tobis at ig.utexas.edu (Michael Tobis) Date: Mon, 23 Mar 2009 14:36:27 -0500 Subject: [IPython-dev] Seeking half-day iPython/Sage tutorial May 21 or May 22 Message-ID: I am writing on behalf of the University of Texas' Scientific Software Days meeting. We are seeking someone to give a tutorial in some broadly useful scientific software tool in Austin TX on May 21 or May 22. iPython as a replacement for Matlab is a good candidate topic. We'd ideally like to see side-by-side comparisons using matplotlib. Speakers who are familiar with Sage and/or with parallel computing features of iPython1 would be especially interesting. We can cover travel, lodging and per diem for a speaker from North America. If anyone on this list would be interested in giving a such a tutorial, please get in touch with me. thanks Michael Tobis University of Texas Institute for Geophysics tobis at ig.utexas.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From walter at livinglogic.de Thu Mar 26 15:11:08 2009 From: walter at livinglogic.de (=?UTF-8?B?V2FsdGVyIETDtnJ3YWxk?=) Date: Thu, 26 Mar 2009 20:11:08 +0100 Subject: [IPython-dev] Two tiny fixes for ipipe Message-ID: <49CBD34C.1040601@livinglogic.de> Hello all! This is my first attempt to use bzr/launchpad, so I might have done something wrong. I have two tiny fixes for ipipe in this branch lp:~doerwalter/ipython/ipipe-fix which I've marked for merging. Could someone merge the fix into the trunk? Servus, Walter From ellisonbg.net at gmail.com Tue Mar 31 17:34:53 2009 From: ellisonbg.net at gmail.com (Brian Granger) Date: Tue, 31 Mar 2009 14:34:53 -0700 Subject: [IPython-dev] Status of pre-0.10 reviews and merges Message-ID: <6ce0ac130903311434k2584a94h6abc6d233e41bc4d@mail.gmail.com> Hi, Things have gone quite for a while now and I wanted to see where we stand on getting ready for 0.10. There are 7 merge proposals listed here: https://code.launchpad.net/ipython/+activereviews Of these 4 seem to be in process and 3 are in need of review. What is the status of the 4 branches that have already been reviewed, but are in need to fixing and/or further review? Of the 3 branches that need review, which are targeted for 0.10? I can partially answer this: * My branch does need to be reviewed and merged before 0.10. I don't know about the other branches. Is there anything else that needs to happen before 0.10 is released? Again, I can give a partial answer... * Everyone who has had branches merged, needs to update the "what's new in ipython" document with information about what they did. * Fernando and I need to sort out the ipdoctest nose extension and the problems that it is causing with some of our Twisted tests. I want to get these things resolved quickly as I am being paid to work on IPython during April these pending merges are currently holding me up. This is because I am going to start making MAJOR changes that will make it extremely difficult if not impossible to merge branches that are children of our current "trunk." We don't need to release 0.10 for me to get started with this work, but we do need to get these branches merged before I branch. If you are working on reviews or branches related to 0.10 please let us know where you stand and a timeframe for finishing up. Cheers, Brian From fperez.net at gmail.com Tue Mar 31 18:23:06 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 31 Mar 2009 15:23:06 -0700 Subject: [IPython-dev] Status of pre-0.10 reviews and merges In-Reply-To: <6ce0ac130903311434k2584a94h6abc6d233e41bc4d@mail.gmail.com> References: <6ce0ac130903311434k2584a94h6abc6d233e41bc4d@mail.gmail.com> Message-ID: Hey, On Tue, Mar 31, 2009 at 2:34 PM, Brian Granger wrote: [...] > * Fernando and I need to sort out the ipdoctest nose extension and the > problems that it is causing with some of our Twisted tests. > > If you are working on reviews or branches related to 0.10 please let > us know where you stand and a timeframe for finishing up. Thanks for pinging back in: here we had a massive sprint for neuroimaging.scipy.org (a single merge was over 92000 lines long, to give you an idea of how much code we moved around :) that I'm only now recovering again from, as grant deadlines pile up. Over the next two weeks I'll plug back into this so we can put 0.10 out. I've been thinking about the twisted bug, and I suspect it has to do with global state. The ipdoctest plugin insantiates a real ipython, and that makes some global changes. I wonder if those are confusing twisted somehow. Can you think of anything in this direction? > I want to get these things resolved quickly as I am being paid to work > on IPython during April these pending merges are currently holding me > up. This is because I am going to start making MAJOR changes that > will make it extremely difficult if not impossible to merge branches > that are children of our current "trunk." We don't need to release > 0.10 for me to get started with this work, but we do need to get these > branches merged before I branch. I know we've talked a bit about this, but before we get started on this major reorganization, let's make sure we do plan it carefully. For better or worse, while ipython is formally 'pre 1.0' and as such, all API changes are in principle fair game, we need to ensure we find a good balance of practicality vs purity. Just like numpy kept some Numeric-inspired baggage that wasn't ideal so that users could transition without unnecessary pain, we should keep the same ideas in mind. We should obviously fix the major messes we have (ipmaker is ridiculous, 3 config systems is crazy, etc), but we should also be willing to tolerate small warts if they make existing user's lives easier. Part of that is communicating a plan for the breakage to everyone in advance, so that feedback can be provided by all affected before anything irreversible has happened. Furthermore, I'd like to suggest that post 0.10 we hold a 'bug and test weeks' (2-4 max) to see if we could do an 0.11 release where we basically try to *only* fix existing tickets and add as many tests as we can, before starting to do major API breakage. There are reasons for this idea: - This would leave 0.11 as an as-bugfixed-as-possible stable point of reference for anyone who might find the upcoming changes too disruptive for a while. - By forcing ourselves to write tons of tests (even just blanketing the codebase with doctests) we would leave the code in a far more robust position for the upcoming reorganization. I really think that the level of code changes you have in mind is a dangerous performance with our current test coverage being so low for the end-user-facing parts. But now that with ipdoctest in place it is *trivial* to write tests against that functionality, it would be possible to protect the code better with tests *before* diving in, so that at least there's a safety net to catch us if we fall. I realize that this would depend on how much time we can actually commit to this. I'd love to do a lot, but my next 12 weeks are insane: *3* grants to write, another full-week sprint to host here, and one major conference coming up. I'm sure others have also hectic schedules. But perhaps by putting a hard time limit on it, we can avoid too much disruption: basically we get done what we get done in that time, little or lots, and then we move on with the reorganization. How does that sound? f