From walter at livinglogic.de  Wed Nov  1 13:52:26 2006
From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=)
Date: Wed, 01 Nov 2006 19:52:26 +0100
Subject: [IPython-dev] IPython.Extensions on sys.path
Message-ID: <4548ECEA.3020605@livinglogic.de>

Hello list!

Currently IPython/__init__.py adds IPython.Extensions to sys.path. IMHO
this is bad, because it can lead to hard to detect bugs: If a module is
imported from both locations you get two modules that look and smell the
same, yet are totally unrelated (chaing one doesn't change the other).
Furthermore modules in IPython.Extensions can shadow modules that are
installed globally which leads to different behaviour depending on
whether code runs in pure Python or in IPython.

Is there a reason why IPython.Extensions is added? Could we drop this in
the next IPython release?

Servus,
   Walter


From vivainio at gmail.com  Wed Nov  1 16:30:20 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Wed, 1 Nov 2006 23:30:20 +0200
Subject: [IPython-dev] IPython.Extensions on sys.path
In-Reply-To: <4548ECEA.3020605@livinglogic.de>
References: <4548ECEA.3020605@livinglogic.de>
Message-ID: <46cb515a0611011330u2c7286e3t4c345be798925959@mail.gmail.com>

On 11/1/06, Walter D?rwald <walter at livinglogic.de> wrote:

> Is there a reason why IPython.Extensions is added? Could we drop this in
> the next IPython release?

There is a reason - it makes the modules in the directory easy to
import (import bah instead of from IPython.Extensions import bah), and
we can ship modules there that could also be available as third party
modules (e.g. 'path'). That way, the same code can be used in and out
of ipython (instead of having to type "from IPython.Extensions.path
import path' or even 'from IPython.path import path', which would be
invalid if IPython was not installed).

A case in point - the "pickleshare" module just imports top level
"path", and as such doesn't require IPython to be usable. The module
is available outside IPython, and I didn't feel like "forking" a
version for IPython (or trying to import path both from top level and
IPython).

One alternative would perhaps be to declare all the stuff in
Extensions as py_modules in setup.py. That way, at least with
setuptools, they would be visible outside the IPython package as top
level modules (yet still reside in the IPython egg directory).

Is there really a problem, apart from the "path" module? If path is
the only problem, we could add 'path' to 'py_modules' and remove it
from Extensions. But the rest should IMHO stay as it is. Many modules
in Extensions are unusable outside IPython anyway.

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From walter at livinglogic.de  Thu Nov  2 03:00:29 2006
From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=)
Date: Thu, 02 Nov 2006 09:00:29 +0100
Subject: [IPython-dev] IPython.Extensions on sys.path
In-Reply-To: <46cb515a0611011330u2c7286e3t4c345be798925959@mail.gmail.com>
References: <4548ECEA.3020605@livinglogic.de>
	<46cb515a0611011330u2c7286e3t4c345be798925959@mail.gmail.com>
Message-ID: <4549A59D.8060307@livinglogic.de>

Ville M. Vainio wrote:

> On 11/1/06, Walter D?rwald <walter at livinglogic.de> wrote:
> 
>> Is there a reason why IPython.Extensions is added? Could we drop this in
>> the next IPython release?
> 
> There is a reason - it makes the modules in the directory easy to
> import (import bah instead of from IPython.Extensions import bah), and
> we can ship modules there that could also be available as third party
> modules (e.g. 'path'). That way, the same code can be used in and out
> of ipython (instead of having to type "from IPython.Extensions.path
> import path' or even 'from IPython.path import path', which would be
> invalid if IPython was not installed).

IMHO the proper way to treat such dependencies would be to document the 
dependency and let the user install the required module. With setuptools 
this can be done automatically:
     setup(
         ...,
         install_requires=[
            "path.py >= 2.1",
         ],
     )

On Windows we already have a dependency on other packages (actually two: 
pyreadline and ctypes) anyway.

> A case in point - the "pickleshare" module just imports top level
> "path", and as such doesn't require IPython to be usable. The module
> is available outside IPython, and I didn't feel like "forking" a
> version for IPython (or trying to import path both from top level and
> IPython).

Would this mean that all modules that are just copies of third-party 
module and can work independently from IPython should be imported via 
"import foo" and modules that depend on IPython should be imported via 
"from IPython.Extensions import foo"?

> One alternative would perhaps be to declare all the stuff in
> Extensions as py_modules in setup.py. That way, at least with
> setuptools, they would be visible outside the IPython package as top
> level modules (yet still reside in the IPython egg directory).

I'm not sure. At least there would no longer be identical copies of the 
same module. But installing IPython would then "overwrite" the original 
version of the module, i.e. when importing it you'll get the IPython 
version instead of the original one.

> Is there really a problem, apart from the "path" module? If path is
> the only problem, we could add 'path' to 'py_modules' and remove it
> from Extensions. But the rest should IMHO stay as it is. Many modules
> in Extensions are unusable outside IPython anyway.

I'd like to use Philip Eby's simplegeneric module 
(http://cheeseshop.python.org/pypi/simplegeneric) for ipipe. xrepr(), 
xiter() and xattrs() could benefit from being generic functions. I 
already experimented in this direction some time ago, but now that a 
simple standalone generic function module is available it no longer 
makes sense to reinvent the wheel.

Philip's OK with us bundling the module, as long as it doesn't conflict 
with any installed version of simplegeneric, which it would (but only 
when running inside IPython).

Servus,
    Walter



From vivainio at gmail.com  Thu Nov  2 03:30:46 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Thu, 2 Nov 2006 10:30:46 +0200
Subject: [IPython-dev] IPython.Extensions on sys.path
In-Reply-To: <4549A59D.8060307@livinglogic.de>
References: <4548ECEA.3020605@livinglogic.de>
	<46cb515a0611011330u2c7286e3t4c345be798925959@mail.gmail.com>
	<4549A59D.8060307@livinglogic.de>
Message-ID: <46cb515a0611020030p1e6bee59rc571779dabcf2e35@mail.gmail.com>

On 11/2/06, Walter D?rwald <walter at livinglogic.de> wrote:

> IMHO the proper way to treat such dependencies would be to document the
> dependency and let the user install the required module. With setuptools
> this can be done automatically:
>      setup(
>          ...,
>          install_requires=[
>             "path.py >= 2.1",
>          ],
>      )

We are still in "no setuptools required" zone (and will for a while,
probably - it didn't end up in python 2.5). And path.py isn't
installable via easy_install at this moment. I'd like to see all these
handy little modules bundled with IPython instead.

> On Windows we already have a dependency on other packages (actually two:
> pyreadline and ctypes) anyway.

ctypes is bundled with python 2.5, and pyreadline is easy_install-able.

> Would this mean that all modules that are just copies of third-party
> module and can work independently from IPython should be imported via
> "import foo" and modules that depend on IPython should be imported via
> "from IPython.Extensions import foo"?

Yes. Though I'm not so sure about "from IPython.Extensions", it's up
to the author of the code to determine which feels more comfortable.
Technically, "Extensions" should all be IPython dependent (IPython
extension plugins) but ATM it also contains stuff that is importable
as top level modules when running inside IPython.

> I'm not sure. At least there would no longer be identical copies of the
> same module. But installing IPython would then "overwrite" the original
> version of the module, i.e. when importing it you'll get the IPython
> version instead of the original one.

Yeah, I don't like that either.

> I'd like to use Philip Eby's simplegeneric module
> (http://cheeseshop.python.org/pypi/simplegeneric) for ipipe. xrepr(),
> xiter() and xattrs() could benefit from being generic functions. I
> already experimented in this direction some time ago, but now that a
> simple standalone generic function module is available it no longer
> makes sense to reinvent the wheel.
>
> Philip's OK with us bundling the module, as long as it doesn't conflict
> with any installed version of simplegeneric, which it would (but only
> when running inside IPython).

Ok. Put it into IPython folder (not IPython/Extensions) and import it
explicitly from there ("from IPython import simplegeneric").

I'm still -1 on removing Extensions from sys.path, though.

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From walter at livinglogic.de  Thu Nov  2 05:55:39 2006
From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=)
Date: Thu, 02 Nov 2006 11:55:39 +0100
Subject: [IPython-dev] IPython.Extensions on sys.path
In-Reply-To: <46cb515a0611020030p1e6bee59rc571779dabcf2e35@mail.gmail.com>
References: <4548ECEA.3020605@livinglogic.de>	
	<46cb515a0611011330u2c7286e3t4c345be798925959@mail.gmail.com>	
	<4549A59D.8060307@livinglogic.de>
	<46cb515a0611020030p1e6bee59rc571779dabcf2e35@mail.gmail.com>
Message-ID: <4549CEAB.5050907@livinglogic.de>

Ville M. Vainio wrote:

> On 11/2/06, Walter D?rwald <walter at livinglogic.de> wrote:
> 
>> IMHO the proper way to treat such dependencies would be to document the
>> dependency and let the user install the required module. With setuptools
>> this can be done automatically:
>>      setup(
>>          ...,
>>          install_requires=[
>>             "path.py >= 2.1",
>>          ],
>>      )
> 
> We are still in "no setuptools required" zone (and will for a while,
> probably - it didn't end up in python 2.5). And path.py isn't
> installable via easy_install at this moment.

Well, easy_install
http://www.jorendorff.com/articles/python/path/path-2.1.zip works, but I
agree that it would be better if path 2.1 was available from the cheeseshop.

> I'd like to see all these
> handy little modules bundled with IPython instead.

OK.

>> On Windows we already have a dependency on other packages (actually two:
>> pyreadline and ctypes) anyway.
> 
> ctypes is bundled with python 2.5, and pyreadline is easy_install-able.

Maybe we should convince Jason Orendorff to upload a new version to the
cheeseshop?

>> Would this mean that all modules that are just copies of third-party
>> module and can work independently from IPython should be imported via
>> "import foo" and modules that depend on IPython should be imported via
>> "from IPython.Extensions import foo"?
> 
> Yes. Though I'm not so sure about "from IPython.Extensions", it's up
> to the author of the code to determine which feels more comfortable.
> Technically, "Extensions" should all be IPython dependent (IPython
> extension plugins) but ATM it also contains stuff that is importable
> as top level modules when running inside IPython.
> 
>> I'm not sure. At least there would no longer be identical copies of the
>> same module. But installing IPython would then "overwrite" the original
>> version of the module, i.e. when importing it you'll get the IPython
>> version instead of the original one.
> 
> Yeah, I don't like that either.
> 
>> I'd like to use Philip Eby's simplegeneric module
>> (http://cheeseshop.python.org/pypi/simplegeneric) for ipipe. xrepr(),
>> xiter() and xattrs() could benefit from being generic functions. I
>> already experimented in this direction some time ago, but now that a
>> simple standalone generic function module is available it no longer
>> makes sense to reinvent the wheel.
>>
>> Philip's OK with us bundling the module, as long as it doesn't conflict
>> with any installed version of simplegeneric, which it would (but only
>> when running inside IPython).
> 
> Ok. Put it into IPython folder (not IPython/Extensions) and import it
> explicitly from there ("from IPython import simplegeneric").

OK, will do.

> I'm still -1 on removing Extensions from sys.path, though.

As long as we have a conflict free way of bundling third-party modules,
I can live with that.

Servus,
   Walter


From vivainio at gmail.com  Thu Nov  2 06:13:51 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Thu, 2 Nov 2006 13:13:51 +0200
Subject: [IPython-dev] IPython.Extensions on sys.path
In-Reply-To: <4549CEAB.5050907@livinglogic.de>
References: <4548ECEA.3020605@livinglogic.de>
	<46cb515a0611011330u2c7286e3t4c345be798925959@mail.gmail.com>
	<4549A59D.8060307@livinglogic.de>
	<46cb515a0611020030p1e6bee59rc571779dabcf2e35@mail.gmail.com>
	<4549CEAB.5050907@livinglogic.de>
Message-ID: <46cb515a0611020313u699fe901m6a13132a667b694c@mail.gmail.com>

On 11/2/06, Walter D?rwald <walter at livinglogic.de> wrote:

> Maybe we should convince Jason Orendorff to upload a new version to the
> cheeseshop?

That would be cool; though I don't think we are quite ready yet to
assume everyone is able/willing to easy_install stuff.

> As long as we have a conflict free way of bundling third-party modules,
> I can live with that.

At the moment "path.py" is the only one that could cause conflicts,
but path.py has been relavitely stable forever, so it doesn't matter
much what version of path.py gets used. For more volatile (and yet
widely used) third party modules, it may indeed make more sense to
use them from inside IPython package.

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From fperez.net at gmail.com  Fri Nov  3 03:14:06 2006
From: fperez.net at gmail.com (Fernando Perez)
Date: Fri, 3 Nov 2006 01:14:06 -0700
Subject: [IPython-dev] IPython.Extensions on sys.path
In-Reply-To: <46cb515a0611020030p1e6bee59rc571779dabcf2e35@mail.gmail.com>
References: <4548ECEA.3020605@livinglogic.de>
	<46cb515a0611011330u2c7286e3t4c345be798925959@mail.gmail.com>
	<4549A59D.8060307@livinglogic.de>
	<46cb515a0611020030p1e6bee59rc571779dabcf2e35@mail.gmail.com>
Message-ID: <db6b5ecc0611030014pf607d42w27d6eadd86008a8a@mail.gmail.com>

On 11/2/06, Ville M. Vainio <vivainio at gmail.com> wrote:
> On 11/2/06, Walter D?rwald <walter at livinglogic.de> wrote:

> Yes. Though I'm not so sure about "from IPython.Extensions", it's up
> to the author of the code to determine which feels more comfortable.
> Technically, "Extensions" should all be IPython dependent (IPython
> extension plugins) but ATM it also contains stuff that is importable
> as top level modules when running inside IPython.
>
> > I'm not sure. At least there would no longer be identical copies of the
> > same module. But installing IPython would then "overwrite" the original
> > version of the module, i.e. when importing it you'll get the IPython
> > version instead of the original one.
>
> Yeah, I don't like that either.
>
> > I'd like to use Philip Eby's simplegeneric module
> > (http://cheeseshop.python.org/pypi/simplegeneric) for ipipe. xrepr(),
> > xiter() and xattrs() could benefit from being generic functions. I
> > already experimented in this direction some time ago, but now that a
> > simple standalone generic function module is available it no longer
> > makes sense to reinvent the wheel.
> >
> > Philip's OK with us bundling the module, as long as it doesn't conflict
> > with any installed version of simplegeneric, which it would (but only
> > when running inside IPython).
>
> Ok. Put it into IPython folder (not IPython/Extensions) and import it
> explicitly from there ("from IPython import simplegeneric").

I'd suggest making a subdir in IPython for third-party modules, so
that we can import them from there.  It doesn't particularly hurt us
to have to type an extra word in the import statements, and it is more
self-documenting that these are simply pulled in for convenience.
Something like IPython/contrib or 'external' can work.

from IPython.external import simplegeneric

lets us clearly know in the future that this comes from the outside.
We can simply document to users which external modules we ship as a
convenience, so they can use them from within ipython.

> I'm still -1 on removing Extensions from sys.path, though.

This one isn't terrible IMO because we're putting it, along with
~/.ipython, at the very end of sys.path, so the danger of it shadowing
a user's expected directory is minimal.  It will shadow an
ImportError, but it won't cover another directory.

What does bother me is that Python (I didn't know this) prepends any
script's execution directory to sys.path.  Try this:

longs[~/test]> pwd
/home/fperez/test

longs[~/test]> cat spath.py
import pprint,sys
pprint.pprint(sys.path)

longs[~/test]> python spath.py
['/home/fperez/test',
 '/home/fperez/test',
 '/home/fperez/tmp/local/lib/python2.4/site-packages',
 '/home/fperez/usr/lib/python',
 '/home/fperez/usr/lib/python2.4/site-packages',
 '/home/fperez/usr/local/lib/python',
 '/home/fperez/usr/local/lib/python2.4/site-packages',
 '/usr/local/lib/python2.4/site-packages',

[etc...]

I just added this code to the main ipython starter script:

# Start by cleaning up sys.path: Python automatically inserts the script's
# base directory into sys.path, at the front.  This can lead to unpleasant
# surprises.
import sys
sys.path.pop(0)

Now, the front of sys.path is identical for a plain python and an
ipython shell.  The only difference is that under ipython, the very
last two entries are:

'/home/fperez/usr/lib/python2.4/site-packages/IPython/Extensions',
 '/home/fperez/.ipython']

which don't exist for plain Python.  I don't see these as a big problem, though.

If others disagree, let us know.

Cheers,


f


From walter at livinglogic.de  Fri Nov  3 05:42:51 2006
From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=)
Date: Fri, 03 Nov 2006 11:42:51 +0100
Subject: [IPython-dev] IPython.Extensions on sys.path
In-Reply-To: <db6b5ecc0611030014pf607d42w27d6eadd86008a8a@mail.gmail.com>
References: <4548ECEA.3020605@livinglogic.de>	
	<46cb515a0611011330u2c7286e3t4c345be798925959@mail.gmail.com>	
	<4549A59D.8060307@livinglogic.de>	
	<46cb515a0611020030p1e6bee59rc571779dabcf2e35@mail.gmail.com>
	<db6b5ecc0611030014pf607d42w27d6eadd86008a8a@mail.gmail.com>
Message-ID: <454B1D2B.509@livinglogic.de>

Fernando Perez wrote:

> On 11/2/06, Ville M. Vainio <vivainio at gmail.com> wrote:
>> On 11/2/06, Walter D?rwald <walter at livinglogic.de> wrote:
> [...]
>> > I'd like to use Philip Eby's simplegeneric module
>> > (http://cheeseshop.python.org/pypi/simplegeneric) for ipipe. xrepr(),
>> > xiter() and xattrs() could benefit from being generic functions. I
>> > already experimented in this direction some time ago, but now that a
>> > simple standalone generic function module is available it no longer
>> > makes sense to reinvent the wheel.
>> >
>> > Philip's OK with us bundling the module, as long as it doesn't conflict
>> > with any installed version of simplegeneric, which it would (but only
>> > when running inside IPython).
>>
>> Ok. Put it into IPython folder (not IPython/Extensions) and import it
>> explicitly from there ("from IPython import simplegeneric").
> 
> I'd suggest making a subdir in IPython for third-party modules, so
> that we can import them from there.  It doesn't particularly hurt us
> to have to type an extra word in the import statements, and it is more
> self-documenting that these are simply pulled in for convenience.
> Something like IPython/contrib or 'external' can work.
> 
> from IPython.external import simplegeneric
> 
> lets us clearly know in the future that this comes from the outside.
> We can simply document to users which external modules we ship as a
> convenience, so they can use them from within ipython.

OK, then I'll add it to a new external subpackage.

> [...]

Servus,
   Walter


From rocky at panix.com  Fri Nov  3 06:41:38 2006
From: rocky at panix.com (R. Bernstein)
Date: Fri, 3 Nov 2006 06:41:38 -0500
Subject: [IPython-dev] ipython/pydb integeration - much progress, more still?
Message-ID: <17739.10994.675434.379180@panix3.panix.com>

Thanks for putting in the remaining patches for pydb integration!

One small debug "print" slipped into magic_pdef around line 672 of
Magic.py. (It was in my patch).

Some other remaining things.

I believe addressing the readline history problem obviates the need
for *all* of the Python version-checking in Debugger.py. In
particular, the code that starts this:

   # Ugly hack: for Python 2.3-2.4, we can't call the parent constructor,
   # because it binds readline and breaks tab-completion.  This means we
   # have to COPY the constructor here.

SVN checkin comments for this portion of the code are this: 
    fperez   - Cleanup [1786], which went in with unfinished stuff by accident
    fperez   - Apply Ville's patch, closes #87

As mentioned previously, readline gets imported on Python 2.5's pdb
unconditionally just as it does in 2.4, so I'm at a loss as to what
was changed in Python 2.5's pdb that "fixed" things. And again, given
the more general readline history fix, is this *still* needed?

I tried a little test on Python 2.4 and don't see a problem. But then
it's not clear to me from the above comments what the problem was or
how one would try to recreate the problem observed.

If this version testing is removed then pydb's post-mortem will work
on all versions of Python. 

Next, I wonder about the need for having to issue "import ipy_pydb".

Why not just uconditionally surround import via a "try/except" block?
If it fails, no problem, there won't be a %pydb magic.

Finally there's the problem of getting ipython's Pdb class used when
calling pydb.runl rather than pydb's Pdb class, so that the
ipython-specific changes get applied when calling via %pydb.

Again, thanks for moving forward on this.







From walter at livinglogic.de  Fri Nov  3 09:07:18 2006
From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=)
Date: Fri, 03 Nov 2006 15:07:18 +0100
Subject: [IPython-dev] IPython.Extensions on sys.path
In-Reply-To: <db6b5ecc0611030014pf607d42w27d6eadd86008a8a@mail.gmail.com>
References: <4548ECEA.3020605@livinglogic.de>	
	<46cb515a0611011330u2c7286e3t4c345be798925959@mail.gmail.com>	
	<4549A59D.8060307@livinglogic.de>	
	<46cb515a0611020030p1e6bee59rc571779dabcf2e35@mail.gmail.com>
	<db6b5ecc0611030014pf607d42w27d6eadd86008a8a@mail.gmail.com>
Message-ID: <454B4D16.90503@livinglogic.de>

Fernando Perez wrote:

> [...]
> I'd suggest making a subdir in IPython for third-party modules, so
> that we can import them from there.  It doesn't particularly hurt us
> to have to type an extra word in the import statements, and it is more
> self-documenting that these are simply pulled in for convenience.
> Something like IPython/contrib or 'external' can work.
> 
> from IPython.external import simplegeneric

OK, done. Should I move path.py too?

Servus,
   Walter


From vivainio at gmail.com  Fri Nov  3 09:27:37 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Fri, 3 Nov 2006 16:27:37 +0200
Subject: [IPython-dev] IPython.Extensions on sys.path
In-Reply-To: <454B4D16.90503@livinglogic.de>
References: <4548ECEA.3020605@livinglogic.de>
	<46cb515a0611011330u2c7286e3t4c345be798925959@mail.gmail.com>
	<4549A59D.8060307@livinglogic.de>
	<46cb515a0611020030p1e6bee59rc571779dabcf2e35@mail.gmail.com>
	<db6b5ecc0611030014pf607d42w27d6eadd86008a8a@mail.gmail.com>
	<454B4D16.90503@livinglogic.de>
Message-ID: <46cb515a0611030627r34a84137v19185c1b5bcc0a95@mail.gmail.com>

On 11/3/06, Walter D?rwald <walter at livinglogic.de> wrote:

> > from IPython.external import simplegeneric
>
> OK, done. Should I move path.py too?

If you add 'external' to sys.path, yes.

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From walter at livinglogic.de  Fri Nov  3 10:21:53 2006
From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=)
Date: Fri, 03 Nov 2006 16:21:53 +0100
Subject: [IPython-dev] IPython.Extensions on sys.path
In-Reply-To: <46cb515a0611030627r34a84137v19185c1b5bcc0a95@mail.gmail.com>
References: <4548ECEA.3020605@livinglogic.de>	
	<46cb515a0611011330u2c7286e3t4c345be798925959@mail.gmail.com>	
	<4549A59D.8060307@livinglogic.de>	
	<46cb515a0611020030p1e6bee59rc571779dabcf2e35@mail.gmail.com>	
	<db6b5ecc0611030014pf607d42w27d6eadd86008a8a@mail.gmail.com>	
	<454B4D16.90503@livinglogic.de>
	<46cb515a0611030627r34a84137v19185c1b5bcc0a95@mail.gmail.com>
Message-ID: <454B5E91.7000808@livinglogic.de>

Ville M. Vainio wrote:

> On 11/3/06, Walter D?rwald <walter at livinglogic.de> wrote:
> 
>> > from IPython.external import simplegeneric
>>
>> OK, done. Should I move path.py too?
> 
> If you add 'external' to sys.path, yes.

But then we're back to where we started: A module in IPython.external
shadows a global module.

Servus,
   Walter



From walter at livinglogic.de  Fri Nov  3 11:23:52 2006
From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=)
Date: Fri, 03 Nov 2006 17:23:52 +0100
Subject: [IPython-dev] IPython.Extensions on sys.path
In-Reply-To: <db6b5ecc0611030014pf607d42w27d6eadd86008a8a@mail.gmail.com>
References: <4548ECEA.3020605@livinglogic.de>	
	<46cb515a0611011330u2c7286e3t4c345be798925959@mail.gmail.com>	
	<4549A59D.8060307@livinglogic.de>	
	<46cb515a0611020030p1e6bee59rc571779dabcf2e35@mail.gmail.com>
	<db6b5ecc0611030014pf607d42w27d6eadd86008a8a@mail.gmail.com>
Message-ID: <454B6D18.9020500@livinglogic.de>

Fernando Perez wrote:

> [...]
> 
> I just added this code to the main ipython starter script:
> 
> # Start by cleaning up sys.path: Python automatically inserts the script's
> # base directory into sys.path, at the front.  This can lead to unpleasant
> # surprises.
> import sys
> sys.path.pop(0)
> 
> Now, the front of sys.path is identical for a plain python and an
> ipython shell.  The only difference is that under ipython, the very
> last two entries are:
> 
> '/home/fperez/usr/lib/python2.4/site-packages/IPython/Extensions',
> '/home/fperez/.ipython']
> 
> which don't exist for plain Python.  I don't see these as a big problem,
> though.
> 
> If others disagree, let us know.

This breaks running the start script directly. With the current version
I get a sys.path that looks like this:

['',
 '/usr/local/lib/python2.5/site-packages/mpcp-1.5-py2.5.egg',
 '/usr/local/lib/python2.5/site-packages/docutils-0.4-py2.5.egg',
 '/usr/local/lib/python2.5/site-packages/ipython-0.7.2-py2.5.egg',
 '/usr/local/lib/python2.5/site-packages/setuptools-0.6c3-py2.5.egg',
 '/usr/local/lib/python2.5/site-packages/cssutils-0.9a6-py2.5.egg',
 '/usr/local/lib/python2.5/site-packages/simplegeneric-0.6-py2.5.egg',
 '/usr/local/lib/python2.5/site-packages/path-2.1-py2.5.egg',
 '/var/home/walter/dist-py',
 '/usr/local/lib/svn-python',
 '/usr/local/lib/python25.zip',
 '/usr/local/lib/python2.5',
 '/usr/local/lib/python2.5/plat-linux2',
 '/usr/local/lib/python2.5/lib-tk',
 '/usr/local/lib/python2.5/lib-dynload',
 '/usr/local/lib/python2.5/site-packages',
 '/var/home/walter/pythonroot',
 '/usr/local/lib/python2.5/site-packages/ipython-0.7.2-py2.5.egg/IPython/Extensions',
 '/var/home/walter/.ipython']

Note that the IPython/Extensions directory from the globally installed
IPython has been added. Commenting out the sys.path.pop(0) gives me:

['',
 '/var/home/walter/checkouts/IPython/current',
 '/usr/local/lib/python2.5/site-packages/mpcp-1.5-py2.5.egg',
 '/usr/local/lib/python2.5/site-packages/docutils-0.4-py2.5.egg',
 '/usr/local/lib/python2.5/site-packages/ipython-0.7.2-py2.5.egg',
 '/usr/local/lib/python2.5/site-packages/setuptools-0.6c3-py2.5.egg',
 '/usr/local/lib/python2.5/site-packages/cssutils-0.9a6-py2.5.egg',
 '/usr/local/lib/python2.5/site-packages/simplegeneric-0.6-py2.5.egg',
 '/usr/local/lib/python2.5/site-packages/path-2.1-py2.5.egg',
 '/var/home/walter/dist-py',
 '/usr/local/lib/svn-python',
 '/usr/local/lib/python25.zip',
 '/usr/local/lib/python2.5',
 '/usr/local/lib/python2.5/plat-linux2',
 '/usr/local/lib/python2.5/lib-tk',
 '/usr/local/lib/python2.5/lib-dynload',
 '/usr/local/lib/python2.5/site-packages',
 '/var/home/walter/pythonroot',
 '/var/home/walter/checkouts/IPython/current/IPython/Extensions',
 '/var/home/walter/.ipython']

Servus,
   Walter


From john.d.kimball at navy.mil  Sat Nov  4 18:09:38 2006
From: john.d.kimball at navy.mil (Kimball, John D CIV NSWCDL G33)
Date: Sat, 4 Nov 2006 18:09:38 -0500
Subject: [IPython-dev] IPython1 unit tests fail
Message-ID: <E02A886D95ED09458A0BC124C790F9C402B3B4DD@NAEANRFKEX04VA.nadsusea.nads.navy.mil>

 
I installed ipython1 and dependecies in accordance with the INSTALL
file.  The unit tests fail, and I'm posting the problem here per the
directions in the INSTALL file.
 
Thanks for your assistance.  You all are doing awesome work!
 
I'm using:
 
enthon 1.0.0 (python 2.4.3)
Windows 2000 SP4 
ZopeInterface-3.1.0c1
The versions of Twisted and Nevow current as of 4PM 11-04-06.
 
I get the following result:
 
Running 29 tests.
 
C:\Python24\lib\site-packages\twisted\trial\runner.py:243:
exceptions.DeprecationWarning: log.startKeepingErrors is deprecated
since Twisted 2.5
ipython1.test.test_controllerpb
  BasicControllerPBTest
    testDeferreds ...
[ERROR]
    cleanup errors
[ERROR]
 
C:\Python24\lib\site-packages\twisted\trial\reporter.py:201:
twisted.trial.reporter.BrokenTestCaseWarning: REACTOR UNCLEAN!
traceback(s) follow: 
Traceback (most recent call last):
  File "C:\Python24\Lib\site-packages\twisted\trial\unittest.py", line
651, in _classCleanUp
    util._Janitor().postClassCleanup()
  File "C:\Python24\Lib\site-packages\twisted\trial\util.py", line 68,
in postClassCleanup
    'cleanPending', 'cleanThreads')
  File "C:\Python24\Lib\site-packages\twisted\trial\util.py", line 72,
in _dispatch
    getattr(self, "do_%s" % attr)()
  File "C:\Python24\Lib\site-packages\twisted\trial\util.py", line 126,
in do_cleanReactor
    raise DirtyReactorError(' '.join(s))
twisted.trial.util.DirtyReactorError: THIS WILL BECOME AN ERROR SOON!
reactor left in unclean state, the following Selectables were left over:
<<class 'twisted.internet.tcp.Port'> of
twisted.spread.pb.PBServerFactory on 10111> <<class
'twisted.internet.tcp.Client'> to ('127.0.0.1', 10111) at 18b1ab0>
<Broker #0 on 10111>
 
    testExecute ...
[ERROR]
    testInterfaces ...
[ERROR]
    testPullNamespace ...
[ERROR]
    testPushPull ...
[ERROR]
    testPushPullSerialized ...
[ERROR]
    testResult ...
[ERROR]
    testScatterGather ...
[ERROR]
ipython1.test.test_controllerservice
  BasicControllerServiceTest
    testDeferreds ...
[ERROR]
    testExecute ...
[ERROR]
    testInterfaces ...
[ERROR]
    testPullNamespace ...
[ERROR]
    testPushPull ...
[ERROR]
    testPushPullSerialized ...
[ERROR]
    testResult ...
[ERROR]
    testScatterGather ...
[ERROR]
                                              [ERROR]
ipython1.test.test_engineservice
  BasicEngineServiceTest
    testCompletedEmptyEngine ...
[ERROR]
    testDeferreds ...
[OK]
    testExecute ...
[OK]
    testGetResult ...
[OK]
    testInterfaces ...
[ERROR]
    testPullNamespace ...
[OK]
    testPushPull ...
[OK]
    testPushPullSerialized ...
[OK]
    testStatus ...
[OK]
                                                     [ERROR]
ipython1.test.test_shell
  BasicShellTest
    testCommand ...
[OK]
    testExecute ...
[OK]
    testPutGet ...
[OK]
    testUpdate ...
[OK]
 
========================================================================
=======
[ERROR]:
ipython1.test.test_controllerpb.BasicControllerPBTest.testDeferreds
 
Traceback (most recent call last):
  File
"C:\Python24\Lib\site-packages\ipython1\test\test_controllerpb.py", line
41, in setUp
    self.addEngine(1)
  File "C:\Python24\Lib\site-packages\ipython1\test\multienginetest.py",
line 20, in addEngine
    e = es.completeEngine(es.EngineService(InteractiveShell))
  File "C:\Python24\Lib\site-packages\ipython1\kernel\engineservice.py",
line 157, in completeEngine
    zi.alsoProvides(engine, IEngineComplete)
exceptions.AttributeError: 'module' object has no attribute
'alsoProvides'
========================================================================
=======
[ERROR]:
ipython1.test.test_controllerpb.BasicControllerPBTest.testExecute
 
Traceback (most recent call last):
  File
"C:\Python24\Lib\site-packages\ipython1\test\test_controllerpb.py", line
36, in setUp
    self.s = reactor.listenTCP(10111, self.sf)
  File "C:\Python24\Lib\site-packages\twisted\internet\posixbase.py",
line 386, in listenTCP
    p.startListening()
  File "C:\Python24\Lib\site-packages\twisted\internet\tcp.py", line
733, in startListening
    raise CannotListenError, (self.interface, self.port, le)
twisted.internet.error.CannotListenError: Couldn't listen on any:10111:
(10048, 'Address already in use').
========================================================================
=======
and so on... (same errors repeated)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20061104/f3f1839a/attachment.html>

From ellisonbg.net at gmail.com  Sat Nov  4 19:48:42 2006
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Sat, 4 Nov 2006 17:48:42 -0700
Subject: [IPython-dev] IPython1 unit tests fail
In-Reply-To: <E02A886D95ED09458A0BC124C790F9C402B3B4DD@NAEANRFKEX04VA.nadsusea.nads.navy.mil>
References: <E02A886D95ED09458A0BC124C790F9C402B3B4DD@NAEANRFKEX04VA.nadsusea.nads.navy.mil>
Message-ID: <6ce0ac130611041648i66e66711md5e641b38382398b@mail.gmail.com>

zi.alsoProvides(engine, IEngineComplete)
exceptions.AttributeError: 'module' object has no attribute 'alsoProvides'

This error from zope.interface makes me think that you have an older
version of zope.interface than ZopeInterface-3.1.0c1.  The
alsoProvides method was new to this release.  Are  you sure you really
have this release of zope.interface?

You are venturing into new territory - all of IPython1 has been
developed on Mac OS X/Linux/AIX.  It should work on Windows, but I am
not sure anyone has really testing everything out.  Have any other
Windows users tried IPython1?

The first step is to get the right version of zope.interface though
and see if that helps.  Let me know what you find and we can debug
these problems.

Brian





On 11/4/06, Kimball, John D CIV NSWCDL G33 <john.d.kimball at navy.mil> wrote:
>
>
>
> I installed ipython1 and dependecies in accordance with the INSTALL file.
> The unit tests fail, and I'm posting the problem here per the directions in
> the INSTALL file.
>
> Thanks for your assistance.  You all are doing awesome work!
>
> I'm using:
>
> enthon 1.0.0 (python 2.4.3)
> Windows 2000 SP4
> ZopeInterface-3.1.0c1
> The versions of Twisted and Nevow current as of 4PM 11-04-06.
>
> I get the following result:
>
> Running 29 tests.
>
> C:\Python24\lib\site-packages\twisted\trial\runner.py:243:
> exceptions.DeprecationWarning: log.startKeepingErrors is deprecated since
> Twisted 2.5
> ipython1.test.test_controllerpb
>   BasicControllerPBTest
>     testDeferreds ...
>             [ERROR]
>     cleanup errors
>         [ERROR]
>
> C:\Python24\lib\site-packages\twisted\trial\reporter.py:201:
> twisted.trial.reporter.BrokenTestCaseWarning: REACTOR
> UNCLEAN! traceback(s) follow:
> Traceback (most recent call last):
>   File
> "C:\Python24\Lib\site-packages\twisted\trial\unittest.py",
> line 651, in _classCleanUp
>     util._Janitor().postClassCleanup()
>   File
> "C:\Python24\Lib\site-packages\twisted\trial\util.py", line
> 68, in postClassCleanup
>     'cleanPending', 'cleanThreads')
>   File
> "C:\Python24\Lib\site-packages\twisted\trial\util.py", line
> 72, in _dispatch
>     getattr(self, "do_%s" % attr)()
>   File
> "C:\Python24\Lib\site-packages\twisted\trial\util.py", line
> 126, in do_cleanReactor
>     raise DirtyReactorError(' '.join(s))
> twisted.trial.util.DirtyReactorError: THIS WILL BECOME AN
> ERROR SOON! reactor left in unclean state, the following Selectables were
> left over:  <<class 'twisted.internet.tcp.Port'> of
> twisted.spread.pb.PBServerFactory on 10111> <<class
> 'twisted.internet.tcp.Client'> to ('127.0.0.1', 10111) at 18b1ab0> <Broker
> #0 on 10111>
>
>     testExecute ...
>             [ERROR]
>     testInterfaces ...
>             [ERROR]
>     testPullNamespace ...
>             [ERROR]
>     testPushPull ...
>             [ERROR]
>     testPushPullSerialized ...
>             [ERROR]
>     testResult ...
>             [ERROR]
>     testScatterGather ...
>             [ERROR]
> ipython1.test.test_controllerservice
>   BasicControllerServiceTest
>     testDeferreds ...
>             [ERROR]
>     testExecute ...
>             [ERROR]
>     testInterfaces ...
>             [ERROR]
>     testPullNamespace ...
>             [ERROR]
>     testPushPull ...
>             [ERROR]
>     testPushPullSerialized ...
>             [ERROR]
>     testResult ...
>             [ERROR]
>     testScatterGather ...
>             [ERROR]
>                                               [ERROR]
> ipython1.test.test_engineservice
>   BasicEngineServiceTest
>     testCompletedEmptyEngine ...
>             [ERROR]
>     testDeferreds ...
>                [OK]
>     testExecute ...
>                [OK]
>     testGetResult ...
>                [OK]
>     testInterfaces ...
>             [ERROR]
>     testPullNamespace ...
>                [OK]
>     testPushPull ...
>                [OK]
>     testPushPullSerialized ...
>                [OK]
>     testStatus ...
>                [OK]
>
> [ERROR]
> ipython1.test.test_shell
>   BasicShellTest
>     testCommand ...
>                [OK]
>     testExecute ...
>                [OK]
>     testPutGet ...
>                [OK]
>     testUpdate ...
>                [OK]
>
> ===============================================================================
> [ERROR]:
> ipython1.test.test_controllerpb.BasicControllerPBTest.testDeferreds
>
> Traceback (most recent call last):
>   File
> "C:\Python24\Lib\site-packages\ipython1\test\test_controllerpb.py",
> line 41, in setUp
>     self.addEngine(1)
>   File
> "C:\Python24\Lib\site-packages\ipython1\test\multienginetest.py",
> line 20, in addEngine
>     e = es.completeEngine(es.EngineService(InteractiveShell))
>   File
> "C:\Python24\Lib\site-packages\ipython1\kernel\engineservice.py",
> line 157, in completeEngine
>     zi.alsoProvides(engine, IEngineComplete)
> exceptions.AttributeError: 'module' object has no attribute 'alsoProvides'
> ===============================================================================
> [ERROR]:
> ipython1.test.test_controllerpb.BasicControllerPBTest.testExecute
>
> Traceback (most recent call last):
>   File
> "C:\Python24\Lib\site-packages\ipython1\test\test_controllerpb.py",
> line 36, in setUp
>     self.s = reactor.listenTCP(10111, self.sf)
>   File
> "C:\Python24\Lib\site-packages\twisted\internet\posixbase.py",
> line 386, in listenTCP
>     p.startListening()
>   File
> "C:\Python24\Lib\site-packages\twisted\internet\tcp.py",
> line 733, in startListening
>     raise CannotListenError, (self.interface, self.port, le)
> twisted.internet.error.CannotListenError: Couldn't listen
> on any:10111: (10048, 'Address already in use').
> ===============================================================================
> and so on... (same errors repeated)
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/ipython-dev
>
>
>


From john.d.kimball at navy.mil  Sat Nov  4 20:32:39 2006
From: john.d.kimball at navy.mil (Kimball, John D CIV NSWCDL G33)
Date: Sat, 4 Nov 2006 20:32:39 -0500
Subject: [IPython-dev] IPython1 unit tests fail
In-Reply-To: <6ce0ac130611041648i66e66711md5e641b38382398b@mail.gmail.com>
Message-ID: <E02A886D95ED09458A0BC124C790F9C402B3B4E6@NAEANRFKEX04VA.nadsusea.nads.navy.mil>


Brian;

THE TESTS NOW PASS!  I deleted the old zope.interface package and
reinstalled 3.1.0c1, and now all the tests pass.

I notice that the svn path in the INSTALL file checks out the Sumo
version of twisted, which claims to include zope interface (version
unspecified) though I couldn't find it in the source code.  Perhaps
setuptools was confused by the existing version?  I installed in the
order in the INSTALL file, and when the problem occurred I attempted to
reinstall zope.interface but apparently it wasn't installed correctly
until I deleted and reinstalled.

Thanks for the prompt response.

John

-----Original Message-----
From: Brian Granger [mailto:ellisonbg.net at gmail.com] 
Sent: Saturday, November 04, 2006 7:49 PM
To: Kimball, John D CIV NSWCDL G33
Cc: ipython-dev at scipy.org
Subject: Re: [IPython-dev] IPython1 unit tests fail


zi.alsoProvides(engine, IEngineComplete)
exceptions.AttributeError: 'module' object has no attribute
'alsoProvides'

This error from zope.interface makes me think that you have an older
version of zope.interface than ZopeInterface-3.1.0c1.  The alsoProvides
method was new to this release.  Are  you sure you really have this
release of zope.interface?

You are venturing into new territory - all of IPython1 has been
developed on Mac OS X/Linux/AIX.  It should work on Windows, but I am
not sure anyone has really testing everything out.  Have any other
Windows users tried IPython1?

The first step is to get the right version of zope.interface though and
see if that helps.  Let me know what you find and we can debug these
problems.

Brian





On 11/4/06, Kimball, John D CIV NSWCDL G33 <john.d.kimball at navy.mil>
wrote:
>
>
>
> I installed ipython1 and dependecies in accordance with the INSTALL 
> file. The unit tests fail, and I'm posting the problem here per the 
> directions in the INSTALL file.
>
> Thanks for your assistance.  You all are doing awesome work!
>
> I'm using:
>
> enthon 1.0.0 (python 2.4.3)
> Windows 2000 SP4
> ZopeInterface-3.1.0c1
> The versions of Twisted and Nevow current as of 4PM 11-04-06.
>
> I get the following result:
>
> Running 29 tests.
>
> C:\Python24\lib\site-packages\twisted\trial\runner.py:243:
> exceptions.DeprecationWarning: log.startKeepingErrors is deprecated 
> since Twisted 2.5 ipython1.test.test_controllerpb
>   BasicControllerPBTest
>     testDeferreds ...
>             [ERROR]
>     cleanup errors
>         [ERROR]
>
> C:\Python24\lib\site-packages\twisted\trial\reporter.py:201:
> twisted.trial.reporter.BrokenTestCaseWarning: REACTOR UNCLEAN! 
> traceback(s) follow: Traceback (most recent call last):
>   File
> "C:\Python24\Lib\site-packages\twisted\trial\unittest.py",
> line 651, in _classCleanUp
>     util._Janitor().postClassCleanup()
>   File
> "C:\Python24\Lib\site-packages\twisted\trial\util.py", line
> 68, in postClassCleanup
>     'cleanPending', 'cleanThreads')
>   File
> "C:\Python24\Lib\site-packages\twisted\trial\util.py", line
> 72, in _dispatch
>     getattr(self, "do_%s" % attr)()
>   File
> "C:\Python24\Lib\site-packages\twisted\trial\util.py", line
> 126, in do_cleanReactor
>     raise DirtyReactorError(' '.join(s))
> twisted.trial.util.DirtyReactorError: THIS WILL BECOME AN
> ERROR SOON! reactor left in unclean state, the following Selectables
were
> left over:  <<class 'twisted.internet.tcp.Port'> of
> twisted.spread.pb.PBServerFactory on 10111> <<class
> 'twisted.internet.tcp.Client'> to ('127.0.0.1', 10111) at 18b1ab0>
<Broker
> #0 on 10111>
>
>     testExecute ...
>             [ERROR]
>     testInterfaces ...
>             [ERROR]
>     testPullNamespace ...
>             [ERROR]
>     testPushPull ...
>             [ERROR]
>     testPushPullSerialized ...
>             [ERROR]
>     testResult ...
>             [ERROR]
>     testScatterGather ...
>             [ERROR]
> ipython1.test.test_controllerservice
>   BasicControllerServiceTest
>     testDeferreds ...
>             [ERROR]
>     testExecute ...
>             [ERROR]
>     testInterfaces ...
>             [ERROR]
>     testPullNamespace ...
>             [ERROR]
>     testPushPull ...
>             [ERROR]
>     testPushPullSerialized ...
>             [ERROR]
>     testResult ...
>             [ERROR]
>     testScatterGather ...
>             [ERROR]
>                                               [ERROR] 
> ipython1.test.test_engineservice
>   BasicEngineServiceTest
>     testCompletedEmptyEngine ...
>             [ERROR]
>     testDeferreds ...
>                [OK]
>     testExecute ...
>                [OK]
>     testGetResult ...
>                [OK]
>     testInterfaces ...
>             [ERROR]
>     testPullNamespace ...
>                [OK]
>     testPushPull ...
>                [OK]
>     testPushPullSerialized ...
>                [OK]
>     testStatus ...
>                [OK]
>
> [ERROR]
> ipython1.test.test_shell
>   BasicShellTest
>     testCommand ...
>                [OK]
>     testExecute ...
>                [OK]
>     testPutGet ...
>                [OK]
>     testUpdate ...
>                [OK]
>
> ======================================================================
> =========
> [ERROR]:
> ipython1.test.test_controllerpb.BasicControllerPBTest.testDeferreds
>
> Traceback (most recent call last):
>   File 
> "C:\Python24\Lib\site-packages\ipython1\test\test_controllerpb.py",
> line 41, in setUp
>     self.addEngine(1)
>   File 
> "C:\Python24\Lib\site-packages\ipython1\test\multienginetest.py",
> line 20, in addEngine
>     e = es.completeEngine(es.EngineService(InteractiveShell))
>   File 
> "C:\Python24\Lib\site-packages\ipython1\kernel\engineservice.py",
> line 157, in completeEngine
>     zi.alsoProvides(engine, IEngineComplete)
> exceptions.AttributeError: 'module' object has no attribute 
> 'alsoProvides' 
>
========================================================================
=======
> [ERROR]:
> ipython1.test.test_controllerpb.BasicControllerPBTest.testExecute
>
> Traceback (most recent call last):
>   File 
> "C:\Python24\Lib\site-packages\ipython1\test\test_controllerpb.py",
> line 36, in setUp
>     self.s = reactor.listenTCP(10111, self.sf)
>   File "C:\Python24\Lib\site-packages\twisted\internet\posixbase.py",
> line 386, in listenTCP
>     p.startListening()
>   File
> "C:\Python24\Lib\site-packages\twisted\internet\tcp.py",
> line 733, in startListening
>     raise CannotListenError, (self.interface, self.port, le)
> twisted.internet.error.CannotListenError: Couldn't listen
> on any:10111: (10048, 'Address already in use').
>
========================================================================
=======
> and so on... (same errors repeated)
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/ipython-dev
>
>
>


From alantrick at gmail.com  Sat Nov  4 20:53:30 2006
From: alantrick at gmail.com (Alan Trick)
Date: Sat, 04 Nov 2006 17:53:30 -0800
Subject: [IPython-dev] breaking readline
Message-ID: <1162691610.19366.12.camel@localhost>

Hi

I'm trying to create a wrapper for sys.stdout. I have some code (link at
bottom) and it works fine on the standards python console, but not so
well in ipython. In ipython it breaks readline so you end up seeing a
lot of weird charachters and certain keys stop working. It works fine in
code.InteractiveConsole. Anyone know what ipython is doing to cause
this.

n.b. I know my code has issues, like using open('/dev/stdout') and a few
other thing, but as it is it *should* work. It's mostly a proof off
concept

Link:
http://terminate.svn.sourceforge.net/viewvc/terminate/trunk/terminate/wrapper.py?revision=68&view=markup



From fperez.net at gmail.com  Sun Nov  5 14:25:31 2006
From: fperez.net at gmail.com (Fernando Perez)
Date: Sun, 5 Nov 2006 12:25:31 -0700
Subject: [IPython-dev] IPython.Extensions on sys.path
In-Reply-To: <454B6D18.9020500@livinglogic.de>
References: <4548ECEA.3020605@livinglogic.de>
	<46cb515a0611011330u2c7286e3t4c345be798925959@mail.gmail.com>
	<4549A59D.8060307@livinglogic.de>
	<46cb515a0611020030p1e6bee59rc571779dabcf2e35@mail.gmail.com>
	<db6b5ecc0611030014pf607d42w27d6eadd86008a8a@mail.gmail.com>
	<454B6D18.9020500@livinglogic.de>
Message-ID: <db6b5ecc0611051125q5e8f358oaa0b57a72659def4@mail.gmail.com>

On 11/3/06, Walter D?rwald <walter at livinglogic.de> wrote:
> Fernando Perez wrote:
>
> > [...]
> >
> > I just added this code to the main ipython starter script:
> >
> > # Start by cleaning up sys.path: Python automatically inserts the script's
> > # base directory into sys.path, at the front.  This can lead to unpleasant
> > # surprises.
> > import sys
> > sys.path.pop(0)
> >
> > Now, the front of sys.path is identical for a plain python and an
> > ipython shell.  The only difference is that under ipython, the very
> > last two entries are:
> >
> > '/home/fperez/usr/lib/python2.4/site-packages/IPython/Extensions',
> > '/home/fperez/.ipython']
> >
> > which don't exist for plain Python.  I don't see these as a big problem,
> > though.
> >
> > If others disagree, let us know.
>
> This breaks running the start script directly. With the current version
> I get a sys.path that looks like this:

Mmh, I also re-checked, and found that plain python also adds that
directory to sys.path.  I must have been mistaken before.  I'll revert
this change now then, to avoid problems.

We should have a very clear policy on how we manipulate sys.path.
Currently (once I revert the sys.path.pop() change) we have:

1. At startup, ipython's sys.path matches a plain python shell, except
for the addition at the end of:

 '/home/fperez/usr/lib/python2.4/site-packages/IPython/Extensions',
 '/home/fperez/.ipython'

Since these two go at the very end, they don't worry me too much: they
are a convenience for interactive configuration and by being at the
end, they can't shadow anything in the default sys.path.  I could see
removing the last one to be a bit more minimalistic, but I'm not sure
it's worth it.

2. There's a more delicate issue: sys.path changes from %run.  In
order to mimic the running of scripts from a system prompt, %run used
to do this:

2384	 	        # find things also in current directory
2385	 	        dname = os.path.dirname(fname)
2386	 	        if not sys.path.count(dname):
2387	 	            sys.path.append(dname)

which I removed in r1874.  I then realized that this is needed so that
scripts can find local things, but because ipython persists the state
of sys.path across runs (something that simply does not occur in a
plain python run from the command line), these changes 'contaminate'
sys.path for the rest of the session.

I'm going to add this back, but in a modified form that will try to
intelligently revert out any changes made to sys.path. I will leave
any changes actually made by the user to persist, in case people have
scripts they use to manipulate sys.path for further interactive use.

Any feedback you may have on this is welcome.

Cheers,

f


From python at koepsell.de  Tue Nov  7 14:08:18 2006
From: python at koepsell.de (Kilian Koepsell)
Date: Tue, 7 Nov 2006 11:08:18 -0800
Subject: [IPython-dev] ipython unit test failed
Message-ID: <65235DB8-FA1A-4E1E-A4AA-E3A8D690CDC9@koepsell.de>

hi,

i just compiled and installed ipython1 on mac-intel.
the 'trial ipython1' command gives the following two errors:

> ====================================================================== 
> =========
> [ERROR]: ipython1.test.test_enginepb
>
>   File "/sw/lib/python2.5/site-packages/twisted/python/reflect.py",  
> line 352, in namedAny
>     topLevelPackage = __import__(trialname)
> <type 'exceptions.SyntaxError'>: Non-ASCII character '\xe2' in  
> file /data/ipython1/ipython1/test/test_enginepb.py on line 7, but  
> no encoding declared; see http://www.python.org/peps/pep-0263.html  
> for details (test_enginepb.py, line 6)
> ====================================================================== 
> =========
> [ERROR]: ipython1.test.test_enginevanilla
>
>   File "/sw/lib/python2.5/site-packages/twisted/python/reflect.py",  
> line 352, in namedAny
>     topLevelPackage = __import__(trialname)
> <type 'exceptions.SyntaxError'>: Non-ASCII character '\xe2' in  
> file /data/ipython1/ipython1/test/test_enginevanilla.py on line 6,  
> but no encoding declared; see http://www.python.org/peps/ 
> pep-0263.html for details (test_enginevanilla.py, line 5)
> ---------------------------------------------------------------------- 
> ---------
> Ran 29 tests in 0.481s
>
> FAILED (errors=2, successes=29)

i am using the following versions:
mac osx 10.4.8
python 2.5
ipython 0.7.2
ipython1 from svn
zopeinterface 3.1.0c1
twisted 2.4.0

any idea what's going wrong?

thanks,
   kilian




From fperez.net at gmail.com  Tue Nov  7 14:47:49 2006
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 7 Nov 2006 12:47:49 -0700
Subject: [IPython-dev] ipython unit test failed
In-Reply-To: <65235DB8-FA1A-4E1E-A4AA-E3A8D690CDC9@koepsell.de>
References: <65235DB8-FA1A-4E1E-A4AA-E3A8D690CDC9@koepsell.de>
Message-ID: <db6b5ecc0611071147h1d1e30a8i4c00057ca4728f70@mail.gmail.com>

On 11/7/06, Kilian Koepsell <python at koepsell.de> wrote:

> i am using the following versions:

> python 2.5

> any idea what's going wrong?

Yup, unfortunately the line above :(

Says Glyf (Twisted lead developer):

http://glyf.livejournal.com/62308.html

The take-home part is:

There's no release of Twisted that works with Python 2.5 yet.

Sorry, but that's outside of our scope.  Obviously they are aware of
the issue, so I hope it will get resolved soon.

I'll add a note to this effect on the wiki so others don't fall in the
same trap.

Cheers,

f


From vivainio at gmail.com  Wed Nov  8 06:02:30 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Wed, 8 Nov 2006 13:02:30 +0200
Subject: [IPython-dev] regarding usefullness of ipconfig() & all
Message-ID: <46cb515a0611080302t51bec859l46a5f498ddaa1fdb@mail.gmail.com>

Just noted the added ipconfig() function and related magic:

- _ip.options already exists for this, and can be directly manipulated.

- I don't think we should have ipANYTHING in user namespace. They are
there for backwards compatibility, bet the documented approach should
be _ip.ANYTHING. I.e. ipmagic -> _ip.magic, etc.

_ip, of course, is the ipapi object.

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From fullung at gmail.com  Wed Nov  8 18:52:18 2006
From: fullung at gmail.com (Albert Strasheim)
Date: Thu, 9 Nov 2006 01:52:18 +0200
Subject: [IPython-dev] Issue in error path in case of invalid command
Message-ID: <01ce01c70390$ec9ffb70$0a83a8c0@ratbert>

Hello all

I'm back to playing with IPython1 on Windows again. As per the instructions 
in

http://projects.scipy.org/ipython/ipython/browser/ipython/branches/chainsaw/README

I started up my controller and my engine. In a script I did:

import ipython1.kernel.api as kernel
rc = kernel.RemoteController(('localhost', 10201))
print rc.getIDs()

When I run this, I get the following error:

2006/11/09 01:45 +0200 [VanillaEngineServerProtocol,1,127.0.0.1] Unhandled 
Error
        Traceback (most recent call last):
          File "C:\Python24\Lib\site-packages\twisted\python\log.py", line 
48, in callWithLogger
            return callWithContext({"system": lp}, func, *args, **kw)
          File "C:\Python24\Lib\site-packages\twisted\python\log.py", line 
33, in callWithContext
            return context.call({ILogContext: newCtx}, func, *args, **kw)
          File "C:\Python24\Lib\site-packages\twisted\python\context.py", 
line 59, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, 
**kw)
          File "C:\Python24\Lib\site-packages\twisted\python\context.py", 
line 37, in callWithContext
            return func(*args,**kw)
        --- <exception caught here> ---
          File 
"C:\Python24\Lib\site-packages\twisted\internet\selectreactor.py", line 139, 
in _doReadOrWrite
            why = getattr(selectable, method)()
          File "C:\Python24\Lib\site-packages\twisted\internet\tcp.py", line 
362, in doRead
            return self.protocol.dataReceived(data)
          File "C:\Python24\Lib\site-packages\twisted\protocols\basic.py", 
line 99, in dataReceived
            self.doData()
          File "C:\Python24\Lib\site-packages\twisted\protocols\basic.py", 
line 62, in doData
            self.stringReceived(self.__buffer)
          File 
"C:\Python24\lib\site-packages\ipython1\kernel\enginevanilla.py", line 558, 
in stringReceived
            self.nextHandler(msg)
          File 
"C:\Python24\lib\site-packages\ipython1\kernel\enginevanilla.py", line 576, 
in dispatch
            self.dieLoudly('Command could not be dispatched: ' + msg)
          File 
"C:\Python24\lib\site-packages\ipython1\kernel\enginevanilla.py", line 588, 
in dieLoudly
            log.msg('Protocol Error [%i]: ' % id + repr(a))
        exceptions.TypeError: int argument required

2006/11/09 01:45 +0200 [VanillaEngineServerProtocol,1,127.0.0.1] 
unregistered engine None
2006/11/09 01:45 +0200 [VanillaEngineServerProtocol,1,127.0.0.1] Unhandled 
Error
        Traceback (most recent call last):
          File "C:\Python24\Scripts\ipcontroller.py", line 51, in main
            reactor.run()
          File 
"C:\Python24\Lib\site-packages\twisted\internet\posixbase.py", line 218, in 
run
            self.mainLoop()
          File 
"C:\Python24\Lib\site-packages\twisted\internet\posixbase.py", line 229, in 
mainLoop
            self.doIteration(t)
          File 
"C:\Python24\Lib\site-packages\twisted\internet\selectreactor.py", line 133, 
in doSelect
            _logrun(selectable, _drdw, selectable, method, dict)
        --- <exception caught here> ---
          File "C:\Python24\Lib\site-packages\twisted\python\log.py", line 
48, in callWithLogger
            return callWithContext({"system": lp}, func, *args, **kw)
          File "C:\Python24\Lib\site-packages\twisted\python\log.py", line 
33, in callWithContext
            return context.call({ILogContext: newCtx}, func, *args, **kw)
          File "C:\Python24\Lib\site-packages\twisted\python\context.py", 
line 59, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, 
**kw)
          File "C:\Python24\Lib\site-packages\twisted\python\context.py", 
line 37, in callWithContext
            return func(*args,**kw)
          File 
"C:\Python24\Lib\site-packages\twisted\internet\selectreactor.py", line 149, 
in _doReadOrWrite
            self._disconnectSelectable(selectable, why, method=="doRead")
          File 
"C:\Python24\Lib\site-packages\twisted\internet\posixbase.py", line 256, in 
_disconnectSelectable
            selectable.connectionLost(failure.Failure(why))
          File "C:\Python24\Lib\site-packages\twisted\internet\tcp.py", line 
416, in connectionLost
            protocol.connectionLost(reason)
          File 
"C:\Python24\lib\site-packages\ipython1\kernel\enginevanilla.py", line 539, 
in connectionLost
            self.factory.unregisterEngine(self.id)
          File 
"C:\Python24\lib\site-packages\ipython1\kernel\enginevanilla.py", line 1026, 
in unregisterEngine
            return self.service.unregisterEngine(id)
          File 
"C:\Python24\Lib\site-packages\ipython1\kernel\controllerservice.py", line 
304, in unregisterEngine
            log.msg("engine %i was not registered" % id)
        exceptions.TypeError: int argument required

In this case, id is None and repr(a) is 'Command could not be dispatched: 
GETIDS'.

Cheers,

Albert 



From fullung at gmail.com  Wed Nov  8 18:58:57 2006
From: fullung at gmail.com (Albert Strasheim)
Date: Thu, 9 Nov 2006 01:58:57 +0200
Subject: [IPython-dev] Issue in error path in case of invalid command
References: <01ce01c70390$ec9ffb70$0a83a8c0@ratbert>
Message-ID: <01e101c70391$daee9ac0$0a83a8c0@ratbert>

Hello all

> I'm back to playing with IPython1 on Windows again. As per the 
> instructions
> in
>
> http://projects.scipy.org/ipython/ipython/browser/ipython/branches/chainsaw/README
>
> I started up my controller and my engine. In a script I did:
>
> import ipython1.kernel.api as kernel
> rc = kernel.RemoteController(('localhost', 10201))
> print rc.getIDs()
>
> When I run this, I get the following error:

I should probably mention that I'm using IPython1 + dependencies fresh (as 
of right now) from SVN.

Cheers,

Albert 



From fperez.net at gmail.com  Wed Nov  8 19:14:19 2006
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 8 Nov 2006 17:14:19 -0700
Subject: [IPython-dev] Issue in error path in case of invalid command
In-Reply-To: <01ce01c70390$ec9ffb70$0a83a8c0@ratbert>
References: <01ce01c70390$ec9ffb70$0a83a8c0@ratbert>
Message-ID: <db6b5ecc0611081614v1713fe13ycea429d9c2ba1bf2@mail.gmail.com>

On 11/8/06, Albert Strasheim <fullung at gmail.com> wrote:
> Hello all
>
> I'm back to playing with IPython1 on Windows again. As per the instructions
> in
>
> http://projects.scipy.org/ipython/ipython/browser/ipython/branches/chainsaw/README
>
> I started up my controller and my engine. In a script I did:
>
> import ipython1.kernel.api as kernel
> rc = kernel.RemoteController(('localhost', 10201))
> print rc.getIDs()

Quick question (Brian may have more ideas later). 10201 is not the
default port, did you start the controller on that port?

I doubt any of us has access to a windows box with enough development
tools to test this, unfortunately.   Up until now I've been able to do
lightweight testing of ipython trunk under windows because it's pure
python, but now we use things that require compilers and scary things
like that.  My windows VMware instance is basically a tax-preparation
tool that runs one or two programs, and I don't have a real
development setup there.

I hope this is a problem that we can track from within a *nix box...

Cheers,

f


From fullung at gmail.com  Wed Nov  8 19:57:40 2006
From: fullung at gmail.com (Albert Strasheim)
Date: Thu, 9 Nov 2006 02:57:40 +0200
Subject: [IPython-dev] Issue in error path in case of invalid command
References: <01ce01c70390$ec9ffb70$0a83a8c0@ratbert>
	<6ce0ac130611081615x690a4d99p3487296572884731@mail.gmail.com>
Message-ID: <000401c7039a$0e83bcf0$0a83a8c0@ratbert>

Thanks, my bad.

Still seems like this points to issues in the lines

log.msg('Protocol Error [%i]: ' % id + repr(a))

and

log.msg("engine %i was not registered" % id)

since by connecting to this wrong port, id was None in both cases, which 
caused the TypeError to be raised.

Cheers,

Albert

----- Original Message ----- 
From: "Brian Granger" <ellisonbg.net at gmail.com>
To: "Albert Strasheim" <fullung at gmail.com>
Sent: Thursday, November 09, 2006 2:15 AM
Subject: Re: [IPython-dev] Issue in error path in case of invalid command


> You are connecting on the wrong port.  10201 is the port the engine
> connect on.  You want port 10105.  If you are running the controler on
> localhost you can simply do:
>
> rc = kernel.RemoteController(kernel.defaultController)
>
> and it will do the right thing.



From fullung at gmail.com  Wed Nov  8 20:07:35 2006
From: fullung at gmail.com (Albert Strasheim)
Date: Thu, 9 Nov 2006 03:07:35 +0200
Subject: [IPython-dev] RemoteController.test issue
Message-ID: <000e01c7039b$70d63850$0a83a8c0@ratbert>

Hello all

With the right port in place, I can get the IDs of my two engines. However, 
when I run the test method of the RemoteController, I get:

The IPython Controller magics only work within IPython.
Connecting to controller:  ('127.0.0.1', 10105)
[0, 1]
### here I run rc.test()
Running tests on 2 engines
Testing execute...
?[0;32m[127.0.0.1:0]?[0;34m In [140]:?[0m a
?[0;32m[127.0.0.1:0]?[0;31m Out[140]:?[0m 1
?[0;32m[127.0.0.1:0]?[0;34m In [141]:?[0m print a
?[0;32m[127.0.0.1:0]?[0;31m Out[141]:?[0m 1
?[0;32m[127.0.0.1:1]?[0;34m In [45]:?[0m print a
?[0;32m[127.0.0.1:1]?[0;31m Out[45]:?[0m 1
?[0;32m[127.0.0.1:0]?[0;34m In [142]:?[0m b-3
?[0;32m[127.0.0.1:0]?[0;31m Out[142]:?[0m -1
?[0;32m[127.0.0.1:1]?[0;34m In [46]:?[0m b-3
?[0;32m[127.0.0.1:1]?[0;31m Out[46]:?[0m -1
Need something to do!
Need something to do!
execute OK
Testing push/pull
Need something to do!
Need something to do!
push/pull FAIL:  list index out of range
Traceback (most recent call last):
  File "C:\home\albert\work2\pyspkrec\frags\frags9.py", line 4, in ?
    rc.test()
  File "C:\Python24\Lib\site-packages\ipython1\kernel\controllerclient.py", 
line 171, in test
    self[1:max(ids)][0]['qwert'] = 3
  File "C:\Python24\Lib\site-packages\ipython1\kernel\controllerclient.py", 
line 336, in __getitem__
    return EngineProxy(self.rc, self._originalIDs[id])
IndexError: list index out of range

Does this test work for you guys?

Cheers,

Albert 



From fperez.net at gmail.com  Wed Nov  8 20:42:48 2006
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 8 Nov 2006 18:42:48 -0700
Subject: [IPython-dev] RemoteController.test issue
In-Reply-To: <000e01c7039b$70d63850$0a83a8c0@ratbert>
References: <000e01c7039b$70d63850$0a83a8c0@ratbert>
Message-ID: <db6b5ecc0611081742g249c9828yd8525ae25f9c932d@mail.gmail.com>

On 11/8/06, Albert Strasheim <fullung at gmail.com> wrote:
> Hello all
>
> With the right port in place, I can get the IDs of my two engines. However,
> when I run the test method of the RemoteController, I get:

> Does this test work for you guys?

I get the same exact failure as you do.  We'll look into it.

You should be able to run things from the tutorial, for example.  I'm
starting to use this stuff in production work as well, so keep the
complaints coming :)  Since I'm on the same boat, we'll be fixing them
as they come.

Cheers,

f


From fullung at gmail.com  Wed Nov  8 22:25:57 2006
From: fullung at gmail.com (Albert Strasheim)
Date: Thu, 9 Nov 2006 05:25:57 +0200
Subject: [IPython-dev] pushAll and executeAll speed issue
Message-ID: <009d01c703ae$c5bc6700$0a83a8c0@ratbert>

Hello all

I'm doing some experiments with IPython1 on Windows. I'm using DeinoMPI's 
GUI on top of mpi4py to start two kernels on my Core 2 Duo system. They 
connect to the controller without problems. I make a RemoteController and 
off I go. This is too cool.

(By the way: as I understand it, MPI is currently only being used to start 
the kernels. From what I can see, it is not yet being used inside things 
like pushAll, etc. Is this the next step to use the MPI functions if they 
are available? Anybody working on this? If not, maybe I can try my hand at 
it. Any hints or suggestions on where to start appreciated.)

Anyway, back to my experiment. The script I'm using is attached.

It starts off with some local benchmarks running on a single core. I get:

cPickle dumps time 4.95649055955
cPickle loads time 2.39126201794
local map time 10.3841574837

Then I push to all and time it. Here I get:

total pushAll time 70.6287321433

I was expecting something proportional to dumpsTime+N*loadsTime (if I 
understand correctly that it is serially pushing out the data to each of the 
N kernels) or maybe even less. I noticed that this section gets dramatically 
slower even with a relatively small increase in data size. While this is 
busy, I see Python using up all the CPU on one core as expected, but I also 
see that the Windows "System" process is very busy.

Next up, the timing over the executeAll call that does the map:

executeAll time for map 139.186242386

but the kernels print:

engine map time 12.8248084603
engine map time 11.3900467797

which is only slightly more than the single core time. However, I have to 
wonder where the heck the rest of the time went?! :-)

Do you guys see similar timings on Linux? Any thoughts on where my time is 
going? Could it be Twisted? I tried with the latest from SVN and the 
official 2.4.0 release for Windows.

Cheers,

Albert 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: frags9.py
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20061109/d17ae91c/attachment.ksh>

From ellisonbg.net at gmail.com  Wed Nov  8 22:50:18 2006
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 8 Nov 2006 20:50:18 -0700
Subject: [IPython-dev] Issue in error path in case of invalid command
In-Reply-To: <000401c7039a$0e83bcf0$0a83a8c0@ratbert>
References: <01ce01c70390$ec9ffb70$0a83a8c0@ratbert>
	<6ce0ac130611081615x690a4d99p3487296572884731@mail.gmail.com>
	<000401c7039a$0e83bcf0$0a83a8c0@ratbert>
Message-ID: <6ce0ac130611081950r69d568d7t36302a848e5a53e7@mail.gmail.com>

Even though you were connecting to the wrong port we need to add some
error checking code so that the controller doesn't die and the error
messages are more understandable.

Brian

On 11/8/06, Albert Strasheim <fullung at gmail.com> wrote:
> Thanks, my bad.
>
> Still seems like this points to issues in the lines
>
> log.msg('Protocol Error [%i]: ' % id + repr(a))
>
> and
>
> log.msg("engine %i was not registered" % id)
>
> since by connecting to this wrong port, id was None in both cases, which
> caused the TypeError to be raised.
>
> Cheers,
>
> Albert
>
> ----- Original Message -----
> From: "Brian Granger" <ellisonbg.net at gmail.com>
> To: "Albert Strasheim" <fullung at gmail.com>
> Sent: Thursday, November 09, 2006 2:15 AM
> Subject: Re: [IPython-dev] Issue in error path in case of invalid command
>
>
> > You are connecting on the wrong port.  10201 is the port the engine
> > connect on.  You want port 10105.  If you are running the controler on
> > localhost you can simply do:
> >
> > rc = kernel.RemoteController(kernel.defaultController)
> >
> > and it will do the right thing.
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/ipython-dev
>


From ellisonbg.net at gmail.com  Wed Nov  8 23:01:55 2006
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 8 Nov 2006 21:01:55 -0700
Subject: [IPython-dev] RemoteController.test issue
In-Reply-To: <db6b5ecc0611081742g249c9828yd8525ae25f9c932d@mail.gmail.com>
References: <000e01c7039b$70d63850$0a83a8c0@ratbert>
	<db6b5ecc0611081742g249c9828yd8525ae25f9c932d@mail.gmail.com>
Message-ID: <6ce0ac130611082001r540c4400mc8dc2d76bf0fde41@mail.gmail.com>

The issue is that the test() method requires at least 4 engines to
run.  I just committed a change to trunk that tests how many engines
you have running and raises an exception if you don't have at least 4.
 It prints an error message indicating what to do.

The tests should pass with 4 engines running.

Cheers,

Brian

On 11/8/06, Fernando Perez <fperez.net at gmail.com> wrote:
> On 11/8/06, Albert Strasheim <fullung at gmail.com> wrote:
> > Hello all
> >
> > With the right port in place, I can get the IDs of my two engines. However,
> > when I run the test method of the RemoteController, I get:
>
> > Does this test work for you guys?
>
> I get the same exact failure as you do.  We'll look into it.
>
> You should be able to run things from the tutorial, for example.  I'm
> starting to use this stuff in production work as well, so keep the
> complaints coming :)  Since I'm on the same boat, we'll be fixing them
> as they come.
>
> Cheers,
>
> f
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/ipython-dev
>


From fperez.net at gmail.com  Wed Nov  8 23:29:09 2006
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 8 Nov 2006 21:29:09 -0700
Subject: [IPython-dev] pushAll and executeAll speed issue
In-Reply-To: <009d01c703ae$c5bc6700$0a83a8c0@ratbert>
References: <009d01c703ae$c5bc6700$0a83a8c0@ratbert>
Message-ID: <db6b5ecc0611082029j1678376ucc26296e1d405b75@mail.gmail.com>

Hey Albert,

Brian will probably pitch in later with more comments, since he knows
the underbelly of this code better than I do.  But in the meantime...

On 11/8/06, Albert Strasheim <fullung at gmail.com> wrote:
> Hello all
>
> I'm doing some experiments with IPython1 on Windows. I'm using DeinoMPI's
> GUI on top of mpi4py to start two kernels on my Core 2 Duo system. They
> connect to the controller without problems. I make a RemoteController and
> off I go. This is too cool.
>
> (By the way: as I understand it, MPI is currently only being used to start
> the kernels. From what I can see, it is not yet being used inside things
> like pushAll, etc. Is this the next step to use the MPI functions if they
> are available? Anybody working on this? If not, maybe I can try my hand at
> it. Any hints or suggestions on where to start appreciated.)

There is MPI code in there already, but currently it is not being used
for many core things.  We should intelligently take advantage of it
when present, while gracefully falling back to pickle when MPI is not
there.

> Anyway, back to my experiment. The script I'm using is attached.
>
> It starts off with some local benchmarks running on a single core. I get:
>
> cPickle dumps time 4.95649055955
> cPickle loads time 2.39126201794
> local map time 10.3841574837
>
> Then I push to all and time it. Here I get:
>
> total pushAll time 70.6287321433
>
> I was expecting something proportional to dumpsTime+N*loadsTime (if I
> understand correctly that it is serially pushing out the data to each of the
> N kernels) or maybe even less. I noticed that this section gets dramatically
> slower even with a relatively small increase in data size. While this is
> busy, I see Python using up all the CPU on one core as expected, but I also
> see that the Windows "System" process is very busy.
>
> Next up, the timing over the executeAll call that does the map:
>
> executeAll time for map 139.186242386
>
> but the kernels print:
>
> engine map time 12.8248084603
> engine map time 11.3900467797
>
> which is only slightly more than the single core time. However, I have to
> wonder where the heck the rest of the time went?! :-)
>
> Do you guys see similar timings on Linux? Any thoughts on where my time is
> going? Could it be Twisted? I tried with the latest from SVN and the
> official 2.4.0 release for Windows.

First, a numpy aside.  Note that I had to change the original line

x = N.arange...

for x = range(), because otherwise I was getting zillions of

Warning: overflow encountered in long_scalars

It seems to come from this: numpy long scalars don't gracefully coerce
to Python longs:

In [7]: x[-1]
Out[7]: 99999

In [8]: x[-1]*x[-1]
Warning: overflow encountered in long_scalars
Out[8]: 1409865409

while the answer should be:

In [22]: y=99999

In [23]: y*y
Out[23]: 9999800001L

But that's for the numpy list :)


Here's what is going on, most likely. The size of that enormous
pickled list is very large:

In [24]: x = range(10000000)

In [25]: len(cPickle.dumps(x))
Out[25]: 98888896

That's a 100MB string, that needs to be shipped over the wire.  At
this point, we are probably being hurt by Twisted, which makes
unnecessary copies of very large objects when transmitting them.  I
know Brian and Min spent some time looking at this, but I'm not sure
how far they got; I think it wasn't too easy to fix directly, though
it /will/ need to be done eventually.

I see two possibilities:

a) Finding how to make Twisted-based sending of pickled strings as
efficient as possible, and with minimal copies.

b) Taking better advantage of MPI if available, so that the string
buffer is sent using MPI directly.

I see fairly similar results to yours:

In [10]: run frags9.py
('127.0.0.1', 10105)
Connecting to controller:  ('127.0.0.1', 10105)
[0, 1]
cPickle dumps time 5.15
cPickle loads time 5.92
local map time 7.06
[127.0.0.1:0] In [29]: import time
[127.0.0.1:1] In [22]: import time
pushing to all...
total pushAll time 1.24
[127.0.0.1:0] In [30]: t = time.clock()
[127.0.0.1:1] In [23]: t = time.clock()
executing on all...
[127.0.0.1:0] In [31]: z = map(lambda y:y*y, x)
[127.0.0.1:1] In [24]: z = map(lambda y:y*y, x)
executeAll time for map 0.0
[127.0.0.1:0] In [32]: t = time.clock() - t
[127.0.0.1:1] In [25]: t = time.clock() - t
total time for push and execute 1.24
[127.0.0.1:0] In [33]: print "engine map time", t
[127.0.0.1:0] Out[33]: engine map time 7.16
[127.0.0.1:1] In [26]: print "engine map time", t
[127.0.0.1:1] Out[26]: engine map time 7.19
(0, 33, 'print "engine map time", t', 'engine map time 7.16\n', '')
(1, 26, 'print "engine map time", t', 'engine map time 7.19\n', '')

I also (under linux) saw an enormous amount of CPU usage from the
kernel; that's typical of large memory allocations being done.
Further, by the end of the runs my 2GB box was almost out of RAM.


For usage of read-only data in multicore machines, it would also be
nice to expose a shared-memory interface, but I'm not really up to
speed on the details of doing that, and whether Python exposes an API
to allow it to be done portably.

Basically, I think that we're starting to see where the performance
bottlenecks are.  This is pre-alpha, after all ;)

We'll keep working on it...

Cheers,

f


From ellisonbg.net at gmail.com  Thu Nov  9 01:14:41 2006
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 8 Nov 2006 23:14:41 -0700
Subject: [IPython-dev] pushAll and executeAll speed issue
In-Reply-To: <009d01c703ae$c5bc6700$0a83a8c0@ratbert>
References: <009d01c703ae$c5bc6700$0a83a8c0@ratbert>
Message-ID: <6ce0ac130611082214q6dfba628od9a3b2d2d3347ea5@mail.gmail.com>

> I'm doing some experiments with IPython1 on Windows. I'm using DeinoMPI's
> GUI on top of mpi4py to start two kernels on my Core 2 Duo system. They
> connect to the controller without problems. I make a RemoteController and
> off I go. This is too cool.

Yes, cool, but it is also dangerous.....read on...

> (By the way: as I understand it, MPI is currently only being used to start
> the kernels. From what I can see, it is not yet being used inside things
> like pushAll, etc. Is this the next step to use the MPI functions if they
> are available? Anybody working on this? If not, maybe I can try my hand at
> it. Any hints or suggestions on where to start appreciated.)

There are two ways one might way to use MPI:

1.  The send data between engines.  This is supported already through
the usage of various python mpi bindings like mpi4py.  Our docs have
some details on how to get started with this, but it can be subtle
depending on your mpi implementation etc.  Please feel free to bug us
about this.

2.  To send data between the client (your python script) and the
engines.  At first this may seem attractive, but it is not really what
you want to do.  The reason is that then you IPython/Python script
also has to be part of the MPi universe.  That make it impossible to
do many important things:

 - Run your script on your laptop, but run the engines/controller on a
remote cluster.
 - Disconnect/reconnect to/from the controller.
 - multiple users connecting to the controller

Because of these reason, we feel that it is a "feature" that MPi is
not used to send data between the client (your python script or an
interactive session) and the engines.

With that said, you shouldn't be sending lots of data between the
client and engines anyway - see below for more discussion of this...

> Anyway, back to my experiment. The script I'm using is attached.
>
> It starts off with some local benchmarks running on a single core. I get:
>
> cPickle dumps time 4.95649055955
> cPickle loads time 2.39126201794
> local map time 10.3841574837
>
> Then I push to all and time it. Here I get:
>
> total pushAll time 70.6287321433
>
> I was expecting something proportional to dumpsTime+N*loadsTime (if I
> understand correctly that it is serially pushing out the data to each of the
> N kernels) or maybe even less. I noticed that this section gets dramatically
> slower even with a relatively small increase in data size. While this is
> busy, I see Python using up all the CPU on one core as expected, but I also
> see that the Windows "System" process is very busy.
>
> Next up, the timing over the executeAll call that does the map:
>
> executeAll time for map 139.186242386
>
> but the kernels print:
>
> engine map time 12.8248084603
> engine map time 11.3900467797
>
> which is only slightly more than the single core time. However, I have to
> wonder where the heck the rest of the time went?! :-)
>
> Do you guys see similar timings on Linux? Any thoughts on where my time is
> going? Could it be Twisted? I tried with the latest from SVN and the
> official 2.4.0 release for Windows.

The current incarnation of your script has a few problems:

1.  It creates multiple copies of a large object on 4 processes on
your system (client, controller, and 2 engines).  Even though the
object is "not that big" when you have multple copies around the
memory usage gets out of controller.  Fernando was seeing his 2 Gb
system nearly maxed out.
It is even worse though than just 4 copies.  If you go through your
script you will see that the client and each engine has 2-3 copies of
the object.  For certain periods of time, the controller can also have
multiple copies.  Most of these problems (for objects the size of
yours) would go away if the controller and engines were all running on
different systems.

2.  There is no real parallelism being expressed.  This is because
pushAll sends the entire object to each engine.  Are you meaning to
use scatterAll?  This partitions a sequence and sends the partitions
to each engine.

3.  One of the golden rules of parallel computing is "never send
something over the wire that can easily be recreated in place"  The
following code is much better:

rc.execute(0,'a = N.arange(0,50000)')
rc.execute(1,'a = N.arnage(50000, 100000)')
rc.executeAll('b = a*a')
result = gatherAll('b')                             # Now gather the
result locally

With this you have full parallelism and only three small strings (not
the actual objects) are sent over the wire.  Code like this will scale
to many processes.

We have really tried to make it easy to send objects between the
client and engines.  But, this usage pattern is not something we are
used to.  By this, I mean it is *very* easy (I have done it many times
with IPython1 :)) to get carried away with how easy it is to send
things around and forget about the realities of what we are really
doing underneath the hood.

With that said, we are very aware of a need to improve how IPython
handles these cases.  There are lots of optimizaitons we can and need
to do and we also need to provide some protections that prevent crazy
things from being done too easily.  We have out work cut out or us and
having users like yourself hammering on things makes our job easier.

Cheers,

Brian


From fullung at gmail.com  Thu Nov  9 17:50:42 2006
From: fullung at gmail.com (Albert Strasheim)
Date: Fri, 10 Nov 2006 00:50:42 +0200
Subject: [IPython-dev] Suggestions for implementing parallel algorithms?
Message-ID: <004e01c70451$7c37eb10$0a83a8c0@ratbert>

Hello all

I'm getting started with using IPython1 and mpi4py for implementing parallel 
algorithms for speaker verification. I'm mostly dealing with data from the 
NIST Speaker Recognition Evaluation, which entails thousands of speaker 
models to train and test.

There are two expectation-maximization algorithms that are of interest to 
me: k-means clustering and Gaussian Mixture Model training. Both of these 
can be implemented in parallel by scattering the training data over a bunch 
of nodes, calculating some statistics, combining the stats in some way and 
repeating this process for a maximum number of iterations or until 
convergence is attained.

I currently have these algorithms implemented on top of NumPy. For every 
algorithm I have a pure-Python version. The Python class implements the 
public interface and does argument checking and other housekeeping and the 
defers to "private" methods to get the real work done. To get some decent 
speed, some of these "private" methods also have C implementations. To get a 
fast version of the algorithm, I mix a class containing only these "private" 
methods that call through to C into my pure-Python class. As an example, in 
the k-means case, this means I end up with two classes, KMeansEstimator 
(pure Python) and CKMeansEstimator (Python on top with some C mixed in).

I would like to adapt these algorithms to run in parallel using IPython1.

Some details about my problem: for training my speaker models, I can simply 
train a number of speaker models per node. This parallelises easily --  
different nodes do different speakers. However, for training my world model, 
I would like to involve all the nodes to work on the same model. This is 
necessary because to train the world model, I have tens to hunderds of hours 
of speech whereas the speaker models are adapted from the world model using 
only a few seconds to a few minutes of speech.

A naive way to implement this in parallel would be to copy my existing 
implementation and call RemoteController's scatterAll/executeAll/gatherAll 
in places where the original algorithm does the loop over the data (the 
expectation step).

I'm hoping I can avoid this duplication. My first idea is to make something 
like a LocalController that implements IPython1's IController interface in a 
way that makes sense for single node operation. This way, I can implement my 
algorithm once in terms of IController operations, test it easily, and later 
by simply setting a controller property on a instance of the class 
implementing the algorithm, decide whether it runs on a single node or in 
parallel.

How do you guys handle these issues in your code? Any suggestions would be 
appreciated.

Cheers,

Albert

P.S. For an example implementation of k-means on top of MPI, see 
http://www.cs.umn.edu/~mnjoshi/PKMeans.pdf 



From ellisonbg.net at gmail.com  Fri Nov 10 00:34:49 2006
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Thu, 9 Nov 2006 22:34:49 -0700
Subject: [IPython-dev] Suggestions for implementing parallel algorithms?
In-Reply-To: <004e01c70451$7c37eb10$0a83a8c0@ratbert>
References: <004e01c70451$7c37eb10$0a83a8c0@ratbert>
Message-ID: <6ce0ac130611092134w754c79c0k293ff81e3c8bdf11@mail.gmail.com>

Albert,

This sounds like a nice application of IPython1.  Fernando and I have
had a number of talks about this exact issue lately.  One things I
should say before all else.  In spite of there being years of research
in parallel computing and algorithms, there is basically no research
on *interactive* parallel computing.  I am not sure you need your
application to be interactive, but if you do, at some level, you are
in new territory.

With that said, there are some things to keep in mind.

First, it is important to realize that all the previous work done on
parallel algorithm/application development still applies even though
the result can be used interactively.  For example, if you need to
move data around between the ipython engines, you should still use MPI
- and all the guidelines for using MPI still apply.

The new and interesting question is really "how and where do you want
to interact with the parallel application as a human user?"  There are
many models of how you would want your application abstracted for
interactive usage.  And at some level, you may want to have the
interactive API very different from the underlying computational model
and parallel algorithm.  You may want to hide the parallelism or you
may find it better to show it explicitely in the API.

In my own work, I have tended to factor my application into units that
can perform the basic computational tasks for both a serial and
parallel versions of the code.  I then use these as building blocks to
build the parallel and serial version.  If the low level components
are factored well, the high level algorithm is typically very short
and I don't mind maintaining both a serial and a parallel version.

For many things I do like the scatterAll/executeAll/gatherAll style of
computation - it is extremely lightweight and easy to implement.  The
one thing to be careful of though is to not use this approach when MPI
is more appropriate.  Testing the scaling of your application with
quickly reveal if there are problems like this.

> I'm hoping I can avoid this duplication. My first idea is to make something
> like a LocalController that implements IPython1's IController interface in a
> way that makes sense for single node operation. This way, I can implement my
> algorithm once in terms of IController operations, test it easily, and later
> by simply setting a controller property on a instance of the class
> implementing the algorithm, decide whether it runs on a single node or in
> parallel.

I had not thought of that before, but it does make sense.  It is sort
of similar to building objects that hide whether the object is being
used in a parallel/serial context.  It is surely worth trying this
approach, but I am not sure how it would turn out in your case.

I don't know if this helps, but I would love to see what you end up
trying and what you find most useful - I am curious about all these
things myself.

Brian

> How do you guys handle these issues in your code? Any suggestions would be
> appreciated.
>
> Cheers,
>
> Albert
>
> P.S. For an example implementation of k-means on top of MPI, see
> http://www.cs.umn.edu/~mnjoshi/PKMeans.pdf
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/ipython-dev
>


From gael.varoquaux at normalesup.org  Fri Nov 10 03:57:05 2006
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 10 Nov 2006 09:57:05 +0100
Subject: [IPython-dev] %bg  %run -i foo.py
Message-ID: <20061110085704.GA16897@clipper.ens.fr>

It seems this doesn't work. I am missing an obvious way to do this.
Elsewhere this would definitely be a nice feature to add.

Another nice magic to add would be a way to reload a module in a brutal
way. Let me explain. I have all the functions that I use often for
interactive work in a few modules. Every once so ever I change one of
them. Of course as I am a bad boy, I have imported this module doing a
"from foo import *", so I cannot reload it using "reload(foo)". The best
way I have found to reload it is something like

import foo
%run -i foo.__file__ 
# Well, this doesn't work, you have to copy and paste foo.__file__ here,
# as %run cannot take python variables as arguments.

I don't know whether I am clear on that, and I don't know if I am the
only one who needs such a magic but I for one would love to see this in
ipython.

Cheers,

Ga?l


From steve at shrogers.com  Fri Nov 10 22:37:50 2006
From: steve at shrogers.com (Steven H. Rogers)
Date: Fri, 10 Nov 2006 20:37:50 -0700
Subject: [IPython-dev] IPython1 on PS3/YDL
Message-ID: <4555458E.7080607@shrogers.com>

The Sony PlayStation 3 running Yellow Dog Linux would seem to be a good 
platform for IPyothon1.  Has anyone given any thought to this?

Regards,
Steve

-- 
Steven H. Rogers, Ph.D., steve at shrogers.com
Weblog: http://shrogers.com/weblog
"He who refuses to do arithmetic is doomed to talk nonsense."
-- John McCarthy
    



From rocky at panix.com  Sat Nov 11 01:21:24 2006
From: rocky at panix.com (R. Bernstein)
Date: Sat, 11 Nov 2006 01:21:24 -0500
Subject: [IPython-dev]  %bg  %run -i foo.py
In-Reply-To: <20061110085704.GA16897@clipper.ens.fr>
References: <20061110085704.GA16897@clipper.ens.fr>
Message-ID: <17749.27620.751949.310833@panix3.panix.com>

Seems like no takers on this, so I'll comment. Possibly you know this
already though.

I don't believe anyone knows how to force an "unimport" in a general
way, especially when a module contains some sort of dynamically
loadable module (such as a DLL or shared object). So yes, a "restart"
is sometimes desirable.

I realized this a while ago so in the context of a debugger (e.g
pydb). So there is a soft restart (called "run") which preserves state
like breakpoints and debugger settings and any variables set. And
there is a "hard" restart (called "restart") which in fact does an
exec using the parameters it saved for the command-line invocation. In
the latter, there is no attempt to save debugger state. In theory, one
could write out breakpoints and other state to a file and read that
back in after restarting. (Maybe one day I'll add two commands to
save/restore debugger state command.)

But in the context of a python shell such as ipython, what would such
a "restart" mean? Pretty simple and straightforward to add would be a
"restart" which does the same thing -- just exec itself using whatever
command-line parameters were given before. I suppose the saved history
would give *some* access to previous commands, even if objects
themselves would be wiped clean. 

Given this, would that be helpful? I don't know. And if not, how would
one mark what to keep and what not to keep? Clearly not everything
should or could be kept: some objects might be the result of
instantiating various classes which did imports. More ambitious would
again be to add two commands to save and restore ipython state for
some definition of state, e.g. the value of "%pdb" (on/off) or the
current working directory.



Gael Varoquaux writes:
 > It seems this doesn't work. I am missing an obvious way to do this.
 > Elsewhere this would definitely be a nice feature to add.
 > 
 > Another nice magic to add would be a way to reload a module in a brutal
 > way. Let me explain. I have all the functions that I use often for
 > interactive work in a few modules. Every once so ever I change one of
 > them. Of course as I am a bad boy, I have imported this module doing a
 > "from foo import *", so I cannot reload it using "reload(foo)". The best
 > way I have found to reload it is something like
 > 
 > import foo
 > %run -i foo.__file__ 
 > # Well, this doesn't work, you have to copy and paste foo.__file__ here,
 > # as %run cannot take python variables as arguments.
 > 
 > I don't know whether I am clear on that, and I don't know if I am the
 > only one who needs such a magic but I for one would love to see this in
 > ipython.
 > 
 > Cheers,
 > 
 > Ga?l
 > _______________________________________________
 > IPython-dev mailing list
 > IPython-dev at scipy.org
 > http://projects.scipy.org/mailman/listinfo/ipython-dev
 > 


From steve at shrogers.com  Sat Nov 11 07:57:15 2006
From: steve at shrogers.com (Steven H. Rogers)
Date: Sat, 11 Nov 2006 05:57:15 -0700
Subject: [IPython-dev] IPython1 on PS3/YDL
In-Reply-To: <6ce0ac130611102140i71a7beb3q970a38b6798242e4@mail.gmail.com>
References: <4555458E.7080607@shrogers.com>
	<6ce0ac130611102140i71a7beb3q970a38b6798242e4@mail.gmail.com>
Message-ID: <4555C8AB.9030303@shrogers.com>

I expect IPython to "just work" on the PPE, which would provide a good 
environment for exploring the Cell's possibilities.  I think it would take 
some work to enable IPython to manage the six available SPE's.  The SPE's 
need to be programmed in C and standard Python to run efficiently if you did 
a straight port to the SPE.  NumPy and PIL are obvious fits for the SPE 
architecture, but processing non-array data structures takes some thought.

Steve
///////////////////

Brian Granger wrote:
> I think it would be fantastic....I just need to get a PS3....
> 
> It sounds like it should "just work," which would be really awesome.
> Let us know if you get to try this out.
> 
> Brian
> 
> On 11/10/06, Steven H. Rogers <steve at shrogers.com> wrote:
>> The Sony PlayStation 3 running Yellow Dog Linux would seem to be a good
>> platform for IPyothon1.  Has anyone given any thought to this?
>>
>> Regards,
>> Steve
>>
>> -- 
>> Steven H. Rogers, Ph.D., steve at shrogers.com
>> Weblog: http://shrogers.com/weblog
>> "He who refuses to do arithmetic is doomed to talk nonsense."
>> -- John McCarthy
>>
>>
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at scipy.org
>> http://projects.scipy.org/mailman/listinfo/ipython-dev
>>
> 
> 

-- 
Steven H. Rogers, Ph.D., steve at shrogers.com
Weblog: http://shrogers.com/weblog
"He who refuses to do arithmetic is doomed to talk nonsense."
-- John McCarthy



From walter at livinglogic.de  Mon Nov 13 09:48:30 2006
From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=)
Date: Mon, 13 Nov 2006 15:48:30 +0100
Subject: [IPython-dev] wxPython GUI for ipipe
Message-ID: <455885BE.8080503@livinglogic.de>

Hello all!

Unfortunately the Google Summer of Code project didn't work out, so we
decided to do our own little Summer of Code. LivingLogic is sponsoring a
student to implement a wxPython GUI for ipipe. In the coming weeks and
months Nikolas Tautenhahn will be implementing a wxPython based GUI for
ipipe that provides the same functionality as ibrowse. The first results
of this can be seen here:

   http://styx.livinglogic.de/~walter/IPython/igrid.py

For this to work, wxPython has to be installed and the current IPython
head is required. Doing "from igrid import *" sets igrid as the default
ipipe display handler. A screenshot can be found here:

   http://styx.livinglogic.de/~walter/IPython/igrid.gif

Servus,
   Walter


From fullung at gmail.com  Mon Nov 13 19:23:48 2006
From: fullung at gmail.com (Albert Strasheim)
Date: Tue, 14 Nov 2006 02:23:48 +0200
Subject: [IPython-dev] Suggestions for implementing parallel algorithms?
In-Reply-To: <6ce0ac130611092134w754c79c0k293ff81e3c8bdf11@mail.gmail.com>
References: <004e01c70451$7c37eb10$0a83a8c0@ratbert>
	<6ce0ac130611092134w754c79c0k293ff81e3c8bdf11@mail.gmail.com>
Message-ID: <20061114002348.GA18968@dogbert.sdsl.sun.ac.za>

Hello all

On Thu, 09 Nov 2006, Brian Granger wrote:

> This sounds like a nice application of IPython1.  Fernando and I have
> had a number of talks about this exact issue lately.  One things I
> should say before all else.  In spite of there being years of research
> in parallel computing and algorithms, there is basically no research
> on *interactive* parallel computing.  I am not sure you need your
> application to be interactive, but if you do, at some level, you are
> in new territory.

My application probably doesn't fall into the interactive category at 
present. But some speaker verification and identification systems could 
definately be used interactively.

> With that said, there are some things to keep in mind.
> 
> First, it is important to realize that all the previous work done on
> parallel algorithm/application development still applies even though
> the result can be used interactively.  For example, if you need to
> move data around between the ipython engines, you should still use MPI
> - and all the guidelines for using MPI still apply.

Okay. This makes sense.
 
> The new and interesting question is really "how and where do you want
> to interact with the parallel application as a human user?"  There are
> many models of how you would want your application abstracted for
> interactive usage.  And at some level, you may want to have the
> interactive API very different from the underlying computational model
> and parallel algorithm.  You may want to hide the parallelism or you
> may find it better to show it explicitely in the API.

Interesting. I see what you're saying. For speaker verification 
systems, you could have an interactive parallel application in the case 
where you provide a system for a bunch of operators to investigate 
phone calls with. The operators might be wearing trenchcoats. ;-)
 
> In my own work, I have tended to factor my application into units that
> can perform the basic computational tasks for both a serial and
> parallel versions of the code.  I then use these as building blocks to
> build the parallel and serial version.  If the low level components
> are factored well, the high level algorithm is typically very short
> and I don't mind maintaining both a serial and a parallel version.

What is still unclear to me is how you call these components? I'm 
trying to avoid putting too much code in strings that get passed to 
executeAll and the like. If you have something foo that does N things 
on each node, how do set things up so that you don't have to write N 
lines in N strings to N executeAll calls? Presumably one would want to 
to just write executeAll('foo(a,b,c)') where you wrote foo as a normal 
Python function.

Something like this:

In [17]: f = lambda x: x
In [18]: rc.pushAll(f=f)
In [19]: rc.pushAll('f(10)')

However, lambda functions don't pickle:

Object cannot be serialized:  f Can't pickle <type 'function'>: 
attribute lookup __builtin__.function failed
Out[18]: False

or something like this:

In [36]: rc.getIDs()
Out[36]: [0, 1]

In [37]: def f(x): return 10*x
   ....:

In [38]: rc.pushAll(f=f)
Out[38]: True

However, this nukes the engines:

exceptions.AttributeError: 'module' object has no attribute 'f'

If I distribute a module containing the functions I want to execute to 
all the nodes beforehand, it might be able to unpickle the function on 
the engines. Still have to try this.

I wonder if it would work with functions declared inside instance 
methods. Where would the engines find these functions? My thinking here 
is that you could write parallel algorithms like this:

class ParallelFoo:
  def parallelop(self, data, controller):
      def something_to_execute_everywhere(x):
          return 10*x
      controller.scatterAll('x', data)
      controller.pushAll(f=something_to_execute_everywhere)
      controller.executeAll('f(x)')
 
Make sense? Any suggestions for doing this kind of thing?

> For many things I do like the scatterAll/executeAll/gatherAll style of
> computation - it is extremely lightweight and easy to implement.  The
> one thing to be careful of though is to not use this approach when MPI
> is more appropriate.  Testing the scaling of your application with
> quickly reveal if there are problems like this.

For my application, getting the data to the engines probably needs some 
thought. Prior to training the world model with K-means and GMM EM I 
need to spread out tens to hundreds of hours of speech between the 
engines.

Correct me if I'm wrong, but since the client and controller aren't 
part of the MPI universe, I have to use something like scatterAll here?

On my client I would typically have a directory with a few hundred 
files that I want to distribute to the engines. A quick 'n dirty 
implementation could read all these files into memory and scatter them 
to the engines.

This approach will run into problems when my dataset becomes bigger 
than the memory on my client machine (reasonably likely) and is 
probably going to be reasonably (very?) slow anyway.

Next idea: scatter filenames (or some other kind of data ID) and let 
the engines query a central server for the data, maybe via HTTP or FTP 
(or maybe do something that exposes a PyTables database to the 
network). Alternatively, keep all the data on each engine machine. My 
data is probably too big for this approach.

Next idea after that: since I have bunch of machines with disk space to 
spare, I could spread out multiple copies of my dataset across these 
machines. Then I could still scatter names and let the engines do some 
kind of lookup to find machines that have the data it is looking for 
and have it then get it from one of these at random. Basically the 
previous idea + load balancing.

Next idea after that: in a separate process on each engine machine, run 
something akin to a BitTorrent client that downloads files from a 
torrent as the local engine needs them. When this starts up, the client 
machine could seed the data until there is at least one copy of each 
file distributed across the engine machines.

Next idea after that: figure out a way to prefer scattering of data to 
kernels that already have the data available. I think the BOINC folks 
call this concept locality scheduling:

http://boinc.berkeley.edu/sched_locality.php

Other idea: instead of all this network I/O, keep a subset of the data 
on each engine machine and use locality scheduling to ensure that only 
machines that have certain data get work related to that data scattered 
to them. At this point, scatter probably isn't the right word anymore.

How fast this stuff is going to be... we'll see. :-)

Do you guys have large datasets to deal with? Any thoughts on doing this 
kind of thing?

> >I'm hoping I can avoid this duplication. My first idea is to make something
> >like a LocalController that implements IPython1's IController interface in 
> >a
> >way that makes sense for single node operation. This way, I can implement 
> >my
> >algorithm once in terms of IController operations, test it easily, and 
> >later
> >by simply setting a controller property on a instance of the class
> >implementing the algorithm, decide whether it runs on a single node or in
> >parallel.
> 
> I had not thought of that before, but it does make sense.  It is sort
> of similar to building objects that hide whether the object is being
> used in a parallel/serial context.  It is surely worth trying this
> approach, but I am not sure how it would turn out in your case.

I'd like to explore this idea further if I can figure out the "right" 
way for algorithms to tell the engines how to do a piece of work from 
inside a method.

> I don't know if this helps, but I would love to see what you end up
> trying and what you find most useful - I am curious about all these
> things myself.

Thanks for your inputs. Much appreciated.

Cheers,

Albert


From gael.varoquaux at normalesup.org  Wed Nov 15 14:17:28 2006
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Wed, 15 Nov 2006 20:17:28 +0100
Subject: [IPython-dev] %bg  %run -i foo.py
In-Reply-To: <20061110085704.GA16897@clipper.ens.fr>
References: <20061110085704.GA16897@clipper.ens.fr>
Message-ID: <20061115191724.GA21139@clipper.ens.fr>

On Fri, Nov 10, 2006 at 09:57:05AM +0100, Gael Varoquaux wrote:
> It seems this doesn't work. I am missing an obvious way to do this.
> Elsewhere this would definitely be a nice feature to add.

OK, revisiting my original mail I see that I wasn't clear:

I would like to do something like

"%bg %run -i foo.py"

but this doesn't work. Is there a way to do this (I may have missed
something in the doc), elsewhere this would be a nice feature to add.

Cheers,

Ga?l


From vivainio at gmail.com  Thu Nov 16 01:26:45 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Thu, 16 Nov 2006 08:26:45 +0200
Subject: [IPython-dev] %bg %run -i foo.py
In-Reply-To: <20061115191724.GA21139@clipper.ens.fr>
References: <20061110085704.GA16897@clipper.ens.fr>
	<20061115191724.GA21139@clipper.ens.fr>
Message-ID: <46cb515a0611152226q39a6ebf7lb91e4e1f2a2d522e@mail.gmail.com>

On 11/15/06, Gael Varoquaux <gael.varoquaux at normalesup.org> wrote:

> On Fri, Nov 10, 2006 at 09:57:05AM +0100, Gael Varoquaux wrote:
> > It seems this doesn't work. I am missing an obvious way to do this.
> > Elsewhere this would definitely be a nice feature to add.
>
> OK, revisiting my original mail I see that I wasn't clear:
>
> I would like to do something like
>
> "%bg %run -i foo.py"

Try something like

%bg _ip.magic('run -i foo.py')

In general, the _ip object object (an instance of "IPApi") is often a
good place to look for stuff. Try %history and you'll see that all
magics are expanded to _ip.magic('blah').

(An unrelated note - I'm moving and w/o home internet connection until
the end of this month, so have some patience).


-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From gael.varoquaux at normalesup.org  Thu Nov 16 02:21:05 2006
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 16 Nov 2006 08:21:05 +0100
Subject: [IPython-dev] %bg %run -i foo.py
Message-ID: <20061116072104.GC5448@clipper.ens.fr>

On Thu, Nov 16, 2006 at 08:26:45AM +0200, Ville M. Vainio wrote:
> %bg _ip.magic('run -i foo.py')

Thanks ! I added this to the wiki

> (An unrelated note - I'm moving and w/o home internet connection until
> the end of this month, so have some patience).

Of course, and thank you for your time spent on ipython, ipython rocks my
days !

Ga?l


From fperez.net at gmail.com  Thu Nov 16 02:30:36 2006
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 16 Nov 2006 00:30:36 -0700
Subject: [IPython-dev] Suggestions for implementing parallel algorithms?
In-Reply-To: <20061114002348.GA18968@dogbert.sdsl.sun.ac.za>
References: <004e01c70451$7c37eb10$0a83a8c0@ratbert>
	<6ce0ac130611092134w754c79c0k293ff81e3c8bdf11@mail.gmail.com>
	<20061114002348.GA18968@dogbert.sdsl.sun.ac.za>
Message-ID: <db6b5ecc0611152330i5a8a58c7pb35f68f9c65cc754@mail.gmail.com>

Hey Albert,

On 11/13/06, Albert Strasheim <fullung at gmail.com> wrote:
> Hello all
>
> On Thu, 09 Nov 2006, Brian Granger wrote:
>
> > This sounds like a nice application of IPython1.  Fernando and I have
> > had a number of talks about this exact issue lately.  One things I
> > should say before all else.  In spite of there being years of research
> > in parallel computing and algorithms, there is basically no research
> > on *interactive* parallel computing.  I am not sure you need your
> > application to be interactive, but if you do, at some level, you are
> > in new territory.
>
> My application probably doesn't fall into the interactive category at
> present. But some speaker verification and identification systems could
> definately be used interactively.

Sorry for the silence... I'm currently working specifically on all of
this, for a large production code in molecular electronic structure
problems.  My private copy of ipython1 has some fixes and enhancements
to make this easier, and I'm writing a little toy example which I'll
commit shortly, and which I'm using to find out for myself what the
'best practices' should be.  We're all learning here, so I don't claim
these solutions will be better than anything else, but at least
they'll come from a production problem, and lots and lots of talks on
the phone and face to face with Brian.

Feel free to ping me of off-SVN copies of any of this if you want them early.

I'm starting to think of using HG for off-SVN development, so we can
all share dev trees more fluidly while SVN is used only for commits
ready for public consumption...
Would you like that?

Cheers,

f


From ellisonbg.net at gmail.com  Thu Nov 16 02:44:51 2006
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Thu, 16 Nov 2006 00:44:51 -0700
Subject: [IPython-dev] Suggestions for implementing parallel algorithms?
In-Reply-To: <db6b5ecc0611152330i5a8a58c7pb35f68f9c65cc754@mail.gmail.com>
References: <004e01c70451$7c37eb10$0a83a8c0@ratbert>
	<6ce0ac130611092134w754c79c0k293ff81e3c8bdf11@mail.gmail.com>
	<20061114002348.GA18968@dogbert.sdsl.sun.ac.za>
	<db6b5ecc0611152330i5a8a58c7pb35f68f9c65cc754@mail.gmail.com>
Message-ID: <6ce0ac130611152344i6e48e2d0o1b68ce309b75d10d@mail.gmail.com>

Sorry I have been so silent as well.  I have a proposal (about
IPython) due in the next
few days so I have been focusing on that.  I will reply early next
week once things settle down.

I definitely have some ideas (and practical experience) in dealing
with large datasets.

But Fernando, do you think that Hg and subversion would play well
together?  I don't know much about Hg.

Brian

On 11/16/06, Fernando Perez <fperez.net at gmail.com> wrote:
> Hey Albert,
>
> On 11/13/06, Albert Strasheim <fullung at gmail.com> wrote:
> > Hello all
> >
> > On Thu, 09 Nov 2006, Brian Granger wrote:
> >
> > > This sounds like a nice application of IPython1.  Fernando and I have
> > > had a number of talks about this exact issue lately.  One things I
> > > should say before all else.  In spite of there being years of research
> > > in parallel computing and algorithms, there is basically no research
> > > on *interactive* parallel computing.  I am not sure you need your
> > > application to be interactive, but if you do, at some level, you are
> > > in new territory.
> >
> > My application probably doesn't fall into the interactive category at
> > present. But some speaker verification and identification systems could
> > definately be used interactively.
>
> Sorry for the silence... I'm currently working specifically on all of
> this, for a large production code in molecular electronic structure
> problems.  My private copy of ipython1 has some fixes and enhancements
> to make this easier, and I'm writing a little toy example which I'll
> commit shortly, and which I'm using to find out for myself what the
> 'best practices' should be.  We're all learning here, so I don't claim
> these solutions will be better than anything else, but at least
> they'll come from a production problem, and lots and lots of talks on
> the phone and face to face with Brian.
>
> Feel free to ping me of off-SVN copies of any of this if you want them early.
>
> I'm starting to think of using HG for off-SVN development, so we can
> all share dev trees more fluidly while SVN is used only for commits
> ready for public consumption...
> Would you like that?
>
> Cheers,
>
> f
>


From fperez.net at gmail.com  Thu Nov 16 03:00:50 2006
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 16 Nov 2006 01:00:50 -0700
Subject: [IPython-dev] Suggestions for implementing parallel algorithms?
In-Reply-To: <6ce0ac130611152344i6e48e2d0o1b68ce309b75d10d@mail.gmail.com>
References: <004e01c70451$7c37eb10$0a83a8c0@ratbert>
	<6ce0ac130611092134w754c79c0k293ff81e3c8bdf11@mail.gmail.com>
	<20061114002348.GA18968@dogbert.sdsl.sun.ac.za>
	<db6b5ecc0611152330i5a8a58c7pb35f68f9c65cc754@mail.gmail.com>
	<6ce0ac130611152344i6e48e2d0o1b68ce309b75d10d@mail.gmail.com>
Message-ID: <db6b5ecc0611160000r32603578n4550c2344b21a8b2@mail.gmail.com>

On 11/16/06, Brian Granger <ellisonbg.net at gmail.com> wrote:
> Sorry I have been so silent as well.  I have a proposal (about
> IPython) due in the next
> few days so I have been focusing on that.  I will reply early next
> week once things settle down.
>
> I definitely have some ideas (and practical experience) in dealing
> with large datasets.
>
> But Fernando, do you think that Hg and subversion would play well
> together?  I don't know much about Hg.

Well, I've lightly used bzr on top of an SVN tree without too many
problems.   Though in that case, I've always used the bzr tree for
fast-paced work and made the SVN commits as separate operations, not
by trying to generate a commit from a bzr changeset.

Recently (both SAGE and Brett Cannon's tests) have me leaning more
towards hg than bzr, but it's probably a bit of a toss-up between
those two.

The main point is just to give each of us an easy way to keep local
version control we can share for pulling changes, while SVN remains
the 'official' tree for commits meant for 'final' distribution.

Just an idea...

Cheers

f


From stefan at sun.ac.za  Thu Nov 16 03:50:29 2006
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Thu, 16 Nov 2006 10:50:29 +0200
Subject: [IPython-dev] Suggestions for implementing parallel algorithms?
In-Reply-To: <db6b5ecc0611160000r32603578n4550c2344b21a8b2@mail.gmail.com>
References: <004e01c70451$7c37eb10$0a83a8c0@ratbert>
	<6ce0ac130611092134w754c79c0k293ff81e3c8bdf11@mail.gmail.com>
	<20061114002348.GA18968@dogbert.sdsl.sun.ac.za>
	<db6b5ecc0611152330i5a8a58c7pb35f68f9c65cc754@mail.gmail.com>
	<6ce0ac130611152344i6e48e2d0o1b68ce309b75d10d@mail.gmail.com>
	<db6b5ecc0611160000r32603578n4550c2344b21a8b2@mail.gmail.com>
Message-ID: <20061116085028.GF1975@mentat.za.net>

On Thu, Nov 16, 2006 at 01:00:50AM -0700, Fernando Perez wrote:
> On 11/16/06, Brian Granger <ellisonbg.net at gmail.com> wrote:
> > Sorry I have been so silent as well.  I have a proposal (about
> > IPython) due in the next
> > few days so I have been focusing on that.  I will reply early next
> > week once things settle down.
> >
> > I definitely have some ideas (and practical experience) in dealing
> > with large datasets.
> >
> > But Fernando, do you think that Hg and subversion would play well
> > together?  I don't know much about Hg.
> 
> Well, I've lightly used bzr on top of an SVN tree without too many
> problems.   Though in that case, I've always used the bzr tree for
> fast-paced work and made the SVN commits as separate operations, not
> by trying to generate a commit from a bzr changeset.
> 
> Recently (both SAGE and Brett Cannon's tests) have me leaning more
> towards hg than bzr, but it's probably a bit of a toss-up between
> those two.

Interesting, I'm interested to know why.  I switched from hg to bzr a
year or so ago, having had problems with things like cherry picking.

I believe Lele's 'tailor' can be useful for the problem mentioned
above:

http://www.darcs.net/DarcsWiki/Tailor

Regards
St?fan


From prabhu at aero.iitb.ac.in  Thu Nov 16 04:57:02 2006
From: prabhu at aero.iitb.ac.in (Prabhu Ramachandran)
Date: Thu, 16 Nov 2006 15:27:02 +0530
Subject: [IPython-dev] Suggestions for implementing parallel algorithms?
In-Reply-To: <6ce0ac130611152344i6e48e2d0o1b68ce309b75d10d@mail.gmail.com>
References: <004e01c70451$7c37eb10$0a83a8c0@ratbert>
	<6ce0ac130611092134w754c79c0k293ff81e3c8bdf11@mail.gmail.com>
	<20061114002348.GA18968@dogbert.sdsl.sun.ac.za>
	<db6b5ecc0611152330i5a8a58c7pb35f68f9c65cc754@mail.gmail.com>
	<6ce0ac130611152344i6e48e2d0o1b68ce309b75d10d@mail.gmail.com>
Message-ID: <17756.13806.598304.997831@prpc.aero.iitb.ac.in>

>>>>> "Brian" == Brian Granger <ellisonbg.net at gmail.com> writes:

    Brian> But Fernando, do you think that Hg and subversion would
    Brian> play well together?  I don't know much about Hg.

What about bzr?

cheers,
prabhu



From vivainio at gmail.com  Thu Nov 16 05:58:46 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Thu, 16 Nov 2006 12:58:46 +0200
Subject: [IPython-dev] Suggestions for implementing parallel algorithms?
In-Reply-To: <17756.13806.598304.997831@prpc.aero.iitb.ac.in>
References: <004e01c70451$7c37eb10$0a83a8c0@ratbert>
	<6ce0ac130611092134w754c79c0k293ff81e3c8bdf11@mail.gmail.com>
	<20061114002348.GA18968@dogbert.sdsl.sun.ac.za>
	<db6b5ecc0611152330i5a8a58c7pb35f68f9c65cc754@mail.gmail.com>
	<6ce0ac130611152344i6e48e2d0o1b68ce309b75d10d@mail.gmail.com>
	<17756.13806.598304.997831@prpc.aero.iitb.ac.in>
Message-ID: <46cb515a0611160258u5a9c477eq5779345a80599c2c@mail.gmail.com>

On 11/16/06, Prabhu Ramachandran <prabhu at aero.iitb.ac.in> wrote:

> >>>>> "Brian" == Brian Granger <ellisonbg.net at gmail.com> writes:
>
>     Brian> But Fernando, do you think that Hg and subversion would
>     Brian> play well together?  I don't know much about Hg.

Hg and SVN should play as well together as local directory and SVN, in
any case. :-)


> What about bzr?

At this moment, googling around would indicate that Hg is technically
better than bzr.

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From prabhu at aero.iitb.ac.in  Thu Nov 16 07:59:39 2006
From: prabhu at aero.iitb.ac.in (Prabhu Ramachandran)
Date: Thu, 16 Nov 2006 18:29:39 +0530
Subject: [IPython-dev] Suggestions for implementing parallel algorithms?
In-Reply-To: <46cb515a0611160258u5a9c477eq5779345a80599c2c@mail.gmail.com>
References: <004e01c70451$7c37eb10$0a83a8c0@ratbert>
	<6ce0ac130611092134w754c79c0k293ff81e3c8bdf11@mail.gmail.com>
	<20061114002348.GA18968@dogbert.sdsl.sun.ac.za>
	<db6b5ecc0611152330i5a8a58c7pb35f68f9c65cc754@mail.gmail.com>
	<6ce0ac130611152344i6e48e2d0o1b68ce309b75d10d@mail.gmail.com>
	<17756.13806.598304.997831@prpc.aero.iitb.ac.in>
	<46cb515a0611160258u5a9c477eq5779345a80599c2c@mail.gmail.com>
Message-ID: <17756.24763.991616.543830@prpc.aero.iitb.ac.in>

>>>>> "Ville" == Ville M Vainio <vivainio at gmail.com> writes:

    Ville> On 11/16/06, Prabhu Ramachandran <prabhu at aero.iitb.ac.in>
    Ville> wrote:
    >> >>>>> "Brian" == Brian Granger <ellisonbg.net at gmail.com>
    >> writes:
    >> 
    Brian> But Fernando, do you think that Hg and subversion would
    Brian> play well together?  I don't know much about Hg.

    Ville> Hg and SVN should play as well together as local directory
    Ville> and SVN, in any case. :-)

    >> What about bzr?

    Ville> At this moment, googling around would indicate that Hg is
    Ville> technically better than bzr.

bzr is slow but that isn't an issue for smaller projects.  IIRC Hg
OTOH can't do directory renames yet which bzr can.  That seems the
difference to me.  I believe the bzr folks are working on speed and
that the Hg folks on the rename.  Personally I can put up with a
slower speed for a small tree but renames are really important which
is why I suggested bzr...

cheers,
prabhu




From stefan at sun.ac.za  Thu Nov 16 08:17:50 2006
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Thu, 16 Nov 2006 15:17:50 +0200
Subject: [IPython-dev] Suggestions for implementing parallel algorithms?
In-Reply-To: <46cb515a0611160258u5a9c477eq5779345a80599c2c@mail.gmail.com>
References: <004e01c70451$7c37eb10$0a83a8c0@ratbert>
	<6ce0ac130611092134w754c79c0k293ff81e3c8bdf11@mail.gmail.com>
	<20061114002348.GA18968@dogbert.sdsl.sun.ac.za>
	<db6b5ecc0611152330i5a8a58c7pb35f68f9c65cc754@mail.gmail.com>
	<6ce0ac130611152344i6e48e2d0o1b68ce309b75d10d@mail.gmail.com>
	<17756.13806.598304.997831@prpc.aero.iitb.ac.in>
	<46cb515a0611160258u5a9c477eq5779345a80599c2c@mail.gmail.com>
Message-ID: <20061116131750.GH1975@mentat.za.net>

On Thu, Nov 16, 2006 at 12:58:46PM +0200, Ville M. Vainio wrote:
> On 11/16/06, Prabhu Ramachandran <prabhu at aero.iitb.ac.in> wrote:
> 
> > >>>>> "Brian" == Brian Granger <ellisonbg.net at gmail.com> writes:
> >
> >     Brian> But Fernando, do you think that Hg and subversion would
> >     Brian> play well together?  I don't know much about Hg.
> 
> Hg and SVN should play as well together as local directory and SVN, in
> any case. :-)
> 
> 
> > What about bzr?
> 
> At this moment, googling around would indicate that Hg is technically
> better than bzr.

Can you provide a URL?  I found several reviews (based on older
versions) which all claim that the two are basically equal, except
when it comes to speed.  In that regard bzr has been vastly improved
with 1.0.  The comparisons are further muddled by some referring to
bzr as in arch/tla.

As for my previous question about mercurial and cherry-picking, it
seems they have started development on a plugin in August:

http://www.selenic.com/mercurial/wiki/index.cgi/TransplantExtension
http://hg.kublai.com/mercurial/transplant

Cheers
St?fan


From fperez.net at gmail.com  Thu Nov 16 11:31:18 2006
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 16 Nov 2006 09:31:18 -0700
Subject: [IPython-dev] Suggestions for implementing parallel algorithms?
In-Reply-To: <20061116131750.GH1975@mentat.za.net>
References: <004e01c70451$7c37eb10$0a83a8c0@ratbert>
	<6ce0ac130611092134w754c79c0k293ff81e3c8bdf11@mail.gmail.com>
	<20061114002348.GA18968@dogbert.sdsl.sun.ac.za>
	<db6b5ecc0611152330i5a8a58c7pb35f68f9c65cc754@mail.gmail.com>
	<6ce0ac130611152344i6e48e2d0o1b68ce309b75d10d@mail.gmail.com>
	<17756.13806.598304.997831@prpc.aero.iitb.ac.in>
	<46cb515a0611160258u5a9c477eq5779345a80599c2c@mail.gmail.com>
	<20061116131750.GH1975@mentat.za.net>
Message-ID: <db6b5ecc0611160831v5dbedc4ewf134e08653cf06b@mail.gmail.com>

On 11/16/06, Stefan van der Walt <stefan at sun.ac.za> wrote:
> On Thu, Nov 16, 2006 at 12:58:46PM +0200, Ville M. Vainio wrote:
> > On 11/16/06, Prabhu Ramachandran <prabhu at aero.iitb.ac.in> wrote:
> >
> > > >>>>> "Brian" == Brian Granger <ellisonbg.net at gmail.com> writes:
> > >
> > >     Brian> But Fernando, do you think that Hg and subversion would
> > >     Brian> play well together?  I don't know much about Hg.
> >
> > Hg and SVN should play as well together as local directory and SVN, in
> > any case. :-)
> >
> >
> > > What about bzr?
> >
> > At this moment, googling around would indicate that Hg is technically
> > better than bzr.
>
> Can you provide a URL?  I found several reviews (based on older
> versions) which all claim that the two are basically equal, except
> when it comes to speed.  In that regard bzr has been vastly improved
> with 1.0.  The comparisons are further muddled by some referring to
> bzr as in arch/tla.

Here's something I recently read:

http://sayspy.blogspot.com/2006/11/bazaar-vs-mercurial-unscientific.html

This was one of my sources for both saying that they were probably
very close, and that hg seemed marginally preferable.  Another reason
for my slight bias towards hg is that SAGE chose it (for speed
reasons, mostly), and I occasionally need to interact with SAGE's dev
code.

I've heard about Tailor before, but haven't actually ever used it.  If
your commits just involve file updates, there's no real need for it
and I prefer to make up a new log message that summarizes the work.
But if you've done structural operations in your hg/bzr work, then you
/really/ want something like Tailor to propagate that correctly into
the SVN operations.  I have no idea how well that actually works in
practice, I can easily imagine it being 'below its advertised
performance' :)

Cheers,

f


From fullung at gmail.com  Thu Nov 16 16:22:41 2006
From: fullung at gmail.com (Albert Strasheim)
Date: Thu, 16 Nov 2006 23:22:41 +0200
Subject: [IPython-dev] Suggestions for implementing parallel algorithms?
In-Reply-To: <db6b5ecc0611152330i5a8a58c7pb35f68f9c65cc754@mail.gmail.com>
References: <004e01c70451$7c37eb10$0a83a8c0@ratbert>
	<6ce0ac130611092134w754c79c0k293ff81e3c8bdf11@mail.gmail.com>
	<20061114002348.GA18968@dogbert.sdsl.sun.ac.za>
	<db6b5ecc0611152330i5a8a58c7pb35f68f9c65cc754@mail.gmail.com>
Message-ID: <20061116212241.GA21364@dogbert.sdsl.sun.ac.za>

Hello all

On Thu, 16 Nov 2006, Fernando Perez wrote:
> On 11/13/06, Albert Strasheim <fullung at gmail.com> wrote:
> >On Thu, 09 Nov 2006, Brian Granger wrote:
> >> This sounds like a nice application of IPython1.  Fernando and I have
> >> had a number of talks about this exact issue lately.  One things I
> >> should say before all else.  In spite of there being years of research
> >> in parallel computing and algorithms, there is basically no research
> >> on *interactive* parallel computing.  I am not sure you need your
> >> application to be interactive, but if you do, at some level, you are
> >> in new territory.
> >
> >My application probably doesn't fall into the interactive category at
> >present. But some speaker verification and identification systems could
> >definately be used interactively.
> 
> Sorry for the silence... I'm currently working specifically on all of
> this, for a large production code in molecular electronic structure
> problems.  My private copy of ipython1 has some fixes and enhancements
> to make this easier, and I'm writing a little toy example which I'll
> commit shortly, and which I'm using to find out for myself what the
> 'best practices' should be.  We're all learning here, so I don't claim
> these solutions will be better than anything else, but at least
> they'll come from a production problem, and lots and lots of talks on
> the phone and face to face with Brian.
> 
> Feel free to ping me of off-SVN copies of any of this if you want them 
> early.

I would love to look at any code you'd be willing to offer. I too am 
trying to figure out the best practices. Thanks in advance!

> I'm starting to think of using HG for off-SVN development, so we can
> all share dev trees more fluidly while SVN is used only for commits
> ready for public consumption...
> Would you like that?

Sounds like a great plan.

Cheers,

Albert


From smelkov at mph1.phys.spbu.ru  Sun Nov 19 12:32:16 2006
From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov)
Date: Sun, 19 Nov 2006 20:32:16 +0300
Subject: [IPython-dev] [patch 0/9] IPython patches from kirr
Message-ID: <20061119173216.561631860@roro-510.zxlink>

Hello up there!

While refreshing IPython recently, I've made some patches against
ipython--trunk. Please apply.

-- 
    ????? ????????, ??????.
    http://landau.phys.spbu.ru/~kirr/aiv/


From smelkov at mph1.phys.spbu.ru  Sun Nov 19 12:32:19 2006
From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov)
Date: Sun, 19 Nov 2006 20:32:19 +0300
Subject: [IPython-dev] [patch 3/9] Kill old bits from %prun doc.
References: <20061119173216.561631860@roro-510.zxlink>
Message-ID: <20061119173434.055551881@roro-510.zxlink>

An embedded and charset-unspecified text was scrubbed...
Name: prun_kill-old-doc.patch
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20061119/fa940529/attachment.ksh>

From smelkov at mph1.phys.spbu.ru  Sun Nov 19 12:32:20 2006
From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov)
Date: Sun, 19 Nov 2006 20:32:20 +0300
Subject: [IPython-dev] [patch 4/9] The manual mentions %ds along %pushd and
	%popd, but %ds is not a valid magic. Maybe it should be %dhist?
References: <20061119173216.561631860@roro-510.zxlink>
Message-ID: <20061119173434.540630873@roro-510.zxlink>

An embedded and charset-unspecified text was scrubbed...
Name: doc-ds-2-dhist.patch
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20061119/dc37d045/attachment.ksh>

From smelkov at mph1.phys.spbu.ru  Sun Nov 19 12:32:18 2006
From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov)
Date: Sun, 19 Nov 2006 20:32:18 +0300
Subject: [IPython-dev] [patch 2/9] Try to use cProfile module (Python-2.5)
	instead of old profile.
References: <20061119173216.561631860@roro-510.zxlink>
Message-ID: <20061119173433.579561522@roro-510.zxlink>

An embedded and charset-unspecified text was scrubbed...
Name: cprofile.patch
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20061119/50a73fc0/attachment.ksh>

From smelkov at mph1.phys.spbu.ru  Sun Nov 19 12:32:21 2006
From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov)
Date: Sun, 19 Nov 2006 20:32:21 +0300
Subject: [IPython-dev] [patch 5/9] ScientificPython webpage has moved to::
	http://dirac.cnrs-orleans.fr/ScientificPython/
References: <20061119173216.561631860@roro-510.zxlink>
Message-ID: <20061119173435.031683281@roro-510.zxlink>

An embedded and charset-unspecified text was scrubbed...
Name: doc-ScientificPython-updates.patch
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20061119/984a26d4/attachment.ksh>

From smelkov at mph1.phys.spbu.ru  Sun Nov 19 12:32:22 2006
From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov)
Date: Sun, 19 Nov 2006 20:32:22 +0300
Subject: [IPython-dev] [patch 6/9] Fix %config crash when invoked without
	args.
References: <20061119173216.561631860@roro-510.zxlink>
Message-ID: <20061119173435.516700533@roro-510.zxlink>

An embedded and charset-unspecified text was scrubbed...
Name: fix-config.patch
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20061119/2a4d4b6f/attachment.ksh>

From smelkov at mph1.phys.spbu.ru  Sun Nov 19 12:32:17 2006
From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov)
Date: Sun, 19 Nov 2006 20:32:17 +0300
Subject: [IPython-dev] [patch 1/9] Fix typos here and there...
References: <20061119173216.561631860@roro-510.zxlink>
Message-ID: <20061119173433.114457240@roro-510.zxlink>

An embedded and charset-unspecified text was scrubbed...
Name: typos.patch
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20061119/6cce5f27/attachment.ksh>

From smelkov at mph1.phys.spbu.ru  Sun Nov 19 12:32:25 2006
From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov)
Date: Sun, 19 Nov 2006 20:32:25 +0300
Subject: [IPython-dev] [patch 9/9] doc: point bug-tracker URL to IPythons
	trac-tickets.
References: <20061119173216.561631860@roro-510.zxlink>
Message-ID: <20061119173436.959526650@roro-510.zxlink>

An embedded and charset-unspecified text was scrubbed...
Name: doc-tickets.patch
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20061119/358c8ff7/attachment.ksh>

From smelkov at mph1.phys.spbu.ru  Sun Nov 19 12:32:23 2006
From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov)
Date: Sun, 19 Nov 2006 20:32:23 +0300
Subject: [IPython-dev] [patch 7/9] Implement %page -r (page in raw mode)
References: <20061119173216.561631860@roro-510.zxlink>
Message-ID: <20061119173436.007608230@roro-510.zxlink>

An embedded and charset-unspecified text was scrubbed...
Name: page-raw.patch
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20061119/e59b2bb3/attachment.ksh>

From smelkov at mph1.phys.spbu.ru  Sun Nov 19 12:32:24 2006
From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov)
Date: Sun, 19 Nov 2006 20:32:24 +0300
Subject: [IPython-dev] [patch 8/9] Fix wrong Qt+IPython+Designer link in
	documentation.
References: <20061119173216.561631860@roro-510.zxlink>
Message-ID: <20061119173436.483781295@roro-510.zxlink>

An embedded and charset-unspecified text was scrubbed...
Name: doc-qt+ipython.patch
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20061119/338c1404/attachment.ksh>

From vivainio at gmail.com  Sun Nov 19 15:12:29 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Sun, 19 Nov 2006 22:12:29 +0200
Subject: [IPython-dev] [patch 0/9] IPython patches from kirr
In-Reply-To: <20061119173216.561631860@roro-510.zxlink>
References: <20061119173216.561631860@roro-510.zxlink>
Message-ID: <46cb515a0611191212q17b55ddfs1a7a169463990dc9@mail.gmail.com>

On 11/19/06, Kirill Smelkov <smelkov at mph1.phys.spbu.ru> wrote:

> While refreshing IPython recently, I've made some patches against
> ipython--trunk. Please apply.

Will do (on tuesday). Many thanks!


-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From fperez.net at gmail.com  Mon Nov 20 01:53:17 2006
From: fperez.net at gmail.com (Fernando Perez)
Date: Sun, 19 Nov 2006 23:53:17 -0700
Subject: [IPython-dev] [patch 0/9] IPython patches from kirr
In-Reply-To: <20061119173216.561631860@roro-510.zxlink>
References: <20061119173216.561631860@roro-510.zxlink>
Message-ID: <db6b5ecc0611192253y45866436oe1bf88c350ba6646@mail.gmail.com>

On 11/19/06, Kirill Smelkov <smelkov at mph1.phys.spbu.ru> wrote:
> Hello up there!
>
> While refreshing IPython recently, I've made some patches against
> ipython--trunk. Please apply.

Thanks!  We'll have a look (Ville already replied).  Much appreciated.

Ville - let me know if there's any one you may have a question with.
Upon a quick look, I noticed that in patch 2/9 would be cleaner by
simply having two separate try/excepts to deal with the
profile-not-in-debian and the cProfile issue separately.     It's
cleaner than retrying the failed import, and the intent is more
obvious.

Also, patch 4/9: the right magic is '%dirs' (for "directory stack"), not %dhist.

Cheers,

f


From vivainio at gmail.com  Tue Nov 21 01:37:47 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Tue, 21 Nov 2006 08:37:47 +0200
Subject: [IPython-dev] [patch 0/9] IPython patches from kirr
In-Reply-To: <db6b5ecc0611192253y45866436oe1bf88c350ba6646@mail.gmail.com>
References: <20061119173216.561631860@roro-510.zxlink>
	<db6b5ecc0611192253y45866436oe1bf88c350ba6646@mail.gmail.com>
Message-ID: <46cb515a0611202237q4d2d478al78cef9f578f06db2@mail.gmail.com>

On 11/20/06, Fernando Perez <fperez.net at gmail.com> wrote:

> Thanks!  We'll have a look (Ville already replied).  Much appreciated.
>
> Ville - let me know if there's any one you may have a question with.

The %config one. I still think %config is redundant and should be
removed, _ip.options already works, is stable (as a part of ipapi)
and is handier anyway (tab completion works etc.).

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From vivainio at gmail.com  Tue Nov 21 15:44:02 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Tue, 21 Nov 2006 22:44:02 +0200
Subject: [IPython-dev] [patch 0/9] IPython patches from kirr
In-Reply-To: <20061121190806.GA5431@landau.phys.spbu.ru>
References: <20061119173216.561631860@roro-510.zxlink>
	<db6b5ecc0611192253y45866436oe1bf88c350ba6646@mail.gmail.com>
	<20061121190806.GA5431@landau.phys.spbu.ru>
Message-ID: <46cb515a0611211244n2d14aa7ctfc42cc34f0c49679@mail.gmail.com>

On 11/21/06, Kirill Smelkov <kirr at landau.phys.spbu.ru> wrote:

> Thanks for your acknowledgement!

It's great that some people pay attention to things such as obsolete
documentation, they are all too easily left dangling...

> This one is ok?

No, your patch can't be applied for some reason. Please send a new one
against the current svn, which has the other patches (apart from the
%config one, because I'd like to remove %config altogether) applied.
And what happened to patch 8/9?

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From smelkov at mph1.phys.spbu.ru  Wed Nov 22 16:34:00 2006
From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov)
Date: Thu, 23 Nov 2006 00:34:00 +0300
Subject: [IPython-dev] [patch 1/2] Try to use cProfile module (Python-2.5)
	instead of old profile.
References: <20061122213359.515229954@roro-510.zxlink>
Message-ID: <20061122213502.540288964@roro-510.zxlink>

An embedded and charset-unspecified text was scrubbed...
Name: cprofile.patch
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20061123/5d8c2729/attachment.ksh>

From smelkov at mph1.phys.spbu.ru  Wed Nov 22 16:34:01 2006
From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov)
Date: Thu, 23 Nov 2006 00:34:01 +0300
Subject: [IPython-dev] [patch 2/2] Fix wrong Qt+IPython+Designer link in
	documentation.
References: <20061122213359.515229954@roro-510.zxlink>
Message-ID: <20061122213503.012163136@roro-510.zxlink>

An embedded and charset-unspecified text was scrubbed...
Name: doc-qt+ipython.patch
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20061123/e403ba02/attachment.ksh>

From smelkov at mph1.phys.spbu.ru  Wed Nov 22 16:33:59 2006
From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov)
Date: Thu, 23 Nov 2006 00:33:59 +0300
Subject: [IPython-dev] [patch 0/2] IPython patches from kirr, iter2
Message-ID: <20061122213359.515229954@roro-510.zxlink>

I'm resending 2 patches out of 3 left unapplied from the first iteration.

-- 
    All the best, Kirill.
    http://landau.phys.spbu.ru/~kirr/aiv/


From rocky at panix.com  Wed Nov 22 23:04:52 2006
From: rocky at panix.com (R. Bernstein)
Date: Wed, 22 Nov 2006 23:04:52 -0500
Subject: [IPython-dev] making %pydb be more pythonic (was:
	Extensions/pydb_ipy.py added)
In-Reply-To: <17731.31618.644784.714712@panix3.panix.com>
References: <46cb515a0610280147u5a310ffev181ec352bc9f83cd@mail.gmail.com>
	<17731.8786.756329.573337@panix3.panix.com>
	<46cb515a0610280347n73539bdt6810659b7584e3a@mail.gmail.com>
	<17731.15518.833930.352892@panix3.panix.com>
	<46cb515a0610280541n1ccc4691o4a47f1ca404b826c@mail.gmail.com>
	<17731.31618.644784.714712@panix3.panix.com>
Message-ID: <17765.7652.52085.132851@panix3.panix.com>

About a month back, there was this discussion:

R. Bernstein writes:
 > Ville M. Vainio writes:
 >  > On 10/28/06, R. Bernstein <rocky at panix.com> wrote:
 >  > 
 >  > > But one other thing regarding call_pydb(). And I'm not really sure how
 >  > > to fix. So if anyone has suggestions...
 >  > >
 >  > > pydb.runv() should be using the ipython Pdb class which has pdef, pdoc
 >  > > commands added and not the pydb one.
 >  > 
 >  > Would it make sense to import pydb.pydb module in pydb_ipy.py, and
 >  > inject the ipython version of Pdb class to the pydb module namespace?
 >  > That way main() would instantiate the ipython version...
 > 
 > Good idea! That's be great!

Nothing has happened on this and I suspect because it wasn't
doable. (At least I couldn't figure out how it would be done. But I
figured if it could that *could* it would be a great thing to do.)

So recently, I've added a parameter to pydb so that runv() can get
passed with an Pdb object extended in the way that ipython wants to
extend it. That will be in the next release of pydb (which again can
be timed so that it is no later than the next ipython release).

I've modified ipy_pydb.py so that it now passes in such a Pdb
object. This was the debug session can make use of pdoc and pinfo. It
also uses the ipython debugger prompt (which personally I'm not a fan
of but I guess it makes it ipython consistent.) There is now the
possibility of using the ipython color scheme, but I couldn't figure
out how to get that passed when creating the Debugger.Pdb object.

And in going over IPython/Extensions/ipy_pydb.py I've noticed a couple
of other things. First there should be a check that pydb can be
imported and that there is a late enough version of pydb. 

Below is a patch.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: ipy_pydb.py.diff
Type: application/octet-stream
Size: 1185 bytes
Desc: Changes to have %pydb be more ipythonic.
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20061122/fbb35919/attachment.obj>

From vivainio at gmail.com  Thu Nov 23 14:06:01 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Thu, 23 Nov 2006 21:06:01 +0200
Subject: [IPython-dev] [patch 1/2] Try to use cProfile module
	(Python-2.5) instead of old profile.
In-Reply-To: <20061122213502.540288964@roro-510.zxlink>
References: <20061122213359.515229954@roro-510.zxlink>
	<20061122213502.540288964@roro-510.zxlink>
Message-ID: <46cb515a0611231106n20ed0cf2ld8d75acd6591f5e4@mail.gmail.com>

On 11/22/06, Kirill Smelkov <smelkov at mph1.phys.spbu.ru> wrote:

> rationale: cProfile is faster.
>  try:
>     import profile,pstats
>  except ImportError:
> -    profile = pstats = None
> +    pass
> +
> +# cProfile was added in Python2.5
> +# if present, we use it instead of old profile, because cProfile is faster
> +try:
> +    import cProfile as profile
> +    import pstats
> +except ImportError:
> +    pass

New you import both profile and cProfile, if both exist. That's a bit
wasteful; why not import cProfile first, and only if that raises
ImportError import profile?

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From vivainio at gmail.com  Thu Nov 23 14:48:05 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Thu, 23 Nov 2006 21:48:05 +0200
Subject: [IPython-dev] making %pydb be more pythonic (was:
	Extensions/pydb_ipy.py added)
In-Reply-To: <17765.7652.52085.132851@panix3.panix.com>
References: <46cb515a0610280147u5a310ffev181ec352bc9f83cd@mail.gmail.com>
	<17731.8786.756329.573337@panix3.panix.com>
	<46cb515a0610280347n73539bdt6810659b7584e3a@mail.gmail.com>
	<17731.15518.833930.352892@panix3.panix.com>
	<46cb515a0610280541n1ccc4691o4a47f1ca404b826c@mail.gmail.com>
	<17731.31618.644784.714712@panix3.panix.com>
	<17765.7652.52085.132851@panix3.panix.com>
Message-ID: <46cb515a0611231148h4ed206fi56adf53bd35ecc8@mail.gmail.com>

On 11/23/06, R. Bernstein <rocky at panix.com> wrote:

>  >  > Would it make sense to import pydb.pydb module in pydb_ipy.py, and
>  >  > inject the ipython version of Pdb class to the pydb module namespace?
>  >  > That way main() would instantiate the ipython version...
>  >
>  > Good idea! That's be great!
>
> Nothing has happened on this and I suspect because it wasn't
> doable. (At least I couldn't figure out how it would be done. But I
> figured if it could that *could* it would be a great thing to do.)

This could have worked (in ipy_pydb - untried code):

-----
import pydb.pydb
import Debugger

pydb.pydb.Pdb = Debugger.Pdb
------

I didn't try it myself because I figured you could have given it more
testing mileage. Additionally, I was off the net for a long while due
to moving.

> So recently, I've added a parameter to pydb so that runv() can get
> passed with an Pdb object extended in the way that ipython wants to
> extend it. That will be in the next release of pydb (which again can
> be timed so that it is no later than the next ipython release).
>
> I've modified ipy_pydb.py so that it now passes in such a Pdb
> object. This was the debug session can make use of pdoc and pinfo. It
> also uses the ipython debugger prompt (which personally I'm not a fan
> of but I guess it makes it ipython consistent.) There is now the
> possibility of using the ipython color scheme, but I couldn't figure
> out how to get that passed when creating the Debugger.Pdb object.
>
> And in going over IPython/Extensions/ipy_pydb.py I've noticed a couple
> of other things. First there should be a check that pydb can be
> imported and that there is a late enough version of pydb.
>
> Below is a patch.


Not a bad solution, I committed the patch. In general, ipy_pydb.py is
"your territory" and I'll gladly commit every patch I get for it
immediately. In theory something like ipy_pydb.py could even be
bundled w/ pydb, just like python-mode.el is bundled with python...

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From smelkov at mph1.phys.spbu.ru  Thu Nov 23 16:44:05 2006
From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov)
Date: Fri, 24 Nov 2006 00:44:05 +0300
Subject: [IPython-dev] [patch 1/2] Try to use cProfile module
	(Python-2.5) instead of old profile.
In-Reply-To: <46cb515a0611231106n20ed0cf2ld8d75acd6591f5e4@mail.gmail.com>
References: <20061122213359.515229954@roro-510.zxlink>
	<20061122213502.540288964@roro-510.zxlink>
	<46cb515a0611231106n20ed0cf2ld8d75acd6591f5e4@mail.gmail.com>
Message-ID: <20061123214404.GA4614@landau.phys.spbu.ru>

On Thu, Nov 23, 2006 at 09:06:01PM +0200, Ville M. Vainio wrote:
> On 11/22/06, Kirill Smelkov <smelkov at mph1.phys.spbu.ru> wrote:
> 
> >rationale: cProfile is faster.
> > try:
> >    import profile,pstats
> > except ImportError:
> >-    profile = pstats = None
> >+    pass
> >+
> >+# cProfile was added in Python2.5
> >+# if present, we use it instead of old profile, because cProfile is faster
> >+try:
> >+    import cProfile as profile
> >+    import pstats
> >+except ImportError:
> >+    pass
> 
> New you import both profile and cProfile, if both exist. That's a bit
> wasteful; why not import cProfile first, and only if that raises
> ImportError import profile?
That's what I originally did (see [patch 2/9], buf Fernando objected, I
can't understand why...

-- 
    ????? ????????, ??????.
    http://landau.phys.spbu.ru/~kirr/aiv/


From vivainio at gmail.com  Thu Nov 23 16:52:42 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Thu, 23 Nov 2006 23:52:42 +0200
Subject: [IPython-dev] [patch 1/2] Try to use cProfile module
	(Python-2.5) instead of old profile.
In-Reply-To: <20061123214404.GA4614@landau.phys.spbu.ru>
References: <20061122213359.515229954@roro-510.zxlink>
	<20061122213502.540288964@roro-510.zxlink>
	<46cb515a0611231106n20ed0cf2ld8d75acd6591f5e4@mail.gmail.com>
	<20061123214404.GA4614@landau.phys.spbu.ru>
Message-ID: <46cb515a0611231352s2c58d47ej5699ad70b448aaf9@mail.gmail.com>

On 11/23/06, Kirill Smelkov <smelkov at mph1.phys.spbu.ru> wrote:

> That's what I originally did (see [patch 2/9], buf Fernando objected, I
> can't understand why...

Oh, I see, he probably just accidentally misread your patch. I'll go
ahead and apply your original patch then (tomorrow).

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From rocky at panix.com  Fri Nov 24 00:39:20 2006
From: rocky at panix.com (R. Bernstein)
Date: Fri, 24 Nov 2006 00:39:20 -0500
Subject: [IPython-dev] autoload and making %pydb be more ipythonic (was:
	Extensions/pydb_ipy.py added)
In-Reply-To: <46cb515a0611231148h4ed206fi56adf53bd35ecc8@mail.gmail.com>
References: <46cb515a0610280147u5a310ffev181ec352bc9f83cd@mail.gmail.com>
	<17731.8786.756329.573337@panix3.panix.com>
	<46cb515a0610280347n73539bdt6810659b7584e3a@mail.gmail.com>
	<17731.15518.833930.352892@panix3.panix.com>
	<46cb515a0610280541n1ccc4691o4a47f1ca404b826c@mail.gmail.com>
	<17731.31618.644784.714712@panix3.panix.com>
	<17765.7652.52085.132851@panix3.panix.com>
	<46cb515a0611231148h4ed206fi56adf53bd35ecc8@mail.gmail.com>
Message-ID: <17766.34184.188910.580317@panix3.panix.com>

Thanks for applying the patch! 

I am currently out of town, but when I get back I'll try as you suggest below.

I have other patches sitting on a disk somewhere too. 

I don't have a problem adding ipy_pydb.py in the pydb package. But the one thing though that I think would somehow be nice is that if one *didn't* have to do run "import ipy_pydb" to allow %pydb to work. Given that one can put this in a "try" block, I don't see any downside of doing this. 

A cool and more general mechanism for ipython extensions would be something along the lines of Emacs Lisp "autoload". Say if I run %xxx ... then if that's not registered, try to run "import ipy_xxx" (where for example xxx = pydb), and then run %xxx ....  Comments?


Ville M. Vainio writes:
 > On 11/23/06, R. Bernstein <rocky at panix.com> wrote:
 > 
 > >  >  > Would it make sense to import pydb.pydb module in pydb_ipy.py, and
 > >  >  > inject the ipython version of Pdb class to the pydb module namespace?
 > >  >  > That way main() would instantiate the ipython version...
 > >  >
 > >  > Good idea! That's be great!
 > >
 > > Nothing has happened on this and I suspect because it wasn't
 > > doable. (At least I couldn't figure out how it would be done. But I
 > > figured if it could that *could* it would be a great thing to do.)
 > 
 > This could have worked (in ipy_pydb - untried code):
 > 
 > -----
 > import pydb.pydb
 > import Debugger
 > 
 > pydb.pydb.Pdb = Debugger.Pdb
 > ------
 > 
 > I didn't try it myself because I figured you could have given it more
 > testing mileage. Additionally, I was off the net for a long while due
 > to moving.
 > 
 > > So recently, I've added a parameter to pydb so that runv() can get
 > > passed with an Pdb object extended in the way that ipython wants to
 > > extend it. That will be in the next release of pydb (which again can
 > > be timed so that it is no later than the next ipython release).
 > >
 > > I've modified ipy_pydb.py so that it now passes in such a Pdb
 > > object. This was the debug session can make use of pdoc and pinfo. It
 > > also uses the ipython debugger prompt (which personally I'm not a fan
 > > of but I guess it makes it ipython consistent.) There is now the
 > > possibility of using the ipython color scheme, but I couldn't figure
 > > out how to get that passed when creating the Debugger.Pdb object.
 > >
 > > And in going over IPython/Extensions/ipy_pydb.py I've noticed a couple
 > > of other things. First there should be a check that pydb can be
 > > imported and that there is a late enough version of pydb.
 > >
 > > Below is a patch.
 > 
 > 
 > Not a bad solution, I committed the patch. In general, ipy_pydb.py is
 > "your territory" and I'll gladly commit every patch I get for it
 > immediately. In theory something like ipy_pydb.py could even be
 > bundled w/ pydb, just like python-mode.el is bundled with python...
 > 
 > -- 
 > Ville M. Vainio - vivainio.googlepages.com
 > blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'
 > 


From Fernando.Perez at colorado.edu  Fri Nov 24 02:59:40 2006
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Fri, 24 Nov 2006 00:59:40 -0700
Subject: [IPython-dev] [patch 1/2] Try to use cProfile module
 (Python-2.5) instead of old profile.
In-Reply-To: <46cb515a0611231352s2c58d47ej5699ad70b448aaf9@mail.gmail.com>
References: <20061122213359.515229954@roro-510.zxlink>	
	<20061122213502.540288964@roro-510.zxlink>	
	<46cb515a0611231106n20ed0cf2ld8d75acd6591f5e4@mail.gmail.com>	
	<20061123214404.GA4614@landau.phys.spbu.ru>
	<46cb515a0611231352s2c58d47ej5699ad70b448aaf9@mail.gmail.com>
Message-ID: <4566A66C.9020008@colorado.edu>

Ville M. Vainio wrote:
> On 11/23/06, Kirill Smelkov <smelkov at mph1.phys.spbu.ru> wrote:
> 
>> That's what I originally did (see [patch 2/9], buf Fernando objected, I
>> can't understand why...
> 
> Oh, I see, he probably just accidentally misread your patch. I'll go
> ahead and apply your original patch then (tomorrow).

Yes, I did misread the original patch.  I looked at it again just now, and 
it's fine.

Thanks again Kirill for these contributions.

Cheers,

f


From vivainio at gmail.com  Fri Nov 24 11:27:57 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Fri, 24 Nov 2006 18:27:57 +0200
Subject: [IPython-dev] autoload and making %pydb be more ipythonic (was:
	Extensions/pydb_ipy.py added)
In-Reply-To: <17766.34184.188910.580317@panix3.panix.com>
References: <46cb515a0610280147u5a310ffev181ec352bc9f83cd@mail.gmail.com>
	<17731.8786.756329.573337@panix3.panix.com>
	<46cb515a0610280347n73539bdt6810659b7584e3a@mail.gmail.com>
	<17731.15518.833930.352892@panix3.panix.com>
	<46cb515a0610280541n1ccc4691o4a47f1ca404b826c@mail.gmail.com>
	<17731.31618.644784.714712@panix3.panix.com>
	<17765.7652.52085.132851@panix3.panix.com>
	<46cb515a0611231148h4ed206fi56adf53bd35ecc8@mail.gmail.com>
	<17766.34184.188910.580317@panix3.panix.com>
Message-ID: <46cb515a0611240827p7b2ac7ei40bb480b72e90d27@mail.gmail.com>

On 11/24/06, R. Bernstein <rocky at panix.com> wrote:

> I don't have a problem adding ipy_pydb.py in the pydb package. But the one thing though that I think would somehow be nice is that if one *didn't* have to do run "import ipy_pydb" to allow %pydb to work. Given that one can put this in a "try" block, I don't see any downside of doing this.

It's just that having to import ipy_pydb would make enabling pydb a
configuration option. Sometimes one may wish to use standard pdb even
when pydb is installed.

And the import need not be manual - it's quite easy to add "import
ipy_pydb" to your personal ipy_user_conf.py.

> A cool and more general mechanism for ipython extensions would be something along the lines of Emacs Lisp "autoload". Say if I run %xxx ... then if that's not registered, try to run "import ipy_xxx" (where for example xxx = pydb), and then run %xxx ....  Comments?

Well, this might save us some startup time & memory footprint. It
should be quite easy to do, esp. for magics (and completers). We can
easily inject stub callables in place of magics, which 1) import a
specified module and 2) try to run the magic again.

I think a good first line of action would be to add an ipapi function
that permanently registers a magic name / completer trigger (or
whatever)  with ipython.

Here's what one could do e.g. in setup.py for pydb:

import IPython.ipapi
ip = IPython.ipapi.get()

ip.register_autoload("%pydb", "pydb.ipy_pydb")

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From smelkov at mph1.phys.spbu.ru  Fri Nov 24 16:29:19 2006
From: smelkov at mph1.phys.spbu.ru (Kirill Smelkov)
Date: Sat, 25 Nov 2006 00:29:19 +0300
Subject: [IPython-dev] [patch 1/2] Try to use cProfile module
	(Python-2.5) instead of old profile.
In-Reply-To: <4566A66C.9020008@colorado.edu>
References: <20061122213359.515229954@roro-510.zxlink>
	<20061122213502.540288964@roro-510.zxlink>
	<46cb515a0611231106n20ed0cf2ld8d75acd6591f5e4@mail.gmail.com>
	<20061123214404.GA4614@landau.phys.spbu.ru>
	<46cb515a0611231352s2c58d47ej5699ad70b448aaf9@mail.gmail.com>
	<4566A66C.9020008@colorado.edu>
Message-ID: <20061124212919.GA4802@landau.phys.spbu.ru>

On Fri, Nov 24, 2006 at 12:59:40AM -0700, Fernando Perez wrote:
> Ville M. Vainio wrote:
> >On 11/23/06, Kirill Smelkov <smelkov at mph1.phys.spbu.ru> wrote:
> >
> >>That's what I originally did (see [patch 2/9], buf Fernando objected, I
> >>can't understand why...
> >
> >Oh, I see, he probably just accidentally misread your patch. I'll go
> >ahead and apply your original patch then (tomorrow).
> 
> Yes, I did misread the original patch.  I looked at it again just now, and 
> it's fine.
Thanks. Can happen with anyone...

-- 
    ????? ????????, ??????.
    http://landau.phys.spbu.ru/~kirr/aiv/


From rocky at panix.com  Sat Nov 25 18:06:30 2006
From: rocky at panix.com (R. Bernstein)
Date: Sat, 25 Nov 2006 18:06:30 -0500
Subject: [IPython-dev] autoload and making %pydb be more ipythonic (was:
	Extensions/pydb_ipy.py added)
In-Reply-To: <46cb515a0611240827p7b2ac7ei40bb480b72e90d27@mail.gmail.com>
References: <46cb515a0610280147u5a310ffev181ec352bc9f83cd@mail.gmail.com>
	<17731.8786.756329.573337@panix3.panix.com>
	<46cb515a0610280347n73539bdt6810659b7584e3a@mail.gmail.com>
	<17731.15518.833930.352892@panix3.panix.com>
	<46cb515a0610280541n1ccc4691o4a47f1ca404b826c@mail.gmail.com>
	<17731.31618.644784.714712@panix3.panix.com>
	<17765.7652.52085.132851@panix3.panix.com>
	<46cb515a0611231148h4ed206fi56adf53bd35ecc8@mail.gmail.com>
	<17766.34184.188910.580317@panix3.panix.com>
	<46cb515a0611240827p7b2ac7ei40bb480b72e90d27@mail.gmail.com>
Message-ID: <17768.52342.879269.903593@panix3.panix.com>

Ville M. Vainio writes:
 ...
 > Well, this might save us some startup time & memory footprint. 

Good!

 > It should be quite easy to do, esp. for magics (and completers). We can
 > easily inject stub callables in place of magics, which 1) import a
 > specified module and 2) try to run the magic again.

Good!
 > 
 > I think a good first line of action would be to add an ipapi function
 > that permanently registers a magic name / completer trigger (or
 > whatever)  with ipython.

So this ipapi function doesn't exist yet, right? 

 > 
 > Here's what one could do e.g. in setup.py for pydb:
 > 
 > import IPython.ipapi
 > ip = IPython.ipapi.get()
 > 
 > ip.register_autoload("%pydb", "pydb.ipy_pydb")

I'm guessing from the above that the ipapi function that needs to get
added is called "register_autoload", correct? (Right now I don't see
"register_autoload" in ipython sources.)

The other thing that is not clear to me where or when this
register_autoload is invoked. It has to have been invoked before
running the first time %pydb is entered by a user since the autoload
indicates what to import. It doesn't make sense to do anything when
installing pydb. And the goal here was to be able to run "%pydb"
without having to run an "import" first. And so it doesn't make sense
to put this as the initialization of an import, either.

The two natural places then would be in ipy_user_conf.py or
ipy_system_conf.py. And it looks like all of the other extensions in
ipython are imported in the latter.


From rocky at panix.com  Sat Nov 25 18:22:38 2006
From: rocky at panix.com (R. Bernstein)
Date: Sat, 25 Nov 2006 18:22:38 -0500
Subject: [IPython-dev] autoload and making %pydb be more ipythonic (was:
	Extensions/pydb_ipy.py added)
In-Reply-To: <46cb515a0611240827p7b2ac7ei40bb480b72e90d27@mail.gmail.com>
References: <46cb515a0610280147u5a310ffev181ec352bc9f83cd@mail.gmail.com>
	<17731.8786.756329.573337@panix3.panix.com>
	<46cb515a0610280347n73539bdt6810659b7584e3a@mail.gmail.com>
	<17731.15518.833930.352892@panix3.panix.com>
	<46cb515a0610280541n1ccc4691o4a47f1ca404b826c@mail.gmail.com>
	<17731.31618.644784.714712@panix3.panix.com>
	<17765.7652.52085.132851@panix3.panix.com>
	<46cb515a0611231148h4ed206fi56adf53bd35ecc8@mail.gmail.com>
	<17766.34184.188910.580317@panix3.panix.com>
	<46cb515a0611240827p7b2ac7ei40bb480b72e90d27@mail.gmail.com>
Message-ID: <17768.53310.125276.760922@panix3.panix.com>

Ville M. Vainio writes:
 > It's just that having to import ipy_pydb would make enabling pydb a
 > configuration option. Sometimes one may wish to use standard pdb even
 > when pydb is installed.

But one can't run %pydb and get pdb, so there's no ambiguity
here. (For pdb one needs "%run -d") 

 > 
 > And the import need not be manual - it's quite easy to add "import
 > ipy_pydb" to your personal ipy_user_conf.py.

Okay, will do for now. Thanks.


From vivainio at gmail.com  Sun Nov 26 03:43:13 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Sun, 26 Nov 2006 10:43:13 +0200
Subject: [IPython-dev] autoload and making %pydb be more ipythonic (was:
	Extensions/pydb_ipy.py added)
In-Reply-To: <17768.52342.879269.903593@panix3.panix.com>
References: <46cb515a0610280147u5a310ffev181ec352bc9f83cd@mail.gmail.com>
	<46cb515a0610280347n73539bdt6810659b7584e3a@mail.gmail.com>
	<17731.15518.833930.352892@panix3.panix.com>
	<46cb515a0610280541n1ccc4691o4a47f1ca404b826c@mail.gmail.com>
	<17731.31618.644784.714712@panix3.panix.com>
	<17765.7652.52085.132851@panix3.panix.com>
	<46cb515a0611231148h4ed206fi56adf53bd35ecc8@mail.gmail.com>
	<17766.34184.188910.580317@panix3.panix.com>
	<46cb515a0611240827p7b2ac7ei40bb480b72e90d27@mail.gmail.com>
	<17768.52342.879269.903593@panix3.panix.com>
Message-ID: <46cb515a0611260043l5052ff22tc2d624676298dfa8@mail.gmail.com>

On 11/26/06, R. Bernstein <rocky at panix.com> wrote:

>  > I think a good first line of action would be to add an ipapi function
>  > that permanently registers a magic name / completer trigger (or
>  > whatever)  with ipython.
>
> So this ipapi function doesn't exist yet, right?

Right.

>  > Here's what one could do e.g. in setup.py for pydb:
>  >
>  > import IPython.ipapi
>  > ip = IPython.ipapi.get()
>  >
>  > ip.register_autoload("%pydb", "pydb.ipy_pydb")
>
> I'm guessing from the above that the ipapi function that needs to get
> added is called "register_autoload", correct? (Right now I don't see
> "register_autoload" in ipython sources.)

Yes. register_autoload doesn't exist yet, I'm still merely
branistroming some ideas.

> indicates what to import. It doesn't make sense to do anything when
> installing pydb. And the goal here was to be able to run "%pydb"

Why doesn't it make sense to do this when installing pydb? Isn't
installation the exact time when things like this should be done?

> The two natural places then would be in ipy_user_conf.py or
> ipy_system_conf.py. And it looks like all of the other extensions in

These are imported every time ipython starts. register_autoload would
be a persistent operation, i.e. it should only be run only once and
ipython stores the autoload map to somewhere in the ".ipython"
directory which it reads on startup.

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From vivainio at gmail.com  Sun Nov 26 03:43:55 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Sun, 26 Nov 2006 10:43:55 +0200
Subject: [IPython-dev] autoload and making %pydb be more ipythonic (was:
	Extensions/pydb_ipy.py added)
In-Reply-To: <17768.53310.125276.760922@panix3.panix.com>
References: <46cb515a0610280147u5a310ffev181ec352bc9f83cd@mail.gmail.com>
	<46cb515a0610280347n73539bdt6810659b7584e3a@mail.gmail.com>
	<17731.15518.833930.352892@panix3.panix.com>
	<46cb515a0610280541n1ccc4691o4a47f1ca404b826c@mail.gmail.com>
	<17731.31618.644784.714712@panix3.panix.com>
	<17765.7652.52085.132851@panix3.panix.com>
	<46cb515a0611231148h4ed206fi56adf53bd35ecc8@mail.gmail.com>
	<17766.34184.188910.580317@panix3.panix.com>
	<46cb515a0611240827p7b2ac7ei40bb480b72e90d27@mail.gmail.com>
	<17768.53310.125276.760922@panix3.panix.com>
Message-ID: <46cb515a0611260043w6dee36eek9881b7f937aea92d@mail.gmail.com>

On 11/26/06, R. Bernstein <rocky at panix.com> wrote:

> Ville M. Vainio writes:
>  > It's just that having to import ipy_pydb would make enabling pydb a
>  > configuration option. Sometimes one may wish to use standard pdb even
>  > when pydb is installed.
>
> But one can't run %pydb and get pdb, so there's no ambiguity
> here. (For pdb one needs "%run -d")

But for post-mortem, there's no such choice.

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From vivainio at gmail.com  Sun Nov 26 05:23:34 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Sun, 26 Nov 2006 12:23:34 +0200
Subject: [IPython-dev] autoload and making %pydb be more ipythonic (was:
	Extensions/pydb_ipy.py added)
In-Reply-To: <46cb515a0611260043l5052ff22tc2d624676298dfa8@mail.gmail.com>
References: <46cb515a0610280147u5a310ffev181ec352bc9f83cd@mail.gmail.com>
	<17731.15518.833930.352892@panix3.panix.com>
	<46cb515a0610280541n1ccc4691o4a47f1ca404b826c@mail.gmail.com>
	<17731.31618.644784.714712@panix3.panix.com>
	<17765.7652.52085.132851@panix3.panix.com>
	<46cb515a0611231148h4ed206fi56adf53bd35ecc8@mail.gmail.com>
	<17766.34184.188910.580317@panix3.panix.com>
	<46cb515a0611240827p7b2ac7ei40bb480b72e90d27@mail.gmail.com>
	<17768.52342.879269.903593@panix3.panix.com>
	<46cb515a0611260043l5052ff22tc2d624676298dfa8@mail.gmail.com>
Message-ID: <46cb515a0611260223o29b3bff9qd607b6dfa8456385@mail.gmail.com>

On 11/26/06, Ville M. Vainio <vivainio at gmail.com> wrote:

> Why doesn't it make sense to do this when installing pydb? Isn't
> installation the exact time when things like this should be done?

Answering myself -

The registration should probably be performed both during installation
and import of the module. In installation the registration only
applies to the user that happens to run setup.py, but other users can
do the registration manually by just importing the module once.

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From rocky at panix.com  Sun Nov 26 10:01:48 2006
From: rocky at panix.com (R. Bernstein)
Date: Sun, 26 Nov 2006 10:01:48 -0500
Subject: [IPython-dev] autoload and making %pydb be more ipythonic (was:
	Extensions/pydb_ipy.py added)
In-Reply-To: <46cb515a0611260223o29b3bff9qd607b6dfa8456385@mail.gmail.com>
References: <46cb515a0610280147u5a310ffev181ec352bc9f83cd@mail.gmail.com>
	<17731.15518.833930.352892@panix3.panix.com>
	<46cb515a0610280541n1ccc4691o4a47f1ca404b826c@mail.gmail.com>
	<17731.31618.644784.714712@panix3.panix.com>
	<17765.7652.52085.132851@panix3.panix.com>
	<46cb515a0611231148h4ed206fi56adf53bd35ecc8@mail.gmail.com>
	<17766.34184.188910.580317@panix3.panix.com>
	<46cb515a0611240827p7b2ac7ei40bb480b72e90d27@mail.gmail.com>
	<17768.52342.879269.903593@panix3.panix.com>
	<46cb515a0611260043l5052ff22tc2d624676298dfa8@mail.gmail.com>
	<46cb515a0611260223o29b3bff9qd607b6dfa8456385@mail.gmail.com>
Message-ID: <17769.44124.934324.948419@panix3.panix.com>

Ville M. Vainio writes:
 > On 11/26/06, Ville M. Vainio <vivainio at gmail.com> wrote:
 > 
 > > Why doesn't it make sense to do this when installing pydb? Isn't
 > > installation the exact time when things like this should be done?
 > 
 > Answering myself -
 > 
 > The registration should probably be performed both during installation
 > and import of the module. In installation the registration only
 > applies to the user that happens to run setup.py, but other users can
 > do the registration manually by just importing the module once.

What happens if I install pydb first and then install ipython? And as
for doing somehting ipython-wise inside a pydb import, you have the
same issue as you brought up before: I may import pydb outside of
ipython, and/or may not want it to get used automatically or every
time inside ipython.

For now, ipy_user_conf.py seems like right place to put an "import
ipy_pydb" to indicate you want %pydb to work.


From rocky at panix.com  Sun Nov 26 10:06:40 2006
From: rocky at panix.com (R. Bernstein)
Date: Sun, 26 Nov 2006 10:06:40 -0500
Subject: [IPython-dev] autoload and making %pydb be more ipythonic (was:
	Extensions/pydb_ipy.py added)
In-Reply-To: <46cb515a0611260043w6dee36eek9881b7f937aea92d@mail.gmail.com>
References: <46cb515a0610280147u5a310ffev181ec352bc9f83cd@mail.gmail.com>
	<46cb515a0610280347n73539bdt6810659b7584e3a@mail.gmail.com>
	<17731.15518.833930.352892@panix3.panix.com>
	<46cb515a0610280541n1ccc4691o4a47f1ca404b826c@mail.gmail.com>
	<17731.31618.644784.714712@panix3.panix.com>
	<17765.7652.52085.132851@panix3.panix.com>
	<46cb515a0611231148h4ed206fi56adf53bd35ecc8@mail.gmail.com>
	<17766.34184.188910.580317@panix3.panix.com>
	<46cb515a0611240827p7b2ac7ei40bb480b72e90d27@mail.gmail.com>
	<17768.53310.125276.760922@panix3.panix.com>
	<46cb515a0611260043w6dee36eek9881b7f937aea92d@mail.gmail.com>
Message-ID: <17769.44416.663746.783851@panix3.panix.com>

Ville M. Vainio writes:
 > On 11/26/06, R. Bernstein <rocky at panix.com> wrote:
 > 
 > > Ville M. Vainio writes:
 > >  > It's just that having to import ipy_pydb would make enabling pydb a
 > >  > configuration option. Sometimes one may wish to use standard pdb even
 > >  > when pydb is installed.
 > >
 > > But one can't run %pydb and get pdb, so there's no ambiguity
 > > here. (For pdb one needs "%run -d")
 > 
 > But for post-mortem, there's no such choice.

Yes, and the way things are done right now in ipython that is a
completly separate issue. This thread was about how not require an
"import ipy_pydb" if someone wants to use %pydb.


From vivainio at gmail.com  Sun Nov 26 10:20:37 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Sun, 26 Nov 2006 17:20:37 +0200
Subject: [IPython-dev] autoload and making %pydb be more ipythonic (was:
	Extensions/pydb_ipy.py added)
In-Reply-To: <17769.44124.934324.948419@panix3.panix.com>
References: <46cb515a0610280147u5a310ffev181ec352bc9f83cd@mail.gmail.com>
	<17731.31618.644784.714712@panix3.panix.com>
	<17765.7652.52085.132851@panix3.panix.com>
	<46cb515a0611231148h4ed206fi56adf53bd35ecc8@mail.gmail.com>
	<17766.34184.188910.580317@panix3.panix.com>
	<46cb515a0611240827p7b2ac7ei40bb480b72e90d27@mail.gmail.com>
	<17768.52342.879269.903593@panix3.panix.com>
	<46cb515a0611260043l5052ff22tc2d624676298dfa8@mail.gmail.com>
	<46cb515a0611260223o29b3bff9qd607b6dfa8456385@mail.gmail.com>
	<17769.44124.934324.948419@panix3.panix.com>
Message-ID: <46cb515a0611260720q68745e6pbc8147f329901f17@mail.gmail.com>

On 11/26/06, R. Bernstein <rocky at panix.com> wrote:

> What happens if I install pydb first and then install ipython? And as

Then you'll need to run "import ipy_pydb" once in ipython.

> for doing somehting ipython-wise inside a pydb import, you have the
> same issue as you brought up before: I may import pydb outside of
> ipython, and/or may not want it to get used automatically or every
> time inside ipython.

IPython.ipapi.get() returns None if you are not running in IPython.
This, and "except ImportError" can be done to safely determine whether
you should do ipython related stuff.

> For now, ipy_user_conf.py seems like right place to put an "import
> ipy_pydb" to indicate you want %pydb to work.

Exactly. And it will stay as the recommended place to load extensions.

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From vivainio at gmail.com  Sun Nov 26 10:25:29 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Sun, 26 Nov 2006 17:25:29 +0200
Subject: [IPython-dev] Mercurial queues
Message-ID: <46cb515a0611260725j5d1a8bb4v75c7503c1e9348f8@mail.gmail.com>

A blast from the past - Stefan Reich?r <stefan at xsteve.at> wrote:

> I also think that bzr is a good solution for that kind of stuff.
> However, I use Mercurial Queues (mq) for tracking svn repositories and
> generating patches.

I took a look at mq myself, and it seems like a very good idea when
working off-SVN:

http://www.selenic.com/mercurial/wiki/index.cgi/MqExtension

All in all, Mercurial looks / feels very promising...

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From vivainio at gmail.com  Sun Nov 26 12:41:07 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Sun, 26 Nov 2006 19:41:07 +0200
Subject: [IPython-dev] Mercurial queues
In-Reply-To: <46cb515a0611260725j5d1a8bb4v75c7503c1e9348f8@mail.gmail.com>
References: <46cb515a0611260725j5d1a8bb4v75c7503c1e9348f8@mail.gmail.com>
Message-ID: <46cb515a0611260941v45b9639ax2b325b3e4c51de5@mail.gmail.com>

On 11/26/06, Ville M. Vainio <vivainio at gmail.com> wrote:

> All in all, Mercurial looks / feels very promising...

And on that tangent, there's now a stock completer for "hg" which you
can invoke by pressing tab after typing the hg command:

[IPython]|8> hg q<TAB>
qapplied   qdiff      qimport    qpop       qrename    qseries
qclone     qfold      qinit      qprev      qrestore   qtop
qcommit    qguard     qnew       qpush      qsave      qunapplied
qdelete    qheader    qnext      qrefresh   qselect
[IPython]|8> hg q

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From vivainio at gmail.com  Sun Nov 26 16:34:30 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Sun, 26 Nov 2006 23:34:30 +0200
Subject: [IPython-dev] svnversion used for Release.py
Message-ID: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com>

I've added a new tool, tools/update_revnum.py that modifies Release.py
(gets the revision number with svnversion command; svn doesn't support
getting global revision command via $Revision$ like tags).

Now the distribution names of svn versions are handier:

http://vivainio.googlepages.com/ipython-0.7.3.svn.r1937.win32.exe

I think we could put out a beta / whatever release soon.

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From stefan at sun.ac.za  Mon Nov 27 03:08:03 2006
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Mon, 27 Nov 2006 10:08:03 +0200
Subject: [IPython-dev] Mercurial queues
In-Reply-To: <46cb515a0611260941v45b9639ax2b325b3e4c51de5@mail.gmail.com>
References: <46cb515a0611260725j5d1a8bb4v75c7503c1e9348f8@mail.gmail.com>
	<46cb515a0611260941v45b9639ax2b325b3e4c51de5@mail.gmail.com>
Message-ID: <20061127080803.GB30805@mentat.za.net>

Hi Ville

Please find a patch for bzr completion attached.

Regards
St?fan

On Sun, Nov 26, 2006 at 07:41:07PM +0200, Ville M. Vainio wrote:
> On 11/26/06, Ville M. Vainio <vivainio at gmail.com> wrote:
> 
> > All in all, Mercurial looks / feels very promising...
> 
> And on that tangent, there's now a stock completer for "hg" which you
> can invoke by pressing tab after typing the hg command:
> 
> [IPython]|8> hg q<TAB>
> qapplied   qdiff      qimport    qpop       qrename    qseries
> qclone     qfold      qinit      qprev      qrestore   qtop
> qcommit    qguard     qnew       qpush      qsave      qunapplied
> qdelete    qheader    qnext      qrefresh   qselect
> [IPython]|8> hg q
-------------- next part --------------
Index: IPython/Extensions/ipy_stock_completers.py
===================================================================
--- IPython/Extensions/ipy_stock_completers.py	(revision 1942)
+++ IPython/Extensions/ipy_stock_completers.py	(working copy)
@@ -113,6 +113,42 @@
 ip.set_hook('complete_command', hg_completer, str_key = 'hg')

 

 

+bzr_commands = """

+add annotate bind branch break-lock bundle-revisions cat check

+checkout commit conflicts deleted diff export gannotate gbranch

+gcommit gdiff help ignore ignored info init init-repository inventory

+log merge missing mkdir mv nick pull push reconcile register-branch

+remerge remove renames resolve revert revno root serve sign-my-commits

+status testament unbind uncommit unknowns update upgrade version

+version-info visualise whoami

+"""

+

+def bzr_completer(self,event):

+    """ Completer for bazaar commands """

+    cmd_param = event.line.split()

+    if event.line.endswith(' '):

+        cmd_param.append('')

+

+    if len(cmd_param) > 2:

+        cmd = cmd_param[1]

+        param = cmd_param[-1]

+        output_file = (param == '--output=')

+        if cmd == 'help':

+            return bzr_commands.split()

+        elif cmd in ['bundle-revisions','conflicts',

+                     'deleted','nick','register-branch',

+                     'serve','unbind','upgrade','version',

+                     'whoami'] and not output_file:

+            return []

+        else:

+            # the rest are probably file names

+            return ip.IP.Completer.file_matches(event.symbol)

+

+    return bzr_commands.split()

+

+ip.set_hook('complete_command', bzr_completer, str_key = 'bzr')

+

+

 def runlistpy(self, event):

     comps = shlex.split(event.line)

     relpath = (len(comps) > 1 and comps[-1] or '')


From vivainio at gmail.com  Mon Nov 27 06:25:28 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Mon, 27 Nov 2006 13:25:28 +0200
Subject: [IPython-dev] Mercurial queues
In-Reply-To: <20061127105147.GA32697@mentat.za.net>
References: <46cb515a0611260725j5d1a8bb4v75c7503c1e9348f8@mail.gmail.com>
	<46cb515a0611260941v45b9639ax2b325b3e4c51de5@mail.gmail.com>
	<20061127080803.GB30805@mentat.za.net>
	<46cb515a0611270149s200603a6n9a1c49d04a656bab@mail.gmail.com>
	<20061127105147.GA32697@mentat.za.net>
Message-ID: <46cb515a0611270325y2314ad6am2780b02914400e1a@mail.gmail.com>

On 11/27/06, Stefan van der Walt <stefan at sun.ac.za> wrote:

> What in the world happened there?!  The file is fine on my machine,
> but I also see the line-feeds in the incoming message.
>
> Here it is again, gzipped.

It worked, thanks! I wonder whether we could someday add also options
for the commands (e.g. svn import --username) - though I suppose it
would have to be after we get the autoload stuff in (post-0.7.3), to
avoid too large databases.

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From stefan at sun.ac.za  Mon Nov 27 07:12:45 2006
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Mon, 27 Nov 2006 14:12:45 +0200
Subject: [IPython-dev] Mercurial queues
In-Reply-To: <46cb515a0611270325y2314ad6am2780b02914400e1a@mail.gmail.com>
References: <46cb515a0611260725j5d1a8bb4v75c7503c1e9348f8@mail.gmail.com>
	<46cb515a0611260941v45b9639ax2b325b3e4c51de5@mail.gmail.com>
	<20061127080803.GB30805@mentat.za.net>
	<46cb515a0611270149s200603a6n9a1c49d04a656bab@mail.gmail.com>
	<20061127105147.GA32697@mentat.za.net>
	<46cb515a0611270325y2314ad6am2780b02914400e1a@mail.gmail.com>
Message-ID: <20061127121245.GC32697@mentat.za.net>

Hi Ville

On Mon, Nov 27, 2006 at 01:25:28PM +0200, Ville M. Vainio wrote:
> On 11/27/06, Stefan van der Walt <stefan at sun.ac.za> wrote:
> 
> >What in the world happened there?!  The file is fine on my machine,
> >but I also see the line-feeds in the incoming message.
> >
> >Here it is again, gzipped.
> 
> It worked, thanks! I wonder whether we could someday add also options
> for the commands (e.g. svn import --username) - though I suppose it
> would have to be after we get the autoload stuff in (post-0.7.3), to
> avoid too large databases.

I thought about this too.  I was trying to imaging what kind of
structure one would need to describe the version control completer.
One would intuitively build something like

vc_commands = ['info','update', 'bundle', ...]
vc_subcommands = {'bundle' : ['--output','-r'], ...}
vc_subcommand_expand = {'--output': file_complete,
		        '-r': revision_complete,
			...}

def revision_complete(self, input):
    rev_re = re.compile('...')
    cur_rev = rev_re.match(commands.getoutput('bzr log -r -1')...)
    
    return ['BASE',cur_rev,'TIP',...]


On the other hand, how do we keep such structures up to date?  It
would be ideal if the program itself could provide a clue to its
completion.

I think these structures already exist in bzr and hg.  In fact,
looking in the directory tree, I think we are looking for

bzrlib/shellcomplete.py

I'll grab the latest copy of bzr and play around.

Cheers
St?fan


From vivainio at gmail.com  Mon Nov 27 07:34:50 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Mon, 27 Nov 2006 14:34:50 +0200
Subject: [IPython-dev] Mercurial queues
In-Reply-To: <20061127121245.GC32697@mentat.za.net>
References: <46cb515a0611260725j5d1a8bb4v75c7503c1e9348f8@mail.gmail.com>
	<46cb515a0611260941v45b9639ax2b325b3e4c51de5@mail.gmail.com>
	<20061127080803.GB30805@mentat.za.net>
	<46cb515a0611270149s200603a6n9a1c49d04a656bab@mail.gmail.com>
	<20061127105147.GA32697@mentat.za.net>
	<46cb515a0611270325y2314ad6am2780b02914400e1a@mail.gmail.com>
	<20061127121245.GC32697@mentat.za.net>
Message-ID: <46cb515a0611270434u7aab0449x3f3492116c1a2126@mail.gmail.com>

On 11/27/06, Stefan van der Walt <stefan at sun.ac.za> wrote:

> I thought about this too.  I was trying to imaging what kind of
> structure one would need to describe the version control completer.
> One would intuitively build something like
>
> vc_commands = ['info','update', 'bundle', ...]
> vc_subcommands = {'bundle' : ['--output','-r'], ...}
> vc_subcommand_expand = {'--output': file_complete,
>                         '-r': revision_complete,
>                         ...}
>
> def revision_complete(self, input):
>     rev_re = re.compile('...')
>     cur_rev = rev_re.match(commands.getoutput('bzr log -r -1')...)
>
>     return ['BASE',cur_rev,'TIP',...]

I think you might be overengineering a little bit. Just looking at the
command and yielding ['--output','-r'] + file_completions would be
just fine for most purposes. That way, if the user types - and presses
tab, she gets the list of options for that command.

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From fperez.net at gmail.com  Tue Nov 28 09:27:07 2006
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 28 Nov 2006 07:27:07 -0700
Subject: [IPython-dev] svnversion used for Release.py
In-Reply-To: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com>
References: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com>
Message-ID: <db6b5ecc0611280627y6dfbdc87ta8ff332a93e07299@mail.gmail.com>

On 11/26/06, Ville M. Vainio <vivainio at gmail.com> wrote:
> I've added a new tool, tools/update_revnum.py that modifies Release.py
> (gets the revision number with svnversion command; svn doesn't support
> getting global revision command via $Revision$ like tags).
>
> Now the distribution names of svn versions are handier:
>
> http://vivainio.googlepages.com/ipython-0.7.3.svn.r1937.win32.exe

Great, thanks.  I'd been wanting this for a long time.

> I think we could put out a beta / whatever release soon.

+1.  I'll be with limited online activity this week, and pretty much
offline the next.  We could target a release for the end of the year.
Feel free to put out the betas sooner though, or let me know and I can
upload one to the dist/testing dir anytime (before Saturday when I go
offline).

Cheers,

f


From fperez.net at gmail.com  Wed Nov 29 01:00:04 2006
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 28 Nov 2006 23:00:04 -0700
Subject: [IPython-dev] svnversion used for Release.py
In-Reply-To: <17772.56028.653337.85310@panix3.panix.com>
References: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com>
	<db6b5ecc0611280627y6dfbdc87ta8ff332a93e07299@mail.gmail.com>
	<17772.56028.653337.85310@panix3.panix.com>
Message-ID: <db6b5ecc0611282200t6db7441cna5a0d7cb18bfc02f@mail.gmail.com>

On 11/28/06, R. Bernstein <rocky at panix.com> wrote:
> Before you go offline and before the next ipython release comes out,
> it would be good to get this from
>   http://projects.scipy.org/pipermail/ipython-dev/2006-October/002413.html
> cleared up. Thanks ...
>
> ----
>
> One last strongly related topic which includes Fernando's comment about
> the necessity of Python 2.4 support. It was the following readline
> workaround in Debugger.py around line 68 that caused me to initially
> decided to punt on Python < 2.5:
>
>
>     if sys.version[:3] >= '2.5':
>         def __init__(self,color_scheme='NoColor',completekey=None,
>                      stdin=None, stdout=None):
>             ... # I'll come back to this below
>     else:
>         # Ugly hack: for Python 2.3-2.4, we can't call the parent constructor,
>         # because it binds readline and breaks tab-completion.  This means we
>         # have to COPY the constructor here.
>         def __init__(self,color_scheme='NoColor'):
>         ... #1
>
> I think the intent at #1 above is basically to copy the code in pdb's
> __init__ constructor omitting "import readline" and the code associated
> with implications of that.
>
> So it looks to me like the test on version 2.5 has more to do with
> testing the version of *pdb* used than anything else version 2.4 or
> version 2.5. Is this correct?
>
> Assuming this is the case, when I look the 2.4 pdb versus the 2.5 pdb,
> I see THAT THEY BOTH IMPORT READLINE. In other words the 2.5 branch
> ("I'll come back to this below") should have pdb code copied minus the
> import readline. This has to be a separate branch still because it is
> different code. But none of it appears, so I'd be curious to learn why
> it's not needed in the 2.5 branch.

No, the problem is a bit more subtle than that: it's not the 'import
readline' that causes problems, but rather the fact that the
constructor fires this call:

2.4:
class Pdb(bdb.Bdb, cmd.Cmd):

    def __init__(self):
        bdb.Bdb.__init__(self)
        cmd.Cmd.__init__(self)

2.5:
class Pdb(bdb.Bdb, cmd.Cmd):

    def __init__(self, completekey='tab', stdin=None, stdout=None):
        bdb.Bdb.__init__(self)
        cmd.Cmd.__init__(self, completekey, stdin, stdout)

The critical difference is that the 2.5 constructor exposes the
'completekey' parameter publicly, and correctly passes it down to the
cmd.Cmd constructor, so we can nicely override it to None and avoid
any readline-related misconfiguration issues.  Prior to 2.5, this
parameter was not accessible from the Pdb constructor and the default
value of 'tab' would always fire.


Does that help?

Cheers,

f


From fperez.net at gmail.com  Wed Nov 29 01:03:11 2006
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 28 Nov 2006 23:03:11 -0700
Subject: [IPython-dev] svnversion used for Release.py
In-Reply-To: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com>
References: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com>
Message-ID: <db6b5ecc0611282203y19b91af6h6efa70842cc173a5@mail.gmail.com>

On 11/26/06, Ville M. Vainio <vivainio at gmail.com> wrote:
> I've added a new tool, tools/update_revnum.py that modifies Release.py
> (gets the revision number with svnversion command; svn doesn't support
> getting global revision command via $Revision$ like tags).
>
> Now the distribution names of svn versions are handier:
>
> http://vivainio.googlepages.com/ipython-0.7.3.svn.r1937.win32.exe
>
> I think we could put out a beta / whatever release soon.

You may have already noticed, but %pdb is systematically crashing.
I'm getting the dreaded:


ipdb>q
*** ERROR ***
This version of pdb has a bug and crashed.
Returning to IPython...
WARNING: Failure executing file: <error.py>

on 2.4 and 2.5 for even the most trivial tests.  I won't be able to
debug this before I leave, but I know it was working fine not very
long ago, b/c I use %pdb pretty regularly.  You might have an idea of
what happened here...

Cheers,

f


From vivainio at gmail.com  Wed Nov 29 02:43:11 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Wed, 29 Nov 2006 09:43:11 +0200
Subject: [IPython-dev] svnversion used for Release.py
In-Reply-To: <db6b5ecc0611282203y19b91af6h6efa70842cc173a5@mail.gmail.com>
References: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com>
	<db6b5ecc0611282203y19b91af6h6efa70842cc173a5@mail.gmail.com>
Message-ID: <46cb515a0611282343p620c24ebi6ffb2c7ccc2a784c@mail.gmail.com>

On 11/29/06, Fernando Perez <fperez.net at gmail.com> wrote:

> You may have already noticed, but %pdb is systematically crashing.
> I'm getting the dreaded:
>
>
> ipdb>q
> *** ERROR ***
> This version of pdb has a bug and crashed.
> Returning to IPython...
> WARNING: Failure executing file: <error.py>
>
> on 2.4 and 2.5 for even the most trivial tests.  I won't be able to
> debug this before I leave, but I know it was working fine not very
> long ago, b/c I use %pdb pretty regularly.  You might have an idea of
> what happened here...

Yes, it's the old problem with "old_all_completions". It's an easy
fix, I'll attend to it today.

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From vivainio at gmail.com  Wed Nov 29 03:04:00 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Wed, 29 Nov 2006 10:04:00 +0200
Subject: [IPython-dev] svnversion used for Release.py
In-Reply-To: <46cb515a0611282343p620c24ebi6ffb2c7ccc2a784c@mail.gmail.com>
References: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com>
	<db6b5ecc0611282203y19b91af6h6efa70842cc173a5@mail.gmail.com>
	<46cb515a0611282343p620c24ebi6ffb2c7ccc2a784c@mail.gmail.com>
Message-ID: <46cb515a0611290004s2a03eb5r49dbedddf1a8ddb8@mail.gmail.com>

On 11/29/06, Ville M. Vainio <vivainio at gmail.com> wrote:

> > on 2.4 and 2.5 for even the most trivial tests.  I won't be able to
> > debug this before I leave, but I know it was working fine not very
> > long ago, b/c I use %pdb pretty regularly.  You might have an idea of
> > what happened here...
>
> Yes, it's the old problem with "old_all_completions". It's an easy
> fix, I'll attend to it today.

Done. rev 1953 is the new "beta 1".

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From rocky at panix.com  Wed Nov 29 03:27:59 2006
From: rocky at panix.com (R. Bernstein)
Date: Wed, 29 Nov 2006 03:27:59 -0500
Subject: [IPython-dev] svnversion used for Release.py
In-Reply-To: <46cb515a0611282343p620c24ebi6ffb2c7ccc2a784c@mail.gmail.com>
References: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com>
	<db6b5ecc0611282203y19b91af6h6efa70842cc173a5@mail.gmail.com>
	<46cb515a0611282343p620c24ebi6ffb2c7ccc2a784c@mail.gmail.com>
Message-ID: <17773.17551.326725.76625@panix3.panix.com>

Looks like it's fixed in revison 1953. There is still a stray "print"
statement I mentioned a while ago.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: pdb.diff
Type: application/octet-stream
Size: 447 bytes
Desc: Remove a stray debug "print" statement
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20061129/c31064e9/attachment.obj>
-------------- next part --------------

Thanks.

Ville M. Vainio writes:
 > On 11/29/06, Fernando Perez <fperez.net at gmail.com> wrote:
 > 
 > > You may have already noticed, but %pdb is systematically crashing.
 > > I'm getting the dreaded:
 > >
 > >
 > > ipdb>q
 > > *** ERROR ***
 > > This version of pdb has a bug and crashed.
 > > Returning to IPython...
 > > WARNING: Failure executing file: <error.py>
 > >
 > > on 2.4 and 2.5 for even the most trivial tests.  I won't be able to
 > > debug this before I leave, but I know it was working fine not very
 > > long ago, b/c I use %pdb pretty regularly.  You might have an idea of
 > > what happened here...
 > 
 > Yes, it's the old problem with "old_all_completions". It's an easy
 > fix, I'll attend to it today.
 > 
 > -- 
 > Ville M. Vainio - vivainio.googlepages.com
 > blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'
 > _______________________________________________
 > IPython-dev mailing list
 > IPython-dev at scipy.org
 > http://projects.scipy.org/mailman/listinfo/ipython-dev
 > 

From rocky at panix.com  Wed Nov 29 03:57:28 2006
From: rocky at panix.com (R. Bernstein)
Date: Wed, 29 Nov 2006 03:57:28 -0500
Subject: [IPython-dev] svnversion used for Release.py
In-Reply-To: <db6b5ecc0611282200t6db7441cna5a0d7cb18bfc02f@mail.gmail.com>
References: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com>
	<db6b5ecc0611280627y6dfbdc87ta8ff332a93e07299@mail.gmail.com>
	<17772.56028.653337.85310@panix3.panix.com>
	<db6b5ecc0611282200t6db7441cna5a0d7cb18bfc02f@mail.gmail.com>
Message-ID: <17773.19320.848097.530795@panix3.panix.com>

Fernando Perez writes:
 > No, the problem is a bit more subtle than that: it's not the 'import
 > readline' that causes problems, but rather the fact that the
 > constructor fires this call:
 > 
 > 2.4:
 > class Pdb(bdb.Bdb, cmd.Cmd):
 > 
 >     def __init__(self):
 >         bdb.Bdb.__init__(self)
 >         cmd.Cmd.__init__(self)
 > 
 > 2.5:
 > class Pdb(bdb.Bdb, cmd.Cmd):
 > 
 >     def __init__(self, completekey='tab', stdin=None, stdout=None):
 >         bdb.Bdb.__init__(self)
 >         cmd.Cmd.__init__(self, completekey, stdin, stdout)
 > 
 > The critical difference is that the 2.5 constructor exposes the
 > 'completekey' parameter publicly, and correctly passes it down to the
 > cmd.Cmd constructor, so we can nicely override it to None and avoid
 > any readline-related misconfiguration issues.  Prior to 2.5, this
 > parameter was not accessible from the Pdb constructor and the default
 > value of 'tab' would always fire.
 > 
 > 
 > Does that help?

This helps a bit, thanks. A suitably recent version of the pydb
follows the Python 2.5 constructor convention - it doesn't matter what
the Python version is. If pydb is used, (and we do check for a
suitably recent version), then that hack is *not* needed. So

class Pdb(OldPdb):
    """Modified Pdb class, does not load readline."""

    if sys.version[:3] >= '2.5'

can become

    if sys.version[:3] >= '2.5' or has_pydb:

I've attached a patch for this below.

I'm still not clear about one other thing though, and this is not
addressed in the patch but perhaps should be.

When you create the debugger you turn off any readline completion,
right? But is this correct whne pydb is used? pydb sets up command
completion correctly for itself, and I'd venture to say that this
would be more helpful in the debugger context than the default ipython
command completion.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: pydb-debug.diff
Type: application/octet-stream
Size: 923 bytes
Desc: Don't need Python 2.3, 2.4 hack if pydb is used.
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20061129/f42ac22d/attachment.obj>

From fperez.net at gmail.com  Thu Nov 30 00:28:26 2006
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 29 Nov 2006 22:28:26 -0700
Subject: [IPython-dev] svnversion used for Release.py
In-Reply-To: <17773.19320.848097.530795@panix3.panix.com>
References: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com>
	<db6b5ecc0611280627y6dfbdc87ta8ff332a93e07299@mail.gmail.com>
	<17772.56028.653337.85310@panix3.panix.com>
	<db6b5ecc0611282200t6db7441cna5a0d7cb18bfc02f@mail.gmail.com>
	<17773.19320.848097.530795@panix3.panix.com>
Message-ID: <db6b5ecc0611292128v45746aabwe7de5d1d1b16710e@mail.gmail.com>

On 11/29/06, R. Bernstein <rocky at panix.com> wrote:

> This helps a bit, thanks. A suitably recent version of the pydb
> follows the Python 2.5 constructor convention - it doesn't matter what
> the Python version is. If pydb is used, (and we do check for a
> suitably recent version), then that hack is *not* needed. So
>
> class Pdb(OldPdb):
>     """Modified Pdb class, does not load readline."""
>
>     if sys.version[:3] >= '2.5'
>
> can become
>
>     if sys.version[:3] >= '2.5' or has_pydb:
>
> I've attached a patch for this below.

I think Ville already applied it, thanks.

> I'm still not clear about one other thing though, and this is not
> addressed in the patch but perhaps should be.
>
> When you create the debugger you turn off any readline completion,
> right? But is this correct whne pydb is used? pydb sets up command
> completion correctly for itself, and I'd venture to say that this
> would be more helpful in the debugger context than the default ipython
> command completion.

IPython simply prevents pdb (the builtin one) from mucking around with
our own completers (it would otherwise call rlcompleter and completely
disable ipython's completions).  It's quite possible that this
disables pydb's completions, and in that case we should try to
accomodate both needs: preventing pdb from breaking ipython's
completion, while allowing pydb's completion to function.  If that's
not the case, patches welcome :)

I'll be 90% offline starting now until about Dec. 12.

I did manage to get a patch in which I'd wanted for a long time; since
it's debugger related, your testing and feedback would be welcome:

%debug

now activates the interactive debugger (including pydb if available)
AFTER an exception has fired, even if you had %pdb off.  This lets you
inspect a crashed code interactively without having to do the old
dance '%pdb on; rerun code; %pdb off'.  I like it a lot more.

Cheers,

f


From vivainio at gmail.com  Thu Nov 30 02:28:02 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Thu, 30 Nov 2006 09:28:02 +0200
Subject: [IPython-dev] svnversion used for Release.py
In-Reply-To: <db6b5ecc0611292128v45746aabwe7de5d1d1b16710e@mail.gmail.com>
References: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com>
	<db6b5ecc0611280627y6dfbdc87ta8ff332a93e07299@mail.gmail.com>
	<17772.56028.653337.85310@panix3.panix.com>
	<db6b5ecc0611282200t6db7441cna5a0d7cb18bfc02f@mail.gmail.com>
	<17773.19320.848097.530795@panix3.panix.com>
	<db6b5ecc0611292128v45746aabwe7de5d1d1b16710e@mail.gmail.com>
Message-ID: <46cb515a0611292328w7cb979a8vdeeed1a50982c0b3@mail.gmail.com>

On 11/30/06, Fernando Perez <fperez.net at gmail.com> wrote:

> now activates the interactive debugger (including pydb if available)
> AFTER an exception has fired, even if you had %pdb off.  This lets you
> inspect a crashed code interactively without having to do the old
> dance '%pdb on; rerun code; %pdb off'.  I like it a lot more.

Sounds great. If it works in all cases, we could as well deprecate
%pdb, which suddenly became quite pointless.

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From fperez.net at gmail.com  Thu Nov 30 02:35:05 2006
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 30 Nov 2006 00:35:05 -0700
Subject: [IPython-dev] svnversion used for Release.py
In-Reply-To: <46cb515a0611292328w7cb979a8vdeeed1a50982c0b3@mail.gmail.com>
References: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com>
	<db6b5ecc0611280627y6dfbdc87ta8ff332a93e07299@mail.gmail.com>
	<17772.56028.653337.85310@panix3.panix.com>
	<db6b5ecc0611282200t6db7441cna5a0d7cb18bfc02f@mail.gmail.com>
	<17773.19320.848097.530795@panix3.panix.com>
	<db6b5ecc0611292128v45746aabwe7de5d1d1b16710e@mail.gmail.com>
	<46cb515a0611292328w7cb979a8vdeeed1a50982c0b3@mail.gmail.com>
Message-ID: <db6b5ecc0611292335o41c5e9fcja4c1b34498149fb6@mail.gmail.com>

On 11/30/06, Ville M. Vainio <vivainio at gmail.com> wrote:
> On 11/30/06, Fernando Perez <fperez.net at gmail.com> wrote:
>
> > now activates the interactive debugger (including pydb if available)
> > AFTER an exception has fired, even if you had %pdb off.  This lets you
> > inspect a crashed code interactively without having to do the old
> > dance '%pdb on; rerun code; %pdb off'.  I like it a lot more.
>
> Sounds great. If it works in all cases, we could as well deprecate
> %pdb, which suddenly became quite pointless.

Not quite: if you're in a rapid edit/run/debug cycle, the automatic
behavior of %pdb is nice.  It's also foolproof: it fires immediately
so there's no chance for a typo on your part to clobber
sys.last_traceback and make %debug useless.

So I think that %debug is what I'll use now 90% of the time, but I
still see a (admittedly marginal) use for %pdb.

Cheers,

f


From rocky at panix.com  Thu Nov 30 04:21:41 2006
From: rocky at panix.com (R. Bernstein)
Date: Thu, 30 Nov 2006 04:21:41 -0500
Subject: [IPython-dev] svnversion used for Release.py
In-Reply-To: <db6b5ecc0611292128v45746aabwe7de5d1d1b16710e@mail.gmail.com>
References: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com>
	<db6b5ecc0611280627y6dfbdc87ta8ff332a93e07299@mail.gmail.com>
	<17772.56028.653337.85310@panix3.panix.com>
	<db6b5ecc0611282200t6db7441cna5a0d7cb18bfc02f@mail.gmail.com>
	<17773.19320.848097.530795@panix3.panix.com>
	<db6b5ecc0611292128v45746aabwe7de5d1d1b16710e@mail.gmail.com>
Message-ID: <17774.41637.724154.424378@panix3.panix.com>

Fernando Perez writes:
 > On 11/29/06, R. Bernstein <rocky at panix.com> wrote:
 > 
 > > This helps a bit, thanks. A suitably recent version of the pydb
 > > follows the Python 2.5 constructor convention - it doesn't matter what
 > > the Python version is. If pydb is used, (and we do check for a
 > > suitably recent version), then that hack is *not* needed. So
 > >
 > > class Pdb(OldPdb):
 > >     """Modified Pdb class, does not load readline."""
 > >
 > >     if sys.version[:3] >= '2.5'
 > >
 > > can become
 > >
 > >     if sys.version[:3] >= '2.5' or has_pydb:
 > >
 > > I've attached a patch for this below.
 > 
 > I think Ville already applied it, thanks.

This part yes. A couple of the others no, and I don't understand why
not. One of the patches was to remove a temporary debugging print in
IPython/Magic.py that accidentally crept in. I've included it in the
patch below.

 > 
 > > I'm still not clear about one other thing though, and this is not
 > > addressed in the patch but perhaps should be.
 > >
 > > When you create the debugger you turn off any readline completion,
 > > right? But is this correct whne pydb is used? pydb sets up command
 > > completion correctly for itself, and I'd venture to say that this
 > > would be more helpful in the debugger context than the default ipython
 > > command completion.
 > 
 > IPython simply prevents pdb (the builtin one) from mucking around with
 > our own completers (it would otherwise call rlcompleter and completely
 > disable ipython's completions).  

I'm having deja vu. This issue isn't strictly a debugger issue. Any
program that uses readline will have this, and ipython users can
invoke any such program run say via %run. So ipython needs to protect
itself in a general way.

A while back Ville noticed command history bleeding between the pydb
(or more generally the thing %run'd and ipython). So that was
addressed in a more general way than had been addressed via pdb. Well
at least from the bleeding into ipython. The bleeding in the other
direction, from ipython to the pydb (or any readline application) is
still there and Ville seems to prefer that.

As for completely disabling ipython's completions, I don't see this
happening at least with pydb and the current patch. I run an ipython
completion before issuing a %pydb or going into post-mortem. Then I
see pydb's completion inside the debugger. I leave that and I get the
same completions in ipython that I had before. However since I'm not
completely certain of the erroneous situation that was happening
before, I am not sure my test is the right one.

 > It's quite possible that this
 > disables pydb's completions, and in that case we should try to
 > accomodate both needs: preventing pdb from breaking ipython's
 > completion, while allowing pydb's completion to function.  

I just checked and ipython does disable pydb's completions. In the
process, I notice that ipython doesn't allow the completion key [tab]
to be specified. Right? (If it does, I would modify the patch
slightly)

 > If that's not the case, patches welcome :)

Okay - here is a patch and the current outstanding patches. If there
are comments, let me know.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: pydb-4.patch
Type: application/octet-stream
Size: 2507 bytes
Desc: Another round of pydb patches. Some new, some old
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20061130/4ccb7650/attachment.obj>
-------------- next part --------------

 > 
 > I'll be 90% offline starting now until about Dec. 12.
 > 
 > I did manage to get a patch in which I'd wanted for a long time; since
 > it's debugger related, your testing and feedback would be welcome:
 > 
 > %debug
 > 
 > now activates the interactive debugger (including pydb if available)
 > AFTER an exception has fired, even if you had %pdb off.  This lets you
 > inspect a crashed code interactively without having to do the old
 > dance '%pdb on; rerun code; %pdb off'.  I like it a lot more.

Well, one thing I like a lot more about it is that it doesn't change
the debugger prompt. :-) 

I've never understood the advantage of changing that. And here's why
it is unhelpful. If you are running py-shell inside Emacs or perhaps
say pick your favorite IDE that allows one to go into ipython as a
shell, then that program need to be sensitive to this change. If the
enclosing program needed to modify its behavior as a result, I could
see the distinction, but my experience has been it doesn't.  Yet, in
the case of py-shell, it means even more complicated regular
expressions.

Also note that recursive debug sessions are indicated by enclosing in
a set of parenthesis. So while the prompt may start off say "ipdb>" if
there is a recursive debug it would become "(ipdb>) ".


 > 
 > Cheers,
 > 
 > f
 > 

Cheers

From vivainio at gmail.com  Thu Nov 30 06:59:34 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Thu, 30 Nov 2006 13:59:34 +0200
Subject: [IPython-dev] svnversion used for Release.py
In-Reply-To: <17774.41637.724154.424378@panix3.panix.com>
References: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com>
	<db6b5ecc0611280627y6dfbdc87ta8ff332a93e07299@mail.gmail.com>
	<17772.56028.653337.85310@panix3.panix.com>
	<db6b5ecc0611282200t6db7441cna5a0d7cb18bfc02f@mail.gmail.com>
	<17773.19320.848097.530795@panix3.panix.com>
	<db6b5ecc0611292128v45746aabwe7de5d1d1b16710e@mail.gmail.com>
	<17774.41637.724154.424378@panix3.panix.com>
Message-ID: <46cb515a0611300359v598c4401w7608455832845f9c@mail.gmail.com>

On 11/30/06, R. Bernstein <rocky at panix.com> wrote:

> This part yes. A couple of the others no, and I don't understand why
> not. One of the patches was to remove a temporary debugging print in
> IPython/Magic.py that accidentally crept in. I've included it in the
> patch below.

Accidental loss of the patch. I'll apply this patch today.

> A while back Ville noticed command history bleeding between the pydb
> (or more generally the thing %run'd and ipython). So that was
> addressed in a more general way than had been addressed via pdb. Well
> at least from the bleeding into ipython. The bleeding in the other
> direction, from ipython to the pydb (or any readline application) is
> still there and Ville seems to prefer that.

Yes, it's the responsibility of called application to clear the
history if they want an empty history.

> I just checked and ipython does disable pydb's completions. In the
> process, I notice that ipython doesn't allow the completion key [tab]
> to be specified. Right? (If it does, I would modify the patch
> slightly)

It's probably a readline config issue, but I think we should not
encourage or even allow using non-tab charactersp

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'