From vivainio at gmail.com  Tue Dec  5 16:10:03 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Tue, 5 Dec 2006 23:10:03 +0200
Subject: [IPython-dev] svnversion used for Release.py
In-Reply-To: <46cb515a0611300359v598c4401w7608455832845f9c@mail.gmail.com>
References: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com>
	<db6b5ecc0611280627y6dfbdc87ta8ff332a93e07299@mail.gmail.com>
	<17772.56028.653337.85310@panix3.panix.com>
	<db6b5ecc0611282200t6db7441cna5a0d7cb18bfc02f@mail.gmail.com>
	<17773.19320.848097.530795@panix3.panix.com>
	<db6b5ecc0611292128v45746aabwe7de5d1d1b16710e@mail.gmail.com>
	<17774.41637.724154.424378@panix3.panix.com>
	<46cb515a0611300359v598c4401w7608455832845f9c@mail.gmail.com>
Message-ID: <46cb515a0612051310j721b9f1bs7f44cbdfcd3533c4@mail.gmail.com>

On 11/30/06, Ville M. Vainio <vivainio at gmail.com> wrote:
> On 11/30/06, R. Bernstein <rocky at panix.com> wrote:
>
> > This part yes. A couple of the others no, and I don't understand why
> > not. One of the patches was to remove a temporary debugging print in
> > IPython/Magic.py that accidentally crept in. I've included it in the
> > patch below.
>
> Accidental loss of the patch. I'll apply this patch today.

Ok, it took more time than I anticipated but it's in now.

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From rocky at panix.com  Tue Dec  5 21:31:42 2006
From: rocky at panix.com (R. Bernstein)
Date: Tue, 5 Dec 2006 21:31:42 -0500
Subject: [IPython-dev] svnversion used for Release.py
In-Reply-To: <46cb515a0612051310j721b9f1bs7f44cbdfcd3533c4@mail.gmail.com>
References: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com>
	<db6b5ecc0611280627y6dfbdc87ta8ff332a93e07299@mail.gmail.com>
	<17772.56028.653337.85310@panix3.panix.com>
	<db6b5ecc0611282200t6db7441cna5a0d7cb18bfc02f@mail.gmail.com>
	<17773.19320.848097.530795@panix3.panix.com>
	<db6b5ecc0611292128v45746aabwe7de5d1d1b16710e@mail.gmail.com>
	<17774.41637.724154.424378@panix3.panix.com>
	<46cb515a0611300359v598c4401w7608455832845f9c@mail.gmail.com>
	<46cb515a0612051310j721b9f1bs7f44cbdfcd3533c4@mail.gmail.com>
Message-ID: <17782.11150.868362.534238@panix3.panix.com>

Getting this in is much appreciated. 

I've done some minimal testing. Things work.  There is/was a small bug
in pydb in calling post_mortem (via %debug) when there is no traceback
around. That's been fixed in pydb's version control and a release will
probably go out Dec 10th.

Thanks again.

Ville M. Vainio writes:
 > On 11/30/06, Ville M. Vainio <vivainio at gmail.com> wrote:
 > > On 11/30/06, R. Bernstein <rocky at panix.com> wrote:
 > >
 > > > This part yes. A couple of the others no, and I don't understand why
 > > > not. One of the patches was to remove a temporary debugging print in
 > > > IPython/Magic.py that accidentally crept in. I've included it in the
 > > > patch below.
 > >
 > > Accidental loss of the patch. I'll apply this patch today.
 > 
 > Ok, it took more time than I anticipated but it's in now.
 > 
 > -- 
 > Ville M. Vainio - vivainio.googlepages.com
 > blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'
 > 


From rocky at panix.com  Wed Dec  6 03:40:56 2006
From: rocky at panix.com (R. Bernstein)
Date: Wed, 6 Dec 2006 03:40:56 -0500
Subject: [IPython-dev] svnversion used for Release.py
In-Reply-To: <17782.11150.868362.534238@panix3.panix.com>
References: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com>
	<db6b5ecc0611280627y6dfbdc87ta8ff332a93e07299@mail.gmail.com>
	<17772.56028.653337.85310@panix3.panix.com>
	<db6b5ecc0611282200t6db7441cna5a0d7cb18bfc02f@mail.gmail.com>
	<17773.19320.848097.530795@panix3.panix.com>
	<db6b5ecc0611292128v45746aabwe7de5d1d1b16710e@mail.gmail.com>
	<17774.41637.724154.424378@panix3.panix.com>
	<46cb515a0611300359v598c4401w7608455832845f9c@mail.gmail.com>
	<46cb515a0612051310j721b9f1bs7f44cbdfcd3533c4@mail.gmail.com>
	<17782.11150.868362.534238@panix3.panix.com>
Message-ID: <17782.33304.256526.453400@panix3.panix.com>

FYI
I mentioned previously:
 ...
 > There is/was a small bug
 > in pydb in calling post_mortem (via %debug) when there is no traceback
 > around. That's been fixed in pydb's version control and a release will
 > probably go out Dec 10th.

Well, actually pdb has the same bug. If you call post_mortem before
there's been an exception, you'll get one - from
post_mortem. Hmmm... perhaps just post_mortem trying to be helpful? 

It may be that ipython wants to test for this before calling
post_mortem. But with respect to pydb, the right thing is to (also)
make a check in post_mortem -- one shouldn't assume the only time
post_mortem will ever get called is from ipython.

Should other pdb users feel this a pdb bug too, some good citizen
might submit a Python bug report. (I probably won't.)



From vivainio at gmail.com  Wed Dec  6 09:22:19 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Wed, 6 Dec 2006 16:22:19 +0200
Subject: [IPython-dev] Servers down? Plus, ignoring SIGINT on 'sh' profile
Message-ID: <46cb515a0612060622q3ff644f0l643f75aafc19fd6a@mail.gmail.com>

Just a heads up - ipython servers seem to be down at the moment.

Another thing:

I thought about monkeypatching IPython.iplib.ipsystem() in the 'sh'
profile to ignore ctrl + C by doing

old_handler = signal.signal(signal.SIGINT, signal.SIG_IGN)

and restoring the old signal handler after running the system command.
It seems to work on win32, but I need to test in Linux as well (at
home).

Does anyone see any problems with this? In this release it wouldn't be
used in any multithreaded profile.

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From fperez.net at gmail.com  Wed Dec  6 21:43:21 2006
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 6 Dec 2006 19:43:21 -0700
Subject: [IPython-dev] Servers down? Plus,
	ignoring SIGINT on 'sh' profile
In-Reply-To: <46cb515a0612060622q3ff644f0l643f75aafc19fd6a@mail.gmail.com>
References: <46cb515a0612060622q3ff644f0l643f75aafc19fd6a@mail.gmail.com>
Message-ID: <db6b5ecc0612061843y5f830f24k95d4448404118567@mail.gmail.com>

On 12/6/06, Ville M. Vainio <vivainio at gmail.com> wrote:
> Just a heads up - ipython servers seem to be down at the moment.
>
> Another thing:
>
> I thought about monkeypatching IPython.iplib.ipsystem() in the 'sh'
> profile to ignore ctrl + C by doing
>
> old_handler = signal.signal(signal.SIGINT, signal.SIG_IGN)
>
> and restoring the old signal handler after running the system command.
> It seems to work on win32, but I need to test in Linux as well (at
> home).
>
> Does anyone see any problems with this? In this release it wouldn't be
> used in any multithreaded profile.

As long as it's only used when running system commands and not normal
python code (which should ALWAYS be interruptible in the normal
non-threaded shells) I don't see a problem.

Note that I will have zero net access til next Monday or Tuesday
starting basically now.

Cheers,

f


From vivainio at gmail.com  Thu Dec  7 15:32:29 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Thu, 7 Dec 2006 22:32:29 +0200
Subject: [IPython-dev] Considerations for 0.7.3 release, branching, rc's
Message-ID: <46cb515a0612071232o1bfc18a9mf5b1f225275849ef@mail.gmail.com>

I didn't branch the beta 2 yet (it's just made from trunk) but once
the beta 2 has been out for a while (next week?) without bug reports,
I think we should create the 0.7.3 branch only only put in bugfixes,
i.e. home in for release candidate.

So if there is something on your plate at this very moment that would
necessitate beta 3 (as opposed to just going for the release
candidates), speak up please. :)

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From ellisonbg.net at gmail.com  Fri Dec  8 14:43:28 2006
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Fri, 8 Dec 2006 12:43:28 -0700
Subject: [IPython-dev] IPython Moin problems
Message-ID: <6ce0ac130612081143i1cfe3644k70d24af0e65e9904@mail.gmail.com>

Just wanted to let folks know that the IPython Moin is having some
problems.  We have updated the Moin version and are having trouble
getting the new version to recognize the right theme.  It should be
functional even though the theme has changed.  Hopefully we can get
this fixed soon.

Brian


From jesus at aguillon.com  Fri Dec  8 17:27:04 2006
From: jesus at aguillon.com (Jesus Aguillon)
Date: Fri, 8 Dec 2006 14:27:04 -0800
Subject: [IPython-dev] edited file is readonly how to change?
Message-ID: <a52480020612081427r452a4757ka138b397e9c180a@mail.gmail.com>

When I edit a file from within IPython and it has a syntax error, the next
time time I try to edit the file it is opened as read-only.  I am using
Windows XP SP2,  IPython 0.7.3b3.r1975 and Gvim 7.0.

I can work around this by writing the file using w! from within vim.  Is
there a way to clear the read-only status on the file without leaving
IPython?  (It is cleared when I exit IPython)

I have tried changing the attribute on the file from Windows, but that
doesn't seem to do it.

Thanks

-- 
jesusATaguillonDOTcom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20061208/3896f996/attachment.html>

From fperez.net at gmail.com  Mon Dec 11 11:53:07 2006
From: fperez.net at gmail.com (Fernando Perez)
Date: Mon, 11 Dec 2006 11:53:07 -0500
Subject: [IPython-dev] svnversion used for Release.py
In-Reply-To: <17782.33304.256526.453400@panix3.panix.com>
References: <46cb515a0611261334v33c2a06ch554c4e16ed667861@mail.gmail.com>
	<17772.56028.653337.85310@panix3.panix.com>
	<db6b5ecc0611282200t6db7441cna5a0d7cb18bfc02f@mail.gmail.com>
	<17773.19320.848097.530795@panix3.panix.com>
	<db6b5ecc0611292128v45746aabwe7de5d1d1b16710e@mail.gmail.com>
	<17774.41637.724154.424378@panix3.panix.com>
	<46cb515a0611300359v598c4401w7608455832845f9c@mail.gmail.com>
	<46cb515a0612051310j721b9f1bs7f44cbdfcd3533c4@mail.gmail.com>
	<17782.11150.868362.534238@panix3.panix.com>
	<17782.33304.256526.453400@panix3.panix.com>
Message-ID: <db6b5ecc0612110853w62b2ba22oc944bf38eb7d75a3@mail.gmail.com>

On 12/6/06, R. Bernstein <rocky at panix.com> wrote:
> FYI
> I mentioned previously:
>  ...
>  > There is/was a small bug
>  > in pydb in calling post_mortem (via %debug) when there is no traceback
>  > around. That's been fixed in pydb's version control and a release will
>  > probably go out Dec 10th.
>
> Well, actually pdb has the same bug. If you call post_mortem before
> there's been an exception, you'll get one - from
> post_mortem. Hmmm... perhaps just post_mortem trying to be helpful?
>
> It may be that ipython wants to test for this before calling
> post_mortem. But with respect to pydb, the right thing is to (also)
> make a check in post_mortem -- one shouldn't assume the only time
> post_mortem will ever get called is from ipython.
>
> Should other pdb users feel this a pdb bug too, some good citizen
> might submit a Python bug report. (I probably won't.)

I just fixed it for IPython, the fix should cover pydb usage as well
(please let me know if it doesn't, I don't have it for testing).  This
is the behavior now (r1977):

In [1]: debug
ERROR: No traceback has been produced, nothing to debug.

Cheers,

f


From ellisonbg.net at gmail.com  Mon Dec 11 18:24:51 2006
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Mon, 11 Dec 2006 16:24:51 -0700
Subject: [IPython-dev] module for blocking on twisted Deferred objects
Message-ID: <6ce0ac130612111524h3fb45e1bq8e91b3e445b2c529@mail.gmail.com>

Hi,

I have written a small python module that provides a synchronous
interface around the asynchronous calls of Twisted.  This has been
something that Fernando and I have wanted for a long time, but until
now I didn't have a solution that was simple and robust.  I could put
this in ipython1, but it could be useful for many folks so I wonder if
putting it into ipython/trunk would make more sense.  Thoughts?

Brian


From barrywark at gmail.com  Tue Dec 12 01:07:08 2006
From: barrywark at gmail.com (Barry Wark)
Date: Mon, 11 Dec 2006 22:07:08 -0800
Subject: [IPython-dev] module for blocking on twisted Deferred objects
In-Reply-To: <6ce0ac130612111524h3fb45e1bq8e91b3e445b2c529@mail.gmail.com>
References: <6ce0ac130612111524h3fb45e1bq8e91b3e445b2c529@mail.gmail.com>
Message-ID: <cd7634ce0612112207w26c9006dh1607169250e8bd4d@mail.gmail.com>

Brian,

I at least will be excited to see it, wherever it appears. What about
putting it in ipython1 (a logical place, it seems), plus having a
stand-alone module/package?

barry

On 12/11/06, Brian Granger <ellisonbg.net at gmail.com> wrote:
> Hi,
>
> I have written a small python module that provides a synchronous
> interface around the asynchronous calls of Twisted.  This has been
> something that Fernando and I have wanted for a long time, but until
> now I didn't have a solution that was simple and robust.  I could put
> this in ipython1, but it could be useful for many folks so I wonder if
> putting it into ipython/trunk would make more sense.  Thoughts?
>
> Brian
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/ipython-dev
>


From ellisonbg.net at gmail.com  Tue Dec 12 01:14:28 2006
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Mon, 11 Dec 2006 23:14:28 -0700
Subject: [IPython-dev] module for blocking on twisted Deferred objects
In-Reply-To: <cd7634ce0612112207w26c9006dh1607169250e8bd4d@mail.gmail.com>
References: <6ce0ac130612111524h3fb45e1bq8e91b3e445b2c529@mail.gmail.com>
	<cd7634ce0612112207w26c9006dh1607169250e8bd4d@mail.gmail.com>
Message-ID: <6ce0ac130612112214t52e9b05endd309a2425b58f5a@mail.gmail.com>

For now I checked it into my sandbox:

http://projects.scipy.org/ipython/ipython/browser/ipython/sandbox/bgranger/blockon.py

I would love feedback, but will probably work on it more over the next few days.

Brian

On 12/11/06, Barry Wark <barrywark at gmail.com> wrote:
> Brian,
>
> I at least will be excited to see it, wherever it appears. What about
> putting it in ipython1 (a logical place, it seems), plus having a
> stand-alone module/package?
>
> barry
>
> On 12/11/06, Brian Granger <ellisonbg.net at gmail.com> wrote:
> > Hi,
> >
> > I have written a small python module that provides a synchronous
> > interface around the asynchronous calls of Twisted.  This has been
> > something that Fernando and I have wanted for a long time, but until
> > now I didn't have a solution that was simple and robust.  I could put
> > this in ipython1, but it could be useful for many folks so I wonder if
> > putting it into ipython/trunk would make more sense.  Thoughts?
> >
> > Brian
> > _______________________________________________
> > IPython-dev mailing list
> > IPython-dev at scipy.org
> > http://projects.scipy.org/mailman/listinfo/ipython-dev
> >
>


From vivainio at gmail.com  Tue Dec 12 14:30:27 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Tue, 12 Dec 2006 21:30:27 +0200
Subject: [IPython-dev] module for blocking on twisted Deferred objects
In-Reply-To: <cd7634ce0612112207w26c9006dh1607169250e8bd4d@mail.gmail.com>
References: <6ce0ac130612111524h3fb45e1bq8e91b3e445b2c529@mail.gmail.com>
	<cd7634ce0612112207w26c9006dh1607169250e8bd4d@mail.gmail.com>
Message-ID: <46cb515a0612121130q34eb9f45ueb7ecb1713bc8187@mail.gmail.com>

On 12/12/06, Barry Wark <barrywark at gmail.com> wrote:

> I at least will be excited to see it, wherever it appears. What about
> putting it in ipython1 (a logical place, it seems), plus having a
> stand-alone module/package?

Seems like standalone module material for me, i.e. not directly
connected to ipython (or even ipython1). Cheeseshop is an apt place
for stuff like this.

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From vivainio at gmail.com  Tue Dec 12 14:33:07 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Tue, 12 Dec 2006 21:33:07 +0200
Subject: [IPython-dev] edited file is readonly how to change?
In-Reply-To: <a52480020612081427r452a4757ka138b397e9c180a@mail.gmail.com>
References: <a52480020612081427r452a4757ka138b397e9c180a@mail.gmail.com>
Message-ID: <46cb515a0612121133kfa260feub356384f5fab733f@mail.gmail.com>

On 12/9/06, Jesus Aguillon <jesus at aguillon.com> wrote:

> When I edit a file from within IPython and it has a syntax error, the next
> time time I try to edit the file it is opened as read-only.  I am using
> Windows XP SP2,  IPython 0.7.3b3.r1975 and Gvim 7.0.
>
> I can work around this by writing the file using w! from within vim.  Is
> there a way to clear the read-only status on the file without leaving
> IPython?  (It is cleared when I exit IPython)

How do you "edit" the file? Executing %edit, or running vim directly?

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From jesus at aguillon.com  Tue Dec 12 14:48:26 2006
From: jesus at aguillon.com (Jesus Aguillon)
Date: Tue, 12 Dec 2006 11:48:26 -0800
Subject: [IPython-dev] edited file is readonly how to change?
In-Reply-To: <46cb515a0612121133kfa260feub356384f5fab733f@mail.gmail.com>
References: <a52480020612081427r452a4757ka138b397e9c180a@mail.gmail.com>
	<46cb515a0612121133kfa260feub356384f5fab733f@mail.gmail.com>
Message-ID: <a52480020612121148l24088118xc84d7aeb6c890231@mail.gmail.com>

I run the %edit command.

thanks!


On 12/12/06, Ville M. Vainio <vivainio at gmail.com> wrote:
>
> On 12/9/06, Jesus Aguillon <jesus at aguillon.com> wrote:
>
> > When I edit a file from within IPython and it has a syntax error, the
> next
> > time time I try to edit the file it is opened as read-only.  I am using
> > Windows XP SP2,  IPython 0.7.3b3.r1975 and Gvim 7.0.
> >
> > I can work around this by writing the file using w! from within vim.  Is
> > there a way to clear the read-only status on the file without leaving
> > IPython?  (It is cleared when I exit IPython)
>
> How do you "edit" the file? Executing %edit, or running vim directly?
>
> --
> Ville M. Vainio - vivainio.googlepages.com
> blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20061212/d80d7fc4/attachment.html>

From fperez.net at gmail.com  Tue Dec 12 16:13:11 2006
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 12 Dec 2006 16:13:11 -0500
Subject: [IPython-dev] module for blocking on twisted Deferred objects
In-Reply-To: <46cb515a0612121130q34eb9f45ueb7ecb1713bc8187@mail.gmail.com>
References: <6ce0ac130612111524h3fb45e1bq8e91b3e445b2c529@mail.gmail.com>
	<cd7634ce0612112207w26c9006dh1607169250e8bd4d@mail.gmail.com>
	<46cb515a0612121130q34eb9f45ueb7ecb1713bc8187@mail.gmail.com>
Message-ID: <db6b5ecc0612121313s76d8039ao5b205ebd2dcc15dc@mail.gmail.com>

On 12/12/06, Ville M. Vainio <vivainio at gmail.com> wrote:
> On 12/12/06, Barry Wark <barrywark at gmail.com> wrote:
>
> > I at least will be excited to see it, wherever it appears. What about
> > putting it in ipython1 (a logical place, it seems), plus having a
> > stand-alone module/package?
>
> Seems like standalone module material for me, i.e. not directly
> connected to ipython (or even ipython1). Cheeseshop is an apt place
> for stuff like this.

Because it's a nice and useful standalone recipe, the Cheeseshop or
the ASPN cookbook are great, high-visibility places to put it in.  But
some form of this code will most definitely be part of ipython1,
albeit included in the larger system.  We simply /need/ this for a lot
of things.

Cheers,

f


From ellisonbg.net at gmail.com  Tue Dec 12 16:50:07 2006
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 12 Dec 2006 14:50:07 -0700
Subject: [IPython-dev] module for blocking on twisted Deferred objects
In-Reply-To: <cd7634ce0612121134r78050c37x6b20ee2a92014b15@mail.gmail.com>
References: <6ce0ac130612111524h3fb45e1bq8e91b3e445b2c529@mail.gmail.com>
	<cd7634ce0612112207w26c9006dh1607169250e8bd4d@mail.gmail.com>
	<6ce0ac130612112214t52e9b05endd309a2425b58f5a@mail.gmail.com>
	<cd7634ce0612121134r78050c37x6b20ee2a92014b15@mail.gmail.com>
Message-ID: <6ce0ac130612121350u3e7745a0v41d0e29f9c0284a9@mail.gmail.com>

Barry,

My understanding is that iterate won't block until there is an event
present.  Underneath it simply calls select/poll with a timeout.
Currently, the weakness of this module is that the event loop will
only run while you are waiting (and blocking on) a particular
deferred.  I could put a timeout that allows the user to wait for a
certain time.  The problem is that once that time has passed, the
event loop will stop and twisted won't be able to process the events
when they do come in.  In IPython1, we handle this by having a
block=True/False option on the execute command.

Another way of saying this is the following.  With the blockon.py
module, there are two options:

1.  You are waiting on the result of a deferred using blockon.py and
twisted is iterating through events.

or

2.  Twisted is not processing events and you have interactive control.

To have both interactive control and have the twisted event loop
running you would have to run the twisted reactor in a second thread
and implement some clever way talking between the two threads.  In
this approach you would have to have all twisted code in that second
thread.  My goal with blockon.py was to let the user directly make
twisted calls in the main thread and provide a simple way of iterating
over events.  If you see a nice way of running the reactor in a
different thread, (twisted has a callFromThread method) we could think
about that approach more.



On 12/12/06, Barry Wark <barrywark at gmail.com> wrote:
> Brian,
>
> Does twisted.internet.reactor.iterate block until an event is present
> in the reactors loop? In other words, is the while loop in your
> BlockingDeferred.blockOn polling every TIMEOUT seconds or is it
> blocking until an event is received?
>
> I suspect you're using this to get synchronous-like behavior with
> distributed computations in ipython1. If the computations take hours
> or days, it seems maybe inefficient to poll that much... you might
> want to have an other parameter to the blockOn call that gives the
> timeout. Then the user can decide the polling resolution.?
>
> barry
>
> On 12/11/06, Brian Granger <ellisonbg.net at gmail.com> wrote:
> > For now I checked it into my sandbox:
> >
> > http://projects.scipy.org/ipython/ipython/browser/ipython/sandbox/bgranger/blockon.py
> >
> > I would love feedback, but will probably work on it more over the next few days.
> >
> > Brian
> >
> > On 12/11/06, Barry Wark <barrywark at gmail.com> wrote:
> > > Brian,
> > >
> > > I at least will be excited to see it, wherever it appears. What about
> > > putting it in ipython1 (a logical place, it seems), plus having a
> > > stand-alone module/package?
> > >
> > > barry
> > >
> > > On 12/11/06, Brian Granger <ellisonbg.net at gmail.com> wrote:
> > > > Hi,
> > > >
> > > > I have written a small python module that provides a synchronous
> > > > interface around the asynchronous calls of Twisted.  This has been
> > > > something that Fernando and I have wanted for a long time, but until
> > > > now I didn't have a solution that was simple and robust.  I could put
> > > > this in ipython1, but it could be useful for many folks so I wonder if
> > > > putting it into ipython/trunk would make more sense.  Thoughts?
> > > >
> > > > Brian
> > > > _______________________________________________
> > > > IPython-dev mailing list
> > > > IPython-dev at scipy.org
> > > > http://projects.scipy.org/mailman/listinfo/ipython-dev
> > > >
> > >
> >
>


From ellisonbg.net at gmail.com  Tue Dec 12 16:57:15 2006
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 12 Dec 2006 14:57:15 -0700
Subject: [IPython-dev] module for blocking on twisted Deferred objects
In-Reply-To: <db6b5ecc0612121313s76d8039ao5b205ebd2dcc15dc@mail.gmail.com>
References: <6ce0ac130612111524h3fb45e1bq8e91b3e445b2c529@mail.gmail.com>
	<cd7634ce0612112207w26c9006dh1607169250e8bd4d@mail.gmail.com>
	<46cb515a0612121130q34eb9f45ueb7ecb1713bc8187@mail.gmail.com>
	<db6b5ecc0612121313s76d8039ao5b205ebd2dcc15dc@mail.gmail.com>
Message-ID: <6ce0ac130612121357n5fc4e052w50744f47bc1305fe@mail.gmail.com>

I am a little hesitant to publicize this module very much.  It is in a
very grey area that many people (maybe myself even) would consider a
hack.  That doesn't change the fact that it is very useful and we will
likely us it in ipython extensively.  The danger is that people would
consider it a general tool that can be used to "make Deferred's
easier."  I more view it as a last resort when you really need Twisted
code to act synchronous.

I will have to think about it more.  It also sort of depends on how it
develops.

Brian

On 12/12/06, Fernando Perez <fperez.net at gmail.com> wrote:
> On 12/12/06, Ville M. Vainio <vivainio at gmail.com> wrote:
> > On 12/12/06, Barry Wark <barrywark at gmail.com> wrote:
> >
> > > I at least will be excited to see it, wherever it appears. What about
> > > putting it in ipython1 (a logical place, it seems), plus having a
> > > stand-alone module/package?
> >
> > Seems like standalone module material for me, i.e. not directly
> > connected to ipython (or even ipython1). Cheeseshop is an apt place
> > for stuff like this.
>
> Because it's a nice and useful standalone recipe, the Cheeseshop or
> the ASPN cookbook are great, high-visibility places to put it in.  But
> some form of this code will most definitely be part of ipython1,
> albeit included in the larger system.  We simply /need/ this for a lot
> of things.
>
> Cheers,
>
> f
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/ipython-dev
>


From vivainio at gmail.com  Wed Dec 13 01:52:16 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Wed, 13 Dec 2006 08:52:16 +0200
Subject: [IPython-dev] [IPython-user] New feature: %run myscript.ipy
In-Reply-To: <6ce0ac130612121516n7a6762b2pe2f42f9b6d68971d@mail.gmail.com>
References: <46cb515a0612121405x4f76db30md4934bfadcbd783e@mail.gmail.com>
	<6ce0ac130612121516n7a6762b2pe2f42f9b6d68971d@mail.gmail.com>
Message-ID: <46cb515a0612122252q28b576a5ma2572fe5ce98eea8@mail.gmail.com>

On 12/13/06, Brian Granger <ellisonbg.net at gmail.com> wrote:

> How is this done?  As I remember, the normal %run basically does an
> exec on the file as a whole.  By that I mean that the file is parsed
> as a whole before being run.  But IPython commands are not able to be
> run in this fashion.  This has come up in the version of run that is
> in ipython1 that works remotely.  In that case we just pass the file
> line by line to the interactive interpreter rather than doing an exec
> on the whole file.  The difficulty is that there are a bunch of funny
> things that you have to watch out for - like you need more than one
> space after an indented code block.  Does the new %run have this
> restriction for ipy files?  I almost think it should be called
> something different because the behavior is different.  Fernando?

Yes, the .ipy file running is a whole different beast, see

http://projects.scipy.org/ipython/ipython/changeset/1981

I think %run is a 'logical' place to do this, in that the user does
think that "he's running an .ipy script", i.e. %run is easy to
remember. The docstring is explicit about the difference (see the trac
link).

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From vivainio at gmail.com  Wed Dec 13 11:39:17 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Wed, 13 Dec 2006 18:39:17 +0200
Subject: [IPython-dev] Branched for 0.7.3 - time for rc1?
Message-ID: <46cb515a0612130839l24bacc5as51c1c992d89749d1@mail.gmail.com>

I just branched the trunk for 0.7.3. If you are tracking svn, do

svn switch http://ipython.scipy.org/svn/ipython/ipython/branches/0.7.3

In the ipython root directory now.

I think this could be the time to put out a release candidate.
Fernando, can you build and upload the candidate (current state of
0.7.3 branch) to the "testing" directory of the official download
site? I already changed Release.py.

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From fperez.net at gmail.com  Wed Dec 13 11:44:14 2006
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 13 Dec 2006 11:44:14 -0500
Subject: [IPython-dev] Branched for 0.7.3 - time for rc1?
In-Reply-To: <46cb515a0612130839l24bacc5as51c1c992d89749d1@mail.gmail.com>
References: <46cb515a0612130839l24bacc5as51c1c992d89749d1@mail.gmail.com>
Message-ID: <db6b5ecc0612130844o678fd15fxa35dbbe68402bd3a@mail.gmail.com>

On 12/13/06, Ville M. Vainio <vivainio at gmail.com> wrote:
> I just branched the trunk for 0.7.3. If you are tracking svn, do

If you can apply that WX patch I sent, that  would be great (to both
trunk and branch).  I haven't heard back, so I'll take silence as
approval.  Edgy runs WX 2.6, so we should be OK.

If you can't, I'll do it and the RC upload in a couple of hours, I
can't right this minute.

Cheers,

f


From vivainio at gmail.com  Wed Dec 13 12:00:08 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Wed, 13 Dec 2006 19:00:08 +0200
Subject: [IPython-dev] Branched for 0.7.3 - time for rc1?
In-Reply-To: <db6b5ecc0612130844o678fd15fxa35dbbe68402bd3a@mail.gmail.com>
References: <46cb515a0612130839l24bacc5as51c1c992d89749d1@mail.gmail.com>
	<db6b5ecc0612130844o678fd15fxa35dbbe68402bd3a@mail.gmail.com>
Message-ID: <46cb515a0612130900o4087c452tf033d09eb99695a@mail.gmail.com>

On 12/13/06, Fernando Perez <fperez.net at gmail.com> wrote:

> On 12/13/06, Ville M. Vainio <vivainio at gmail.com> wrote:
> > I just branched the trunk for 0.7.3. If you are tracking svn, do
>
> If you can apply that WX patch I sent, that  would be great (to both
> trunk and branch).  I haven't heard back, so I'll take silence as
> approval.  Edgy runs WX 2.6, so we should be OK.

Ok, it's in.

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From fperez.net at gmail.com  Wed Dec 13 16:10:21 2006
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 13 Dec 2006 16:10:21 -0500
Subject: [IPython-dev] Branched for 0.7.3 - time for rc1?
In-Reply-To: <db6b5ecc0612130844o678fd15fxa35dbbe68402bd3a@mail.gmail.com>
References: <46cb515a0612130839l24bacc5as51c1c992d89749d1@mail.gmail.com>
	<db6b5ecc0612130844o678fd15fxa35dbbe68402bd3a@mail.gmail.com>
Message-ID: <db6b5ecc0612131310gcf75ba2g167ff227fd148ecc@mail.gmail.com>

On 12/13/06, Fernando Perez <fperez.net at gmail.com> wrote:
> On 12/13/06, Ville M. Vainio <vivainio at gmail.com> wrote:
> > I just branched the trunk for 0.7.3. If you are tracking svn, do
>
> If you can apply that WX patch I sent, that  would be great (to both
> trunk and branch).  I haven't heard back, so I'll take silence as
> approval.  Edgy runs WX 2.6, so we should be OK.
>
> If you can't, I'll do it and the RC upload in a couple of hours, I
> can't right this minute.

It's currently uploading to

http://ipython.scipy.org/dist/testing

all the files should be there in a minute.

Minor note: I just saw that the revision number it creates
(0.7.3rc1.r1986) is a bit different from that which svn reports from
the top level of the 0.7.3 branch dir:

funes[ipython]> svn info | grep Revision
Revision: 1993

Do you see that as well (it could be a fluke of my  local setup)?

Cheers,

f


From dfj225 at gmail.com  Wed Dec 13 16:44:53 2006
From: dfj225 at gmail.com (Douglas Jones)
Date: Wed, 13 Dec 2006 16:44:53 -0500
Subject: [IPython-dev] IPython1 parallel questions
Message-ID: <1315be7e0612131344w40a21830td32d982de9b0fff4@mail.gmail.com>

Hello IPython group!

I recently discovered IPython1 and its parallel capabilities. Once I
read about the feature set, I became extremely excited as IPython1
seems to solve many of the interactive parallel computing problems
that I had hoped to solve for a project that I am a developer on.

Let me start by framing my project. Our code currently exists as
library written in C++. Our library does parallel computation using
MPI. We currently have code that exposes this to python in an
interactive manner, and have been looking for ways to expand this to a
fuller more robust implementation. I'm hoping that IPython1 will be
the perfect piece to create our collaborative, interactive
environment.

That said, as I investigate IPython further and start to develop some
prototype code, I have some questions for the community and
developers.

1) Are there any outstanding issues or problems with the IPython1 code
base that inhibits parallel computation? I've been experimenting with
the code and its features for the past few days in addition to looking
through the bug list, and from what I can tell there are none. Are
there any features mentioned in IPython's documentation or the
presentations on the site that haven't been implemented yet?

2) Is there any authentication or security between the client shell
and the IPython controller?

3) What are the proper ways to configure the operation of the client
and the controller or the engines? From what I've seen it looks like
there is an API that allows the user to set certain options. Can
configuration files be used?

Some configuration issues I'm thinking of right now involve output.
For instance, is there a way to turn off the feedback from each node
for every line of code? I would imagine that if you had a cluster with
many nodes running, you would not want to see this feedback. Another
issue is how output on stdout from C++ code is handled. I noticed that
right now, any output on stdout for a simple C++ module that I have
exposed to python and loaded in each engine simply is simply written
by the engine to the console. Is there some way to redirect this
output?

4) When objects are sent from the kernel to the client, is the only
prerequisite to this that the object being sent is able to be pickled?
I suppose the client would also have to have the class code for any
objects imported?

5) Are there any known projects that use IPython1 for interactive
scientific computing? I'd be really interested to see ones that also
support visualization of distributed data.

Well, I think that covers all my initial questions. Thank you for
bearing this long post and my newbie questions. I'm really looking
forward to working more with IPython as it seems like an amazing piece
of software.

Thank you,
~doug


From vivainio at gmail.com  Wed Dec 13 17:24:41 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Thu, 14 Dec 2006 00:24:41 +0200
Subject: [IPython-dev] Branched for 0.7.3 - time for rc1?
In-Reply-To: <db6b5ecc0612131310gcf75ba2g167ff227fd148ecc@mail.gmail.com>
References: <46cb515a0612130839l24bacc5as51c1c992d89749d1@mail.gmail.com>
	<db6b5ecc0612130844o678fd15fxa35dbbe68402bd3a@mail.gmail.com>
	<db6b5ecc0612131310gcf75ba2g167ff227fd148ecc@mail.gmail.com>
Message-ID: <46cb515a0612131424m4c057a25p3c34c023bd804683@mail.gmail.com>

On 12/13/06, Fernando Perez <fperez.net at gmail.com> wrote:

> Minor note: I just saw that the revision number it creates
> (0.7.3rc1.r1986) is a bit different from that which svn reports from
> the top level of the 0.7.3 branch dir:
>
> funes[ipython]> svn info | grep Revision
> Revision: 1993
>
> Do you see that as well (it could be a fluke of my  local setup)?

Ah yes, I forgot to run update_revnum.py. I just ran it on the branch,
so if it's not a big problem you could re-build and upload...

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From vivainio at gmail.com  Thu Dec 14 02:45:13 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Thu, 14 Dec 2006 09:45:13 +0200
Subject: [IPython-dev] Creating shortcuts on windows should overwrite old
	shortcuts
Message-ID: <46cb515a0612132345x55cfe448xca740b79caff4b3a@mail.gmail.com>

As it stands (rc1), the shortcuts are not created by the postinstall
script if a shortcut already exists at target. This is wrong IMO - if
there is an older version of IPython installed, it remains pointing at
the old ipython. This only becomes a problem if there are 2 python
versions installed; my old shortcut used python 2.4, and hence
launched the version of ipython that was installed for python 2.4
(even if I selected python 2.5 on the installer).

Are there some projected problems in ditching create_shorcut_safe?

I'd also like to change the "pysh" shortcut to use the "sh" profile.

I think we should put out a rc2 with these modifications.

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From vivainio at gmail.com  Thu Dec 14 11:08:48 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Thu, 14 Dec 2006 18:08:48 +0200
Subject: [IPython-dev] rc2 is ready
Message-ID: <46cb515a0612140808p1fc148a2x3fa21553b23b71ff@mail.gmail.com>

I changed the windows post install script to always create shortcuts,
even if it needs to overwrite the old ones. I also bumped up the
release number to 0.7.3rc2.r1998.

Fernando, could you upload this to 'testing' directory? With some
installation feedback, I think we are ready to go for 0.7.3 final real
soon now.

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From ellisonbg.net at gmail.com  Thu Dec 14 12:51:19 2006
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Thu, 14 Dec 2006 10:51:19 -0700
Subject: [IPython-dev] IPython1 parallel questions
In-Reply-To: <1315be7e0612131344w40a21830td32d982de9b0fff4@mail.gmail.com>
References: <1315be7e0612131344w40a21830td32d982de9b0fff4@mail.gmail.com>
Message-ID: <6ce0ac130612140951v58f1299fq825dfd7077a9d762@mail.gmail.com>

Douglas,

See my comments inline below.

> I recently discovered IPython1 and its parallel capabilities. Once I
> read about the feature set, I became extremely excited as IPython1
> seems to solve many of the interactive parallel computing problems
> that I had hoped to solve for a project that I am a developer on.
>
> Let me start by framing my project. Our code currently exists as
> library written in C++. Our library does parallel computation using
> MPI. We currently have code that exposes this to python in an
> interactive manner, and have been looking for ways to expand this to a
> fuller more robust implementation. I'm hoping that IPython1 will be
> the perfect piece to create our collaborative, interactive
> environment.

Nice, this is one of the main usage cases we had in mind when
designing ipython1.  It should work well for this.

> That said, as I investigate IPython further and start to develop some
> prototype code, I have some questions for the community and
> developers.
>
> 1) Are there any outstanding issues or problems with the IPython1 code
> base that inhibits parallel computation? I've been experimenting with
> the code and its features for the past few days in addition to looking
> through the bug list, and from what I can tell there are none. Are
> there any features mentioned in IPython's documentation or the
> presentations on the site that haven't been implemented yet?

Not really.  IPython1 is already being used for real science and
people don't seem to be limited in any significant way.  With that
said, there are a number of things we are still working on.  Here is a
taste:

1.  Optimizing how large objects are send between the client and
engines.  In our current approach, the controller become a bottleneck
when you try to use push/pull to send really big things (> 100's of
MBs).  With that said, if you are wanting to send large objects
around, you might want to rethink how you are parallelizing you
algorithm.

2.  Task farming.  While the architecture is setup for task farming,
we have not implemented a few parts of it.  We are actively (as in
this week) working on this.

3.  Security.

4.  Scalability.  Currently all the engines connect to a single
controller.  We have tested this on 128 processors and it works fine.
The problem is that most systems have a per process file-descriptor
limit that is not much higher than that (like 256).  We would like to
explore ways (multiple controllers?) of getting it to scale beyond
128-256.

> 2) Is there any authentication or security between the client shell
> and the IPython controller?

Not yet, but we are working on it.

> 3) What are the proper ways to configure the operation of the client
> and the controller or the engines? From what I've seen it looks like
> there is an API that allows the user to set certain options. Can
> configuration files be used?

Yes, we have a very powerful configuration system.  For examples see
the ipython1/configfiles directory.  You can simply copy these over to
your ~/.ipython directory and start playing around.
The documentation on this part of things is still not great though.
Please let us know if you have questions.

> Some configuration issues I'm thinking of right now involve output.
> For instance, is there a way to turn off the feedback from each node
> for every line of code? I would imagine that if you had a cluster with
> many nodes running, you would not want to see this feedback. Another
> issue is how output on stdout from C++ code is handled. I noticed that
> right now, any output on stdout for a simple C++ module that I have
> exposed to python and loaded in each engine simply is simply written
> by the engine to the console. Is there some way to redirect this
> output?

There are two ways of executing code: blocking and non-blocking:

rc.executeAll('a=10', block=True)

This mode will wait until the command has been run and it will bring
back the stdout/stderr and print it to the screen.  If the remote
command is long running, your local ipython session will remain
blocked until the command is complete.

rc.executeAll('time.sleep(1000000)', block=False)

In non-blocking mode, execute returns immediately after _submitting_
the command.  Furthermore, it won't automatically bring back and print
the stdout/stderr of the command.

You can also set all command to block or not by using the block
attribute of the RemoteController object:

rc.block = False

I usually use block=True for debugging, but then set block=False for
long running things.  Also, you can always get the stdout/stderr of a
previously run command by using the %result magic:

%result                             # print the stdout/stderr of the
last remote commad

%result 10                       # print the stdout/stderr of the 10th
remote command

> 4) When objects are sent from the kernel to the client, is the only
> prerequisite to this that the object being sent is able to be pickled?
> I suppose the client would also have to have the class code for any
> objects imported?

Yes.  Numpy arrays are sent using their raw buffers so for that case
you don't have to pay the price of pickling.

> 5) Are there any known projects that use IPython1 for interactive
> scientific computing? I'd be really interested to see ones that also
> support visualization of distributed data.

Yes.  Two examples:

1.  At my company (Tech-X), there are a group of people using ipython1
to do parallel data analysis on supercomputers.  They start with
50-100 GBs of data in 1000-2000 hdf5 data files and need to do a bunch
of calculations that involve data from multiple files.  They use
ipython1 to first reduce the data they want (in parallel) to a single
file and then then run an algorithm (again in parallel) over a set of
parameters.  There were able to parallelize this code in 2 days and
its shows nice linear scaling.

2.  Fernando Perez (the creator of ipython) is using ipython1 in his
research in applied mathematics.  His algorithm uses multiresolution
analysis to solve high dimensional partial differential equations.  He
has just begun (in the last 2 weeks) to parallelize his code using
ipython1, so I don't think he is in production mode.  His case is also
non-trivial as it 1) needs automatic load balancing and 2) has lots of
communications.

The reason we started working on this, is that both Fernando and I are
scientists (both got our Ph.D.'s in Theoretical Physics) and we wanted
these tools to exist so we could use them for our own research.

There are some other folks on the list that have playing around with
ipython1, but I am not sure if anyone else has moved into production
mode yet.

> Well, I think that covers all my initial questions. Thank you for
> bearing this long post and my newbie questions. I'm really looking
> forward to working more with IPython as it seems like an amazing piece
> of software.

Thanks!  Please let us know if you have more questions/comments or ideas.

Brian

> Thank you,
> ~doug
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/ipython-dev
>


From vivainio at gmail.com  Thu Dec 14 16:21:54 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Thu, 14 Dec 2006 23:21:54 +0200
Subject: [IPython-dev] rc2 is ready
In-Reply-To: <46cb515a0612140808p1fc148a2x3fa21553b23b71ff@mail.gmail.com>
References: <46cb515a0612140808p1fc148a2x3fa21553b23b71ff@mail.gmail.com>
Message-ID: <46cb515a0612141321i3f8270b1ic81949a2ab01c24d@mail.gmail.com>

On 12/14/06, Ville M. Vainio <vivainio at gmail.com> wrote:

> I changed the windows post install script to always create shortcuts,
> even if it needs to overwrite the old ones. I also bumped up the
> release number to 0.7.3rc2.r1998.

Make that r2001. I did some last minute cr/lf -> lf normalizations.

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From fperez.net at gmail.com  Thu Dec 14 22:49:27 2006
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 14 Dec 2006 22:49:27 -0500
Subject: [IPython-dev] rc2 is ready
In-Reply-To: <46cb515a0612141321i3f8270b1ic81949a2ab01c24d@mail.gmail.com>
References: <46cb515a0612140808p1fc148a2x3fa21553b23b71ff@mail.gmail.com>
	<46cb515a0612141321i3f8270b1ic81949a2ab01c24d@mail.gmail.com>
Message-ID: <db6b5ecc0612141949y62efc1f4vf533ad6bc0239fdb@mail.gmail.com>

On 12/14/06, Ville M. Vainio <vivainio at gmail.com> wrote:
> On 12/14/06, Ville M. Vainio <vivainio at gmail.com> wrote:
>
> > I changed the windows post install script to always create shortcuts,
> > even if it needs to overwrite the old ones. I also bumped up the
> > release number to 0.7.3rc2.r1998.
>
> Make that r2001. I did some last minute cr/lf -> lf normalizations.

Done.  Sorry for the delay, down with a bad cold.

Cheers,

f


From vivainio at gmail.com  Fri Dec 15 03:25:33 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Fri, 15 Dec 2006 10:25:33 +0200
Subject: [IPython-dev] rc2 is ready
In-Reply-To: <006501c71fcf$78abba00$0a83a8c0@ratbert>
References: <46cb515a0612140808p1fc148a2x3fa21553b23b71ff@mail.gmail.com>
	<46cb515a0612141321i3f8270b1ic81949a2ab01c24d@mail.gmail.com>
	<006501c71fcf$78abba00$0a83a8c0@ratbert>
Message-ID: <46cb515a0612150025w34332b1jce39636461fea71e@mail.gmail.com>

On 12/15/06, Albert Strasheim <fullung at gmail.com> wrote:

> If you have time to sort out another minor Windows annoyance, perhaps you
> could fix this:
>
> http://projects.scipy.org/ipython/ipython/ticket/99

Ah, it was reported to be in the "chainsaw" component so I ignored it.

I think I'll just copy (as opposed to rename) the current "ipython"
script to ipython.py on win32, to avoid breaking things that may rely
on existence of plain "ipython" w/o extension.

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From vivainio at gmail.com  Fri Dec 15 05:19:01 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Fri, 15 Dec 2006 12:19:01 +0200
Subject: [IPython-dev] rc2 is ready
In-Reply-To: <46cb515a0612150025w34332b1jce39636461fea71e@mail.gmail.com>
References: <46cb515a0612140808p1fc148a2x3fa21553b23b71ff@mail.gmail.com>
	<46cb515a0612141321i3f8270b1ic81949a2ab01c24d@mail.gmail.com>
	<006501c71fcf$78abba00$0a83a8c0@ratbert>
	<46cb515a0612150025w34332b1jce39636461fea71e@mail.gmail.com>
Message-ID: <46cb515a0612150219t3cfb54f1j3fe63d882c792267@mail.gmail.com>

On 12/15/06, Ville M. Vainio <vivainio at gmail.com> wrote:

> I think I'll just copy (as opposed to rename) the current "ipython"
> script to ipython.py on win32, to avoid breaking things that may rely
> on existence of plain "ipython" w/o extension.

Done in 0.7.3 branch.

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From fperez.net at gmail.com  Fri Dec 15 07:13:48 2006
From: fperez.net at gmail.com (Fernando Perez)
Date: Fri, 15 Dec 2006 07:13:48 -0500
Subject: [IPython-dev] rc2 is ready
In-Reply-To: <46cb515a0612150219t3cfb54f1j3fe63d882c792267@mail.gmail.com>
References: <46cb515a0612140808p1fc148a2x3fa21553b23b71ff@mail.gmail.com>
	<46cb515a0612141321i3f8270b1ic81949a2ab01c24d@mail.gmail.com>
	<006501c71fcf$78abba00$0a83a8c0@ratbert>
	<46cb515a0612150025w34332b1jce39636461fea71e@mail.gmail.com>
	<46cb515a0612150219t3cfb54f1j3fe63d882c792267@mail.gmail.com>
Message-ID: <db6b5ecc0612150413r2d17fffeq3250af77befc05f8@mail.gmail.com>

On 12/15/06, Ville M. Vainio <vivainio at gmail.com> wrote:
> On 12/15/06, Ville M. Vainio <vivainio at gmail.com> wrote:
>
> > I think I'll just copy (as opposed to rename) the current "ipython"
> > script to ipython.py on win32, to avoid breaking things that may rely
> > on existence of plain "ipython" w/o extension.
>
> Done in 0.7.3 branch.

Great.  Let's keep in mind to do a sync to trunk after the release.

Cheers,

f


From dfj225 at gmail.com  Fri Dec 15 19:43:30 2006
From: dfj225 at gmail.com (Douglas Jones)
Date: Fri, 15 Dec 2006 19:43:30 -0500
Subject: [IPython-dev] IPython1 parallel questions
In-Reply-To: <6ce0ac130612140951v58f1299fq825dfd7077a9d762@mail.gmail.com>
References: <1315be7e0612131344w40a21830td32d982de9b0fff4@mail.gmail.com>
	<6ce0ac130612140951v58f1299fq825dfd7077a9d762@mail.gmail.com>
Message-ID: <1315be7e0612151643l76547041k98b6674ae5562706@mail.gmail.com>

Brian,

Thank you for the response.

I've been moving forward with the integration of my library with
IPython. Unfortunately, I've come across an issue that has prevented
our library from functioning correctly.

First, let me give a little background. Right now, our library is
loaded in each engine by importing it as a python module. There are
cases in our library when a function will be executed on each node,
but one node may take slightly longer to process. For instance, when
reading input node 0 is responsible for scanning the file and
distributing data to each process. Code of this fashion takes
advantage of MPI_Win_lock and MPI_Win_unlock when sending data to
other processes.

The issue I've come across is that at times these calls to MPI code
will deadlock in certain instances. What I think is going on is that
the functions in our library that do relatively little will return,
causing the IPEngine to continue to its input handler code (in effect
blocking on the engine's socket). If, after this has happened, another
node tries to do MPI_Win_lock or unlock the call will deadlock because
MPI is unable to communicate with the other nodes that are now blocked
in IPEngine code.

At least, this is my educated guess as to what is happening.

Is this an issue that you've had to work out before?

Is there some way to ensure that each engine will not block on its
socket until every engine has had a chance to execute the user code
completely?

I guess what I'm thinking of is similar to an MPI_Barrier such that
each engine would

execute(user_code)
Barrier()
#once each engine is at this point, the engines can return to their internal
#communication code

I greatly appreciate your feedback.

Thank you,
~doug


On 12/14/06, Brian Granger <ellisonbg.net at gmail.com> wrote:
> Douglas,
>
> See my comments inline below.
>
> > I recently discovered IPython1 and its parallel capabilities. Once I
> > read about the feature set, I became extremely excited as IPython1
> > seems to solve many of the interactive parallel computing problems
> > that I had hoped to solve for a project that I am a developer on.
> >
> > Let me start by framing my project. Our code currently exists as
> > library written in C++. Our library does parallel computation using
> > MPI. We currently have code that exposes this to python in an
> > interactive manner, and have been looking for ways to expand this to a
> > fuller more robust implementation. I'm hoping that IPython1 will be
> > the perfect piece to create our collaborative, interactive
> > environment.
>
> Nice, this is one of the main usage cases we had in mind when
> designing ipython1.  It should work well for this.
>
> > That said, as I investigate IPython further and start to develop some
> > prototype code, I have some questions for the community and
> > developers.
> >
> > 1) Are there any outstanding issues or problems with the IPython1 code
> > base that inhibits parallel computation? I've been experimenting with
> > the code and its features for the past few days in addition to looking
> > through the bug list, and from what I can tell there are none. Are
> > there any features mentioned in IPython's documentation or the
> > presentations on the site that haven't been implemented yet?
>
> Not really.  IPython1 is already being used for real science and
> people don't seem to be limited in any significant way.  With that
> said, there are a number of things we are still working on.  Here is a
> taste:
>
> 1.  Optimizing how large objects are send between the client and
> engines.  In our current approach, the controller become a bottleneck
> when you try to use push/pull to send really big things (> 100's of
> MBs).  With that said, if you are wanting to send large objects
> around, you might want to rethink how you are parallelizing you
> algorithm.
>
> 2.  Task farming.  While the architecture is setup for task farming,
> we have not implemented a few parts of it.  We are actively (as in
> this week) working on this.
>
> 3.  Security.
>
> 4.  Scalability.  Currently all the engines connect to a single
> controller.  We have tested this on 128 processors and it works fine.
> The problem is that most systems have a per process file-descriptor
> limit that is not much higher than that (like 256).  We would like to
> explore ways (multiple controllers?) of getting it to scale beyond
> 128-256.
>
> > 2) Is there any authentication or security between the client shell
> > and the IPython controller?
>
> Not yet, but we are working on it.
>
> > 3) What are the proper ways to configure the operation of the client
> > and the controller or the engines? From what I've seen it looks like
> > there is an API that allows the user to set certain options. Can
> > configuration files be used?
>
> Yes, we have a very powerful configuration system.  For examples see
> the ipython1/configfiles directory.  You can simply copy these over to
> your ~/.ipython directory and start playing around.
> The documentation on this part of things is still not great though.
> Please let us know if you have questions.
>
> > Some configuration issues I'm thinking of right now involve output.
> > For instance, is there a way to turn off the feedback from each node
> > for every line of code? I would imagine that if you had a cluster with
> > many nodes running, you would not want to see this feedback. Another
> > issue is how output on stdout from C++ code is handled. I noticed that
> > right now, any output on stdout for a simple C++ module that I have
> > exposed to python and loaded in each engine simply is simply written
> > by the engine to the console. Is there some way to redirect this
> > output?
>
> There are two ways of executing code: blocking and non-blocking:
>
> rc.executeAll('a=10', block=True)
>
> This mode will wait until the command has been run and it will bring
> back the stdout/stderr and print it to the screen.  If the remote
> command is long running, your local ipython session will remain
> blocked until the command is complete.
>
> rc.executeAll('time.sleep(1000000)', block=False)
>
> In non-blocking mode, execute returns immediately after _submitting_
> the command.  Furthermore, it won't automatically bring back and print
> the stdout/stderr of the command.
>
> You can also set all command to block or not by using the block
> attribute of the RemoteController object:
>
> rc.block = False
>
> I usually use block=True for debugging, but then set block=False for
> long running things.  Also, you can always get the stdout/stderr of a
> previously run command by using the %result magic:
>
> %result                             # print the stdout/stderr of the
> last remote commad
>
> %result 10                       # print the stdout/stderr of the 10th
> remote command
>
> > 4) When objects are sent from the kernel to the client, is the only
> > prerequisite to this that the object being sent is able to be pickled?
> > I suppose the client would also have to have the class code for any
> > objects imported?
>
> Yes.  Numpy arrays are sent using their raw buffers so for that case
> you don't have to pay the price of pickling.
>
> > 5) Are there any known projects that use IPython1 for interactive
> > scientific computing? I'd be really interested to see ones that also
> > support visualization of distributed data.
>
> Yes.  Two examples:
>
> 1.  At my company (Tech-X), there are a group of people using ipython1
> to do parallel data analysis on supercomputers.  They start with
> 50-100 GBs of data in 1000-2000 hdf5 data files and need to do a bunch
> of calculations that involve data from multiple files.  They use
> ipython1 to first reduce the data they want (in parallel) to a single
> file and then then run an algorithm (again in parallel) over a set of
> parameters.  There were able to parallelize this code in 2 days and
> its shows nice linear scaling.
>
> 2.  Fernando Perez (the creator of ipython) is using ipython1 in his
> research in applied mathematics.  His algorithm uses multiresolution
> analysis to solve high dimensional partial differential equations.  He
> has just begun (in the last 2 weeks) to parallelize his code using
> ipython1, so I don't think he is in production mode.  His case is also
> non-trivial as it 1) needs automatic load balancing and 2) has lots of
> communications.
>
> The reason we started working on this, is that both Fernando and I are
> scientists (both got our Ph.D.'s in Theoretical Physics) and we wanted
> these tools to exist so we could use them for our own research.
>
> There are some other folks on the list that have playing around with
> ipython1, but I am not sure if anyone else has moved into production
> mode yet.
>
> > Well, I think that covers all my initial questions. Thank you for
> > bearing this long post and my newbie questions. I'm really looking
> > forward to working more with IPython as it seems like an amazing piece
> > of software.
>
> Thanks!  Please let us know if you have more questions/comments or ideas.
>
> Brian
>
> > Thank you,
> > ~doug
> > _______________________________________________
> > IPython-dev mailing list
> > IPython-dev at scipy.org
> > http://projects.scipy.org/mailman/listinfo/ipython-dev
> >
>


From ellisonbg.net at gmail.com  Sat Dec 16 12:10:28 2006
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Sat, 16 Dec 2006 10:10:28 -0700
Subject: [IPython-dev] IPython1 parallel questions
In-Reply-To: <1315be7e0612151643l76547041k98b6674ae5562706@mail.gmail.com>
References: <1315be7e0612131344w40a21830td32d982de9b0fff4@mail.gmail.com>
	<6ce0ac130612140951v58f1299fq825dfd7077a9d762@mail.gmail.com>
	<1315be7e0612151643l76547041k98b6674ae5562706@mail.gmail.com>
Message-ID: <6ce0ac130612160910l228672edjac230238d341c2d6@mail.gmail.com>

On 12/15/06, Douglas Jones <dfj225 at gmail.com> wrote:
> Brian,
>
> Thank you for the response.
>
> I've been moving forward with the integration of my library with
> IPython. Unfortunately, I've come across an issue that has prevented
> our library from functioning correctly.
>
> First, let me give a little background. Right now, our library is
> loaded in each engine by importing it as a python module. There are
> cases in our library when a function will be executed on each node,
> but one node may take slightly longer to process. For instance, when
> reading input node 0 is responsible for scanning the file and
> distributing data to each process. Code of this fashion takes
> advantage of MPI_Win_lock and MPI_Win_unlock when sending data to
> other processes.

I have never used these MPI calls, but hopefully I can help clarify
what ipython1 does and the issues to watch out for (see below).

> The issue I've come across is that at times these calls to MPI code
> will deadlock in certain instances. What I think is going on is that
> the functions in our library that do relatively little will return,
> causing the IPEngine to continue to its input handler code (in effect
> blocking on the engine's socket). If, after this has happened, another
> node tries to do MPI_Win_lock or unlock the call will deadlock because
> MPI is unable to communicate with the other nodes that are now blocked
> in IPEngine code.
>
> At least, this is my educated guess as to what is happening.

This is a very plausible scenarrio.

> Is this an issue that you've had to work out before?

At some level yes.  I think it is best to describe the execution flow
in each engine and how that relates to the synchronization issues of
MPi.  Each engine repeats the following loop:

1.  Wait for the controller to send it a command.  During this period,
it doesn't have anything to do so it is idle.  But, it doesn't "block"
on the socket as it uses non-blocking sockets.  Underneath it is
iterating through a select/poll loop that is watching for incoming
data.

2.  When the controller send the engine a command to execute, the
engine's event loop begins the execution of the command.  While the
command is running, the event loop that is watching for network
traffic effectively stops.

3.  When the command is done, the engine sends the results back to the engine.

4.  The engine waits for further commands.

The one important point is that each user command can be many atomic
commands underneath.  For instance, you might make one python function
call that in turns calls a C++ class method, which in turn makes
dozens of MPi calls.

In terms of synchronizing the different engines for "safe" MPI usage,
ipython1 makes the same guarentees that you get in regular MPi
programs:  you must use explicit MPI Barrier calls to enforce
synchronization.  At that level the only difference between using
ipython1 compared with a normal MPI code is where the commands are
coming from (over the network vs from compiled code).

> Is there some way to ensure that each engine will not block on its
> socket until every engine has had a chance to execute the user code
> completely?

Each command that is sent to an engine is always completed before the
engine begins to get another command.

> I guess what I'm thinking of is similar to an MPI_Barrier such that
> each engine would
>
> execute(user_code)
> Barrier()
> #once each engine is at this point, the engines can return to their internal
> #communication code

I would instead, put the MPI_Barrier() in the user code at the right places.

> I greatly appreciate your feedback.

Let me know if this solves some of your problems.  If the problems
persist, you might want to email the mpi4py mailing list.  Lisandro
Dalcin is a true MPI pro.

Brian

> Thank you,
> ~doug
>
>
> On 12/14/06, Brian Granger <ellisonbg.net at gmail.com> wrote:
> > Douglas,
> >
> > See my comments inline below.
> >
> > > I recently discovered IPython1 and its parallel capabilities. Once I
> > > read about the feature set, I became extremely excited as IPython1
> > > seems to solve many of the interactive parallel computing problems
> > > that I had hoped to solve for a project that I am a developer on.
> > >
> > > Let me start by framing my project. Our code currently exists as
> > > library written in C++. Our library does parallel computation using
> > > MPI. We currently have code that exposes this to python in an
> > > interactive manner, and have been looking for ways to expand this to a
> > > fuller more robust implementation. I'm hoping that IPython1 will be
> > > the perfect piece to create our collaborative, interactive
> > > environment.
> >
> > Nice, this is one of the main usage cases we had in mind when
> > designing ipython1.  It should work well for this.
> >
> > > That said, as I investigate IPython further and start to develop some
> > > prototype code, I have some questions for the community and
> > > developers.
> > >
> > > 1) Are there any outstanding issues or problems with the IPython1 code
> > > base that inhibits parallel computation? I've been experimenting with
> > > the code and its features for the past few days in addition to looking
> > > through the bug list, and from what I can tell there are none. Are
> > > there any features mentioned in IPython's documentation or the
> > > presentations on the site that haven't been implemented yet?
> >
> > Not really.  IPython1 is already being used for real science and
> > people don't seem to be limited in any significant way.  With that
> > said, there are a number of things we are still working on.  Here is a
> > taste:
> >
> > 1.  Optimizing how large objects are send between the client and
> > engines.  In our current approach, the controller become a bottleneck
> > when you try to use push/pull to send really big things (> 100's of
> > MBs).  With that said, if you are wanting to send large objects
> > around, you might want to rethink how you are parallelizing you
> > algorithm.
> >
> > 2.  Task farming.  While the architecture is setup for task farming,
> > we have not implemented a few parts of it.  We are actively (as in
> > this week) working on this.
> >
> > 3.  Security.
> >
> > 4.  Scalability.  Currently all the engines connect to a single
> > controller.  We have tested this on 128 processors and it works fine.
> > The problem is that most systems have a per process file-descriptor
> > limit that is not much higher than that (like 256).  We would like to
> > explore ways (multiple controllers?) of getting it to scale beyond
> > 128-256.
> >
> > > 2) Is there any authentication or security between the client shell
> > > and the IPython controller?
> >
> > Not yet, but we are working on it.
> >
> > > 3) What are the proper ways to configure the operation of the client
> > > and the controller or the engines? From what I've seen it looks like
> > > there is an API that allows the user to set certain options. Can
> > > configuration files be used?
> >
> > Yes, we have a very powerful configuration system.  For examples see
> > the ipython1/configfiles directory.  You can simply copy these over to
> > your ~/.ipython directory and start playing around.
> > The documentation on this part of things is still not great though.
> > Please let us know if you have questions.
> >
> > > Some configuration issues I'm thinking of right now involve output.
> > > For instance, is there a way to turn off the feedback from each node
> > > for every line of code? I would imagine that if you had a cluster with
> > > many nodes running, you would not want to see this feedback. Another
> > > issue is how output on stdout from C++ code is handled. I noticed that
> > > right now, any output on stdout for a simple C++ module that I have
> > > exposed to python and loaded in each engine simply is simply written
> > > by the engine to the console. Is there some way to redirect this
> > > output?
> >
> > There are two ways of executing code: blocking and non-blocking:
> >
> > rc.executeAll('a=10', block=True)
> >
> > This mode will wait until the command has been run and it will bring
> > back the stdout/stderr and print it to the screen.  If the remote
> > command is long running, your local ipython session will remain
> > blocked until the command is complete.
> >
> > rc.executeAll('time.sleep(1000000)', block=False)
> >
> > In non-blocking mode, execute returns immediately after _submitting_
> > the command.  Furthermore, it won't automatically bring back and print
> > the stdout/stderr of the command.
> >
> > You can also set all command to block or not by using the block
> > attribute of the RemoteController object:
> >
> > rc.block = False
> >
> > I usually use block=True for debugging, but then set block=False for
> > long running things.  Also, you can always get the stdout/stderr of a
> > previously run command by using the %result magic:
> >
> > %result                             # print the stdout/stderr of the
> > last remote commad
> >
> > %result 10                       # print the stdout/stderr of the 10th
> > remote command
> >
> > > 4) When objects are sent from the kernel to the client, is the only
> > > prerequisite to this that the object being sent is able to be pickled?
> > > I suppose the client would also have to have the class code for any
> > > objects imported?
> >
> > Yes.  Numpy arrays are sent using their raw buffers so for that case
> > you don't have to pay the price of pickling.
> >
> > > 5) Are there any known projects that use IPython1 for interactive
> > > scientific computing? I'd be really interested to see ones that also
> > > support visualization of distributed data.
> >
> > Yes.  Two examples:
> >
> > 1.  At my company (Tech-X), there are a group of people using ipython1
> > to do parallel data analysis on supercomputers.  They start with
> > 50-100 GBs of data in 1000-2000 hdf5 data files and need to do a bunch
> > of calculations that involve data from multiple files.  They use
> > ipython1 to first reduce the data they want (in parallel) to a single
> > file and then then run an algorithm (again in parallel) over a set of
> > parameters.  There were able to parallelize this code in 2 days and
> > its shows nice linear scaling.
> >
> > 2.  Fernando Perez (the creator of ipython) is using ipython1 in his
> > research in applied mathematics.  His algorithm uses multiresolution
> > analysis to solve high dimensional partial differential equations.  He
> > has just begun (in the last 2 weeks) to parallelize his code using
> > ipython1, so I don't think he is in production mode.  His case is also
> > non-trivial as it 1) needs automatic load balancing and 2) has lots of
> > communications.
> >
> > The reason we started working on this, is that both Fernando and I are
> > scientists (both got our Ph.D.'s in Theoretical Physics) and we wanted
> > these tools to exist so we could use them for our own research.
> >
> > There are some other folks on the list that have playing around with
> > ipython1, but I am not sure if anyone else has moved into production
> > mode yet.
> >
> > > Well, I think that covers all my initial questions. Thank you for
> > > bearing this long post and my newbie questions. I'm really looking
> > > forward to working more with IPython as it seems like an amazing piece
> > > of software.
> >
> > Thanks!  Please let us know if you have more questions/comments or ideas.
> >
> > Brian
> >
> > > Thank you,
> > > ~doug
> > > _______________________________________________
> > > IPython-dev mailing list
> > > IPython-dev at scipy.org
> > > http://projects.scipy.org/mailman/listinfo/ipython-dev
> > >
> >
>


From dfj225 at gmail.com  Mon Dec 18 15:52:33 2006
From: dfj225 at gmail.com (Douglas Jones)
Date: Mon, 18 Dec 2006 15:52:33 -0500
Subject: [IPython-dev] IPython1 parallel questions
In-Reply-To: <6ce0ac130612160910l228672edjac230238d341c2d6@mail.gmail.com>
References: <1315be7e0612131344w40a21830td32d982de9b0fff4@mail.gmail.com>
	<6ce0ac130612140951v58f1299fq825dfd7077a9d762@mail.gmail.com>
	<1315be7e0612151643l76547041k98b6674ae5562706@mail.gmail.com>
	<6ce0ac130612160910l228672edjac230238d341c2d6@mail.gmail.com>
Message-ID: <1315be7e0612181252o48c6b0a1y42194d3d89399b2c@mail.gmail.com>

Brian,

Thank you for your responses, they have been very helpful.

Another question that came up involves the termination of a running command.

As an example, say that I do

rc.executeAll('commandX()')

and commandX happens to be a  long running command. Shortly after
hiting enter, I realize using commandX was a mistake.

Is there some way to halt the execution of commandX?

Based on your description of how the engines operate, I suppose this
currently isn't feasible. Have you considered creating a way to halt
commands on the engine without destroying the state of the system? I
think something like this would be useful for interactive computing
with large datasets.

Thank you,
~Doug


From ellisonbg.net at gmail.com  Mon Dec 18 18:21:29 2006
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Mon, 18 Dec 2006 16:21:29 -0700
Subject: [IPython-dev] IPython1 parallel questions
In-Reply-To: <1315be7e0612181252o48c6b0a1y42194d3d89399b2c@mail.gmail.com>
References: <1315be7e0612131344w40a21830td32d982de9b0fff4@mail.gmail.com>
	<6ce0ac130612140951v58f1299fq825dfd7077a9d762@mail.gmail.com>
	<1315be7e0612151643l76547041k98b6674ae5562706@mail.gmail.com>
	<6ce0ac130612160910l228672edjac230238d341c2d6@mail.gmail.com>
	<1315be7e0612181252o48c6b0a1y42194d3d89399b2c@mail.gmail.com>
Message-ID: <6ce0ac130612181521h5b0beb62w8512f4861a785cb9@mail.gmail.com>

This is absolutely something we have on our radar, but we don't have
it implemented yet.  It is especially complicated if commandX()
happens to run C/C++ code.  Then the signal handlers in Python won't
do a thing.  To get that to work, you need:

1.  Another process on the same machine that send OS level signals.

2.  The author of the C-code must use a macro to install the right
signal handlers.

This becomes even more complicated if MPI is involved.  This that
said, Fernando has thought quite a bit about this before and we plan
to eventually work on it.

Brian

On 12/18/06, Douglas Jones <dfj225 at gmail.com> wrote:
> Brian,
>
> Thank you for your responses, they have been very helpful.
>
> Another question that came up involves the termination of a running command.
>
> As an example, say that I do
>
> rc.executeAll('commandX()')
>
> and commandX happens to be a  long running command. Shortly after
> hiting enter, I realize using commandX was a mistake.
>
> Is there some way to halt the execution of commandX?
>
> Based on your description of how the engines operate, I suppose this
> currently isn't feasible. Have you considered creating a way to halt
> commands on the engine without destroying the state of the system? I
> think something like this would be useful for interactive computing
> with large datasets.
>
> Thank you,
> ~Doug
>


From novak at ucolick.org  Tue Dec 19 19:15:20 2006
From: novak at ucolick.org (Gregory Novak)
Date: Tue, 19 Dec 2006 16:15:20 -0800
Subject: [IPython-dev] Apropos for Python
Message-ID: <m2bqlzjucn.fsf@ucolick.org>

This seems like something that surely someone has written, but I
couldn't find it so away I went...

I was trying to find a variable to check to see whether or not IPython
had been loaded with the -pylab option.  Emacs has nice apropos,
apropos-variable, apropos-function, and apropos-documentation
functions.  I sought to emulate them in Python.  Attached is what I
came up with.

A call like apropos_name("pylab", IPython) will look for any attribute
with pylab in its name (through dir(IPython)).  Then it'll recursively
search through any compound objects available through IPython, etc.  

It keeps a list of the ids of the objects it has already searched, so
it doens't waste (too much) time and circular references shouldn't be
a problem.  

There's also apropos_value (searching the string representation of
objects), apropos_doc, (searching docstrings), apropos_name_regexp,
apropos_value_regexp, and apropos_doc_regexp.  Finally, there's plain
old apropos() that allows you to specify a function that looks for and
searches the bit of the object you care about.

So there it is.  Comments and feedback welcome.  The very likely
comment "That's nice and all, but you should be using
super-py-apropos-deluxe that everyone else uses and is 100x better" is
welcome, too.

By the way, the call apropos_name('pylab', IPython) led me to:

IPython.ConfigLoader.ultraTB.pydoc.__builtin__._ip.options.pylab
and
IPython.ConfigLoader.ultraTB.pydoc.__builtin__._ip.options.opts.pylab

Is that really where I have to look to find out whether or not IPython
was started with the pylab option?  That's even more buried than I
thought it would be.

Cheers,
Greg

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: apropos.py
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20061219/3223c3e6/attachment.ksh>

From vivainio at gmail.com  Wed Dec 20 02:11:15 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Wed, 20 Dec 2006 09:11:15 +0200
Subject: [IPython-dev] Apropos for Python
In-Reply-To: <m2bqlzjucn.fsf@ucolick.org>
References: <m2bqlzjucn.fsf@ucolick.org>
Message-ID: <46cb515a0612192311s5302cf50j956f00b68b037f08@mail.gmail.com>

On 12/20/06, Gregory Novak <novak at ucolick.org> wrote:

> IPython.ConfigLoader.ultraTB.pydoc.__builtin__._ip.options.pylab
> and
> IPython.ConfigLoader.ultraTB.pydoc.__builtin__._ip.options.opts.pylab
>
> Is that really where I have to look to find out whether or not IPython
> was started with the pylab option?  That's even more buried than I
> thought it would be.

In interactive shell, it's:

_ip.options.pylab

In non-interactive session, it is

import IPython.ipapi
ip = IPython.ipapi.get()
print ip.options.pylab


And thanks for the submission! Still, it would be more convenient as a
magic function... see

http://ipython.scipy.org/moin/Cookbook/CustomMagic

and

http://ipython.scipy.org/moin/IpythonExtensionApi

on how to do it (it's pretty easy).

BTW, this is a bit similar to %psearch, see "%psearch?".

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From vivainio at gmail.com  Wed Dec 20 10:31:16 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Wed, 20 Dec 2006 17:31:16 +0200
Subject: [IPython-dev] Trunk is open
Message-ID: <46cb515a0612200731w35c56095l1f8c4889eadeb96a@mail.gmail.com>

With the release of 0.7.3, trunk is again open for normal development.
I just merged all the mods made in 0.7.3 branch to trunk.

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From novak at ucolick.org  Wed Dec 20 12:38:10 2006
From: novak at ucolick.org (Gregory Novak)
Date: Wed, 20 Dec 2006 09:38:10 -0800
Subject: [IPython-dev] Apropos for Python
In-Reply-To: <46cb515a0612192311s5302cf50j956f00b68b037f08@mail.gmail.com>
	(Ville M. Vainio's message of "Wed, 20 Dec 2006 09:11:15 +0200")
References: <m2bqlzjucn.fsf@ucolick.org>
	<46cb515a0612192311s5302cf50j956f00b68b037f08@mail.gmail.com>
Message-ID: <m27iwmjwn1.fsf@ucolick.org>

"Ville M. Vainio" <vivainio at gmail.com> writes:
> BTW, this is a bit similar to %psearch, see "%psearch?".

The documentation seemed to indicate that it did what I wanted, but
when I played with it I didn't get what I expected.  So I didn't
explore further.

Thanks,
Greg


From vivainio at gmail.com  Wed Dec 20 13:33:13 2006
From: vivainio at gmail.com (Ville M. Vainio)
Date: Wed, 20 Dec 2006 20:33:13 +0200
Subject: [IPython-dev] Apropos for Python
In-Reply-To: <m2bqlzjucn.fsf@ucolick.org>
References: <m2bqlzjucn.fsf@ucolick.org>
Message-ID: <46cb515a0612201033h2a5e0e57n439e890df7b02f78@mail.gmail.com>

On 12/20/06, Gregory Novak <novak at ucolick.org> wrote:

> This seems like something that surely someone has written, but I
> couldn't find it so away I went...

Here's a glitch:

q:\dl\apropos.py in search_doc(needle='file', name='_Environ',
obj=<class os._Environ at 0x009D2E70>)
    114 def search_value(needle, name, obj): return needle in str(obj)
    115 def search_doc(needle, name, obj):   return hasattr(obj,
'__doc__') and \
--> 116     needle in obj.__doc__
        needle = 'file'
        obj.__doc__ = None
    117
    118 def search_name_regexp(needle, name, obj):  return
re.search(needle, name)
<type 'exceptions.TypeError'>: argument of type 'NoneType' is not iterable

i.e. if there is __doc__ but it's None, it fails. Easy to fix, of course.


-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From Fernando.Perez at colorado.edu  Sun Dec 31 20:27:30 2006
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Sun, 31 Dec 2006 18:27:30 -0700
Subject: [IPython-dev] Auto-discard notification
In-Reply-To: <mailman.58359.1164384313.2761.ipython-user@scipy.org>
References: <mailman.58359.1164384313.2761.ipython-user@scipy.org>
Message-ID: <45986382.3030204@colorado.edu>

Hi Andreas,

[ cc'd to the user list for archiving ]

ipython-user-bounces at scipy.org wrote:
> The attached message has been automatically discarded.
> 
> 
> ------------------------------------------------------------------------
> 
> Subject:
> remote ipython shell
> From:
> Andreas <A.Schoeller at web.de>
> Date:
> Fri, 24 Nov 2006 16:02:08 +0000 (UTC)
> To:
> ipython-user at scipy.net
> 
> To:
> ipython-user at scipy.net
> 
> 
> Hello,
> 
> i'm trying to talk to a embedded remote ipython-process via Rpyc. 
> The embedded-ipython-shell starts 
> and is ready for input,but the readline-support is gone. 
> Somebd recommended to call the completer-function directly.
> This should work as described in the ipython source by :
> x = "test"
> __IP.complete('x.l')
> but it does not. It returnes []
> Remote or local doesn't matter, the return is always empty ?
> Am i doing smth wrong here ?


No, we are :(  This is a bug, since what you're doing is basically the 
docstring example and should work.  We'll look into it, thanks for the report.

Here's the ticket to track this one:

http://projects.scipy.org/ipython/ipython/ticket/109


Mmh, this used to work...  Time to finish up the testing stuff I've been 
writing so we can actually unittest these docstrings as we work.

Cheers,

f