From fperez.net at gmail.com  Sun Feb  1 03:01:38 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Sun, 1 Feb 2009 00:01:38 -0800
Subject: [IPython-dev] review request
In-Reply-To: <db6b5ecc0901310036u58cb609k9af6cc8db5567186@mail.gmail.com>
References: <49778077.4080005@bostream.nu>
	<db6b5ecc0901310036u58cb609k9af6cc8db5567186@mail.gmail.com>
Message-ID: <db6b5ecc0902010001ja40faa2g49800251c994b43b@mail.gmail.com>

On Sat, Jan 31, 2009 at 12:36 AM, Fernando Perez <fperez.net at gmail.com> wrote:

> Sorry, I've been MIA for a while.  I'm going to bed now, but I'll
> finish this review tomorrow.  I'll try to knock out a few more reviews
> as well if I can over the weekend.

Done in the lp review, if anyone else has comments, please post them there...

Cheers,

f


From fperez.net at gmail.com  Sun Feb  1 03:15:54 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Sun, 1 Feb 2009 00:15:54 -0800
Subject: [IPython-dev] Useful posts on launchpad/bzr use
Message-ID: <db6b5ecc0902010015i49df0beakb52ba42d4b171866@mail.gmail.com>

Howdy,

in case anyone is interested, I found a couple of useful posts on
using bzr and launchpad:

http://theironlion.net/blog/2009/01/23/launchpad-and-bazaar-how-do-it/
http://theironlion.net/blog/2009/01/13/using-bazaar-launchpad-making-pushing-easy/

Ironically I was googling for 'why is launchpad so slow', because
right now it's timing out for me on large branch diffs :)  His first
post actually speaks of coming speedups, let's hope it's true...  But
there's some good info in there, at least for me (I feel like I'm
still learning there).

Cheers,

f


From laurent.dufrechou at gmail.com  Sun Feb  1 08:05:29 2009
From: laurent.dufrechou at gmail.com (=?us-ascii?Q?Laurent_Dufrechou?=)
Date: Sun, 1 Feb 2009 14:05:29 +0100
Subject: [IPython-dev] Merging to main
Message-ID: <49859e1f.0437560a.7980.ffffee56@mx.google.com>

Hi fernando,

It's a long time since i worked on my branch. Hopefully I had some little
time recently and cleaned up some things, added option to disable threading,
and commited 2 fix that close two bug reported by users.

What is the process to merge my work in the main branch?

I've clicked on some option in launchpad, requesting a review but don't know
if it the 'good' process..

 

Regards,

Laurent

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090201/753b41f5/attachment.html>

From laurent.dufrechou at gmail.com  Sun Feb  1 08:11:40 2009
From: laurent.dufrechou at gmail.com (=?iso-8859-1?Q?Laurent_Dufr=E9chou?=)
Date: Sun, 1 Feb 2009 14:11:40 +0100
Subject: [IPython-dev] bzr, whoami and email hiding
Message-ID: <49859f92.1358560a.1877.16a1@mx.google.com>

Hello,
Reading fernando last link i?ve seen that?:

<<<<<<<<<<<<<<<<<<
The importance of bzr whoami
One of the really important things that lots of people forget is to start
off with is setting your name is Bazaar. This can be done really easily by
doing bzr whoami "Your Name <email at example.com>" It's important that you set
this properly, or your revisions aren't going to get your name on them. The
email address you set here also needs to be an email that Launchpad knows
about (a confirmed address). This way, you get the karma for the revisions,
and revision listings in a branch page will link back to your profile.
>>>>>>>>>>>>>>>>>>

It is said that I need to give a true email address,(the one given to
launchpad in fact). But is it sure the email address will be hidden from
launchpad site? I mean will it be written in clear text to anybody going to
launchpad (and also robots...)?

Thx for any comments on this!

Laurent



From fperez.net at gmail.com  Sun Feb  1 15:45:12 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Sun, 1 Feb 2009 12:45:12 -0800
Subject: [IPython-dev] Merging to main
In-Reply-To: <49859e1f.0437560a.7980.ffffee56@mx.google.com>
References: <49859e1f.0437560a.7980.ffffee56@mx.google.com>
Message-ID: <db6b5ecc0902011245n2f9aac39q4ee96127928d9376@mail.gmail.com>

Hi Laurent,

On Sun, Feb 1, 2009 at 5:05 AM, Laurent Dufrechou
<laurent.dufrechou at gmail.com> wrote:
> Hi fernando,
>
> It's a long time since i worked on my branch? Hopefully I had some little
> time recently and cleaned up some things, added option to disable threading,
> and commited 2 fix that close two bug reported by users.
>
> What is the process to merge my work in the main branch?
>
> I've clicked on some option in launchpad, requesting a review but don't know
> if it the 'good' process..

Yes, that's the right way to do it.  I've just finished reviewing
Jorgen's branch, and I'll try over the next few days to review any
others pending.  Others from the team are welcome to review as well,
the idea is simply to have at least two sets of careful eyes go over
all code: the author and someone else who's used to the process.

In your case it would be great if Gael has a chance to give it a look,
since he's our other resident Wx person, but I'll do my best as well.
Ville has some code pending review that I should look at first,
though, since I've been falling behind on that pretty badly.  I'll do
my best to catch up with all of them now.

Cheers,

f


From fperez.net at gmail.com  Sun Feb  1 16:40:25 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Sun, 1 Feb 2009 13:40:25 -0800
Subject: [IPython-dev] bzr, whoami and email hiding
In-Reply-To: <49859f92.1358560a.1877.16a1@mx.google.com>
References: <49859f92.1358560a.1877.16a1@mx.google.com>
Message-ID: <db6b5ecc0902011340y3741054dkdda148d57a299168@mail.gmail.com>

Hi Laurent,

On Sun, Feb 1, 2009 at 5:11 AM, Laurent Dufr?chou
<laurent.dufrechou at gmail.com> wrote:
> Hello,
> Reading fernando last link i've seen that :
>
> <<<<<<<<<<<<<<<<<<
> The importance of bzr whoami
> One of the really important things that lots of people forget is to start
> off with is setting your name is Bazaar. This can be done really easily by
> doing bzr whoami "Your Name <email at example.com>" It's important that you set
> this properly, or your revisions aren't going to get your name on them. The
> email address you set here also needs to be an email that Launchpad knows
> about (a confirmed address). This way, you get the karma for the revisions,
> and revision listings in a branch page will link back to your profile.
>>>>>>>>>>>>>>>>>>>
>
> It is said that I need to give a true email address,(the one given to
> launchpad in fact). But is it sure the email address will be hidden from
> launchpad site? I mean will it be written in clear text to anybody going to
> launchpad (and also robots...)?
>
> Thx for any comments on this!

As far as I can see, launchpad hides email addresses for all
logged-out users.  If I check your user page:

https://launchpad.net/~laurent-dufrechou

from a firefox session where I've logged out, it won't show it to me,
but my logged-in session does show it.  So I suppose that if someone
writes a spam spider that uses a fake account and scrapes the user
pages, it's in principle possible to get those addresses, but they do
not appear to 'normal' robots.

Cheers,

f


From fperez.net at gmail.com  Sun Feb  1 16:42:23 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Sun, 1 Feb 2009 13:42:23 -0800
Subject: [IPython-dev] Useful posts on launchpad/bzr use
In-Reply-To: <db6b5ecc0902010015i49df0beakb52ba42d4b171866@mail.gmail.com>
References: <db6b5ecc0902010015i49df0beakb52ba42d4b171866@mail.gmail.com>
Message-ID: <db6b5ecc0902011342v15d82d47j9baabedc30c850df@mail.gmail.com>

On Sun, Feb 1, 2009 at 12:15 AM, Fernando Perez <fperez.net at gmail.com> wrote:
> Howdy,
>
> in case anyone is interested, I found a couple of useful posts on
> using bzr and launchpad:

Incidentally, I forgot yesterday to link to another good post on this same site:

http://theironlion.net/blog/2009/01/15/launchpad-code-reviews-without-browser/

about doing the whole review cycle purely via email.  That blog seems
like a good resource to keep in mind...

Cheers,

f


From gael.varoquaux at normalesup.org  Sun Feb  1 16:55:27 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Sun, 1 Feb 2009 22:55:27 +0100
Subject: [IPython-dev] Merging to main
In-Reply-To: <db6b5ecc0902011245n2f9aac39q4ee96127928d9376@mail.gmail.com>
References: <49859e1f.0437560a.7980.ffffee56@mx.google.com>
	<db6b5ecc0902011245n2f9aac39q4ee96127928d9376@mail.gmail.com>
Message-ID: <20090201215526.GI931@phare.normalesup.org>

On Sun, Feb 01, 2009 at 12:45:12PM -0800, Fernando Perez wrote:
> In your case it would be great if Gael has a chance to give it a look,
> since he's our other resident Wx person, but I'll do my best as well.
> Ville has some code pending review that I should look at first,
> though, since I've been falling behind on that pretty badly.  I'll do
> my best to catch up with all of them now.

Done.

Ga?l


From fperez.net at gmail.com  Sun Feb  1 16:58:57 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Sun, 1 Feb 2009 13:58:57 -0800
Subject: [IPython-dev] Merging to main
In-Reply-To: <20090201215526.GI931@phare.normalesup.org>
References: <49859e1f.0437560a.7980.ffffee56@mx.google.com>
	<db6b5ecc0902011245n2f9aac39q4ee96127928d9376@mail.gmail.com>
	<20090201215526.GI931@phare.normalesup.org>
Message-ID: <db6b5ecc0902011358t60ebf066n5d8c1c95edf9ffd0@mail.gmail.com>

On Sun, Feb 1, 2009 at 1:55 PM, Gael Varoquaux
> Done.

Awesome, thanks.  My comments added as well.


Cheers,

f


From dsdale24 at gmail.com  Sun Feb  1 19:28:27 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Sun, 1 Feb 2009 19:28:27 -0500
Subject: [IPython-dev] Useful posts on launchpad/bzr use
In-Reply-To: <db6b5ecc0902011342v15d82d47j9baabedc30c850df@mail.gmail.com>
References: <db6b5ecc0902010015i49df0beakb52ba42d4b171866@mail.gmail.com>
	<db6b5ecc0902011342v15d82d47j9baabedc30c850df@mail.gmail.com>
Message-ID: <a08e5f80902011628p5a98bdaauc197f6e0dd8c4aa4@mail.gmail.com>

On Sun, Feb 1, 2009 at 4:42 PM, Fernando Perez <fperez.net at gmail.com> wrote:

> On Sun, Feb 1, 2009 at 12:15 AM, Fernando Perez <fperez.net at gmail.com>
> wrote:
> > Howdy,
> >
> > in case anyone is interested, I found a couple of useful posts on
> > using bzr and launchpad:
>
> Incidentally, I forgot yesterday to link to another good post on this same
> site:
>
>
> http://theironlion.net/blog/2009/01/15/launchpad-code-reviews-without-browser/
>
> about doing the whole review cycle purely via email.  That blog seems
> like a good resource to keep in mind...
>

Thanks for pointing these out, this was really useful information. I ended
up signing up for the rss feed.

I spent the morning setting things up like he discussed in advanced
concepts. Here is my .bazaar/locations.conf, which allows you to integrate
the ideas he talked about in "making pushing easy"  with the layout in
advanced concepts:

[/home/darren/Projects/repos]
push_location = lp:~dsdale24/
push_location:policy = appendpath
public_branch = lp:~dsdale24/
public_branch:policy = appendpath

[/home/darren/Projects]
cbranch_target = /home/darren/Projects/repos
cbranch_target:policy = appendpath

Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090201/67502316/attachment.html>

From dsdale24 at gmail.com  Sun Feb  1 19:35:03 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Sun, 1 Feb 2009 19:35:03 -0500
Subject: [IPython-dev] bzr, whoami and email hiding
In-Reply-To: <db6b5ecc0902011340y3741054dkdda148d57a299168@mail.gmail.com>
References: <49859f92.1358560a.1877.16a1@mx.google.com>
	<db6b5ecc0902011340y3741054dkdda148d57a299168@mail.gmail.com>
Message-ID: <a08e5f80902011635u63fa3fd3yf48d340cdc66ed@mail.gmail.com>

On Sun, Feb 1, 2009 at 4:40 PM, Fernando Perez <fperez.net at gmail.com> wrote:

> Hi Laurent,
>
> On Sun, Feb 1, 2009 at 5:11 AM, Laurent Dufr?chou
> <laurent.dufrechou at gmail.com> wrote:
> > Hello,
> > Reading fernando last link i've seen that :
> >
> > <<<<<<<<<<<<<<<<<<
> > The importance of bzr whoami
> > One of the really important things that lots of people forget is to start
> > off with is setting your name is Bazaar. This can be done really easily
> by
> > doing bzr whoami "Your Name <email at example.com>" It's important that you
> set
> > this properly, or your revisions aren't going to get your name on them.
> The
> > email address you set here also needs to be an email that Launchpad knows
> > about (a confirmed address). This way, you get the karma for the
> revisions,
> > and revision listings in a branch page will link back to your profile.
> >>>>>>>>>>>>>>>>>>>
> >
> > It is said that I need to give a true email address,(the one given to
> > launchpad in fact). But is it sure the email address will be hidden from
> > launchpad site? I mean will it be written in clear text to anybody going
> to
> > launchpad (and also robots...)?
> >
> > Thx for any comments on this!
>
> As far as I can see, launchpad hides email addresses for all
> logged-out users.  If I check your user page:
>
> https://launchpad.net/~laurent-dufrechou<https://launchpad.net/%7Elaurent-dufrechou>
>
> from a firefox session where I've logged out, it won't show it to me,
> but my logged-in session does show it.  So I suppose that if someone
> writes a spam spider that uses a fake account and scrapes the user
> pages, it's in principle possible to get those addresses, but they do
> not appear to 'normal' robots.
>

There is an option on your launchpad profile page to hide your email address
from other launchpad users. That might hide your email from everyone.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090201/4e792209/attachment.html>

From fperez.net at gmail.com  Sun Feb  1 19:53:47 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Sun, 1 Feb 2009 16:53:47 -0800
Subject: [IPython-dev] looking for advice on a custom dict-like completer
In-Reply-To: <a08e5f80901311613k73453b6at95b9e3df51099cef@mail.gmail.com>
References: <a08e5f80901310637n39a6411ei63475c10a0b9a905@mail.gmail.com>
	<db6b5ecc0901310934v3019728dt9590a1e8a9fd82f@mail.gmail.com>
	<a08e5f80901311613k73453b6at95b9e3df51099cef@mail.gmail.com>
Message-ID: <db6b5ecc0902011653y54332d11sf455b616d91678a6@mail.gmail.com>

Hi Darren,

On Sat, Jan 31, 2009 at 4:13 PM, Darren Dale <dsdale24 at gmail.com> wrote:
> Hi Fernando,
>
> Thank you for the advice. I've been looking at this most of the day, and I
> think I'm stuck. I have an object I'm trying to navigate:

[...]

unfortunately I don't have a quick answer.  I've been trying to catch
up on many fronts today and this one I don't know how to answer off
the top of my head.  I'll keep it on my pile and will try to help you
soon, but don't hold your breath :)  Ville has done a lot of work on
the completions machinery, so he might have a good suggestion at the
ready though...

Cheers,

f


From fperez.net at gmail.com  Sun Feb  1 21:59:50 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Sun, 1 Feb 2009 18:59:50 -0800
Subject: [IPython-dev] Useful posts on launchpad/bzr use
In-Reply-To: <a08e5f80902011628p5a98bdaauc197f6e0dd8c4aa4@mail.gmail.com>
References: <db6b5ecc0902010015i49df0beakb52ba42d4b171866@mail.gmail.com>
	<db6b5ecc0902011342v15d82d47j9baabedc30c850df@mail.gmail.com>
	<a08e5f80902011628p5a98bdaauc197f6e0dd8c4aa4@mail.gmail.com>
Message-ID: <db6b5ecc0902011859m66667952kcd71a37531f10c9c@mail.gmail.com>

Hi,

On Sun, Feb 1, 2009 at 4:28 PM, Darren Dale <dsdale24 at gmail.com> wrote:

> Thanks for pointing these out, this was really useful information. I ended
> up signing up for the rss feed.
>
> I spent the morning setting things up like he discussed in advanced
> concepts. Here is my .bazaar/locations.conf, which allows you to integrate
> the ideas he talked about in "making pushing easy"  with the layout in
> advanced concepts:
>
> [/home/darren/Projects/repos]
> push_location = lp:~dsdale24/
> push_location:policy = appendpath
> public_branch = lp:~dsdale24/
> public_branch:policy = appendpath
>
> [/home/darren/Projects]
> cbranch_target = /home/darren/Projects/repos
> cbranch_target:policy = appendpath

I read his posts but haven't changed my setup yet, partly because my
typical layout is a little different.  I tend to have repos on  a
per-project directory rather than a global one, but I could easily (I
think) use those ideas with my layout, just by having several
repo/cbranch entries.

My question to you is: do you see any downside of using that split
between repos and checkouts at all?  I like that separation between
the repo storage and the working trees, but before I change things on
my laptop, I'd like to think through any possible disadvantages.

Thanks!

f


From vivainio at gmail.com  Mon Feb  2 08:07:31 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Mon, 2 Feb 2009 15:07:31 +0200
Subject: [IPython-dev] looking for advice on a custom dict-like completer
In-Reply-To: <a08e5f80901311613k73453b6at95b9e3df51099cef@mail.gmail.com>
References: <a08e5f80901310637n39a6411ei63475c10a0b9a905@mail.gmail.com>
	<db6b5ecc0901310934v3019728dt9590a1e8a9fd82f@mail.gmail.com>
	<a08e5f80901311613k73453b6at95b9e3df51099cef@mail.gmail.com>
Message-ID: <46cb515a0902020507n1292714vda7253c503222b81@mail.gmail.com>

On Sun, Feb 1, 2009 at 2:13 AM, Darren Dale <dsdale24 at gmail.com> wrote:

> Thank you for the advice. I've been looking at this most of the day, and I
> think I'm stuck. I have an object I'm trying to navigate:

> It's not clear to me how I would get the object at level_3 and then hook
> into ipython's usual completion chain to filter attributes with leading
> underscores, or the traits completer, for example. Is this possible?

You just need to eval the whole thing, and see what attributes that
eval'ed object has. That's what I did in ipy_greedycompleter. It's
sort of "complete at your own risk" scheme, but it's the best thing
you can get without putting in lots of work (and it will do what you
want anyhow). The only work you need to do is determine how much you
need to eval (here, it's everything before period).


-- 
Ville M. Vainio
http://tinyurl.com/vainio


From dsdale24 at gmail.com  Mon Feb  2 12:39:35 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Mon, 2 Feb 2009 12:39:35 -0500
Subject: [IPython-dev] Useful posts on launchpad/bzr use
In-Reply-To: <db6b5ecc0902011859m66667952kcd71a37531f10c9c@mail.gmail.com>
References: <db6b5ecc0902010015i49df0beakb52ba42d4b171866@mail.gmail.com>
	<db6b5ecc0902011342v15d82d47j9baabedc30c850df@mail.gmail.com>
	<a08e5f80902011628p5a98bdaauc197f6e0dd8c4aa4@mail.gmail.com>
	<db6b5ecc0902011859m66667952kcd71a37531f10c9c@mail.gmail.com>
Message-ID: <a08e5f80902020939v45b150dcqc052064fd146ba68@mail.gmail.com>

On Sun, Feb 1, 2009 at 9:59 PM, Fernando Perez <fperez.net at gmail.com> wrote:

> Hi,
>
> On Sun, Feb 1, 2009 at 4:28 PM, Darren Dale <dsdale24 at gmail.com> wrote:
>
> > Thanks for pointing these out, this was really useful information. I
> ended
> > up signing up for the rss feed.
> >
> > I spent the morning setting things up like he discussed in advanced
> > concepts. Here is my .bazaar/locations.conf, which allows you to
> integrate
> > the ideas he talked about in "making pushing easy"  with the layout in
> > advanced concepts:
> >
> > [/home/darren/Projects/repos]
> > push_location = lp:~dsdale24/
> > push_location:policy = appendpath
> > public_branch = lp:~dsdale24/
> > public_branch:policy = appendpath
> >
> > [/home/darren/Projects]
> > cbranch_target = /home/darren/Projects/repos
> > cbranch_target:policy = appendpath
>
> I read his posts but haven't changed my setup yet, partly because my
> typical layout is a little different.  I tend to have repos on  a
> per-project directory rather than a global one, but I could easily (I
> think) use those ideas with my layout, just by having several
> repo/cbranch entries.
>
> My question to you is: do you see any downside of using that split
> between repos and checkouts at all?  I like that separation between
> the repo storage and the working trees, but before I change things on
> my laptop, I'd like to think through any possible disadvantages.
>
>
I like this layout, and my experience so far has been trouble-free. I
switched to this layout for the two projects I have under bzr/lp revision
control, on my laptop yesterday and on my office computer this morning. I
havent seen any disadvantages yet, and it was really simple to set up the
split layout. If you have reservations, maybe you could try backing up and
splitting just one of your project directories. I might follow your lead and
make a repository for each project at some point.

Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090202/001caa0f/attachment.html>

From fperez.net at gmail.com  Mon Feb  2 13:30:46 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Mon, 2 Feb 2009 10:30:46 -0800
Subject: [IPython-dev] Useful posts on launchpad/bzr use
In-Reply-To: <a08e5f80902020939v45b150dcqc052064fd146ba68@mail.gmail.com>
References: <db6b5ecc0902010015i49df0beakb52ba42d4b171866@mail.gmail.com>
	<db6b5ecc0902011342v15d82d47j9baabedc30c850df@mail.gmail.com>
	<a08e5f80902011628p5a98bdaauc197f6e0dd8c4aa4@mail.gmail.com>
	<db6b5ecc0902011859m66667952kcd71a37531f10c9c@mail.gmail.com>
	<a08e5f80902020939v45b150dcqc052064fd146ba68@mail.gmail.com>
Message-ID: <db6b5ecc0902021030l6e94ca7cj8da2162535dd261@mail.gmail.com>

Hi Darren,

On Mon, Feb 2, 2009 at 9:39 AM, Darren Dale <dsdale24 at gmail.com> wrote:

> I like this layout, and my experience so far has been trouble-free. I
> switched to this layout for the two projects I have under bzr/lp revision
> control, on my laptop yesterday and on my office computer this morning. I
> havent seen any disadvantages yet, and it was really simple to set up the
> split layout. If you have reservations, maybe you could try backing up and
> splitting just one of your project directories. I might follow your lead and
> make a repository for each project at some point.

OK, thanks for your feedback.  Yes, I think what I'll do then is set
things up with one repo per project that I'm seriously involved with
and for which I expect lots of branches, plus one generic 'projects'
repo where I can branch various odds and ends I may occasionally want
to track or use.  Then, from each repo I can use this checkout model,
keeping a separation between the actual repo and the workspace, which
I like.

I'll drop a line in a few weeks if I see any glaring problems with
this setup, but it sounds like a good way to organize things for
bzr/lp work.

Cheers,

f


From dsdale24 at gmail.com  Mon Feb  2 16:06:33 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Mon, 2 Feb 2009 16:06:33 -0500
Subject: [IPython-dev] looking for advice on a custom dict-like completer
In-Reply-To: <46cb515a0902020507n1292714vda7253c503222b81@mail.gmail.com>
References: <a08e5f80901310637n39a6411ei63475c10a0b9a905@mail.gmail.com>
	<db6b5ecc0901310934v3019728dt9590a1e8a9fd82f@mail.gmail.com>
	<a08e5f80901311613k73453b6at95b9e3df51099cef@mail.gmail.com>
	<46cb515a0902020507n1292714vda7253c503222b81@mail.gmail.com>
Message-ID: <a08e5f80902021306l18a33375m3c5ded7d4afc63dd@mail.gmail.com>

On Mon, Feb 2, 2009 at 8:07 AM, Ville M. Vainio <vivainio at gmail.com> wrote:

> On Sun, Feb 1, 2009 at 2:13 AM, Darren Dale <dsdale24 at gmail.com> wrote:
>
> > Thank you for the advice. I've been looking at this most of the day, and
> I
> > think I'm stuck. I have an object I'm trying to navigate:
>
> > It's not clear to me how I would get the object at level_3 and then hook
> > into ipython's usual completion chain to filter attributes with leading
> > underscores, or the traits completer, for example. Is this possible?
>
> You just need to eval the whole thing, and see what attributes that
> eval'ed object has. That's what I did in ipy_greedycompleter. It's
> sort of "complete at your own risk" scheme, but it's the best thing
> you can get without putting in lots of work (and it will do what you
> want anyhow). The only work you need to do is determine how much you
> need to eval (here, it's everything before period).
>
>
Thanks for the suggestion. Once I determine how much to eval, do you know if
it is possible to hook back into the standard completer pipeline to benefit
from things like ipy_traits_completer?

Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090202/2ac08af5/attachment.html>

From dsdale24 at gmail.com  Tue Feb  3 09:03:30 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Tue, 3 Feb 2009 09:03:30 -0500
Subject: [IPython-dev] looking for advice on a custom dict-like completer
In-Reply-To: <a08e5f80902021306l18a33375m3c5ded7d4afc63dd@mail.gmail.com>
References: <a08e5f80901310637n39a6411ei63475c10a0b9a905@mail.gmail.com>
	<db6b5ecc0901310934v3019728dt9590a1e8a9fd82f@mail.gmail.com>
	<a08e5f80901311613k73453b6at95b9e3df51099cef@mail.gmail.com>
	<46cb515a0902020507n1292714vda7253c503222b81@mail.gmail.com>
	<a08e5f80902021306l18a33375m3c5ded7d4afc63dd@mail.gmail.com>
Message-ID: <a08e5f80902030603i6f0fad2aw4b880b88ef489d9d@mail.gmail.com>

On Mon, Feb 2, 2009 at 4:06 PM, Darren Dale <dsdale24 at gmail.com> wrote:

> On Mon, Feb 2, 2009 at 8:07 AM, Ville M. Vainio <vivainio at gmail.com>wrote:
>
>> On Sun, Feb 1, 2009 at 2:13 AM, Darren Dale <dsdale24 at gmail.com> wrote:
>>
>> > Thank you for the advice. I've been looking at this most of the day, and
>> I
>> > think I'm stuck. I have an object I'm trying to navigate:
>>
>> > It's not clear to me how I would get the object at level_3 and then hook
>> > into ipython's usual completion chain to filter attributes with leading
>> > underscores, or the traits completer, for example. Is this possible?
>>
>> You just need to eval the whole thing, and see what attributes that
>> eval'ed object has. That's what I did in ipy_greedycompleter. It's
>> sort of "complete at your own risk" scheme, but it's the best thing
>> you can get without putting in lots of work (and it will do what you
>> want anyhow). The only work you need to do is determine how much you
>> need to eval (here, it's everything before period).
>>
>>
> Thanks for the suggestion. Once I determine how much to eval, do you know
> if it is possible to hook back into the standard completer pipeline to
> benefit from things like ipy_traits_completer?
>

I'd like to make a suggestion for the regular expression in the greedy
completer. Prepending  r"(?:.*\=+)?" should allow you to do things like:

a=b=''.att<tab>

since it won't include a=b= in the match.

I'm making progress on a custom completer for a dict-like object that doesnt
call arbitrary functions (regular expressions are a pain to relearn, but
they are incredibly useful). I'll share what I have as soon as I tie what I
have together.

Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090203/46399217/attachment.html>

From dsdale24 at gmail.com  Tue Feb  3 12:56:14 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Tue, 3 Feb 2009 12:56:14 -0500
Subject: [IPython-dev] looking for advice on a custom dict-like completer
In-Reply-To: <a08e5f80902030603i6f0fad2aw4b880b88ef489d9d@mail.gmail.com>
References: <a08e5f80901310637n39a6411ei63475c10a0b9a905@mail.gmail.com>
	<db6b5ecc0901310934v3019728dt9590a1e8a9fd82f@mail.gmail.com>
	<a08e5f80901311613k73453b6at95b9e3df51099cef@mail.gmail.com>
	<46cb515a0902020507n1292714vda7253c503222b81@mail.gmail.com>
	<a08e5f80902021306l18a33375m3c5ded7d4afc63dd@mail.gmail.com>
	<a08e5f80902030603i6f0fad2aw4b880b88ef489d9d@mail.gmail.com>
Message-ID: <a08e5f80902030956r3d67d17cm1754158db6b95558@mail.gmail.com>

On Tue, Feb 3, 2009 at 9:03 AM, Darren Dale <dsdale24 at gmail.com> wrote:

>
>
> On Mon, Feb 2, 2009 at 4:06 PM, Darren Dale <dsdale24 at gmail.com> wrote:
>
>> On Mon, Feb 2, 2009 at 8:07 AM, Ville M. Vainio <vivainio at gmail.com>wrote:
>>
>>> On Sun, Feb 1, 2009 at 2:13 AM, Darren Dale <dsdale24 at gmail.com> wrote:
>>>
>>> > Thank you for the advice. I've been looking at this most of the day,
>>> and I
>>> > think I'm stuck. I have an object I'm trying to navigate:
>>>
>>> > It's not clear to me how I would get the object at level_3 and then
>>> hook
>>> > into ipython's usual completion chain to filter attributes with leading
>>> > underscores, or the traits completer, for example. Is this possible?
>>>
>>> You just need to eval the whole thing, and see what attributes that
>>> eval'ed object has. That's what I did in ipy_greedycompleter. It's
>>> sort of "complete at your own risk" scheme, but it's the best thing
>>> you can get without putting in lots of work (and it will do what you
>>> want anyhow). The only work you need to do is determine how much you
>>> need to eval (here, it's everything before period).
>>>
>>>
>> Thanks for the suggestion. Once I determine how much to eval, do you know
>> if it is possible to hook back into the standard completer pipeline to
>> benefit from things like ipy_traits_completer?
>>
>
> I'd like to make a suggestion for the regular expression in the greedy
> completer. Prepending  r"(?:.*\=+)?" should allow you to do things like:
>
> a=b=''.att<tab>
>
> since it won't include a=b= in the match.
>
> I'm making progress on a custom completer for a dict-like object that
> doesnt call arbitrary functions (regular expressions are a pain to relearn,
> but they are incredibly useful). I'll share what I have as soon as I tie
> what I have together.
>


I am nearly finished with this project, I'm just having trouble evaluating
doing the following:

d={'test':'ok'}
a={'nested':d}

a['nes<tab>

yields:

a['nested'

I'd like it to yield

a['nested']

but thats a minor point. A major sticking point is:

a['nested'].ite<tab>

That doesnt yield anything, but I know that a list of possible completions
is being generated, I can print the list in my completer, but its not
propagating back to ipython for some reason. Could someone please have a
look? Its easy to try it, just drop it in your ~/.ipython directory and add
this to ~/.ipython/ipy_user_conf.main:

from ipy_h5py_completer import activate
activate()

I know it says h5py, but it works on plain old dictionaries for now.

Thanks,
Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090203/3583acca/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ipy_h5py_completer.py
Type: text/x-python
Size: 3207 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090203/3583acca/attachment.py>

From vivainio at gmail.com  Tue Feb  3 13:29:18 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Tue, 3 Feb 2009 20:29:18 +0200
Subject: [IPython-dev] looking for advice on a custom dict-like completer
In-Reply-To: <a08e5f80902030956r3d67d17cm1754158db6b95558@mail.gmail.com>
References: <a08e5f80901310637n39a6411ei63475c10a0b9a905@mail.gmail.com>
	<db6b5ecc0901310934v3019728dt9590a1e8a9fd82f@mail.gmail.com>
	<a08e5f80901311613k73453b6at95b9e3df51099cef@mail.gmail.com>
	<46cb515a0902020507n1292714vda7253c503222b81@mail.gmail.com>
	<a08e5f80902021306l18a33375m3c5ded7d4afc63dd@mail.gmail.com>
	<a08e5f80902030603i6f0fad2aw4b880b88ef489d9d@mail.gmail.com>
	<a08e5f80902030956r3d67d17cm1754158db6b95558@mail.gmail.com>
Message-ID: <46cb515a0902031029x299a74d5tb0085ab3b8901902@mail.gmail.com>

On Tue, Feb 3, 2009 at 7:56 PM, Darren Dale <dsdale24 at gmail.com> wrote:

> but thats a minor point. A major sticking point is:
>
> a['nested'].ite<tab>
>
> That doesnt yield anything, but I know that a list of possible completions
> is being generated, I can print the list in my completer, but its not
> propagating back to ipython for some reason. Could someone please have a

This happens because readline completer delimiters.

Try doing:

readline.set_completer_delims(' ')

then your completer will work properly.

-- 
Ville M. Vainio
http://tinyurl.com/vainio


From vivainio at gmail.com  Tue Feb  3 13:31:46 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Tue, 3 Feb 2009 20:31:46 +0200
Subject: [IPython-dev] looking for advice on a custom dict-like completer
In-Reply-To: <46cb515a0902031029x299a74d5tb0085ab3b8901902@mail.gmail.com>
References: <a08e5f80901310637n39a6411ei63475c10a0b9a905@mail.gmail.com>
	<db6b5ecc0901310934v3019728dt9590a1e8a9fd82f@mail.gmail.com>
	<a08e5f80901311613k73453b6at95b9e3df51099cef@mail.gmail.com>
	<46cb515a0902020507n1292714vda7253c503222b81@mail.gmail.com>
	<a08e5f80902021306l18a33375m3c5ded7d4afc63dd@mail.gmail.com>
	<a08e5f80902030603i6f0fad2aw4b880b88ef489d9d@mail.gmail.com>
	<a08e5f80902030956r3d67d17cm1754158db6b95558@mail.gmail.com>
	<46cb515a0902031029x299a74d5tb0085ab3b8901902@mail.gmail.com>
Message-ID: <46cb515a0902031031m6b00a4b2h5ea0a16f3ac3e20a@mail.gmail.com>

On Tue, Feb 3, 2009 at 8:29 PM, Ville M. Vainio <vivainio at gmail.com> wrote:

> This happens because readline completer delimiters.

Because OF ...

I think it may be time to get rid of that mess that the completer
delimiters are... but it would require extensive testing on all
platform.

And, it will screw up mac stuff, obviously ;-)

-- 
Ville M. Vainio
http://tinyurl.com/vainio


From dsdale24 at gmail.com  Tue Feb  3 14:25:52 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Tue, 3 Feb 2009 14:25:52 -0500
Subject: [IPython-dev] looking for advice on a custom dict-like completer
In-Reply-To: <46cb515a0902031031m6b00a4b2h5ea0a16f3ac3e20a@mail.gmail.com>
References: <a08e5f80901310637n39a6411ei63475c10a0b9a905@mail.gmail.com>
	<db6b5ecc0901310934v3019728dt9590a1e8a9fd82f@mail.gmail.com>
	<a08e5f80901311613k73453b6at95b9e3df51099cef@mail.gmail.com>
	<46cb515a0902020507n1292714vda7253c503222b81@mail.gmail.com>
	<a08e5f80902021306l18a33375m3c5ded7d4afc63dd@mail.gmail.com>
	<a08e5f80902030603i6f0fad2aw4b880b88ef489d9d@mail.gmail.com>
	<a08e5f80902030956r3d67d17cm1754158db6b95558@mail.gmail.com>
	<46cb515a0902031029x299a74d5tb0085ab3b8901902@mail.gmail.com>
	<46cb515a0902031031m6b00a4b2h5ea0a16f3ac3e20a@mail.gmail.com>
Message-ID: <a08e5f80902031125l33b24355r90e9d464fd667dc@mail.gmail.com>

On Tue, Feb 3, 2009 at 1:31 PM, Ville M. Vainio <vivainio at gmail.com> wrote:

> On Tue, Feb 3, 2009 at 8:29 PM, Ville M. Vainio <vivainio at gmail.com>
> wrote:
>
> > This happens because readline completer delimiters.
>
> Because OF ...
>
> I think it may be time to get rid of that mess that the completer
> delimiters are... but it would require extensive testing on all
> platform.
>
> And, it will screw up mac stuff, obviously ;-)
>

I put this at the end of my activate():

    import readline
    readline.set_completer_delims(" ")


but it looks like that is not the right place to put it. It makes attribute
completion work, but it breaks item completion. Where should it go?

Thanks,
Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090203/22c6883b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ipy_h5py_completer.py
Type: text/x-python
Size: 3167 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090203/22c6883b/attachment.py>

From vivainio at gmail.com  Tue Feb  3 14:43:20 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Tue, 3 Feb 2009 21:43:20 +0200
Subject: [IPython-dev] looking for advice on a custom dict-like completer
In-Reply-To: <a08e5f80902031125l33b24355r90e9d464fd667dc@mail.gmail.com>
References: <a08e5f80901310637n39a6411ei63475c10a0b9a905@mail.gmail.com>
	<db6b5ecc0901310934v3019728dt9590a1e8a9fd82f@mail.gmail.com>
	<a08e5f80901311613k73453b6at95b9e3df51099cef@mail.gmail.com>
	<46cb515a0902020507n1292714vda7253c503222b81@mail.gmail.com>
	<a08e5f80902021306l18a33375m3c5ded7d4afc63dd@mail.gmail.com>
	<a08e5f80902030603i6f0fad2aw4b880b88ef489d9d@mail.gmail.com>
	<a08e5f80902030956r3d67d17cm1754158db6b95558@mail.gmail.com>
	<46cb515a0902031029x299a74d5tb0085ab3b8901902@mail.gmail.com>
	<46cb515a0902031031m6b00a4b2h5ea0a16f3ac3e20a@mail.gmail.com>
	<a08e5f80902031125l33b24355r90e9d464fd667dc@mail.gmail.com>
Message-ID: <46cb515a0902031143nb2f32d6pd2f34c6728dd407@mail.gmail.com>

On Tue, Feb 3, 2009 at 9:25 PM, Darren Dale <dsdale24 at gmail.com> wrote:

> I put this at the end of my activate():
>
>     import readline
>     readline.set_completer_delims(" ")
>
>
> but it looks like that is not the right place to put it. It makes attribute
> completion work, but it breaks item completion. Where should it go?

Actually, it was the right place to put it. I tried the same and it
broke item completion for me as well. Apparently, you need ' in your
completer_delims for item completion, and it breaks attribute
completion. Oh joy.

You might want to try finding some kind of compromise from:

- magic completer_delims that will work
- returning something apart from the whole strings. E.g. the string
after the ' character. This kind of hackery was necessary for
completion of file names with spaces in them.
- Setting completer delims in your completer (!?)

-- 
Ville M. Vainio
http://tinyurl.com/vainio


From dsdale24 at gmail.com  Tue Feb  3 15:59:39 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Tue, 3 Feb 2009 15:59:39 -0500
Subject: [IPython-dev] looking for advice on a custom dict-like completer
In-Reply-To: <46cb515a0902031143nb2f32d6pd2f34c6728dd407@mail.gmail.com>
References: <a08e5f80901310637n39a6411ei63475c10a0b9a905@mail.gmail.com>
	<a08e5f80901311613k73453b6at95b9e3df51099cef@mail.gmail.com>
	<46cb515a0902020507n1292714vda7253c503222b81@mail.gmail.com>
	<a08e5f80902021306l18a33375m3c5ded7d4afc63dd@mail.gmail.com>
	<a08e5f80902030603i6f0fad2aw4b880b88ef489d9d@mail.gmail.com>
	<a08e5f80902030956r3d67d17cm1754158db6b95558@mail.gmail.com>
	<46cb515a0902031029x299a74d5tb0085ab3b8901902@mail.gmail.com>
	<46cb515a0902031031m6b00a4b2h5ea0a16f3ac3e20a@mail.gmail.com>
	<a08e5f80902031125l33b24355r90e9d464fd667dc@mail.gmail.com>
	<46cb515a0902031143nb2f32d6pd2f34c6728dd407@mail.gmail.com>
Message-ID: <a08e5f80902031259k2de3decer9aa93bcc2e2eb169@mail.gmail.com>

On Tue, Feb 3, 2009 at 2:43 PM, Ville M. Vainio <vivainio at gmail.com> wrote:

> On Tue, Feb 3, 2009 at 9:25 PM, Darren Dale <dsdale24 at gmail.com> wrote:
>
> > I put this at the end of my activate():
> >
> >     import readline
> >     readline.set_completer_delims(" ")
> >
> >
> > but it looks like that is not the right place to put it. It makes
> attribute
> > completion work, but it breaks item completion. Where should it go?
>
> Actually, it was the right place to put it. I tried the same and it
> broke item completion for me as well. Apparently, you need ' in your
> completer_delims for item completion, and it breaks attribute
> completion. Oh joy.
>
> You might want to try finding some kind of compromise from:
>
> - magic completer_delims that will work


completer delims are perfectly mystical to me, I might as will be readiing
linear A.

>
> - returning something apart from the whole strings. E.g. the string
> after the ' character. This kind of hackery was necessary for
> completion of file names with spaces in them.


I think I am only returning the string after the ' character.


>
> - Setting completer delims in your completer (!?)
>

I didn't think this could work, since there is no opportunity to reset the
delims back to the ipython defaults. But I tried it, and it works ok,
sometimes you need to hit tab twice to get a list of possible completions.
Not bad. It would be nice to find a better solution to the delims issue but
I think I'm out of my depth.

I'm attaching what seems to be a working copy. It only works for string
keys, thats all h5py expects and I didnt have time to try to make it more
general.

Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090203/eaf63592/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ipy_dict_completer.py
Type: text/x-python
Size: 2668 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090203/eaf63592/attachment.py>

From dsdale24 at gmail.com  Tue Feb  3 16:00:20 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Tue, 3 Feb 2009 16:00:20 -0500
Subject: [IPython-dev] looking for advice on a custom dict-like completer
In-Reply-To: <a08e5f80902031259k2de3decer9aa93bcc2e2eb169@mail.gmail.com>
References: <a08e5f80901310637n39a6411ei63475c10a0b9a905@mail.gmail.com>
	<46cb515a0902020507n1292714vda7253c503222b81@mail.gmail.com>
	<a08e5f80902021306l18a33375m3c5ded7d4afc63dd@mail.gmail.com>
	<a08e5f80902030603i6f0fad2aw4b880b88ef489d9d@mail.gmail.com>
	<a08e5f80902030956r3d67d17cm1754158db6b95558@mail.gmail.com>
	<46cb515a0902031029x299a74d5tb0085ab3b8901902@mail.gmail.com>
	<46cb515a0902031031m6b00a4b2h5ea0a16f3ac3e20a@mail.gmail.com>
	<a08e5f80902031125l33b24355r90e9d464fd667dc@mail.gmail.com>
	<46cb515a0902031143nb2f32d6pd2f34c6728dd407@mail.gmail.com>
	<a08e5f80902031259k2de3decer9aa93bcc2e2eb169@mail.gmail.com>
Message-ID: <a08e5f80902031300u903bbdcs5fe53bb39937e6a4@mail.gmail.com>

I forgot the most important part. Thank you Ville and Fernando for your
help.

Darren

On Tue, Feb 3, 2009 at 3:59 PM, Darren Dale <dsdale24 at gmail.com> wrote:

>
>
> On Tue, Feb 3, 2009 at 2:43 PM, Ville M. Vainio <vivainio at gmail.com>wrote:
>
>> On Tue, Feb 3, 2009 at 9:25 PM, Darren Dale <dsdale24 at gmail.com> wrote:
>>
>> > I put this at the end of my activate():
>> >
>> >     import readline
>> >     readline.set_completer_delims(" ")
>> >
>> >
>> > but it looks like that is not the right place to put it. It makes
>> attribute
>> > completion work, but it breaks item completion. Where should it go?
>>
>> Actually, it was the right place to put it. I tried the same and it
>> broke item completion for me as well. Apparently, you need ' in your
>> completer_delims for item completion, and it breaks attribute
>> completion. Oh joy.
>>
>> You might want to try finding some kind of compromise from:
>>
>> - magic completer_delims that will work
>
>
> completer delims are perfectly mystical to me, I might as will be readiing
> linear A.
>
>>
>> - returning something apart from the whole strings. E.g. the string
>> after the ' character. This kind of hackery was necessary for
>> completion of file names with spaces in them.
>
>
> I think I am only returning the string after the ' character.
>
>
>>
>> - Setting completer delims in your completer (!?)
>>
>
> I didn't think this could work, since there is no opportunity to reset the
> delims back to the ipython defaults. But I tried it, and it works ok,
> sometimes you need to hit tab twice to get a list of possible completions.
> Not bad. It would be nice to find a better solution to the delims issue but
> I think I'm out of my depth.
>
> I'm attaching what seems to be a working copy. It only works for string
> keys, thats all h5py expects and I didnt have time to try to make it more
> general.
>
> Darren
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090203/1f37082d/attachment.html>

From dsdale24 at gmail.com  Tue Feb  3 16:13:07 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Tue, 3 Feb 2009 16:13:07 -0500
Subject: [IPython-dev] looking for advice on a custom dict-like completer
In-Reply-To: <a08e5f80902031300u903bbdcs5fe53bb39937e6a4@mail.gmail.com>
References: <a08e5f80901310637n39a6411ei63475c10a0b9a905@mail.gmail.com>
	<a08e5f80902021306l18a33375m3c5ded7d4afc63dd@mail.gmail.com>
	<a08e5f80902030603i6f0fad2aw4b880b88ef489d9d@mail.gmail.com>
	<a08e5f80902030956r3d67d17cm1754158db6b95558@mail.gmail.com>
	<46cb515a0902031029x299a74d5tb0085ab3b8901902@mail.gmail.com>
	<46cb515a0902031031m6b00a4b2h5ea0a16f3ac3e20a@mail.gmail.com>
	<a08e5f80902031125l33b24355r90e9d464fd667dc@mail.gmail.com>
	<46cb515a0902031143nb2f32d6pd2f34c6728dd407@mail.gmail.com>
	<a08e5f80902031259k2de3decer9aa93bcc2e2eb169@mail.gmail.com>
	<a08e5f80902031300u903bbdcs5fe53bb39937e6a4@mail.gmail.com>
Message-ID: <a08e5f80902031313s5e18bf44vbf208484c873db3b@mail.gmail.com>

And here is the h5py completer.

On Tue, Feb 3, 2009 at 4:00 PM, Darren Dale <dsdale24 at gmail.com> wrote:

> I forgot the most important part. Thank you Ville and Fernando for your
> help.
>
> Darren
>
>
> On Tue, Feb 3, 2009 at 3:59 PM, Darren Dale <dsdale24 at gmail.com> wrote:
>
>>
>>
>> On Tue, Feb 3, 2009 at 2:43 PM, Ville M. Vainio <vivainio at gmail.com>wrote:
>>
>>> On Tue, Feb 3, 2009 at 9:25 PM, Darren Dale <dsdale24 at gmail.com> wrote:
>>>
>>> > I put this at the end of my activate():
>>> >
>>> >     import readline
>>> >     readline.set_completer_delims(" ")
>>> >
>>> >
>>> > but it looks like that is not the right place to put it. It makes
>>> attribute
>>> > completion work, but it breaks item completion. Where should it go?
>>>
>>> Actually, it was the right place to put it. I tried the same and it
>>> broke item completion for me as well. Apparently, you need ' in your
>>> completer_delims for item completion, and it breaks attribute
>>> completion. Oh joy.
>>>
>>> You might want to try finding some kind of compromise from:
>>>
>>> - magic completer_delims that will work
>>
>>
>> completer delims are perfectly mystical to me, I might as will be readiing
>> linear A.
>>
>>>
>>> - returning something apart from the whole strings. E.g. the string
>>> after the ' character. This kind of hackery was necessary for
>>> completion of file names with spaces in them.
>>
>>
>> I think I am only returning the string after the ' character.
>>
>>
>>>
>>> - Setting completer delims in your completer (!?)
>>>
>>
>> I didn't think this could work, since there is no opportunity to reset the
>> delims back to the ipython defaults. But I tried it, and it works ok,
>> sometimes you need to hit tab twice to get a list of possible completions.
>> Not bad. It would be nice to find a better solution to the delims issue but
>> I think I'm out of my depth.
>>
>> I'm attaching what seems to be a working copy. It only works for string
>> keys, thats all h5py expects and I didnt have time to try to make it more
>> general.
>>
>> Darren
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090203/1835c8ee/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ipy_h5py_completer.py
Type: text/x-python
Size: 2796 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090203/1835c8ee/attachment.py>

From dsdale24 at gmail.com  Tue Feb  3 18:25:43 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Tue, 3 Feb 2009 18:25:43 -0500
Subject: [IPython-dev] looking for advice on a custom dict-like completer
In-Reply-To: <a08e5f80902031313s5e18bf44vbf208484c873db3b@mail.gmail.com>
References: <a08e5f80901310637n39a6411ei63475c10a0b9a905@mail.gmail.com>
	<a08e5f80902030603i6f0fad2aw4b880b88ef489d9d@mail.gmail.com>
	<a08e5f80902030956r3d67d17cm1754158db6b95558@mail.gmail.com>
	<46cb515a0902031029x299a74d5tb0085ab3b8901902@mail.gmail.com>
	<46cb515a0902031031m6b00a4b2h5ea0a16f3ac3e20a@mail.gmail.com>
	<a08e5f80902031125l33b24355r90e9d464fd667dc@mail.gmail.com>
	<46cb515a0902031143nb2f32d6pd2f34c6728dd407@mail.gmail.com>
	<a08e5f80902031259k2de3decer9aa93bcc2e2eb169@mail.gmail.com>
	<a08e5f80902031300u903bbdcs5fe53bb39937e6a4@mail.gmail.com>
	<a08e5f80902031313s5e18bf44vbf208484c873db3b@mail.gmail.com>
Message-ID: <a08e5f80902031525t34bf83d0ydbe74d5f826baadc@mail.gmail.com>

I found and corrected a few glitches, here are some improved completers.

On Tue, Feb 3, 2009 at 4:13 PM, Darren Dale <dsdale24 at gmail.com> wrote:

> And here is the h5py completer.
>
>
> On Tue, Feb 3, 2009 at 4:00 PM, Darren Dale <dsdale24 at gmail.com> wrote:
>
>> I forgot the most important part. Thank you Ville and Fernando for your
>> help.
>>
>> Darren
>>
>>
>> On Tue, Feb 3, 2009 at 3:59 PM, Darren Dale <dsdale24 at gmail.com> wrote:
>>
>>>
>>>
>>> On Tue, Feb 3, 2009 at 2:43 PM, Ville M. Vainio <vivainio at gmail.com>wrote:
>>>
>>>> On Tue, Feb 3, 2009 at 9:25 PM, Darren Dale <dsdale24 at gmail.com> wrote:
>>>>
>>>> > I put this at the end of my activate():
>>>> >
>>>> >     import readline
>>>> >     readline.set_completer_delims(" ")
>>>> >
>>>> >
>>>> > but it looks like that is not the right place to put it. It makes
>>>> attribute
>>>> > completion work, but it breaks item completion. Where should it go?
>>>>
>>>> Actually, it was the right place to put it. I tried the same and it
>>>> broke item completion for me as well. Apparently, you need ' in your
>>>> completer_delims for item completion, and it breaks attribute
>>>> completion. Oh joy.
>>>>
>>>> You might want to try finding some kind of compromise from:
>>>>
>>>> - magic completer_delims that will work
>>>
>>>
>>> completer delims are perfectly mystical to me, I might as will be
>>> readiing linear A.
>>>
>>>>
>>>> - returning something apart from the whole strings. E.g. the string
>>>> after the ' character. This kind of hackery was necessary for
>>>> completion of file names with spaces in them.
>>>
>>>
>>> I think I am only returning the string after the ' character.
>>>
>>>
>>>>
>>>> - Setting completer delims in your completer (!?)
>>>>
>>>
>>> I didn't think this could work, since there is no opportunity to reset
>>> the delims back to the ipython defaults. But I tried it, and it works ok,
>>> sometimes you need to hit tab twice to get a list of possible completions.
>>> Not bad. It would be nice to find a better solution to the delims issue but
>>> I think I'm out of my depth.
>>>
>>> I'm attaching what seems to be a working copy. It only works for string
>>> keys, thats all h5py expects and I didnt have time to try to make it more
>>> general.
>>>
>>> Darren
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090203/e03c204b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ipy_dict_completer.py
Type: text/x-python
Size: 2667 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090203/e03c204b/attachment.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ipy_h5py_completer.py
Type: text/x-python
Size: 2940 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090203/e03c204b/attachment-0001.py>

From p.c.degroot at tudelft.nl  Tue Feb  3 18:45:37 2009
From: p.c.degroot at tudelft.nl (Pieter Cristiaan de Groot)
Date: Wed, 04 Feb 2009 00:45:37 +0100
Subject: [IPython-dev] ctrl-c bug using -gthread on windows,
	with possible solution
Message-ID: <4988D721.3060301@tudelft.nl>

Dear all,

running ipython 0.9.1 on windows using -gthread gives problems using 
ctrl-c. It gives an IOError when ctrl-c is pushed during the sleep(0.01) 
(File: Shell.py line 835-843), otherwise everyghing goes ok. This does 
not happen on a linux machine.

Here's the code:

    def on_timer(self):
        """Called when GTK is idle.
            Must return True always, otherwise GTK stops calling it"""
  
        update_tk(self.tk)
        self.IP.runcode()
        time.sleep(0.01)
        return True

Replacing the bottom part with this seems to solve it:

        try:
            update_tk(self.tk)
            self.IP.runcode()
            time.sleep(0.01)
            return True
        except IOError, e:
            return True



I hope this helps,

Pieter


From p.c.degroot at tudelft.nl  Tue Feb  3 18:47:59 2009
From: p.c.degroot at tudelft.nl (Pieter Cristiaan de Groot)
Date: Wed, 04 Feb 2009 00:47:59 +0100
Subject: [IPython-dev] ctrl-c bug using -gthread on windows,
 with possible solution
In-Reply-To: <4988D721.3060301@tudelft.nl>
References: <4988D721.3060301@tudelft.nl>
Message-ID: <4988D7AF.9090809@tudelft.nl>

What I forgot to mention is that because of this IOError, threading dies.

Pieter

Pieter Cristiaan de Groot wrote:
> Dear all,
>
> running ipython 0.9.1 on windows using -gthread gives problems using 
> ctrl-c. It gives an IOError when ctrl-c is pushed during the sleep(0.01) 
> (File: Shell.py line 835-843), otherwise everyghing goes ok. This does 
> not happen on a linux machine.
>
> Here's the code:
>
>     def on_timer(self):
>         """Called when GTK is idle.
>             Must return True always, otherwise GTK stops calling it"""
>   
>         update_tk(self.tk)
>         self.IP.runcode()
>         time.sleep(0.01)
>         return True
>
> Replacing the bottom part with this seems to solve it:
>
>         try:
>             update_tk(self.tk)
>             self.IP.runcode()
>             time.sleep(0.01)
>             return True
>         except IOError, e:
>             return True
>
>
>
> I hope this helps,
>
> Pieter
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
>   


From vivainio at gmail.com  Wed Feb  4 11:25:03 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Wed, 4 Feb 2009 18:25:03 +0200
Subject: [IPython-dev] looking for advice on a custom dict-like completer
In-Reply-To: <a08e5f80902031525t34bf83d0ydbe74d5f826baadc@mail.gmail.com>
References: <a08e5f80901310637n39a6411ei63475c10a0b9a905@mail.gmail.com>
	<a08e5f80902030956r3d67d17cm1754158db6b95558@mail.gmail.com>
	<46cb515a0902031029x299a74d5tb0085ab3b8901902@mail.gmail.com>
	<46cb515a0902031031m6b00a4b2h5ea0a16f3ac3e20a@mail.gmail.com>
	<a08e5f80902031125l33b24355r90e9d464fd667dc@mail.gmail.com>
	<46cb515a0902031143nb2f32d6pd2f34c6728dd407@mail.gmail.com>
	<a08e5f80902031259k2de3decer9aa93bcc2e2eb169@mail.gmail.com>
	<a08e5f80902031300u903bbdcs5fe53bb39937e6a4@mail.gmail.com>
	<a08e5f80902031313s5e18bf44vbf208484c873db3b@mail.gmail.com>
	<a08e5f80902031525t34bf83d0ydbe74d5f826baadc@mail.gmail.com>
Message-ID: <46cb515a0902040825h50e12fcat271c6ea8a93b6c36@mail.gmail.com>

On Wed, Feb 4, 2009 at 1:25 AM, Darren Dale <dsdale24 at gmail.com> wrote:

> I found and corrected a few glitches, here are some improved completers.

Mind if I add these to ipython-extensions package?

https://code.launchpad.net/~ipython-dev/ipython/ipython-extensions

-- 
Ville M. Vainio
http://tinyurl.com/vainio


From dsdale24 at gmail.com  Wed Feb  4 11:52:11 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Wed, 4 Feb 2009 11:52:11 -0500
Subject: [IPython-dev] looking for advice on a custom dict-like completer
In-Reply-To: <46cb515a0902040825h50e12fcat271c6ea8a93b6c36@mail.gmail.com>
References: <a08e5f80901310637n39a6411ei63475c10a0b9a905@mail.gmail.com>
	<46cb515a0902031029x299a74d5tb0085ab3b8901902@mail.gmail.com>
	<46cb515a0902031031m6b00a4b2h5ea0a16f3ac3e20a@mail.gmail.com>
	<a08e5f80902031125l33b24355r90e9d464fd667dc@mail.gmail.com>
	<46cb515a0902031143nb2f32d6pd2f34c6728dd407@mail.gmail.com>
	<a08e5f80902031259k2de3decer9aa93bcc2e2eb169@mail.gmail.com>
	<a08e5f80902031300u903bbdcs5fe53bb39937e6a4@mail.gmail.com>
	<a08e5f80902031313s5e18bf44vbf208484c873db3b@mail.gmail.com>
	<a08e5f80902031525t34bf83d0ydbe74d5f826baadc@mail.gmail.com>
	<46cb515a0902040825h50e12fcat271c6ea8a93b6c36@mail.gmail.com>
Message-ID: <a08e5f80902040852o30f54782gc7a8008b406334b@mail.gmail.com>

On Wed, Feb 4, 2009 at 11:25 AM, Ville M. Vainio <vivainio at gmail.com> wrote:

> On Wed, Feb 4, 2009 at 1:25 AM, Darren Dale <dsdale24 at gmail.com> wrote:
>
> > I found and corrected a few glitches, here are some improved completers.
>
> Mind if I add these to ipython-extensions package?
>
> https://code.launchpad.net/~ipython-dev/ipython/ipython-extensions<https://code.launchpad.net/%7Eipython-dev/ipython/ipython-extensions>
>

I'm not familiar with this package, what is it, like a staging area or a
home for code with licenses that are incompatible with BSD? I'll release
these under the python or bsd license, but I would prefer to wait a couple
days on the h5py completer to continue testing it. This morning I added a
feature at the h5py author's request, hw is considering including it in his
next release. The dict completer still needs some work, it punts when you
try to item-complete a dictionary containing keys that are not strings.

Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090204/26fbe808/attachment.html>

From vivainio at gmail.com  Wed Feb  4 12:41:09 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Wed, 4 Feb 2009 19:41:09 +0200
Subject: [IPython-dev] looking for advice on a custom dict-like completer
In-Reply-To: <a08e5f80902040852o30f54782gc7a8008b406334b@mail.gmail.com>
References: <a08e5f80901310637n39a6411ei63475c10a0b9a905@mail.gmail.com>
	<46cb515a0902031031m6b00a4b2h5ea0a16f3ac3e20a@mail.gmail.com>
	<a08e5f80902031125l33b24355r90e9d464fd667dc@mail.gmail.com>
	<46cb515a0902031143nb2f32d6pd2f34c6728dd407@mail.gmail.com>
	<a08e5f80902031259k2de3decer9aa93bcc2e2eb169@mail.gmail.com>
	<a08e5f80902031300u903bbdcs5fe53bb39937e6a4@mail.gmail.com>
	<a08e5f80902031313s5e18bf44vbf208484c873db3b@mail.gmail.com>
	<a08e5f80902031525t34bf83d0ydbe74d5f826baadc@mail.gmail.com>
	<46cb515a0902040825h50e12fcat271c6ea8a93b6c36@mail.gmail.com>
	<a08e5f80902040852o30f54782gc7a8008b406334b@mail.gmail.com>
Message-ID: <46cb515a0902040941y5447e7cdm7444733acee80571@mail.gmail.com>

On Wed, Feb 4, 2009 at 6:52 PM, Darren Dale <dsdale24 at gmail.com> wrote:

> I'm not familiar with this package, what is it, like a staging area or a
> home for code with licenses that are incompatible with BSD? I'll release
> these under the python or bsd license, but I would prefer to wait a couple
> days on the h5py completer to continue testing it. This morning I added a
> feature at the h5py author's request, hw is considering including it in his
> next release. The dict completer still needs some work, it punts when you
> try to item-complete a dictionary containing keys that are not strings.

There is a description at:

http://pypi.python.org/pypi/ipython-extensions/0.2

I like to think of it as a place to put extensions that are not
coupled to ipython release cycle - think of it as 'contrib' folder.
In addition to allowing independent release cycle, it can be faster to
put stuff 'in-development' extensions there, since there are no
centralized quality requirements (no unit tests needed) - code is
released "with the hope that it will be useful".

It's also easier to tell users to pull a new version
ipython-extensions than to ask them to upgrade their ipython
installation (which may come from their linux distribution).

-- 
Ville M. Vainio
http://tinyurl.com/vainio


From fperez.net at gmail.com  Fri Feb  6 03:10:00 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Fri, 6 Feb 2009 00:10:00 -0800
Subject: [IPython-dev] looking for advice on a custom dict-like completer
In-Reply-To: <46cb515a0902040941y5447e7cdm7444733acee80571@mail.gmail.com>
References: <a08e5f80901310637n39a6411ei63475c10a0b9a905@mail.gmail.com>
	<a08e5f80902031125l33b24355r90e9d464fd667dc@mail.gmail.com>
	<46cb515a0902031143nb2f32d6pd2f34c6728dd407@mail.gmail.com>
	<a08e5f80902031259k2de3decer9aa93bcc2e2eb169@mail.gmail.com>
	<a08e5f80902031300u903bbdcs5fe53bb39937e6a4@mail.gmail.com>
	<a08e5f80902031313s5e18bf44vbf208484c873db3b@mail.gmail.com>
	<a08e5f80902031525t34bf83d0ydbe74d5f826baadc@mail.gmail.com>
	<46cb515a0902040825h50e12fcat271c6ea8a93b6c36@mail.gmail.com>
	<a08e5f80902040852o30f54782gc7a8008b406334b@mail.gmail.com>
	<46cb515a0902040941y5447e7cdm7444733acee80571@mail.gmail.com>
Message-ID: <db6b5ecc0902060010h496c67a6n5bb5c3da2939d2fb@mail.gmail.com>

On Wed, Feb 4, 2009 at 9:41 AM, Ville M. Vainio <vivainio at gmail.com> wrote:

> I like to think of it as a place to put extensions that are not
> coupled to ipython release cycle - think of it as 'contrib' folder.
> In addition to allowing independent release cycle, it can be faster to
> put stuff 'in-development' extensions there, since there are no
> centralized quality requirements (no unit tests needed) - code is
> released "with the hope that it will be useful".
>
> It's also easier to tell users to pull a new version
> ipython-extensions than to ask them to upgrade their ipython
> installation (which may come from their linux distribution).

It's worth noting that if Darren is willing to add tests/docs for it,
I'd be happy to have it in ipython proper.  But we are trying to be
serious about not committing any code without docs and tests into the
core, so in the end it's your choice, Darren, of where to send it.

Cheers,

f


From p.c.degroot at tudelft.nl  Fri Feb  6 05:05:54 2009
From: p.c.degroot at tudelft.nl (Pieter Cristiaan de Groot)
Date: Fri, 06 Feb 2009 11:05:54 +0100
Subject: [IPython-dev] ctrl-c bug using -gthread on windows,
 with possible solution
In-Reply-To: <4988D7AF.9090809@tudelft.nl>
References: <4988D721.3060301@tudelft.nl> <4988D7AF.9090809@tudelft.nl>
Message-ID: <498C0B82.2030404@tudelft.nl>

Hi all,

any comments on the ctrl-c issue?
Or did I post to the wrong list?

Pieter

Pieter Cristiaan de Groot wrote:
> What I forgot to mention is that because of this IOError, threading dies.
>
> Pieter
>
> Pieter Cristiaan de Groot wrote:
>   
>> Dear all,
>>
>> running ipython 0.9.1 on windows using -gthread gives problems using 
>> ctrl-c. It gives an IOError when ctrl-c is pushed during the sleep(0.01) 
>> (File: Shell.py line 835-843), otherwise everyghing goes ok. This does 
>> not happen on a linux machine.
>>
>> Here's the code:
>>
>>     def on_timer(self):
>>         """Called when GTK is idle.
>>             Must return True always, otherwise GTK stops calling it"""
>>   
>>         update_tk(self.tk)
>>         self.IP.runcode()
>>         time.sleep(0.01)
>>         return True
>>
>> Replacing the bottom part with this seems to solve it:
>>
>>         try:
>>             update_tk(self.tk)
>>             self.IP.runcode()
>>             time.sleep(0.01)
>>             return True
>>         except IOError, e:
>>             return True
>>
>>
>>
>> I hope this helps,
>>
>> Pieter
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at scipy.org
>> http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
>>   
>>     
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
>   


From dsdale24 at gmail.com  Fri Feb  6 10:21:01 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Fri, 6 Feb 2009 10:21:01 -0500
Subject: [IPython-dev] looking for advice on a custom dict-like completer
In-Reply-To: <db6b5ecc0902060010h496c67a6n5bb5c3da2939d2fb@mail.gmail.com>
References: <a08e5f80901310637n39a6411ei63475c10a0b9a905@mail.gmail.com>
	<46cb515a0902031143nb2f32d6pd2f34c6728dd407@mail.gmail.com>
	<a08e5f80902031259k2de3decer9aa93bcc2e2eb169@mail.gmail.com>
	<a08e5f80902031300u903bbdcs5fe53bb39937e6a4@mail.gmail.com>
	<a08e5f80902031313s5e18bf44vbf208484c873db3b@mail.gmail.com>
	<a08e5f80902031525t34bf83d0ydbe74d5f826baadc@mail.gmail.com>
	<46cb515a0902040825h50e12fcat271c6ea8a93b6c36@mail.gmail.com>
	<a08e5f80902040852o30f54782gc7a8008b406334b@mail.gmail.com>
	<46cb515a0902040941y5447e7cdm7444733acee80571@mail.gmail.com>
	<db6b5ecc0902060010h496c67a6n5bb5c3da2939d2fb@mail.gmail.com>
Message-ID: <a08e5f80902060721v5db9c643qe3a55059af390de4@mail.gmail.com>

On Fri, Feb 6, 2009 at 3:10 AM, Fernando Perez <fperez.net at gmail.com> wrote:

> On Wed, Feb 4, 2009 at 9:41 AM, Ville M. Vainio <vivainio at gmail.com>
> wrote:
>
> > I like to think of it as a place to put extensions that are not
> > coupled to ipython release cycle - think of it as 'contrib' folder.
> > In addition to allowing independent release cycle, it can be faster to
> > put stuff 'in-development' extensions there, since there are no
> > centralized quality requirements (no unit tests needed) - code is
> > released "with the hope that it will be useful".
> >
> > It's also easier to tell users to pull a new version
> > ipython-extensions than to ask them to upgrade their ipython
> > installation (which may come from their linux distribution).
>
> It's worth noting that if Darren is willing to add tests/docs for it,
> I'd be happy to have it in ipython proper.  But we are trying to be
> serious about not committing any code without docs and tests into the
> core, so in the end it's your choice, Darren, of where to send it.
>

I submitted the h5py completer to the author of h5py, he is considering
including it in h5py-1.1. I think it might make more sense for it to be
distributed as a part of h5py. Do you think ipython is a better home for it?

I hadn't really intended to continue work on the ipy_dict_completer, I was
only using it as a stepping stone, but it might be a good candidate for
inclusion in ipython at some point. It was easy enough to implement the
items completer for h5py since it only accepts strings for the keys. For a
regular dict that can use any hashable object as a key, it should be
possible to check the user namespace for possible completions, and then
filter that list based on whether those objects appear as keys in the dict
being completed. Although, I can imagine entanglements like:

dict_a = {11:'eleven'}
dict_b= {12: dict_a}

dict_a[dict_b[dict_<tab>

which might be simple enough to do with recursive calls, but I've never
gotten an answer on whether or not it is possible to hook back into the
standard ipython completion pipeline from within a completer. Plus, I really
need to use the time I have available to work on my physical quantities
package, which is nearly ready to submit to numpy-discussion for comments. I
think I'd rather submit the dict completer to ipython-extensions for now, in
case anyone would like to put the finishing touches on it.

Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090206/e7e187e4/attachment.html>

From vivainio at gmail.com  Fri Feb  6 14:03:47 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Fri, 6 Feb 2009 21:03:47 +0200
Subject: [IPython-dev] looking for advice on a custom dict-like completer
In-Reply-To: <a08e5f80902060721v5db9c643qe3a55059af390de4@mail.gmail.com>
References: <a08e5f80901310637n39a6411ei63475c10a0b9a905@mail.gmail.com>
	<a08e5f80902031259k2de3decer9aa93bcc2e2eb169@mail.gmail.com>
	<a08e5f80902031300u903bbdcs5fe53bb39937e6a4@mail.gmail.com>
	<a08e5f80902031313s5e18bf44vbf208484c873db3b@mail.gmail.com>
	<a08e5f80902031525t34bf83d0ydbe74d5f826baadc@mail.gmail.com>
	<46cb515a0902040825h50e12fcat271c6ea8a93b6c36@mail.gmail.com>
	<a08e5f80902040852o30f54782gc7a8008b406334b@mail.gmail.com>
	<46cb515a0902040941y5447e7cdm7444733acee80571@mail.gmail.com>
	<db6b5ecc0902060010h496c67a6n5bb5c3da2939d2fb@mail.gmail.com>
	<a08e5f80902060721v5db9c643qe3a55059af390de4@mail.gmail.com>
Message-ID: <46cb515a0902061103xc69540el2cfe3565386956cc@mail.gmail.com>

On Fri, Feb 6, 2009 at 5:21 PM, Darren Dale <dsdale24 at gmail.com> wrote:

> I submitted the h5py completer to the author of h5py, he is considering
> including it in h5py-1.1. I think it might make more sense for it to be
> distributed as a part of h5py. Do you think ipython is a better home for it?

That's true, it's always the best to ship such things in the actual
library it supports if possible.

> inclusion in ipython at some point. It was easy enough to implement the
> items completer for h5py since it only accepts strings for the keys. For a

String keys are the most important use case for a dict completer anyway.

> regular dict that can use any hashable object as a key, it should be
> possible to check the user namespace for possible completions, and then
> filter that list based on whether those objects appear as keys in the dict
> being completed. Although, I can imagine entanglements like:

Sounds a bit too involved & agressive for basic ipython usage ;-). I
can mostly envision support for strings and ints - anything that
doesn't work for those should just use normal completions that work
with all names available in the user_ns.

> package, which is nearly ready to submit to numpy-discussion for comments. I
> think I'd rather submit the dict completer to ipython-extensions for now, in
> case anyone would like to put the finishing touches on it.

Yeah, do that. It's better to get it out in the wild than let it rot
in mail attachments and personal sandboxes.

-- 
Ville M. Vainio
http://tinyurl.com/vainio


From ellisonbg.net at gmail.com  Sat Feb  7 11:00:39 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Sat, 7 Feb 2009 08:00:39 -0800
Subject: [IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook
Message-ID: <6ce0ac130902070800s16e7365bp9bde1277f06b973f@mail.gmail.com>

Hello all,

I am trying to understand the current situation of how IPython
integrates with GUI event loops.  I am going to be doing some work on
the core soon, so I want to make sure I understand all of the
subtleties.  From looking at things, here is my take:

1. The IPython Core and any GUI event loop must be run in the same
thread and that thread must be the main thread if IPython is to offer
interactive GUI support.

2. PyOS_InputHook allows a function to be registered that will be
called periodically (by readline) while input is being entered at the
command line. This opens the door for GUI event loops to continue
running while a command line IPython is waiting for input. This is
only relevant for terminal based IPython shells though.  From what I
can tell, only Tk and recent releases of GTK and Qt4 support this
though.

Does Wx support this?

If not, can we implement this ourselves?

I tried to use PyOS_InputHook this morning from ctypes, but it didn't
work.  Has anyone figured this out? Is PyOS_InputHook relevent for
non-teminal based IPython frontends?

In other words: can we get rid of the messy threading code in
IPython?!!! (please say yes)

3. SIgnals.  I know that one of the most subtle aspect of IPython is
getting signals working in the multithreaded shells.  How does this
change if we move to using PyOS_InputHook rather than our current GUI
shells?

4. There is an IPython commit by Darren Dale saying that the
PyOS_InputHook approach wasn't working fully, so it was disabled.  Did
we get to the bottom of this?

Cheers,

Brian


From vivainio at gmail.com  Sat Feb  7 11:38:45 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Sat, 7 Feb 2009 18:38:45 +0200
Subject: [IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook
In-Reply-To: <6ce0ac130902070800s16e7365bp9bde1277f06b973f@mail.gmail.com>
References: <6ce0ac130902070800s16e7365bp9bde1277f06b973f@mail.gmail.com>
Message-ID: <46cb515a0902070838i68ba9e2cp24a4e1bf45303ea5@mail.gmail.com>

On Sat, Feb 7, 2009 at 6:00 PM, Brian Granger <ellisonbg.net at gmail.com> wrote:

> 1. The IPython Core and any GUI event loop must be run in the same
> thread and that thread must be the main thread if IPython is to offer
> interactive GUI support.

Depends what you mean by "IPython core". The important thing is that
every line entered by the user gets run in the gui mainloop (which is
what the current threaded shells do).

> 2. PyOS_InputHook allows a function to be registered that will be
> called periodically (by readline) while input is being entered at the
> command line. This opens the door for GUI event loops to continue
> running while a command line IPython is waiting for input. This is
> only relevant for terminal based IPython shells though.

Exactly.

> Does Wx support this?

I don't know - probably not. On linux, wx is based on gtk so with
future gtk it could work.

> If not, can we implement this ourselves?

See above ;-).

> work.  Has anyone figured this out? Is PyOS_InputHook relevent for
> non-teminal based IPython frontends?

It's not relevant at all. Since you control the event loop, there is
no "input blocking", so the hack that PyOS_InputHook is can be
ignored.

> In other words: can we get rid of the messy threading code in
> IPython?!!! (please say yes)

Definitely yes, we can (and it's long overdue).

> 3. SIgnals.  I know that one of the most subtle aspect of IPython is
> getting signals working in the multithreaded shells.  How does this
> change if we move to using PyOS_InputHook rather than our current GUI
> shells?

We just have to disable the crash handler, the event loop typically
deals with ctrl+C itself.

> 4. There is an IPython commit by Darren Dale saying that the
> PyOS_InputHook approach wasn't working fully, so it was disabled.  Did
> we get to the bottom of this?

Wasn't it about pdb?

-- 
Ville M. Vainio
http://tinyurl.com/vainio


From dsdale24 at gmail.com  Sat Feb  7 11:48:07 2009
From: dsdale24 at gmail.com (Darren Dale)
Date: Sat, 7 Feb 2009 11:48:07 -0500
Subject: [IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook
In-Reply-To: <6ce0ac130902070800s16e7365bp9bde1277f06b973f@mail.gmail.com>
References: <6ce0ac130902070800s16e7365bp9bde1277f06b973f@mail.gmail.com>
Message-ID: <a08e5f80902070848h46fc835fta658bbe25d4baf3c@mail.gmail.com>

On Sat, Feb 7, 2009 at 11:00 AM, Brian Granger <ellisonbg.net at gmail.com>wrote:

> Hello all,
>
> I am trying to understand the current situation of how IPython
> integrates with GUI event loops.  I am going to be doing some work on
> the core soon, so I want to make sure I understand all of the
> subtleties.  From looking at things, here is my take:
>
> 1. The IPython Core and any GUI event loop must be run in the same
> thread and that thread must be the main thread if IPython is to offer
> interactive GUI support.
>
> 2. PyOS_InputHook allows a function to be registered that will be
> called periodically (by readline) while input is being entered at the
> command line. This opens the door for GUI event loops to continue
> running while a command line IPython is waiting for input. This is
> only relevant for terminal based IPython shells though.  From what I
> can tell, only Tk and recent releases of GTK and Qt4 support this
> though.
>
> Does Wx support this?
>
> If not, can we implement this ourselves?
>
> I tried to use PyOS_InputHook this morning from ctypes, but it didn't
> work.  Has anyone figured this out? Is PyOS_InputHook relevent for
> non-teminal based IPython frontends?
>
> In other words: can we get rid of the messy threading code in
> IPython?!!! (please say yes)
>
> 3. SIgnals.  I know that one of the most subtle aspect of IPython is
> getting signals working in the multithreaded shells.  How does this
> change if we move to using PyOS_InputHook rather than our current GUI
> shells?
>
> 4. There is an IPython commit by Darren Dale saying that the
> PyOS_InputHook approach wasn't working fully, so it was disabled.  Did
> we get to the bottom of this?
>

I think you are referring to a comment from 2008-02-07 in
docs/attic/Changelog, is that right? IPython's Qt threading support is
implemented the same way as the other gui toolkits, using IPython's
infrastructure, which was conflicting with Qt's use of PyOS_InputHook. I
dont remember all the details, there was some discussion on the ipython
mailing list around the the time of that commit, it boiled down to starting
ipython with -q4thread, and attempting to "run qt4_foo.py" which started the
main loop by calling QApplication.exec_(), which blocked. Phil Thompson
suggested the call to remove pyqt4's use of the input hook. I don't think we
have ever tried relying solely on pyqt4's built in support for interactive
interpreters.

Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090207/77b31b0a/attachment.html>

From vivainio at gmail.com  Sat Feb  7 11:53:34 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Sat, 7 Feb 2009 18:53:34 +0200
Subject: [IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook
In-Reply-To: <a08e5f80902070848h46fc835fta658bbe25d4baf3c@mail.gmail.com>
References: <6ce0ac130902070800s16e7365bp9bde1277f06b973f@mail.gmail.com>
	<a08e5f80902070848h46fc835fta658bbe25d4baf3c@mail.gmail.com>
Message-ID: <46cb515a0902070853g4505319bm483b7133ed76951f@mail.gmail.com>

On Sat, Feb 7, 2009 at 6:48 PM, Darren Dale <dsdale24 at gmail.com> wrote:

> suggested the call to remove pyqt4's use of the input hook. I don't think we
> have ever tried relying solely on pyqt4's built in support for interactive
> interpreters.

I've tried it and it worked:

http://ipython0.wordpress.com/2008/05/15/embedding-ipython-in-gui-apps-is-trivial/

-- 
Ville M. Vainio
http://tinyurl.com/vainio


From gael.varoquaux at normalesup.org  Sat Feb  7 12:11:33 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Sat, 7 Feb 2009 18:11:33 +0100
Subject: [IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook
In-Reply-To: <6ce0ac130902070800s16e7365bp9bde1277f06b973f@mail.gmail.com>
References: <6ce0ac130902070800s16e7365bp9bde1277f06b973f@mail.gmail.com>
Message-ID: <20090207171133.GC24193@phare.normalesup.org>

On Sat, Feb 07, 2009 at 08:00:39AM -0800, Brian Granger wrote:
> 1. The IPython Core and any GUI event loop must be run in the same
> thread and that thread must be the main thread if IPython is to offer
> interactive GUI support.

Correct.

> 2. PyOS_InputHook allows a function to be registered that will be
> called periodically (by readline) while input is being entered at the
> command line. This opens the door for GUI event loops to continue
> running while a command line IPython is waiting for input. This is
> only relevant for terminal based IPython shells though.  From what I
> can tell, only Tk and recent releases of GTK and Qt4 support this
> though.

> Does Wx support this?

I don't know.

> If not, can we implement this ourselves?

Don't know either.

> In other words: can we get rid of the messy threading code in
> IPython?!!! (please say yes)

It would be really great. A common case of crashing for me is
tab-completing on a object with a property, and this property making a
call in the event loop.

I am not sure I can help you, however :(.

Ga?l


From vivainio at gmail.com  Sat Feb  7 12:34:25 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Sat, 7 Feb 2009 19:34:25 +0200
Subject: [IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook
In-Reply-To: <20090207171133.GC24193@phare.normalesup.org>
References: <6ce0ac130902070800s16e7365bp9bde1277f06b973f@mail.gmail.com>
	<20090207171133.GC24193@phare.normalesup.org>
Message-ID: <46cb515a0902070934j4883503ehb060928d80aede96@mail.gmail.com>

On Sat, Feb 7, 2009 at 7:11 PM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:

> It would be really great. A common case of crashing for me is
> tab-completing on a object with a property, and this property making a
> call in the event loop.

This should be fixable - just ask the mainloop thread to do the
completion. Perhaps call runsource('__compl = _ip.complete(.... )')

Of course stuff like this is a crutch compared to getting rid of the
threads in the first place.

-- 
Ville M. Vainio
http://tinyurl.com/vainio


From ellisonbg.net at gmail.com  Sat Feb  7 12:51:45 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Sat, 7 Feb 2009 09:51:45 -0800
Subject: [IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook
In-Reply-To: <a08e5f80902070848h46fc835fta658bbe25d4baf3c@mail.gmail.com>
References: <6ce0ac130902070800s16e7365bp9bde1277f06b973f@mail.gmail.com>
	<a08e5f80902070848h46fc835fta658bbe25d4baf3c@mail.gmail.com>
Message-ID: <6ce0ac130902070951j716fc1an5b5bfb71b705a7a@mail.gmail.com>

> I think you are referring to a comment from 2008-02-07 in
> docs/attic/Changelog, is that right? IPython's Qt threading support is
> implemented the same way as the other gui toolkits, using IPython's
> infrastructure, which was conflicting with Qt's use of PyOS_InputHook. I
> dont remember all the details, there was some discussion on the ipython
> mailing list around the the time of that commit, it boiled down to starting
> ipython with -q4thread, and attempting to "run qt4_foo.py" which started the
> main loop by calling QApplication.exec_(), which blocked. Phil Thompson
> suggested the call to remove pyqt4's use of the input hook. I don't think we
> have ever tried relying solely on pyqt4's built in support for interactive
> interpreters.

Ah, OK this makes sense, especially because the default for PyQT4 is
to set the inputhook.


> Darren
>


From ellisonbg.net at gmail.com  Sat Feb  7 12:58:32 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Sat, 7 Feb 2009 09:58:32 -0800
Subject: [IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook
In-Reply-To: <46cb515a0902070838i68ba9e2cp24a4e1bf45303ea5@mail.gmail.com>
References: <6ce0ac130902070800s16e7365bp9bde1277f06b973f@mail.gmail.com>
	<46cb515a0902070838i68ba9e2cp24a4e1bf45303ea5@mail.gmail.com>
Message-ID: <6ce0ac130902070958k53bec6d1m43c076b058db3e1b@mail.gmail.com>

>> If not, can we implement this ourselves?
>
> See above ;-).

I don't see where "above" you talk about this.  For what it is worth,
here is what I tried:

>>> import readline
>>> import ctypes
>>> ctypes.pythonapi.PyOS_InputHook
<_FuncPtr object at 0x76420>
>>> def my_callback():
...   print "In am here"
...   return 0
...
>>> cbf = ctypes.CFUNCTYPE(ctypes.c_int)(my_callback)
>>> cbf
<CFunctionType object at 0x769d0>
>>> ctypes.pythonapi.PyOS_InputHook = cbf

If I understand the python source code correctly, readline should now
be calling my callback, but it doesn't seem to.  Am I misunderstanding
how this works?  It doesn't sound like wx has current support for
PyOS_InputHook, but if we could implement it ourselves using ctypes,
that would solve the problem.

>> work.  Has anyone figured this out? Is PyOS_InputHook relevent for
>> non-teminal based IPython frontends?
>
> It's not relevant at all. Since you control the event loop, there is
> no "input blocking", so the hack that PyOS_InputHook is can be
> ignored.

OK ,at least I am thinking about this correctly.  So, if we had GUI
based IPython's only, we would *never* have to worry about this :-)

>> In other words: can we get rid of the messy threading code in
>> IPython?!!! (please say yes)
>
> Definitely yes, we can (and it's long overdue).

Great!

>> 3. SIgnals.  I know that one of the most subtle aspect of IPython is
>> getting signals working in the multithreaded shells.  How does this
>> change if we move to using PyOS_InputHook rather than our current GUI
>> shells?
>
> We just have to disable the crash handler, the event loop typically
> deals with ctrl+C itself.

Again, sounds good.

>> 4. There is an IPython commit by Darren Dale saying that the
>> PyOS_InputHook approach wasn't working fully, so it was disabled.  Did
>> we get to the bottom of this?
>
> Wasn't it about pdb?

I hope not.

Brian

> --
> Ville M. Vainio
> http://tinyurl.com/vainio
>


From ellisonbg.net at gmail.com  Sat Feb  7 13:00:17 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Sat, 7 Feb 2009 10:00:17 -0800
Subject: [IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook
In-Reply-To: <20090207171133.GC24193@phare.normalesup.org>
References: <6ce0ac130902070800s16e7365bp9bde1277f06b973f@mail.gmail.com>
	<20090207171133.GC24193@phare.normalesup.org>
Message-ID: <6ce0ac130902071000o6de94a83y181bec34757b517e@mail.gmail.com>

> It would be really great. A common case of crashing for me is
> tab-completing on a object with a property, and this property making a
> call in the event loop.

Does this happen in a terminal based ipython -wthread setting or in
your wx ipython GUI?

If it happens in your wx GUI, can you clarify how you are using
threads, etc?  The reason that I ask is that you should be able to get
away with not using threads.

Cheers,

Brian



> I am not sure I can help you, however :(.
>
> Ga?l
>


From gael.varoquaux at normalesup.org  Sat Feb  7 13:04:10 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Sat, 7 Feb 2009 19:04:10 +0100
Subject: [IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook
In-Reply-To: <6ce0ac130902071000o6de94a83y181bec34757b517e@mail.gmail.com>
References: <6ce0ac130902070800s16e7365bp9bde1277f06b973f@mail.gmail.com>
	<20090207171133.GC24193@phare.normalesup.org>
	<6ce0ac130902071000o6de94a83y181bec34757b517e@mail.gmail.com>
Message-ID: <20090207180410.GC22215@phare.normalesup.org>

On Sat, Feb 07, 2009 at 10:00:17AM -0800, Brian Granger wrote:
> > It would be really great. A common case of crashing for me is
> > tab-completing on a object with a property, and this property making a
> > call in the event loop.

> Does this happen in a terminal based ipython -wthread setting or in
> your wx ipython GUI?

In a terminal-based "ipython -wthread"

> If it happens in your wx GUI, can you clarify how you are using
> threads, etc?  The reason that I ask is that you should be able to get
> away with not using threads.

I am not using any thread in the wx GUI. This was a very important design
choice, IMHO. I am fiddling a lot with the mainloop to keep as much a
possible the terminal reactive when a computation is going on, but there
are no threads (except when a subprocess is used, but I was very careful
to make them play well with the mainloop).

Ga?l


From ellisonbg.net at gmail.com  Sat Feb  7 13:09:05 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Sat, 7 Feb 2009 10:09:05 -0800
Subject: [IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook
In-Reply-To: <20090207180410.GC22215@phare.normalesup.org>
References: <6ce0ac130902070800s16e7365bp9bde1277f06b973f@mail.gmail.com>
	<20090207171133.GC24193@phare.normalesup.org>
	<6ce0ac130902071000o6de94a83y181bec34757b517e@mail.gmail.com>
	<20090207180410.GC22215@phare.normalesup.org>
Message-ID: <6ce0ac130902071009i592a68bal59dcaab44819f01d@mail.gmail.com>

> I am not using any thread in the wx GUI. This was a very important design
> choice, IMHO. I am fiddling a lot with the mainloop to keep as much a
> possible the terminal reactive when a computation is going on, but there
> are no threads (except when a subprocess is used, but I was very careful
> to make them play well with the mainloop).

Because we are likely moving in this direction, I want to understand
this better.  What do you mean by "fiddling with the mainloop?"  Is
this "fiddling" optional?  If it causes bugs in your GUI (is this what
you are saying?) why do you do it?  You imply that the terminal was
not responsive until you did this extra stuff.  Can you say a bit
more?  Hopefully, this will enable us to better support the usage case
that you have!

Cheers,

Brian

> Ga?l
>


From vivainio at gmail.com  Sat Feb  7 13:37:12 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Sat, 7 Feb 2009 20:37:12 +0200
Subject: [IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook
In-Reply-To: <6ce0ac130902070958k53bec6d1m43c076b058db3e1b@mail.gmail.com>
References: <6ce0ac130902070800s16e7365bp9bde1277f06b973f@mail.gmail.com>
	<46cb515a0902070838i68ba9e2cp24a4e1bf45303ea5@mail.gmail.com>
	<6ce0ac130902070958k53bec6d1m43c076b058db3e1b@mail.gmail.com>
Message-ID: <46cb515a0902071037x9ffa7e8o36254b377c4840a2@mail.gmail.com>

On Sat, Feb 7, 2009 at 7:58 PM, Brian Granger <ellisonbg.net at gmail.com> wrote:

>>> If not, can we implement this ourselves?
>>
>> See above ;-).
>
> I don't see where "above" you talk about this.  For what it is worth,
> here is what I tried:

I was talking about the fact that wx is based on gtk on linux, so it
might get it "for free".

>>>> import readline
>>>> import ctypes
>>>> ctypes.pythonapi.PyOS_InputHook
> <_FuncPtr object at 0x76420>
>>>> def my_callback():
> ...   print "In am here"
> ...   return 0
> ...
>>>> cbf = ctypes.CFUNCTYPE(ctypes.c_int)(my_callback)
>>>> cbf
> <CFunctionType object at 0x769d0>
>>>> ctypes.pythonapi.PyOS_InputHook = cbf
>
> If I understand the python source code correctly, readline should now
> be calling my callback, but it doesn't seem to.  Am I misunderstanding

Perhaps trying to change the value of static C variable may be a bit
much for ctypes?

If so, perhaps a python extension whose sole purpose would be the
ability to set the input hook would be in order. Perhaps one of the
"ipython-friendly" projects (numpy?) that have C side stuff could
sneak it into their next release.. ;-).

Though really, wx should get it for free with new gtk in the future,
and for windows wx it should be just a matter of contributing a patch.

-- 
Ville M. Vainio
http://tinyurl.com/vainio


From ellisonbg.net at gmail.com  Sat Feb  7 13:58:03 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Sat, 7 Feb 2009 10:58:03 -0800
Subject: [IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook
In-Reply-To: <46cb515a0902071037x9ffa7e8o36254b377c4840a2@mail.gmail.com>
References: <6ce0ac130902070800s16e7365bp9bde1277f06b973f@mail.gmail.com>
	<46cb515a0902070838i68ba9e2cp24a4e1bf45303ea5@mail.gmail.com>
	<6ce0ac130902070958k53bec6d1m43c076b058db3e1b@mail.gmail.com>
	<46cb515a0902071037x9ffa7e8o36254b377c4840a2@mail.gmail.com>
Message-ID: <6ce0ac130902071058j4ccc772fr7aaf0faddb191eac@mail.gmail.com>

>> If I understand the python source code correctly, readline should now
>> be calling my callback, but it doesn't seem to.  Am I misunderstanding
>
> Perhaps trying to change the value of static C variable may be a bit
> much for ctypes?

That is my thought.

> If so, perhaps a python extension whose sole purpose would be the
> ability to set the input hook would be in order. Perhaps one of the
> "ipython-friendly" projects (numpy?) that have C side stuff could
> sneak it into their next release.. ;-).

Or we could simply create a simple Cython extension to do this.  I
will give this a shot next.

> Though really, wx should get it for free with new gtk in the future,
> and for windows wx it should be just a matter of contributing a patch.

Here's for hoping that we just get it for free.  But, the big thing is
that this overall approach is *very* worth our pursuing, even if it
means we contribute some upstream patches.

Cheers,

Brian

> --
> Ville M. Vainio
> http://tinyurl.com/vainio
>


From ellisonbg.net at gmail.com  Sat Feb  7 17:42:12 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Sat, 7 Feb 2009 14:42:12 -0800
Subject: [IPython-dev] Cython based PyOS_InputHook hack
Message-ID: <6ce0ac130902071442g710c0743of3432d4cb378f6c2@mail.gmail.com>

So,

I have figured out how to call PyOS_InputHook from Cython code.  It is
pretty simple.  If you have the latest Cython, just drop the attached
files in a directory, open up IPython and do the commands listed
below.

My hack doesn't do anything with a GUI - it simply shows 1) what
exactly PyOS_InputHook does and 2) Shows how to call it from Cython.

I think the only reason we need to think about this is to get wx
working in the PyOS_InputHook mode.  Ville thought that because wx is
based on gtk on Linux we don't need to worry about this.  But, that
leaves Win32 and OS X out in the dark.  Also, I think the
PyOS_InputHook logic is in the Python bindings (pygtk), which I don't
think wx uses.

So, in order for this to really work, someone who knows wx well
(Gael?, someone at Enthought?) has to create a real PyOS_InputHook
that does the right things for wx.   The basic idea is that the hook
needs to watch for stdin to be active and then run the event loop.
The best place to look for ideas is:

* In the pyqt source code - just search for PyOS_InputHook
* In the _tkinter.c source code of Python itself.

Here is what you type to run my example
===============================

In [1]: import pyximport     # This Cython package compiles
inputhook.pyx on the fly!!!

In [2]: pyximport.install()

In [3]: import inputhook

In [4]: inputhook.count
Out[4]: 0

In [5]: inputhook.set_input_hook()

In [6]: inputhook.count
Out[6]: 27

In [7]: inputhook.count
Out[7]: 37

In [8]: inputhook.count
Out[8]: 47

In [9]: inputhook.count
Out[9]: 70

In [10]: inputhook.clear_input_hook()

In [11]: inputhook.count
Out[11]: 124

In [12]: inputhook.count
Out[12]: 124
-------------- next part --------------
#include <Python.h>

static void set_input_hook_c(int (*f)(void))
{
    PyOS_InputHook = f;
}

static void clear_input_hook_c(void)
{
    PyOS_InputHook = NULL;
}
-------------- next part --------------
A non-text attachment was scrubbed...
Name: inputhook.pyx
Type: application/octet-stream
Size: 723 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090207/d9bfc691/attachment.obj>

From gael.varoquaux at normalesup.org  Sun Feb  8 05:22:00 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Sun, 8 Feb 2009 11:22:00 +0100
Subject: [IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook
In-Reply-To: <6ce0ac130902071009i592a68bal59dcaab44819f01d@mail.gmail.com>
References: <6ce0ac130902070800s16e7365bp9bde1277f06b973f@mail.gmail.com>
	<20090207171133.GC24193@phare.normalesup.org>
	<6ce0ac130902071000o6de94a83y181bec34757b517e@mail.gmail.com>
	<20090207180410.GC22215@phare.normalesup.org>
	<6ce0ac130902071009i592a68bal59dcaab44819f01d@mail.gmail.com>
Message-ID: <20090208102200.GA14469@phare.normalesup.org>

On Sat, Feb 07, 2009 at 10:09:05AM -0800, Brian Granger wrote:
> > I am not using any thread in the wx GUI. This was a very important design
> > choice, IMHO. I am fiddling a lot with the mainloop to keep as much a
> > possible the terminal reactive when a computation is going on, but there
> > are no threads (except when a subprocess is used, but I was very careful
> > to make them play well with the mainloop).

> Because we are likely moving in this direction, I want to understand
> this better.  What do you mean by "fiddling with the mainloop?"  Is
> this "fiddling" optional?  If it causes bugs in your GUI (is this what
> you are saying?) why do you do it?  You imply that the terminal was
> not responsive until you did this extra stuff.  Can you say a bit
> more?

Sorry, I was out.

When I say fiddling with the event loop, I mean many things. The big deal
is that you cannot get a refresh, or a key event without running the
event loop. The user is executing code, and executing it in the event
loop, bocking it, as we are going non-multithreaded, but it is
imperative, sequential code, and we really need event-driven code that
yields to the event-loop in order to get refreshes and events. So I
overloaded a few operations, such as print and raw_input to yield back to
the event loop. Now I couldn't simply use wx.Yield blindly, because it is
fragile, and getting this right was a bit tricky. 

For screen refresh, you want to refresh ASAP, because you don't want
debug statements of a long running calculation to pop up only after it
finish. So I had to trigger controlled refresh events to do refreshes on
prints. See method 'write', line 135 of
http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/frontend/wx/console_widget.py
This method is injected in a sys.stdout and sys.stderr replacement. I had
to be careful not to get a refresh called when we where already in a
wx.Yield. This is why I had to add the refresh keyword, and use it
cleverly. In addition, printing a lot of text to the terminal would slow
down a lot, because a refresh is expensive. This is why I introduced the
notion of a delayed refresh: there is a timer, and a queue. Trying to
refresh triggers the timer, and the refresh happens delayed, so that the
queue fills in. Obviously this means that if you are unlucky and the
write method does not get called for a long time while you where waiting
for the timer to empty, you don't get a refresh, because you don't enter
the event loop again.

I had to be careful with all other possible interactions, such as
keyboard callbacks, not to trigger the Yield. Also, I had to write an
OS-level output redirection that would weave the captured flux for
compiled code with the Python flux, and call the refresh callback ASAP,
this is done in
http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/kernel/core/redirector_output_trap.py

For the raw_input, I had to start a second event loop, to process
keyboard event while the main event loop was blocked by the user code
executing. See method 'raw_input', line 176 of
http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/frontend/wx/wx_frontend.py
There is readline-like logic that is used here to determine when this
second event loop is terminated and the user code can resume.

Finally, the trickiest part was running subprocesses. The reason being
that I needed both synchronous prints, and key-event processing. The
subprocess in run in a separate thread, see
http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/frontend/_process/pipedprocess.py
Its input stream is fed in by a raw_input-like mechanism, with a separate
event-loop running, see method system_call, line 197 of
http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/frontend/wx/wx_frontend.py
. I have to use a separate event-loop because the call to a subprocess is
blocking for the user code. The output stream is polled (yeah, I am not
proud of the polling, I couldn't figure out something better), and fed to
a 'buffered_write' method. No need to do a wx.Yield, here, as an
event-loop is running in a separate thread. Which just need to reduce the
number of write called per unit of time to avoid flooding the event loop,
and to make sure we are thread-safe, see method 'buffered_write', line
264 of
http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/frontend/wx/wx_frontend.py
The termination of the subprocess is wired to a callback which flushes
the buffer and terminates the inner event loop.

I believe that this is mostly it. There is a lot more of little subtle
things you have to be careful with: not calling your callbacks in
infinite loops, making sure methods that can be called in other thread
are thread safe, making sure you don't screw the order of events. This is
quite tricky code, I believe. The good news is that all event loops are
quite similar (the Tk one is different, in Python, as it runs in a
separate OS thread, but we can forget that, and use it like the others).
It would be possible to build an object that would abstract scintilla and
the event-loop control out of different GUI toolkits, and expose a common
interface. This would actually be fairly easy and would get us an
interactive frontend for all toolkits.

Ga?l


From gael.varoquaux at normalesup.org  Sun Feb  8 05:28:35 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Sun, 8 Feb 2009 11:28:35 +0100
Subject: [IPython-dev] Cython based PyOS_InputHook hack
In-Reply-To: <6ce0ac130902071442g710c0743of3432d4cb378f6c2@mail.gmail.com>
References: <6ce0ac130902071442g710c0743of3432d4cb378f6c2@mail.gmail.com>
Message-ID: <20090208102835.GB14469@phare.normalesup.org>

On Sat, Feb 07, 2009 at 02:42:12PM -0800, Brian Granger wrote:
> So, in order for this to really work, someone who knows wx well
> (Gael?, someone at Enthought?)

Sorry, Brain, I don't know wx well-enough to do this without a
significant time investement (these things are tricky). I don't see myself
finding this time soon, as my coding time is taken by Mayavi and
NeuroImaging-related stuff.

I can put this on the pile of stuff to do. I believe I can give it a try
in a few months. Maybe not before begining of July. I hate to be that
pessimistic, but I need to free some time where I can focus and think for
this, I believe.

Hopefuly someone else will volunteer. Robert Kern knows wx much better
than I do :).

Ga?l


From vivainio at gmail.com  Sun Feb  8 06:59:25 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Sun, 8 Feb 2009 13:59:25 +0200
Subject: [IPython-dev] Cython based PyOS_InputHook hack
In-Reply-To: <20090208102835.GB14469@phare.normalesup.org>
References: <6ce0ac130902071442g710c0743of3432d4cb378f6c2@mail.gmail.com>
	<20090208102835.GB14469@phare.normalesup.org>
Message-ID: <46cb515a0902080359w72ea78c9x1a5f5efbdf75e922@mail.gmail.com>

On Sun, Feb 8, 2009 at 12:28 PM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:

> Sorry, Brain, I don't know wx well-enough to do this without a
> significant time investement (these things are tricky). I don't see myself

Inferring from your other mail, it would only seem necessary to call
the mainloops Yield in the input hook function.

-- 
Ville M. Vainio
http://tinyurl.com/vainio


From vivainio at gmail.com  Sun Feb  8 07:05:41 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Sun, 8 Feb 2009 14:05:41 +0200
Subject: [IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook
In-Reply-To: <20090208102200.GA14469@phare.normalesup.org>
References: <6ce0ac130902070800s16e7365bp9bde1277f06b973f@mail.gmail.com>
	<20090207171133.GC24193@phare.normalesup.org>
	<6ce0ac130902071000o6de94a83y181bec34757b517e@mail.gmail.com>
	<20090207180410.GC22215@phare.normalesup.org>
	<6ce0ac130902071009i592a68bal59dcaab44819f01d@mail.gmail.com>
	<20090208102200.GA14469@phare.normalesup.org>
Message-ID: <46cb515a0902080405t51a619c0ged894d328b8cd8f5@mail.gmail.com>

On Sun, Feb 8, 2009 at 12:22 PM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:

> Finally, the trickiest part was running subprocesses. The reason being
> that I needed both synchronous prints, and key-event processing. The

In qt, dealing with processes is luckily very simple; you can just use QProcess.

http://doc.trolltech.com/4.3/qprocess.html

It allows you to just react to the following signals:

void readyReadStandardError ()
void readyReadStandardOutput ()

"QProcess allows you to treat a process as a sequential I/O device.
You can write to and read from the process just as you would access a
network connection using QTcpSocket. You can then write to the
process's standard input by calling write(), and read the standard
output by calling read(), readLine(), and getChar()."

-- 
Ville M. Vainio
http://tinyurl.com/vainio


From gael.varoquaux at normalesup.org  Sun Feb  8 07:07:16 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Sun, 8 Feb 2009 13:07:16 +0100
Subject: [IPython-dev] Cython based PyOS_InputHook hack
In-Reply-To: <46cb515a0902080359w72ea78c9x1a5f5efbdf75e922@mail.gmail.com>
References: <6ce0ac130902071442g710c0743of3432d4cb378f6c2@mail.gmail.com>
	<20090208102835.GB14469@phare.normalesup.org>
	<46cb515a0902080359w72ea78c9x1a5f5efbdf75e922@mail.gmail.com>
Message-ID: <20090208120716.GD14469@phare.normalesup.org>

On Sun, Feb 08, 2009 at 01:59:25PM +0200, Ville M. Vainio wrote:
> On Sun, Feb 8, 2009 at 12:28 PM, Gael Varoquaux
> <gael.varoquaux at normalesup.org> wrote:

> > Sorry, Brain, I don't know wx well-enough to do this without a
	    ^
	Sorry, Brian, this was a typo.
> > significant time investement (these things are tricky). I don't see myself

> Inferring from your other mail, it would only seem necessary to call
> the mainloops Yield in the input hook function.

I believe this is very risky. Yield is a very fragile function. For
instance, under Windows, calling a Yield inside a Yield is not possible.
So if you have a GUI callback that calls a raw_input, you are likely to
end up in this situation (recursive call of Yield) when plugging readline
in the GUI as currently done with "ipython -wthread".

I am not saying this is not the way to go. I am saying it needs thinking
and validation.

Ga?l


From gael.varoquaux at normalesup.org  Sun Feb  8 07:10:06 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Sun, 8 Feb 2009 13:10:06 +0100
Subject: [IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook
In-Reply-To: <46cb515a0902080405t51a619c0ged894d328b8cd8f5@mail.gmail.com>
References: <6ce0ac130902070800s16e7365bp9bde1277f06b973f@mail.gmail.com>
	<20090207171133.GC24193@phare.normalesup.org>
	<6ce0ac130902071000o6de94a83y181bec34757b517e@mail.gmail.com>
	<20090207180410.GC22215@phare.normalesup.org>
	<6ce0ac130902071009i592a68bal59dcaab44819f01d@mail.gmail.com>
	<20090208102200.GA14469@phare.normalesup.org>
	<46cb515a0902080405t51a619c0ged894d328b8cd8f5@mail.gmail.com>
Message-ID: <20090208121006.GE14469@phare.normalesup.org>

On Sun, Feb 08, 2009 at 02:05:41PM +0200, Ville M. Vainio wrote:
> On Sun, Feb 8, 2009 at 12:22 PM, Gael Varoquaux
> <gael.varoquaux at normalesup.org> wrote:

> > Finally, the trickiest part was running subprocesses. The reason being
> > that I needed both synchronous prints, and key-event processing. The

> In qt, dealing with processes is luckily very simple; you can just use QProcess.

> http://doc.trolltech.com/4.3/qprocess.html

Will this give you reliable killing of the process and all the processes
it spawned, and this accross platforms? This is important, as elsewhere
you can very easily lock your IPython session.

Ga?l


From vivainio at gmail.com  Sun Feb  8 07:46:46 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Sun, 8 Feb 2009 14:46:46 +0200
Subject: [IPython-dev] Cython based PyOS_InputHook hack
In-Reply-To: <20090208120716.GD14469@phare.normalesup.org>
References: <6ce0ac130902071442g710c0743of3432d4cb378f6c2@mail.gmail.com>
	<20090208102835.GB14469@phare.normalesup.org>
	<46cb515a0902080359w72ea78c9x1a5f5efbdf75e922@mail.gmail.com>
	<20090208120716.GD14469@phare.normalesup.org>
Message-ID: <46cb515a0902080446o3adc3b5bya6975cb6773818ad@mail.gmail.com>

On Sun, Feb 8, 2009 at 2:07 PM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:

>> Inferring from your other mail, it would only seem necessary to call
>> the mainloops Yield in the input hook function.
>
> I believe this is very risky. Yield is a very fragile function. For
> instance, under Windows, calling a Yield inside a Yield is not possible.
> So if you have a GUI callback that calls a raw_input, you are likely to
> end up in this situation (recursive call of Yield) when plugging readline
> in the GUI as currently done with "ipython -wthread".

Ok, is there a better wx equivalent for gtk.mainiteration()? A simple
workaround for possible re-entrancy is just a static variable
executing_inputhook that would not run the mainloop iteration if it's
already running it.

-- 
Ville M. Vainio
http://tinyurl.com/vainio


From vivainio at gmail.com  Sun Feb  8 07:49:07 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Sun, 8 Feb 2009 14:49:07 +0200
Subject: [IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook
In-Reply-To: <20090208121006.GE14469@phare.normalesup.org>
References: <6ce0ac130902070800s16e7365bp9bde1277f06b973f@mail.gmail.com>
	<20090207171133.GC24193@phare.normalesup.org>
	<6ce0ac130902071000o6de94a83y181bec34757b517e@mail.gmail.com>
	<20090207180410.GC22215@phare.normalesup.org>
	<6ce0ac130902071009i592a68bal59dcaab44819f01d@mail.gmail.com>
	<20090208102200.GA14469@phare.normalesup.org>
	<46cb515a0902080405t51a619c0ged894d328b8cd8f5@mail.gmail.com>
	<20090208121006.GE14469@phare.normalesup.org>
Message-ID: <46cb515a0902080449m4e3f8a8ar2c29979836cbde8c@mail.gmail.com>

On Sun, Feb 8, 2009 at 2:10 PM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:

>> http://doc.trolltech.com/4.3/qprocess.html
>
> Will this give you reliable killing of the process and all the processes
> it spawned, and this accross platforms? This is important, as elsewhere
> you can very easily lock your IPython session.

Probably, it has both kill() and terminate() - i.e. the polite way and
the hard way.

What it doesn't do is shell-like argument parsing.

-- 
Ville M. Vainio
http://tinyurl.com/vainio


From gael.varoquaux at normalesup.org  Sun Feb  8 07:55:57 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Sun, 8 Feb 2009 13:55:57 +0100
Subject: [IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook
In-Reply-To: <46cb515a0902080449m4e3f8a8ar2c29979836cbde8c@mail.gmail.com>
References: <6ce0ac130902070800s16e7365bp9bde1277f06b973f@mail.gmail.com>
	<20090207171133.GC24193@phare.normalesup.org>
	<6ce0ac130902071000o6de94a83y181bec34757b517e@mail.gmail.com>
	<20090207180410.GC22215@phare.normalesup.org>
	<6ce0ac130902071009i592a68bal59dcaab44819f01d@mail.gmail.com>
	<20090208102200.GA14469@phare.normalesup.org>
	<46cb515a0902080405t51a619c0ged894d328b8cd8f5@mail.gmail.com>
	<20090208121006.GE14469@phare.normalesup.org>
	<46cb515a0902080449m4e3f8a8ar2c29979836cbde8c@mail.gmail.com>
Message-ID: <20090208125557.GH14469@phare.normalesup.org>

On Sun, Feb 08, 2009 at 02:49:07PM +0200, Ville M. Vainio wrote:
> On Sun, Feb 8, 2009 at 2:10 PM, Gael Varoquaux
> <gael.varoquaux at normalesup.org> wrote:

> >> http://doc.trolltech.com/4.3/qprocess.html

> > Will this give you reliable killing of the process and all the processes
> > it spawned, and this accross platforms? This is important, as elsewhere
> > you can very easily lock your IPython session.

> Probably, it has both kill() and terminate() - i.e. the polite way and
> the hard way.

I saw that. We just need to validate that it runs robustly on all
platforms. I had problems with killing subprocesses under Windows. As a
result I had zombies that users could not see or kill. Coming from Qt it
may very well be coded robustly, though.

Ga?l


From vivainio at gmail.com  Sun Feb  8 08:01:06 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Sun, 8 Feb 2009 15:01:06 +0200
Subject: [IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook
In-Reply-To: <20090208125557.GH14469@phare.normalesup.org>
References: <6ce0ac130902070800s16e7365bp9bde1277f06b973f@mail.gmail.com>
	<20090207171133.GC24193@phare.normalesup.org>
	<6ce0ac130902071000o6de94a83y181bec34757b517e@mail.gmail.com>
	<20090207180410.GC22215@phare.normalesup.org>
	<6ce0ac130902071009i592a68bal59dcaab44819f01d@mail.gmail.com>
	<20090208102200.GA14469@phare.normalesup.org>
	<46cb515a0902080405t51a619c0ged894d328b8cd8f5@mail.gmail.com>
	<20090208121006.GE14469@phare.normalesup.org>
	<46cb515a0902080449m4e3f8a8ar2c29979836cbde8c@mail.gmail.com>
	<20090208125557.GH14469@phare.normalesup.org>
Message-ID: <46cb515a0902080501q20caf355o528602b006341d4f@mail.gmail.com>

On Sun, Feb 8, 2009 at 2:55 PM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:

>
> I saw that. We just need to validate that it runs robustly on all
> platforms. I had problems with killing subprocesses under Windows. As a
> result I had zombies that users could not see or kill. Coming from Qt it
> may very well be coded robustly, though.

Yeah, I sort of trust the Qt guys on this as well. However, even if it
sucked badly, we still have the PID and get rid of the process on our
own.

-- 
Ville M. Vainio
http://tinyurl.com/vainio


From gael.varoquaux at normalesup.org  Sun Feb  8 08:25:34 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Sun, 8 Feb 2009 14:25:34 +0100
Subject: [IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook
In-Reply-To: <46cb515a0902080501q20caf355o528602b006341d4f@mail.gmail.com>
References: <20090207171133.GC24193@phare.normalesup.org>
	<6ce0ac130902071000o6de94a83y181bec34757b517e@mail.gmail.com>
	<20090207180410.GC22215@phare.normalesup.org>
	<6ce0ac130902071009i592a68bal59dcaab44819f01d@mail.gmail.com>
	<20090208102200.GA14469@phare.normalesup.org>
	<46cb515a0902080405t51a619c0ged894d328b8cd8f5@mail.gmail.com>
	<20090208121006.GE14469@phare.normalesup.org>
	<46cb515a0902080449m4e3f8a8ar2c29979836cbde8c@mail.gmail.com>
	<20090208125557.GH14469@phare.normalesup.org>
	<46cb515a0902080501q20caf355o528602b006341d4f@mail.gmail.com>
Message-ID: <20090208132534.GI14469@phare.normalesup.org>

On Sun, Feb 08, 2009 at 03:01:06PM +0200, Ville M. Vainio wrote:
> Yeah, I sort of trust the Qt guys on this as well. However, even if it
> sucked badly, we still have the PID and get rid of the process on our
> own.

The big deal are the processes initiated by the child process, and not
the child process itself. This is what is hard, and the pid doesn't solve
it. The gpid does, and this is how I solved the problem under Unix, but I
had to set the process in a separate group first. And under windows
things are quite different.

Ga?l


From vivainio at gmail.com  Sun Feb  8 09:18:46 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Sun, 8 Feb 2009 16:18:46 +0200
Subject: [IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook
In-Reply-To: <20090208132534.GI14469@phare.normalesup.org>
References: <20090207171133.GC24193@phare.normalesup.org>
	<20090207180410.GC22215@phare.normalesup.org>
	<6ce0ac130902071009i592a68bal59dcaab44819f01d@mail.gmail.com>
	<20090208102200.GA14469@phare.normalesup.org>
	<46cb515a0902080405t51a619c0ged894d328b8cd8f5@mail.gmail.com>
	<20090208121006.GE14469@phare.normalesup.org>
	<46cb515a0902080449m4e3f8a8ar2c29979836cbde8c@mail.gmail.com>
	<20090208125557.GH14469@phare.normalesup.org>
	<46cb515a0902080501q20caf355o528602b006341d4f@mail.gmail.com>
	<20090208132534.GI14469@phare.normalesup.org>
Message-ID: <46cb515a0902080618u38469d88p7df8c8b7717739a1@mail.gmail.com>

On Sun, Feb 8, 2009 at 3:25 PM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:

> On Sun, Feb 08, 2009 at 03:01:06PM +0200, Ville M. Vainio wrote:
>> Yeah, I sort of trust the Qt guys on this as well. However, even if it
>> sucked badly, we still have the PID and get rid of the process on our
>> own.
>
> The big deal are the processes initiated by the child process, and not
> the child process itself. This is what is hard, and the pid doesn't solve
> it. The gpid does, and this is how I solved the problem under Unix, but I
> had to set the process in a separate group first. And under windows
> things are quite different.

Was part of your problem by any chance due to using shell
(use_shell=True in subprocess module calls) to launch the processes?
That has led to its share of problems for me...

-- 
Ville M. Vainio
http://tinyurl.com/vainio


From gael.varoquaux at normalesup.org  Sun Feb  8 09:20:17 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Sun, 8 Feb 2009 15:20:17 +0100
Subject: [IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook
In-Reply-To: <46cb515a0902080618u38469d88p7df8c8b7717739a1@mail.gmail.com>
References: <20090207180410.GC22215@phare.normalesup.org>
	<6ce0ac130902071009i592a68bal59dcaab44819f01d@mail.gmail.com>
	<20090208102200.GA14469@phare.normalesup.org>
	<46cb515a0902080405t51a619c0ged894d328b8cd8f5@mail.gmail.com>
	<20090208121006.GE14469@phare.normalesup.org>
	<46cb515a0902080449m4e3f8a8ar2c29979836cbde8c@mail.gmail.com>
	<20090208125557.GH14469@phare.normalesup.org>
	<46cb515a0902080501q20caf355o528602b006341d4f@mail.gmail.com>
	<20090208132534.GI14469@phare.normalesup.org>
	<46cb515a0902080618u38469d88p7df8c8b7717739a1@mail.gmail.com>
Message-ID: <20090208142017.GA25619@phare.normalesup.org>

On Sun, Feb 08, 2009 at 04:18:46PM +0200, Ville M. Vainio wrote:
> Was part of your problem by any chance due to using shell
> (use_shell=True in subprocess module calls) to launch the processes?
> That has led to its share of problems for me...

Absolutely. I ended needing this to be able to processing arbitrary
shebang lines from the user.

Ga?l


From mabshoff at googlemail.com  Sun Feb  8 13:45:24 2009
From: mabshoff at googlemail.com (Michael Abshoff)
Date: Sun, 8 Feb 2009 10:45:24 -0800
Subject: [IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook
In-Reply-To: <20090208125557.GH14469@phare.normalesup.org>
References: <6ce0ac130902070800s16e7365bp9bde1277f06b973f@mail.gmail.com>
	<20090207171133.GC24193@phare.normalesup.org>
	<6ce0ac130902071000o6de94a83y181bec34757b517e@mail.gmail.com>
	<20090207180410.GC22215@phare.normalesup.org>
	<6ce0ac130902071009i592a68bal59dcaab44819f01d@mail.gmail.com>
	<20090208102200.GA14469@phare.normalesup.org>
	<46cb515a0902080405t51a619c0ged894d328b8cd8f5@mail.gmail.com>
	<20090208121006.GE14469@phare.normalesup.org>
	<46cb515a0902080449m4e3f8a8ar2c29979836cbde8c@mail.gmail.com>
	<20090208125557.GH14469@phare.normalesup.org>
Message-ID: <c98f3e570902081045q7a224ec0k4d93598a2992ac19@mail.gmail.com>

On Sun, Feb 8, 2009 at 4:55 AM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Sun, Feb 08, 2009 at 02:49:07PM +0200, Ville M. Vainio wrote:
>> On Sun, Feb 8, 2009 at 2:10 PM, Gael Varoquaux
>> <gael.varoquaux at normalesup.org> wrote:
>
>> >> http://doc.trolltech.com/4.3/qprocess.html
>
>> > Will this give you reliable killing of the process and all the processes
>> > it spawned, and this accross platforms? This is important, as elsewhere
>> > you can very easily lock your IPython session.
>
>> Probably, it has both kill() and terminate() - i.e. the polite way and
>> the hard way.
>
> I saw that. We just need to validate that it runs robustly on all
> platforms. I had problems with killing subprocesses under Windows. As a
> result I had zombies that users could not see or kill. Coming from Qt it
> may very well be coded robustly, though.

In my experience Qt's QProcess module last year on Windows using MinGW
was less that stellar and I often ended up with run away processes
that I had to kill manually. I never deeply investigated what
happened, but was surprised since I expected to run into fewer bugs
with Qt 4.x in general on such a common platform.  The project I was
working on died, so I have not been testing that code since about
early 2008, so Qt might have this bug fixed in a subsequent release.

> Ga?l

Cheers,

Michael

> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
>


From ellisonbg.net at gmail.com  Sun Feb  8 15:15:53 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Sun, 8 Feb 2009 12:15:53 -0800
Subject: [IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook
In-Reply-To: <20090208102200.GA14469@phare.normalesup.org>
References: <6ce0ac130902070800s16e7365bp9bde1277f06b973f@mail.gmail.com>
	<20090207171133.GC24193@phare.normalesup.org>
	<6ce0ac130902071000o6de94a83y181bec34757b517e@mail.gmail.com>
	<20090207180410.GC22215@phare.normalesup.org>
	<6ce0ac130902071009i592a68bal59dcaab44819f01d@mail.gmail.com>
	<20090208102200.GA14469@phare.normalesup.org>
Message-ID: <6ce0ac130902081215i2304c6e4y92a06f7eba7cffbe@mail.gmail.com>

Gael,

Wow, thanks for taking the time to give us all the details of the
issues you ran into.  This is *super* helpful as we think about how to
move forward with refactoring the core to better support usage cases
like this.

At the same time, this type of thing will keep me up at night.  Our
core problem right now with the IPython core is that it is welded and
duct taped to the terminal and readline, making it difficult to use
outside of that setting.  The things you have done to get this working
(in my mind) are simply welding the Core to wx.  If we have to do
things like this for each GUI toolkit, we are back to having a core
that is non-reusable and difficult to maintain, test, extend, debug,
etc.  To make matters worse, these types of hacks are notoriously
fragile.

The killer though is that, at the end of the day, if a user runs a
long computation (let's say a big PyMC computation for example), the
entire GUI will become unresponsive.  To me, the above hacks seems
like an awful lot of work to get a solution that is still
sub-standard.

I think all of us have a vision that
IPython/numpy/scipy/matplotlib/sympy/ets/etc will allow us to build
scientific computing environments that blow the doors off of
Mathematica, Matlab, Maple, IDL, etc.  However, if our software always
has to be explained with "oh, that, yeh, when you run that type of
code, the GUI will freeze," we will never reach that goal.

So, let's try to figure out a real solution to these problems that
isn't crippled and let's us write robust, reusable and hackable code.

Cheers,

Brian




On Sun, Feb 8, 2009 at 2:22 AM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Sat, Feb 07, 2009 at 10:09:05AM -0800, Brian Granger wrote:
>> > I am not using any thread in the wx GUI. This was a very important design
>> > choice, IMHO. I am fiddling a lot with the mainloop to keep as much a
>> > possible the terminal reactive when a computation is going on, but there
>> > are no threads (except when a subprocess is used, but I was very careful
>> > to make them play well with the mainloop).
>
>> Because we are likely moving in this direction, I want to understand
>> this better.  What do you mean by "fiddling with the mainloop?"  Is
>> this "fiddling" optional?  If it causes bugs in your GUI (is this what
>> you are saying?) why do you do it?  You imply that the terminal was
>> not responsive until you did this extra stuff.  Can you say a bit
>> more?
>
> Sorry, I was out.
>
> When I say fiddling with the event loop, I mean many things. The big deal
> is that you cannot get a refresh, or a key event without running the
> event loop. The user is executing code, and executing it in the event
> loop, bocking it, as we are going non-multithreaded, but it is
> imperative, sequential code, and we really need event-driven code that
> yields to the event-loop in order to get refreshes and events. So I
> overloaded a few operations, such as print and raw_input to yield back to
> the event loop. Now I couldn't simply use wx.Yield blindly, because it is
> fragile, and getting this right was a bit tricky.
>
> For screen refresh, you want to refresh ASAP, because you don't want
> debug statements of a long running calculation to pop up only after it
> finish. So I had to trigger controlled refresh events to do refreshes on
> prints. See method 'write', line 135 of
> http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/frontend/wx/console_widget.py
> This method is injected in a sys.stdout and sys.stderr replacement. I had
> to be careful not to get a refresh called when we where already in a
> wx.Yield. This is why I had to add the refresh keyword, and use it
> cleverly. In addition, printing a lot of text to the terminal would slow
> down a lot, because a refresh is expensive. This is why I introduced the
> notion of a delayed refresh: there is a timer, and a queue. Trying to
> refresh triggers the timer, and the refresh happens delayed, so that the
> queue fills in. Obviously this means that if you are unlucky and the
> write method does not get called for a long time while you where waiting
> for the timer to empty, you don't get a refresh, because you don't enter
> the event loop again.
>
> I had to be careful with all other possible interactions, such as
> keyboard callbacks, not to trigger the Yield. Also, I had to write an
> OS-level output redirection that would weave the captured flux for
> compiled code with the Python flux, and call the refresh callback ASAP,
> this is done in
> http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/kernel/core/redirector_output_trap.py
>
> For the raw_input, I had to start a second event loop, to process
> keyboard event while the main event loop was blocked by the user code
> executing. See method 'raw_input', line 176 of
> http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/frontend/wx/wx_frontend.py
> There is readline-like logic that is used here to determine when this
> second event loop is terminated and the user code can resume.
>
> Finally, the trickiest part was running subprocesses. The reason being
> that I needed both synchronous prints, and key-event processing. The
> subprocess in run in a separate thread, see
> http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/frontend/_process/pipedprocess.py
> Its input stream is fed in by a raw_input-like mechanism, with a separate
> event-loop running, see method system_call, line 197 of
> http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/frontend/wx/wx_frontend.py
> . I have to use a separate event-loop because the call to a subprocess is
> blocking for the user code. The output stream is polled (yeah, I am not
> proud of the polling, I couldn't figure out something better), and fed to
> a 'buffered_write' method. No need to do a wx.Yield, here, as an
> event-loop is running in a separate thread. Which just need to reduce the
> number of write called per unit of time to avoid flooding the event loop,
> and to make sure we are thread-safe, see method 'buffered_write', line
> 264 of
> http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/frontend/wx/wx_frontend.py
> The termination of the subprocess is wired to a callback which flushes
> the buffer and terminates the inner event loop.
>
> I believe that this is mostly it. There is a lot more of little subtle
> things you have to be careful with: not calling your callbacks in
> infinite loops, making sure methods that can be called in other thread
> are thread safe, making sure you don't screw the order of events. This is
> quite tricky code, I believe. The good news is that all event loops are
> quite similar (the Tk one is different, in Python, as it runs in a
> separate OS thread, but we can forget that, and use it like the others).
> It would be possible to build an object that would abstract scintilla and
> the event-loop control out of different GUI toolkits, and expose a common
> interface. This would actually be fairly easy and would get us an
> interactive frontend for all toolkits.
>
> Ga?l
>


From gael.varoquaux at normalesup.org  Sun Feb  8 15:53:54 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Sun, 8 Feb 2009 21:53:54 +0100
Subject: [IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook
In-Reply-To: <6ce0ac130902081215i2304c6e4y92a06f7eba7cffbe@mail.gmail.com>
References: <6ce0ac130902070800s16e7365bp9bde1277f06b973f@mail.gmail.com>
	<20090207171133.GC24193@phare.normalesup.org>
	<6ce0ac130902071000o6de94a83y181bec34757b517e@mail.gmail.com>
	<20090207180410.GC22215@phare.normalesup.org>
	<6ce0ac130902071009i592a68bal59dcaab44819f01d@mail.gmail.com>
	<20090208102200.GA14469@phare.normalesup.org>
	<6ce0ac130902081215i2304c6e4y92a06f7eba7cffbe@mail.gmail.com>
Message-ID: <20090208205354.GU14469@phare.normalesup.org>

On Sun, Feb 08, 2009 at 12:15:53PM -0800, Brian Granger wrote:
> The things you have done to get this working (in my mind) are simply
> welding the Core to wx.  If we have to do things like this for each GUI
> toolkit, we are back to having a core that is non-reusable and
> difficult to maintain, test, extend, debug, etc. 

I am actually not too worried about this. I believe the problems can be
abstracted to a small set of well-defined methods to be implemented by a
frontend.

> To make matters worse, these types of hacks are notoriously fragile.

Indeed, this worries me a lot/

> The killer though is that, at the end of the day, if a user runs a
> long computation (let's say a big PyMC computation for example), the
> entire GUI will become unresponsive.  To me, the above hacks seems
> like an awful lot of work to get a solution that is still
> sub-standard.

> I think all of us have a vision that
> IPython/numpy/scipy/matplotlib/sympy/ets/etc will allow us to build
> scientific computing environments that blow the doors off of
> Mathematica, Matlab, Maple, IDL, etc.  However, if our software always
> has to be explained with "oh, that, yeh, when you run that type of
> code, the GUI will freeze," we will never reach that goal.

> So, let's try to figure out a real solution to these problems that
> isn't crippled and let's us write robust, reusable and hackable code.

I have given a lot of thought to this, and I believe that to avoid
freezing, there is no other option than multiple processes. If you think
of all the interactive environments that are user friendly, they all run
code in a sandbox that has nothing to do with the GUI (sage included). If
you don't go multi-processes, you are bound to expose some of the
oddities of the event loop, or worst, threads, to the user.

The problem with multiple processes is that if I want to do interactive
data visualization or debugging, I end up copying my data between
processes, and this is not good at all. Maybe the shared array that we
have been discussing on the scipy-users mailing list is an answer to this
problem, but it is not there yet, and it seems like a sledge hammer
approach.

If you only want a friendly, interactive terminal that does not freeze
when you run computations, ie an IPython++, I believe something like the
sage approach is excellent. In other words, you have a canvas where you
lay out commands and their output, that lives in a totally different
process than the execution engine. Not too much data flows between the
canvas and the execution engine, so there is no shared memory problem.
Interactive visualization can be done by running a GUI loop in the
execution engine, and having things like matplotlib or Mayavi living in
the same process. I believe the best approach so far would be to build
upon something like knoboo (http://knoboo.com/, which is unfortunatey
GPL) for the front, and an execution engine with an eventloop built upon
IPython1. Maybe the frontend could be made to look cooler by building a
GUI around an embedded Webkit, but that's probably the easiest part.

Interactive visualization of very large dataset is an important feature
for my current work. Mayavi + ipython is fantastic in this regard (sorry
for my lack of humility). It is a great debug tool to be able to
visualize the very exact huge datasets that you are working with, and the
larger the dataset, the more important it is to have interactivity, to be
able to explore it better.

These are the conclusions I came to at the end of my work at Enthought
this summer. The experience I gained from developing the frontend,
integrating it in Mayavi, and developing SciPylab (a scientific
environment integrating all that, that we have been quite quiet about
because it is far from being ready), changed a lot my initial vision of
the problem, I must admit.

My 2 cents,

Ga?l


From gael.varoquaux at normalesup.org  Sun Feb  8 16:38:52 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Sun, 8 Feb 2009 22:38:52 +0100
Subject: [IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook
In-Reply-To: <6ce0ac130902081331r20ba7b54p65795b7935117176@mail.gmail.com>
References: <6ce0ac130902070800s16e7365bp9bde1277f06b973f@mail.gmail.com>
	<20090207171133.GC24193@phare.normalesup.org>
	<6ce0ac130902071000o6de94a83y181bec34757b517e@mail.gmail.com>
	<20090207180410.GC22215@phare.normalesup.org>
	<6ce0ac130902071009i592a68bal59dcaab44819f01d@mail.gmail.com>
	<20090208102200.GA14469@phare.normalesup.org>
	<6ce0ac130902081215i2304c6e4y92a06f7eba7cffbe@mail.gmail.com>
	<20090208205354.GU14469@phare.normalesup.org>
	<6ce0ac130902081331r20ba7b54p65795b7935117176@mail.gmail.com>
Message-ID: <20090208213852.GB12866@phare.normalesup.org>

Copying to the list, which you probably forgot.

On Sun, Feb 08, 2009 at 01:31:27PM -0800, Brian Granger wrote:
> On Sun, Feb 8, 2009 at 12:53 PM, Gael Varoquaux
> <gael.varoquaux at normalesup.org> wrote:
> > On Sun, Feb 08, 2009 at 12:15:53PM -0800, Brian Granger wrote:
> >> The things you have done to get this working (in my mind) are simply
> >> welding the Core to wx.  If we have to do things like this for each GUI
> >> toolkit, we are back to having a core that is non-reusable and
> >> difficult to maintain, test, extend, debug, etc.

> > I am actually not too worried about this. I believe the problems can be
> > abstracted to a small set of well-defined methods to be implemented by a
> > frontend.

> I agree that we should be able to abstract these things out.  But, if
> you believe that all abstractions are leaky (I do) then what you
> really end up with is 1) nice clean abstractions and 2) duct tape to
> plug the leaks.  The question is how much do they leak.

Yup, agreed.

> >> To make matters worse, these types of hacks are notoriously fragile.

> > Indeed, this worries me a lot/

> It is this fragility that leads me to think that the above
> abstractions will have leaks and require duct taping.

Yup.

> > I have given a lot of thought to this, and I believe that to avoid
> > freezing, there is no other option than multiple processes. If you think
> > of all the interactive environments that are user friendly, they all run
> > code in a sandbox that has nothing to do with the GUI (sage included). If
> > you don't go multi-processes, you are bound to expose some of the
> > oddities of the event loop, or worst, threads, to the user.

> Yes, as far as I know all the commercial product (Mathematica, Matlab,
> etc) all do this as well.  What is interesting is that they also do
> interactive plotting.  I wonder if they share data?  I too think that
> a multiprocess solution is key.

> > The problem with multiple processes is that if I want to do interactive
> > data visualization or debugging, I end up copying my data between
> > processes, and this is not good at all. Maybe the shared array that we
> > have been discussing on the scipy-users mailing list is an answer to this
> > problem, but it is not there yet, and it seems like a sledge hammer
> > approach.

> Visualization:  yes, this could require data movement if the
> visualization GUI is running in a process where the data isn't.

I would really like to be able not to have to copy data. In some of the
work that I have been doing currently, I have been able to achieve
results that others have not simply because I was more clever with memory
(hint: 64 bit + a lot of memmapping). My raw data takes more than 2Gb. I
can't copy it many times before I kill my 8Gb box.

> Debugging:  I think we could make a debugger work across processes.
> After all, an interactive debugger just takes input and write output.
> That IO can be sent over the wire.

Yes, but that not the scientific debugger than I am dreaming of, in which
I could do visualization.

Thinking these things over, it seems that we should have visualization at
both ends. To use frontend visualization, we would have data copying, but
no freezing during calculation and debugging. A good UI paradigm would be
needed to make it clear what happens on which end.

> > If you only want a friendly, interactive terminal that does not freeze
> > when you run computations, ie an IPython++, I believe something like the
> > sage approach is excellent. In other words, you have a canvas where you
> > lay out commands and their output, that lives in a totally different
> > process than the execution engine. Not too much data flows between the
> > canvas and the execution engine, so there is no shared memory problem.
> > Interactive visualization can be done by running a GUI loop in the
> > execution engine, and having things like matplotlib or Mayavi living in
> > the same process. I believe the best approach so far would be to build
> > upon something like knoboo (http://knoboo.com/, which is unfortunatey
> > GPL) for the front, and an execution engine with an eventloop built upon
> > IPython1. Maybe the frontend could be made to look cooler by building a
> > GUI around an embedded Webkit, but that's probably the easiest part.

> This is the type of design that I have in mind.  We just need to think
> about how things like print, raw_input will be handled.  But it should
> be possible.  I will try to sketch out some of these ideas.

> > Interactive visualization of very large dataset is an important feature
> > for my current work. Mayavi + ipython is fantastic in this regard (sorry
> > for my lack of humility). It is a great debug tool to be able to
> > visualize the very exact huge datasets that you are working with, and the
> > larger the dataset, the more important it is to have interactivity, to be
> > able to explore it better.

> I agree that interactive viz is an absolute requirement.  It has to
> work or we can't get science done!!!

Thanks for working on that, Brian, these are tricky and important
questions.

Good luck,

Ga?l


From ellisonbg.net at gmail.com  Sun Feb  8 16:47:55 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Sun, 8 Feb 2009 13:47:55 -0800
Subject: [IPython-dev] Status of IPython+GUI+Threads+PyOS_InputHook
In-Reply-To: <20090208213852.GB12866@phare.normalesup.org>
References: <6ce0ac130902070800s16e7365bp9bde1277f06b973f@mail.gmail.com>
	<20090207171133.GC24193@phare.normalesup.org>
	<6ce0ac130902071000o6de94a83y181bec34757b517e@mail.gmail.com>
	<20090207180410.GC22215@phare.normalesup.org>
	<6ce0ac130902071009i592a68bal59dcaab44819f01d@mail.gmail.com>
	<20090208102200.GA14469@phare.normalesup.org>
	<6ce0ac130902081215i2304c6e4y92a06f7eba7cffbe@mail.gmail.com>
	<20090208205354.GU14469@phare.normalesup.org>
	<6ce0ac130902081331r20ba7b54p65795b7935117176@mail.gmail.com>
	<20090208213852.GB12866@phare.normalesup.org>
Message-ID: <6ce0ac130902081347h35b1bc8fjdfd5fb7c06d097ce@mail.gmail.com>

> Yes, but that not the scientific debugger than I am dreaming of, in which
> I could do visualization.

But, if both the debugger and a GUI event loop are running in the
IPython process, this would be possible.  The only thing that this
process wouldn't handle is input and text output?

But, yes, interactive viz in a debugger makes lots of sense.

> Thinking these things over, it seems that we should have visualization at
> both ends. To use frontend visualization, we would have data copying, but
> no freezing during calculation and debugging. A good UI paradigm would be
> needed to make it clear what happens on which end.

Yep, we might need this.  I think coming up with a good paradigm will
really help us.

Cheers,

Brian


From ellisonbg.net at gmail.com  Sun Feb  8 16:53:13 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Sun, 8 Feb 2009 13:53:13 -0800
Subject: [IPython-dev] Cython based PyOS_InputHook hack
In-Reply-To: <20090208102835.GB14469@phare.normalesup.org>
References: <6ce0ac130902071442g710c0743of3432d4cb378f6c2@mail.gmail.com>
	<20090208102835.GB14469@phare.normalesup.org>
Message-ID: <6ce0ac130902081353va425c88tbdf3b5b92d8f0dae@mail.gmail.com>

> Sorry, Brain, I don't know wx well-enough to do this without a
> significant time investement (these things are tricky). I don't see myself
> finding this time soon, as my coding time is taken by Mayavi and
> NeuroImaging-related stuff.>

As Ville points out, your other email possibly gives the answer.  You
know more about wx than you think!

> I can put this on the pile of stuff to do. I believe I can give it a try
> in a few months. Maybe not before begining of July. I hate to be that
> pessimistic, but I need to free some time where I can focus and think for
> this, I believe.

Don't worry about it.  If we simply remove the threaded shells,
someone familiar with wx will step up to the plate ;-)

That said, I might try to see i a simple call to Yield will do it.

Brian


> Ga?l
>


From ellisonbg.net at gmail.com  Sun Feb  8 16:55:10 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Sun, 8 Feb 2009 13:55:10 -0800
Subject: [IPython-dev] Cython based PyOS_InputHook hack
In-Reply-To: <20090208120716.GD14469@phare.normalesup.org>
References: <6ce0ac130902071442g710c0743of3432d4cb378f6c2@mail.gmail.com>
	<20090208102835.GB14469@phare.normalesup.org>
	<46cb515a0902080359w72ea78c9x1a5f5efbdf75e922@mail.gmail.com>
	<20090208120716.GD14469@phare.normalesup.org>
Message-ID: <6ce0ac130902081355y6937ac45n4d5e08a2dfc9f5a3@mail.gmail.com>

>> > Sorry, Brain, I don't know wx well-enough to do this without a
>            ^
>        Sorry, Brian, this was a typo.

That is funny.  My first driver's license had this typo - I first
noticed it when I got pulled over by a cop and had to expain that my
name was not "Brain".

>> > significant time investement (these things are tricky). I don't see myself
>
>> Inferring from your other mail, it would only seem necessary to call
>> the mainloops Yield in the input hook function.
>
> I believe this is very risky. Yield is a very fragile function. For
> instance, under Windows, calling a Yield inside a Yield is not possible.
> So if you have a GUI callback that calls a raw_input, you are likely to
> end up in this situation (recursive call of Yield) when plugging readline
> in the GUI as currently done with "ipython -wthread".

But, I think there are protections for this:

Calling the Yield method on the application object while in Yield is
a bug, i.e. Yield is not reentrant. The debug version of wxWidgets
throws an assert in this case (that bug is assigned to me). If you
need to call Yield and you're not absolutely sure you're not being
called from Yield, you should pass True to Yield, i.e. wx.GetApp
().Yield(True), which will return instead of recursing in this case.

> I am not saying this is not the way to go. I am saying it needs thinking
> and validation.

For sure, would $20 that a simple call to Yield won't work in all cases.

Cheers,

Brian

> Ga?l
>


From ellisonbg.net at gmail.com  Sun Feb  8 19:08:31 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Sun, 8 Feb 2009 16:08:31 -0800
Subject: [IPython-dev] Interactive wx/pylab with no threads (PyOS_InputHook)
Message-ID: <6ce0ac130902081608ha95d3c3mbb404e6d04968323@mail.gmail.com>

IPython and matplotlib devs,

Over the weekend I have been playing around to see if it is possible
to do interactive GUI work with wx from IPython *without using
threads*.  The idea here is to use PyOS_InputHook.  Currently, recent
versions of PyQt4 and PyGTK do this and if we can get wx working, we
can probably get rid of IPython's subtle threaded shells that
currently allow interactive GUIs to work.

I am attaching a Cython module that mostly works.  Here is a simple
example that works in IPython (without the -wthread option!)

In [1]: import pyximport

In [2]: pyximport.install()

In [3]: import inputhook

In [4]: inputhook.set_input_hook()

In [5]: import wx

In [6]: app = wx.PySimpleApp()

In [7]: app.MainLoop()

In [8]: f = wx.Frame(None,-1,"Look mom, no threads!")

In [9]: f.Show()
Out[9]: True

The docstring of the module also has a matplotlib example.  This
really does work and I am pretty sure it will also work in plain
vanilla python as well.  There are a few issues to work out:

* When frame are closed by clicking the red button or the "X", the
Windows don't close.  In addition, in matplotlib, this causes further
problems.

* In the current matplotlib backend wx.Yield() is called in a way that
is not safe as far as protecting against recursive calls to Yield.  I
think it should be called in this way:

app = wx.GetApp()
if app is not None:
  app.Yield(True)

* I don't think that interupts work yet, but I haven't tested this
thoroughly yet.

I don't have any more time to work on this right now, but I at least
wanted to share my findings with both IPython and matplotlib devs.  It
would be great if someone familiar with wx could try to figure out the
remaining issues.  If there are no takers here, I might eventually see
if wxpython itself is interested in this code (that is probably where
it really belongs anyway).

Cheers,

Brian
-------------- next part --------------
#include <Python.h>

static PyThreadState *event_tstate = NULL;

static void set_input_hook_c(int (*f)(void))
{
    event_tstate = PyThreadState_Get();
    PyOS_InputHook = f;
}

static void clear_input_hook_c(void)
{
    PyOS_InputHook = NULL;
}
-------------- next part --------------
A non-text attachment was scrubbed...
Name: inputhook.pyx
Type: application/octet-stream
Size: 2384 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090208/d35533f8/attachment.obj>

From barrywark at gmail.com  Sun Feb  8 23:49:07 2009
From: barrywark at gmail.com (Barry Wark)
Date: Sun, 8 Feb 2009 20:49:07 -0800
Subject: [IPython-dev] Interactive wx/pylab with no threads
	(PyOS_InputHook)
In-Reply-To: <6ce0ac130902081608ha95d3c3mbb404e6d04968323@mail.gmail.com>
References: <6ce0ac130902081608ha95d3c3mbb404e6d04968323@mail.gmail.com>
Message-ID: <cd7634ce0902082049n6a29ee12wbfdf7f0b6ae39567@mail.gmail.com>

I appologize for getting to the discussion late... There have been
several very interesting threads in the last couple of days. I'm
replying to this one arbitrarily, am intending to write more detailed
responses to some of the other threads, but wanted to throw this
general idea out there...

>From my reading, it seems that Brian is well on his way to having a
threading-free core for Qt and GTK. With the awesome new (in a recent
version, I forget which) OS X backend for matplotlib and the existing
ability of  Twisted's _threadedselect reactor to work with OS X's
CFRunLoop via PyObjC (it uses a very similar mechanism to the
PyOS_InputHook, no?), I suspect that a similar approach will work with
OS X as well. I know this may sound heretical to some, but if
supporting Wx GUI loops natively requires compromising the hackability
and testability of the IPython core, do we *need* to support it?

Just a 1c thought.

-Barry



On Sun, Feb 8, 2009 at 4:08 PM, Brian Granger <ellisonbg.net at gmail.com> wrote:
> IPython and matplotlib devs,
>
> Over the weekend I have been playing around to see if it is possible
> to do interactive GUI work with wx from IPython *without using
> threads*.  The idea here is to use PyOS_InputHook.  Currently, recent
> versions of PyQt4 and PyGTK do this and if we can get wx working, we
> can probably get rid of IPython's subtle threaded shells that
> currently allow interactive GUIs to work.
>
> I am attaching a Cython module that mostly works.  Here is a simple
> example that works in IPython (without the -wthread option!)
>
> In [1]: import pyximport
>
> In [2]: pyximport.install()
>
> In [3]: import inputhook
>
> In [4]: inputhook.set_input_hook()
>
> In [5]: import wx
>
> In [6]: app = wx.PySimpleApp()
>
> In [7]: app.MainLoop()
>
> In [8]: f = wx.Frame(None,-1,"Look mom, no threads!")
>
> In [9]: f.Show()
> Out[9]: True
>
> The docstring of the module also has a matplotlib example.  This
> really does work and I am pretty sure it will also work in plain
> vanilla python as well.  There are a few issues to work out:
>
> * When frame are closed by clicking the red button or the "X", the
> Windows don't close.  In addition, in matplotlib, this causes further
> problems.
>
> * In the current matplotlib backend wx.Yield() is called in a way that
> is not safe as far as protecting against recursive calls to Yield.  I
> think it should be called in this way:
>
> app = wx.GetApp()
> if app is not None:
>  app.Yield(True)
>
> * I don't think that interupts work yet, but I haven't tested this
> thoroughly yet.
>
> I don't have any more time to work on this right now, but I at least
> wanted to share my findings with both IPython and matplotlib devs.  It
> would be great if someone familiar with wx could try to figure out the
> remaining issues.  If there are no takers here, I might eventually see
> if wxpython itself is interested in this code (that is probably where
> it really belongs anyway).
>
> Cheers,
>
> Brian
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
>
>


From gael.varoquaux at normalesup.org  Mon Feb  9 01:00:18 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 9 Feb 2009 07:00:18 +0100
Subject: [IPython-dev] Interactive wx/pylab with no
	threads	(PyOS_InputHook)
In-Reply-To: <cd7634ce0902082049n6a29ee12wbfdf7f0b6ae39567@mail.gmail.com>
References: <6ce0ac130902081608ha95d3c3mbb404e6d04968323@mail.gmail.com>
	<cd7634ce0902082049n6a29ee12wbfdf7f0b6ae39567@mail.gmail.com>
Message-ID: <20090209060018.GA26350@phare.normalesup.org>

On Sun, Feb 08, 2009 at 08:49:07PM -0800, Barry Wark wrote:
> I appologize for getting to the discussion late... There have been
> several very interesting threads in the last couple of days. I'm
> replying to this one arbitrarily, am intending to write more detailed
> responses to some of the other threads, but wanted to throw this
> general idea out there...

> > From my reading, it seems that Brian is well on his way to having a
> > threading-free core for Qt and GTK. With the awesome new (in a recent
> > version, I forget which) OS X backend for matplotlib and the existing
> > ability of  Twisted's _threadedselect reactor to work with OS X's
> > CFRunLoop via PyObjC (it uses a very similar mechanism to the
> > PyOS_InputHook, no?), I suspect that a similar approach will work with
> > OS X as well. I know this may sound heretical to some, but if
> > supporting Wx GUI loops natively requires compromising the hackability
> > and testability of the IPython core, do we *need* to support it?

All the libraries I use, I use them through Wx GUI loops. And it is the
case of most people under Windows (I am under Linux). The reason behind
both statement is that Wx has been historically the easy way to have a
powerful GUI shared between major operating systems. Maybe Qt is the way
forward now, but it is not yet established. The problem with Wx
(recursive Yeilds) comes from the fact that it supports different event
loop with different semantics on different platforms, so it comes from
that richness.

Wx is the toolkit where you will find the largest amount of mature
contributions. I know that matplotlib has a variety of different toolkit
support, but matplotlib is an exemption.

Do we need wx support? I guess not, but I would be jumping ship, and many
people would.

Ga?l


From gael.varoquaux at normalesup.org  Mon Feb  9 01:29:45 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 9 Feb 2009 07:29:45 +0100
Subject: [IPython-dev] [matplotlib-devel] Interactive wx/pylab with no
	threads	(PyOS_InputHook)
In-Reply-To: <6ce0ac130902081608ha95d3c3mbb404e6d04968323@mail.gmail.com>
References: <6ce0ac130902081608ha95d3c3mbb404e6d04968323@mail.gmail.com>
Message-ID: <20090209062945.GC26350@phare.normalesup.org>

On Sun, Feb 08, 2009 at 04:08:31PM -0800, Brian Granger wrote:
> * In the current matplotlib backend wx.Yield() is called in a way that
> is not safe as far as protecting against recursive calls to Yield.  I
> think it should be called in this way:

> app = wx.GetApp()
> if app is not None:
>   app.Yield(True)

The problem I see with this approach is that arbitrary wx programs will
always be doing this. The matplotlib guys can fix matplotib not to do
this. I can fix Mayavi not to do this, but there are many more wx
programs. And anyhow, most of the time, Yield should not be called, as it
is a hack. Unfortunately, you often end up having to call it. :(

Ga?l


From fperez.net at gmail.com  Mon Feb  9 03:53:18 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Mon, 9 Feb 2009 00:53:18 -0800
Subject: [IPython-dev] Interactive wx/pylab with no threads
	(PyOS_InputHook)
In-Reply-To: <20090209060018.GA26350@phare.normalesup.org>
References: <6ce0ac130902081608ha95d3c3mbb404e6d04968323@mail.gmail.com>
	<cd7634ce0902082049n6a29ee12wbfdf7f0b6ae39567@mail.gmail.com>
	<20090209060018.GA26350@phare.normalesup.org>
Message-ID: <db6b5ecc0902090053q5a3a31a2we8f975e61667b910@mail.gmail.com>

On Sun, Feb 8, 2009 at 10:00 PM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Sun, Feb 08, 2009 at 08:49:07PM -0800, Barry Wark wrote:

>> > From my reading, it seems that Brian is well on his way to having a
>> > threading-free core for Qt and GTK. With the awesome new (in a recent
>> > version, I forget which) OS X backend for matplotlib and the existing
>> > ability of  Twisted's _threadedselect reactor to work with OS X's
>> > CFRunLoop via PyObjC (it uses a very similar mechanism to the
>> > PyOS_InputHook, no?), I suspect that a similar approach will work with
>> > OS X as well. I know this may sound heretical to some, but if
>> > supporting Wx GUI loops natively requires compromising the hackability
>> > and testability of the IPython core, do we *need* to support it?
>
> All the libraries I use, I use them through Wx GUI loops. And it is the
> case of most people under Windows (I am under Linux). The reason behind
> both statement is that Wx has been historically the easy way to have a
> powerful GUI shared between major operating systems. Maybe Qt is the way
> forward now, but it is not yet established. The problem with Wx
> (recursive Yeilds) comes from the fact that it supports different event
> loop with different semantics on different platforms, so it comes from
> that richness.
>
> Wx is the toolkit where you will find the largest amount of mature
> contributions. I know that matplotlib has a variety of different toolkit
> support, but matplotlib is an exemption.
>
> Do we need wx support? I guess not, but I would be jumping ship, and many
> people would.

Barry,  I think you make a very valid point, but quite unfortunately,
I have to side with Gael on this one: Wx *today* gives us Chaco,
Mayavi2, Envisage, TraitsGUI and friends.  That's a LOT of
functionality that many of us need and use, and we can't really dump
that user base.  We can look into perhaps refactoring 'ipython
-wthread' into some ghetto where it doesn't "pollute" the rest of the
code quite so much, and I'm totally open to that.  But I think *right
now* dropping Wx would be a mistake.

However, note above my emphasis on "now".  That's because I'm very
much hoping that PyQt will go LGPL and that over time, Qt will come to
replace Wx in the roles we're interested in.  If/when that comes to
pass, we may very well see this discussion in a very different light.

Cheers,

f


From vivainio at gmail.com  Mon Feb  9 04:44:09 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Mon, 9 Feb 2009 11:44:09 +0200
Subject: [IPython-dev] Interactive wx/pylab with no threads
	(PyOS_InputHook)
In-Reply-To: <db6b5ecc0902090053q5a3a31a2we8f975e61667b910@mail.gmail.com>
References: <6ce0ac130902081608ha95d3c3mbb404e6d04968323@mail.gmail.com>
	<cd7634ce0902082049n6a29ee12wbfdf7f0b6ae39567@mail.gmail.com>
	<20090209060018.GA26350@phare.normalesup.org>
	<db6b5ecc0902090053q5a3a31a2we8f975e61667b910@mail.gmail.com>
Message-ID: <46cb515a0902090144g28f4585dlea8e4b50cce313a0@mail.gmail.com>

On Mon, Feb 9, 2009 at 10:53 AM, Fernando Perez <fperez.net at gmail.com> wrote:

> that user base.  We can look into perhaps refactoring 'ipython
> -wthread' into some ghetto where it doesn't "pollute" the rest of the
> code quite so much, and I'm totally open to that.  But I think *right
> now* dropping Wx would be a mistake.

Actually, it's not all that bad. The 'ghetto' is already the
MTInteractiveShell, and it's easy to avoid that by just using standard
InteractiveShell in a single-threaded fashion. And since Brian already
implemented the PyOS_InputHook for wx, it's just a matter of putting
it under extensive testing.

Note that now that wx can also supports this, you can throw in twisted
wxreactor and it should work out of the box.

-- 
Ville M. Vainio
http://tinyurl.com/vainio


From gael.varoquaux at normalesup.org  Mon Feb  9 05:38:20 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 9 Feb 2009 11:38:20 +0100
Subject: [IPython-dev] Interactive wx/pylab with no
	threads	(PyOS_InputHook)
In-Reply-To: <db6b5ecc0902090053q5a3a31a2we8f975e61667b910@mail.gmail.com>
References: <6ce0ac130902081608ha95d3c3mbb404e6d04968323@mail.gmail.com>
	<cd7634ce0902082049n6a29ee12wbfdf7f0b6ae39567@mail.gmail.com>
	<20090209060018.GA26350@phare.normalesup.org>
	<db6b5ecc0902090053q5a3a31a2we8f975e61667b910@mail.gmail.com>
Message-ID: <20090209103819.GB32331@phare.normalesup.org>

On Mon, Feb 09, 2009 at 12:53:18AM -0800, Fernando Perez wrote:
> Wx *today* gives us Chaco, Mayavi2, Envisage, TraitsGUI and friends.
> That's a LOT of functionality that many of us need and use, and we
> can't really dump that user base. 

It is not only but these tools. It is that if I need to develop and
deploy a GUI on all major platforms, and I need it to be BSD or LGPL, wx
is the best option I have, regardless of Traits. It is also the most used
in the Python community. I wouldn't be surprised to find out that many
labs have their custom tools that use wx.

Ga?l


From vivainio at gmail.com  Mon Feb  9 05:42:42 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Mon, 9 Feb 2009 12:42:42 +0200
Subject: [IPython-dev] Interactive wx/pylab with no threads
	(PyOS_InputHook)
In-Reply-To: <20090209103819.GB32331@phare.normalesup.org>
References: <6ce0ac130902081608ha95d3c3mbb404e6d04968323@mail.gmail.com>
	<cd7634ce0902082049n6a29ee12wbfdf7f0b6ae39567@mail.gmail.com>
	<20090209060018.GA26350@phare.normalesup.org>
	<db6b5ecc0902090053q5a3a31a2we8f975e61667b910@mail.gmail.com>
	<20090209103819.GB32331@phare.normalesup.org>
Message-ID: <46cb515a0902090242h71140424w58c68dc1b93819bc@mail.gmail.com>

On Mon, Feb 9, 2009 at 12:38 PM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:

> It is not only but these tools. It is that if I need to develop and
> deploy a GUI on all major platforms, and I need it to be BSD or LGPL, wx
> is the best option I have, regardless of Traits. It is also the most used
> in the Python community. I wouldn't be surprised to find out that many
> labs have their custom tools that use wx.

Luckily, with Qt gone LGPL, I would guess PyQt to follow suit at some
point, and the license issue disappears. I'd also assume PyQt use to
increase rapidly after that.

But, as said previously, there is no need to worry about wx support in
particular, since it clearly can be made to work without threads.

-- 
Ville M. Vainio
http://tinyurl.com/vainio


From barrywark at gmail.com  Mon Feb  9 12:17:02 2009
From: barrywark at gmail.com (Barry Wark)
Date: Mon, 9 Feb 2009 09:17:02 -0800
Subject: [IPython-dev] Interactive wx/pylab with no threads
	(PyOS_InputHook)
In-Reply-To: <20090209060018.GA26350@phare.normalesup.org>
References: <6ce0ac130902081608ha95d3c3mbb404e6d04968323@mail.gmail.com>
	<cd7634ce0902082049n6a29ee12wbfdf7f0b6ae39567@mail.gmail.com>
	<20090209060018.GA26350@phare.normalesup.org>
Message-ID: <cd7634ce0902090917x114ae22aya1293f20a47b45eb@mail.gmail.com>

On Sun, Feb 8, 2009 at 10:00 PM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Sun, Feb 08, 2009 at 08:49:07PM -0800, Barry Wark wrote:
>> I appologize for getting to the discussion late... There have been
>> several very interesting threads in the last couple of days. I'm
>> replying to this one arbitrarily, am intending to write more detailed
>> responses to some of the other threads, but wanted to throw this
>> general idea out there...
>
>> > From my reading, it seems that Brian is well on his way to having a
>> > threading-free core for Qt and GTK. With the awesome new (in a recent
>> > version, I forget which) OS X backend for matplotlib and the existing
>> > ability of  Twisted's _threadedselect reactor to work with OS X's
>> > CFRunLoop via PyObjC (it uses a very similar mechanism to the
>> > PyOS_InputHook, no?), I suspect that a similar approach will work with
>> > OS X as well. I know this may sound heretical to some, but if
>> > supporting Wx GUI loops natively requires compromising the hackability
>> > and testability of the IPython core, do we *need* to support it?
>
> All the libraries I use, I use them through Wx GUI loops. And it is the
> case of most people under Windows (I am under Linux). The reason behind
> both statement is that Wx has been historically the easy way to have a
> powerful GUI shared between major operating systems. Maybe Qt is the way
> forward now, but it is not yet established. The problem with Wx
> (recursive Yeilds) comes from the fact that it supports different event
> loop with different semantics on different platforms, so it comes from
> that richness.
>
> Wx is the toolkit where you will find the largest amount of mature
> contributions. I know that matplotlib has a variety of different toolkit
> support, but matplotlib is an exemption.
>
> Do we need wx support? I guess not, but I would be jumping ship, and many
> people would.

That's the answer. I was just throwing out the idea. Sounds like it's
not a realistic approach at this time.

>
> Ga?l
>


From gael.varoquaux at normalesup.org  Mon Feb  9 12:19:14 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 9 Feb 2009 18:19:14 +0100
Subject: [IPython-dev] Interactive wx/pylab with no
	threads	(PyOS_InputHook)
In-Reply-To: <cd7634ce0902090917x114ae22aya1293f20a47b45eb@mail.gmail.com>
References: <6ce0ac130902081608ha95d3c3mbb404e6d04968323@mail.gmail.com>
	<cd7634ce0902082049n6a29ee12wbfdf7f0b6ae39567@mail.gmail.com>
	<20090209060018.GA26350@phare.normalesup.org>
	<cd7634ce0902090917x114ae22aya1293f20a47b45eb@mail.gmail.com>
Message-ID: <20090209171914.GF27832@phare.normalesup.org>

On Mon, Feb 09, 2009 at 09:17:02AM -0800, Barry Wark wrote:
> > Do we need wx support? I guess not, but I would be jumping ship, and many
> > people would.

> That's the answer. I was just throwing out the idea. Sounds like it's
> not a realistic approach at this time.

I just want to point out that I am ashamed of the above sentence
('jumping ship'). I was overreacting and being rude, as usual. I would be
happier if I hadn't writen that sentence.

Ga?l


From barrywark at gmail.com  Mon Feb  9 12:20:02 2009
From: barrywark at gmail.com (Barry Wark)
Date: Mon, 9 Feb 2009 09:20:02 -0800
Subject: [IPython-dev] Interactive wx/pylab with no threads
	(PyOS_InputHook)
In-Reply-To: <db6b5ecc0902090053q5a3a31a2we8f975e61667b910@mail.gmail.com>
References: <6ce0ac130902081608ha95d3c3mbb404e6d04968323@mail.gmail.com>
	<cd7634ce0902082049n6a29ee12wbfdf7f0b6ae39567@mail.gmail.com>
	<20090209060018.GA26350@phare.normalesup.org>
	<db6b5ecc0902090053q5a3a31a2we8f975e61667b910@mail.gmail.com>
Message-ID: <cd7634ce0902090920j46c4e85bnebe57301dbc154c9@mail.gmail.com>

On Mon, Feb 9, 2009 at 12:53 AM, Fernando Perez <fperez.net at gmail.com> wrote:
> On Sun, Feb 8, 2009 at 10:00 PM, Gael Varoquaux
> <gael.varoquaux at normalesup.org> wrote:
>> On Sun, Feb 08, 2009 at 08:49:07PM -0800, Barry Wark wrote:
>
>>> > From my reading, it seems that Brian is well on his way to having a
>>> > threading-free core for Qt and GTK. With the awesome new (in a recent
>>> > version, I forget which) OS X backend for matplotlib and the existing
>>> > ability of  Twisted's _threadedselect reactor to work with OS X's
>>> > CFRunLoop via PyObjC (it uses a very similar mechanism to the
>>> > PyOS_InputHook, no?), I suspect that a similar approach will work with
>>> > OS X as well. I know this may sound heretical to some, but if
>>> > supporting Wx GUI loops natively requires compromising the hackability
>>> > and testability of the IPython core, do we *need* to support it?
>>
>> All the libraries I use, I use them through Wx GUI loops. And it is the
>> case of most people under Windows (I am under Linux). The reason behind
>> both statement is that Wx has been historically the easy way to have a
>> powerful GUI shared between major operating systems. Maybe Qt is the way
>> forward now, but it is not yet established. The problem with Wx
>> (recursive Yeilds) comes from the fact that it supports different event
>> loop with different semantics on different platforms, so it comes from
>> that richness.
>>
>> Wx is the toolkit where you will find the largest amount of mature
>> contributions. I know that matplotlib has a variety of different toolkit
>> support, but matplotlib is an exemption.
>>
>> Do we need wx support? I guess not, but I would be jumping ship, and many
>> people would.
>
> Barry,  I think you make a very valid point, but quite unfortunately,
> I have to side with Gael on this one: Wx *today* gives us Chaco,
> Mayavi2, Envisage, TraitsGUI and friends.  That's a LOT of
> functionality that many of us need and use, and we can't really dump
> that user base.  We can look into perhaps refactoring 'ipython
> -wthread' into some ghetto where it doesn't "pollute" the rest of the
> code quite so much, and I'm totally open to that.  But I think *right
> now* dropping Wx would be a mistake.
>
> However, note above my emphasis on "now".  That's because I'm very
> much hoping that PyQt will go LGPL and that over time, Qt will come to
> replace Wx in the roles we're interested in.  If/when that comes to
> pass, we may very well see this discussion in a very different light.

Yes, I agree. It would be a shame to loose all of the Enthought suite
(though don't they also have a Qt backend? I suppose it's not as well
maintained as the Wx backend?), as well as the many other tools.
Once/if PyQt is LGPL, hopefully the community can *start* to
consolidate. If IPython comes out strongly in favor of Qt for future
development, that might help, though I undestand if the IPython
community doesn't want to make that stand.

cheers,
B

>
> Cheers,
>
> f
>


From barrywark at gmail.com  Mon Feb  9 12:20:58 2009
From: barrywark at gmail.com (Barry Wark)
Date: Mon, 9 Feb 2009 09:20:58 -0800
Subject: [IPython-dev] Interactive wx/pylab with no threads
	(PyOS_InputHook)
In-Reply-To: <20090209171914.GF27832@phare.normalesup.org>
References: <6ce0ac130902081608ha95d3c3mbb404e6d04968323@mail.gmail.com>
	<cd7634ce0902082049n6a29ee12wbfdf7f0b6ae39567@mail.gmail.com>
	<20090209060018.GA26350@phare.normalesup.org>
	<cd7634ce0902090917x114ae22aya1293f20a47b45eb@mail.gmail.com>
	<20090209171914.GF27832@phare.normalesup.org>
Message-ID: <cd7634ce0902090920w4f78101fu39ed0bf0a533ee2e@mail.gmail.com>

On Mon, Feb 9, 2009 at 9:19 AM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Mon, Feb 09, 2009 at 09:17:02AM -0800, Barry Wark wrote:
>> > Do we need wx support? I guess not, but I would be jumping ship, and many
>> > people would.
>
>> That's the answer. I was just throwing out the idea. Sounds like it's
>> not a realistic approach at this time.
>
> I just want to point out that I am ashamed of the above sentence
> ('jumping ship'). I was overreacting and being rude, as usual. I would be
> happier if I hadn't writen that sentence.

I, for one, didn't take it as rude; just a concise statement of a
functional requirement for future development ;)

>
> Ga?l
>


From vivainio at gmail.com  Mon Feb  9 13:41:31 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Mon, 9 Feb 2009 20:41:31 +0200
Subject: [IPython-dev] [IPython-user] Problems with "_" syntax output
	cache (ipy_bzr, gettext).
Message-ID: <46cb515a0902091041g6fd14f61idf279a2f7a2b685e@mail.gmail.com>

Apparently, it's ipy_bzr that screws up _. This happens because
ipython does not want to clobber _ when using gettext, and the bzr
plugin(s) use gettext and reassing _.

You can work around this by doing

import __builtin__
del __builtin__.__dict__['_']

After ip.load('ipy_bzr')

Erik, can you try it for a while and report any problems? I'll add
that to ipy_bzr.py if you have good experience with it.

Relevant discussion below.


On Sun, Feb 8, 2009 at 11:58 PM, Erik Tollerud <etolleru at uci.edu> wrote:
> I do not in the profiles, but in ipy_user_conf.py
>
> This prompted me to look through all my settings in ipy_user_conf.py,
> though, and I discovered the offending line:
> ip.load('ipy_bzr')
>
> Apparently the bzr module is somehow screwing up the _ bindings ...
> this is less than ideal, as I would like both to be available...
> should I post it as a bug, or is this something related to my bzr
> setup or something otherwise unique to me?  I tried looking through
> the ipy_bzr.py file and saw nothing obvious but I'm by no means an
> ipapi expert.
>
> Another problem apparently related to switching from ipythonrc:  I no
> longer get tab completion for files - that is, it used to be that I
> could type "open('filename." and hit tab to see all the possible files
> that fit... now this doesn't seem to do anything.  Do you no what (if
> any) option will enable that behavior? It works fine with cd
> filename.<tab> , but it used to work whenever I had any filename
> inside a string...
>
> On Sun, Feb 8, 2009 at 12:32 AM, Ville M. Vainio <vivainio at gmail.com> wrote:
>> On Sun, Feb 8, 2009 at 3:39 AM, Erik Tollerud <etolleru at uci.edu> wrote:
>>> I'm still facing this problem... has no one else encountered it?
>>
>> Not yet.
>>
>> Do you have ipy_defaults imported in your profiles?
>>
>> --
>> Ville M. Vainio
>> http://tinyurl.com/vainio
>>
>
>
>
> --
> Erik Tollerud
> Graduate Student
> Center For Cosmology
> Department of Physics and Astronomy
> 2142 Frederick Reines Hall
> University of California, Irvine
> Office Phone: (949)824-2587
> Cell: (651)307-9409
> etolleru at uci.edu
>



-- 
Ville M. Vainio
http://tinyurl.com/vainio


From ellisonbg.net at gmail.com  Mon Feb  9 16:10:55 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Mon, 9 Feb 2009 13:10:55 -0800
Subject: [IPython-dev] Interactive wx/pylab with no threads
	(PyOS_InputHook)
In-Reply-To: <20090209171914.GF27832@phare.normalesup.org>
References: <6ce0ac130902081608ha95d3c3mbb404e6d04968323@mail.gmail.com>
	<cd7634ce0902082049n6a29ee12wbfdf7f0b6ae39567@mail.gmail.com>
	<20090209060018.GA26350@phare.normalesup.org>
	<cd7634ce0902090917x114ae22aya1293f20a47b45eb@mail.gmail.com>
	<20090209171914.GF27832@phare.normalesup.org>
Message-ID: <6ce0ac130902091310r79cdff6co4e99fd0be9e062ed@mail.gmail.com>

> I just want to point out that I am ashamed of the above sentence
> ('jumping ship'). I was overreacting and being rude, as usual. I would be
> happier if I hadn't writen that sentence.

Don't worry, if you jump ship, we will surely throw you a life
preserver and turn the boat around to pick you up.

Brian


From ondrej at certik.cz  Tue Feb 10 14:52:10 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Tue, 10 Feb 2009 11:52:10 -0800
Subject: [IPython-dev] status of the ipython web frontend
Message-ID: <85b5c3130902101152q31b8cd9bwc04b81a886c0f9e2@mail.gmail.com>

Hi,

I'd like to host some python library on the web, so that people can
play with it, see here for more info:

http://groups.google.com/group/sage-devel/browse_thread/thread/7b904c2d4b5084da

what is the current status with the ipython web frontend? And future
plans? I know we played with that the Sage Days 8 in Austin, but
unfortunately I didn't have time since then to work on that.

Basically, there is sage notebook, knoboo and ipython web frontend, as
far as I know.
The problem is, that it needs so much work to fix all the small bugs,
so that the web frontend is actually usable. Sage notebook is probably
the most advanced, so I think I will just use that.


Ondrej


From ellisonbg.net at gmail.com  Tue Feb 10 15:05:18 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 10 Feb 2009 12:05:18 -0800
Subject: [IPython-dev] [matplotlib-devel] Interactive wx/pylab with no
	threads (PyOS_InputHook)
In-Reply-To: <223008.13097.qm@web62401.mail.re1.yahoo.com>
References: <6ce0ac130902081608ha95d3c3mbb404e6d04968323@mail.gmail.com>
	<223008.13097.qm@web62401.mail.re1.yahoo.com>
Message-ID: <6ce0ac130902101205l14e6f307k7d5d4cc8b46d101@mail.gmail.com>

Michiel,

Thanks for jumping into the discussion.

> I wrote the code in PyGTK that uses PyOS_InputHook for interactivity, as well as the Mac OS X native backend for matplotlib that uses PyOS_InputHook in exactly the same way. PyQT and Tkinter also use PyOS_InputHook, though the code is a bit kludgy on Windows. So I definitely agree that PyOS_InputHook is the right way to go.

Great, I was wondering how the Mac OS X backend works - now I know.  I
will have a look at the code for both PyGTK and OS X.  Hopefully that
will show me more of the best way of handling this.

> Your current code should work, but there's a better way to do it. If I understand the code correctly, you rely on the fact that PyOS_InputHook is called repeatedly by readline, and you use PyOS_InputHook to process wx events that need to be processed at that time.

Yes, at least that is my understanding.  I put in some debug
statements and you could see that it was being called repeatedly.

A better way is to use PyOS_InputHook to start the wx event loop, but
have this event loop check stdin. As soon as some input is available
on stdin, you exit the event loop, which means that PyOS_InputHook
returns, and Python can proceed to handle the command that was just
entered by the user on stdin.
>
> Essentially, think of wx's event loop as sitting in a call to select(), waiting for the next wx event to arrive. You want to add fileno(stdin) to the set of file descriptors watched by select().

I have seen that this is how the PyQt4 implementation handles it.

> There are two advantages to this approach. First, it does not rely on readline calling PyOS_InputHook repeatedly. This is important, since Python may not be using readline at all, and if it is, depending on the Python version and how readline was installed it may call PyOS_InputHook only once.

OK, I was wondering about this.  But, what happens if PyOS_InputHook
is called repeatedly.  Are you not then starting the event loop
multiple times.  Can you say more about what happens in this case?

Second, this approach is more efficient (not wasting processor cycles
going back and forth between readline and PyOS_InputHook), and gives a
better response time (essentially immediate).

That would be very nice as my implementation is less responsive.

> The best place to put this code is in wxPython. Hopefully (I haven't checked this), wx exposes enough of the event loop to allow you to have it watch stdin. This may be an issue, since for example qt4 does not on Windows, which is why the event loop is kludgy with PyQT on Windows. You could have a look at the PyOS_InputHook code in PyGTK (you'll need to get the developer's version of PyGTK, since this code is not yet in an official release). It's actually quite straightforward and you may be able to modify it directly for wx.

Yes, I fully agree with this.  I might end up contacting the wx devs
to get their help on this.  I actually don't know wx at all, so I am
amazed that I got this far.  I will have a look at the PyGTK
implementation.

> It's actually unfortunate that we have to use PyOS_InputHook; all this would be a lot easier if Python itself supported event loops.

Yes, that would be nice!!!

Cheers,

Brian

> --Michiel
>
> --- On Sun, 2/8/09, Brian Granger <ellisonbg.net at gmail.com> wrote:
>
>> From: Brian Granger <ellisonbg.net at gmail.com>
>> Subject: [matplotlib-devel] Interactive wx/pylab with no threads (PyOS_InputHook)
>> To: matplotlib-devel at lists.sourceforge.net, "IPython Development list" <ipython-dev at scipy.org>
>> Date: Sunday, February 8, 2009, 7:08 PM
>> IPython and matplotlib devs,
>>
>> Over the weekend I have been playing around to see if it is
>> possible
>> to do interactive GUI work with wx from IPython *without
>> using
>> threads*.  The idea here is to use PyOS_InputHook.
>> Currently, recent
>> versions of PyQt4 and PyGTK do this and if we can get wx
>> working, we
>> can probably get rid of IPython's subtle threaded
>> shells that
>> currently allow interactive GUIs to work.
>>
>> I am attaching a Cython module that mostly works.  Here is
>> a simple
>> example that works in IPython (without the -wthread
>> option!)
>>
>> In [1]: import pyximport
>>
>> In [2]: pyximport.install()
>>
>> In [3]: import inputhook
>>
>> In [4]: inputhook.set_input_hook()
>>
>> In [5]: import wx
>>
>> In [6]: app = wx.PySimpleApp()
>>
>> In [7]: app.MainLoop()
>>
>> In [8]: f = wx.Frame(None,-1,"Look mom, no
>> threads!")
>>
>> In [9]: f.Show()
>> Out[9]: True
>>
>> The docstring of the module also has a matplotlib example.
>> This
>> really does work and I am pretty sure it will also work in
>> plain
>> vanilla python as well.  There are a few issues to work
>> out:
>>
>> * When frame are closed by clicking the red button or the
>> "X", the
>> Windows don't close.  In addition, in matplotlib, this
>> causes further
>> problems.
>>
>> * In the current matplotlib backend wx.Yield() is called in
>> a way that
>> is not safe as far as protecting against recursive calls to
>> Yield.  I
>> think it should be called in this way:
>>
>> app = wx.GetApp()
>> if app is not None:
>>   app.Yield(True)
>>
>> * I don't think that interupts work yet, but I
>> haven't tested this
>> thoroughly yet.
>>
>> I don't have any more time to work on this right now,
>> but I at least
>> wanted to share my findings with both IPython and
>> matplotlib devs.  It
>> would be great if someone familiar with wx could try to
>> figure out the
>> remaining issues.  If there are no takers here, I might
>> eventually see
>> if wxpython itself is interested in this code (that is
>> probably where
>> it really belongs anyway).
>>
>> Cheers,
>>
>> Brian
>> ------------------------------------------------------------------------------
>> Create and Deploy Rich Internet Apps outside the browser
>> with Adobe(R)AIR(TM)
>> software. With Adobe AIR, Ajax developers can use existing
>> skills and code to
>> build responsive, highly engaging applications that combine
>> the power of local
>> resources and data with the reach of the web. Download the
>> Adobe AIR SDK and
>> Ajax docs to start building applications
>> today-http://p.sf.net/sfu/adobe-com_______________________________________________
>> Matplotlib-devel mailing list
>> Matplotlib-devel at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
>
>
>
>


From ellisonbg.net at gmail.com  Tue Feb 10 15:06:55 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 10 Feb 2009 12:06:55 -0800
Subject: [IPython-dev] status of the ipython web frontend
In-Reply-To: <85b5c3130902101152q31b8cd9bwc04b81a886c0f9e2@mail.gmail.com>
References: <85b5c3130902101152q31b8cd9bwc04b81a886c0f9e2@mail.gmail.com>
Message-ID: <6ce0ac130902101206i16fe387hfa12e769efc02e38@mail.gmail.com>

For now, you should absolutely use the sage notebook.  It looks like
we may get some funding to work on the IPython core this spring and
summer, which would move things forward.  But, if you want something
today, Sage is hard to beat!

Cheers,

Brian

On Tue, Feb 10, 2009 at 11:52 AM, Ondrej Certik <ondrej at certik.cz> wrote:
> Hi,
>
> I'd like to host some python library on the web, so that people can
> play with it, see here for more info:
>
> http://groups.google.com/group/sage-devel/browse_thread/thread/7b904c2d4b5084da
>
> what is the current status with the ipython web frontend? And future
> plans? I know we played with that the Sage Days 8 in Austin, but
> unfortunately I didn't have time since then to work on that.
>
> Basically, there is sage notebook, knoboo and ipython web frontend, as
> far as I know.
> The problem is, that it needs so much work to fix all the small bugs,
> so that the web frontend is actually usable. Sage notebook is probably
> the most advanced, so I think I will just use that.
>
>
> Ondrej
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
>


From ondrej at certik.cz  Tue Feb 10 15:18:09 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Tue, 10 Feb 2009 12:18:09 -0800
Subject: [IPython-dev] status of the ipython web frontend
In-Reply-To: <6ce0ac130902101206i16fe387hfa12e769efc02e38@mail.gmail.com>
References: <85b5c3130902101152q31b8cd9bwc04b81a886c0f9e2@mail.gmail.com>
	<6ce0ac130902101206i16fe387hfa12e769efc02e38@mail.gmail.com>
Message-ID: <85b5c3130902101218s3e566cf1ne74510ac327099a5@mail.gmail.com>

On Tue, Feb 10, 2009 at 12:06 PM, Brian Granger <ellisonbg.net at gmail.com> wrote:
> For now, you should absolutely use the sage notebook.  It looks like
> we may get some funding to work on the IPython core this spring and

I see, that'd be cool.

> summer, which would move things forward.  But, if you want something
> today, Sage is hard to beat!

Yes, It's good. So I'll use it.

Thanks,
Ondrej


From fperez.net at gmail.com  Tue Feb 10 16:09:51 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 10 Feb 2009 13:09:51 -0800
Subject: [IPython-dev] IPython in the 'IMSL studio'...
Message-ID: <db6b5ecc0902101309l44d10f7cw94fff175df20b24@mail.gmail.com>

http://www.vni.com/products/imsl/pyimslstudio/keyFeatures.php

Cool :)

f


From fperez.net at gmail.com  Tue Feb 10 20:32:54 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 10 Feb 2009 17:32:54 -0800
Subject: [IPython-dev] Looking towards 0.10...
Message-ID: <db6b5ecc0902101732r10f37d8xf995410c8705a3f9@mail.gmail.com>

Hi all,

there's been a bunch of recent work done on various branches, and we
should probably look into putting an 0.10 out in a few weeks, once we
can merge everything in.  I have my own branch to clean up before I
post it up for review, but even if things aren't perfect, I think we
should make a new version available.

Right off the top of my head I know we have:

- Laurent's recent branch, Gael finished the review and it's ready to merge.
- Brian's recent win32 fixes, I reviewed it and with a few fixes it can go.
- Jorgen's had some lingering test failures, but once those get
resolved it can go, he's addressed all other concerns.
- My main trunk needs to be pushed up for review.

Pending review, these two look easy:

- https://code.launchpad.net/~villemvainio/ipython/ip-sets-warn/+merge/2456
- https://code.launchpad.net/~vvatsa/ipython/ipcluster-dev/+merge/2352

This one seemed to be almost ready:
- https://code.launchpad.net/~villemvainio/ipython/ip0-unittests-1/+merge/1075

- I'll push my trunk hopefully in a couple of hours.

- What am I missing?


If this looks like it's everything, we should be able to wrap up in a
couple of weeks, me thinks...

I think I'd rather put 0.10 out basically as a big merge of this work,
so that we can then pay some attention to a lot of recent things that
have been put on the  list but that are not in any branches.  I'm
thinking of things like Robert's lprof and  pretty.py, his magics
code, and others that  I probably let slip by...

What do y'all think?

Cheers,

f


From ldufrechou at marport.com  Wed Feb 11 16:21:16 2009
From: ldufrechou at marport.com (=?ISO-8859-1?Q?Laurent_Dufr=E9chou?=)
Date: Wed, 11 Feb 2009 22:21:16 +0100
Subject: [IPython-dev] Hum... tried to merge gael branch...
Message-ID: <7ced318f0902111321v6a9dff3ds405780485b625478@mail.gmail.com>

Hello,
I've tried to merge gael branch into mine to keep it's frontend in
sync with my wx-gui.
(Is it the "way ti do it?" or do I need to wait 0.10 merging?)
Sadly the effects where (seen so far :) ):

R * scripts/wxIpython => scripts/ipython-wx
   * IPython/gui/wx/wxIPython.py


/script/wxIPython to have been erased and a new one appeared
ipython-wx.(that don't work because don't got a def main())
/Ipython/gui/wx/wxIPython.py has been chmoded -x :(
(what does the little '*' means? if you don't know will google :))

So: 1) can I keep my chmod +x ? it so easier, and I want it! :p
       2) If you want to rename the script name,  should we warn
package maintainer? (I've seen a recent mail from debian maintainer
requesting doc)
            Is it automatic? For the def main(), well no problem I can
add something to support that.

Laurent


From ldufrechou at marport.com  Wed Feb 11 16:39:20 2009
From: ldufrechou at marport.com (=?ISO-8859-1?Q?Laurent_Dufr=E9chou?=)
Date: Wed, 11 Feb 2009 22:39:20 +0100
Subject: [IPython-dev] Hum... tried to merge gael branch...
In-Reply-To: <7ced318f0902111321v6a9dff3ds405780485b625478@mail.gmail.com>
References: <7ced318f0902111321v6a9dff3ds405780485b625478@mail.gmail.com>
Message-ID: <7ced318f0902111339t4810f646hc38e2db7b1a3b617@mail.gmail.com>

Hum forget about the def Main... I've modified my code recently that
make the script die...
Will correct this. In fact I think the script has only be renamed!
shame on me :)

2009/2/11 Laurent Dufr?chou <ldufrechou at marport.com>:
> Hello,
> I've tried to merge gael branch into mine to keep it's frontend in
> sync with my wx-gui.
> (Is it the "way ti do it?" or do I need to wait 0.10 merging?)
> Sadly the effects where (seen so far :) ):
>
> R * scripts/wxIpython => scripts/ipython-wx
>   * IPython/gui/wx/wxIPython.py
>
>
> /script/wxIPython to have been erased and a new one appeared
> ipython-wx.(that don't work because don't got a def main())
> /Ipython/gui/wx/wxIPython.py has been chmoded -x :(
> (what does the little '*' means? if you don't know will google :))
>
> So: 1) can I keep my chmod +x ? it so easier, and I want it! :p
>       2) If you want to rename the script name,  should we warn
> package maintainer? (I've seen a recent mail from debian maintainer
> requesting doc)
>            Is it automatic? For the def main(), well no problem I can
> add something to support that.
>
> Laurent
>


From laurent.dufrechou at gmail.com  Wed Feb 11 17:05:34 2009
From: laurent.dufrechou at gmail.com (Laurent Dufrechou)
Date: Wed, 11 Feb 2009 23:05:34 +0100
Subject: [IPython-dev] ctrl+c in gui, event loops,
	sure i'm missing something...
Message-ID: <7ced318f0902111405y1118b2d9ya75e4c2ff825afbf@mail.gmail.com>

Hi guys, want to have a reflexion with the gui gurus over there ;)

Since i've written my wx ipython gui I've wanted from the beginning to
support ctrl+c.
Because it is so easy to do a code that lock it is i think a must have feature.
My gui run a wx main loop. A specific widget is created to instanciate
a scintilla textctrl. An ipython object is also created that handle
the current state of the ipython interpreter.
when the user push enter, as in ipython console application I push the
line in the interpreter and this one run the code loop.
Here is how i solved the problem:

If user want to have ctrl+c support, I has to activate a "threading option".
When this option is activated each time a line is pushed in the
interpreter the line is pushed in thread code executor that makes the
ipython core run the line  in a little tread. When the line is
processed, the result is displayed and the thread killed.
If user launched an infinite loop:
while 1:
  a=1

then I can catch the ctrl+c key becuase my gui still responsive and i
send a ctrl+c to the thread. That's all and it works well so far.

Now that i'm much satisfisfied with current state, i want to improve
the whole reflexion a step further.
In ipython console (in bash) you can easily do a ctrl+c.
WHY CAN'T I do that without threading in a gui... It could be easily
removed if the could get the ctrl+c and then send it to the gui to
make it stop.

So is there a way perhaps to do it the other way, that is:
make the gui run the ipython line and get another "event magic loop"
in another thread to catch the ctrl+c and send it to wx.mainloop?
(do it the other way in fact :)) so it will remove all the thread
constraint of my code regarding ipython integration. Well there is not
a lot of constraints but...

The fact is that i'm not aware of how the wx/qt/... event loop works
and perhaps someone there has better idea ?

Any clever magic idea welcomed!

Cheers,
Laurent


From gael.varoquaux at normalesup.org  Wed Feb 11 17:05:28 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Wed, 11 Feb 2009 23:05:28 +0100
Subject: [IPython-dev] Hum... tried to merge gael branch...
In-Reply-To: <7ced318f0902111339t4810f646hc38e2db7b1a3b617@mail.gmail.com>
References: <7ced318f0902111321v6a9dff3ds405780485b625478@mail.gmail.com>
	<7ced318f0902111339t4810f646hc38e2db7b1a3b617@mail.gmail.com>
Message-ID: <20090211220528.GB22749@phare.normalesup.org>

On Wed, Feb 11, 2009 at 10:39:20PM +0100, Laurent Dufr?chou wrote:
> Hum forget about the def Main... I've modified my code recently that
> make the script die...
> Will correct this. In fact I think the script has only be renamed!
> shame on me :)

And in addition this has not happened in my branch, but in the main one
(trunk), and I only got the change by syncing with the trunk. You can
check this by doing 'bzr log ipython-wx'.

You should sync to the trunk every once in a while, it would make your
life easier tracking what happens.

Cheers,

Ga?l


From ldufrechou at marport.com  Wed Feb 11 17:19:35 2009
From: ldufrechou at marport.com (=?ISO-8859-1?Q?Laurent_Dufr=E9chou?=)
Date: Wed, 11 Feb 2009 23:19:35 +0100
Subject: [IPython-dev] Hum... tried to merge gael branch...
In-Reply-To: <20090211220528.GB22749@phare.normalesup.org>
References: <7ced318f0902111321v6a9dff3ds405780485b625478@mail.gmail.com>
	<7ced318f0902111339t4810f646hc38e2db7b1a3b617@mail.gmail.com>
	<20090211220528.GB22749@phare.normalesup.org>
Message-ID: <7ced318f0902111419r7608733ao7c9245204f9dcae2@mail.gmail.com>

Yeah! This is what I've just seen :)
In fact i think only the chmod +x has changed I think...

If I merge with trunk, then I need to commit back to my branch. Does
it will have some side effect when merging back my branch????

For example can I safely do:
bzr merge ipython main branch +
bzr merge gael branch

and than later merge my code into main one?

If that works, wow bzr is good :)!

Sorry for newbiness question, but I don't want to destroy my branch!!!!

2009/2/11 Gael Varoquaux <gael.varoquaux at normalesup.org>:
> On Wed, Feb 11, 2009 at 10:39:20PM +0100, Laurent Dufr?chou wrote:
>> Hum for just get about the def Main... I've modified my code recently that
>> make the script die...
>> Will correct this. In fact I think the script has only be renamed!
>> shame on me :)
>
> And in addition this has not happened in my branch, but in the main one
> (trunk), and I only got the change by syncing with the trunk. You can
> check this by doing 'bzr log ipython-wx'.
>
> You should sync to the trunk every once in a while, it would make your
> life easier tracking what happens.
>
> Cheers,
>
> Ga?l
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
>


From gael.varoquaux at normalesup.org  Wed Feb 11 17:46:35 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Wed, 11 Feb 2009 23:46:35 +0100
Subject: [IPython-dev] Hum... tried to merge gael branch...
In-Reply-To: <7ced318f0902111419r7608733ao7c9245204f9dcae2@mail.gmail.com>
References: <7ced318f0902111321v6a9dff3ds405780485b625478@mail.gmail.com>
	<7ced318f0902111339t4810f646hc38e2db7b1a3b617@mail.gmail.com>
	<20090211220528.GB22749@phare.normalesup.org>
	<7ced318f0902111419r7608733ao7c9245204f9dcae2@mail.gmail.com>
Message-ID: <20090211224635.GC22749@phare.normalesup.org>

On Wed, Feb 11, 2009 at 11:19:35PM +0100, Laurent Dufr?chou wrote:
> Yeah! This is what I've just seen :)
> In fact i think only the chmod +x has changed I think...

> If I merge with trunk, then I need to commit back to my branch. Does
> it will have some side effect when merging back my branch????

> For example can I safely do:
> bzr merge ipython main branch +
> bzr merge gael branch

> and than later merge my code into main one?

> If that works, wow bzr is good :)!

As long as there are no conflicts between my branch and trunk, this will
work. Avoiding conflicts with a branch and trunk is a good policy,
because it renders this kind of development pattern possible. If you have
to create conflicts, then you are on your own...

Ga?l


From ldufrechou at marport.com  Wed Feb 11 18:26:16 2009
From: ldufrechou at marport.com (=?ISO-8859-1?Q?Laurent_Dufr=E9chou?=)
Date: Thu, 12 Feb 2009 00:26:16 +0100
Subject: [IPython-dev] Hum... tried to merge gael branch...
In-Reply-To: <20090211224635.GC22749@phare.normalesup.org>
References: <7ced318f0902111321v6a9dff3ds405780485b625478@mail.gmail.com>
	<7ced318f0902111339t4810f646hc38e2db7b1a3b617@mail.gmail.com>
	<20090211220528.GB22749@phare.normalesup.org>
	<7ced318f0902111419r7608733ao7c9245204f9dcae2@mail.gmail.com>
	<20090211224635.GC22749@phare.normalesup.org>
Message-ID: <7ced318f0902111526g724d1b9fu3bd807a793cc7eb2@mail.gmail.com>

Ok, I've just managed to merge lp:ipython in to my branch.
Last question, if I merge your branch gael , and don't know why, but
your branch will not be merged in next release.
(for any reason, or for example, you say your code is not good and you
will merge instead gael2_newbranch)
Does my branch will then be delayed (because of the dependance) ?
Or when final merge will arrive, fernando will be able to select the
"good" part of my branch...
Perhaps the bad thing is that I've  branched the  full main dev repos
instead of branching only ./gui directory?

Once your branch will be merged be prepared for a set of question on
how to derivate and tune your frontend to make it the same (in
appearance) to the one i did ;)
Hope it will be easy! Ideally and if possible and want to get the same
behaviour as mine, like classical ipython console.

By the way have you seen bug
https://bugs.launchpad.net/ipython/+bug/311490?

Got the same bug there, it is related to
iplib.py:389:        self.user_ns['_oh'] = self.output_hist
iplib.py:394:        self.user_ns['Out'] = self.output_hist

but can't find where self.output_hist is refreshed. Will continue to
dig this tomorow.
If I found something will tell you.

Cheers,
Laurent

2009/2/11 Gael Varoquaux <gael.varoquaux at normalesup.org>:
> On Wed, Feb 11, dmanage2009 at 11:19:35PM +0100, Laurent Dufr?chou wrote:
>> Yeah! This is what I've just seen :)
>> In fact i think only the chmod +x has changed I think...
>
>> If I merge with trunk, then I need to commit back to my branch. Does
>> it will have some side effect when merging back my branch????
>
>> For example can I safely do:
>> bzr merge ipython main branch +
>> bzr merge gael branch
>
>> and than later merge my code into main one?
>
>> If that works, wow bzr is good :)!
>
> As long as there are no conflicts between my branch and trunk, this will
> work. Avoiding conflicts with a branch and trunk is a good policy,
> because it renders this kind of development pattern possible. If you have
> to create conflicts, then you are on your own...
>
> Ga?l
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
>


From ellisonbg.net at gmail.com  Wed Feb 11 19:42:30 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 11 Feb 2009 16:42:30 -0800
Subject: [IPython-dev] Careful about locks on lp/bzr
Message-ID: <6ce0ac130902111642o277029f0s8935d9c78fb151f3@mail.gmail.com>

Hi,

I just went to push to trunk after I did a merge and got this:

Unable to obtain lock lp-45628560:///~ipython-dev/ipython/trunk/.bzr/branch/lock
held by villemvainio at bazaar.launchpad.net on host crowberry [process #12127]
locked 28 hours, 57 minutes ago
Will continue to try until 00:37:10, unless you press Ctrl-C
If you're sure that it's not being modified, use bzr break-lock
lp-45628560:///~ipython-dev/ipython/trunk/.bzr/branch/lock
bzr: ERROR: Could not acquire lock "(remote lock)"

I went ahead and did the break-lock thing.  But in the future, let's
be careful not to put locks on trunk.  So, next question - how does
this happen in the first place?

Cheers,

Brian


From gael.varoquaux at normalesup.org  Thu Feb 12 00:39:09 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 12 Feb 2009 06:39:09 +0100
Subject: [IPython-dev] Careful about locks on lp/bzr
In-Reply-To: <6ce0ac130902111642o277029f0s8935d9c78fb151f3@mail.gmail.com>
References: <6ce0ac130902111642o277029f0s8935d9c78fb151f3@mail.gmail.com>
Message-ID: <20090212053909.GB30330@phare.normalesup.org>

On Wed, Feb 11, 2009 at 04:42:30PM -0800, Brian Granger wrote:
> Unable to obtain lock lp-45628560:///~ipython-dev/ipython/trunk/.bzr/branch/lock
> held by villemvainio at bazaar.launchpad.net on host crowberry [process #12127]
> locked 28 hours, 57 minutes ago
> Will continue to try until 00:37:10, unless you press Ctrl-C
> If you're sure that it's not being modified, use bzr break-lock
> lp-45628560:///~ipython-dev/ipython/trunk/.bzr/branch/lock
> bzr: ERROR: Could not acquire lock "(remote lock)"

> I went ahead and did the break-lock thing.  But in the future, let's
> be careful not to put locks on trunk.  So, next question - how does
> this happen in the first place?

Strange, I had the same problem and this is the first time that happened
to me. And it was with the same hostname (crowberry?), but my user, and
on my branch.

Ga?l


From vivainio at gmail.com  Thu Feb 12 01:45:40 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Thu, 12 Feb 2009 08:45:40 +0200
Subject: [IPython-dev] Careful about locks on lp/bzr
In-Reply-To: <6ce0ac130902111642o277029f0s8935d9c78fb151f3@mail.gmail.com>
References: <6ce0ac130902111642o277029f0s8935d9c78fb151f3@mail.gmail.com>
Message-ID: <46cb515a0902112245r3534beend58cfaad6e0423b8@mail.gmail.com>

On Thu, Feb 12, 2009 at 2:42 AM, Brian Granger <ellisonbg.net at gmail.com> wrote:

> I went ahead and did the break-lock thing.  But in the future, let's
> be careful not to put locks on trunk.  So, next question - how does
> this happen in the first place?

It happens when a push is interrupted in mid-air (ctrl+c etc).

It's no biggie to do break-lock, however.

-- 
Ville M. Vainio
http://tinyurl.com/vainio


From gael.varoquaux at normalesup.org  Thu Feb 12 03:52:56 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 12 Feb 2009 09:52:56 +0100
Subject: [IPython-dev] ctrl+c in gui, event loops,
	sure i'm missing	something...
In-Reply-To: <7ced318f0902111405y1118b2d9ya75e4c2ff825afbf@mail.gmail.com>
References: <7ced318f0902111405y1118b2d9ya75e4c2ff825afbf@mail.gmail.com>
Message-ID: <20090212085256.GC20814@phare.normalesup.org>

On Wed, Feb 11, 2009 at 11:05:34PM +0100, Laurent Dufrechou wrote:
> Any clever magic idea welcomed!

This is a recurrent problem that we have been discussing more than once
on this mailing list, although not in these terms.

You are aware of the problems with a single event loop (blocks when doing
calculations) and with threads (fundamentally unreliable and prone to
crashing). If I can rephrase to questions that I can try to answer, I
would say that you have two questions: i) why are we in a situation
different than eg Matlab, or IPython in a terminal? ii) How can we do
better and have something robust, and yet provide reaction to events
(including interruption) when calculations are running.

i) Why are we in a situation different than eg Matlab, or IPython in a
terminal?

With a wxPython frontend for IPython, we are in a very specific
situation: we have an environment for interaction with the user (our GUI)
that is completely embedded and mixed with our execution engine. This is
very powerful, because this means we can couple interaction with what is
being run in the execution engine. 

However, Python is not a language terribly good at isolation. Not only
does it have an incredible amount of global states, but also it has no
thread-safe memory model. Memory coherence across threads in a shared
memory model (ie not an actor-like or message-passing model) is a very
hard problem. AFAIK only Java and .net have solved it, and with a
significant investment in the development of their virtual machine.

We are thus cursed with the fact that our user-interaction code and our
execution engine, when running in the same Python virtual machine cannot
be isolated, and trying to simulate this isolation will lead to quirks,
crashes, or both.

Matlab, or simply running IPython in a terminal, are quite different,
because the interaction with the user is separated from the execution
environment. In this regard, if you want to look at a Python
implementation of this concept, the Sage notebook is a good example, and
Ctrl-C does work quite reliably there.

ii) How can we do better and have something robust, and yet provide
reaction to events (including interruption) when calculations are
running.

I am convinced the solution is going to multiple processes. Actually the
important thing is to have multiple virtual machines between which we
share a	limited amount of states. Going multi-process is a simple way of
doing this. From a pragmatic point of view, you have two option: using
IPython1 to push the execution engine out of process with the IPython1
mechanism, or using the multiprocessing module to keep an thread-like
API, but with several processes.

Brian and Fernando had to fight with a similar problem when exposing the
execution engine over the network: the network-listening code could be
freezed by the execution engine. I seem to remember that they had to use
two processes for an engine: one to run the code, and the other one to
supervise it, but it would be interesting to hear them share there
experience.

My 2 centimes,

Ga?l



From laurent.dufrechou at gmail.com  Thu Feb 12 04:09:32 2009
From: laurent.dufrechou at gmail.com (=?us-ascii?Q?Laurent_Dufrechou?=)
Date: Thu, 12 Feb 2009 10:09:32 +0100
Subject: [IPython-dev] ctrl+c in gui, event loops,
	sure i'm missing 	something...
In-Reply-To: <6ce0ac130902111615k253615b7p32ead9647c9ca92c@mail.gmail.com>
References: <7ced318f0902111405y1118b2d9ya75e4c2ff825afbf@mail.gmail.com>
	<6ce0ac130902111615k253615b7p32ead9647c9ca92c@mail.gmail.com>
Message-ID: <4993e74e.06c8100a.6899.188f@mx.google.com>

Hi Brian,

> > If user want to have ctrl+c support, I has to activate a "threading
> option".
> > When this option is activated each time a line is pushed in the
> > interpreter the line is pushed in thread code executor that makes the
> > ipython core run the line  in a little tread.
> 
> The IPython core cannot run user code in a thread (at least, not in a
> thread safe manner).  Here is the reason:
> 
> If there are wx/GUI objects in the IPython core that the user is
> working with (for plotting through matplotlib, etc/), both wx itself
> and user code can and will perform operations on those objects.
> Unless you protect all such operations with locks, you can easily get
> into trouble.  Summary:  a GUI enabled IPython has to be run in the
> same thread as the GUI itself.
> 
> Plus, I don't see what running the code in a thread buys you in this
> case.  In fact, because of the subtle interaction between signals and
> threads, I would even say that using threads makes things more
> complicated.  Each time user code is run, can't you just install a
> signal handler that will interupt the user code directly?
> 

Well, thx to asynchroneous raise in the thread I'm able to interrupt the
code.
Interaction is not that bad, just being careful when calling wx code inside
ipython.
So that's true that you can't run wx code into the shell currently.
That's why I'm looking for another possible solution.

> 
> When the line is
> > processed, the result is displayed and the thread killed.
> > If user launched an infinite loop:
> > while 1:
> >  a=1
> >
> > then I can catch the ctrl+c key becuase my gui still responsive and i
> > send a ctrl+c to the thread. That's all and it works well so far.
> 
> I don't think you can send signals to threads - only processes.
> Fernando has hacked this type of thing before though, so he might know
> better.
> 

Yep you can, found a thread class to do that. Check ./gui/wx/thread_ex.py.
With this I'm able to raise asynchroneous interrupts.

> > Now that i'm much satisfisfied with current state, i want to improve
> > the whole reflexion a step further.
> > In ipython console (in bash) you can easily do a ctrl+c.
> > WHY CAN'T I do that without threading in a gui... It could be easily
> > removed if the could get the ctrl+c and then send it to the gui to
> > make it stop.
> 
> I think you can, you may have to install your own signal handlers if
> wx/qt install their own (which they probably do).
>

Hum, will write to wx mailing list to see if it is possible to catch such
event, the problem I see
is that I'm processing the event 'enter' when running ipython loop. So not
sure another event can be catched at the same time asynchroneously :(





From laurent.dufrechou at gmail.com  Thu Feb 12 05:11:40 2009
From: laurent.dufrechou at gmail.com (Laurent Dufrechou)
Date: Thu, 12 Feb 2009 11:11:40 +0100
Subject: [IPython-dev] ctrl+c in gui, event loops,
	sure i'm missing 	something...
In-Reply-To: <20090212085256.GC20814@phare.normalesup.org>
References: <7ced318f0902111405y1118b2d9ya75e4c2ff825afbf@mail.gmail.com>
	<20090212085256.GC20814@phare.normalesup.org>
Message-ID: <7ced318f0902120211t45381d63k37a3015c4f7f5db2@mail.gmail.com>

Ok thanks a lot gael, I better understand for the console now.
I really understand the virtual machine concept and isolation. And all
you describe is true when using treading.
(the isolation process si sooo true...)
So my next step would be to use ipython1 frontend + network (to
simulate process).
Since what you've written it is clear that it is the only way ... the
ipython1 engine :)

Perhaps it has ever been discussed, but reformulating in my own words
help me clarify the things :/

So let speak about a plan. ("ipython1" is for me the new core able to
launch remote code)
I've played a little with "ipython1" before, and was able to launch
client /server to execute deported script on multiple machines.
So I think we'll need to have the same thing here:
A gui will need to launch a 'ipython1" core engine (easy no?) that
will remotly(or locally) connect to a slave engine.
(if local machine, has to be launched by the gui)
(perhaps all I say is a little bit outdated, so feel free to crush me :) )

So the fact is if networking/ipython1 core is needed, then wx must be
integrataed with twisted, and I think it is not the case currently.
(If I well understood the concept and from recent mail i've readen). True?
--> Is that hard, looking current "ipython1" state to play with a
twisted wx loop?
Anyone ever played with it? Anything planned?
--> Is the wx sycnhroneous frontend is ready and able to deal with ipython1?
--> If not, is it a good basis, and how far are we from this state?

Regards,
Laurent


From prabhu at aero.iitb.ac.in  Thu Feb 12 05:25:46 2009
From: prabhu at aero.iitb.ac.in (Prabhu Ramachandran)
Date: Thu, 12 Feb 2009 15:55:46 +0530
Subject: [IPython-dev] [matplotlib-devel] Interactive wx/pylab with no
 threads	(PyOS_InputHook)
In-Reply-To: <20090209062945.GC26350@phare.normalesup.org>
References: <6ce0ac130902081608ha95d3c3mbb404e6d04968323@mail.gmail.com>
	<20090209062945.GC26350@phare.normalesup.org>
Message-ID: <4993F92A.5070107@aero.iitb.ac.in>

On 02/09/09 11:59, Gael Varoquaux wrote:
> On Sun, Feb 08, 2009 at 04:08:31PM -0800, Brian Granger wrote:
>> * In the current matplotlib backend wx.Yield() is called in a way that
>> is not safe as far as protecting against recursive calls to Yield.  I
>> think it should be called in this way:
> 
>> app = wx.GetApp()
>> if app is not None:
>>   app.Yield(True)
> 
> The problem I see with this approach is that arbitrary wx programs will
> always be doing this. The matplotlib guys can fix matplotib not to do
> this. I can fix Mayavi not to do this, but there are many more wx
> programs. And anyhow, most of the time, Yield should not be called, as it
> is a hack. Unfortunately, you often end up having to call it. :(

Just nit picking: You'd really have to modify traits and pyface for 
this. Mayavi doesn't start the mainloop.

cheers,
prabhu


From gael.varoquaux at normalesup.org  Thu Feb 12 05:39:19 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 12 Feb 2009 11:39:19 +0100
Subject: [IPython-dev] ctrl+c in gui, event loops,
	sure i'm missing	something...
In-Reply-To: <7ced318f0902120211t45381d63k37a3015c4f7f5db2@mail.gmail.com>
References: <7ced318f0902111405y1118b2d9ya75e4c2ff825afbf@mail.gmail.com>
	<20090212085256.GC20814@phare.normalesup.org>
	<7ced318f0902120211t45381d63k37a3015c4f7f5db2@mail.gmail.com>
Message-ID: <20090212103919.GA24736@phare.normalesup.org>

On Thu, Feb 12, 2009 at 11:11:40AM +0100, Laurent Dufrechou wrote:
> So the fact is if networking/ipython1 core is needed, then wx must be
> integrataed with twisted, and I think it is not the case currently.
> (If I well understood the concept and from recent mail i've readen). True?
> --> Is that hard, looking current "ipython1" state to play with a
> twisted wx loop?
> Anyone ever played with it? Anything planned?
> --> Is the wx sycnhroneous frontend is ready and able to deal with ipython1?

No as it is, but the execution engine (an IPython1 instance) is
reasonnably well abstracted, and you should be able to instanciate one
that is out of processes.

Now you will have a few big problems. The first one is that you basically
become asynchronous. You have to deal with deferred and all that. You
need to look at the work by Barry Wark, in the asynchronous front end
base, and will need to derive a new class using the asynchronous base and
the frontend base, rather than the synchronous base. The code has been
thought to allow this, but of course, there will be glitches.

The second problem is that the code allowing to have ipython0 features is
a hack that is incompatible with out-of-process execution. Briefly, you
need to share the namespace of the two engine (the ipython0 and the
ipython1) to get magics and code completion. The clean way out of this is
to refactor ipython0 to port features to an ipython1 engine. This is what
we have been talking about doing for a long while and has been delayed
all the time. I don't have a good answer here.

Ga?l


From prabhu at aero.iitb.ac.in  Thu Feb 12 05:40:21 2009
From: prabhu at aero.iitb.ac.in (Prabhu Ramachandran)
Date: Thu, 12 Feb 2009 16:10:21 +0530
Subject: [IPython-dev] Interactive wx/pylab with no threads
	(PyOS_InputHook)
In-Reply-To: <6ce0ac130902081608ha95d3c3mbb404e6d04968323@mail.gmail.com>
References: <6ce0ac130902081608ha95d3c3mbb404e6d04968323@mail.gmail.com>
Message-ID: <4993FC95.9070705@aero.iitb.ac.in>

On 02/09/09 05:38, Brian Granger wrote:
> IPython and matplotlib devs,
> 
> Over the weekend I have been playing around to see if it is possible
> to do interactive GUI work with wx from IPython *without using
> threads*.  The idea here is to use PyOS_InputHook.  Currently, recent
> versions of PyQt4 and PyGTK do this and if we can get wx working, we
> can probably get rid of IPython's subtle threaded shells that
> currently allow interactive GUIs to work.
> 
> I am attaching a Cython module that mostly works.  Here is a simple
> example that works in IPython (without the -wthread option!)
[...]
> I don't have any more time to work on this right now, but I at least
> wanted to share my findings with both IPython and matplotlib devs.  It
> would be great if someone familiar with wx could try to figure out the
> remaining issues.  If there are no takers here, I might eventually see
> if wxpython itself is interested in this code (that is probably where
> it really belongs anyway).

This is cool!  It works with mayavi, which is a pretty demanding test. 
I did run into problems with pyximport messing up on some imports but 
manually importing the inputhook.so fixed those.  The interactive 
response is not snappy but it definitely works without a problem.

One problem I can see with this is that while it does eliminate the 
threading, there are still issues with multiple toolkits.  It would be 
neat if there were a system where you could mix toolkits too -- it looks 
like it should be possible to support this though.

cheers,
prabhu


From vivainio at gmail.com  Thu Feb 12 06:16:00 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Thu, 12 Feb 2009 13:16:00 +0200
Subject: [IPython-dev] ctrl+c in gui, event loops,
	sure i'm missing 	something...
In-Reply-To: <20090212085256.GC20814@phare.normalesup.org>
References: <7ced318f0902111405y1118b2d9ya75e4c2ff825afbf@mail.gmail.com>
	<20090212085256.GC20814@phare.normalesup.org>
Message-ID: <46cb515a0902120316t2ae90aeaxb06d5bbe0a07d771@mail.gmail.com>

On Thu, Feb 12, 2009 at 10:52 AM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:

> I am convinced the solution is going to multiple processes. Actually the
> important thing is to have multiple virtual machines between which we
> share a limited amount of states. Going multi-process is a simple way of

I agree that multi-process approach is the way to go, when ipython
does not need to access the gui directly (i.e. you can't do rapid gui
code prototyping). But the whole thing should not be overly
complicated since we already have multiple gui frontends - and all
they do is in essence passing strings around (text to execute, text to
display). It shouldn't really be that problematic to put a process
boundary in between. Perhaps the key is to avoid using sockets and
such, but just

- Launch frontend
- From frontend, launch backend using subprocess.Popen (or QProcess, or ...)
- Everything going to stdout can be displayed normally on frontend,
unless it's something special like get_completions_reply data
structure. We don't need to do any stdout redirection because
subprocess launch did it already
- On ctrl+C / interrupt button, send SIGINT to backend
- Standard input will be special, because it may contain "control"
commands like get_completions('foo...') that require special handling
- On linux, we could experiment with pty's to make the subprocess responsive

Perhaps the backend will benefit from a twisted mainloop that can hook
into stdin? And in any case, I think we *do* want some kind of
mainloop in the backend to avoid threads as much as possible.

Pros/cons of this approach:

Pros:

- No stdout/stdin redirection/capture code in frontend. It's just
normal pipe handling.
- Can be done as a library (no changes needed for ipython core)
- Launched processes (in backend) should behave naturally with the
stdin/stdout, since they inherit the pipe from ipython backend
process. IPython backend process itself can safely block waiting for
command to finish. No control commands should be sent to ipython
process while it's blocking, since they would be sent to launched
slave process instead. raw_input() in %run'd scripts should work fine
as well

Cons:

- Ptys are platform specific (i.e. the windows implementation may have
less "interactive" feel")
- Can't reconnect to backend if frontend crashes
- Probably other nasty issues that show up only when it's done

-- 
Ville M. Vainio
http://tinyurl.com/vainio


From vivainio at gmail.com  Thu Feb 12 06:22:46 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Thu, 12 Feb 2009 13:22:46 +0200
Subject: [IPython-dev] ctrl+c in gui, event loops,
	sure i'm missing 	something...
In-Reply-To: <20090212103919.GA24736@phare.normalesup.org>
References: <7ced318f0902111405y1118b2d9ya75e4c2ff825afbf@mail.gmail.com>
	<20090212085256.GC20814@phare.normalesup.org>
	<7ced318f0902120211t45381d63k37a3015c4f7f5db2@mail.gmail.com>
	<20090212103919.GA24736@phare.normalesup.org>
Message-ID: <46cb515a0902120322l79d019e1ye9b7ade5dbeac095@mail.gmail.com>

On Thu, Feb 12, 2009 at 12:39 PM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:

> The second problem is that the code allowing to have ipython0 features is
> a hack that is incompatible with out-of-process execution. Briefly, you
> need to share the namespace of the two engine (the ipython0 and the
> ipython1) to get magics and code completion. The clean way out of this is

Why is that, really? If code completion happens at backend, there is
nothing apart from a list of strings that frontend needs to know (and
it only needs to tell backend about the string that is to be
completed). Magics, otoh, are just a backend issue that frontend
doesn't need to know about...

-- 
Ville M. Vainio
http://tinyurl.com/vainio


From p.c.degroot at tudelft.nl  Thu Feb 12 08:06:08 2009
From: p.c.degroot at tudelft.nl (Pieter Cristiaan de Groot)
Date: Thu, 12 Feb 2009 14:06:08 +0100
Subject: [IPython-dev] ctrl-c bug using -gthread on windows,
 with possible solution
In-Reply-To: <498C0B82.2030404@tudelft.nl>
References: <4988D721.3060301@tudelft.nl> <4988D7AF.9090809@tudelft.nl>
	<498C0B82.2030404@tudelft.nl>
Message-ID: <49941EC0.7010006@tudelft.nl>

Hi all,

this is my last attempt to get the attention to this post. Even if this 
is the wrong place to post to,
could somebody explain where it should be posted?

Pieter

Pieter Cristiaan de Groot wrote:
> Hi all,
>
> any comments on the ctrl-c issue?
> Or did I post to the wrong list?
>
> Pieter
>
> Pieter Cristiaan de Groot wrote:
>   
>> What I forgot to mention is that because of this IOError, threading dies.
>>
>> Pieter
>>
>> Pieter Cristiaan de Groot wrote:
>>   
>>     
>>> Dear all,
>>>
>>> running ipython 0.9.1 on windows using -gthread gives problems using 
>>> ctrl-c. It gives an IOError when ctrl-c is pushed during the sleep(0.01) 
>>> (File: Shell.py line 835-843), otherwise everyghing goes ok. This does 
>>> not happen on a linux machine.
>>>
>>> Here's the code:
>>>
>>>     def on_timer(self):
>>>         """Called when GTK is idle.
>>>             Must return True always, otherwise GTK stops calling it"""
>>>   
>>>         update_tk(self.tk)
>>>         self.IP.runcode()
>>>         time.sleep(0.01)
>>>         return True
>>>
>>> Replacing the bottom part with this seems to solve it:
>>>
>>>         try:
>>>             update_tk(self.tk)
>>>             self.IP.runcode()
>>>             time.sleep(0.01)
>>>             return True
>>>         except IOError, e:
>>>             return True
>>>
>>>
>>>
>>> I hope this helps,
>>>
>>> Pieter
>>> _______________________________________________
>>> IPython-dev mailing list
>>> IPython-dev at scipy.org
>>> http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
>>>   
>>>     
>>>       
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at scipy.org
>> http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
>>   
>>     
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
>   


From vivainio at gmail.com  Thu Feb 12 08:54:40 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Thu, 12 Feb 2009 15:54:40 +0200
Subject: [IPython-dev] ctrl-c bug using -gthread on windows,
	with possible 	solution
In-Reply-To: <49941EC0.7010006@tudelft.nl>
References: <4988D721.3060301@tudelft.nl> <4988D7AF.9090809@tudelft.nl>
	<498C0B82.2030404@tudelft.nl> <49941EC0.7010006@tudelft.nl>
Message-ID: <46cb515a0902120554l28037c50i14d70f21f1833755@mail.gmail.com>

On Thu, Feb 12, 2009 at 3:06 PM, Pieter Cristiaan de Groot
<p.c.degroot at tudelft.nl> wrote:

> this is my last attempt to get the attention to this post. Even if this
> is the wrong place to post to,
> could somebody explain where it should be posted?

This is the right place, you probably just didn't get lucky in drawing
people's attention. Perhaps because people don't use gthread on
windows all that much.

Can you try catching the exception just for the sleep command?

-- 
Ville M. Vainio
http://tinyurl.com/vainio


From gael.varoquaux at normalesup.org  Thu Feb 12 11:09:10 2009
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 12 Feb 2009 17:09:10 +0100
Subject: [IPython-dev] ctrl+c in gui, event loops,
	sure i'm missing	something...
In-Reply-To: <46cb515a0902120322l79d019e1ye9b7ade5dbeac095@mail.gmail.com>
References: <7ced318f0902111405y1118b2d9ya75e4c2ff825afbf@mail.gmail.com>
	<20090212085256.GC20814@phare.normalesup.org>
	<7ced318f0902120211t45381d63k37a3015c4f7f5db2@mail.gmail.com>
	<20090212103919.GA24736@phare.normalesup.org>
	<46cb515a0902120322l79d019e1ye9b7ade5dbeac095@mail.gmail.com>
Message-ID: <20090212160910.GA8194@phare.normalesup.org>

On Thu, Feb 12, 2009 at 01:22:46PM +0200, Ville M. Vainio wrote:
> On Thu, Feb 12, 2009 at 12:39 PM, Gael Varoquaux
> <gael.varoquaux at normalesup.org> wrote:

> > The second problem is that the code allowing to have ipython0 features is
> > a hack that is incompatible with out-of-process execution. Briefly, you
> > need to share the namespace of the two engine (the ipython0 and the
> > ipython1) to get magics and code completion. The clean way out of this is

> Why is that, really? If code completion happens at backend, there is
> nothing apart from a list of strings that frontend needs to know (and
> it only needs to tell backend about the string that is to be
> completed). Magics, otoh, are just a backend issue that frontend
> doesn't need to know about...

In theory yes. But because all the cool feature of ipython0 couldn't be
abstracted from ipython0, I had to instanciate an ipython0 instance, and
feed it the namespace of the ipython1 engine. There needs to be a huge
refactoring of ipython0, right now it is a tangle of objects calling each
other, heavily coupled.

Ga?l


From vivainio at gmail.com  Thu Feb 12 13:27:12 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Thu, 12 Feb 2009 20:27:12 +0200
Subject: [IPython-dev] ctrl+c in gui, event loops,
	sure i'm missing 	something...
In-Reply-To: <20090212160910.GA8194@phare.normalesup.org>
References: <7ced318f0902111405y1118b2d9ya75e4c2ff825afbf@mail.gmail.com>
	<20090212085256.GC20814@phare.normalesup.org>
	<7ced318f0902120211t45381d63k37a3015c4f7f5db2@mail.gmail.com>
	<20090212103919.GA24736@phare.normalesup.org>
	<46cb515a0902120322l79d019e1ye9b7ade5dbeac095@mail.gmail.com>
	<20090212160910.GA8194@phare.normalesup.org>
Message-ID: <46cb515a0902121027t67a359bdr382a26f08751bfdf@mail.gmail.com>

On Thu, Feb 12, 2009 at 6:09 PM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:

> feed it the namespace of the ipython1 engine. There needs to be a huge
> refactoring of ipython0, right now it is a tangle of objects calling each
> other, heavily coupled.

I do not disagree. Unfortunately, the most tangled places are the ones
that add most value for IPython. Also, a lot of the mess is there
because there are just so many features and "intricate details" (as is
evident from reading all the comments in the code). So it may be a
mess but it's a valuable mess with years of battle testing that can't
all be factored out. It's a delicate balance with using the effort to
1) get what you want or 2) have the clean code you want.

As I see it, if you are just using the current ipython0 core as a
server, you can ignore a lot of the messier parts because the code is
not executed at all. Also, you can see it as a black box that executes
code you push to it, producing the Right Thing on stdout.

-- 
Ville M. Vainio
http://tinyurl.com/vainio


From vivainio at gmail.com  Thu Feb 12 14:15:43 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Thu, 12 Feb 2009 21:15:43 +0200
Subject: [IPython-dev] ctrl+c in gui, event loops,
	sure i'm missing 	something...
In-Reply-To: <46cb515a0902120316t2ae90aeaxb06d5bbe0a07d771@mail.gmail.com>
References: <7ced318f0902111405y1118b2d9ya75e4c2ff825afbf@mail.gmail.com>
	<20090212085256.GC20814@phare.normalesup.org>
	<46cb515a0902120316t2ae90aeaxb06d5bbe0a07d771@mail.gmail.com>
Message-ID: <46cb515a0902121115xad98d02ia51abfd04ada2979@mail.gmail.com>

On Thu, Feb 12, 2009 at 1:16 PM, Ville M. Vainio <vivainio at gmail.com> wrote:

> Pros:
>
> - No stdout/stdin redirection/capture code in frontend. It's just
> normal pipe handling.
> - Can be done as a library (no changes needed for ipython core)

Here's a concrete action plan:

- Implement something like ipy_server or ipy_vimserver that uses
stdin, stdout+stderr for communication.
- stdin is the "control channel" that gives commands that get executed
with ip.runlines. It may read pickles, or whatever.
- If the command coming from stdin is not "run this block"
(concretely, "complete" would be one such command), the response will
be written to stderr. Otherwise,
stdout and stderr will behave normally (i.e. stderr can contain both
control channel replies and normal stderr stuff).
- Try how well it works with qt ui (because that's the simplest to
hack for me). it still doesn't have proper stdin handling (unlike
Gaels ui), but that's not a dealbreaker yet. I can do stdin by some
quick hack first as a proof of concept.
- Report back if/when I have real code.

-- 
Ville M. Vainio
http://tinyurl.com/vainio


From ellisonbg.net at gmail.com  Thu Feb 12 15:23:07 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Thu, 12 Feb 2009 12:23:07 -0800
Subject: [IPython-dev] ctrl+c in gui, event loops,
	sure i'm missing 	something...
In-Reply-To: <46cb515a0902121115xad98d02ia51abfd04ada2979@mail.gmail.com>
References: <7ced318f0902111405y1118b2d9ya75e4c2ff825afbf@mail.gmail.com>
	<20090212085256.GC20814@phare.normalesup.org>
	<46cb515a0902120316t2ae90aeaxb06d5bbe0a07d771@mail.gmail.com>
	<46cb515a0902121115xad98d02ia51abfd04ada2979@mail.gmail.com>
Message-ID: <6ce0ac130902121223vf368809t36937c20d2122059@mail.gmail.com>

> - Implement something like ipy_server or ipy_vimserver that uses
> stdin, stdout+stderr for communication.

These don't use stdin/stdout for communication.  They use Unix
sockets.  Plus, this is an unsafe thing to do (yes our current
versions have this problem).  Here is why.  These codes run an event
loop in a thread.  They then call the ipython0 core from that thread.
If the user does something interactively with IPython, you have two
threads simultaneiously operating on the ipython0 core and the users
namespace.  Because the ipython0 core is not thread safe, this is
entirely unsafe from the threading perspective.

> - stdin is the "control channel" that gives commands that get executed
> with ip.runlines. It may read pickles, or whatever.
> - If the command coming from stdin is not "run this block"
> (concretely, "complete" would be one such command), the response will
> be written to stderr. Otherwise,
> stdout and stderr will behave normally (i.e. stderr can contain both
> control channel replies and normal stderr stuff).

I think stdin/stdout should be reserved for the usual meanings.  There
are many robust ways of doing IPC over various types of sockets that
should be used instead for control.  This is what things like
vim_server do already - we just need to think carefully about how we
can make these things thread safe.

Cheers,

Brian


From ellisonbg.net at gmail.com  Thu Feb 12 15:26:34 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Thu, 12 Feb 2009 12:26:34 -0800
Subject: [IPython-dev] ctrl+c in gui, event loops,
	sure i'm missing 	something...
In-Reply-To: <46cb515a0902121027t67a359bdr382a26f08751bfdf@mail.gmail.com>
References: <7ced318f0902111405y1118b2d9ya75e4c2ff825afbf@mail.gmail.com>
	<20090212085256.GC20814@phare.normalesup.org>
	<7ced318f0902120211t45381d63k37a3015c4f7f5db2@mail.gmail.com>
	<20090212103919.GA24736@phare.normalesup.org>
	<46cb515a0902120322l79d019e1ye9b7ade5dbeac095@mail.gmail.com>
	<20090212160910.GA8194@phare.normalesup.org>
	<46cb515a0902121027t67a359bdr382a26f08751bfdf@mail.gmail.com>
Message-ID: <6ce0ac130902121226g66b89777jb2f8c3893e3cf9a0@mail.gmail.com>

> I do not disagree. Unfortunately, the most tangled places are the ones
> that add most value for IPython. Also, a lot of the mess is there
> because there are just so many features and "intricate details" (as is
> evident from reading all the comments in the code).

> So it may be a
> mess but it's a valuable mess with years of battle testing

I completely agree with this.

> that can't all be factored out.

I disagree with this.  The main issue with ipython0 is that it is
welded to the terminal through readline.  There is absolutely no
reason this can't be factored out.  It may not be easy, but to support
the things that people want, we must try.

Cheers,

Brian


From vivainio at gmail.com  Thu Feb 12 15:52:28 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Thu, 12 Feb 2009 22:52:28 +0200
Subject: [IPython-dev] ctrl+c in gui, event loops,
	sure i'm missing 	something...
In-Reply-To: <6ce0ac130902121223vf368809t36937c20d2122059@mail.gmail.com>
References: <7ced318f0902111405y1118b2d9ya75e4c2ff825afbf@mail.gmail.com>
	<20090212085256.GC20814@phare.normalesup.org>
	<46cb515a0902120316t2ae90aeaxb06d5bbe0a07d771@mail.gmail.com>
	<46cb515a0902121115xad98d02ia51abfd04ada2979@mail.gmail.com>
	<6ce0ac130902121223vf368809t36937c20d2122059@mail.gmail.com>
Message-ID: <46cb515a0902121252m4f9db8e1r845fa8aad9def4af@mail.gmail.com>

On Thu, Feb 12, 2009 at 10:23 PM, Brian Granger <ellisonbg.net at gmail.com> wrote:

>> - Implement something like ipy_server or ipy_vimserver that uses
>> stdin, stdout+stderr for communication.
>
> These don't use stdin/stdout for communication.  They use Unix
> sockets.  Plus, this is an unsafe thing to do (yes our current

True, and I'd like to change them to use just natural pipes that come
from process startup. Well, actually I won't be using in code in
ipy_*server at all, since I want to avoid threads. Other forms of IPC
are easier to wrangle, but I just want to retain a tight connection
with the processes by using these "natural", inherited pipes. They are
also necessary for subprocesses launched by the backend.

And it's not like I couldn't add another socket connection to the game
if need arises ;-).

> versions have this problem).  Here is why.  These codes run an event
> loop in a thread.  They then call the ipython0 core from that thread.

Yeah, no thread will enter my implementation. ipy_server.py was just a
quick hack I did to prove some point, iirc).

> I think stdin/stdout should be reserved for the usual meanings.  There

But the idea here *is* to use them for their usual meaning, with some
extra features added.

> are many robust ways of doing IPC over various types of sockets that
> should be used instead for control.  This is what things like
> vim_server do already - we just need to think carefully about how we
> can make these things thread safe.

Those scripts don't really do that much. I think that by hooking all
the three I/O sockects we will have something much more interesting.

These scripts could be mado threadsafe by removing threads from them.
Threading was just the easy way to write such scripts, but without
much work you could throw in asyncore or twisted.


-- 
Ville M. Vainio
http://tinyurl.com/vainio


From p.c.degroot at tudelft.nl  Thu Feb 12 17:05:10 2009
From: p.c.degroot at tudelft.nl (Pieter Cristiaan de Groot)
Date: Thu, 12 Feb 2009 23:05:10 +0100
Subject: [IPython-dev] ctrl-c bug using -gthread on windows,
 with possible 	solution
In-Reply-To: <46cb515a0902120554l28037c50i14d70f21f1833755@mail.gmail.com>
References: <4988D721.3060301@tudelft.nl> <4988D7AF.9090809@tudelft.nl>	
	<498C0B82.2030404@tudelft.nl> <49941EC0.7010006@tudelft.nl>
	<46cb515a0902120554l28037c50i14d70f21f1833755@mail.gmail.com>
Message-ID: <49949D16.6020100@tudelft.nl>


Just to be sure I answer your question fully:

case 1) the original 0.9.1 ipython code gives this error if I start it 
with -gthread and push ctrl-c

##########################

CODE (in Shell.py, around linenr. 840):

        update_tk(self.tk)
        self.IP.runcode()
        time.sleep(0.01)
        return True

ERROR:

KeyboardInterrupt - Press <Enter> to 
continue.---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)

c:\python25\lib\site-packages\IPython\Shell.pyc in on_timer(self)
    840         update_tk(self.tk)
    841         self.IP.runcode()
--> 842         time.sleep(0.01)
    843         return True
    844

IOError: [Errno 4] Interrupted function call

#################

case 2) catching the exception (with some prints):

###############

CODE:

        try:
            update_tk(self.tk)
            self.IP.runcode()
            time.sleep(0.01)
            return True
        except Exception, e:
            print type(e)
            print(e)
            return True

ERROR:

KeyboardInterrupt - Press <Enter> to continue. <type 'exceptions.IOError'>
[Errno 4] Interrupted function call

#########################

case 3) if I understood correctly this is what you wanted to see:

#####################

CODE:

        update_tk(self.tk)
        self.IP.runcode()
        try:
            time.sleep(0.01)
            return True
        except Exception ,e:
            print type(e)
            print e
            return True

ERROR:

KeyboardInterrupt - Press <Enter> to continue. <type 'exceptions.IOError'>
[Errno 4] Interrupted function call

##########################

case 3a) occasionally (1 in ~20 times), if you interrupt the code while 
it is not in the sleep, then everything works fine. It gives this message:

############################

CODE: (same as case 3, and probably the same for 1 and 2)

ERROR:

KeyboardInterrupt - Press <Enter> to continue.

###########################

I hope this answers your question. If you want me to check any more 
things, please ask.

Pieter

Ville M. Vainio wrote:
> On Thu, Feb 12, 2009 at 3:06 PM, Pieter Cristiaan de Groot
> <p.c.degroot at tudelft.nl> wrote:
>
>   
>> this is my last attempt to get the attention to this post. Even if this
>> is the wrong place to post to,
>> could somebody explain where it should be posted?
>>     
>
> This is the right place, you probably just didn't get lucky in drawing
> people's attention. Perhaps because people don't use gthread on
> windows all that much.
>
> Can you try catching the exception just for the sleep command?
>
>   


From vivainio at gmail.com  Fri Feb 13 01:37:32 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Fri, 13 Feb 2009 08:37:32 +0200
Subject: [IPython-dev] ctrl-c bug using -gthread on windows,
	with possible 	solution
In-Reply-To: <49949D16.6020100@tudelft.nl>
References: <4988D721.3060301@tudelft.nl> <4988D7AF.9090809@tudelft.nl>
	<498C0B82.2030404@tudelft.nl> <49941EC0.7010006@tudelft.nl>
	<46cb515a0902120554l28037c50i14d70f21f1833755@mail.gmail.com>
	<49949D16.6020100@tudelft.nl>
Message-ID: <46cb515a0902122237i72111fefl9f0b3d5e840c61ca@mail.gmail.com>

On Fri, Feb 13, 2009 at 12:05 AM, Pieter Cristiaan de Groot
<p.c.degroot at tudelft.nl> wrote:

> I hope this answers your question. If you want me to check any more things,
> please ask

No need, this is enough to add case 3 to ipython. Thank you!

-- 
Ville M. Vainio
http://tinyurl.com/vainio


From vivainio at gmail.com  Fri Feb 13 15:55:13 2009
From: vivainio at gmail.com (Ville M. Vainio)
Date: Fri, 13 Feb 2009 22:55:13 +0200
Subject: [IPython-dev] Looking towards 0.10...
In-Reply-To: <db6b5ecc0902101732r10f37d8xf995410c8705a3f9@mail.gmail.com>
References: <db6b5ecc0902101732r10f37d8xf995410c8705a3f9@mail.gmail.com>
Message-ID: <46cb515a0902131255q28fec489ib81cd0e12105e13f@mail.gmail.com>

On Wed, Feb 11, 2009 at 3:32 AM, Fernando Perez <fperez.net at gmail.com> wrote:

> This one seemed to be almost ready:
> - https://code.launchpad.net/~villemvainio/ipython/ip0-unittests-1/+merge/1075

It's really not - I disabled the autocall test case, then
test_magic.py still wrote stuff to screen. Essentially I'd have to
disable all the test cases, which is as good as not merging in the
branch at all. As you suggest it may be a nose issue - certainly it
should be the one handling the output capture, not individual test
cases.

This should not really prevent a release, since it's just tests
anyway, and users don't need to run them. This can be merged after the
release.

I added my trunk-dev for review, though.
-- 
Ville M. Vainio
http://tinyurl.com/vainio


From fperez.net at gmail.com  Fri Feb 13 18:07:54 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Fri, 13 Feb 2009 15:07:54 -0800
Subject: [IPython-dev] Looking towards 0.10...
In-Reply-To: <46cb515a0902131255q28fec489ib81cd0e12105e13f@mail.gmail.com>
References: <db6b5ecc0902101732r10f37d8xf995410c8705a3f9@mail.gmail.com>
	<46cb515a0902131255q28fec489ib81cd0e12105e13f@mail.gmail.com>
Message-ID: <db6b5ecc0902131507x63162acdm567a77dc22d288fe@mail.gmail.com>

On Fri, Feb 13, 2009 at 12:55 PM, Ville M. Vainio <vivainio at gmail.com> wrote:
> On Wed, Feb 11, 2009 at 3:32 AM, Fernando Perez <fperez.net at gmail.com> wrote:
>
>> This one seemed to be almost ready:
>> - https://code.launchpad.net/~villemvainio/ipython/ip0-unittests-1/+merge/1075
>
> It's really not - I disabled the autocall test case, then
> test_magic.py still wrote stuff to screen. Essentially I'd have to
> disable all the test cases, which is as good as not merging in the
> branch at all. As you suggest it may be a nose issue - certainly it
> should be the one handling the output capture, not individual test
> cases.
>
> This should not really prevent a release, since it's just tests
> anyway, and users don't need to run them. This can be merged after the
> release.

OK, we'll merge it when it's fixed then, post-0.10.  If it requires
looking at nose, so be it.

But we most certainly do users want running the test suite, always.
We need to make every effort possible so that testing ipython is
trivial, always, for anyone.  There are many bugs that happen only on
the user's end (platform, language, encoding, etc), so we want our
users always running our tests.  For this reason, the test code has to
be in tip-top shape: if running the tests and reporting back is a
nightmare, not only does it make life unpleasant for us, but more
importantly, we lose critical feedback from our users.

> I added my trunk-dev for review, though.

Cool, thanks.   I'll  have a go at it.


Anything else anyone can think of that I missed in the previous email?

Cheers,

f


From ellisonbg.net at gmail.com  Fri Feb 20 18:08:22 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Fri, 20 Feb 2009 15:08:22 -0800
Subject: [IPython-dev] IPython test suite is slow on OS X fast on Windows
Message-ID: <6ce0ac130902201508v6a14c8faga061fdf6a88a8879@mail.gmail.com>

Hi,

Last summer Min and I noticed that the IPython test suite is quite
fast on Windows, but really slow on OS X:

1.  2.53 Ghz Intel Core 2 Duo, Mac OS X 10.5.  Test suite takes 67 seconds

2.  *Same machine*, running Windows XP as VM in VMWare.  Test suite
takes 17 seconds.

I have profiled the test suite and found some interesting and confusing things:

On the OS X, calls to tokenize from ultraTB take up 30 seconds!  On
Windows, just 0.6 seconds.  Does anyone have an idea of what might be
going on here?

To run the test suite using the profiler, I did:

trial --profile IPython

There are probably other ways of running this.

Thoughts?

Brian


From fperez.net at gmail.com  Fri Feb 20 19:48:08 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Fri, 20 Feb 2009 16:48:08 -0800
Subject: [IPython-dev] IPython test suite is slow on OS X fast on Windows
In-Reply-To: <6ce0ac130902201508v6a14c8faga061fdf6a88a8879@mail.gmail.com>
References: <6ce0ac130902201508v6a14c8faga061fdf6a88a8879@mail.gmail.com>
Message-ID: <db6b5ecc0902201648t19288ccfif66d777de2966301@mail.gmail.com>

On Fri, Feb 20, 2009 at 3:08 PM, Brian Granger <ellisonbg.net at gmail.com> wrote:
> Hi,
>
> Last summer Min and I noticed that the IPython test suite is quite
> fast on Windows, but really slow on OS X:
>
> 1.  2.53 Ghz Intel Core 2 Duo, Mac OS X 10.5.  Test suite takes 67 seconds
>
> 2.  *Same machine*, running Windows XP as VM in VMWare.  Test suite
> takes 17 seconds.
>
> I have profiled the test suite and found some interesting and confusing things:
>
> On the OS X, calls to tokenize from ultraTB take up 30 seconds!  On
> Windows, just 0.6 seconds.  Does anyone have an idea of what might be
> going on here?

That's very odd... On my fairly fast desktop using 64-bit ubuntu 8.10, I get

Ran 431 tests in 28.629s

Are you sure that the XP tests are really all completing identically?

The tokenize stuff is very bizarre.  That's part of the stdlib, do you
have a little self-contained example that we could use to track down
the difference across machines?

Cheers,

f


From ellisonbg.net at gmail.com  Fri Feb 20 19:53:44 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Fri, 20 Feb 2009 16:53:44 -0800
Subject: [IPython-dev] IPython test suite is slow on OS X fast on Windows
In-Reply-To: <db6b5ecc0902201648t19288ccfif66d777de2966301@mail.gmail.com>
References: <6ce0ac130902201508v6a14c8faga061fdf6a88a8879@mail.gmail.com>
	<db6b5ecc0902201648t19288ccfif66d777de2966301@mail.gmail.com>
Message-ID: <6ce0ac130902201653t12b12858v96ca223732dba032@mail.gmail.com>

After more investigation, I think it may be how the built-in Python on
OS X is built:

On OS X:

In [1]: def f():
  ...:     for i in range(100000):
  ...:         a = 1 + i
In [2]: %timeit f()
100 loops, best of 3: 11.8 ms per loop

Same hardware, but Windows:

In [1]: def f():
  ...:     for i in range(100000):
  ...:         a = 1 + i
In [2]: %timeit f()
100 loops, best of 3: 8.33 ms per loop

The ratio of these times is about the same as the ratio of the test suite times.

That's also depressing...

But, what is really puzzling is why the profiler shows that tokenize
is the bottleneck.

Thanks for timing this on your system Fernando.  I will post to the
list if I figure out anything more.

Brian

On Fri, Feb 20, 2009 at 4:48 PM, Fernando Perez <fperez.net at gmail.com> wrote:
> On Fri, Feb 20, 2009 at 3:08 PM, Brian Granger <ellisonbg.net at gmail.com> wrote:
>> Hi,
>>
>> Last summer Min and I noticed that the IPython test suite is quite
>> fast on Windows, but really slow on OS X:
>>
>> 1.  2.53 Ghz Intel Core 2 Duo, Mac OS X 10.5.  Test suite takes 67 seconds
>>
>> 2.  *Same machine*, running Windows XP as VM in VMWare.  Test suite
>> takes 17 seconds.
>>
>> I have profiled the test suite and found some interesting and confusing things:
>>
>> On the OS X, calls to tokenize from ultraTB take up 30 seconds!  On
>> Windows, just 0.6 seconds.  Does anyone have an idea of what might be
>> going on here?
>
> That's very odd... On my fairly fast desktop using 64-bit ubuntu 8.10, I get
>
> Ran 431 tests in 28.629s
>
> Are you sure that the XP tests are really all completing identically?
>
> The tokenize stuff is very bizarre.  That's part of the stdlib, do you
> have a little self-contained example that we could use to track down
> the difference across machines?
>
> Cheers,
>
> f
>


From allen.fowler at yahoo.com  Wed Feb 25 20:27:18 2009
From: allen.fowler at yahoo.com (Allen Fowler)
Date: Wed, 25 Feb 2009 17:27:18 -0800 (PST)
Subject: [IPython-dev] quick way to reset & run a program?
Message-ID: <105574.85628.qm@web45604.mail.sp1.yahoo.com>

Hello,

Is there a quick/easy way to reset the interpreter's state, and then run an external file?

Basically I'm looking for a quicker way to execute: "%reset, confirm [yes], %run". (Optional clearing the cached output history. )

Thank you



      



From fperez.net at gmail.com  Thu Feb 26 02:19:24 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 25 Feb 2009 23:19:24 -0800
Subject: [IPython-dev] quick way to reset & run a program
In-Reply-To: <105574.85628.qm@web45604.mail.sp1.yahoo.com>
References: <105574.85628.qm@web45604.mail.sp1.yahoo.com>
Message-ID: <db6b5ecc0902252319u3738a9a2yba3688f25f22a6f1@mail.gmail.com>

On Wed, Feb 25, 2009 at 5:27 PM, Allen Fowler <allen.fowler at yahoo.com> wrote:
> Hello,
>
> Is there a quick/easy way to reset the interpreter's state, and then run an external file?
>
> Basically I'm looking for a quicker way to execute: "%reset, confirm [yes], %run". (Optional clearing the cached output history. )

Not out of the box, but writing a little magic to do this is and
installing it in your home dir is pretty easy.  Unfortunatley you
can't quite just make your magic do 'reset -y, run' because %reset
asks unconditionally.  But %reset is small enough (see magic_reset in
Magic.py) that you can simply copy its contents into your own personal
%rrun and then have that call %run.  Let us know if you have any
problems with this and need further help.

Cheers,

f


From fperez.net at gmail.com  Thu Feb 26 13:17:16 2009
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 26 Feb 2009 10:17:16 -0800
Subject: [IPython-dev] Very bizarre bug... any ideas?
Message-ID: <db6b5ecc0902261017tc2ec9f2n6a7de137d23f38a7@mail.gmail.com>

Hi all,

morning puzzle:

https://bugs.launchpad.net/ipython/+bug/334744

If anyone can think of what could be going on here, I'm all ears.

Cheers,

f


From allen.fowler at yahoo.com  Thu Feb 26 16:46:46 2009
From: allen.fowler at yahoo.com (Allen Fowler)
Date: Thu, 26 Feb 2009 13:46:46 -0800 (PST)
Subject: [IPython-dev] quick way to reset & run a program
In-Reply-To: <db6b5ecc0902252319u3738a9a2yba3688f25f22a6f1@mail.gmail.com>
References: <105574.85628.qm@web45604.mail.sp1.yahoo.com>
	<db6b5ecc0902252319u3738a9a2yba3688f25f22a6f1@mail.gmail.com>
Message-ID: <565024.41290.qm@web45611.mail.sp1.yahoo.com>

> > Is there a quick/easy way to reset the interpreter's state, and then run an 

> external file?
> >
> > Basically I'm looking for a quicker way to execute: "%reset, confirm [yes], 
> %run". (Optional clearing the cached output history. )
> 
> Not out of the box, but writing a little magic to do this is and
> installing it in your home dir is pretty easy.  Unfortunatley you
> can't quite just make your magic do 'reset -y, run' because %reset
> asks unconditionally.  But %reset is small enough (see magic_reset in
> Magic.py) that you can simply copy its contents into your own personal
> %rrun and then have that call %run.  Let us know if you have any
> problems with this and need further help.
> 
> 


Sounds like fun.    :) 

Can you point me to the relevant docs?


I am a new ipython user, so out of curiosity, what kind of ipython work flows do people use to help develop software?

Thank you,
:)



      



From ronena at gmail.com  Sat Feb 28 16:52:41 2009
From: ronena at gmail.com (Ronen Abravanel)
Date: Sat, 28 Feb 2009 23:52:41 +0200
Subject: [IPython-dev] Ipython plugin for Editra
Message-ID: <2498a980902281352q48df3928v1b6bb05662f31ef3@mail.gmail.com>

Editra, an extensible programmer's editor written in wxPython, contain a
cool IPython plugin. This mail is a follow up for a discussion with
Laurent, The author of the plugin, in his blog, continue here
according to his
request. The Orginal discassion can be found in
http://ipython0.wordpress.com/2008/04/25/editra-ipython-plugin-under-work/or
at the end of this mail.


Using papi to set\redefine IPython's magic word partly do the job, The other
part is to execute some arbitrary python code inside The ipython shell.
Invoking papis "ex" method (Using papi from *ipython_panel.IP._IP.getapi() *)
let the Code run in IPython's context, but the output is printed to Editra's
stdout (The console in wich I invoked editra's from) and not in the IPython
in the shelf.  My attempts to set sys.stdout to an handler from, let say,
_term.cout or others, cused IPython to stack. futher more, the Exeption
throws while I call ip.ex, is cought within Editra and withn IPython.

Further more - I wish to change, If I can, Only IPyShell/__init__.py and not
the stuff in "IPython" dir, as it seems as a "pure" copy of IPython's
orginal source (is it?)


Thanks,
Ronen.


--------------------------------------
The orginal discassion:

   1. Ronen Says:
   February 21, 2009 at 8:18
pm<http://ipython0.wordpress.com/2008/04/25/editra-ipython-plugin-under-work/#comment-25>

   That?s a one cool Plugin.

   What will make it ideal, is better integration with Editra?s Editor:
   Make ipython?s %edit Open the edit-buffer in Editra?s current Windows,
   and Enable poeple to run The code they edit in Editra inside the open
   IPython windows?
   Any hint where should I try to hack in order to achieve those?

   Thanks,
   Ronen.
   2.  Laurent Says:
   February 22, 2009 at 10:45
am<http://ipython0.wordpress.com/2008/04/25/editra-ipython-plugin-under-work/#comment-26>

   Hello,
   You can have access to the plugin repos:

   # Non-members may check out a read-only working copy anonymously over
   HTTP.
   svn checkout
http://editra-plugins.googlecode.com/svn/trunk/editra-plugins-read-only

   The plugin name is ipyshell, in it?s directory you?ve got the whole
   ipython source + the plugin source.

   It need a little refresh for sure [image: :)] .

   If you want to try to hack it take a look @Ipython/gui/wx the whole code
   is in ipython_view and ipshellnonblocking.

   ipython_view -> gui interface
   ipshellnonblocking -> the interface with ipython

   You can add:
   ip = IPython.ipapi.get()
   def myEdit(self, arg):
   print ?do what you want?
   ip.expose_magic(?edit?, myEdit)

   to WxNonBlockingIPShell inside ipython_view.py

   Please note that if ipython is ever installed on your system you?ll have
   to make a eggsetup.py develop to make sure to use your new local copy.

   We can discuss about all of this in ipython-dev mailing list [image: :)]

   Cheers,
   Laurent
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090228/aad1e02c/attachment.html>

From laurent.dufrechou at gmail.com  Sat Feb 28 20:33:51 2009
From: laurent.dufrechou at gmail.com (=?UTF-8?Q?Laurent_Dufr=C3=A9chou?=)
Date: Sun, 1 Mar 2009 02:33:51 +0100
Subject: [IPython-dev] Ipython plugin for Editra
In-Reply-To: <2498a980902281352q48df3928v1b6bb05662f31ef3@mail.gmail.com>
References: <2498a980902281352q48df3928v1b6bb05662f31ef3@mail.gmail.com>
Message-ID: <49a9e60b.07eb300a.6f2e.010c@mx.google.com>

Hi Ronen,

 

Happy to see you?ve tried to hack the editra plugin ;)

 

IPyShell/__init__.py contains only the interface with editra. 

I do not cleanly understand the step your are trying to do. You?ve used ? %ed? than it launched editra new source panel? And closing it executed your code, but stdin/out goes outside ipython shell?

Is that what you tried?

 

About the files to modify. __init__.py in editra plugn is the definition of the plugin interface and how you interact with editra.

Ipython dir is a pure copy that I was used to upload from time to time. I have to take a look at some updates that cody did. (some compatibility issue with python 2.6 and ipython tree)

The fact is that the update you?re currently trying to di will surely need some update into /gui/wx I think L. But well we?ll see.

Depending on what you?ll want to do we will perhaps expose some methods so plugin will be able to configure the things you need. (ex being able to define an editor)

(I think we can use ipython config file but I?ve never used them since then so?)

 

I will try to upload updated sources for python tree tomorrow, so you?ll have a bleeding edge wx integration. I?ve corrected some bugs since then.

Sure we will find a way to workaround the stdin/out issue you saw.

 

Send me your current hack, I?ll take a look at your issue.

 

Cheers,

Laurent

 

De : ipython-dev-bounces at scipy.org [mailto:ipython-dev-bounces at scipy.org] De la part de Ronen Abravanel
Envoy? : samedi 28 f?vrier 2009 22:53
? : ipython-dev at scipy.org
Objet : [IPython-dev] Ipython plugin for Editra

 

Editra, an extensible programmer's editor written in wxPython, contain a cool IPython plugin. This mail is a follow up for a discussion with Laurent , The author of the plugin, in his blog, continue here according to his request. The Orginal discassion can be found in http://ipython0.wordpress.com/2008/04/25/editra-ipython-plugin-under-work/ or at the end of this mail.


Using papi to set\redefine IPython's magic word partly do the job, The other part is to execute some arbitrary python code inside The ipython shell. Invoking papis "ex" method (Using papi from ipython_panel.IP._IP.getapi() ) let the Code run in IPython's context, but the output is printed to Editra's stdout (The console in wich I invoked editra's from) and not in the IPython in the shelf.  My attempts to set sys.stdout to an handler from, let say, _term.cout or others, cused IPython to stack. futher more, the Exeption throws while I call ip.ex, is cought within Editra and withn IPython.

Further more - I wish to change, If I can, Only IPyShell/__init__.py and not the stuff in "IPython" dir, as it seems as a "pure" copy of IPython's orginal source (is it?)


Thanks,
Ronen.


--------------------------------------
The orginal discassion:

1.	Ronen Says: 
February 21, 2009 at 8:18 pm <http://ipython0.wordpress.com/2008/04/25/editra-ipython-plugin-under-work/#comment-25>  

That?s a one cool Plugin.

What will make it ideal, is better integration with Editra?s Editor:
Make ipython?s %edit Open the edit-buffer in Editra?s current Windows, and Enable poeple to run The code they edit in Editra inside the open IPython windows?
Any hint where should I try to hack in order to achieve those? 

Thanks,
Ronen.

2.	  <http://1.gravatar.com/avatar/7d4c274b8ecba154d3ee55838145bb23?s=32&d=identicon> Laurent Says: 
February 22, 2009 at 10:45 am <http://ipython0.wordpress.com/2008/04/25/editra-ipython-plugin-under-work/#comment-26>  

Hello,
You can have access to the plugin repos:

# Non-members may check out a read-only working copy anonymously over HTTP.
svn checkout http://editra-plugins.googlecode.com/svn/trunk/ editra-plugins-read-only 

The plugin name is ipyshell, in it?s directory you?ve got the whole ipython source + the plugin source.

It need a little refresh for sure  :) <http://s.wordpress.com/wp-includes/images/smilies/icon_smile.gif> .

If you want to try to hack it take a look @Ipython/gui/wx the whole code is in ipython_view and ipshellnonblocking.

ipython_view -> gui interface
ipshellnonblocking -> the interface with ipython

You can add:
ip = IPython.ipapi.get()
def myEdit(self, arg):
print ?do what you want?
ip.expose_magic(?edit?, myEdit)

to WxNonBlockingIPShell inside ipython_view.py

Please note that if ipython is ever installed on your system you?ll have to make a eggsetup.py develop to make sure to use your new local copy.

We can discuss about all of this in ipython-dev mailing list  :) <http://s.wordpress.com/wp-includes/images/smilies/icon_smile.gif> 

Cheers,
Laurent

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090301/b48239c3/attachment.html>