From Fernando.Perez at colorado.edu  Wed Mar  2 03:54:29 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Wed, 02 Mar 2005 01:54:29 -0700
Subject: [IPython-dev] [ANN] IPython 0.6.12 is out.
Message-ID: <42257F45.4080704@colorado.edu>

Hi all,

I'm glad to announce the release of IPython 0.6.12.  This is mainly a bugfix 
release.

IPython's homepage is at:

http://ipython.scipy.org

and downloads are at:

http://ipython.scipy.org/dist

I've provided RPMs (for Python 2.3 and 2.4, built under Fedora Core 3), plus
source downloads (.tar.gz).  We now also have a native win32 installer which
should work correctly for both Python 2.3 and 2.4.

Debian, Fink and BSD packages for this version should be coming soon, as the
respective maintainers (many thanks to Jack Moffit, Andrea Riciputi and Dryice
Liu) have the time to follow their packaging procedures.

Many thanks to Enthought for their continued hosting support for IPython, and
to all the users who contributed ideas, fixes and reports.


Release notes
-------------

As always, the NEWS file can be found at http://ipython.scipy.org/NEWS, and
the full ChangeLog at http://ipython.scipy.org/ChangeLog.

* New hooks system.  This can be considered the first step in cleaning up 
ipython to expose its internals in a more rational manner.  IPython/hooks.py 
contains the hooks which are meant to be easily overridden by users for 
customizations.  Currently only the editor hook (called by %edit) exists, but 
a framework is in place.

* Fix a subtle bug with corrupted init files under win32, thanks to Ville 
Vainio for trackign this.

* Fix for Debian, which recently removed the profiler module from the default 
python distribution and moved it to non-free.

* Allow empty and left-aligned output prompts, if users so want them.

* New %time magic to time statements and expressions.  It reports CPU and wall 
clock time.

* Other small fixes and cleanups.


Enjoy, and as usual please report any problems.

Regards,

Fernando.



From Fernando.Perez at colorado.edu  Wed Mar  2 10:58:58 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Wed, 02 Mar 2005 08:58:58 -0700
Subject: [IPython-dev] [ANN] IPython 0.6.12 is out.
In-Reply-To: <mailman.259.1109763083.905.ipython-dev@scipy.net>
References: <mailman.259.1109763083.905.ipython-dev@scipy.net>
Message-ID: <4225E2C2.7050003@colorado.edu>

[ Norbert, your message originally was dropped b/c the ipython lists don't 
allow non-subscriber postings.  I've whitelisted you manually]

> Subject: Re: [IPython-dev] [ANN] IPython 0.6.12 is out.
> From: Norbert Tretkowski <nobse at debian.org>
> Date: Wed, 2 Mar 2005 12:31:13 +0100
> To:  ipython-dev at scipy.net
> 
> * Fernando Perez wrote:
> 
>>* Fix for Debian, which recently removed the profiler module from
>>the default python distribution and moved it to non-free.
> 
> 
> Great, so we can remove that patch. Thanks.
> 
> Debian packages are available[0], and will be uploaded to Debian
> unstable as soon as I get an answer from Jack Moffitt, the current
> maintainer of Debian's ipython package.
> 
> Regards, Norbert
> 
> [0]: http://people.debian.org/~nobse/debian/unstable/

Thanks.  I should have been more explicit in the announcement, in stating the 
fact I used the debian patch with only minor format modifications.  This is 
the full changelog entry:

	* IPython/Magic.py (Magic.profile_missing_notice): Don't crash if
	the profiler can't be imported.  Fix for Debian, which removed
	profile.py because of License issues.  I applied a slightly
	modified version of the original Debian patch at
	http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=294500.

Regards,

f



From Fernando.Perez at colorado.edu  Fri Mar  4 13:44:45 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Fri, 04 Mar 2005 11:44:45 -0700
Subject: [IPython-dev] Offline until March 15th
Message-ID: <4228AC9D.6040600@colorado.edu>

Hi all,

just a note to let you know that I'll have very limited, if any, email access 
from tomorrow until March 15th approximately.

Regards,

f



From mantegazza at ill.fr  Wed Mar 16 10:56:15 2005
From: mantegazza at ill.fr (=?iso-8859-15?q?Fr=E9d=E9ric_Mantegazza?=)
Date: Wed, 16 Mar 2005 16:56:15 +0100
Subject: [IPython-dev] Completer in ipython 0.6.12
Message-ID: <200503161656.15525.mantegazza@ill.fr>

Hello Fernando,

A few months ago, with your help, I made a patch to add my own matcher, but 
it does not work anymore in the 0.6.12 version. Now, it does nothing when I 
hit the TAB key, in any case :o(

Here is the code used:

def proxy_matches(self, text, state):
    """ Get the attribute of a remove Pyro object.
    """
    log = Logger('client')

    # Another option, seems to work great. Catches things like ''.<tab>
    m = re.match(r"(\S+(\.\w+)*)\.(\w*)$", text)

    if not m:
        return []
    expr, attr = m.group(1, 3)
    matches = []
    try:
        object = eval(expr, self.namespace)
        if isinstance(object, Pyro.core.DynamicProxyWithAttrs):
            words = object.getAvailableAttributes()
            matches = []
            n = len(attr)
            if words:
                for word in words:
                    if word[:n] == attr and word != "__builtins__":
                        matches.append("%s.%s" % (expr, word))

    except NameError:
        pass
        
    except Pyro.errors.ConnectionClosedError:
        log.error("Connexion closed")
        object.adapter.rebindURI() # Should be moved to runcode()
        matches = [""]

    return matches

ipshell.IP.Completer.matchers.insert(0, 'proxy_matches')
IPython.iplib.MagicCompleter.proxy_matches = proxy_matches

I saw that you changed lot of things in the iplib.py file, and I don't know 
how to have my patch working. Could your explain me how things works, now ?

Thank's for your help,

-- 
   Fr?d?ric



From mantegazza at ill.fr  Wed Mar 16 10:56:16 2005
From: mantegazza at ill.fr (=?iso-8859-15?q?Fr=E9d=E9ric_Mantegazza?=)
Date: Wed, 16 Mar 2005 16:56:16 +0100
Subject: [IPython-dev] Some ideas for more hooks
Message-ID: <200503161656.16609.mantegazza@ill.fr>

Hello Fernando,

Here is a little paper to explain the (unusual) way we are using ipython in 
our application (PyMAD). This is a summary of our needs of internal 
interaction with ipython to make something very powerfull.

I hope it will help you to re-design ipython with user customizable hooks.

PS : PyMAD is a software we are developping to control spectrometers, big 
neutrons instruments.

-- 
   Fr?d?ric
-------------- next part --------------
This document describes the interactions needed by PyMAD with ipython.
======================================================================

1) Introduction
---------------

The philosophy of PyMAD is to give to the user a high-level set of tools to
drive a spectrometer. The idea is to directly make some internal python
objects available to the user, and let him combine them.

So, The final user just calls some methods of a few high-level objects to drive
the spectrometer. This as the advantage to make all the python scripting stuff
available, to build some new high level tools.

As we also need a system which can be used from several places, we use a
client/server framework, with Pyro. Pyro is a distributed objects server. It
just make some remote python objects available to a client as if they where
local.

2) User interaction
-------------------

To avoid the need for the final user to write python scripts and run them to do
something, we need a simple working environment which gives the possibility to
interactively use the server objects. That's where ipython solves lots of
problems.

ipython is an enhanced python shell. It let the user runs python code, but has
many powerfull features :

 - history, even between sessions
 - colorized and customizable traceback
 - code completion
 - magic commands
 - much more...
 
ipython is on the client-side of PyMAD. In fact, there is a special client which
connects to the remote objects of PyMAD, launch an embbeded ipython instance,
and make the remote objects available to the user, in the global namespace of
ipython. This way, the user can drive the spectrometer through the methods of
these objects, or can build complex scripts to do complex actions.

PyMAD also use the magic commands to make a simple MAD-like command interpreter.
The magic functions use TPG (Toy Parser Generator), a easy-to-use parser based
on regexp. These MAD-like commands are for users which don't know about python,
but also to make shortcuts, to avoid the need to write several lines of normal
python code to do some complex but repetitive tasks.

One important point is that PyMAD can understand both syntax, which can be
combined. Most of the time, MAD-like commands will be used, but python code can
be more powerfull to do expert measures (with automatic feedback interaction
according to the results), or to prototype a new complex command.

3) ipython needs
----------------

In order to give users all these powerfull features, PyMAD needs to interacts
very closely with ipython. In the actual state of ipython we patch some internal
classes, by rebinding methods to custom ones. This is not very clean, and can
lead to problems if internal structures of new ipyton releases change.

So, here is the main PyMAD interactions needed :

  Catch custom (PyMAD) exceptions (now done with rebinding
IPython.iplib.InteractiveShell.runcode() method), with the possibility to
get the inital text code involved in the exception. For the moment, in the
runcode() method, we only get the code object, from which it is impossible to
retreive the text.

  Add some new matchers for completion. As PyMAD use remote
objects, completion only shows the client Pyro proxy. So we added a new
matcher by adding a IPython.iplib.MagicCompleter.proxy_matches() method,
and insert this matcher in ipshell.IP.Completer.matchers list. The new
matcher get the object (from the text param), call a special method on this
object which returns all available attributes (in fact, only these we want
to show to the user). But it does not work anymore in 0.6.12 :o(

  In the same way as matchers, get the docstring from the
remote object instead of the client one when using 'object?' syntaxe.(don't now
how to do this, but it could be done on the same idea: calling a special
method on the object, method returning the doc of our remote object).

  New exception handler.

  Prevent objects from beeing deleted from global namespace.

  Prompt modification at run-time (already solved. I have to do some tests)
 
  Access to the command-line interpreter, to have ipython interprets code has if
  it was entered through keyboard (ie make difference between magic commands and
  normal python code).

From Fernando.Perez at colorado.edu  Wed Mar 16 14:18:11 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Wed, 16 Mar 2005 12:18:11 -0700
Subject: [IPython-dev] IPython officially in Fedora Extras
Message-ID: <42388673.3060900@colorado.edu>

Hi all,

thanks to S. Walton for this heads-up:

http://download.fedora.redhat.com/pub/fedora/linux/extras/3/i386/ipython-0.6.11-2.noarch.rpm

IPython is now officially a Fedora Extras package :)

Besides making me feel warm and fuzzy for 5 seconds, those of you on Fedora 
can now forget about downloading it manually, since it will come to you via 
the Extras repo.  In Fedora 3 the Extras repo needs to be added manually, but 
it will be active by default in FC4.

To whoever was responsible for getting this into Fedora, many thanks (I have 
no idea who did it, since I was never contacted).  If you are on this list, 
let me know in case you want to be added directly to a short list of package 
maintainers who get a special heads-up message from me whenever I make a release.

Regards,

f



From Fernando.Perez at colorado.edu  Wed Mar 16 14:38:19 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Wed, 16 Mar 2005 12:38:19 -0700
Subject: [IPython-dev] Completer in ipython 0.6.12
In-Reply-To: <200503161656.15525.mantegazza@ill.fr>
References: <200503161656.15525.mantegazza@ill.fr>
Message-ID: <42388B2B.6040700@colorado.edu>

Fr?d?ric Mantegazza wrote:

> ipshell.IP.Completer.matchers.insert(0, 'proxy_matches')
> IPython.iplib.MagicCompleter.proxy_matches = proxy_matches
> 
> I saw that you changed lot of things in the iplib.py file, and I don't know 
> how to have my patch working. Could your explain me how things works, now ?

It should be relatively straightforward.  It's just that now instead of being 
called by name, the matchers list actually contains the raw python methods. 
You probably want to try something along the follwing lines (untested, typed 
up in my mail client):

import new

ipshell.IP.Completer.proxy_matches = new.instancemethod(proxy_matches,
                                                         ipshell.IP.Completer)

ipshell.IP.Completer.matchers.insert(0, ipshell.IP.Completer.proxy_matches)


Or some minor variation thereof.  Please let me know if it works, and the 
actual syntax that makes it work.  This is something which I can then expose 
with a public function that wraps some of this boilerplate.

Best,

f



From Fernando.Perez at colorado.edu  Wed Mar 16 14:44:48 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Wed, 16 Mar 2005 12:44:48 -0700
Subject: [IPython-dev] Some ideas for more hooks
In-Reply-To: <200503161656.16609.mantegazza@ill.fr>
References: <200503161656.16609.mantegazza@ill.fr>
Message-ID: <42388CB0.4070903@colorado.edu>

Fr?d?ric Mantegazza wrote:
> Hello Fernando,
> 
> Here is a little paper to explain the (unusual) way we are using ipython in 
> our application (PyMAD). This is a summary of our needs of internal 
> interaction with ipython to make something very powerfull.
> 
> I hope it will help you to re-design ipython with user customizable hooks.

OK, this is excellent.  Some of these will probably take time, but I'll use 
this as a driving document for how to best integrate ipython with a 
third-party project.

And anyone else out there who uses ipython for their engine: please send me 
something like this, which spells out which public parts make sense for your 
project to be cleanly exposed.  It's hard for me to guess ahead of time what 
people may need, as much as I try.   A document like this allows me to 
actually organize things for real-world usability, instead of theoretical (and 
perhaps irrelevant) benefits I can come up with.

> PS : PyMAD is a software we are developping to control spectrometers, big 
> neutrons instruments.

Do you have a public page mentioning it?  I've made a little list of links to 
projects which use ipython, I'd like to add yours.

> 3) ipython needs
> ----------------

[...]

for now, if I don't comment on something, assume I got it and will put it on 
my todo list.

>   New exception handler.

I need more specific details than this.

>   Prevent objects from beeing deleted from global namespace.

What do you mean?  Do you want to block the 'del' operator?  Please give me 
better details.  IPython's namespace handling is a bit delicate, so this has 
to be done with care.

>   Prompt modification at run-time (already solved. I have to do some tests)

Let me know if it worked.

>   Access to the command-line interpreter, to have ipython interprets code has if
>   it was entered through keyboard (ie make difference between magic commands and
>   normal python code).

This is the hardest, and yet it is very important.  I'll have to do it for 
other reasons, but it may take a bit longer.

Again, thanks for the details.  A document like this is actually _useful_ to 
me, and from it I can move on with specific improvements.

Best,

f



From vivainio at kolumbus.fi  Wed Mar 16 16:49:45 2005
From: vivainio at kolumbus.fi (Ville Vainio)
Date: Wed, 16 Mar 2005 23:49:45 +0200
Subject: [IPython-dev] Expanding $a to 'f1 f2' when $a is ['f1','f2']
Message-ID: <1111009785.9409.7.camel@localhost.localdomain>

One often deals with lists of filenames when doing shellish stuff. It
would be quite useful if you could use $flist in shell commands to say
'list of files', i.e. a list of file names separated by space. As it
stands, it expands to the string representation of the list, which is
almost never useful in shell commands. I've done

for f in junk:
    rm $f

way too often lately, I'd just like to do 

rm $junk



From Fernando.Perez at colorado.edu  Wed Mar 16 17:18:43 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Wed, 16 Mar 2005 15:18:43 -0700
Subject: [IPython-dev] Expanding $a to 'f1 f2' when $a is ['f1','f2']
In-Reply-To: <1111009785.9409.7.camel@localhost.localdomain>
References: <1111009785.9409.7.camel@localhost.localdomain>
Message-ID: <4238B0C3.5000801@colorado.edu>

Ville Vainio wrote:
> One often deals with lists of filenames when doing shellish stuff. It
> would be quite useful if you could use $flist in shell commands to say
> 'list of files', i.e. a list of file names separated by space. As it
> stands, it expands to the string representation of the list, which is
> almost never useful in shell commands. I've done
> 
> for f in junk:
>     rm $f
> 
> way too often lately, I'd just like to do 
> 
> rm $junk

Sorry, but the amount of special-casing that this would require is almost 
guaranteed to break somewhere.  Keep in mind that you can evaluate arbitrary 
code in $ expressions, so you can do:

rm ${' '.join(junk)}

Not the cleanest, but it works.

The problem is that I simply don't see a way to generically distinguish when 
$foo should be evaluated as a list joined on whitespace and when it should be 
treated as a list.  The $ evaluation mechanism is actually a full-blown 
generic interpolation module, Ka-Ping Yee's Itpl, from PEP-215:

http://www.python.org/peps/pep-0215.html

How do you propose such a mechanism do what you want in a generic, safe way? 
How can I determine when each kind of expansion to apply?  Unless a clean 
solution can be provided, this one stays as is.

Best,

f



From mantegazza at ill.fr  Thu Mar 17 07:18:23 2005
From: mantegazza at ill.fr (=?iso-8859-15?q?Fr=E9d=E9ric_Mantegazza?=)
Date: Thu, 17 Mar 2005 13:18:23 +0100
Subject: [IPython-dev] Completer in ipython 0.6.12
In-Reply-To: <42388B2B.6040700@colorado.edu>
References: <200503161656.15525.mantegazza@ill.fr>
	<42388B2B.6040700@colorado.edu>
Message-ID: <200503171318.23890.mantegazza@ill.fr>

Le Mercredi 16 Mars 2005 20:38, Fernando Perez a ?crit :

> > ipshell.IP.Completer.matchers.insert(0, 'proxy_matches')
> > IPython.iplib.MagicCompleter.proxy_matches = proxy_matches
>
> ipshell.IP.Completer.proxy_matches = new.instancemethod(proxy_matches,
                                                          ipshell.IP.Completer)
>
> ipshell.IP.Completer.matchers.insert(0,
                                       ipshell.IP.Completer.proxy_matches) 
>
>
> Or some minor variation thereof.  Please let me know if it works, and the
> actual syntax that makes it work.

Well, it works, but not exactly in the same way as before. Now, I get all 
matches, ie the standard ones, and mine. But I would like to see only 
mines, ie do not call other matchers if mine returns something.

> This is something which I can then expose with a public function that
> wraps some of this boilerplate. 

Yes, please ! If youd do, it may be possible to the have both behaviours: as 
it is now, ie returning all matches from all matcher methods; or only 
return matches from the first matcher method which returns somethings. 
Then, matcher methods order is to take in account.

Thank's,

-- 
   Fr?d?ric



From mantegazza at ill.fr  Thu Mar 17 07:33:26 2005
From: mantegazza at ill.fr (=?iso-8859-15?q?Fr=E9d=E9ric_Mantegazza?=)
Date: Thu, 17 Mar 2005 13:33:26 +0100
Subject: [IPython-dev] Some ideas for more hooks
In-Reply-To: <42388CB0.4070903@colorado.edu>
References: <200503161656.16609.mantegazza@ill.fr>
	<42388CB0.4070903@colorado.edu>
Message-ID: <200503171333.27199.mantegazza@ill.fr>

(Fernando, you sent your answer both on my personnal address and on ipython 
list, but I only received it from my address. Is it normal ? My mailer does 
not remove any message, even if it is cross-posted...)

Le Mercredi 16 Mars 2005 20:44, Fernando Perez a ?crit :

> > I hope it will help you to re-design ipython with user customizable
> > hooks.
>
> OK, this is excellent.  Some of these will probably take time, but I'll
> use this as a driving document for how to best integrate ipython with a
> third-party project.
>
> And anyone else out there who uses ipython for their engine: please send
> me something like this, which spells out which public parts make sense
> for your project to be cleanly exposed.  It's hard for me to guess ahead
> of time what people may need, as much as I try.   A document like this
> allows me to actually organize things for real-world usability, instead
> of theoretical (and perhaps irrelevant) benefits I can come up with.

I will update my document if I have new ideas or needs ;o)

> > PS : PyMAD is a software we are developping to control spectrometers,
> > big neutrons instruments.
>
> Do you have a public page mentioning it?  I've made a little list of
> links to projects which use ipython, I'd like to add yours.

We do not have public page, but Ican make a little paper for you (I already 
did for the Pyro project page).

> >   New exception handler.
>
> I need more specific details than this.

I didn't post it, but I have a little piece of code to overload 
IPython.iplib.InteractiveShell.runcode() method. In this method, I just 
catch our PyMAD project exceptions, and print a message instead of the 
traceback.

I also catch a Pyro exception. I would like to be able here to get the code 
which lead to the error, because I would like to make an automatic ribind 
on the pyro object which as failed.

> >   Prevent objects from beeing deleted from global namespace.
>
> What do you mean?  Do you want to block the 'del' operator?  Please give
> me better details.  IPython's namespace handling is a bit delicate, so
> this has to be done with care.

Yes, I want to avoid the user to delete some our Pyro remote object proxies, 
through the del operator. The idea is to give ipython a list of objects 
which can't be deleted.

But I understand it is not very easy to do. Let it at the bottom of your 
todo list !

-- 
   Fr?d?ric



From vivainio at kolumbus.fi  Thu Mar 17 10:07:27 2005
From: vivainio at kolumbus.fi (Ville Vainio)
Date: Thu, 17 Mar 2005 17:07:27 +0200
Subject: [IPython-dev] Expanding $a to 'f1 f2' when $a is ['f1','f2']
In-Reply-To: <4238B0C3.5000801@colorado.edu>
References: <1111009785.9409.7.camel@localhost.localdomain>
	<4238B0C3.5000801@colorado.edu>
Message-ID: <1111072047.16515.6.camel@localhost.localdomain>

On Wed, 2005-03-16 at 15:18 -0700, Fernando Perez wrote:

> Sorry, but the amount of special-casing that this would require is almost 
> guaranteed to break somewhere.  Keep in mind that you can evaluate arbitrary 
> code in $ expressions, so you can do:
> 
> rm ${' '.join(junk)}
> 
> Not the cleanest, but it works.
> 
> The problem is that I simply don't see a way to generically distinguish when 
> $foo should be evaluated as a list joined on whitespace and when it should be 
> treated as a list.  The $ evaluation mechanism is actually a full-blown 
> generic interpolation module, Ka-Ping Yee's Itpl, from PEP-215:
> 
> http://www.python.org/peps/pep-0215.html
> 
> How do you propose such a mechanism do what you want in a generic, safe way? 

1. If it's just $junk (i.e. not ${junk})

2. not isinstance(junk, basestring)

3. iter(junk) does not raise TypeError

-> Expand it as " ".join(junk)





From Fernando.Perez at colorado.edu  Thu Mar 17 13:28:50 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu, 17 Mar 2005 11:28:50 -0700
Subject: [IPython-dev] Expanding $a to 'f1 f2' when $a is ['f1','f2']
In-Reply-To: <1111072047.16515.6.camel@localhost.localdomain>
References: <1111009785.9409.7.camel@localhost.localdomain>
	<4238B0C3.5000801@colorado.edu>
	<1111072047.16515.6.camel@localhost.localdomain>
Message-ID: <4239CC62.7020608@colorado.edu>

Ville Vainio wrote:
> On Wed, 2005-03-16 at 15:18 -0700, Fernando Perez wrote:
> 
> 
>>Sorry, but the amount of special-casing that this would require is almost 
>>guaranteed to break somewhere.  Keep in mind that you can evaluate arbitrary 
>>code in $ expressions, so you can do:
>>
>>rm ${' '.join(junk)}
>>
>>Not the cleanest, but it works.
>>
>>The problem is that I simply don't see a way to generically distinguish when 
>>$foo should be evaluated as a list joined on whitespace and when it should be 
>>treated as a list.  The $ evaluation mechanism is actually a full-blown 
>>generic interpolation module, Ka-Ping Yee's Itpl, from PEP-215:
>>
>>http://www.python.org/peps/pep-0215.html
>>
>>How do you propose such a mechanism do what you want in a generic, safe way? 
> 
> 
> 1. If it's just $junk (i.e. not ${junk})
> 
> 2. not isinstance(junk, basestring)
> 
> 3. iter(junk) does not raise TypeError
> 
> -> Expand it as " ".join(junk)

OK, your proposal sounds very reasonable in principle.  However, I'm afraid 
that considering how the Itpl code is written, it's not that easy to do. 
These strings with $ in them get fed wholesale to the itpl machinery, which 
does the splitting on chunks at instantiation time, and then runs a fast 
eval() loop over all chunks when str() is called.

I simply don't see a clean way to inject all this special-casing logic into 
it, and I'm afraid it may break things (besides having a performance impact). 
    I really think the effort/benefit ratio is against this one, sorry.

Best,

f



From Fernando.Perez at colorado.edu  Thu Mar 17 13:39:00 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu, 17 Mar 2005 11:39:00 -0700
Subject: [IPython-dev] Completer in ipython 0.6.12
In-Reply-To: <200503171318.23890.mantegazza@ill.fr>
References: <200503161656.15525.mantegazza@ill.fr>
	<42388B2B.6040700@colorado.edu> <200503171318.23890.mantegazza@ill.fr>
Message-ID: <4239CEC4.1080009@colorado.edu>

Fr?d?ric Mantegazza wrote:

> Well, it works, but not exactly in the same way as before. Now, I get all 
> matches, ie the standard ones, and mine. But I would like to see only 
> mines, ie do not call other matchers if mine returns something.
> 
> 
>>This is something which I can then expose with a public function that
>>wraps some of this boilerplate. 
> 
> 
> Yes, please ! If youd do, it may be possible to the have both behaviours: as 
> it is now, ie returning all matches from all matcher methods; or only 
> return matches from the first matcher method which returns somethings. 
> Then, matcher methods order is to take in account.

OK, there was a change (by user request) in .12 to merge all namespaces in 
completions.  Now it turns out you don't like it :)  It should be possible to 
make this configurable, though by default the behavior will be the new one. 
But I'll give you a flag you can set to restore the old one.  Give me a few 
days though.  And I'll wrap the matcher addition into a simple public function.

Best,

f



From Fernando.Perez at colorado.edu  Thu Mar 17 13:43:04 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu, 17 Mar 2005 11:43:04 -0700
Subject: [IPython-dev] Some ideas for more hooks
In-Reply-To: <200503171333.27199.mantegazza@ill.fr>
References: <200503161656.16609.mantegazza@ill.fr>
	<42388CB0.4070903@colorado.edu> <200503171333.27199.mantegazza@ill.fr>
Message-ID: <4239CFB8.1010701@colorado.edu>

Fr?d?ric Mantegazza wrote:
> (Fernando, you sent your answer both on my personnal address and on ipython 
> list, but I only received it from my address. Is it normal ? My mailer does 
> not remove any message, even if it is cross-posted...)

No, it's not normal.  I sent it to the list, so you should have gotten that.

>>>PS : PyMAD is a software we are developping to control spectrometers,
>>>big neutrons instruments.
>>
>>Do you have a public page mentioning it?  I've made a little list of
>>links to projects which use ipython, I'd like to add yours.
> 
> 
> We do not have public page, but Ican make a little paper for you (I already 
> did for the Pyro project page).

That would be very nice.

>>>  New exception handler.
>>
>>I need more specific details than this.
> 
> 
> I didn't post it, but I have a little piece of code to overload 
> IPython.iplib.InteractiveShell.runcode() method. In this method, I just 
> catch our PyMAD project exceptions, and print a message instead of the 
> traceback.
> 
> I also catch a Pyro exception. I would like to be able here to get the code 
> which lead to the error, because I would like to make an automatic ribind 
> on the pyro object which as failed.

I need to see this example.  Would you mind completing the little draft you 
sent with this info and code examples?  I'd like to keep that whole document 
as a guide for the next few weeks worth of work, and it's much better if it's 
organized into a real document rather than scattered in pieces on an email thread.

>>>  Prevent objects from beeing deleted from global namespace.
>>
>>What do you mean?  Do you want to block the 'del' operator?  Please give
>>me better details.  IPython's namespace handling is a bit delicate, so
>>this has to be done with care.
> 
> 
> Yes, I want to avoid the user to delete some our Pyro remote object proxies, 
> through the del operator. The idea is to give ipython a list of objects 
> which can't be deleted.
> 
> But I understand it is not very easy to do. Let it at the bottom of your 
> todo list !

You could add a syntax prefilter yourself, which blocks on 'del foo' with an 
error message.  There is no way to prevent all avenues for deletion (exec can 
fool you), but this would cover most normal cases.  And you can do that today, 
with the existing machinery in ipython.  Look at the manual for examples of 
syntax prefilters.

Best,

f



From vivainio at kolumbus.fi  Thu Mar 17 16:39:52 2005
From: vivainio at kolumbus.fi (Ville Vainio)
Date: Thu, 17 Mar 2005 23:39:52 +0200
Subject: [IPython-dev] Expanding $a to 'f1 f2' when $a is ['f1','f2']
In-Reply-To: <4239CC62.7020608@colorado.edu>
References: <1111009785.9409.7.camel@localhost.localdomain>
	<4238B0C3.5000801@colorado.edu>
	<1111072047.16515.6.camel@localhost.localdomain>
	<4239CC62.7020608@colorado.edu>
Message-ID: <1111095592.8989.2.camel@localhost.localdomain>

On Thu, 2005-03-17 at 11:28 -0700, Fernando Perez wrote:

> OK, your proposal sounds very reasonable in principle.  However, I'm afraid 
> that considering how the Itpl code is written, it's not that easy to do. 

Fair enough - I can get by 

rm $flist(files)

so it's not a showstopper for me anyway.




From mantegazza at ill.fr  Fri Mar 18 02:14:16 2005
From: mantegazza at ill.fr (=?iso-8859-15?q?Fr=E9d=E9ric_Mantegazza?=)
Date: Fri, 18 Mar 2005 08:14:16 +0100
Subject: [IPython-dev] Completer in ipython 0.6.12
In-Reply-To: <4239CEC4.1080009@colorado.edu>
References: <200503161656.15525.mantegazza@ill.fr>
	<200503171318.23890.mantegazza@ill.fr> <4239CEC4.1080009@colorado.edu>
Message-ID: <200503180814.16837.mantegazza@ill.fr>

Le Jeudi 17 Mars 2005 19:39, Fernando Perez a ?crit :

> OK, there was a change (by user request) in .12 to merge all namespaces
> in completions.  Now it turns out you don't like it :)  It should be
> possible to make this configurable, though by default the behavior will
> be the new one. But I'll give you a flag you can set to restore the old
> one.  Give me a few days though.  And I'll wrap the matcher addition into
> a simple public function.

This is not an urgency. As far as I can get at least my own matcher, this is 
enought for now.

-- 
   Fr?d?ric



From mantegazza at ill.fr  Fri Mar 18 02:20:54 2005
From: mantegazza at ill.fr (=?iso-8859-15?q?Fr=E9d=E9ric_Mantegazza?=)
Date: Fri, 18 Mar 2005 08:20:54 +0100
Subject: [IPython-dev] Some ideas for more hooks
In-Reply-To: <4239CFB8.1010701@colorado.edu>
References: <200503161656.16609.mantegazza@ill.fr>
	<200503171333.27199.mantegazza@ill.fr> <4239CFB8.1010701@colorado.edu>
Message-ID: <200503180820.54893.mantegazza@ill.fr>

Le Jeudi 17 Mars 2005 19:43, Fernando Perez a ?crit :

> No, it's not normal.  I sent it to the list, so you should have gotten
> that.

Strange. I think that the mailing-list robot as seen that you cross-posted 
both to me and the list, detected that I'm on the list, so didn't send the 
message by the list way...  Maybe...

> > We do not have public page, but Ican make a little paper for you (I
> > already did for the Pyro project page).
>
> That would be very nice.

I will spend this morning to work on both updating my document and writing 
this paper.

> I need to see this example.  Would you mind completing the little draft
> you sent with this info and code examples?  I'd like to keep that whole
> document as a guide for the next few weeks worth of work, and it's much
> better if it's organized into a real document rather than scattered in
> pieces on an email thread.

Ok.

> You could add a syntax prefilter yourself, which blocks on 'del foo' with
> an error message.  There is no way to prevent all avenues for deletion
> (exec can fool you), but this would cover most normal cases.  And you can
> do that today, with the existing machinery in ipython.  Look at the
> manual for examples of syntax prefilters.

I forgot this filters ! I think I can do the job with this feature.

-- 
   Fr?d?ric



From mantegazza at ill.fr  Fri Mar 18 07:21:31 2005
From: mantegazza at ill.fr (=?iso-8859-15?q?Fr=E9d=E9ric_Mantegazza?=)
Date: Fri, 18 Mar 2005 13:21:31 +0100
Subject: [IPython-dev] PyMAD/IPython which list updated
Message-ID: <200503181321.31361.mantegazza@ill.fr>

Hello,

Here is an updated version of our which list for IPython. I attached both 
text (reStructured) and html versions.

Cheers,

-- 
   Fr?d?ric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20050318/f5319e84/attachment.html>
-------------- next part --------------
======================================================================
This document describes the interactions needed by PyMAD with IPython.
======================================================================

:Authors: 
    Fr?d?ric Mantegazza, 

:Version: $Rev: 1291 $

1) Introduction
---------------

The philosophy of **PyMAD** is to give to the user a high-level set of tools to
drive a spectrometer. The idea is to directly make some internal python
objects available to the user, and let him combine them.

So, The final user just calls some methods of a few high-level objects to drive
the spectrometer. This as the advantage to make all the python scripting stuff
available, to build some new high level tools.

As we also need a system which can be used from several places, we use a
client/server framework, with **Pyro**. **Pyro** is a distributed objects
server. It just make some remote python objects available to a client as if they
where local.

2) User interaction
-------------------

To avoid the need for the final user to write python scripts and run them to do
something, we need a simple working environment which gives the possibility to
interactively use the server objects. That's where **IPython** solves lots of
problems !

**IPython** is an enhanced python shell. It let the user runs python code, but
has many powerfull features :

- history, even between sessions
- colorized and customizable traceback
- code completion
- magic commands
- much more...
 
**IPython** is on the client-side of **PyMAD**. In fact, there is a special
client which connects to the remote objects of **PyMAD**, launch an embbeded
**IPython** instance, and make the remote objects available to the user, in the
global namespace of **IPython**. This way, the user can drive the spectrometer
through the methods of these objects, or can build complex scripts to do complex
actions.

**PyMAD** also use the magic commands to make a simple command interpreter.
The magic functions use TPG (Toy Parser Generator), a easy-to-use parser based
on regexp. These MAD-like commands are for users which don't know about python,
but also to make shortcuts, to avoid the need to write several lines of normal
python code to do some complex but repetitive tasks.

One important point is that **PyMAD** can understand both syntax, which can be
combined. Most of the time, simple commands will be used, but python code can
be more powerfull to do expert measures (with automatic feedback interaction
according to the results), or to prototype a new complex command.

3) **IPython** needs
--------------------

In order to give users all these powerfull features, **PyMAD** needs to
interacts very closely with **IPython**. In the actual state of IPython we patch
some internal classes, by rebinding methods to custom ones. This is not very
clean, and can lead to problems if internal structures of new ipyton releases
change.

So, here is the main **PyMAD** interactions needed:

1. Catch custom *PyMADError* exceptions (now done with rebinding
   IPython.iplib.InteractiveShell.runcode() method), **with the possibility to
   get the inital text code involved in the exception**. For the moment, in the
   runcode() method, we only get the code object, from which it is impossible to
   retreive the text. Here is the code used::

    def runcode(self, code_obj):
        """Execute a code object.
    
        When an exception occurs, self.showtraceback() is called to display a
        traceback."""
        log = Logger()
        message = MessageLogger()
    
        # Set our own excepthook in case the user code tries to call it
        # directly, so that the IPython crash handler doesn't get triggered
        old_excepthook,sys.excepthook = sys.excepthook, self.excepthook
        try:
            try:
                exec code_obj in self.locals
            finally:
                # Reset our crash handler in place
                sys.excepthook = old_excepthook
    
        except SystemExit, message:
            if str(message)=='IPythonExit':
                raise
            else:
                self.resetbuffer()
                self.showtraceback()
                warn( __builtin__.exit,level=1)
                
        # We just add this few lines
        #except AttributeError, message:
            #print message
            # TODO: Use the same syntax (color) as IPython
            
        except PyMADError, exc:
            message.error(exc.standardStr())
            #log.exception("Console execution error")
       
        except Pyro.errors.ConnectionClosedError:
            message.critical("Pyro connexion closed")
            log.exception("Console execution error")
            # TODO: get the object and call rebindURI()
    
        except:
            self.showtraceback()
        else:
            if code.softspace(sys.stdout, 0):
                print

2. Add some new matchers for completion. As **PyMAD** uses remote objects,
   completion only shows the client Pyro proxy. So we added a new matcher by
   adding a IPython.iplib.MagicCompleter.proxy_matches() method, and insert this
   matcher in ipshell.IP.Completer.matchers list. The new matcher get the object
   (from the text param), call a special method on this object which returns all
   available attributes (in fact, only these we want to show to the user). Give
   the possibility to return all matchers, or only the no None first. Here is
   the code used::

    def proxy_matches(self, text, state):
        """ Get the attribute of a remove Pyro object.
        """
        log = Logger('client')
    
        # Another option, seems to work great. Catches things like ''.<tab>
        m = re.match(r"(\S+(\.\w+)*)\.(\w*)$", text)
    
        if not m:
            return []
        expr, attr = m.group(1, 3)
        matches = []
        try:
            object = eval(expr, self.namespace)
            if isinstance(object, Pyro.core.DynamicProxyWithAttrs):
                words = object.getAvailableAttributes()
                #if hasattr(object,'__class__'):
                    #words.append('__class__')
                    #words = words + get_class_members(object.__class__)
                matches = []
                n = len(attr)
                if words:
                    for word in words:
                        if word[:n] == attr and word != "__builtins__":
                            matches.append("%s.%s" % (expr, word))
    
        except NameError:
            pass
            
        except Pyro.errors.ConnectionClosedError:
            log.error("Connexion closed")
            object.adapter.rebindURI() # Should be moved to runcode()
            matches = [""]
    
        return matches
    
    ipshell.IP.Completer.proxy_matches = new.instancemethod(proxy_matches,
                                                            ipshell.IP.Completer)
    ipshell.IP.Completer.matchers.insert(0, ipshell.IP.Completer.proxy_matches)

3. In the same way as matchers, get the docstring from the remote object instead
   of the client one when using 'object?' syntaxe. This could be done on the
   same idea: calling a special method on the object, method returning the doc
   of our remote object).

4. New exception handler. Here, the idea is to be able to present different kind
   of exceptions in different ways. Some will only print a simple message, some
   others will print the entire traceback (maybe a modified traceback).

5. Prevent objects from beeing deleted by *del* keyword [1]_.

6. Prompt modification at run-time [2]_.
 
7. Access to the command-line interpreter, to have **IPython** interprets code
   has if it was entered through keyboard (ie make difference between magic
   commands and normal python code).

.. [1] Can be done with pre-filters. Have to be tested, but should work.
.. [2] Can be done with prompt_specials_color dict. Have to be tested,
       but should work.

From vivainio at kolumbus.fi  Fri Mar 18 10:42:55 2005
From: vivainio at kolumbus.fi (Ville Vainio)
Date: Fri, 18 Mar 2005 17:42:55 +0200
Subject: [IPython-dev] Re: [IPython-user] New feature in CVS, testers?
In-Reply-To: <1111159421.14626.14.camel@localhost.localdomain>
References: <423AA1BD.3090801@colorado.edu>
	<1111159421.14626.14.camel@localhost.localdomain>
Message-ID: <1111160576.14626.21.camel@localhost.localdomain>

(My apologies for spamming the list repeatedly w/ separate mails)

> Tried it now, there is the problem that ipython steals the stdin and
> stdout by using popen, and as such you don't get interactive feedback
> from the progress of the command (try 'find /' to see what I mean) and
> programs that want the full control of the terminal (try 'vi') won't
> work at all.
> 
> I wonder whether 'subprocess' module would be helpful here, and whether
> it can be used w/ ipython due to backwards compatibility reasons.

Of course it's called 'subprocess'
http://docs.python.org/lib/module-subprocess.html

It's also available for older Python versions as separate download I
think.

It might work to create a subprocess.Popen, inheriting stdin from the
current process and capturing stdout and stderr to file-like objects
that both print the output and store it in strings for future
processing.





From Fernando.Perez at colorado.edu  Fri Mar 18 14:14:30 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Fri, 18 Mar 2005 12:14:30 -0700
Subject: [IPython-dev] Re: [IPython-user] New feature in CVS, testers?
In-Reply-To: <1111160576.14626.21.camel@localhost.localdomain>
References: <423AA1BD.3090801@colorado.edu>
	<1111159421.14626.14.camel@localhost.localdomain>
	<1111160576.14626.21.camel@localhost.localdomain>
Message-ID: <423B2896.4080507@colorado.edu>

Ville Vainio wrote:
> (My apologies for spamming the list repeatedly w/ separate mails)
> 
> 
>>Tried it now, there is the problem that ipython steals the stdin and
>>stdout by using popen, and as such you don't get interactive feedback
>>from the progress of the command (try 'find /' to see what I mean) and
>>programs that want the full control of the terminal (try 'vi') won't
>>work at all.
>>
>>I wonder whether 'subprocess' module would be helpful here, and whether
>>it can be used w/ ipython due to backwards compatibility reasons.
> 
> 
> Of course it's called 'subprocess'
> http://docs.python.org/lib/module-subprocess.html
> 
> It's also available for older Python versions as separate download I
> think.

OK, this looks very promising, many thanks.  If I can backport it/get it for 
2.2, I can just shove it into the distribution of ipython itself.  Thanks for 
catching the stdin problems, I shouldn't have sent this after a late night 
coding frenzy with a cold :)

I'll work on it this weekend, stay tuned.

Best,

f



From Fernando.Perez at colorado.edu  Fri Mar 18 18:40:33 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Fri, 18 Mar 2005 16:40:33 -0700
Subject: [IPython-dev] A plea for help with a threading bug...
Message-ID: <423B66F1.3060705@colorado.edu>

Hi all,

I got this bug report:

http://www.scipy.net/roundup/ipython/issue30

The short of it:

Presing 'tab' in order to activate the completer sometimes
segfaults ipython, when one of the thread flags is used
(gthread, wthread or pylab). Although this happens more
often with gthread, a similar crash is also possible with
wthread.

Since this is caused by some race condition, in order to
reproduce the problem, the TIMEOUT variable of
IPShellWX/IPShellGTK can be set to 1.

Steps to Reproduce:
1. start ipython -gthread
2. press 'tab' a couple of times
3. 'Segmentation fault'

or:
1. start ipython -wthread
2. press 'tab' a couple of times
3. 'Fatal Python error: GC object already tracked'


The bad news: I don't have a clue what to do here.  I'm hoping someone on the 
list who knows more about threading than I do might point me in the right 
direction.   Anything will help, since I have exactly _zero_ ideas at this point.

Best,

f



From Fernando.Perez at colorado.edu  Fri Mar 18 22:10:18 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Fri, 18 Mar 2005 20:10:18 -0700
Subject: [IPython-dev] Re: [IPython-user] A plea for help with a threading
	bug...
In-Reply-To: <mailman.2584.1111201306.905.ipython-dev@scipy.net>
References: <mailman.2584.1111201306.905.ipython-dev@scipy.net>
Message-ID: <423B981A.7080209@colorado.edu>

> Subject: Re: [IPython-user] A plea for help with a threading bug...
> From:  Andrew Straw <astraw at caltech.edu>
> Date: Fri, 18 Mar 2005 19:01:37 -0800
> To:  IPython-dev List <ipython-dev at scipy.net>> 
> 
> Well, since you asked, here's a guess:
> 
> Make sure there are no calls happening to the Python C API between 
> Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS.  If ipython is pure 
> Python, then this would suggest a problem with some other code written in C.
> 
> HTH,
> Andrew

Well, ipython _is_ 100% pure, pristine python code.  <TAB> calls the 
completion code, which ultimately will peek into the readline module, which 
does have a C component.  But that's part of python itself, not my code.

The question is, how in the world do I file a bug report against python on 
this, when I don't even know who may be causing it?  readline may not be the 
culprit: there are also pygtk and wx calls being made in there...

I hate threads.  I hate threads.  I hate threads.

Thanks, though.

f



From astraw at caltech.edu  Sat Mar 19 00:03:43 2005
From: astraw at caltech.edu (Andrew Straw)
Date: Fri, 18 Mar 2005 21:03:43 -0800
Subject: [IPython-dev] Re: [IPython-user] A plea for help with a threading
	bug...
In-Reply-To: <423B981A.7080209@colorado.edu>
References: <mailman.2584.1111201306.905.ipython-dev@scipy.net>
	<423B981A.7080209@colorado.edu>
Message-ID: <f18cef3a7f2dfa984f9ec9679d1147a4@caltech.edu>

I'm not sure how to run GDB with ipython, but with straight Python, do 
this:

Use a debug build of Python, and use it to build all extension modules 
so that all debugging symbols are included.  Copy 
Python-2.x/Misc/gdbinit to .gdbinit.  Run gdb.  Then type "target exec 
/path/to/your/debug/build/of/python" now type "run yourscript.py any 
args here".

Your program should now be running under gdb.  When a segfault happens, 
you can get a backtrace of the C stack (type "bt"), but you can also 
get a python stack backtrace with the magic provided by .gdbinit.  It's 
been a little while, but you type something like "up 2" do move 
yourself up the stack to a C call like "Py_Eval".  This Py_Eval 
(whatever it's really called) is the name of the C function that 
executes the Python bytecode.  So, once you're here you can execute the 
.gdbinit macros like "pyframe" and "pystack".  This way, you can find 
the Python stack the resulted in the bad call.

IIRC, gdb will tell you which thread crashed, which are running, and so 
on.  This may help track down the problem.

If the problem really is that some code was using the Python C API 
while the GIL is released (between Py_BEGIN_ALLOW_THREADS and 
Py_END_END_THREADS), all of this fancy debugging will probably only 
reveal the innards of some bit of code that eventually crashed, not 
necessarily the culprit.  Something like this could happen, for 
example, if one thread de-references an object which then gets 
de-alloced, but a call from another thread also de-references it 
without having acquired the GIL.

Hope that helps...
Andrew

On Mar 18, 2005, at 7:10 PM, Fernando Perez wrote:

>> Subject: Re: [IPython-user] A plea for help with a threading bug...
>> From:  Andrew Straw <astraw at caltech.edu>
>> Date: Fri, 18 Mar 2005 19:01:37 -0800
>> To:  IPython-dev List <ipython-dev at scipy.net>> Well, since you asked, 
>> here's a guess:
>> Make sure there are no calls happening to the Python C API between 
>> Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS.  If ipython is pure 
>> Python, then this would suggest a problem with some other code 
>> written in C.
>> HTH,
>> Andrew
>
> Well, ipython _is_ 100% pure, pristine python code.  <TAB> calls the 
> completion code, which ultimately will peek into the readline module, 
> which does have a C component.  But that's part of python itself, not 
> my code.
>
> The question is, how in the world do I file a bug report against 
> python on this, when I don't even know who may be causing it?  
> readline may not be the culprit: there are also pygtk and wx calls 
> being made in there...
>
> I hate threads.  I hate threads.  I hate threads.
>
> Thanks, though.
>
> f
>




From prabhu_r at users.sf.net  Sat Mar 19 05:58:24 2005
From: prabhu_r at users.sf.net (Prabhu Ramachandran)
Date: Sat, 19 Mar 2005 16:28:24 +0530
Subject: [IPython-dev] A plea for help with a threading bug...
In-Reply-To: <423B66F1.3060705@colorado.edu>
References: <423B66F1.3060705@colorado.edu>
Message-ID: <16956.1488.412879.700277@monster.linux.in>

>>>>> "FP" == Fernando Perez <Fernando.Perez at colorado.edu> writes:

    FP> Hi all, I got this bug report:

    FP> http://www.scipy.net/roundup/ipython/issue30
[...]
    FP> Since this is caused by some race condition, in order to
    FP> reproduce the problem, the TIMEOUT variable of
    FP> IPShellWX/IPShellGTK can be set to 1.

Well, why should you set the TIMEOUT to 1 in the first place?  Thats 1
millisecond between calls to the on_timer function.  Theoretically, if
it takes more than 1 millisecond to execute the runcode method this is
asking for trouble isn't it?

cheers,
prabhu



From prabhu_r at users.sf.net  Sat Mar 19 13:32:42 2005
From: prabhu_r at users.sf.net (Prabhu Ramachandran)
Date: Sun, 20 Mar 2005 00:02:42 +0530
Subject: [IPython-dev] A plea for help with a threading bug...
In-Reply-To: <1111241861.8673.15.camel@dxeon.localdomain>
References: <423B66F1.3060705@colorado.edu>
	<16956.1488.412879.700277@monster.linux.in>
	<1111241861.8673.15.camel@dxeon.localdomain>
Message-ID: <16956.28746.166250.987589@monster.linux.in>

>>>>> "YI" == Yariv Ido <yariv at vipe.technion.ac.il> writes:

    YI> Hi, First of all, the segmentation fault is caused by some
    YI> race condition.  Setting the TIMEOUT to 1 just increases the
    YI> probability of that happening, so you could debug it more
    YI> efficiently. If you would like to avoid setting it, try
    YI> running ipython with the -gthread flag, and pressing 'tab'
    YI> repeatedly...  More thoughts on the subject:

Well, I can't seem to reproduce this.  I even set TIMEOUT to 1.
Debian, ipython-0.6.12_cvs (updated a week or two ago), pygtk-2.6.2.
python-2.3.5.  

cheers,
prabhu



From Fernando.Perez at colorado.edu  Sat Mar 19 14:47:47 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Sat, 19 Mar 2005 12:47:47 -0700
Subject: [Fwd: Re: [IPython-dev] A plea for help with a threading bug...]
Message-ID: <423C81E3.30105@colorado.edu>

I'm forwarding Yariv's response, which was killed by the non-subscriber filter 
(Yariv, I just whitelisted this address for you manually, let me know if you 
use others).

best,

f

-------- Original Message --------
Subject: Re: [IPython-dev] A plea for help with a threading bug...
Date: Sat, 19 Mar 2005 16:17:41 +0200
From: Yariv Ido <yariv at vipe.stud.technion.ac.il>
Reply-To: yariv at vipe.stud.technion.ac.il
To: Prabhu Ramachandran <prabhu_r at users.sourceforge.net>
CC: Fernando Perez <Fernando.Perez at colorado.edu>,        IPython-dev List 
<ipython-dev at scipy.net>,        IPython-user List <ipython-user at scipy.net>, 
      Yariv <yariv at vipe.stud.technion.ac.il>
References: <423B66F1.3060705 at colorado.edu>	 
<16956.1488.412879.700277 at monster.linux.in>

Hi,

First of all, the segmentation fault is caused by some race condition.
Setting the TIMEOUT to 1 just increases the probability of that
happening, so you could debug it more efficiently. If you would like to
avoid setting it, try running ipython with the -gthread flag, and
pressing 'tab' repeatedly...
More thoughts on the subject:
1. This bug has nothing to do with the runcode method. You can just
return True as the first statement of the on_timer() method, and a
segfault would still occur.
2. I haven't had enough time to debug it, but by taking a look at the
readline built-in module, it seems that it releases the GIL every now
and then. My guess is that this is somehow related.
3. I've noticed that IPython's code has no
threads_enter()/threads_leave(). I don't know if it would solve this
particular problem, but it may fix the threads_init() call in win32 (And
thus fix the tremendous slowdown as reported on the bug tracker).

Yariv

On Sat, 2005-03-19 at 16:28 +0530, Prabhu Ramachandran wrote:
> >>>>> "FP" == Fernando Perez <Fernando.Perez at colorado.edu> writes:
> 
>     FP> Hi all, I got this bug report:
> 
>     FP> http://www.scipy.net/roundup/ipython/issue30
> [...]
>     FP> Since this is caused by some race condition, in order to
>     FP> reproduce the problem, the TIMEOUT variable of
>     FP> IPShellWX/IPShellGTK can be set to 1.
> 
> Well, why should you set the TIMEOUT to 1 in the first place?  Thats 1
> millisecond between calls to the on_timer function.  Theoretically, if
> it takes more than 1 millisecond to execute the runcode method this is
> asking for trouble isn't it?
> 
> cheers,
> prabhu
> 



From Fernando.Perez at colorado.edu  Sat Mar 19 14:50:48 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Sat, 19 Mar 2005 12:50:48 -0700
Subject: [IPython-dev] Re: [IPython-user] A plea for help with a threading
	bug...
In-Reply-To: <f18cef3a7f2dfa984f9ec9679d1147a4@caltech.edu>
References: <mailman.2584.1111201306.905.ipython-dev@scipy.net>
	<423B981A.7080209@colorado.edu>
	<f18cef3a7f2dfa984f9ec9679d1147a4@caltech.edu>
Message-ID: <423C8298.5040208@colorado.edu>

Andrew Straw wrote:
> I'm not sure how to run GDB with ipython, but with straight Python, do 
> this:
> 
> Use a debug build of Python, and use it to build all extension modules 
> so that all debugging symbols are included.  Copy 
> Python-2.x/Misc/gdbinit to .gdbinit.  Run gdb.  Then type "target exec 
> /path/to/your/debug/build/of/python" now type "run yourscript.py any 
> args here".

[ great, detailed instructions on thread debugging ]

Well, and here I was, hoping you'd say "Ah! you just forgot to run

import easy_threads

at the top of your code.  That should do it".  But hey, who am I to complain...

It seems like Yariv has narrowed it down to a readline bug, so I'm going to 
leave it at that (too swamped on my end with other things).  But many thanks 
for this message, since these instructions will come in handy more than once 
in the future, I'm afraid.

Take care,

f



From Fernando.Perez at colorado.edu  Sat Mar 19 14:53:19 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Sat, 19 Mar 2005 12:53:19 -0700
Subject: [Fwd: Re: [IPython-dev] A plea for help with a threading bug...]
In-Reply-To: <423C81E3.30105@colorado.edu>
References: <423C81E3.30105@colorado.edu>
Message-ID: <423C832F.4070409@colorado.edu>

[Yariv]

> 3. I've noticed that IPython's code has no
> threads_enter()/threads_leave(). I don't know if it would solve this
> particular problem, but it may fix the threads_init() call in win32 (And
> thus fix the tremendous slowdown as reported on the bug tracker).

Could you be more specific on where this should go?  And is there anyway 
anyone could test this on win32 and let me know if it helps any?  I'll be 
happy to add the changes, but only if I know they will help, and where to put 
them.

best,

f



From Fernando.Perez at colorado.edu  Sat Mar 19 20:06:00 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Sat, 19 Mar 2005 18:06:00 -0700
Subject: [IPython-dev] Re: [IPython-user] New feature in CVS, testers?
In-Reply-To: <1111160576.14626.21.camel@localhost.localdomain>
References: <423AA1BD.3090801@colorado.edu>
	<1111159421.14626.14.camel@localhost.localdomain>
	<1111160576.14626.21.camel@localhost.localdomain>
Message-ID: <423CCC78.6090803@colorado.edu>

Ville Vainio wrote:

> Of course it's called 'subprocess'
> http://docs.python.org/lib/module-subprocess.html
> 
> It's also available for older Python versions as separate download I
> think.
> 
> It might work to create a subprocess.Popen, inheriting stdin from the
> current process and capturing stdout and stderr to file-like objects
> that both print the output and store it in strings for future
> processing.

OK, I admit defeat and retreat.  I tried quite hard to get this to work across 
all cases using subprocess (it runs fine on 2.3), but couldn't.

I am therefore forced to back off the autocapture for shell commands, I'm 
afraid.  If you or anyone can figure out how to make this work reliably, 
addressing the various points you mentioned earlier (non-blocking stdin/out), 
I'd love to see a solution and will gladly incorporate it.  But after spending 
a fair amount of effort and failing to find a solid solution, I have to move on.

Some good came out of this, though: the %sc/%sx magics now capture to these 
special objects with the .l/.s/.n attributes which make their interactive use 
a lot easier.  So not all is lost:

In [6]: sc a=ls s*py

In [7]: a
Out[7]: 'sanner.py\nscopes.py\nsplot.py\nstrings.py'

In [8]: a.s
Out[8]: 'sanner.py scopes.py splot.py strings.py'

In [9]: a.l
Out[9]: ['sanner.py', 'scopes.py', 'splot.py', 'strings.py']


In fact, now that we have these, I have a question on which I'd like feedback. 
  I am very tempted to change the behavior of !! to capture to a string 
instead of a list.  This would make it a bit more uniform, since !!foo would 
just capture directly whatever !foo did, without any changes.  Using _.l would 
give you the list form if that's what you want, at any time.

I guess it's a question of which use is more common: the list form or the 
string form?  Capturing just to a list is also less expensive, since no 
additional processing is necessary, so I'm leaning strongly in that direction. 
This makes the capture very inexpensive, and the list form is only a .l away 
for those who need it, and they pay for it only when needed (instead of doing 
so on every capture).

So unless I hear a good counter argument, I'll switch !!foo over to plain 
string capture into one of these new objects.

And many thanks for the feedback and testing on this.

Best,

f



From Fernando.Perez at colorado.edu  Sun Mar 20 06:20:17 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Sun, 20 Mar 2005 04:20:17 -0700
Subject: [IPython-dev] PyMAD/IPython which list updated
In-Reply-To: <200503181321.31361.mantegazza@ill.fr>
References: <200503181321.31361.mantegazza@ill.fr>
Message-ID: <423D5C71.9090903@colorado.edu>

Thanks for this update.  Below follow a few more comments, it would be great 
if you could keep the doc updated as we hash out the issues.  I'm thinking a 
wiki would be great for this, but for now we don't have one.  So while I code, 
you can be the Fr?d?Wiki ;-)  I actually think most things will be cleared up 
in short order, so in reality there's no need to do much.

Fr?d?ric Mantegazza wrote:

>    1.
> 
>       Catch custom /PyMADError/ exceptions (now done with rebinding
>       IPython.iplib.InteractiveShell.runcode() method), *with the possibility to
>       get the inital text code involved in the exception*. For the moment, in
>       the runcode() method, we only get the code object, from which it is
>       impossible to retreive the text. Here is the code used:

OK, now, IPython's runsource looks as follows:

         # Case 3
         # We store the code source and object so that threaded shells and
         # custom exception handlers can access all this info if needed.
         self.code_to_run_src = source
         self.code_to_run = code
         # now actually execute the code object
         self.runcode(code)

This means that if you modify runcode(), when you catch an exception you can 
use the self.code_to_run_src attribute to get the source which was being run, 
and do whatever you want with it.  Note that the new runcode() at the end has 
a bit of flushing to do, which your modified version will also need to do:

...
         except:
             self.showtraceback()
         else:
             if code.softspace(sys.stdout, 0):
                 print
         # Flush out code object which has been run (and source)
         self.code_to_run = None
         self.code_to_run_src = ''


I still think you'll have to rebind runcode() to your own method.  I don't see 
an easy way to make this more flexible, without jumping through convoluted 
contortions.  It's technically possible to store an arbitrary list of 
additional exceptions and handlers.  But I think at this point, rather than 
trying to parametrize every possible combination in the universe, it's simply 
cleaner for you to override one method and do exactly what you want.  This 
part of the code rarely changes, so I think the risk for you is really minimal.

[... deleted big rant about how hard this was ...]

Scratch the previous paragraph,  I just realized this was rather easy to do.

It's funny how python does these things to you. You start arguing how 
something is not a good idea, because it's too complicated to do.  And when 
you make a little test to show why it's so, you simply end up solving the 
problem.  And it takes longer to argue about why its' hard, than to
actually implement a working solution.  No wonder we get hooked on this 
language, try doing the same thing in C, Perl, or just about anything else.

In case anyone is curious, here's a simple example (outside of ipython):

import sys

def myhandler(etype,value,tb):
     print '*** my handler ***'
     print 'etype:',etype
     print 'value:',value
     print 'tb:   ',tb
     print

def customcatch(src,exc,handler):
     try:
         exec src
     except exc:
         etype,value,tb = sys.exc_info()
         handler(etype,value,tb)

customcatch('range(3)[10]',(IndexError,ZeroDivisionError),myhandler)
customcatch('1/0',(IndexError,ZeroDivisionError),myhandler)
customcatch('1/0',(ZeroDivisionError,),myhandler)

# this won't be caught:
customcatch('{1:2}[10]',(IndexError,ZeroDivisionError),myhandler)

You can run the above, even with normal python, to see the effect.

Following this, IPython now has a facility for custom exception hadling. 
IPython can take a list of custom exceptions and a handler, and call that 
handler if any exception in the list matches.  Note that this can only be done 
for a single handler, because I can't (short of doing dynamic code generation, 
which I don't want to get into)  have multiple except clauses.  So your one 
handler would get the exception info for _any_ of the matched exceptions, and 
it would have to deal with them.

Please see the set_custom_exc method for details.  You can call it at runtime 
anywhere (__IPYTHON__.set_custom_exc(...)), or you can pass the class 
constructor the relevant information, via the

   custom_exceptions=((),None)

parameter.  This param has the list of custom exceptions and their handler 
(None selects an internal dummy default).  Here's what you get with

   custom_exceptions=((ZeroDivisionError,IndexError),None) :

In [1]: x=99

In [2]: y=0

In [3]: x/y
*** Simple custom exception handler ***
Exception type : exceptions.ZeroDivisionError
Exception value: integer division or modulo by zero
Traceback      : <traceback object at 0x40379cfc>
Source code    : x/y

In [4]: a=range(3)

In [5]: a[4]
*** Simple custom exception handler ***
Exception type : exceptions.IndexError
Exception value: list index out of range
Traceback      : <traceback object at 0x40379e14>
Source code    : a[4]


This system allows you to do basically anything you want, as far as exception 
reporting and control is concerned.  This is in CVS, please let me know how it 
goes.

>    2.
> 
>       Add some new matchers for completion. As *PyMAD* uses remote objects,
>       completion only shows the client Pyro proxy. So we added a new matcher by
>       adding a IPython.iplib.MagicCompleter.proxy_matches() method, and insert
>       this matcher in ipshell.IP.Completer.matchers list. The new matcher get
>       the object (from the text param), call a special method on this object
>       which returns all available attributes (in fact, only these we want to
>       show to the user). Give the possibility to return all matchers, or only
>       the no None first. Here is the code used:

Done.  A new rc parameter, readline_merge_completions, controls this (true by 
default).

New completers can now be added by using the __IPYTHON__.set_custom_completer 
method.  Plase see its docstring for details.

>    3.
> 
>       In the same way as matchers, get the docstring from the remote object
>       instead of the client one when using 'object?' syntaxe. This could be done
>       on the same idea: calling a special method on the object, method returning
>       the doc of our remote object).

OK, this is easy.  Just make sure that the objects IPython sees have a 
.getdoc() method; I've added a call to that into my object introspection code. 
  Have a look at inspect.getdoc, you can probably have your .getdoc method be 
like that, since it does a good job of cleaning up indentation for the 
docstring.  Your method should conform to the same API as inspect.getdoc: 
return None if nothing is found, return a string (formatted to your liking) 
otherwise.

In fact, the change is already in CVS, so feel free to play with it.  See 
OInspect.py's getdoc() function for details.

>    4.
> 
>       New exception handler. Here, the idea is to be able to present different
>       kind of exceptions in different ways. Some will only print a simple
>       message, some others will print the entire traceback (maybe a modified
>       traceback).

Well, there are already three exception handlers in ipython, switchable at 
runtime and init time via xmode.  If you need more flexibility, the first step 
would be to simply subclass AutoFormattedTB from ultraTB.py.  This is what 
IPython uses for its exception handler:

         # TraceBack handlers:
         # Need two, one for syntax errors and one for other exceptions.
         self.SyntaxTB = ultraTB.ListTB(color_scheme='NoColor')
         # This one is initialized with an offset, meaning we always want to
         # remove the topmost item in the traceback, which is our own internal
         # code. Valid modes: ['Plain','Context','Verbose']
         self.InteractiveTB = ultraTB.AutoFormattedTB(mode = 'Plain',
                                                      color_scheme='NoColor',
                                                      tb_offset = 1)

You could then reassign self.InteractiveTB to be an instance of your modified 
one (you may want to also modify the convenience %xmode magic to recognize 
your new modes).  This should give you full control over exception reporting. 
    Once you are satisfied with a clean design, we can look into whether some 
of this flexibility can be put into a decent API, but for now I think it's 
easier if you just subclass and play with whatever you want.

Alternatively, with the custom exception handler mechanism (see above), you 
can just catch specifically the exceptions you'd like to treat differently and 
deal with them directly.  Which of the two approaches you take will depend 
exactly on what you want to do.

>    7.
> 
>       Access to the command-line interpreter, to have *IPython* interprets code
>       has if it was entered through keyboard (ie make difference between magic
>       commands and normal python code).

Why can't you just use the push() method?  That's pretty much what it does, 
see its docstring.  If this is not enough, you'll need to clarify to me 
exactly why not, and what else you'd need.


OK, that's quite a lot.  All of this is in CVS, please test it extensively and 
let me know how it all goes.

Regards,

f



From mantegazza at ill.fr  Mon Mar 21 03:36:04 2005
From: mantegazza at ill.fr (=?iso-8859-15?q?Fr=E9d=E9ric_Mantegazza?=)
Date: Mon, 21 Mar 2005 09:36:04 +0100
Subject: [IPython-dev] PyMAD/IPython which list updated
In-Reply-To: <423D5C71.9090903@colorado.edu>
References: <200503181321.31361.mantegazza@ill.fr>
	<423D5C71.9090903@colorado.edu>
Message-ID: <200503210936.04583.mantegazza@ill.fr>

Le Dimanche 20 Mars 2005 12:20, Fernando Perez a ?crit :

> Thanks for this update.  Below follow a few more comments, it would be
> great if you could keep the doc updated as we hash out the issues.  I'm
> thinking a wiki would be great for this, but for now we don't have one. 
> So while I code, you can be the Fr?d?Wiki ;-)  I actually think most
> things will be cleared up in short order, so in reality there's no need
> to do much.

Ok, I'll update the doc each time I implment and make a solution run. But 
the wiki is a very good idea :o)

> [...]
> OK, that's quite a lot.  All of this is in CVS, please test it
> extensively and let me know how it all goes.

Fernando, this is great ! Thank you very much for all this big work. I will 
spend the next days to implement all this features, and also have a closer 
look at the documentation, because it seems that I missed some nice already 
implemented features (push(), %sc, %sx, macros...).

And I have some more ideas/needs ;o)

-- 
   Fr?d?ric



From Fernando.Perez at colorado.edu  Mon Mar 21 18:30:04 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Mon, 21 Mar 2005 16:30:04 -0700
Subject: [IPython-dev] A plea for help with a threading bug...
In-Reply-To: <mailman.2806.1111327775.905.ipython-dev@scipy.net>
References: <mailman.2806.1111327775.905.ipython-dev@scipy.net>
Message-ID: <423F58FC.1080006@colorado.edu>

ipython-dev-bounces at scipy.net wrote:
> The attached message has been automatically discarded.

[ Originally sent by Yariv, reforwarded by me.  The non-quoted stuff are my 
questions.

I've whitelisted one address for you, but you might want to just subscribe if 
you use many.

f]

> ------------------------------------------------------------------------
> 
> Subject: Re: [Fwd: Re: [IPython-dev] A plea for help with a threading bug...]
> From: Yariv Ido <yariv at vipe.technion.ac.il>
> Date:  Sun, 20 Mar 2005 16:04:02 +0200
> To: Fernando Perez <Fernando.Perez at colorado.edu>
> CC: Yariv Ido <yariv at vipe.stud.technion.ac.il>, IPython-dev List 
> <ipython-dev at scipy.net>
> 
> 
> Hi,
> 
> Well, first of all, the self.gtk.threads_init() call should be made on
> win32 systems too. As far as I know, this isn't a platform specific
> hack...
> Second, in order to avoid lockups ( Which occur if you do call
> threads_init() on win32 ), you need to wrap the call to gtk's mainloop
> with threads_enter() and threads_leave() calls:
> 
> 					self.gtk.threads_enter()
>         self.gtk_mainloop()	->	self.gtk_mainloop()
> 					self.gtk.threads_leave()
> 
> In theory, every call to some gtk functionality made outside the main
> thread should also be wrapped with this lock mechanism. Unfortunately, I
> think it's not trivial in the case of IPython, especially when the user
> decides to create new threads which mess around with GTK's objects... I
> don't know what are the odds of that happening...

Well, I can at least wrap the calls _I_ make in the above.  I just tried it, 
and I see no ill effects.  As for protecting the user's calls, I don't know... 
  Unless someone tells me otherwise (I know so little about threads and GTK, 
that I rely on your collective wisdom here), this change will go into CVS.

> 
> There's a short explanation about all this at:
> <http://www.moeraki.com/pygtkreference/pygtk2reference/gdk-functions.html#function-gdk--threads-enter>
> 
> Y.
> 
> On Sat, 2005-03-19 at 12:53 -0700, Fernando Perez wrote:
> 
>>[Yariv]
>>
>>
>>>3. I've noticed that IPython's code has no
>>>threads_enter()/threads_leave(). I don't know if it would solve this
>>>particular problem, but it may fix the threads_init() call in win32 (And
>>>thus fix the tremendous slowdown as reported on the bug tracker).
>>
>>Could you be more specific on where this should go?  And is there anyway 
>>anyone could test this on win32 and let me know if it helps any?  I'll be 
>>happy to add the changes, but only if I know they will help, and where to put 
>>them.
>>
>>best,
>>
>>f



From mantegazza at ill.fr  Wed Mar 23 07:57:10 2005
From: mantegazza at ill.fr (=?iso-8859-15?q?Fr=E9d=E9ric_Mantegazza?=)
Date: Wed, 23 Mar 2005 13:57:10 +0100
Subject: [IPython-dev] PyMAD/IPython which list updated
In-Reply-To: <423D5C71.9090903@colorado.edu>
References: <200503181321.31361.mantegazza@ill.fr>
	<423D5C71.9090903@colorado.edu>
Message-ID: <200503231357.10760.mantegazza@ill.fr>

Le Dimanche 20 Mars 2005 12:20, Fernando Perez a ?crit :

> Thanks for this update.  Below follow a few more comments, it would be
> great if you could keep the doc updated as we hash out the issues.  I'm
> thinking a wiki would be great for this, but for now we don't have one. 
> So while I code, you can be the Fr?d?Wiki ;-)  I actually think most
> things will be cleared up in short order, so in reality there's no need
> to do much.

I've tried and implemented some of the new features. And I updated the 
Fr?d?Wiki ;o)

> 1.
> Following this, IPython now has a facility for custom exception handling.
> IPython can take a list of custom exceptions and a handler, and call that
> handler if any exception in the list matches.  Note that this can only be
> done for a single handler, because I can't (short of doing dynamic code
> generation, which I don't want to get into)  have multiple except
> clauses.  So your one handler would get the exception info for _any_ of
> the matched exceptions, and it would have to deal with them.

Ok, works fine :o)

> 2.
> Done.  A new rc parameter, readline_merge_completions, controls this
> (true by default).
>
> New completers can now be added by using the
> __IPYTHON__.set_custom_completer method.  Plase see its docstring for
> details.

Working too :o)

> 3.
> OK, this is easy.  Just make sure that the objects IPython sees have a
> .getdoc() method; I've added a call to that into my object introspection
> code. Have a look at inspect.getdoc, you can probably have your .getdoc
> method be like that, since it does a good job of cleaning up indentation
> for the docstring.  Your method should conform to the same API as
> inspect.getdoc: return None if nothing is found, return a string
> (formatted to your liking) otherwise.
>
> In fact, the change is already in CVS, so feel free to play with it.  See
> OInspect.py's getdoc() function for details.

It seems to work, but I need some more explanation of the usage, and how 
things work.

I'm using the inspect.getdoc() code in my Spectro object. But I would like 
to have something more accurate. To sumarize:

obj? should give me the docstring of the class definition of obj.
obj.method? should give me the docstring of obj.method().

Is it possible ?

> 4.
> Alternatively, with the custom exception handler mechanism (see above),
> you can just catch specifically the exceptions you'd like to treat
> differently and deal with them directly.  Which of the two approaches you
> take will depend exactly on what you want to do.

Yes, this is enough for our needs for the moment.

> 5.
Last, I tried to use pre-filters, to avoid user deleting some of our 
objects, but I didn't find documentation about them. Where is it ?

> 6.
Prompt is also working fine, but I get some strange behaviour when using 
multi-lines prompt. Yes, I think that you don't support multi-line prompt, 
but it works, except:

- Going up in the history change the color of the prompt (turn back to 
  white).
- The indentation for outputs should be computed with the lenght of the last 
  line of the prompt, and not with all the prompt.

This is just a remark, but it may not be difficult to solve these problems.

> 7.
> Why can't you just use the push() method?  That's pretty much what it
> does, see its docstring.  If this is not enough, you'll need to clarify
> to me exactly why not, and what else you'd need.

See my question on the user ML. I have problems using it.

> 8.
In fact, my first idea before making a dynamic prompt was to have a status 
bar (dynamically refreshed) at the bottom of IPython, as it is done in 
minicom, for example. Is it difficult to implement ?

Thank's,

-- 
   Fr?d?ric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20050323/4e91b4e9/attachment.html>
-------------- next part --------------
======================================================================
This document describes the interactions needed by PyMAD with IPython.
======================================================================

:Authors: 
    Fr?d?ric Mantegazza, 

:Version: $Rev: 1324 $

1) Introduction
---------------

The philosophy of **PyMAD** is to give to the user a high-level set of tools to
drive a spectrometer. The idea is to directly make some internal python
objects available to the user, and let him combine them.

So, The final user just calls some methods of a few high-level objects to drive
the spectrometer. This as the advantage to make all the python scripting stuff
available, to build some new high level tools.

As we also need a system which can be used from several places, we use a
client/server framework, with **Pyro**. **Pyro** is a distributed objects
server. It just make some remote python objects available to a client as if they
where local.

2) User interaction
-------------------

To avoid the need for the final user to write python scripts and run them to do
something, we need a simple working environment which gives the possibility to
interactively use the server objects. That's where **IPython** solves lots of
problems !

**IPython** is an enhanced python shell. It let the user runs python code, but
has many powerfull features :

- history, even between sessions
- colorized and customizable traceback
- code completion
- magic commands
- much more...
 
**IPython** is on the client-side of **PyMAD**. In fact, there is a special
client which connects to the remote objects of **PyMAD**, launch an embbeded
**IPython** instance, and make the remote objects available to the user, in the
global namespace of **IPython**. This way, the user can drive the spectrometer
through the methods of these objects, or can build complex scripts to do complex
actions.

**PyMAD** also use the magic commands to make a simple command interpreter.
The magic functions use TPG (Toy Parser Generator), a easy-to-use parser based
on regexp. These MAD-like commands are for users which don't know about python,
but also to make shortcuts, to avoid the need to write several lines of normal
python code to do some complex but repetitive tasks.

One important point is that **PyMAD** can understand both syntax, which can be
combined. Most of the time, simple commands will be used, but python code can
be more powerfull to do expert measures (with automatic feedback interaction
according to the results), or to prototype a new complex command.

3) **IPython** interactions
---------------------------

In all the following, ipshell is a IPShellEmbed instance.

Here is the main **PyMAD/IPython** interactions:

1. Catch custom *PyMADError* exceptions. The solution (from CVS version), is to
   use the new set_custom_sxc() method of __IP object (ie ishell.IP object)::

    def pymadHandler(self, exceptionType, exceptionValue, traceback):
        """ This is a special handler to handle custom exceptions in IPython.
        """
        
        # Handle PyMAD exceptions
        if issubclass(exceptionType, PyMADError):
            MessageLogger().error(exceptionValue.standardStr())
            
        #?Handle Pyro exceptions
        elif issubclass(exceptionType, Pyro.errors.ConnectionClosedError):
            MessageLogger().critical("Connexion with server lost. Please restart it...")
            if self.code_to_run_src.find('.') != -1:
                code = self.code_to_run_src.split(".")
                obj = eval(code[0], self.user_ns)
                obj.adapter.rebindURI()
                MessageLogger().warning("Server found again. Running last command...")
                eval(self.code_to_run_src, self.user_ns)
            else:
                MessageLogger().warning("Could not automatically rebind to server. Better restart console")
    
        #?Handle AttributeError exception
        elif exceptionType is AttributeError:
            if self.code_to_run_src.find('.') != -1:
                code = self.code_to_run_src.split(".")
                obj = eval(code[0], self.user_ns)
                if isinstance(obj, Pyro.core.DynamicProxyWithAttrs):
                    MessageLogger().error("%s has no attribute %s" % (code[0], code[1]))
                else:
                    self.showtraceback()
            else:
                self.showtraceback()
    
        # Others
        else:
            self.showtraceback()
            print "\n\n"
            print "*** Unknown exception ***"
            print "Exception type :", exceptionType
            print "Exception value:", exceptionValue
            print "Traceback      :", traceback
            print "Source code    :", self.code_to_run_src

    ipshell.IP.set_custom_exc((PyMADError, Pyro.errors.ConnectionClosedError, AttributeError), pymadHandler)

2. Add some new matchers for completion. As **PyMAD** uses remote objects,
   completion only shows the client Pyro proxy. So we create a new matcher which
   get the object (from the text param), call a special method on this object
   which returns all available attributes (in fact, only these we want to show
   to the user). Here is the code used::

    def proxy_matches(self, text, state):
        """ Get the attributes of a remove Pyro object.
        """
        message = MessageLogger()
    
        # Another option, seems to work great. Catches things like ''.<tab>
        m = re.match(r"(\S+(\.\w+)*)\.(\w*)$", text)
    
        matches = []
        
        if m:
            expr, attr = m.group(1, 3)
            try:
                obj = eval(expr, self.namespace)
                if isinstance(obj, Pyro.core.DynamicProxyWithAttrs):
                    words = obj.getAvailableAttributes()
                    n = len(attr)
                    if words:
                        for word in words:
                            if word[:n] == attr and word != "__builtins__":
                                matches.append("%s.%s" % (expr, word))
        
            except NameError:
                pass
                
            except Pyro.errors.ConnectionClosedError:
                message.critical("Connexion with server lost.")
            
        return matches

    ipshell.IP.set_custom_completer(proxy_matches)
    ipshell.IP.Completer.merge_completions = False

3. In the same way as matchers, get the docstring from the remote object instead
   of the client one when using 'obj?' and 'obj.method?' syntax. This can be
   done by adding a getdoc() method to the object. But it only works for the
   first syntax. And if object provides __call__ method, it adds some more infos
   not wanted.

4. New exception handler. Here, the idea is to be able to present different kind
   of exceptions in different ways. Some will only print a simple message, some
   others will print the entire traceback (maybe a modified traceback).
   This is done in the point 1.

5. Prevent objects from being deleted by *del* keyword. This can be done with
   pre-filters, but I didn't find how such filters are defined.

6. Dynamic Prompt. Works fine using the PEP 215 syntax. When using multi-line
   prompt, color changed when going up in the history. Also, the indentation for
   output should be computed from the lenght of the last prompt line, and not
   the whole prompt. Here is the prompt used::

    argv = ["-pi1",
            "Ki=${globals()['Spectro'].Ki} Kf=${globals()['Spectro'].Kf}\n\
            mode=${globals()['Spectro'].driveMode} flipper=${globals()['Spectro'].flipperBeam}\n\
            PyMAD@${os.path.expandvars('$PYRO_NS_HOSTNAME')}>>> ",
            "-po",
            " ",
            "-profile",
            "pymad"]
 
7. Access to the command-line interpreter, to have **IPython** interprets code
   has if it was entered through keyboard (ie make difference between magic
   commands and normal python code). Can be done with the ipshell.IP.push()
   method, but it does not work. This will be used in the magic 'do' method.
   At the moment, it can only interprets magic commands (using magic() method).

8. Status bar at the bottom line. Like prompt, should be dynamically refreshde.

Open points are 3 (for obj.method? syntax), 5, 6 (for multi-line prompt), 7, 8.

From Fernando.Perez at colorado.edu  Wed Mar 23 10:16:48 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Wed, 23 Mar 2005 08:16:48 -0700
Subject: [IPython-dev] PyMAD/IPython which list updated
In-Reply-To: <200503231357.10760.mantegazza@ill.fr>
References: <200503181321.31361.mantegazza@ill.fr>
	<423D5C71.9090903@colorado.edu> <200503231357.10760.mantegazza@ill.fr>
Message-ID: <42418860.7000209@colorado.edu>

Fr?d?ric Mantegazza wrote:

>>OK, this is easy.  Just make sure that the objects IPython sees have a
>>.getdoc() method; I've added a call to that into my object introspection
>>code. Have a look at inspect.getdoc, you can probably have your .getdoc
>>method be like that, since it does a good job of cleaning up indentation
>>for the docstring.  Your method should conform to the same API as
>>inspect.getdoc: return None if nothing is found, return a string
>>(formatted to your liking) otherwise.
>>
>>In fact, the change is already in CVS, so feel free to play with it.  See
>>OInspect.py's getdoc() function for details.
> 
> 
> It seems to work, but I need some more explanation of the usage, and how 
> things work.
> 
> I'm using the inspect.getdoc() code in my Spectro object. But I would like 
> to have something more accurate. To sumarize:
> 
> obj? should give me the docstring of the class definition of obj.
> obj.method? should give me the docstring of obj.method().
> 
> Is it possible ?

Just have a look at OInspect.getdoc, it shows exactly what is done.  IPython 
will call getdoc() on anything you pass to it, so as long as you provide 
getdoc() in the proper places, you can do anything you want.  This is just a 
protocol for getting information, it's up to you to provide whatever 
information you want over that protocol.

>>5.
> 
> Last, I tried to use pre-filters, to avoid user deleting some of our 
> objects, but I didn't find documentation about them. Where is it ?

http://ipython.scipy.org/doc/manual/node11.html

>>6.
> 
> Prompt is also working fine, but I get some strange behaviour when using 
> multi-lines prompt. Yes, I think that you don't support multi-line prompt, 
> but it works, except:
> 
> - Going up in the history change the color of the prompt (turn back to 
>   white).
> - The indentation for outputs should be computed with the lenght of the last 
>   line of the prompt, and not with all the prompt.
> 
> This is just a remark, but it may not be difficult to solve these problems.

I'm sorry, but I don't see any problems.  With the following:

prompt_in1 '${genutils.getoutput("date")} - In [\#]: '

in my rc file, I get this:

Wed Mar 23 08:14:16 MST 2005 - In [1]: for i in (1,2):
                                   ...:      print i,
                                   ...:
1 2

As far as I can see, that's all correct in terms of indentation, and using 
up-arrow doesn't change any colors.  If you have any problems here, please, 
instead of trying to describe it, paste the _exact_ output that you get, along 
with indicating what you think it should be.  There may well be a problem, but 
I simply don't see any in the cases I've tested (and I cant test your actual 
prompt, which depends on all sorts of things which are local to your environment).

>>8.
> 
> In fact, my first idea before making a dynamic prompt was to have a status 
> bar (dynamically refreshed) at the bottom of IPython, as it is done in 
> minicom, for example. Is it difficult to implement ?

Plain impossible.  IPython is not a curses program, and those things can only 
be done via curses.  A line-oriented program like ipython simply can't do that 
kind of thing, since it has no access to the screen as a 2-d space, only to 
the current line.  You could write a curses wrapper which runs ipython inside 
of it, and does what you want.  But that's clearly beyond what ipython can do, 
because curses is not portable to win32 platforms (and is in general a PITA to 
use).

Best,

f



From Fernando.Perez at colorado.edu  Wed Mar 23 10:29:02 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Wed, 23 Mar 2005 08:29:02 -0700
Subject: [IPython-dev] PyMAD/IPython which list updated
In-Reply-To: <42418860.7000209@colorado.edu>
References: <200503181321.31361.mantegazza@ill.fr>
	<423D5C71.9090903@colorado.edu> <200503231357.10760.mantegazza@ill.fr>
	<42418860.7000209@colorado.edu>
Message-ID: <42418B3E.60108@colorado.edu>

Fernando Perez wrote:
> I'm sorry, but I don't see any problems.  With the following:
> 
> prompt_in1 '${genutils.getoutput("date")} - In [\#]: '
> 
> in my rc file, I get this:
> 
> Wed Mar 23 08:14:16 MST 2005 - In [1]: for i in (1,2):
>                                    ...:      print i,

Just to clarify, the above looks wrong because of how my mail client pasted 
spaces.  On my screen, the two ':' prompt characters in both lines are 
correctly aligned.

Best,

f



From mantegazza at ill.fr  Wed Mar 23 11:53:11 2005
From: mantegazza at ill.fr (=?iso-8859-15?q?Fr=E9d=E9ric_Mantegazza?=)
Date: Wed, 23 Mar 2005 17:53:11 +0100
Subject: [IPython-dev] PyMAD/IPython which list updated
In-Reply-To: <42418860.7000209@colorado.edu>
References: <200503181321.31361.mantegazza@ill.fr>
	<200503231357.10760.mantegazza@ill.fr> <42418860.7000209@colorado.edu>
Message-ID: <200503231753.11773.mantegazza@ill.fr>

Le Mercredi 23 Mars 2005 16:16, Fernando Perez a ?crit :

> 3.
> Just have a look at OInspect.getdoc, it shows exactly what is done. 
> IPython will call getdoc() on anything you pass to it, so as long as you
> provide getdoc() in the proper places, you can do anything you want. 
> This is just a protocol for getting information, it's up to you to
> provide whatever information you want over that protocol.

Ok, so there is no way to have obj.method? syntax works the way I want, 
since IPython will call obj.method.getdoc(), which has no sens...

In the case a __call__ method is defined, using obj? syntax will print 
additionnal informations:

>>> Spectro?
Type:           instance
Base Class:     Pyro.core.DynamicProxyWithAttrs
String Form:    <DynamicProxyWithAttrs for
                PYRO://sunceng:7772/c239dc101add619f4fb8359d7176c6d6>
Namespace:      Interactive
Docstring:
    Proxy for Spectrometer.

    This is a test of docstring.
Callable:       Yes
Call def:       Spectro(self, *vargs, **kargs)

Could it be possible to print the Docstring informations after all other 
ones ?

> 5.
> http://ipython.scipy.org/doc/manual/node11.html

Thank's !

> 6.
> As far as I can see, that's all correct in terms of indentation, and
> using up-arrow doesn't change any colors.  If you have any problems here,
> please, instead of trying to describe it, paste the _exact_ output that
> you get, along with indicating what you think it should be.  There may
> well be a problem, but I simply don't see any in the cases I've tested
> (and I cant test your actual prompt, which depends on all sorts of things
> which are local to your environment).

The 'problem' occurs only when you have a multi-lineprompt, ie when using 
one or more '\n' char:

prompt_in1 '${genutils.getoutput("date")} \n In [\#]: '

Then, the output is:

mer mar 23 16:46:29 CET 2005
 In [1]: for i in xrange(3):
                                   ...:     print i
                                   ...:
0
1
2

For the color problem (sorry, can't repoduce it here !), it only occures on 
the second line (in fact, the last one) when going up in the history. In 
the above example, only 'In [' turns to white; not the remaining of the 
line.

Well, put this at the end of your todo list, as it is not very important ! 
I've just made this test to simulate a status bar...

> 8.
> Plain impossible.  IPython is not a curses program, and those things can
> only be done via curses.  A line-oriented program like ipython simply
> can't do that kind of thing, since it has no access to the screen as a
> 2-d space, only to the current line.  You could write a curses wrapper
> which runs ipython inside of it, and does what you want.  But that's
> clearly beyond what ipython can do, because curses is not portable to
> win32 platforms (and is in general a PITA to use).

Ok, this was a bad idea !

-- 
   Fr?d?ric



From Fernando.Perez at colorado.edu  Wed Mar 23 12:16:13 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Wed, 23 Mar 2005 10:16:13 -0700
Subject: [IPython-dev] PyMAD/IPython which list updated
In-Reply-To: <200503231753.11773.mantegazza@ill.fr>
References: <200503181321.31361.mantegazza@ill.fr>
	<200503231357.10760.mantegazza@ill.fr> <42418860.7000209@colorado.edu>
	<200503231753.11773.mantegazza@ill.fr>
Message-ID: <4241A45D.4030605@colorado.edu>

Fr?d?ric Mantegazza wrote:

> Ok, so there is no way to have obj.method? syntax works the way I want, 
> since IPython will call obj.method.getdoc(), which has no sens...
> 
> In the case a __call__ method is defined, using obj? syntax will print 
> additionnal informations:
> 
> 
>>>>Spectro?
> 
> Type:           instance
> Base Class:     Pyro.core.DynamicProxyWithAttrs
> String Form:    <DynamicProxyWithAttrs for
>                 PYRO://sunceng:7772/c239dc101add619f4fb8359d7176c6d6>
> Namespace:      Interactive
> Docstring:
>     Proxy for Spectrometer.
> 
>     This is a test of docstring.
> Callable:       Yes
> Call def:       Spectro(self, *vargs, **kargs)
> 
> Could it be possible to print the Docstring informations after all other 
> ones ?

I don't understand what you mean here, please be as specific as possible.  Do 
you just want the order to be changed, so docstrings go last?

You could just override getdoc in OInspect, but I have a feeling that some of 
what you want to do is simply impossible.  You'll need to be very clear and 
specific in describing (make up a mock example if you want) _exactly_ what you 
need to do.  Only in that case will I be able to help you further.


>>6.
>>As far as I can see, that's all correct in terms of indentation, and
>>using up-arrow doesn't change any colors.  If you have any problems here,
>>please, instead of trying to describe it, paste the _exact_ output that
>>you get, along with indicating what you think it should be.  There may
>>well be a problem, but I simply don't see any in the cases I've tested
>>(and I cant test your actual prompt, which depends on all sorts of things
>>which are local to your environment).
> 
> 
> The 'problem' occurs only when you have a multi-lineprompt, ie when using 
> one or more '\n' char:
> 
> prompt_in1 '${genutils.getoutput("date")} \n In [\#]: '
> 
> Then, the output is:
> 
> mer mar 23 16:46:29 CET 2005
>  In [1]: for i in xrange(3):
>                                    ...:     print i
>                                    ...:
> 0
> 1
> 2
> 
> For the color problem (sorry, can't repoduce it here !), it only occures on 
> the second line (in fact, the last one) when going up in the history. In 
> the above example, only 'In [' turns to white; not the remaining of the 
> line.
> 
> Well, put this at the end of your todo list, as it is not very important ! 
> I've just made this test to simulate a status bar...

With a proper explanation, the fix was actually trivial.  Checked into CVS, 
let me know if it does what you need.

Best,

f



From mantegazza at ill.fr  Thu Mar 24 03:02:45 2005
From: mantegazza at ill.fr (=?iso-8859-15?q?Fr=E9d=E9ric_Mantegazza?=)
Date: Thu, 24 Mar 2005 09:02:45 +0100
Subject: [IPython-dev] PyMAD/IPython which list updated
In-Reply-To: <4241A45D.4030605@colorado.edu>
References: <200503181321.31361.mantegazza@ill.fr>
	<200503231753.11773.mantegazza@ill.fr> <4241A45D.4030605@colorado.edu>
Message-ID: <200503240902.46187.mantegazza@ill.fr>

Le Mercredi 23 Mars 2005 18:16, Fernando Perez a ?crit :

> > Could it be possible to print the Docstring informations after all
> > other ones ?
>
> I don't understand what you mean here, please be as specific as possible.
>  Do you just want the order to be changed, so docstrings go last?

Yes.

> You could just override getdoc in OInspect, but I have a feeling that
> some of what you want to do is simply impossible.  You'll need to be very
> clear and specific in describing (make up a mock example if you want)
> _exactly_ what you need to do.  Only in that case will I be able to help
> you further.

The problem is not simple, as I'm working on remote objects through proxies 
(if we met at Grenoble, I'll explain you exactly how it works, but it is 
very difficult for me to describe all the stuff in english :o/). I have an 
idea how to do the job. It is not easy for me to explain, so I will post 
the code as soon as I can make it work. But it is very specific to my 
problem, and don't have to be implemented in IPython. Maybe just have some 
minor changes to make things easier to do... I'll let you know.

> With a proper explanation, the fix was actually trivial.  Checked into
> CVS, let me know if it does what you need.

Great ! Again, sorry for my poor english, leading to these lack of 
understanding...

-- 
   Fr?d?ric



From Fernando.Perez at colorado.edu  Fri Mar 25 06:58:12 2005
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Fri, 25 Mar 2005 04:58:12 -0700
Subject: [IPython-dev] PyMAD/IPython which list updated
In-Reply-To: <200503240902.46187.mantegazza@ill.fr>
References: <200503181321.31361.mantegazza@ill.fr>
	<200503231753.11773.mantegazza@ill.fr> <4241A45D.4030605@colorado.edu>
	<200503240902.46187.mantegazza@ill.fr>
Message-ID: <4243FCD4.6080700@colorado.edu>

Fr?d?ric Mantegazza wrote:
> Le Mercredi 23 Mars 2005 18:16, Fernando Perez a ?crit :
> 
> 
>>>Could it be possible to print the Docstring informations after all
>>>other ones ?
>>
>>I don't understand what you mean here, please be as specific as possible.
>> Do you just want the order to be changed, so docstrings go last?
> 
> 
> Yes.

OK, then the answer is no :)  I'm sorry, but I don't want to lump all 
docstrings at the end, because they serve different purposes.  IPython tries 
to give obj.__doc__ by default, but if it is a class/callable, it tries to add 
some extra information which may be useful.  But the constructor docstring 
only makes sense next to the rest of the constructor information.  It would be 
bad (usability-wise) to lump all docstrings into one ball at the end, 
disconnected from the structures they represent.

>>You could just override getdoc in OInspect, but I have a feeling that
>>some of what you want to do is simply impossible.  You'll need to be very
>>clear and specific in describing (make up a mock example if you want)
>>_exactly_ what you need to do.  Only in that case will I be able to help
>>you further.
> 
> 
> The problem is not simple, as I'm working on remote objects through proxies 
> (if we met at Grenoble, I'll explain you exactly how it works, but it is 
> very difficult for me to describe all the stuff in english :o/). I have an 
> idea how to do the job. It is not easy for me to explain, so I will post 
> the code as soon as I can make it work. But it is very specific to my 
> problem, and don't have to be implemented in IPython. Maybe just have some 
> minor changes to make things easier to do... I'll let you know.

OK, let me know once you have something implemented, and it may be possible to 
make it easier for you on the ipython side.  But ultimately it seems you are 
trying to do something odd enough, that some hacking on your side is going to 
be unavoidable.

>>With a proper explanation, the fix was actually trivial.  Checked into
>>CVS, let me know if it does what you need.
> 
> 
> Great ! Again, sorry for my poor english, leading to these lack of 
> understanding...

No worries.

best,

f



From mantegazza at ill.fr  Fri Mar 25 07:55:29 2005
From: mantegazza at ill.fr (=?iso-8859-15?q?Fr=E9d=E9ric_Mantegazza?=)
Date: Fri, 25 Mar 2005 13:55:29 +0100
Subject: [IPython-dev] PyMAD/IPython which list updated
In-Reply-To: <4243FCD4.6080700@colorado.edu>
References: <200503181321.31361.mantegazza@ill.fr>
	<200503240902.46187.mantegazza@ill.fr> <4243FCD4.6080700@colorado.edu>
Message-ID: <200503251355.29760.mantegazza@ill.fr>

Le Vendredi 25 Mars 2005 12:58, Fernando Perez a ?crit :

> OK, then the answer is no :)  I'm sorry, but I don't want to lump all
> docstrings at the end, because they serve different purposes.  IPython
> tries to give obj.__doc__ by default, but if it is a class/callable, it
> tries to add some extra information which may be useful.  But the
> constructor docstring only makes sense next to the rest of the
> constructor information.  It would be bad (usability-wise) to lump all
> docstrings into one ball at the end, disconnected from the structures
> they represent.

Ok, I understand.

> OK, let me know once you have something implemented, and it may be
> possible to make it easier for you on the ipython side.  But ultimately
> it seems you are trying to do something odd enough, that some hacking on
> your side is going to be unavoidable.

;o)

-- 
   Fr?d?ric