From kriehl at enthought.com  Fri Feb  2 15:20:40 2007
From: kriehl at enthought.com (Katrina Riehl)
Date: Fri, 02 Feb 2007 14:20:40 -0600
Subject: [IPython-dev] IPython1 installation
Message-ID: <1170447640.3220.26.camel@katrina.enthought>

Hello,

I wondered if anyone on the list could help me.  I recently checked out
ipython1 from svn.  When I try to use it from python, I get this
traceback:

Python 2.4.4 (#1, Oct 23 2006, 13:58:00) 
[GCC 4.1.1 20061011 (Red Hat 4.1.1-30)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ipython1
>>> import ipython1.kernel.api
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/usr/lib/python2.4/site-packages/ipython1/kernel/api.py", line
63, in ?
    import ipython1.config.api as config
  File "/usr/lib/python2.4/site-packages/ipython1/config/api.py", line
63, in ?
    from ipython1.config.objects import configClasses
  File "/usr/lib/python2.4/site-packages/ipython1/config/objects.py",
line 57, in ?
    from ipython1.kernel.enginevanilla import \
  File
"/usr/lib/python2.4/site-packages/ipython1/kernel/enginevanilla.py",
line 47, in ?
    from ipython1.kernel import error, protocols
  File "/usr/lib/python2.4/site-packages/ipython1/kernel/protocols.py",
line 35, in ?
    class NetstringProducer(object):
  File "/usr/lib/python2.4/site-packages/zope/interface/advice.py", line
132, in advise
    return callback(newClass)
  File
"/usr/lib/python2.4/site-packages/zope/interface/declarations.py", line
550, in _implements_advice
    classImplements(cls, *interfaces)
  File
"/usr/lib/python2.4/site-packages/zope/interface/declarations.py", line
527, in classImplements
    spec.declared += tuple(_normalizeargs(interfaces))
  File
"/usr/lib/python2.4/site-packages/zope/interface/declarations.py", line
1345, in _normalizeargs
    _normalizeargs(v, output)
  File
"/usr/lib/python2.4/site-packages/zope/interface/declarations.py", line
1344, in _normalizeargs
    for v in sequence:
TypeError: Error when calling the metaclass bases
    iteration over non-sequence
>>> 

Does this mean that there is a problem with my zope interface?  Or did I
not set ipython1 up correctly?


I really appreciate your help.
Thanks,
Katrina




From bryanv at enthought.com  Mon Feb 12 10:27:03 2007
From: bryanv at enthought.com (Bryan Van de Ven)
Date: Mon, 12 Feb 2007 09:27:03 -0600
Subject: [IPython-dev] missing import
Message-ID: <45D08747.8000906@enthought.com>

causes test failures in "trial ipython1"

--
Bryan Van de Ven
-------------- next part --------------
A non-text attachment was scrubbed...
Name: patch.diff
Type: text/x-patch
Size: 332 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20070212/72a7317b/attachment.bin>

From ellisonbg.net at gmail.com  Tue Feb 13 13:52:54 2007
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 13 Feb 2007 11:52:54 -0700
Subject: [IPython-dev] Let's clear out/update all the old Trac tickets
Message-ID: <6ce0ac130702131052s486eac90m1a6e2c3713aec8c@mail.gmail.com>

Hi all,

I was just updating the tickets for the ipython1 saw branch.  There
are a number of older tickets on the IPython Trac that seem "dead."
Could everyone clean out old tickets on the Trac.  I have done this on
tickets related to the saw branch.  Fernando, many of the oldest
tickets are related to the nbdoc/nbshell.  I am not sure if you want
to keep those around.  My feeling is that when we get around to
working on the notebook, most of those tickets won't apply.

Cheers,

Brian


From fperez.net at gmail.com  Wed Feb 14 02:04:13 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 14 Feb 2007 00:04:13 -0700
Subject: [IPython-dev] Let's clear out/update all the old Trac tickets
In-Reply-To: <6ce0ac130702131052s486eac90m1a6e2c3713aec8c@mail.gmail.com>
References: <6ce0ac130702131052s486eac90m1a6e2c3713aec8c@mail.gmail.com>
Message-ID: <db6b5ecc0702132304l76adbc57ub11caa3cd76333f0@mail.gmail.com>

On 2/13/07, Brian Granger <ellisonbg.net at gmail.com> wrote:
> Hi all,
>
> I was just updating the tickets for the ipython1 saw branch.  There
> are a number of older tickets on the IPython Trac that seem "dead."
> Could everyone clean out old tickets on the Trac.  I have done this on
> tickets related to the saw branch.  Fernando, many of the oldest
> tickets are related to the nbdoc/nbshell.  I am not sure if you want
> to keep those around.  My feeling is that when we get around to
> working on the notebook, most of those tickets won't apply.

I don't want to remove them altogether, because there's a certain
amount of information in there that /may/ prove useful in the future.
But given that right now they are not likely to be relevant, I bumped
the priority for all of them way down, so that they show up at the
bottom of the reports:

http://projects.scipy.org/ipython/ipython/report/1

As we identify ones (I didn't actually read them carefully, I just
went over all of them) which we're /certain/ are never going to be
relevant, it's certainly OK to just close them.  But at least now
they're reasonably out of the way.

I didn't find a 'postpone' option in Trac, so I used priority lowering
as the next-best solution.  If you prefer an alternative, let me know.

Cheers,

f


From fperez.net at gmail.com  Sun Feb 18 02:10:34 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Sun, 18 Feb 2007 00:10:34 -0700
Subject: [IPython-dev] pydoc and introspective features
In-Reply-To: <mailman.14355.1171777712.8339.ipython-dev@scipy.org>
References: <mailman.14355.1171777712.8339.ipython-dev@scipy.org>
Message-ID: <45D7FBEA.4080702@gmail.com>

Hi Laurent,

I'm forwarding your message to the list because it was originally 
auto-discarded.  The ipython lists don't allow non-subscriber posts due to the 
inordinate amout of spam we were getting.  I manually whitelisted you address 
so you can post.

I'll post my actual reply separately, this is just your original message for 
the list.

Cheers,

f

> Subject:  pydoc and introspective features
> From: "Laurent Gautier" <lgautier at gmail.com>
> Date: Sun, 18 Feb 2007 13:48:28 +0800
> To: ipython-dev at scipy.org
> 
> 
> Hi,
> 
> We are two people in the process of rewriting pydoc for the standard python
> (with python2.6 as a target), splitting the whole into functional units (modules
> in a package) and creating the units with extensions outside pydoc in mind.
> In all honesty, we have no warranty that it will get included but we
> are committed
> to release a python module that could replace the existing pydoc anyway.
> 
> I have been (and remain) an happy user of ipython, and I was thinking that
> that some of the introspective documention-centered capabilities that we are
> developing could be of interest to the ipython project.
> 
> I see that ipython is being rewritten, and that might well be the right moment
> for me to bring that up and call for comments.
> 
> Anyone to comment ?
> 
> Cheers,
> 
> Laurent


From fperez.net at gmail.com  Sun Feb 18 16:29:08 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Sun, 18 Feb 2007 14:29:08 -0700
Subject: [IPython-dev] missing import
In-Reply-To: <45D08747.8000906@enthought.com>
References: <45D08747.8000906@enthought.com>
Message-ID: <db6b5ecc0702181329p1b16c2d2qedbf9e81b2e631c0@mail.gmail.com>

Hey Bryan,

On 2/12/07, Bryan Van de Ven <bryanv at enthought.com> wrote:
> causes test failures in "trial ipython1"

Thanks for the fix.  I did apply it, but note that the chainsaw branch
is mostly dead.  We're getting ready to release the saw branch as the
new development one, with enough functionality to really replace
chainsaw, and a LOT of new features, more tests, etc.   Brian Granger
is going to PyCon (I unfortunately can't make it this time, so he'll
be representing the entire IPython team :), and will give a talk about
it.  I know some from the Enthought team will be there as well, so if
you're using chainsaw, at that point you'll be able to talk to him and
hopefully move over to the nicer new codebase.

Cheers,

f


From dfj225 at gmail.com  Tue Feb 20 10:54:14 2007
From: dfj225 at gmail.com (Douglas Jones)
Date: Tue, 20 Feb 2007 10:54:14 -0500
Subject: [IPython-dev] ipython1 remote exceptions causing local crashes
Message-ID: <1315be7e0702200754i65fbc002g2e1bd71b10d7bdb1@mail.gmail.com>

Hi all,

I've been looking at the new ipython1 saw branch and the implications
of the changes to a project that I'm working on.

I've noticed in my testing that running some code on the engine that
causes an exception crashes my ipython client. I'm using recent svn
checkouts of both ipython1(revision 2104) and the ipython shell (which
lists itself as version IPython 0.7.4.svn.r2010).

Is this a known issue?

If you need more information, I'd be happy to provide some of the
traceback information.

The only thing I haven't updated is the version of Twisted installed
on the machine. Is it necessary to have a newer version of Twisted for
the ipython1 saw branch? I think Twisted that is installed is version
2.4.0.

Thanks,
~doug


From dfj225 at gmail.com  Tue Feb 20 10:54:14 2007
From: dfj225 at gmail.com (Douglas Jones)
Date: Tue, 20 Feb 2007 10:54:14 -0500
Subject: [IPython-dev] ipython1 remote exceptions causing local crashes
Message-ID: <1315be7e0702200754i65fbc002g2e1bd71b10d7bdb1@mail.gmail.com>

Hi all,

I've been looking at the new ipython1 saw branch and the implications
of the changes to a project that I'm working on.

I've noticed in my testing that running some code on the engine that
causes an exception crashes my ipython client. I'm using recent svn
checkouts of both ipython1(revision 2104) and the ipython shell (which
lists itself as version IPython 0.7.4.svn.r2010).

Is this a known issue?

If you need more information, I'd be happy to provide some of the
traceback information.

The only thing I haven't updated is the version of Twisted installed
on the machine. Is it necessary to have a newer version of Twisted for
the ipython1 saw branch? I think Twisted that is installed is version
2.4.0.

Thanks,
~doug


From fperez.net at gmail.com  Tue Feb 20 16:00:33 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 20 Feb 2007 14:00:33 -0700
Subject: [IPython-dev] ipython1 remote exceptions causing local crashes
In-Reply-To: <1315be7e0702200754i65fbc002g2e1bd71b10d7bdb1@mail.gmail.com>
References: <1315be7e0702200754i65fbc002g2e1bd71b10d7bdb1@mail.gmail.com>
Message-ID: <db6b5ecc0702201300sa9065b4of7e3ac79a2ed160c@mail.gmail.com>

Hi Douglas,

On 2/20/07, Douglas Jones <dfj225 at gmail.com> wrote:
> Hi all,
>
> I've been looking at the new ipython1 saw branch and the implications
> of the changes to a project that I'm working on.
>
> I've noticed in my testing that running some code on the engine that
> causes an exception crashes my ipython client. I'm using recent svn
> checkouts of both ipython1(revision 2104) and the ipython shell (which
> lists itself as version IPython 0.7.4.svn.r2010).
>
> Is this a known issue?

Well, it's a feature, not a bug :)  It was actually something that was
deliberately added recently, to ensure that remote exceptions aren't
silently swallowed and that you are forced to know about them.  The
reason is that otherwise swallowing remote exceptions makes for
incredibly hard debugging: your engine threw an exception which went
silently into a log file only, and the client keeps on happily feeding
it new commands to execute, most of which are likely to produce
garbage because some previous quantity you expected to have computed
is not there.

We are trying to bring, within the realm of what's possible, a smooth
transition from a model of single-process synchronous programming to
distributed computing, and exceptions are a particularly thorny issue
here.  We recently spoke to DOE people about  the next-generation
language projects for distributed computing that the DOE/DARPA are
looking into (X10, Fortress and Chapel, see
http://www.ahpcrc.org/conferences/PGAS2006/presentations/Yelick.pdf).
As far as we can tell, no system yet has a completely satisfactory
answer for this, partly because I don't think you /can/ have such an
answer.

We hope we'll offer a model that's reasonable for end users (we are
end users of this, so we're eating our own dog food here), and that
will hopefully balance:

- Feeling like an analytic continuation of the serial programming
model (though remember that you may always find surprises off the real
axis :)

- Offering some utilities that make the common synchronization tasks
needed for regular algorithmic development easy to achieve.

- Exposing enough of the distributed objects to allow more
sophisticated uses. It is still possible that we'll limit access to
parts of the underlying asynchronous machinery for reasons I don't
quite have time to explain right now.  I can go into more detail later
if needed.


But in all of this, feedback from users will be critically important.
We've put a LOT of thought into this, but it's very easy to travel
down a dead end without noticing it, so feel free to speak up.  If we
get this right, this system should be very useful for many people and
for a while, so this is the time to complain if you think we're
barking up the wrong tree on any issue.

> If you need more information, I'd be happy to provide some of the
> traceback information.
>
> The only thing I haven't updated is the version of Twisted installed
> on the machine. Is it necessary to have a newer version of Twisted for
> the ipython1 saw branch? I think Twisted that is installed is version
> 2.4.0.

Yes, saw relies of bugfixes to Twisted that are only part of Twisted
2.5.0.  We do now ship twisted's web2 module internally so you don't
have to track SVN for that.  Basically if you get Twisted 2.5.0
installed (which requires zope interfaces), you should be good to go.

Cheers,

f


From dfj225 at gmail.com  Tue Feb 20 16:56:56 2007
From: dfj225 at gmail.com (Douglas Jones)
Date: Tue, 20 Feb 2007 16:56:56 -0500
Subject: [IPython-dev] ipython1 remote exceptions causing local crashes
In-Reply-To: <db6b5ecc0702201300sa9065b4of7e3ac79a2ed160c@mail.gmail.com>
References: <1315be7e0702200754i65fbc002g2e1bd71b10d7bdb1@mail.gmail.com>
	<db6b5ecc0702201300sa9065b4of7e3ac79a2ed160c@mail.gmail.com>
Message-ID: <1315be7e0702201356h78201814w27becc7294760912@mail.gmail.com>

Fernando,

Thank you for the detailed response.

One thing I meant to mention in my previous post, is that the crash
only happens when using %autopx mode. Doing the same thing in using
rc.executeAll(...) causes the exception and the Traceback to be
displayed. Since you said this is a feature, I suppose you know this.
What I don't understand is why the behavior between the two should be
different.

That said, I understand the issue you are trying to solve by having
the client behave in this manner, however I think it might be
undesirable behavior for some projects.

A lot of my experience is based on the default python (CPython) shell,
which I think the previous "chainsaw" version of ipython1 did a good
job of emulating in regard to exceptions coming from the cluster.

Please forgive me if some of the issues I bring up have already been
discussed or have already been addressed in the code, as I haven't
been able to look through all of the new features and changes.

For the project that I am working on, it seems that one of the big
features is to use your cluster in an interactive manner. The "feel"
of this for our current implementation is much like typing any other
command into your python shell. Ipython1 goes a long way to providing
the features to make this necessary.

That said, it seems like an overreaction to have the user, more or
less, ejected from their interactive session for what may have been a
trivial mistake. I think one of the nice features of working
interactively is that you are given a chance to fix the mistake and
continue along with minimal fuss.

So, I can see the behavior that you have being useful for modes where
there is not a human at the keyboard. I agree that you wouldn't want
code to continue to run on the assumption that a previous command was
successful when it really wasn't.

Because I think IPython will be used for both situations (interactive
and "batch"), the handling of exceptions should at least be
configurable or changed internally based on the mode of use.

I realize this is a work in progress, but the current handling of
exceptions is misleading, I think. The user is presented with the
normal IPython has crashed error messages with instructions to mail
the developers along with a long traceback. This really makes it seem
like the error was internal to IPython and not something that was a
result of the user's own code being executed on the cluster. It was
this confusion that prompted me to mail the list with my "issue".

I hope you'll take into consideration the things I've said here. I'd
be happy to provide more information or go into more detail about how
I envision things working. I'd also be interested in hearing some of
what has already been decided and discussed as well as any future
development along this line that has been planned, but not implemented
yet.

Thank you,
~Doug


On 2/20/07, Fernando Perez <fperez.net at gmail.com> wrote:
> Hi Douglas,
>
> On 2/20/07, Douglas Jones <dfj225 at gmail.com> wrote:
> > Hi all,
> >
> > I've been looking at the new ipython1 saw branch and the implications
> > of the changes to a project that I'm working on.
> >
> > I've noticed in my testing that running some code on the engine that
> > causes an exception crashes my ipython client. I'm using recent svn
> > checkouts of both ipython1(revision 2104) and the ipython shell (which
> > lists itself as version IPython 0.7.4.svn.r2010).
> >
> > Is this a known issue?
>
> Well, it's a feature, not a bug :)  It was actually something that was
> deliberately added recently, to ensure that remote exceptions aren't
> silently swallowed and that you are forced to know about them.  The
> reason is that otherwise swallowing remote exceptions makes for
> incredibly hard debugging: your engine threw an exception which went
> silently into a log file only, and the client keeps on happily feeding
> it new commands to execute, most of which are likely to produce
> garbage because some previous quantity you expected to have computed
> is not there.
>
> We are trying to bring, within the realm of what's possible, a smooth
> transition from a model of single-process synchronous programming to
> distributed computing, and exceptions are a particularly thorny issue
> here.  We recently spoke to DOE people about  the next-generation
> language projects for distributed computing that the DOE/DARPA are
> looking into (X10, Fortress and Chapel, see
> http://www.ahpcrc.org/conferences/PGAS2006/presentations/Yelick.pdf).
> As far as we can tell, no system yet has a completely satisfactory
> answer for this, partly because I don't think you /can/ have such an
> answer.
>
> We hope we'll offer a model that's reasonable for end users (we are
> end users of this, so we're eating our own dog food here), and that
> will hopefully balance:
>
> - Feeling like an analytic continuation of the serial programming
> model (though remember that you may always find surprises off the real
> axis :)
>
> - Offering some utilities that make the common synchronization tasks
> needed for regular algorithmic development easy to achieve.
>
> - Exposing enough of the distributed objects to allow more
> sophisticated uses. It is still possible that we'll limit access to
> parts of the underlying asynchronous machinery for reasons I don't
> quite have time to explain right now.  I can go into more detail later
> if needed.
>
>
> But in all of this, feedback from users will be critically important.
> We've put a LOT of thought into this, but it's very easy to travel
> down a dead end without noticing it, so feel free to speak up.  If we
> get this right, this system should be very useful for many people and
> for a while, so this is the time to complain if you think we're
> barking up the wrong tree on any issue.
>
> > If you need more information, I'd be happy to provide some of the
> > traceback information.
> >
> > The only thing I haven't updated is the version of Twisted installed
> > on the machine. Is it necessary to have a newer version of Twisted for
> > the ipython1 saw branch? I think Twisted that is installed is version
> > 2.4.0.
>
> Yes, saw relies of bugfixes to Twisted that are only part of Twisted
> 2.5.0.  We do now ship twisted's web2 module internally so you don't
> have to track SVN for that.  Basically if you get Twisted 2.5.0
> installed (which requires zope interfaces), you should be good to go.
>
> Cheers,
>
> f
>


From fperez.net at gmail.com  Tue Feb 20 17:05:36 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 20 Feb 2007 15:05:36 -0700
Subject: [IPython-dev] ipython1 remote exceptions causing local crashes
In-Reply-To: <1315be7e0702201356h78201814w27becc7294760912@mail.gmail.com>
References: <1315be7e0702200754i65fbc002g2e1bd71b10d7bdb1@mail.gmail.com>
	<db6b5ecc0702201300sa9065b4of7e3ac79a2ed160c@mail.gmail.com>
	<1315be7e0702201356h78201814w27becc7294760912@mail.gmail.com>
Message-ID: <db6b5ecc0702201405p1d2f4ab2h48371f0f65612a8e@mail.gmail.com>

Hi,

On 2/20/07, Douglas Jones <dfj225 at gmail.com> wrote:

> Please forgive me if some of the issues I bring up have already been
> discussed or have already been addressed in the code, as I haven't
> been able to look through all of the new features and changes.
>
> For the project that I am working on, it seems that one of the big
> features is to use your cluster in an interactive manner. The "feel"
> of this for our current implementation is much like typing any other
> command into your python shell. Ipython1 goes a long way to providing
> the features to make this necessary.
>
> That said, it seems like an overreaction to have the user, more or
> less, ejected from their interactive session for what may have been a
> trivial mistake. I think one of the nice features of working
> interactively is that you are given a chance to fix the mistake and
> continue along with minimal fuss.

[...]

Ah! What you are mentionig is most definitely a bug, though.  Our
current expectation is that when you get a remote exception, you
should see an /exception/ in the local interactive shell, with a nice
traceback and all that.  But if you're actually getting dumped /out/
of ipython with a 'mail this crash report' message, that is a bug,
plain and simple.

Please do send us that crash report and whatever other details you
feel are relevant, and we'll try to hunt this one down ASAP.

Our intent is that the remote-enabled usage should feel very close to
the 'regular' ipython, so crashing out is definitely not a feature :)

Sorry that I misunderstood your original message.

Cheers,

f


From dfj225 at gmail.com  Wed Feb 21 09:44:22 2007
From: dfj225 at gmail.com (Douglas Jones)
Date: Wed, 21 Feb 2007 09:44:22 -0500
Subject: [IPython-dev] ipython1 remote exceptions causing local crashes
In-Reply-To: <db6b5ecc0702201405p1d2f4ab2h48371f0f65612a8e@mail.gmail.com>
References: <1315be7e0702200754i65fbc002g2e1bd71b10d7bdb1@mail.gmail.com>
	<db6b5ecc0702201300sa9065b4of7e3ac79a2ed160c@mail.gmail.com>
	<1315be7e0702201356h78201814w27becc7294760912@mail.gmail.com>
	<db6b5ecc0702201405p1d2f4ab2h48371f0f65612a8e@mail.gmail.com>
Message-ID: <1315be7e0702210644v14c407b8x4a786be8b7fd0047@mail.gmail.com>

Ah, it is good to hear that this is not a feature!

Sorry that my original message wasn't clear as to what was actually happening.

I've attached the crash report generated by IPython. In addition, I've
copied from my terminal some information that was printed out after
the "Mail this crash" message and doesn't seem to be in the auto
generated report.

First, some basic info about my machine:
Gentoo Linux 2.6.18 SMP
Python version: 2.4.3
Twisted version: 2.5.0 (I updated from 2.4.0 since my original
message, but the crash persists)
IPython version: 0.7.4.svn.r2010 (Crash was the same with earlier versions)
IPython1 "saw" branch: svn revision 2104

You can see the full list of commands that I've executed in the crash
report attached, but the basic overview is as follows:

rc.executeAll('a') # causes name exception, but does not crash the client
%autopx
a                      # same exception is generated here, but IPython
client crashes

If you need more information than what I've provided here, please let
me know and I will try to get whatever is needed.

Thanks,
~Doug


Here is what I've copied from the terminal after the IPython "mail
this crash" message.

Disconnecting from  ('localhost', 10111)
Unhandled error in Deferred:
(debug:  C: Deferred was created:
 C:  File "/usr/bin/ipython", line 27, in ?
 C:    IPython.Shell.start().mainloop()
 C:  File "/usr/lib64/python2.4/site-packages/IPython/Shell.py", line
59, in mainloop
 C:    self.IP.mainloop(banner)
 C:  File "/usr/lib64/python2.4/site-packages/IPython/iplib.py", line
1500, in mainloop
 C:    self.interact(banner)
 C:  File "/usr/lib64/python2.4/site-packages/IPython/iplib.py", line
1647, in interact
 C:    more = self.push(line)
 C:  File "/usr/lib64/python2.4/site-packages/IPython/iplib.py", line
1948, in push
 C:    more = self.runsource('\n'.join(self.buffer), self.filename)
 C:  File "/home/djones/local/ipython_saw/lib64/python2.4/site-packages/ipython1/kernel/magic.py",
line 139, in pxrunsource
 C:    self.activeController.iexecuteAll(source)
 C:  File "/home/djones/local/ipython_saw//lib64/python2.4/site-packages/ipython1/kernel/multiengineclient.py",
line 294, in iexecuteAll
 C:    return self.iexecute('all', lines)
 C:  File "/home/djones/local/ipython_saw//lib64/python2.4/site-packages/ipython1/kernel/multiengineclient.py",
line 286, in iexecute
 C:    d = self.execute(targets, lines, block=False)
 C:  File "/home/djones/local/ipython_saw//lib64/python2.4/site-packages/ipython1/kernel/multiengineclient.py",
line 114, in execute
 C:    return self.multiengine.execute(targets, lines, block)
 C:  File "/home/djones/local/ipython_saw//lib64/python2.4/site-packages/ipython1/kernel/multienginepb.py",
line 417, in execute
 C:    d2 = self._getActualDeferred(d)
 C:  File "/home/djones/local/ipython_saw//lib64/python2.4/site-packages/ipython1/kernel/multienginepb.py",
line 398, in _getActualDeferred
 C:    d2 = self._getPendingDeferred(deferredID)
 C:  File "/home/djones/local/ipython_saw//lib64/python2.4/site-packages/ipython1/kernel/multienginepb.py",
line 392, in _getPendingDeferred
 C:    d = self.callRemote('getPendingDeferred', deferredID)
 C:  File "/usr/lib64/python2.4/site-packages/twisted/spread/pb.py",
line 343, in callRemote
 C:    _name, args, kw)
 C:  File "/usr/lib64/python2.4/site-packages/twisted/spread/pb.py",
line 822, in _sendMessage
 C:    rval = defer.Deferred()
 I: First Invoker was:
 I:  File "/usr/bin/ipython", line 27, in ?
 I:    IPython.Shell.start().mainloop()
 I:  File "/usr/lib64/python2.4/site-packages/IPython/Shell.py", line
59, in mainloop
 I:    self.IP.mainloop(banner)
 I:  File "/usr/lib64/python2.4/site-packages/IPython/iplib.py", line
1500, in mainloop
 I:    self.interact(banner)
 I:  File "/usr/lib64/python2.4/site-packages/IPython/iplib.py", line
1647, in interact
 I:    more = self.push(line)
 I:  File "/usr/lib64/python2.4/site-packages/IPython/iplib.py", line
1948, in push
 I:    more = self.runsource('\n'.join(self.buffer), self.filename)
 I:  File "/home/djones/local/ipython_saw/lib64/python2.4/site-packages/ipython1/kernel/magic.py",
line 139, in pxrunsource
 I:    self.activeController.iexecuteAll(source)
 I:  File "/home/djones/local/ipython_saw//lib64/python2.4/site-packages/ipython1/kernel/multiengineclient.py",
line 294, in iexecuteAll
 I:    return self.iexecute('all', lines)
 I:  File "/home/djones/local/ipython_saw//lib64/python2.4/site-packages/ipython1/kernel/multiengineclient.py",
line 289, in iexecute
 I:    return self.blockOn(d)
 I:  File "/home/djones/local/ipython_saw//lib64/python2.4/site-packages/ipython1/kernel/multiengineclient.py",
line 87, in blockOn
 I:    return blockOn(d)
 I:  File "/home/djones/local/ipython_saw//lib64/python2.4/site-packages/ipython1/kernel/blockon.py",
line 156, in blockOn
 I:    return bd.blockOn()
 I:  File "/home/djones/local/ipython_saw//lib64/python2.4/site-packages/ipython1/kernel/blockon.py",
line 94, in blockOn
 I:    reactor.iterate(TIMEOUT)
 I:  File "/usr/lib64/python2.4/site-packages/twisted/internet/base.py",
line 382, in iterate
 I:    self.doIteration(delay)
 I:  File "/usr/lib64/python2.4/site-packages/twisted/internet/selectreactor.py",
line 133, in doSelect
 I:    _logrun(selectable, _drdw, selectable, method, dict)
 I:  File "/usr/lib64/python2.4/site-packages/twisted/python/log.py",
line 48, in callWithLogger
 I:    return callWithContext({"system": lp}, func, *args, **kw)
 I:  File "/usr/lib64/python2.4/site-packages/twisted/python/log.py",
line 33, in callWithContext
 I:    return context.call({ILogContext: newCtx}, func, *args, **kw)
 I:  File "/usr/lib64/python2.4/site-packages/twisted/python/context.py",
line 59, in callWithContext
 I:    return self.currentContext().callWithContext(ctx, func, *args, **kw)
 I:  File "/usr/lib64/python2.4/site-packages/twisted/python/context.py",
line 37, in callWithContext
 I:    return func(*args,**kw)
 I:  File "/usr/lib64/python2.4/site-packages/twisted/internet/selectreactor.py",
line 139, in _doReadOrWrite
 I:    why = getattr(selectable, method)()
 I:  File "/usr/lib64/python2.4/site-packages/twisted/internet/tcp.py",
line 362, in doRead
 I:    return self.protocol.dataReceived(data)
 I:  File "/usr/lib64/python2.4/site-packages/twisted/spread/banana.py",
line 218, in dataReceived
 I:    gotItem(item)
 I:  File "/usr/lib64/python2.4/site-packages/twisted/spread/banana.py",
line 148, in gotItem
 I:    self.callExpressionReceived(item)
 I:  File "/usr/lib64/python2.4/site-packages/twisted/spread/banana.py",
line 113, in callExpressionReceived
 I:    self.expressionReceived(obj)
 I:  File "/usr/lib64/python2.4/site-packages/twisted/spread/pb.py",
line 522, in expressionReceived
 I:    method(*sexp[1:])
 I:  File "/usr/lib64/python2.4/site-packages/twisted/spread/pb.py",
line 891, in proto_answer
 I:    d.callback(self.unserialize(netResult))
)
Traceback (most recent call last):
  File "/usr/lib64/python2.4/site-packages/twisted/spread/pb.py", line
847, in _recvMessage
    netResult = object.remoteMessageReceived(self, message, netArgs, netKw)
  File "/usr/lib64/python2.4/site-packages/twisted/spread/flavors.py",
line 119, in remoteMessageReceived
    state = method(*args, **kw)
  File "/home/djones/local/ipython_saw/lib64/python2.4/site-packages/ipython1/kernel/enginepb.py",
line 288, in remote_execute
    d = self.service.execute(lines)
  File "/home/djones/local/ipython_saw/lib64/python2.4/site-packages/ipython1/kernel/engineservice.py",
line 227, in execute
    d = defer.execute(self.shell.execute, lines)
--- <exception caught here> ---
  File "/usr/lib64/python2.4/site-packages/twisted/internet/defer.py",
line 79, in execute
    result = callable(*args, **kw)
  File "/home/djones/local/ipython_saw/lib64/python2.4/site-packages/ipython1/core/shell.py",
line 209, in execute
    raise self._tracebackTuple
exceptions.NameError:



On 2/20/07, Fernando Perez <fperez.net at gmail.com> wrote:
> Hi,
>
> On 2/20/07, Douglas Jones <dfj225 at gmail.com> wrote:
>
> > Please forgive me if some of the issues I bring up have already been
> > discussed or have already been addressed in the code, as I haven't
> > been able to look through all of the new features and changes.
> >
> > For the project that I am working on, it seems that one of the big
> > features is to use your cluster in an interactive manner. The "feel"
> > of this for our current implementation is much like typing any other
> > command into your python shell. Ipython1 goes a long way to providing
> > the features to make this necessary.
> >
> > That said, it seems like an overreaction to have the user, more or
> > less, ejected from their interactive session for what may have been a
> > trivial mistake. I think one of the nice features of working
> > interactively is that you are given a chance to fix the mistake and
> > continue along with minimal fuss.
>
> [...]
>
> Ah! What you are mentionig is most definitely a bug, though.  Our
> current expectation is that when you get a remote exception, you
> should see an /exception/ in the local interactive shell, with a nice
> traceback and all that.  But if you're actually getting dumped /out/
> of ipython with a 'mail this crash report' message, that is a bug,
> plain and simple.
>
> Please do send us that crash report and whatever other details you
> feel are relevant, and we'll try to hunt this one down ASAP.
>
> Our intent is that the remote-enabled usage should feel very close to
> the 'regular' ipython, so crashing out is definitely not a feature :)
>
> Sorry that I misunderstood your original message.
>
> Cheers,
>
> f
>
-------------- next part --------------
***************************************************************************

IPython post-mortem report

IPython version: 0.7.4.svn.r2010 

SVN revision   : 2010M 

Platform info  : os.name -> posix, sys.platform -> linux2

***************************************************************************

Current user configuration structure:

{'Version': 0,
 '__allownew': False,
 'alias': [],
 'args': [],
 'autocall': 1,
 'autoedit_syntax': 0,
 'autoindent': 1,
 'automagic': 1,
 'banner': 1,
 'c': '',
 'cache_size': 1000,
 'classic': 0,
 'color_info': 1,
 'colors': 'Linux',
 'confirm_exit': 1,
 'debug': 0,
 'deep_reload': 0,
 'editor': '/usr/bin/vim',
 'embedded': False,
 'execfile': [],
 'execute': [''],
 'gthread': 0,
 'help': 0,
 'ignore': 0,
 'import_all': [],
 'import_mod': [],
 'import_some': [[]],
 'include': [],
 'ipythondir': '/home/djones/.ipython',
 'log': 0,
 'logfile': '',
 'logplay': '',
 'magic_docstrings': 0,
 'messages': 1,
 'multi_line_specials': 1,
 'nosep': 0,
 'object_info_string_level': 0,
 'opts': Struct({'__allownew': True}),
 'pdb': 0,
 'pprint': 1,
 'profile': '',
 'prompt_in1': 'In [\\#]: ',
 'prompt_in2': '   .\\D.: ',
 'prompt_out': 'Out[\\#]: ',
 'prompts_pad_left': 1,
 'pylab': 0,
 'q4thread': 0,
 'qthread': 0,
 'quick': 0,
 'quiet': 0,
 'rcfile': 'ipythonrc',
 'readline': 1,
 'readline_merge_completions': 1,
 'readline_omit__names': 0,
 'readline_parse_and_bind': ['tab: complete',
                             '"\\C-l": possible-completions',
                             'set show-all-if-ambiguous on',
                             '"\\C-o": tab-insert',
                             '"\\M-i": "    "',
                             '"\\M-o": "\\d\\d\\d\\d"',
                             '"\\M-I": "\\d\\d\\d\\d"',
                             '"\\C-r": reverse-search-history',
                             '"\\C-s": forward-search-history',
                             '"\\C-p": history-search-backward',
                             '"\\C-n": history-search-forward',
                             '"\\e[A": history-search-backward',
                             '"\\e[B": history-search-forward',
                             '"\\C-k": kill-line',
                             '"\\C-u": unix-line-discard'],
 'readline_remove_delims': '-/~',
 'screen_length': -2,
 'separate_in': '\n',
 'separate_out': '',
 'separate_out2': '',
 'system_header': 'IPython system call: ',
 'system_verbose': 0,
 'term_title': 1,
 'tk': 0,
 'upgrade': 0,
 'wildcards_case_sensitive': 1,
 'wthread': 0,
 'wxversion': '0',
 'xmode': 'Context'}

***************************************************************************

Crash traceback:

---------------------------------------------------------------------------
exceptions.NameError                                     Python 2.4.3: /usr/bin/python
                                                   Wed Feb 21 09:26:41 2007
A problem occured executing Python code.  Here is the sequence of function
calls leading up to the error, with the most recent (innermost) call last.

/usr/bin/ipython 
     12 IPython.Shell.IPShell().mainloop(sys_exit=1)
     13 
     14 [or simply IPython.Shell.IPShell().mainloop(1) ]
     15 
     16 and IPython will be your working environment when you start python. The final
     17 sys.exit() call will make python exit transparently when IPython finishes, so
     18 you don't have an extra prompt to get out of.
     19 
     20 This is probably useful to developers who manage multiple Python versions and
     21 don't want to have correspondingly multiple IPython versions. Note that in
     22 this mode, there is no way to pass IPython any command-line options, as those
     23 are trapped first by Python itself.
     24 """
     25 
     26 import IPython
---> 27 IPython.Shell.start().mainloop()
        IPython.Shell.start.mainloop = undefined
     28 
     29 
     30 
     31 
     32 
     33 
     34 
     35 
     36 
     37 
     38 
     39 
     40 
     41 
     42 

/usr/lib64/python2.4/site-packages/IPython/Shell.py in mainloop(self=<IPython.Shell.IPShell instance>, sys_exit=0, banner=None)
     44 #-----------------------------------------------------------------------------
     45 # This class is trivial now, but I want to have it in to publish a clean
     46 # interface. Later when the internals are reorganized, code that uses this
     47 # shouldn't have to change.
     48 
     49 class IPShell:
     50     """Create an IPython instance."""
     51     
     52     def __init__(self,argv=None,user_ns=None,user_global_ns=None,
     53                  debug=1,shell_class=InteractiveShell):
     54         self.IP = make_IPython(argv,user_ns=user_ns,
     55                                user_global_ns=user_global_ns,
     56                                debug=debug,shell_class=shell_class)
     57 
     58     def mainloop(self,sys_exit=0,banner=None):
---> 59         self.IP.mainloop(banner)
        self.IP.mainloop = <bound method InteractiveShell.mainloop of <IPython.iplib.InteractiveShell object at 0x2b0675e2f210>>
        banner = None
     60         if sys_exit:
     61             sys.exit()
     62 
     63 #-----------------------------------------------------------------------------
     64 class IPShellEmbed:
     65     """Allow embedding an IPython shell into a running program.
     66 
     67     Instances of this class are callable, with the __call__ method being an
     68     alias to the embed() method of an InteractiveShell instance.
     69 
     70     Usage (see also the example-embed.py file for a running example):
     71 
     72     ipshell = IPShellEmbed([argv,banner,exit_msg,rc_override])
     73 
     74     - argv: list containing valid command-line options for IPython, as they

/usr/lib64/python2.4/site-packages/IPython/iplib.py in mainloop(self=<IPython.iplib.InteractiveShell object>, banner='Python 2.4.3 (#1, Oct 25 2006, 17:53:09) \nType "...ut \'object\'. ?object also works, ?? prints more.\n')
   1485 
   1486         If an optional banner argument is given, it will override the
   1487         internally created default banner."""
   1488 
   1489         if self.rc.c:  # Emulate Python's -c option
   1490             self.exec_init_cmd()
   1491         if banner is None:
   1492             if not self.rc.banner:
   1493                 banner = ''
   1494             # banner is string? Use it directly!
   1495             elif isinstance(self.rc.banner,basestring):
   1496                 banner = self.rc.banner
   1497             else:                
   1498                 banner = self.BANNER+self.banner2
   1499 
-> 1500         self.interact(banner)
        self.interact = <bound method InteractiveShell.interact of <IPython.iplib.InteractiveShell object at 0x2b0675e2f210>>
        banner = 'Python 2.4.3 (#1, Oct 25 2006, 17:53:09) \nType "copyright", "credits" or "license" for more information.\n\nIPython 0.7.4.svn.r2010 -- An enhanced Interactive Python.\n?       -> Introduction to IPython\'s features.\n%magic  -> Information about IPython\'s \'magic\' % functions.\nhelp    -> Python\'s own help system.\nobject? -> Details about \'object\'. ?object also works, ?? prints more.\n'
   1501 
   1502     def exec_init_cmd(self):
   1503         """Execute a command given at the command line.
   1504 
   1505         This emulates Python's -c option."""
   1506 
   1507         #sys.argv = ['-c']
   1508         self.push(self.rc.c)
   1509 
   1510     def embed_mainloop(self,header='',local_ns=None,global_ns=None,stack_depth=0):
   1511         """Embeds IPython into a running python program.
   1512 
   1513         Input:
   1514 
   1515           - header: An optional header message can be specified.

/usr/lib64/python2.4/site-packages/IPython/iplib.py in interact(self=<IPython.iplib.InteractiveShell object>, banner='Python 2.4.3 (#1, Oct 25 2006, 17:53:09) \nType "...ut \'object\'. ?object also works, ?? prints more.\n')
   1632             except EOFError:
   1633                 if self.autoindent:
   1634                     self.readline_startup_hook(None)
   1635                 self.write('\n')
   1636                 self.exit()
   1637             except bdb.BdbQuit:
   1638                 warn('The Python debugger has exited with a BdbQuit exception.\n'
   1639                      'Because of how pdb handles the stack, it is impossible\n'
   1640                      'for IPython to properly format this particular exception.\n'
   1641                      'IPython will resume normal operation.')
   1642             except:
   1643                 # exceptions here are VERY RARE, but they can be triggered
   1644                 # asynchronously by signal handlers, for example.
   1645                 self.showtraceback()
   1646             else:
-> 1647                 more = self.push(line)
        more = False
        self.push = <bound method InteractiveShell.push of <IPython.iplib.InteractiveShell object at 0x2b0675e2f210>>
        line = 'a'
   1648                 if (self.SyntaxTB.last_syntax_error and
   1649                     self.rc.autoedit_syntax):
   1650                     self.edit_syntax_error()
   1651             
   1652         # We are off again...
   1653         __builtin__.__dict__['__IPYTHON__active'] -= 1
   1654 
   1655     def excepthook(self, etype, value, tb):
   1656       """One more defense for GUI apps that call sys.excepthook.
   1657 
   1658       GUI frameworks like wxPython trap exceptions and call
   1659       sys.excepthook themselves.  I guess this is a feature that
   1660       enables them to keep running after exceptions that would
   1661       otherwise kill their mainloop. This is a bother for IPython
   1662       which excepts to catch all of the program exceptions with a try:

/usr/lib64/python2.4/site-packages/IPython/iplib.py in push(self=<IPython.iplib.InteractiveShell object>, line='a')
   1933         is reset; otherwise, the command is incomplete, and the buffer
   1934         is left as it was after the line was appended.  The return
   1935         value is 1 if more input is required, 0 if the line was dealt
   1936         with in some way (this is the same as runsource()).
   1937         """
   1938 
   1939         # autoindent management should be done here, and not in the
   1940         # interactive loop, since that one is only seen by keyboard input.  We
   1941         # need this done correctly even for code run via runlines (which uses
   1942         # push).
   1943 
   1944         #print 'push line: <%s>' % line  # dbg
   1945         for subline in line.splitlines():
   1946             self.autoindent_update(subline)
   1947         self.buffer.append(line)
-> 1948         more = self.runsource('\n'.join(self.buffer), self.filename)
        more = undefined
        self.runsource.join = undefined
        self.buffer = ['a']
        self.filename = '<ipython console>'
   1949         if not more:
   1950             self.resetbuffer()
   1951         return more
   1952 
   1953     def resetbuffer(self):
   1954         """Reset the input buffer."""
   1955         self.buffer[:] = []
   1956         
   1957     def raw_input(self,prompt='',continue_prompt=False):
   1958         """Write a prompt and read a line.
   1959 
   1960         The returned line does not include the trailing newline.
   1961         When the user enters the EOF key sequence, EOFError is raised.
   1962 
   1963         Optional inputs:

/home/djones/local/ipython_saw/lib64/python2.4/site-packages/ipython1/kernel/magic.py in pxrunsource(self=<IPython.iplib.InteractiveShell object>, source='a', filename='<ipython console>', symbol='single')
    124         # Case 1
    125         self.showsyntaxerror(filename)
    126         return None
    127 
    128     if code is None:
    129         # Case 2
    130         return True
    131 
    132     # Case 3
    133     # Because autopx is enabled, we now call executeAll or disable autopx if
    134     # %autopx or autopx has been called
    135     if '_ip.magic("%autopx' in source or '_ip.magic("autopx' in source:
    136         _disable_autopx(self)
    137         return False
    138     else:
--> 139         self.activeController.iexecuteAll(source)
        self.activeController.iexecuteAll = <bound method PBInteractiveMultiEngineClient.iexecuteAll of <ipython1.kernel.multienginepb.PBInteractiveMultiEngineClient object at 0x2b067b832e10>>
        source = 'a'
    140         return False
    141         
    142 def magic_autopx(self, parameter_s=''):
    143     """Toggles auto parallel mode for the active IPython Controller.
    144     
    145     To activate a Controller in IPython, first create it and then call
    146     the activate() method.
    147     
    148     Then you can do the following:
    149      
    150     >>> %autopx                    # Now all commands are executed in parallel
    151     Auto Parallel Enabled
    152     Type %autopx to disable
    153     ...
    154     >>> %autopx                    # Now all commands are locally executed

/home/djones/local/ipython_saw/lib64/python2.4/site-packages/ipython1/kernel/multiengineclient.py in iexecuteAll(self=<ipython1.kernel.multienginepb.PBInteractiveMultiEngineClient object>, lines='a')
    279                 if cmd_stderr:
    280                     print "%s[%s:%i]%s Err[%i]:\n%s %s" % \
    281                         (green, self.addr[0], target,
    282                         red, cmd_num, normal, cmd_stderr)
    283         return None
    284         
    285     def iexecute(self, targets, lines, block=None):
    286         d = self.execute(targets, lines, block=False)
    287         d.addCallback(self._printResult)
    288         if (self.block and block is None) or block:
    289             return self.blockOn(d)
    290         else:
    291             return d
    292         
    293     def iexecuteAll(self, lines):
--> 294         return self.iexecute('all', lines)
        self.iexecute = <bound method PBInteractiveMultiEngineClient.iexecute of <ipython1.kernel.multienginepb.PBInteractiveMultiEngineClient object at 0x2b067b832e10>>
        lines = 'a'
    295         
    296     def igetResult(self, targets, i=None):
    297         saveBlock = self.block
    298         self.block = False
    299         d = self.getResult(targets, i)
    300         d.addCallback(self._printResult)
    301         self.block = saveBlock
    302         return self._blockOrNot(d)   
    303     
    304     def igetResultAll(self, i=None):
    305         return self.igetResult('all', i)
    306     
    307     def _printQueueStatus(self, status):
    308         for e in status:
    309             print "Engine: ", e[0]

/home/djones/local/ipython_saw/lib64/python2.4/site-packages/ipython1/kernel/multiengineclient.py in iexecute(self=<ipython1.kernel.multienginepb.PBInteractiveMultiEngineClient object>, targets='all', lines='a', block=None)
    274                     blue, cmd_num, normal, cmd_stdin)
    275                 if cmd_stdout:
    276                     print "%s[%s:%i]%s Out[%i]:%s %s" % \
    277                         (green, self.addr[0], target,
    278                         red, cmd_num, normal, cmd_stdout)
    279                 if cmd_stderr:
    280                     print "%s[%s:%i]%s Err[%i]:\n%s %s" % \
    281                         (green, self.addr[0], target,
    282                         red, cmd_num, normal, cmd_stderr)
    283         return None
    284         
    285     def iexecute(self, targets, lines, block=None):
    286         d = self.execute(targets, lines, block=False)
    287         d.addCallback(self._printResult)
    288         if (self.block and block is None) or block:
--> 289             return self.blockOn(d)
        self.blockOn = <bound method PBInteractiveMultiEngineClient.blockOn of <ipython1.kernel.multienginepb.PBInteractiveMultiEngineClient object at 0x2b067b832e10>>
        d = <Deferred at 0x2b067b6f93b0  current result: <twisted.python.failure.Failure exceptions.NameError>>
    290         else:
    291             return d
    292         
    293     def iexecuteAll(self, lines):
    294         return self.iexecute('all', lines)
    295         
    296     def igetResult(self, targets, i=None):
    297         saveBlock = self.block
    298         self.block = False
    299         d = self.getResult(targets, i)
    300         d.addCallback(self._printResult)
    301         self.block = saveBlock
    302         return self._blockOrNot(d)   
    303     
    304     def igetResultAll(self, i=None):

/home/djones/local/ipython_saw/lib64/python2.4/site-packages/ipython1/kernel/multiengineclient.py in blockOn(self=<ipython1.kernel.multienginepb.PBInteractiveMultiEngineClient object>, d=<Deferred at 0x2b067b6f93b0  current result: <twisted.python.failure.Failure exceptions.NameError>>)
     72             
     73     def _setBlock(self, block):
     74         self._block = block
     75         if self.multiengine is not None:
     76             self.multiengine.block = block
     77     
     78     block = property(_getBlock, _setBlock, None, None)
     79     
     80     def __init__(self, addr):
     81         self.addr = addr
     82         self.multiengine = None
     83         self._block = True
     84         self.connected = False
     85                            
     86     def blockOn(self, d):
---> 87         return blockOn(d)
        global blockOn = <function blockOn at 0x2b067b46d758>
        d = <Deferred at 0x2b067b6f93b0  current result: <twisted.python.failure.Failure exceptions.NameError>>
     88             
     89     def _blockOrNot(self, d):
     90         if self._block:
     91             return self.blockOn(d)
     92         else:
     93             return d
     94 
     95     #---------------------------------------------------------------------------
     96     # Methods for subclasses to override
     97     #---------------------------------------------------------------------------
     98             
     99     def connect(self):
    100         """Create self.multiengine and set self.connected to True.
    101         
    102         Implementers of this method must also (after creating self.multiengine)

/home/djones/local/ipython_saw/lib64/python2.4/site-packages/ipython1/kernel/blockon.py in blockOn(deferrable=[<Deferred at 0x2b067b6f93b0  current result: <twisted.python.failure.Failure exceptions.NameError>>], fireOnOneCallback=0, fireOnOneErrback=0, consumeErrors=0)
    141         deferrable = [deferrable]
    142     
    143     # Add a check to simply pass through plain objects.
    144     for i in range(len(deferrable)):
    145         if hasattr(deferrable[i], '__defer__'):
    146             deferrable[i] = deferrable[i].__defer__()
    147     
    148     d = gatherBoth(deferrable,
    149                    fireOnOneCallback, 
    150                    fireOnOneErrback,
    151                    consumeErrors,
    152                    logErrors=0)
    153     if not fireOnOneCallback:
    154         d.addCallback(_parseResults)
    155     bd = BlockingDeferred(d)
--> 156     return bd.blockOn()
        bd.blockOn = <bound method BlockingDeferred.blockOn of <ipython1.kernel.blockon.BlockingDeferred object at 0x2b067b90b410>>
    157     
    158 
    159 # Start the reactor
    160 startReactor()
    161 
    162 
    163 
    164 
    165 
    166 
    167 
    168 
    169 
    170 
    171 

/home/djones/local/ipython_saw/lib64/python2.4/site-packages/ipython1/kernel/blockon.py in blockOn(self=<ipython1.kernel.blockon.BlockingDeferred object>)
     85         On success this will return the result.
     86         
     87         On failure, it will raise an exception.
     88         """
     89         
     90         self.d.addBoth(self.gotResult)
     91         self.d.addErrback(self.gotFailure)
     92         
     93         while not self.finished:
     94             reactor.iterate(TIMEOUT)
     95             self.count += 1
     96         
     97         if isinstance(self.d.result, dict):
     98             f = self.d.result.get('failure', None)
     99             if isinstance(f, failure.Failure):
--> 100                 f.raiseException()
        f.raiseException = <bound method Failure.raiseException of <twisted.python.failure.Failure exceptions.NameError>>
    101         return self.d.result
    102 
    103     def gotResult(self, result):
    104         self.finished = True
    105         return result
    106         
    107     def gotFailure(self, f):
    108         self.finished = True
    109         # Now make it look like a success so the failure isn't unhandled
    110         return {'failure':f}
    111         
    112         
    113 def _parseResults(result):
    114     if isinstance(result, (list, tuple)):
    115         if len(result) == 1:

/usr/lib64/python2.4/site-packages/twisted/python/failure.py in raiseException(self=<twisted.python.failure.Failure exceptions.NameError>)
    244         @returns: the matching L{Exception} type, or None if no match.
    245         """
    246         for error in errorTypes:
    247             err = error
    248             if inspect.isclass(error) and issubclass(error, Exception):
    249                 err = reflect.qual(error)
    250             if err in self.parents:
    251                 return error
    252         return None
    253 
    254     def raiseException(self):
    255         """
    256         raise the original exception, preserving traceback
    257         information if available.
    258         """
--> 259         raise self.type, self.value, self.tb
        self.type = <class exceptions.NameError at 0x2b0675de65f0>
        self.value = <exceptions.NameError instance at 0x2b067b6f8518>
        self.tb = None
    260 
    261 
    262     def __repr__(self):
    263         return "<%s %s>" % (self.__class__, self.type)
    264 
    265     def __str__(self):
    266         return "[Failure instance: %s]" % self.getBriefTraceback()
    267 
    268     def __getstate__(self):
    269         """Avoid pickling objects in the traceback.
    270         """
    271         if self.pickled:
    272             return self.__dict__
    273         c = self.__dict__.copy()
    274         

NameError: 

***************************************************************************

History of session input:
import ipython1.kernel.api as api
rc = api.RemoteController(('localhost',10111))
rc.connect()
rc.activate()
rc.executeAll("a")
_ip.magic("%autopx ")
a

*** Last line of input (may not be in above history):
a

From fperez.net at gmail.com  Wed Feb 21 16:47:28 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 21 Feb 2007 14:47:28 -0700
Subject: [IPython-dev] ipython1 remote exceptions causing local crashes
In-Reply-To: <1315be7e0702210644v14c407b8x4a786be8b7fd0047@mail.gmail.com>
References: <1315be7e0702200754i65fbc002g2e1bd71b10d7bdb1@mail.gmail.com>
	<db6b5ecc0702201300sa9065b4of7e3ac79a2ed160c@mail.gmail.com>
	<1315be7e0702201356h78201814w27becc7294760912@mail.gmail.com>
	<db6b5ecc0702201405p1d2f4ab2h48371f0f65612a8e@mail.gmail.com>
	<1315be7e0702210644v14c407b8x4a786be8b7fd0047@mail.gmail.com>
Message-ID: <db6b5ecc0702211347x1a6db6adjdc274a7244df402d@mail.gmail.com>

On 2/21/07, Douglas Jones <dfj225 at gmail.com> wrote:
> Ah, it is good to hear that this is not a feature!
>
> Sorry that my original message wasn't clear as to what was actually happening.
>
> I've attached the crash report generated by IPython. In addition, I've
> copied from my terminal some information that was printed out after
> the "Mail this crash" message and doesn't seem to be in the auto
> generated report.

Thanks a lot, Douglas.  This info should be enough.  Brian is leaving
tonight for PyCon and he's been working a lot on the saw internals, so
it may be a few days before we fix this.  I'll still have a look and
might be able to do it early if it doesn't require messing with the
code that Brian is currently manipulating.

But in any case, we'll work on it and will fix it.

Regards,

f


From david.huard at gmail.com  Thu Feb 22 14:18:40 2007
From: david.huard at gmail.com (David Huard)
Date: Thu, 22 Feb 2007 14:18:40 -0500
Subject: [IPython-dev] Setup problem with ipython1
Message-ID: <pan.2007.02.22.19.18.28.579505@gmail.com>

Hi, 

I just updated the saw branch and I have problems with the installation;
it complains about being unable to import ipython1.external. 

I added a ipython1/external/twisted/web2 to `with_packages` in setup.py,
but then, by calling import ipython1.kernel.api I get

/usr/local/lib/python2.4/site-packages/ipython1/external/twisted/web2/__init__.py
      9
     10 """
     11
---> 12 from ipython1.external.twisted.web2._version import version
     13 __version__ = version.short()

/usr/local/lib/python2.4/site-packages/ipython1/external/twisted/web2/_version.py
      1 # This is an auto-generated file. Use admin/change-versions to update.
----> 2 from twisted.python import versions
      3 version = versions.Version(__name__[:__name__.rfind('.')], 0, 2, 0)

my twisted version is 2.2.0, is this outdated ?

Thanks, 

David Huard




From fperez.net at gmail.com  Thu Feb 22 14:51:07 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 22 Feb 2007 12:51:07 -0700
Subject: [IPython-dev] Setup problem with ipython1
In-Reply-To: <pan.2007.02.22.19.18.28.579505@gmail.com>
References: <pan.2007.02.22.19.18.28.579505@gmail.com>
Message-ID: <db6b5ecc0702221151w701d5f04i83914829a57fd390@mail.gmail.com>

Hi David,

On 2/22/07, David Huard <david.huard at gmail.com> wrote:
> Hi,
>
> I just updated the saw branch and I have problems with the installation;
> it complains about being unable to import ipython1.external.
>
> I added a ipython1/external/twisted/web2 to `with_packages` in setup.py,
> but then, by calling import ipython1.kernel.api I get
>
> /usr/local/lib/python2.4/site-packages/ipython1/external/twisted/web2/__init__.py
>       9
>      10 """
>      11
> ---> 12 from ipython1.external.twisted.web2._version import version
>      13 __version__ = version.short()
>
> /usr/local/lib/python2.4/site-packages/ipython1/external/twisted/web2/_version.py
>       1 # This is an auto-generated file. Use admin/change-versions to update.
> ----> 2 from twisted.python import versions
>       3 version = versions.Version(__name__[:__name__.rfind('.')], 0, 2, 0)
>
> my twisted version is 2.2.0, is this outdated ?

Yes, I'm afraid so.  Twisted 2.5.0 is required for this to run.

Having said that, saw is currently undergoing massive changes, so I'd
recommend waiting until next week for the dust to settle just a
little.  That's why we haven't really announced much of anything
publicly yet :)

Cheers,

f


From fperez.net at gmail.com  Sun Feb 25 17:00:06 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Sun, 25 Feb 2007 15:00:06 -0700
Subject: [IPython-dev] pydoc and introspective features
In-Reply-To: <45D7FBEA.4080702@gmail.com>
References: <mailman.14355.1171777712.8339.ipython-dev@scipy.org>
	<45D7FBEA.4080702@gmail.com>
Message-ID: <db6b5ecc0702251400w5505472te85ca5a0d86dee75@mail.gmail.com>

Sorry for not replying earlier, I was caught up in the prep for
PyCon'07 (which I didn't personally attend, but where Brian presented
yesterday two ipython talks).

> > Subject:  pydoc and introspective features
> > From: "Laurent Gautier" <lgautier at gmail.com>
> > Date: Sun, 18 Feb 2007 13:48:28 +0800
> > To: ipython-dev at scipy.org
> >
> >
> > Hi,
> >
> > We are two people in the process of rewriting pydoc for the standard python
> > (with python2.6 as a target), splitting the whole into functional units (modules
> > in a package) and creating the units with extensions outside pydoc in mind.
> > In all honesty, we have no warranty that it will get included but we
> > are committed
> > to release a python module that could replace the existing pydoc anyway.
> >
> > I have been (and remain) an happy user of ipython, and I was thinking that
> > that some of the introspective documention-centered capabilities that we are
> > developing could be of interest to the ipython project.
> >
> > I see that ipython is being rewritten, and that might well be the right moment
> > for me to bring that up and call for comments.
> >
> > Anyone to comment ?

Yes, I think that a more modular pydoc would be great.  Off the top of
my head, a few things:

- search.  This is probably the biggest gripe everyone has with python
vs. commercial interactive systems (such as Matlab or Mathematica).
Tab-completion and 'foo?' work great, but if you don't even know where
to begin looking for something, you're stuck.    A builtin indexing
system that could be either exposed via a web browser or to a
command-line program (such as ipython) would be very welcome by a lot
of users.

- unification with epydoc (or something else)?  I don't know if this
is on the table, but I think that it would be great if the
html-doc-generator from the stdlib were so good that we didn't have to
look for replacement third-party tools.  But I'm not terribly aware of
the internal details and issues, so perhaps you perceive these as
tools with separate purposes.

- easy extensibility.  IPython allows objects to define their own
.getdoc() method to provide documentation in ways that can't be
achieved with a static __doc__ string attribute.  This was added at
user's request, so there's a real-world need for it.  My approach was
a bit ad-hoc, but it would be nice to have a standardized protocol for
objects to expose information about themselves.  If pydoc leads the
way with an officially accepted convention here, ipython will follow
suit.

  This is an area that will require some thought, because ideally
objects should be able to provide information about themselves in the
richest possible way, but that way is dependent on the requester's
environment: for example, is the client capable of HTML or graphical
display, or is it just a terminal?  A generic solution should allow
objects to know the capabilities of the requester, so it can
accordingly provide as rich an information display as the requester
can handle.  A bit of experimentation here may show what a sensible
API for this would be.

- You may want to have a look in ipython's OInspect module, which is
responsible for extracting information about objects.  If pydoc moves
in that direction, feel free to take any code from this that you may
find useful.


Regards,

f


From danmil at comcast.net  Mon Feb 26 13:17:25 2007
From: danmil at comcast.net (Dan Milstein)
Date: Mon, 26 Feb 2007 12:17:25 -0600
Subject: [IPython-dev] Auto-calling question
Message-ID: <29FC48C7-1223-4C59-9F65-3E6EB97CDFEB@comcast.net>

I'm working my way through the prefilter code and I have a question  
which isn't clear from the docs (to me, at least):

Are the ';', ',', and '/' escapes only supposed to be live if the  
user has %autocall on?  Or are they supposed to be live all the time?

Currently, they're only live if %autocall *is* on, but the code is a  
bit confusing on that front, and I'm not 100% certain that's what's  
intended.

I.e., here's the current behavior:

In [1]: ;len 1 2 3
------> len("1 2 3")
Out[1]: 5

In [2]: %autocall 0
Automatic calling is: OFF

In [3]: ;len 1 2 3
------------------------------------------------------------
    File "<ipython console>", line 1
      ;len 1 2 3
      ^
SyntaxError: invalid syntax


If this *is* the right behavior, I have a small patch to clean up  
some confusing stuff with iplib.InteractiveShell.esc_handlers.

-Dan

p.s. There's a comment that sort of suggests that, at some point,  
those were live all the time.  The 'original' re for line_split looks  
like all escape characters were created equal. (iplib.py l. 495)


From fperez.net at gmail.com  Mon Feb 26 13:43:27 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Mon, 26 Feb 2007 11:43:27 -0700
Subject: [IPython-dev] Auto-calling question
In-Reply-To: <29FC48C7-1223-4C59-9F65-3E6EB97CDFEB@comcast.net>
References: <29FC48C7-1223-4C59-9F65-3E6EB97CDFEB@comcast.net>
Message-ID: <db6b5ecc0702261043x28c53522ka64c3c8edbc830fa@mail.gmail.com>

On 2/26/07, Dan Milstein <danmil at comcast.net> wrote:
> I'm working my way through the prefilter code and I have a question
> which isn't clear from the docs (to me, at least):
>
> Are the ';', ',', and '/' escapes only supposed to be live if the
> user has %autocall on?  Or are they supposed to be live all the time?
>
> Currently, they're only live if %autocall *is* on, but the code is a
> bit confusing on that front, and I'm not 100% certain that's what's
> intended.

That code is hideously confusing and brittle, unfortunately.  I
/think/ (I honestly don't exactly remember) the original intent was
for the special first-character escapes to be always active, with the
autocall behavior controlling only the automatic addition of parens
without any user input.

So if you have a patch that restores this functionality without
breaking anything else in the beautiful plate of spaghetti that
_prefilter() is, by all means send it in :)

Regards,

f


From danmil at comcast.net  Mon Feb 26 13:47:28 2007
From: danmil at comcast.net (Dan Milstein)
Date: Mon, 26 Feb 2007 12:47:28 -0600
Subject: [IPython-dev] Auto-calling question
In-Reply-To: <db6b5ecc0702261043x28c53522ka64c3c8edbc830fa@mail.gmail.com>
References: <29FC48C7-1223-4C59-9F65-3E6EB97CDFEB@comcast.net>
	<db6b5ecc0702261043x28c53522ka64c3c8edbc830fa@mail.gmail.com>
Message-ID: <478DFE6E-2908-4574-9E84-7E94C7629EC1@comcast.net>

That's way helpful -- it seemed like it *should* work that way, but  
it doesn't (i.e. the other escape chars are always active -- why not  
those?).

I do have an idea for cleaning up all that prefilter code, so I'll  
see if I can fix that all together once I get the tests up and running.

Will hopefully have a decent set of tests to submit soonish (which  
will fail for this behavior ;-).

-D


On Feb 26, 2007, at 12:43 PM, Fernando Perez wrote:

> On 2/26/07, Dan Milstein <danmil at comcast.net> wrote:
>> I'm working my way through the prefilter code and I have a question
>> which isn't clear from the docs (to me, at least):
>>
>> Are the ';', ',', and '/' escapes only supposed to be live if the
>> user has %autocall on?  Or are they supposed to be live all the time?
>>
>> Currently, they're only live if %autocall *is* on, but the code is a
>> bit confusing on that front, and I'm not 100% certain that's what's
>> intended.
>
> That code is hideously confusing and brittle, unfortunately.  I
> /think/ (I honestly don't exactly remember) the original intent was
> for the special first-character escapes to be always active, with the
> autocall behavior controlling only the automatic addition of parens
> without any user input.
>
> So if you have a patch that restores this functionality without
> breaking anything else in the beautiful plate of spaghetti that
> _prefilter() is, by all means send it in :)
>
> Regards,
>
> f



From vivainio at gmail.com  Mon Feb 26 14:21:58 2007
From: vivainio at gmail.com (Ville M. Vainio)
Date: Mon, 26 Feb 2007 20:21:58 +0100
Subject: [IPython-dev] Auto-calling question
In-Reply-To: <478DFE6E-2908-4574-9E84-7E94C7629EC1@comcast.net>
References: <29FC48C7-1223-4C59-9F65-3E6EB97CDFEB@comcast.net>
	<db6b5ecc0702261043x28c53522ka64c3c8edbc830fa@mail.gmail.com>
	<478DFE6E-2908-4574-9E84-7E94C7629EC1@comcast.net>
Message-ID: <46cb515a0702261121g2c59f168y74831e1de3a83f2c@mail.gmail.com>

On 2/26/07, Dan Milstein <danmil at comcast.net> wrote:

> I do have an idea for cleaning up all that prefilter code, so I'll
> see if I can fix that all together once I get the tests up and running.

It would be groovy if it involved input_prefilter hook (see
Extensions/jobctrl.py and ext_rescapture.py).

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From danmil at comcast.net  Mon Feb 26 14:33:58 2007
From: danmil at comcast.net (Dan Milstein)
Date: Mon, 26 Feb 2007 13:33:58 -0600
Subject: [IPython-dev] Auto-calling question
In-Reply-To: <46cb515a0702261121g2c59f168y74831e1de3a83f2c@mail.gmail.com>
References: <29FC48C7-1223-4C59-9F65-3E6EB97CDFEB@comcast.net>
	<db6b5ecc0702261043x28c53522ka64c3c8edbc830fa@mail.gmail.com>
	<478DFE6E-2908-4574-9E84-7E94C7629EC1@comcast.net>
	<46cb515a0702261121g2c59f168y74831e1de3a83f2c@mail.gmail.com>
Message-ID: <9262B4B6-8117-4734-972F-2714B19E1623@comcast.net>

One of my goals is very much to make the system play nicer with  
extensions (and, hopefully, to put something in place so you can  
insert your own rewriting phases into the middle of a coherent  
rewrite sequence).

Actually, since you bring that up, I have a question about  
input_prefilter:

As far as I can tell from the code, if you install an input_prefilter  
hook, then, for each line:

  If your hook changes the line at all, then IPython does no more  
transformations (no magic, no autocall, etc)

  If your hook runs but does *not* change the line, then IPython  
*does* run it's usual transformations


a) Is that right?

b) Is that the desired behavior?

I can imagine situations where someone would like to add a modest  
extension to IPython's transformations, but this forces you into an  
all-or-nothing situation.

-D

On Feb 26, 2007, at 1:21 PM, Ville M. Vainio wrote:

> On 2/26/07, Dan Milstein <danmil at comcast.net> wrote:
>
>> I do have an idea for cleaning up all that prefilter code, so I'll
>> see if I can fix that all together once I get the tests up and  
>> running.
>
> It would be groovy if it involved input_prefilter hook (see
> Extensions/jobctrl.py and ext_rescapture.py).
>
> -- 
> Ville M. Vainio - vivainio.googlepages.com
> blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'



From vivainio at gmail.com  Mon Feb 26 15:38:41 2007
From: vivainio at gmail.com (Ville M. Vainio)
Date: Mon, 26 Feb 2007 21:38:41 +0100
Subject: [IPython-dev] Auto-calling question
In-Reply-To: <9262B4B6-8117-4734-972F-2714B19E1623@comcast.net>
References: <29FC48C7-1223-4C59-9F65-3E6EB97CDFEB@comcast.net>
	<db6b5ecc0702261043x28c53522ka64c3c8edbc830fa@mail.gmail.com>
	<478DFE6E-2908-4574-9E84-7E94C7629EC1@comcast.net>
	<46cb515a0702261121g2c59f168y74831e1de3a83f2c@mail.gmail.com>
	<9262B4B6-8117-4734-972F-2714B19E1623@comcast.net>
Message-ID: <46cb515a0702261238m38f44b65xf989bc8d3cae21e3@mail.gmail.com>

On 2/26/07, Dan Milstein <danmil at comcast.net> wrote:

> Actually, since you bring that up, I have a question about
> input_prefilter:
>
> As far as I can tell from the code, if you install an input_prefilter
> hook, then, for each line:
>
>   If your hook changes the line at all, then IPython does no more
> transformations (no magic, no autocall, etc)
>
>   If your hook runs but does *not* change the line, then IPython
> *does* run it's usual transformations
>
>
> a) Is that right?

No, ipapi.TryNext takes optional args that modify what the next hook
gets as an arg.

See ipapi.TryNext and hooks.CommandChainDispatcher (all the normal
hooks end up in CommandChainDispatcher "chains").

Let us know if this is still unclear, with the current lack of example
use. And thank you for taking initiative in this!

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From danmil at comcast.net  Mon Feb 26 16:08:17 2007
From: danmil at comcast.net (Dan Milstein)
Date: Mon, 26 Feb 2007 16:08:17 -0500
Subject: [IPython-dev] Auto-calling question
In-Reply-To: <46cb515a0702261238m38f44b65xf989bc8d3cae21e3@mail.gmail.com>
References: <29FC48C7-1223-4C59-9F65-3E6EB97CDFEB@comcast.net>
	<db6b5ecc0702261043x28c53522ka64c3c8edbc830fa@mail.gmail.com>
	<478DFE6E-2908-4574-9E84-7E94C7629EC1@comcast.net>
	<46cb515a0702261121g2c59f168y74831e1de3a83f2c@mail.gmail.com>
	<9262B4B6-8117-4734-972F-2714B19E1623@comcast.net>
	<46cb515a0702261238m38f44b65xf989bc8d3cae21e3@mail.gmail.com>
Message-ID: <E02AD708-F40A-4CA6-B429-38925590DD26@comcast.net>

I understand the CommandChainDispatcher idea, but it looks like the  
hooks get called in iplib._prefilter (l. 2081), like so:

    rewritten = self.hooks.input_prefilter(stripped)
    if rewritten != stripped: # ok, some prefilter did something
        rewritten = pre + rewritten  # add indentation
        return self.handle_normal(rewritten)
    # Rest of _prefilter() continues here -- checking for %magic,  
alias, etc

So that all hook-installed prefilters will play well together, but  
that, if any of those do anything, then self.handle_normal is called  
and nothing else from the core IPython system happens.  Is there  
something I'm missing?

-D

On Feb 26, 2007, at 3:38 PM, Ville M. Vainio wrote:

> On 2/26/07, Dan Milstein <danmil at comcast.net> wrote:
>
>> Actually, since you bring that up, I have a question about
>> input_prefilter:
>>
>> As far as I can tell from the code, if you install an input_prefilter
>> hook, then, for each line:
>>
>>   If your hook changes the line at all, then IPython does no more
>> transformations (no magic, no autocall, etc)
>>
>>   If your hook runs but does *not* change the line, then IPython
>> *does* run it's usual transformations
>>
>>
>> a) Is that right?
>
> No, ipapi.TryNext takes optional args that modify what the next hook
> gets as an arg.
>
> See ipapi.TryNext and hooks.CommandChainDispatcher (all the normal
> hooks end up in CommandChainDispatcher "chains").
>
> Let us know if this is still unclear, with the current lack of example
> use. And thank you for taking initiative in this!
>
> -- 
> Ville M. Vainio - vivainio.googlepages.com
> blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'



From danmil at comcast.net  Mon Feb 26 16:13:09 2007
From: danmil at comcast.net (Dan Milstein)
Date: Mon, 26 Feb 2007 16:13:09 -0500
Subject: [IPython-dev] Auto-calling question
In-Reply-To: <E02AD708-F40A-4CA6-B429-38925590DD26@comcast.net>
References: <29FC48C7-1223-4C59-9F65-3E6EB97CDFEB@comcast.net>
	<db6b5ecc0702261043x28c53522ka64c3c8edbc830fa@mail.gmail.com>
	<478DFE6E-2908-4574-9E84-7E94C7629EC1@comcast.net>
	<46cb515a0702261121g2c59f168y74831e1de3a83f2c@mail.gmail.com>
	<9262B4B6-8117-4734-972F-2714B19E1623@comcast.net>
	<46cb515a0702261238m38f44b65xf989bc8d3cae21e3@mail.gmail.com>
	<E02AD708-F40A-4CA6-B429-38925590DD26@comcast.net>
Message-ID: <8F368E20-0446-48ED-ACAD-417CFF09AD1B@comcast.net>

Just to follow up: when I say "nothing else from the core IPython  
system happens", I just mean the prefiltering stuff -- expanding  
aliases, interpreting %magic, etc.

-D

On Feb 26, 2007, at 4:08 PM, Dan Milstein wrote:

> I understand the CommandChainDispatcher idea, but it looks like the
> hooks get called in iplib._prefilter (l. 2081), like so:
>
>     rewritten = self.hooks.input_prefilter(stripped)
>     if rewritten != stripped: # ok, some prefilter did something
>         rewritten = pre + rewritten  # add indentation
>         return self.handle_normal(rewritten)
>     # Rest of _prefilter() continues here -- checking for %magic,
> alias, etc
>
> So that all hook-installed prefilters will play well together, but
> that, if any of those do anything, then self.handle_normal is called
> and nothing else from the core IPython system happens.  Is there
> something I'm missing?
>
> -D
>
> On Feb 26, 2007, at 3:38 PM, Ville M. Vainio wrote:
>
>> On 2/26/07, Dan Milstein <danmil at comcast.net> wrote:
>>
>>> Actually, since you bring that up, I have a question about
>>> input_prefilter:
>>>
>>> As far as I can tell from the code, if you install an  
>>> input_prefilter
>>> hook, then, for each line:
>>>
>>>   If your hook changes the line at all, then IPython does no more
>>> transformations (no magic, no autocall, etc)
>>>
>>>   If your hook runs but does *not* change the line, then IPython
>>> *does* run it's usual transformations
>>>
>>>
>>> a) Is that right?
>>
>> No, ipapi.TryNext takes optional args that modify what the next hook
>> gets as an arg.
>>
>> See ipapi.TryNext and hooks.CommandChainDispatcher (all the normal
>> hooks end up in CommandChainDispatcher "chains").
>>
>> Let us know if this is still unclear, with the current lack of  
>> example
>> use. And thank you for taking initiative in this!
>>
>> -- 
>> Ville M. Vainio - vivainio.googlepages.com
>> blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev



From vivainio at gmail.com  Mon Feb 26 16:39:11 2007
From: vivainio at gmail.com (Ville M. Vainio)
Date: Mon, 26 Feb 2007 22:39:11 +0100
Subject: [IPython-dev] Auto-calling question
In-Reply-To: <E02AD708-F40A-4CA6-B429-38925590DD26@comcast.net>
References: <29FC48C7-1223-4C59-9F65-3E6EB97CDFEB@comcast.net>
	<db6b5ecc0702261043x28c53522ka64c3c8edbc830fa@mail.gmail.com>
	<478DFE6E-2908-4574-9E84-7E94C7629EC1@comcast.net>
	<46cb515a0702261121g2c59f168y74831e1de3a83f2c@mail.gmail.com>
	<9262B4B6-8117-4734-972F-2714B19E1623@comcast.net>
	<46cb515a0702261238m38f44b65xf989bc8d3cae21e3@mail.gmail.com>
	<E02AD708-F40A-4CA6-B429-38925590DD26@comcast.net>
Message-ID: <46cb515a0702261339m672784efm7ebf4c589a60ddf8@mail.gmail.com>

On 2/26/07, Dan Milstein <danmil at comcast.net> wrote:

> So that all hook-installed prefilters will play well together, but
> that, if any of those do anything, then self.handle_normal is called
> and nothing else from the core IPython system happens.  Is there
> something I'm missing?

No, you are not missing anything. The magics and other "system" stuff
are currently not implemented through the hook system, and that's one
of the reasons I hoped your cleanup scheme would "harmonize" the
'built-in' prefilters (actually, I'm not sure I even like the idea of
built in prefilter too much) with the hooks system. In the meantime,
you can easily convert %foo to _ip.magic("foo") if you want, to get
the magics working.

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From mmetz at astro.uni-bonn.de  Tue Feb 27 07:55:40 2007
From: mmetz at astro.uni-bonn.de (Manuel Metz)
Date: Tue, 27 Feb 2007 13:55:40 +0100
Subject: [IPython-dev] gtk - Warning
Message-ID: <45E42A4C.8010701@astro.uni-bonn.de>

Hi,
since today after an update on Ubuntu/feisty I get the following warning 
when starting ipython:

/usr/lib/python2.5/site-packages/IPython/Shell.py:664: 
GtkDeprecationWarning: gtk.threads_leave is deprecated, use 
gtk.gdk.threads_leave instead
   self.gtk.threads_leave()

Should be trivial to fix since solution is already given ;-)

Manuel


From fperez.net at gmail.com  Tue Feb 27 10:50:48 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 27 Feb 2007 08:50:48 -0700
Subject: [IPython-dev] gtk - Warning
In-Reply-To: <45E42A4C.8010701@astro.uni-bonn.de>
References: <45E42A4C.8010701@astro.uni-bonn.de>
Message-ID: <db6b5ecc0702270750u3574afcdk25d1ac5b1c306467@mail.gmail.com>

On 2/27/07, Manuel Metz <mmetz at astro.uni-bonn.de> wrote:
> Hi,
> since today after an update on Ubuntu/feisty I get the following warning
> when starting ipython:
>
> /usr/lib/python2.5/site-packages/IPython/Shell.py:664:
> GtkDeprecationWarning: gtk.threads_leave is deprecated, use
> gtk.gdk.threads_leave instead
>    self.gtk.threads_leave()

I just committed a fix in SVN, thanks for the note (there were a few
more similar warnings after fixing that one).

It would be very good if people on older GTK setups could report of
any problems, I don't know the history of their API in detail, so
there's a chance the new call form may break.  If it does, please let
me know what the value of

gtk.gtk_version

is on your system, so I can protect the new call with version number checks.

Cheers,

f


From danmil at comcast.net  Wed Feb 28 14:17:27 2007
From: danmil at comcast.net (Dan Milstein)
Date: Wed, 28 Feb 2007 14:17:27 -0500
Subject: [IPython-dev] Tests for prefilter
Message-ID: <855AA20F-C120-4D45-9795-40B732C15FF9@comcast.net>

ipython-folk,

Okay, as promised, here's a script which runs fairly exhaustive tests  
on the input prefiltering.

Comments in the code, but some highlights:

  - Run it as python test_prefilter.py (i.e. *not* using ipython)

  - It's quiet by default, but you can pass in -v to get dots

  - I'm only testing to make sure the right handle_X methods get  
called.  I am *not* testing those methods at all.  In fact, I swap  
them out as part of the tests.

  - However, that leaves plenty to test ;-).  Basically, it's making  
sure that the right combination of %options, namespace contents and  
input patterns trigger the right expansions.  The gorgeous mess that  
is _prefilter().  136 tests total.

  - I found a bunch of things which look like bugs, which I'm listing  
below.  I am *not* having the tests complain about those yet.  I'm  
wanting it to quietly succeed so that I can use it to test a rewrite  
of prefiltering.  If you'd like the failing tests turned back on, I  
can do that pretty easily.  (I left XXX comments in the code where  
tests can/should be added back in).

Bugs / Questions
================

  1) Even with %autocall off, '/', ',' and ';' should trigger  
autocalls. Does not work. (I asked about this Monday)

  2) In several places, the prefilter checks for an input line which  
looks like, e.g.:

thing = rhs

And, if it finds it, doesn't try to look up 'thing' as an alias, etc,  
so that a normal python expr won't get shadowed by ipython magic  
bits.  However, in some places, the set of characters which turn off  
the magic bits are: '!=()', and in others, they are '!=()<>'. I  
*think* they should be the same in both places.  See, e.g. line 2121  
v line 2135.  Or, just possibly it should be all python binary ops,  
which is a much bigger list.

  3) Can aliases have '.' in their names?  If so, there's a problem:  
such aliases *do* expand with %autocall off, but they *don't* expand  
when it's on (because of a subtlety of how ._ofind is used or avoided).

  4) More autocall fun -- what should ipython do (with autocall on),  
with the two following (the comments are my understanding of how it's  
supposed to work currently):

 > "abc".join range(4)     # Should *not* autocall and doesn't

 > /"abc".join range(4)    #2 *Should* autocall, but doesn't.

Currently, #1 should *not* autocall the join method, because autocall  
only triggers for things which look like identifiers mixed with '.'.   
Is that, in fact, how the system should work?

However, #2 also doesn't autocall the join (and it should, I think).   
In fact, it totally blows up -- ipython somehow ends up with a '//'  
at the start of the line and has no idea what to do.  I think I know  
why this is happening.


  5) Binary ops and autocall

Autocalling gets turned off for *most* binary operators, so that,  
e.g. 'fun in lst' won't become 'fun(in list)', even if fun is  
callable.   However, it's missing the % operator.  So that, e.g. 'fun  
% s' will become 'fun(%s)'.


That's what I've got...
-Dan

-------------- next part --------------
A non-text attachment was scrubbed...
Name: test_prefilter.py
Type: text/x-python-script
Size: 12937 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20070228/82369e0c/attachment.bin>

From novak at ucolick.org  Wed Feb 28 15:53:44 2007
From: novak at ucolick.org (Greg Novak)
Date: Wed, 28 Feb 2007 12:53:44 -0800 (PST)
Subject: [IPython-dev] pydoc and introspective features
In-Reply-To: <db6b5ecc0702251400w5505472te85ca5a0d86dee75@mail.gmail.com>
References: <mailman.14355.1171777712.8339.ipython-dev@scipy.org>
	<45D7FBEA.4080702@gmail.com>
	<db6b5ecc0702251400w5505472te85ca5a0d86dee75@mail.gmail.com>
Message-ID: <20070228.125344.126718244.novak@ucolick.org>

"Fernando Perez" <fperez.net at gmail.com> wrote:
> - search.  This is probably the biggest gripe everyone has with python
> vs. commercial interactive systems (such as Matlab or Mathematica).
> Tab-completion and 'foo?' work great, but if you don't even know where
> to begin looking for something, you're stuck.    A builtin indexing
> system that could be either exposed via a web browser or to a
> command-line program (such as ipython) would be very welcome by a lot
> of users.

While we're on the subject, I humbly submit my slow-as-a-slug but
fairly general code to recursively search python objects looking for
things.  

It looks inside modules, dicts, tuples, lists, and instances looking
for things based on name, value, or docstring.  It's also pretty easy
to extend it either to look inside different objects or else have a
different definition of a 'match.'  

It returns a list of strings that tell you how to get to the thing you
want.  A typical call would be:

aproposName('needle', compoundObject)

returns:
['arg[foo].bar[3]']
Ie: "There's something named 'needle' in the third element of the
attribute named bar of the object with dict key foo in the object
passed as the argument."

I've posted this before--this version fixes major problems (ie, some
things I thought worked didn't work in the previously posted version).

I've also attached test code.  

This is probably more useful as food for thought than for anything
practical.  On the other hand it solves a somewhat more general
problem, being able to look inside live object as opposed to searching
only doc strings.  

Greg
-------------- next part --------------
import unittest;

import apropos as aproposModule
from apropos import *

class AproposTest(unittest.TestCase):
    # Untested functions, but I think it's ok that way:
    # _apropos  apropos

    def testAproposName(self):
        class Composite:
            def __init__(self):
                self.a = 1
                self.foo = 'bar'
                self.b = 3
        self.assertEqual(aproposName('foo', [1,'foo',2]),
                         [])
        self.assertEqual(aproposName('foo', (1,'foo',3)),
                         [])
        self.assertEqual(aproposName('foo', dict(a=1,foo='bar',b=3)),
                         ['arg[foo]'])
        self.assertEqual(aproposName('foo', Composite()),
                         ['arg.foo'])

        lst = aproposName('aproposName', aproposModule, exclude='_')
        self.assertTrue('apropos.aproposName' in lst)
        self.assertTrue('apropos.aproposNameRegexp' in lst)
        self.assertFalse('apropos.__builtins__[_ip].user_ns[aproposName]'
                         in lst)

        self.assertEqual(aproposName('foo', Composite(), name='name'),
                         ['name.foo'])

    def testMaxDepth(self):
        lst = aproposName('foo', dict(foo=dict(foo=1, bar=2), b=3),
                          maxDepth=0)
        self.assertFalse('arg][foo][foo]' in lst)
        self.assertFalse('arg][foo]' in lst)

        lst = aproposName('foo', dict(foo=dict(foo=1, bar=2), b=3),
                          maxDepth=1)
        self.assertFalse('arg[foo][foo]' in lst)
        self.assertTrue('arg[foo]' in lst)

        lst = aproposName('foo', dict(foo=dict(foo=1, bar=2), b=3),
                          maxDepth=2)
        self.assertTrue('arg[foo][foo]' in lst)
        self.assertTrue('arg[foo]' in lst)

        lst = aproposName('foo', dict(foo=dict(foo=1, bar=2), b=3))
        self.assertTrue('arg[foo][foo]' in lst)
        self.assertTrue('arg[foo]' in lst)

    # FIXME -- Sometimes causes bus error?
    def disable_testModuleSearch(self):
        # Sequester the long-running test.
        lst = aproposName('aproposName', aproposModule)
        self.assertTrue('apropos.aproposName' in lst)
        self.assertTrue('apropos.aproposNameRegexp' in lst)
        self.assertTrue('apropos.__builtins__[_ip].user_ns[aproposName]'
                        in lst)

    def testSyntax(self):
        """Functionality has been tested... just make sure that these
        functions can be called"""
        class Composite:
            def __init__(self, str):
                self.__doc__ = str

        self.assertEqual(aproposValue('foo', dict(a=1, bar='foo')),
                         ['arg[bar]'])
        self.assertEqual(aproposDoc('foo', Composite('foo')),
                         ['arg'])
        self.assertEqual(aproposNameRegexp ('^foo', dict(foo=1, barfoo=2)),
                         ['arg[foo]'])
        self.assertEqual(aproposValueRegexp ('^foo', dict(bar='foo',
                                                          the='afoo')),
                         ['arg[bar]'])
        self.assertEqual(aproposDocRegexp ('^foo', Composite('foo')),
                         ['arg'])
        self.assertEqual(aproposDocRegexp ('^foo', Composite('theFoo')),
                         [])
            
    def testNullIntrospector(self):
        i = NullIntrospector()
        # I think this is how this is supposed to work
        self.assertEqual(id(i), id(i.__iter__()))
        self.assertRaises(StopIteration, i.next)

        # make sure code doens't freak out
        i = NullIntrospector(exclude='_')

    def testListIntrospector(self):
        i = ListIntrospector([1,2])
        self.assertEqual(id(i), id(i.__iter__()))
        self.assertEqual(i.next(), (1, None, '[0]'))
        self.assertEqual(i.next(), (2, None, '[1]'))
        self.assertRaises(StopIteration, i.next)

        # make sure code doens't freak out
        i = ListIntrospector([1,2], exclude='_')

    def testInstanceIntrospector(self):
        class Composite:
            pass

        c = Composite()
        c.a = 1
        c.b = 2

        lst = [el for el in InstanceIntrospector(c)]
        # depending on how I'm running the test, one or the other of
        # these should be in the list
        self.assertTrue(('test_apropos', '__module__', '.__module__') in lst
                        or ('__builtin__', '__module__', '.__module__') in lst)
        self.assertTrue((None, '__doc__', '.__doc__') in lst)
        self.assertTrue((1, 'a', '.a') in lst)
        self.assertTrue((2, 'b', '.b') in lst)
        self.assertEqual(len(lst), 4)

        lst = [el for el in InstanceIntrospector(c, exclude='_')]
        self.assertFalse(() in lst)
        self.assertFalse((None, None, '.__doc__') in lst)
        self.assertEqual(len(lst), 2)

    def testDictIntrospector(self):
        lst = [el for el in DictIntrospector(dict(a=1,_b=2))]

        self.assertEqual(len(lst), 2)
        self.assertTrue((1, 'a', '[a]') in lst)
        self.assertTrue((2, '_b', '[_b]') in lst)

        lst = [el for el in DictIntrospector(dict(a=1,_b=2), exclude='_')]
        self.assertEqual(len(lst), 1)
        self.assertTrue((1, 'a', '[a]') in lst)
        self.assertFalse((2, '_b', '[_b]') in lst)            

    def testSearchName(self):
        self.assertTrue(searchName('needle', 'the needle', None))
        self.assertTrue(searchName('needle', 'needle more', None))
        self.assertTrue(searchName('needle', 'the needle more', None))

        # Make sure function doesn't freak out for no name
        self.assertFalse(searchName('needle', None, None))
        
    def testSearchValue(self):
        class Composite:
            def __init__(self, str):
                self._str = str
            def __repr__(self):
                return self._str
            def __str__(self):
                return self._str
            
        self.assertTrue(searchValue('needle', None,
                                    Composite('the needle')))
        self.assertTrue(searchValue('needle', None,
                                    Composite('needle more')))
        self.assertTrue(searchValue('needle', None,
                                    Composite('the needle more')))
        # These are not true because searchValue doens't split
        # apart built-in containers
        self.assertFalse(searchValue('needle', None,
                                    ['needle', 2, 3]))
        self.assertFalse(searchValue('needle', None,
                                    ('needle', 2, 3)))
        self.assertFalse(searchValue('needle', None,
                                    dict(a='needle', b=2, c=3)))

        
    def testSearchDoc(self):   
        class Composite:
            def __init__(self, str):
                self.__doc__ = str

        self.assertTrue(searchDoc('needle', None,
                                  Composite('the needle')))
        self.assertTrue(searchDoc('needle', None,
                                  Composite('needle more')))
        self.assertTrue(searchDoc('needle', None,
                                  Composite('the needle more')))

        # Make sure search fn doesn't freak out
        self.assertFalse(searchDoc('needle', None,
                                   Composite(None)))

        
    def testSearchNameRegexp(self):  
        self.assertFalse(searchNameRegexp('^needle', 'the needle', None))
        self.assertTrue(searchNameRegexp('^needle', 'needle more', None))
        self.assertFalse(searchNameRegexp('^needle', 'the needle more', None))

        # Make sure function doesn't freak out for no name
        self.assertFalse(searchName('^needle', None, None))

    def testSearchValueRegexp(self): 
        class Composite:
            def __init__(self, str):
                self._str = str
            def __repr__(self):
                return self._str
            def __str__(self):
                return self._str
            
        self.assertFalse(searchValueRegexp('^needle', None,
                                           Composite('the needle')))
        self.assertTrue(searchValueRegexp('^needle', None,
                                          Composite('needle more')))
        self.assertFalse(searchValueRegexp('^needle', None,
                                           Composite('the needle more')))

        # Make sure we don't search inside containers
        self.assertFalse(searchValueRegexp('needle', None,
                                           ['needle', 2, 3]))
        self.assertFalse(searchValueRegexp('needle', None,
                                           ('needle', 2, 3)))
        self.assertFalse(searchValueRegexp('needle', None,
                                           dict(a='needle', b=2, c=3)))

    def testSearchDocRegexp(self):   
        class Composite:
            def __init__(self, str):
                self.__doc__ = str

        self.assertFalse(searchDocRegexp('^needle', None,
                                         Composite('the needle')))
        self.assertTrue(searchDocRegexp('^needle', None,
                                        Composite('needle more')))
        self.assertFalse(searchDocRegexp('^needle', None,
                                         Composite('the needle more')))

        # Make sure function doesn't freak out if no doc
        self.assertFalse(searchDocRegexp('^needle', None,
                                         Composite(None)))
        
def suite():
    suites = [unittest.TestLoader().loadTestsFromTestCase(test)
              for test in (AproposTest,)]
    return unittest.TestSuite(suites)

def test():
    unittest.TextTestRunner().run(suite())

def itest():
    suite().debug()
-------------- next part --------------
import types
import re

__version__ = 0.2
__author__ = "Greg Novak <novak at ucolick.org"
# Date: January 14, 2007
# Code released public domain.  Do whatever you want with it.

# You can add your own types to these lists if you want apropos to
# descend into them.  If you have a container that you want apropos to
# search, but it doesn't respond appropriately to the methods listed
# below, you can give it a function called __apropos__.  This function
# takes no arguments and should return an iterator.  The iterator
# should return the contents of the object, as tuples of
# (elementObject, nameString, accessString)

# Must respond to __iter__ and [string].  Designed for things you
# access via [string]
dictTypes = [types.DictType]
# Must respond to __iter__().  Designed for things you access via
# [int]
listTypes = [types.ListType, types.TupleType]
# Must give sensible results to dir(), getattr().  Designed for things
# you access via .
instanceTypes = [types.InstanceType, types.ModuleType]

##################################################
## Interface

## Common Usage
def aproposName(needle, haystack=None, **kw):
    """Recursively search for attributes with where needle is a
    substring of the name.  See apropos() for addtional keyword
    arguments.  Typical usage is aproposName('string', module).

    Return a list of strings showing the path to reach the matching
    object"""
    return apropos(needle, haystack, searchFn=searchName, **kw)

def aproposValue(needle, haystack=None, **kw):
    """Recursively search for attributes with where needle is a
    substring the string representation of the object.  See apropos()
    for addtional keyword arguments.  Typical usage is
    aproposValue('string', module).

    Return a list of strings showing the path to reach the matching
    object"""
    return apropos(needle, haystack, searchFn=searchValue, **kw)

def aproposDoc(needle, haystack=None, **kw):
    """Recursively search for attributes with where needle is a
    substring of the documentation string of the object.  See
    apropos() for addtional keyword arguments.  Typical usage is
    aproposDoc('string', module).

    Return a list of strings showing the path to reach the matching
    object"""
    return apropos(needle, haystack, searchFn=searchDoc, **kw)

def aproposNameRegexp (needle, haystack=None, **kw):
    """Recursively search for attributes with where needle is a regexp
    matching the name.  See apropos() for addtional keyword arguments.
    Typical usage is aproposNameRegexp('string', module).

    Return a list of strings showing the path to reach the matching
    object"""
    return apropos(needle, haystack, searchFn=searchNameRegexp, **kw)

def aproposValueRegexp(needle, haystack=None, **kw):
    """Recursively search for attributes with where needle is a regexp
    matching the string representation of the object.  See apropos()
    for addtional keyword arguments.  Typical usage is
    aproposValueRegexp('string', module).

    Return a list of strings showing the path to reach the matching
    object"""
    return apropos(needle, haystack, searchFn=searchValueRegexp, **kw)

def aproposDocRegexp(needle, haystack=None, **kw):
    """Recursively search for attributes with where needle is a regexp
    matching the docstring of the object.  See apropos() for addtional
    keyword arguments.  Typical usage is aproposDocRegexp('string',
    module).

    Return a list of strings showing the path to reach the matching
    object"""
    return apropos(needle, haystack, searchFn=searchDocRegexp, **kw)

## Handles default values of arguments
def apropos(needle, haystack=None, name=None,
            searchFn=None, **kw):
    """Recursively search through haystack looking for needle.
    Typical usage is apropos('string', module).
    
    haystack can be any python object.  Typically it's a module.  If
    it's not given, it's the dict returned by globals() (ie, watch
    out, it's going to take a while).

    name is the name of the top level object.  It's first bit of the
    'accessor' strings that are returned.  If not specified, defaults
    to 'arg'.
    
    Matches determined by searchFn.  searchFn(needle, name, obj)
    returns true if the object should be considered a match.  By
    default, searchFn matches if needle is a substring of the name of
    the object.

    Return a list of strings showing the path to reach the matching
    object"""
    if haystack is None:
        haystack = globals()
        name = ''
    elif name is None:
        if hasattr(haystack, "__name__"):
            name = haystack.__name__
        else:
            name = 'arg'
    
    if searchFn is None: searchFn = searchName

    return _apropos(needle, haystack, name, searchFn, **kw)

##################################################
## Common search functions

def searchName(needle, name, obj):
    return name and needle in name    

def searchValue(needle, name, obj):
    # String representation of dicts, lists, and tuples includes the
    # objects within them, so don't consider that to be a match on the
    # desired value.  Wait to get inside the container class...
    #
    # TODO What I really want to do is match the container if none of
    # its contents matched.
    if type(obj) not in (types.TupleType, types.ListType,
                         types.DictType):
        return needle in str(obj)
# NOTE -- should be repr()?

def searchDoc(needle, name, obj):
    return hasattr(obj, '__doc__') and obj.__doc__ \
           and needle in obj.__doc__
    
def searchNameRegexp(needle, name, obj):
    return name and re.search(needle, name)

def searchValueRegexp(needle, name, obj):
    if type(obj) not in (types.TupleType, types.ListType,
                         types.DictType):
        return re.search(needle, str(obj))

def searchDocRegexp(needle, name, obj):
    return hasattr(obj, '__doc__') \
           and obj.__doc__ \
           and re.search(needle, obj.__doc__)

##################################################
## The guts

def _apropos(needle, haystack, haystackName,
             searchFn, maxDepth=None, **kw):
    """Recursively search through haystack looking for needle.

    haystack can be any python object.  Typically it's a module.  If
    it's not given, it's the dict returned by globals() (ie, watch
    out, it's going to take a while).
    
    Matches determined by searchFn.  searchFn(needle, name, obj)
    returns true if the object should be considered a match.  By
    default, searchFn matches if needle is a substring of the name of
    the object.  

    name is the name of the top level object.  It's first bit of the
    'accessor' strings that are returned.  If not specified, defaults
    to 'arg'.

    Return a list of strings showing the path to reach the matching
    object."""
    def search(haystack, haystackName, fullName, depth):
        '''Free variable: needle, searchTypes'''
        # print "Searched", len(searchedIds), "Searching", depth, fullName
        if searchFn(needle, haystackName, haystack):
            found.append(fullName)

        # break apart if obj is not already searched
        if type(haystack) in searchTypes \
                and (not maxDepth or depth < maxDepth) \
                and id(haystack) not in searchedIds:
            # Prevent loops with circular references by setting this
            # _before_ descending into sub-objects
            searchedIds.append(id(haystack))

            for hay, hayName, hayAccess in introspect(haystack, **kw):
                search(hay, hayName, fullName + hayAccess, depth+1)

    searchedIds = []
    found = []
    searchTypes = dictTypes + listTypes + instanceTypes

    search(haystack, haystackName, haystackName, 0)
    return found

def introspect(obj, **kw):
    if type(obj) in dictTypes:
        return DictIntrospector(obj, **kw)
    if type(obj) in listTypes:
        return ListIntrospector(obj, **kw)
    if type(obj) in instanceTypes:
        return InstanceIntrospector(obj, **kw)

    # User objects
    if hasattr(obj, '__apropos__'):
        return obj.__apropos__(**kw)

    # Stymied
    print "apropos.py: Warning, don't know how to deal with " + str(obj)
    return NullIntrospector()

# NOTE These introspectors simplify the code, but they seem to take about five
# times as long, very unfortunately.
class Introspector (object):
    def __iter__(self):
        return self

    def next(self):
        pass

class NullIntrospector (Introspector):
    def __init__(self, **kw):
        pass

    def next(self):
        raise StopIteration

class DictIntrospector (Introspector):
    # types that respond to __iter__, obj.[key] to get a value
    def __init__(self, dict, exclude=None):
        self.dict = dict
        self.iter = self.dict.__iter__()        
        self.exclude = exclude
        
    def next(self):
        # return tuple of obj, name, accessName
        k = self.iter.next()
        # FIXME -- completely skip non-string key entries
        while type(k) is not types.StringType \
              or (self.exclude and k.startswith(self.exclude)):
            k = self.iter.next()
        return self.dict[k], k, '[' + k + ']'

class ListIntrospector (Introspector):
    # types that respond to __iter__
    def __init__(self, list, exclude=None):
        self.list = list
        self.iter = self.list.__iter__()
        self.i = 0

    def next(self):
        # return tuple of obj, name, accessName
        self.i += 1
        return self.iter.next(), None, '[' + str(self.i-1) + ']'

class InstanceIntrospector (Introspector):
    # classes that respond to dir and getattr
    def __init__(self, inst, exclude=None):
        self.inst = inst
        self.iter = dir(self.inst).__iter__()
        self.exclude = exclude

    def next(self):
        # return tuple of obj, name, accessName

        # IPython structs allow non-string attributes.  Filter them
        # out because they cause problems.  That is, you have to
        # access them via obj[1], not getattr(obj, 1) or
        # getattr(obj, '1')    
        # FIXME -- filter out non-string things that appear in dir()

        name = self.iter.next()
        while type(name) is not types.StringType \
              or (self.exclude and name.startswith(self.exclude)):
            name = self.iter.next()
        return getattr(self.inst, name), name, "." + name


From ellisonbg.net at gmail.com  Wed Feb 28 17:36:43 2007
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 28 Feb 2007 15:36:43 -0700
Subject: [IPython-dev] Tests for prefilter
In-Reply-To: <855AA20F-C120-4D45-9795-40B732C15FF9@comcast.net>
References: <855AA20F-C120-4D45-9795-40B732C15FF9@comcast.net>
Message-ID: <6ce0ac130702281436v74697bedwd0a53e3d05f210f5@mail.gmail.com>

Dan,

I haven't had a chance to look at the attachment yet, but it is great
you've started looking at this.

I have been sick since getting back to Boulder on Monday, so I haven't
been checking email.  I will probably feel good enough to work
tomorrow - I'll send you a longer reply then.

Cheers,

Brian

On 2/28/07, Dan Milstein <danmil at comcast.net> wrote:
> ipython-folk,
>
> Okay, as promised, here's a script which runs fairly exhaustive tests
> on the input prefiltering.
>
> Comments in the code, but some highlights:
>
>   - Run it as python test_prefilter.py (i.e. *not* using ipython)
>
>   - It's quiet by default, but you can pass in -v to get dots
>
>   - I'm only testing to make sure the right handle_X methods get
> called.  I am *not* testing those methods at all.  In fact, I swap
> them out as part of the tests.
>
>   - However, that leaves plenty to test ;-).  Basically, it's making
> sure that the right combination of %options, namespace contents and
> input patterns trigger the right expansions.  The gorgeous mess that
> is _prefilter().  136 tests total.
>
>   - I found a bunch of things which look like bugs, which I'm listing
> below.  I am *not* having the tests complain about those yet.  I'm
> wanting it to quietly succeed so that I can use it to test a rewrite
> of prefiltering.  If you'd like the failing tests turned back on, I
> can do that pretty easily.  (I left XXX comments in the code where
> tests can/should be added back in).
>
> Bugs / Questions
> ================
>
>   1) Even with %autocall off, '/', ',' and ';' should trigger
> autocalls. Does not work. (I asked about this Monday)
>
>   2) In several places, the prefilter checks for an input line which
> looks like, e.g.:
>
> thing = rhs
>
> And, if it finds it, doesn't try to look up 'thing' as an alias, etc,
> so that a normal python expr won't get shadowed by ipython magic
> bits.  However, in some places, the set of characters which turn off
> the magic bits are: '!=()', and in others, they are '!=()<>'. I
> *think* they should be the same in both places.  See, e.g. line 2121
> v line 2135.  Or, just possibly it should be all python binary ops,
> which is a much bigger list.
>
>   3) Can aliases have '.' in their names?  If so, there's a problem:
> such aliases *do* expand with %autocall off, but they *don't* expand
> when it's on (because of a subtlety of how ._ofind is used or avoided).
>
>   4) More autocall fun -- what should ipython do (with autocall on),
> with the two following (the comments are my understanding of how it's
> supposed to work currently):
>
>  > "abc".join range(4)     # Should *not* autocall and doesn't
>
>  > /"abc".join range(4)    #2 *Should* autocall, but doesn't.
>
> Currently, #1 should *not* autocall the join method, because autocall
> only triggers for things which look like identifiers mixed with '.'.
> Is that, in fact, how the system should work?
>
> However, #2 also doesn't autocall the join (and it should, I think).
> In fact, it totally blows up -- ipython somehow ends up with a '//'
> at the start of the line and has no idea what to do.  I think I know
> why this is happening.
>
>
>   5) Binary ops and autocall
>
> Autocalling gets turned off for *most* binary operators, so that,
> e.g. 'fun in lst' won't become 'fun(in list)', even if fun is
> callable.   However, it's missing the % operator.  So that, e.g. 'fun
> % s' will become 'fun(%s)'.
>
>
> That's what I've got...
> -Dan
>
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
>
>
>


From fperez.net at gmail.com  Wed Feb 28 17:41:07 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 28 Feb 2007 15:41:07 -0700
Subject: [IPython-dev] Tests for prefilter
In-Reply-To: <6ce0ac130702281436v74697bedwd0a53e3d05f210f5@mail.gmail.com>
References: <855AA20F-C120-4D45-9795-40B732C15FF9@comcast.net>
	<6ce0ac130702281436v74697bedwd0a53e3d05f210f5@mail.gmail.com>
Message-ID: <db6b5ecc0702281441g7954b7b9v87b7c4e48e3a6398@mail.gmail.com>

On 2/28/07, Brian Granger <ellisonbg.net at gmail.com> wrote:
> Dan,
>
> I haven't had a chance to look at the attachment yet, but it is great
> you've started looking at this.
>
> I have been sick since getting back to Boulder on Monday, so I haven't
> been checking email.  I will probably feel good enough to work
> tomorrow - I'll send you a longer reply then.
>

Thanks, Brian.  Between the win32/terminal stuff and the scipy book
stuff, my allotted daily quota for free software has been already
exceeded, I'm afraid :)



Cheers,

f


From lgautier at gmail.com  Wed Feb 28 22:20:58 2007
From: lgautier at gmail.com (Laurent Gautier)
Date: Thu, 1 Mar 2007 11:20:58 +0800
Subject: [IPython-dev] pydoc and introspective features
In-Reply-To: <20070228.125344.126718244.novak@ucolick.org>
References: <mailman.14355.1171777712.8339.ipython-dev@scipy.org>
	<45D7FBEA.4080702@gmail.com>
	<db6b5ecc0702251400w5505472te85ca5a0d86dee75@mail.gmail.com>
	<20070228.125344.126718244.novak@ucolick.org>
Message-ID: <27d1e6020702281920k46db643fmceb30037bcac00a5@mail.gmail.com>

Nice. Thanks.

We already have a recursive data structure for objects, and implemented
search on both the pydoc string and on the elements of an object:
http://pydoc-r.svn.sourceforge.net/viewvc/pydoc-r/branches/pydoc_dag/pydoc/search.py?view=markup

I am very pleased to see that this is sort of design is suggested as well !
We do seem to have a similiar approach in spirit (we have different classes
for different object types), with the extra twist that we are trying
to model both
object sitting on the disk (in unloaded modules/python files) and
'live' objects
(that is 'in memory') under a unified structure.

I will look into getting in all the search goodies you propose.

Thanks again,

Laurent





2007/3/1, Greg Novak <novak at ucolick.org>:
> "Fernando Perez" <fperez.net at gmail.com> wrote:
> > - search.  This is probably the biggest gripe everyone has with python
> > vs. commercial interactive systems (such as Matlab or Mathematica).
> > Tab-completion and 'foo?' work great, but if you don't even know where
> > to begin looking for something, you're stuck.    A builtin indexing
> > system that could be either exposed via a web browser or to a
> > command-line program (such as ipython) would be very welcome by a lot
> > of users.
>
> While we're on the subject, I humbly submit my slow-as-a-slug but
> fairly general code to recursively search python objects looking for
> things.
>
> It looks inside modules, dicts, tuples, lists, and instances looking
> for things based on name, value, or docstring.  It's also pretty easy
> to extend it either to look inside different objects or else have a
> different definition of a 'match.'
>
> It returns a list of strings that tell you how to get to the thing you
> want.  A typical call would be:
>
> aproposName('needle', compoundObject)
>
> returns:
> ['arg[foo].bar[3]']
> Ie: "There's something named 'needle' in the third element of the
> attribute named bar of the object with dict key foo in the object
> passed as the argument."
>
> I've posted this before--this version fixes major problems (ie, some
> things I thought worked didn't work in the previously posted version).
>
> I've also attached test code.
>
> This is probably more useful as food for thought than for anything
> practical.  On the other hand it solves a somewhat more general
> problem, being able to look inside live object as opposed to searching
> only doc strings.
>
> Greg
>
> import unittest;
>
> import apropos as aproposModule
> from apropos import *
>
> class AproposTest(unittest.TestCase):
>     # Untested functions, but I think it's ok that way:
>     # _apropos  apropos
>
>     def testAproposName(self):
>         class Composite:
>             def __init__(self):
>                 self.a = 1
>                 self.foo = 'bar'
>                 self.b = 3
>         self.assertEqual(aproposName('foo', [1,'foo',2]),
>                          [])
>         self.assertEqual(aproposName('foo', (1,'foo',3)),
>                          [])
>         self.assertEqual(aproposName('foo', dict(a=1,foo='bar',b=3)),
>                          ['arg[foo]'])
>         self.assertEqual(aproposName('foo', Composite()),
>                          ['arg.foo'])
>
>         lst = aproposName('aproposName', aproposModule, exclude='_')
>         self.assertTrue('apropos.aproposName' in lst)
>         self.assertTrue('apropos.aproposNameRegexp' in lst)
>         self.assertFalse('apropos.__builtins__[_ip].user_ns[aproposName]'
>                          in lst)
>
>         self.assertEqual(aproposName('foo', Composite(), name='name'),
>                          ['name.foo'])
>
>     def testMaxDepth(self):
>         lst = aproposName('foo', dict(foo=dict(foo=1, bar=2), b=3),
>                           maxDepth=0)
>         self.assertFalse('arg][foo][foo]' in lst)
>         self.assertFalse('arg][foo]' in lst)
>
>         lst = aproposName('foo', dict(foo=dict(foo=1, bar=2), b=3),
>                           maxDepth=1)
>         self.assertFalse('arg[foo][foo]' in lst)
>         self.assertTrue('arg[foo]' in lst)
>
>         lst = aproposName('foo', dict(foo=dict(foo=1, bar=2), b=3),
>                           maxDepth=2)
>         self.assertTrue('arg[foo][foo]' in lst)
>         self.assertTrue('arg[foo]' in lst)
>
>         lst = aproposName('foo', dict(foo=dict(foo=1, bar=2), b=3))
>         self.assertTrue('arg[foo][foo]' in lst)
>         self.assertTrue('arg[foo]' in lst)
>
>     # FIXME -- Sometimes causes bus error?
>     def disable_testModuleSearch(self):
>         # Sequester the long-running test.
>         lst = aproposName('aproposName', aproposModule)
>         self.assertTrue('apropos.aproposName' in lst)
>         self.assertTrue('apropos.aproposNameRegexp' in lst)
>         self.assertTrue('apropos.__builtins__[_ip].user_ns[aproposName]'
>                         in lst)
>
>     def testSyntax(self):
>         """Functionality has been tested... just make sure that these
>         functions can be called"""
>         class Composite:
>             def __init__(self, str):
>                 self.__doc__ = str
>
>         self.assertEqual(aproposValue('foo', dict(a=1, bar='foo')),
>                          ['arg[bar]'])
>         self.assertEqual(aproposDoc('foo', Composite('foo')),
>                          ['arg'])
>         self.assertEqual(aproposNameRegexp ('^foo', dict(foo=1, barfoo=2)),
>                          ['arg[foo]'])
>         self.assertEqual(aproposValueRegexp ('^foo', dict(bar='foo',
>                                                           the='afoo')),
>                          ['arg[bar]'])
>         self.assertEqual(aproposDocRegexp ('^foo', Composite('foo')),
>                          ['arg'])
>         self.assertEqual(aproposDocRegexp ('^foo', Composite('theFoo')),
>                          [])
>
>     def testNullIntrospector(self):
>         i = NullIntrospector()
>         # I think this is how this is supposed to work
>         self.assertEqual(id(i), id(i.__iter__()))
>         self.assertRaises(StopIteration, i.next)
>
>         # make sure code doens't freak out
>         i = NullIntrospector(exclude='_')
>
>     def testListIntrospector(self):
>         i = ListIntrospector([1,2])
>         self.assertEqual(id(i), id(i.__iter__()))
>         self.assertEqual(i.next(), (1, None, '[0]'))
>         self.assertEqual(i.next(), (2, None, '[1]'))
>         self.assertRaises(StopIteration, i.next)
>
>         # make sure code doens't freak out
>         i = ListIntrospector([1,2], exclude='_')
>
>     def testInstanceIntrospector(self):
>         class Composite:
>             pass
>
>         c = Composite()
>         c.a = 1
>         c.b = 2
>
>         lst = [el for el in InstanceIntrospector(c)]
>         # depending on how I'm running the test, one or the other of
>         # these should be in the list
>         self.assertTrue(('test_apropos', '__module__', '.__module__') in lst
>                         or ('__builtin__', '__module__', '.__module__') in lst)
>         self.assertTrue((None, '__doc__', '.__doc__') in lst)
>         self.assertTrue((1, 'a', '.a') in lst)
>         self.assertTrue((2, 'b', '.b') in lst)
>         self.assertEqual(len(lst), 4)
>
>         lst = [el for el in InstanceIntrospector(c, exclude='_')]
>         self.assertFalse(() in lst)
>         self.assertFalse((None, None, '.__doc__') in lst)
>         self.assertEqual(len(lst), 2)
>
>     def testDictIntrospector(self):
>         lst = [el for el in DictIntrospector(dict(a=1,_b=2))]
>
>         self.assertEqual(len(lst), 2)
>         self.assertTrue((1, 'a', '[a]') in lst)
>         self.assertTrue((2, '_b', '[_b]') in lst)
>
>         lst = [el for el in DictIntrospector(dict(a=1,_b=2), exclude='_')]
>         self.assertEqual(len(lst), 1)
>         self.assertTrue((1, 'a', '[a]') in lst)
>         self.assertFalse((2, '_b', '[_b]') in lst)
>
>     def testSearchName(self):
>         self.assertTrue(searchName('needle', 'the needle', None))
>         self.assertTrue(searchName('needle', 'needle more', None))
>         self.assertTrue(searchName('needle', 'the needle more', None))
>
>         # Make sure function doesn't freak out for no name
>         self.assertFalse(searchName('needle', None, None))
>
>     def testSearchValue(self):
>         class Composite:
>             def __init__(self, str):
>                 self._str = str
>             def __repr__(self):
>                 return self._str
>             def __str__(self):
>                 return self._str
>
>         self.assertTrue(searchValue('needle', None,
>                                     Composite('the needle')))
>         self.assertTrue(searchValue('needle', None,
>                                     Composite('needle more')))
>         self.assertTrue(searchValue('needle', None,
>                                     Composite('the needle more')))
>         # These are not true because searchValue doens't split
>         # apart built-in containers
>         self.assertFalse(searchValue('needle', None,
>                                     ['needle', 2, 3]))
>         self.assertFalse(searchValue('needle', None,
>                                     ('needle', 2, 3)))
>         self.assertFalse(searchValue('needle', None,
>                                     dict(a='needle', b=2, c=3)))
>
>
>     def testSearchDoc(self):
>         class Composite:
>             def __init__(self, str):
>                 self.__doc__ = str
>
>         self.assertTrue(searchDoc('needle', None,
>                                   Composite('the needle')))
>         self.assertTrue(searchDoc('needle', None,
>                                   Composite('needle more')))
>         self.assertTrue(searchDoc('needle', None,
>                                   Composite('the needle more')))
>
>         # Make sure search fn doesn't freak out
>         self.assertFalse(searchDoc('needle', None,
>                                    Composite(None)))
>
>
>     def testSearchNameRegexp(self):
>         self.assertFalse(searchNameRegexp('^needle', 'the needle', None))
>         self.assertTrue(searchNameRegexp('^needle', 'needle more', None))
>         self.assertFalse(searchNameRegexp('^needle', 'the needle more', None))
>
>         # Make sure function doesn't freak out for no name
>         self.assertFalse(searchName('^needle', None, None))
>
>     def testSearchValueRegexp(self):
>         class Composite:
>             def __init__(self, str):
>                 self._str = str
>             def __repr__(self):
>                 return self._str
>             def __str__(self):
>                 return self._str
>
>         self.assertFalse(searchValueRegexp('^needle', None,
>                                            Composite('the needle')))
>         self.assertTrue(searchValueRegexp('^needle', None,
>                                           Composite('needle more')))
>         self.assertFalse(searchValueRegexp('^needle', None,
>                                            Composite('the needle more')))
>
>         # Make sure we don't search inside containers
>         self.assertFalse(searchValueRegexp('needle', None,
>                                            ['needle', 2, 3]))
>         self.assertFalse(searchValueRegexp('needle', None,
>                                            ('needle', 2, 3)))
>         self.assertFalse(searchValueRegexp('needle', None,
>                                            dict(a='needle', b=2, c=3)))
>
>     def testSearchDocRegexp(self):
>         class Composite:
>             def __init__(self, str):
>                 self.__doc__ = str
>
>         self.assertFalse(searchDocRegexp('^needle', None,
>                                          Composite('the needle')))
>         self.assertTrue(searchDocRegexp('^needle', None,
>                                         Composite('needle more')))
>         self.assertFalse(searchDocRegexp('^needle', None,
>                                          Composite('the needle more')))
>
>         # Make sure function doesn't freak out if no doc
>         self.assertFalse(searchDocRegexp('^needle', None,
>                                          Composite(None)))
>
> def suite():
>     suites = [unittest.TestLoader().loadTestsFromTestCase(test)
>               for test in (AproposTest,)]
>     return unittest.TestSuite(suites)
>
> def test():
>     unittest.TextTestRunner().run(suite())
>
> def itest():
>     suite().debug()
>
> import types
> import re
>
> __version__ = 0.2
> __author__ = "Greg Novak <novak at ucolick.org"
> # Date: January 14, 2007
> # Code released public domain.  Do whatever you want with it.
>
> # You can add your own types to these lists if you want apropos to
> # descend into them.  If you have a container that you want apropos to
> # search, but it doesn't respond appropriately to the methods listed
> # below, you can give it a function called __apropos__.  This function
> # takes no arguments and should return an iterator.  The iterator
> # should return the contents of the object, as tuples of
> # (elementObject, nameString, accessString)
>
> # Must respond to __iter__ and [string].  Designed for things you
> # access via [string]
> dictTypes = [types.DictType]
> # Must respond to __iter__().  Designed for things you access via
> # [int]
> listTypes = [types.ListType, types.TupleType]
> # Must give sensible results to dir(), getattr().  Designed for things
> # you access via .
> instanceTypes = [types.InstanceType, types.ModuleType]
>
> ##################################################
> ## Interface
>
> ## Common Usage
> def aproposName(needle, haystack=None, **kw):
>     """Recursively search for attributes with where needle is a
>     substring of the name.  See apropos() for addtional keyword
>     arguments.  Typical usage is aproposName('string', module).
>
>     Return a list of strings showing the path to reach the matching
>     object"""
>     return apropos(needle, haystack, searchFn=searchName, **kw)
>
> def aproposValue(needle, haystack=None, **kw):
>     """Recursively search for attributes with where needle is a
>     substring the string representation of the object.  See apropos()
>     for addtional keyword arguments.  Typical usage is
>     aproposValue('string', module).
>
>     Return a list of strings showing the path to reach the matching
>     object"""
>     return apropos(needle, haystack, searchFn=searchValue, **kw)
>
> def aproposDoc(needle, haystack=None, **kw):
>     """Recursively search for attributes with where needle is a
>     substring of the documentation string of the object.  See
>     apropos() for addtional keyword arguments.  Typical usage is
>     aproposDoc('string', module).
>
>     Return a list of strings showing the path to reach the matching
>     object"""
>     return apropos(needle, haystack, searchFn=searchDoc, **kw)
>
> def aproposNameRegexp (needle, haystack=None, **kw):
>     """Recursively search for attributes with where needle is a regexp
>     matching the name.  See apropos() for addtional keyword arguments.
>     Typical usage is aproposNameRegexp('string', module).
>
>     Return a list of strings showing the path to reach the matching
>     object"""
>     return apropos(needle, haystack, searchFn=searchNameRegexp, **kw)
>
> def aproposValueRegexp(needle, haystack=None, **kw):
>     """Recursively search for attributes with where needle is a regexp
>     matching the string representation of the object.  See apropos()
>     for addtional keyword arguments.  Typical usage is
>     aproposValueRegexp('string', module).
>
>     Return a list of strings showing the path to reach the matching
>     object"""
>     return apropos(needle, haystack, searchFn=searchValueRegexp, **kw)
>
> def aproposDocRegexp(needle, haystack=None, **kw):
>     """Recursively search for attributes with where needle is a regexp
>     matching the docstring of the object.  See apropos() for addtional
>     keyword arguments.  Typical usage is aproposDocRegexp('string',
>     module).
>
>     Return a list of strings showing the path to reach the matching
>     object"""
>     return apropos(needle, haystack, searchFn=searchDocRegexp, **kw)
>
> ## Handles default values of arguments
> def apropos(needle, haystack=None, name=None,
>             searchFn=None, **kw):
>     """Recursively search through haystack looking for needle.
>     Typical usage is apropos('string', module).
>
>     haystack can be any python object.  Typically it's a module.  If
>     it's not given, it's the dict returned by globals() (ie, watch
>     out, it's going to take a while).
>
>     name is the name of the top level object.  It's first bit of the
>     'accessor' strings that are returned.  If not specified, defaults
>     to 'arg'.
>
>     Matches determined by searchFn.  searchFn(needle, name, obj)
>     returns true if the object should be considered a match.  By
>     default, searchFn matches if needle is a substring of the name of
>     the object.
>
>     Return a list of strings showing the path to reach the matching
>     object"""
>     if haystack is None:
>         haystack = globals()
>         name = ''
>     elif name is None:
>         if hasattr(haystack, "__name__"):
>             name = haystack.__name__
>         else:
>             name = 'arg'
>
>     if searchFn is None: searchFn = searchName
>
>     return _apropos(needle, haystack, name, searchFn, **kw)
>
> ##################################################
> ## Common search functions
>
> def searchName(needle, name, obj):
>     return name and needle in name
>
> def searchValue(needle, name, obj):
>     # String representation of dicts, lists, and tuples includes the
>     # objects within them, so don't consider that to be a match on the
>     # desired value.  Wait to get inside the container class...
>     #
>     # TODO What I really want to do is match the container if none of
>     # its contents matched.
>     if type(obj) not in (types.TupleType, types.ListType,
>                          types.DictType):
>         return needle in str(obj)
> # NOTE -- should be repr()?
>
> def searchDoc(needle, name, obj):
>     return hasattr(obj, '__doc__') and obj.__doc__ \
>            and needle in obj.__doc__
>
> def searchNameRegexp(needle, name, obj):
>     return name and re.search(needle, name)
>
> def searchValueRegexp(needle, name, obj):
>     if type(obj) not in (types.TupleType, types.ListType,
>                          types.DictType):
>         return re.search(needle, str(obj))
>
> def searchDocRegexp(needle, name, obj):
>     return hasattr(obj, '__doc__') \
>            and obj.__doc__ \
>            and re.search(needle, obj.__doc__)
>
> ##################################################
> ## The guts
>
> def _apropos(needle, haystack, haystackName,
>              searchFn, maxDepth=None, **kw):
>     """Recursively search through haystack looking for needle.
>
>     haystack can be any python object.  Typically it's a module.  If
>     it's not given, it's the dict returned by globals() (ie, watch
>     out, it's going to take a while).
>
>     Matches determined by searchFn.  searchFn(needle, name, obj)
>     returns true if the object should be considered a match.  By
>     default, searchFn matches if needle is a substring of the name of
>     the object.
>
>     name is the name of the top level object.  It's first bit of the
>     'accessor' strings that are returned.  If not specified, defaults
>     to 'arg'.
>
>     Return a list of strings showing the path to reach the matching
>     object."""
>     def search(haystack, haystackName, fullName, depth):
>         '''Free variable: needle, searchTypes'''
>         # print "Searched", len(searchedIds), "Searching", depth, fullName
>         if searchFn(needle, haystackName, haystack):
>             found.append(fullName)
>
>         # break apart if obj is not already searched
>         if type(haystack) in searchTypes \
>                 and (not maxDepth or depth < maxDepth) \
>                 and id(haystack) not in searchedIds:
>             # Prevent loops with circular references by setting this
>             # _before_ descending into sub-objects
>             searchedIds.append(id(haystack))
>
>             for hay, hayName, hayAccess in introspect(haystack, **kw):
>                 search(hay, hayName, fullName + hayAccess, depth+1)
>
>     searchedIds = []
>     found = []
>     searchTypes = dictTypes + listTypes + instanceTypes
>
>     search(haystack, haystackName, haystackName, 0)
>     return found
>
> def introspect(obj, **kw):
>     if type(obj) in dictTypes:
>         return DictIntrospector(obj, **kw)
>     if type(obj) in listTypes:
>         return ListIntrospector(obj, **kw)
>     if type(obj) in instanceTypes:
>         return InstanceIntrospector(obj, **kw)
>
>     # User objects
>     if hasattr(obj, '__apropos__'):
>         return obj.__apropos__(**kw)
>
>     # Stymied
>     print "apropos.py: Warning, don't know how to deal with " + str(obj)
>     return NullIntrospector()
>
> # NOTE These introspectors simplify the code, but they seem to take about five
> # times as long, very unfortunately.
> class Introspector (object):
>     def __iter__(self):
>         return self
>
>     def next(self):
>         pass
>
> class NullIntrospector (Introspector):
>     def __init__(self, **kw):
>         pass
>
>     def next(self):
>         raise StopIteration
>
> class DictIntrospector (Introspector):
>     # types that respond to __iter__, obj.[key] to get a value
>     def __init__(self, dict, exclude=None):
>         self.dict = dict
>         self.iter = self.dict.__iter__()
>         self.exclude = exclude
>
>     def next(self):
>         # return tuple of obj, name, accessName
>         k = self.iter.next()
>         # FIXME -- completely skip non-string key entries
>         while type(k) is not types.StringType \
>               or (self.exclude and k.startswith(self.exclude)):
>             k = self.iter.next()
>         return self.dict[k], k, '[' + k + ']'
>
> class ListIntrospector (Introspector):
>     # types that respond to __iter__
>     def __init__(self, list, exclude=None):
>         self.list = list
>         self.iter = self.list.__iter__()
>         self.i = 0
>
>     def next(self):
>         # return tuple of obj, name, accessName
>         self.i += 1
>         return self.iter.next(), None, '[' + str(self.i-1) + ']'
>
> class InstanceIntrospector (Introspector):
>     # classes that respond to dir and getattr
>     def __init__(self, inst, exclude=None):
>         self.inst = inst
>         self.iter = dir(self.inst).__iter__()
>         self.exclude = exclude
>
>     def next(self):
>         # return tuple of obj, name, accessName
>
>         # IPython structs allow non-string attributes.  Filter them
>         # out because they cause problems.  That is, you have to
>         # access them via obj[1], not getattr(obj, 1) or
>         # getattr(obj, '1')
>         # FIXME -- filter out non-string things that appear in dir()
>
>         name = self.iter.next()
>         while type(name) is not types.StringType \
>               or (self.exclude and name.startswith(self.exclude)):
>             name = self.iter.next()
>         return getattr(self.inst, name), name, "." + name
>
>
>