From novak at ucolick.org  Fri Jan  5 14:30:50 2007
From: novak at ucolick.org (Gregory Novak)
Date: Fri, 05 Jan 2007 11:30:50 -0800
Subject: [IPython-dev] Strange bug
Message-ID: <m2zm8xw9sl.fsf@ucolick.org>

I just spent the better part of a day trying to figure out what on
earth was going on with a rather strange bug.  The dramatis personae:

pyIDL is a bit of python + c code that allows you call IDL
(http://www.ittvis.com/idl/) code from Python
(http://www.its.caltech.edu/~mmckerns/software.html) 

(for Mike: IPython is an enhanced interactive python shell
http://ipython.scipy.org/moin/) 

pyIDL provides getattr/setattr functions to access variables in IDL so
that instead of typing idl_instance.get('x') you can type
idl_instance.x  

There's also a function called eval that evaluates IDL expressions,
like so:
idl_instance.eval('y=x')

When I called idl_instance.eval('y=x') under plain vanilla python, it
worked fine.  When I called the same thing under IPython, I got
strange errors coming from IDL:

% Attempt to call undefined procedure/function: 'EVAL'.
% Execution halted at: $MAIN$          

This error message is coming from IDL, because apparently the string
"eval(...)" is getting passed to the IDL instance rather than
executing the python idl_instance.eval function.

What on Earth is going on?  It works in Python, but not IPython, and
the failure mode is very strange... an extra string "escapes" into the
IDL instance.

Well, I spent a bunch of time fooling around with providing my own
__getattribute__, __setattribute__, etc functions that printed
messages in the idl_instance object, so that I could see how it was
treated differently in Python vs. IPython.  

It turns out that apparently IPython is calling 
idl_instance.__getattribute__('eval("y=x")).  Then the string
'eval("y=x")' is passed to pyIDL's __getattr__ method, which checks to
see if 'eval("y=x")' is an IDL variable.  Needless to say, it isn't,
and it triggers the above error message.

I'd love to provide a backtrace but when I run the code under pdb,
there's no problem (__getattribute__ is only called on strings like
'eval', not 'eval("y=x")').  And when I try to throw an exception in
the offending call to __getattribute__ in order to get a backtrace,
it's caught somewhere higher up, but I can't see where.  I've disabled
all of the try:/except: blocks whithin pyIDL, and none of them seem to
be catching it.  So it's either in C code or there's soemthing weirder
going on.

What's the point of all this?  Well, it definitely qualifies as a bug,
but there's an easy workaround: type idl_instance.eval("y = x").  The
spaces apparently prevent IPython from doing its strange
__getattribute__ call.  

That's all...
Greg


From fperez.net at gmail.com  Fri Jan  5 14:44:01 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Fri, 5 Jan 2007 12:44:01 -0700
Subject: [IPython-dev] Strange bug
In-Reply-To: <m2zm8xw9sl.fsf@ucolick.org>
References: <m2zm8xw9sl.fsf@ucolick.org>
Message-ID: <db6b5ecc0701051144u693d26cdo90a6f8e5cec22b52@mail.gmail.com>

On 1/5/07, Gregory Novak <novak at ucolick.org> wrote:
> I just spent the better part of a day trying to figure out what on
> earth was going on with a rather strange bug.  The dramatis personae:
>
> pyIDL is a bit of python + c code that allows you call IDL
> (http://www.ittvis.com/idl/) code from Python
> (http://www.its.caltech.edu/~mmckerns/software.html)
>
> (for Mike: IPython is an enhanced interactive python shell
> http://ipython.scipy.org/moin/)

He knows :)   (Mike and I have met several times at Caltech, at the
various SciPy conferences).


[...]

OK: you've given me enough information that I may be able to improve
the automatic evaluation parser to avoid this kind of situation,
though I won't get to it today.  Thanks for the excellent report.

In the meantime, try disabling autocalling:

In [1]: %autocall
Automatic calling is: OFF

and see if it helps.  If not, disable automagic as well:

In [2]: %automagic

Automagic is OFF, % prefix IS needed for magic functions.


These may give you a temporary workaround, til I get to the bottom of this.

Regards,

f


From fperez.net at gmail.com  Fri Jan  5 20:46:36 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Fri, 5 Jan 2007 18:46:36 -0700
Subject: [IPython-dev] Strange bug
In-Reply-To: <Pine.GSO.4.58.0701051205100.2063@sue>
References: <m2zm8xw9sl.fsf@ucolick.org>
	<db6b5ecc0701051144u693d26cdo90a6f8e5cec22b52@mail.gmail.com>
	<Pine.GSO.4.58.0701051205100.2063@sue>
Message-ID: <db6b5ecc0701051746m3da9de0dn5d62dc6c26b14fd1@mail.gmail.com>

[ I've added you to the whitelist on ipython-dev so your messages don't bounce ]


On 1/5/07, Michael McKerns <mmckerns at its.caltech.edu> wrote:
> Fernando,
>
> I have a evaluation parser inside of pyIDL too.
> Out of curiosity, you may want to take a look
> at the algorithm.  Everything after "END PUBLIC
> METHODS" in rsiidl.py relates to the parsing of
> IDL commands... but the logic is similar to python
> expressions, and for most cases I use re patterns.
>
> This reminds me that we shold talk again soon. :)

Yup :)

Thanks for the pointer.  In this case, I'm pretty sure the fix is
relatively simple once I dig deep enough into the part of the code
that does this parsing; Greg's report was detailed enough that I can
track it down based on his description.

Cheers,

f


From fperez.net at gmail.com  Mon Jan  8 12:58:14 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Mon, 8 Jan 2007 10:58:14 -0700
Subject: [IPython-dev] Strange bug
In-Reply-To: <m23b6m78a6.fsf@euterpe.local>
References: <m2zm8xw9sl.fsf@ucolick.org>
	<db6b5ecc0701051144u693d26cdo90a6f8e5cec22b52@mail.gmail.com>
	<m23b6m78a6.fsf@euterpe.local>
Message-ID: <db6b5ecc0701080958t1e170661k48b7b5451df2fb10@mail.gmail.com>

[ This was meant originally for the list, resending ]

On 1/7/07, Gregory Novak <novak at ucolick.org> wrote:

> In the meantime, I have another strange IPython specific bug.  This
> uses GDL (http://gnudatalanguage.sourceforge.net/), an independent
> implementation of the IDL syntax.  (I'd like to have two independent
> solutions so my calling-IDL-from-python problem in case one of them
> ends up giving me trouble for one reason or another).  GDL allows
> itself to be built as a python module, and then you can do things
> like:
>
> >>> import GDL
> >>> print GDL.function("sin",(1,))
> 0.841470956802

Mmh, this one is not ipython-specific.  I get the same crash under
Ubuntu Edgy with 2.4.4 using pure python (not ipython).  The following
is running python under gdb:

>>> import GDL
>>> print GDL.function("sin",(1,))

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 1075537584 (LWP 1616)]
0x40026bd9 in initreadline () from /usr/lib/python2.4/lib-dynload/readline.so
(gdb) bt
#0  0x40026bd9 in initreadline () from
/usr/lib/python2.4/lib-dynload/readline.so
#1  0x080f9a5b in PyOS_Readline ()
#2  0x08057549 in PyTokenizer_FromString ()
#3  0x08057cbd in PyTokenizer_Get ()
#4  0x08055dac in Py_Main ()
#5  0x080dd277 in PyRun_InteractiveOneFlags ()
#6  0x080dd466 in PyRun_InteractiveLoopFlags ()
#7  0x080ddf72 in PyRun_AnyFileExFlags ()
#8  0x08055cc2 in Py_Main ()
#9  0x08055132 in main ()


It very much seems like you've found a bug in readline related to
threads issues, which doesn't surprise me terribly.  Other ipython
users have in the past found similar bugs, which did get fixed:

http://sourceforge.net/tracker/index.php?func=detail&aid=1166660&group_id=5470&atid=105470

The reason why for you plain python doesn't crash is because your
plain python session is not initializing readline at all, which I do
via my $PYTHONSTARTUP.  Here's how to crash plain python (at least it
works reliably on my ubuntu box).  Paste this into an interactive
session (don't try to execfile, you have to paste it as if you had
typed it interactively so readline has to work):

import os
import readline

''' Some gibberish here is needed that's long enough to trigger the bug.  I'm
not sure how much data is actually required...

12jd lsjlksajsa jfl;ksaj34567890
123asdjf lkasdj flkajs l;kfd4567890
123asdflkasdj fl;saj f;a4567890
12jd lsjlksajsa jfl;ksaj34567890
123asdjf lkasdj flkajs l;kfd4567890
123asdflkasdj fl;saj f;a4567890
12jd lsjlksajsa jfl;ksaj34567890
123asdjf lkasdj flkajs l;kfd4567890
123asdflkasdj fl;saj f;a4567890
'''

try:
   __histfile = os.path.join(os.environ['HOME'],'.python-history')
   readline.read_history_file(__histfile)
except IOError:
   pass  # History file doesn't exist yet.

# This will now cause a crash:

import GDL
print GDL.function("sin",(1,))

# EOF

At this point, I get a segfault, so it's safe to say we have a python
bug here, not an ipython one.  IPython just happens to activate so
much of readline at startup that it forces the bug to the surface.

A good rule of thumb is: if you see a segfault, it's not an ipython
bug (except under Windows).  IPython has no extension code, so by
itself in principle it can't segfault.  It may /expose/ python bugs
(such as in this case, and it has happened before), but it can't cause
them by itself.

This statement is qualified because under windows, IPython's readline
support uses ctypes, and once you load ctypes, it's trivial to
segfault python with 'pure python', since at that point you're really
hooking directly into C libs and have bypassed all of the safeties of
the Python VM.

Best,

f


From vivainio at gmail.com  Wed Jan 17 13:19:08 2007
From: vivainio at gmail.com (Ville M. Vainio)
Date: Wed, 17 Jan 2007 19:19:08 +0100
Subject: [IPython-dev] Fwd: IPython Crash Report
In-Reply-To: <BB13E4C4D649234F84A01F6DF64BB87F017E16F9@ccs-ex01.coldstorage.com>
References: <BB13E4C4D649234F84A01F6DF64BB87F017E16F9@ccs-ex01.coldstorage.com>
Message-ID: <46cb515a0701171019w127d2014m349e6c1c273fae33@mail.gmail.com>

Posting this to ipython-dev in case someone on windows has time to look at
it. I'm currently running only linux and can't play with it for a while.

---------- Forwarded message ----------
From: Gerrat Rickert <grickert at coldstorage.com>
Date: Jan 9, 2007 4:09 PM
Subject: IPython Crash Report
To: vivainio at gmail.com

 ?this isn't real important, but just more of a FYI.



Basically, under Windows (XP):

1)  launch IPython,

2)  type Ctrl-D to exit,

3)  hit Enter to begin a new line

4) type pretty much anything that isn't a command ? IPython reports that it
has crashed.  It should probably just give a little error about [whatever
you typed] is not defined.











-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20070117/753e6ba2/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.jpg
Type: image/jpeg
Size: 11152 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20070117/753e6ba2/attachment.jpg>
-------------- next part --------------
***************************************************************************

IPython post-mortem report

IPython version: 0.7.3 

SVN revision   : 2007 

Platform info  : os.name -> nt, sys.platform -> win32

***************************************************************************

Current user configuration structure:

{'Version': 0,
 '__allownew': False,
 'alias': [],
 'args': [],
 'autocall': 1,
 'autoedit_syntax': 0,
 'autoindent': 1,
 'automagic': 1,
 'banner': 1,
 'c': '',
 'cache_size': 1000,
 'classic': 0,
 'color_info': 1,
 'colors': 'Linux',
 'confirm_exit': 1,
 'debug': 0,
 'deep_reload': 0,
 'editor': 'notepad',
 'embedded': False,
 'execfile': [],
 'execute': [''],
 'gthread': 0,
 'help': 0,
 'ignore': 0,
 'import_all': [],
 'import_mod': [],
 'import_some': [[]],
 'include': [],
 'ipythondir': 'C:\\Documents and Settings\\grickert\\_ipython',
 'log': 0,
 'logfile': '',
 'logplay': '',
 'magic_docstrings': 0,
 'messages': 1,
 'multi_line_specials': 1,
 'nosep': 0,
 'object_info_string_level': 0,
 'opts': Struct({'__allownew': True}),
 'pdb': 1,
 'pprint': 1,
 'profile': '',
 'prompt_in1': 'In [\\#]: ',
 'prompt_in2': '   .\\D.: ',
 'prompt_out': 'Out[\\#]: ',
 'prompts_pad_left': 1,
 'pylab': 0,
 'q4thread': 0,
 'qthread': 0,
 'quick': 0,
 'quiet': 0,
 'rcfile': 'ipythonrc.ini',
 'readline': 1,
 'readline_merge_completions': 1,
 'readline_omit__names': 0,
 'readline_parse_and_bind': ['tab: complete',
                             '"\\C-l": possible-completions',
                             'set show-all-if-ambiguous on',
                             '"\\C-o": tab-insert',
                             '"\\M-i": "    "',
                             '"\\M-o": "\\d\\d\\d\\d"',
                             '"\\M-I": "\\d\\d\\d\\d"',
                             '"\\C-r": reverse-search-history',
                             '"\\C-s": forward-search-history',
                             '"\\C-p": history-search-backward',
                             '"\\C-n": history-search-forward',
                             '"\\e[A": history-search-backward',
                             '"\\e[B": history-search-forward'],
 'readline_remove_delims': '-/~',
 'screen_length': -2,
 'separate_in': '\n',
 'separate_out': '',
 'separate_out2': '',
 'system_header': 'IPython system call: ',
 'system_verbose': 0,
 'tk': 0,
 'upgrade': 0,
 'wildcards_case_sensitive': 1,
 'wthread': 0,
 'wxversion': '0',
 'xmode': 'Context'}

***************************************************************************

Crash traceback:

---------------------------------------------------------------------------
exceptions.NameError                              Python 2.4.3: c:\python24\python.exe
                                                   Tue Jan 09 10:02:21 2007
A problem occured executing Python code.  Here is the sequence of function
calls leading up to the error, with the most recent (innermost) call last.

C:\Misc\Admin\python\<stdin> 
----> 1 
      2 
      3 
      4 
      5 
      6 
      7 
      8 
      9 
     10 
     11 
     12 
     13 
     14 
     15 
     16 
     17 
     18 
     19 
     20 
     21 
     22 
     23 
     24 
     25 
     26 
     27 
     28 
     29 
     30 
     31 

NameError: name 'exit' is not defined

***************************************************************************

History of session input:

*** Last line of input (may not be in above history):

From vivian at vdesmedt.com  Thu Jan 18 03:03:25 2007
From: vivian at vdesmedt.com (Vivian De Smedt)
Date: Thu, 18 Jan 2007 09:03:25 +0100
Subject: [IPython-dev] Fwd: IPython Crash Report
In-Reply-To: <46cb515a0701171019w127d2014m349e6c1c273fae33@mail.gmail.com>
References: <BB13E4C4D649234F84A01F6DF64BB87F017E16F9@ccs-ex01.coldstorage.com>
	<46cb515a0701171019w127d2014m349e6c1c273fae33@mail.gmail.com>
Message-ID: <45AF29CD.2080408@vdesmedt.com>

Just for information. I run IPython on WindowsXP and I'cant reproduce 
your problem with that small procedure

1)  launch IPython,
2)  type Ctrl-D to exit,
3)  hit Enter to begin a new line
4) type pretty much anything that isn't a command ? IPython reports that 
it has crashed.  It should probably just give a little error about 
[whatever you typed] is not defined

Vivian.




From Fernando.Perez at colorado.edu  Thu Jan 18 14:50:55 2007
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu, 18 Jan 2007 12:50:55 -0700
Subject: [IPython-dev] Auto-discard notification
In-Reply-To: <91cf711d0701180721o5fbe36c4ve2b4d0fac4cfc179@mail.gmail.com>
References: <mailman.2341.1168900215.11042.ipython-dev@scipy.org>	
	<45AF18E3.3080600@colorado.edu>
	<91cf711d0701180721o5fbe36c4ve2b4d0fac4cfc179@mail.gmail.com>
Message-ID: <45AFCF9F.5040506@colorado.edu>

David Huard wrote:
> Thanks a lot Fernando,
> 
> Effectively, it doesn't seem to work, unless I'm doing something wrong.

Small bug, here's the fix.  Let me know if it works (as you noticed, this code 
has seen precious little testing):

Index: ipython1/kernel/controllervanilla.py
===================================================================
--- ipython1/kernel/controllervanilla.py        (revision 2018)
+++ ipython1/kernel/controllervanilla.py        (working copy)
@@ -169,7 +169,7 @@

          self.pushAll(tar_fileString=fileString)
          self.executeAll("tar_file = open('%s','wb')" % \
-            tarball_name, block=False)
+            tarballName, block=False)
          self.executeAll("tar_file.write(tar_fileString)", block=False)
          self.executeAll("tar_file.close()", block=False)
          self.executeAll("import os", block=False)


I've already committed it, so you can either apply it manually or update your 
SVN checkout.

For future discussions on this, I'd really appreciate it if you could use the 
list.  Much of this code has not been written by me directly, so it's critical 
that we get other developers' feedback as well.

Thanks for the report!

Cheers,

f


From david.huard at gmail.com  Thu Jan 18 16:03:11 2007
From: david.huard at gmail.com (David Huard)
Date: Thu, 18 Jan 2007 16:03:11 -0500
Subject: [IPython-dev] Auto-discard notification
In-Reply-To: <45AFCF9F.5040506@colorado.edu>
References: <mailman.2341.1168900215.11042.ipython-dev@scipy.org>
	<45AF18E3.3080600@colorado.edu>
	<91cf711d0701180721o5fbe36c4ve2b4d0fac4cfc179@mail.gmail.com>
	<45AFCF9F.5040506@colorado.edu>
Message-ID: <91cf711d0701181303u4120a8f7s7e7362f93a730434@mail.gmail.com>

Ok, now I can push the module, but the import fails.

So for instance

>>> from tests import model_1
>>> rc.pushModule(model_1)
Locating the module...
Module (model_1) found in:
tests
[127.0.0.1:0] In [25]: os.system('tar -xf model_1.tar')
[127.0.0.1:0] Out[25]: 0
[127.0.0.1:1] In [34]: os.system('tar -xf model_1.tar')
[127.0.0.1:1] Out[34]: 0
[127.0.0.1:2] In [23]: os.system('tar -xf model_1.tar')
[127.0.0.1:2] Out[23]: 0
[127.0.0.1:3] In [18]: os.system('tar -xf model_1.tar')
[127.0.0.1:3] Out[18]: 0

>>> rc.executeAll('import model_1')
[127.0.0.1:1] In [35]: import model_1
[127.0.0.1:1] Err[35]:
 Traceback (most recent call last):
  File "<console>", line 1, in ?
ImportError: No module named model_1

The model_1.tar file is in the local directory, but the engines doesn't seem
to find the module on their side.
Where do the engines live ? rc.executeAll('os.getcwd()') return the local
dir, but the engines themselves must be elsewhere, since they can't find the
modules locally.

Cheers,

David

P.S. Fernando, don't bother sending the patch, svn checking is easier for
both of us.



2007/1/18, Fernando Perez <Fernando.Perez at colorado.edu>:
>
> David Huard wrote:
> > Thanks a lot Fernando,
> >
> > Effectively, it doesn't seem to work, unless I'm doing something wrong.
>
> Small bug, here's the fix.  Let me know if it works (as you noticed, this
> code
> has seen precious little testing):
>
> Index: ipython1/kernel/controllervanilla.py
> ===================================================================
> --- ipython1/kernel/controllervanilla.py        (revision 2018)
> +++ ipython1/kernel/controllervanilla.py        (working copy)
> @@ -169,7 +169,7 @@
>
>           self.pushAll(tar_fileString=fileString)
>           self.executeAll("tar_file = open('%s','wb')" % \
> -            tarball_name, block=False)
> +            tarballName, block=False)
>           self.executeAll("tar_file.write(tar_fileString)", block=False)
>           self.executeAll("tar_file.close()", block=False)
>           self.executeAll("import os", block=False)
>
>
> I've already committed it, so you can either apply it manually or update
> your
> SVN checkout.
>
> For future discussions on this, I'd really appreciate it if you could use
> the
> list.  Much of this code has not been written by me directly, so it's
> critical
> that we get other developers' feedback as well.
>
> Thanks for the report!
>
> Cheers,
>
> f
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20070118/d7da63db/attachment.html>

From wvarner at purdue.edu  Tue Jan 23 23:04:30 2007
From: wvarner at purdue.edu (Winship Varner)
Date: Tue, 23 Jan 2007 23:04:30 -0500
Subject: [IPython-dev] problems I've having with the ls and dir commands on
	Windows XP
Message-ID: <45B6DACE.9040503@purdue.edu>

Hello,

I just recently joined this mailing list because I'm having the 
following problems with IPython. Here's what I get when I type in the 
following:

    In [1]: ls
    IPython system call: dir /on /on

This is *all* its says. It does not display the contents of the 
directory. When I type:

    In [2]: dir
    Out[2]: <built-in function dir>

Again, this is all I get. And then finally, when I type in the following:

    In [3]: !!ls *txt
    IPython system call: ls *txt

The program seems to freeze (I cannot get it to quit using Ctl-C or 
Ctl-Z), and I have to close the window. I'm using the newest versions of 
the program and its prerequisites. Any help or workarounds would be 
greatly appreciated. Thanks.

-- Winship Varner


From vivainio at gmail.com  Wed Jan 24 00:55:41 2007
From: vivainio at gmail.com (Ville M. Vainio)
Date: Wed, 24 Jan 2007 06:55:41 +0100
Subject: [IPython-dev] problems I've having with the ls and dir commands
	on Windows XP
In-Reply-To: <45B6DACE.9040503@purdue.edu>
References: <45B6DACE.9040503@purdue.edu>
Message-ID: <46cb515a0701232155r437c57dcp4955b86750d0876a@mail.gmail.com>

On 1/24/07, Winship Varner <wvarner at purdue.edu> wrote:

> The program seems to freeze (I cannot get it to quit using Ctl-C or
> Ctl-Z), and I have to close the window. I'm using the newest versions of
> the program and its prerequisites. Any help or workarounds would be
> greatly appreciated. Thanks.

Is there something wrong with your ls binary (broken cygwin/whatever)?
Does "!ls" work normally? How about "!dir"?

After finding out what works, (let's say 'ls -F'), do

> alias ls ls -F

> %store ls

and it should be fixed for all the sessions.

-- 
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'


From ellisonbg.net at gmail.com  Wed Jan 24 15:34:19 2007
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 24 Jan 2007 13:34:19 -0700
Subject: [IPython-dev] Update on IPython recent developments
Message-ID: <6ce0ac130701241234l19235b20v3d20d22bbbf349e2@mail.gmail.com>

Hello all,

The purpose of this email is to update IPython developers on the state
of IPython1, which is currently focused on bringing interactive
parallel and distributed capabilities to IPython.  Over the past month
or so, we have done a good amount of refactoring of IPython1.  As of
now, the current development branch for IPython1 is "saw" rather than
"chainsaw."  Because people are still using chainsaw, we will maintain
it for a while (maybe another month), but the saw branch has a lot of
new things.  I should say, that we are not yet recommending that
averge users start to use "saw" yet.  There are a few more things we
need to do before it is a full replacement of the "chainsaw" branch.
But anyone following the development of IPython1 closely, definitely
should look at "saw."

Below is a summary of the new things in the saw branch.  For an
overview of our overall design, checkout the following wiki pages:

http://ipython.scipy.org/moin/IPython1
http://ipython.scipy.org/moin/Parallel_Computing

Design Changes
=============

The core design elements are documented in the engine and controller
interfaces.  These can be found here:

For the engine:

http://projects.scipy.org/ipython/ipython/browser/ipython/branches/saw/ipython1/kernel/engineservice.py

For the controller:

http://projects.scipy.org/ipython/ipython/browser/ipython/branches/saw/ipython1/kernel/controllerservice.py

The big different in the saw branch is that the controller interface
has been simplified dramatically.
The controller is now basically something to which engines connect.
How those engines
are presented to clients is no longer specified in the controller interface.

The reason we have done this is to make is easy for developers to
quickly develop (using)
adapters new high level client interfaces to a a controller (a set fo
engines).  The client interface
that was in the chainsaw branch is now called the multiengine
interface and is documented here:

http://projects.scipy.org/ipython/ipython/browser/ipython/branches/saw/ipython1/kernel/multiengine.py

This controller interface is exposed to various network protocols in
the modules:

multienginepb.py
multienginevanilla.py

It should now be very easy for people to use the multiengine case as
an example in developing new controller interfaces.  Some of the
interfaces that we are thinking about:

A task farming interface
A map/reduce style interface

The nice thing about this new design is that by using adaption a
single controller instance can be adapted to multiple interfaces and
each of those interfaces can be exposed to the network
in multiple network protocols.

Please feel free to email us if you have questions about these things.

Testing Facilities
==============

We have been working hard to get our tests up to speed.  There have
been two big developments on this front lately:

* New testing utilities.  Fernando has done a great job in adding new
testing utilities that allow lineline doctests, standalone doctests
and standard unittests to be integrated together in a seamless
fashion.  These utilities also allow IPython code to be tested in
doctests :)  Have a look here for these new utilities, examples and
templates.

http://projects.scipy.org/ipython/ipython/browser/ipython/branches/saw/ipython1/test

* Much better test coverage.  With the refactoring we have done, we
also have improved our test coverage a lot.  There is more to do, but
we are beginning to have a good deal of confidence in the core parts
of IPython1.

Error Propagation
==============

We are now able to propagate most types of exceptions over the
network.  While some of this was being done before, the new
implementation is much better:

* The exception types are preserved as the hop along the network.
* Exceptions are raised in the users namespace
* Errors are now reported in an agressive manner.  This means that any
error is immediately sent back to the user
the instant is happens.

Client Changes
============

The old client that is documented on our wiki pages is still
supported, but will be depreciated soon.
We are working on a new version that will use a better network
protocol.  This new version will have the same exact API so any
existing code that uses that client will still work.

We have a new client based on Twistsed's Perspective Broker protocol.
This client is fully asynchronous and in combination with our new
blockon.py module, code can be written in a nice sort of hybrod
synchronous/asynchronous manner.  We still don't have documenation of
this, but that should emerge soon.  Have a look at

http://projects.scipy.org/ipython/ipython/browser/ipython/branches/saw/ipython1/kernel/blockon.py
http://projects.scipy.org/ipython/ipython/browser/ipython/branches/saw/ipython1/kernel/multienginepb.py

If you are interested in this client.

When the "saw" branch in ready to replace "chainsaw" formally  we will
let people know on the dev and users lists.

Cheers,

Brian


From stefan at sun.ac.za  Wed Jan 24 19:28:42 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Thu, 25 Jan 2007 02:28:42 +0200
Subject: [IPython-dev] Update on IPython recent developments
In-Reply-To: <6ce0ac130701241234l19235b20v3d20d22bbbf349e2@mail.gmail.com>
References: <6ce0ac130701241234l19235b20v3d20d22bbbf349e2@mail.gmail.com>
Message-ID: <20070125002842.GA5772@mentat.za.net>

Hi Brian

Thanks for the update, and for all the hard work you guys have put
into the new ipython kernel so far!  I'm especially excited about the
new test framework (they all pass here, woohoo!).

A couple of beginner's questions:

- When a task is executed on many engines, and some of them fail, how
  do I find out on which node it failed, and with which error message?

- Is there a limit on the size of data sent from the controller to an
  engine?  When working on one machine (with more than one engine), is
  data being sent around exactly as on a network (i.e. are there no
  optimisations made for working on a local machine?)  Would a UNIX
  socket be a viable alternative (to a socket) in this case?

- I'd like to code algorithms so they can be executed either with or
  without using the new kernel.  How difficult would it be to
  implement a dummy controller interface that can easily be included
  with a package to avoid a dependency on ipython1?  I.e. (drastically
  over-simplified):

  def getIDs():
      return [0]

  def push(nodes,**vars):
      for var,value in vars.iteritems():
          # set self._cache[var] = value

  def execute(node,cmd):
      # In cached namespace, execute cmd

  etc.

Hope these questions make sense on some level.

Regards.
St?fan

On Wed, Jan 24, 2007 at 01:34:19PM -0700, Brian Granger wrote:
> Hello all,
> 
> The purpose of this email is to update IPython developers on the state
> of IPython1, which is currently focused on bringing interactive
> parallel and distributed capabilities to IPython.  Over the past month
> or so, we have done a good amount of refactoring of IPython1.  As of
> now, the current development branch for IPython1 is "saw" rather than
> "chainsaw."  Because people are still using chainsaw, we will maintain
> it for a while (maybe another month), but the saw branch has a lot of
> new things.  I should say, that we are not yet recommending that
> averge users start to use "saw" yet.  There are a few more things we
> need to do before it is a full replacement of the "chainsaw" branch.
> But anyone following the development of IPython1 closely, definitely
> should look at "saw."


From ellisonbg.net at gmail.com  Wed Jan 24 22:45:51 2007
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 24 Jan 2007 20:45:51 -0700
Subject: [IPython-dev] Update on IPython recent developments
In-Reply-To: <20070125002842.GA5772@mentat.za.net>
References: <6ce0ac130701241234l19235b20v3d20d22bbbf349e2@mail.gmail.com>
	<20070125002842.GA5772@mentat.za.net>
Message-ID: <6ce0ac130701241945t618463ebu5073c26a0f20d0f7@mail.gmail.com>

> Thanks for the update, and for all the hard work you guys have put
> into the new ipython kernel so far!  I'm especially excited about the
> new test framework (they all pass here, woohoo!).

Great, we are also very excited about passing tests.

> A couple of beginner's questions:
>
> - When a task is executed on many engines, and some of them fail, how
>   do I find out on which node it failed, and with which error message?

Currently if any action (execute, push, pull) is run against multiple
engines, the error handling goes as follows:

If any single engine fires an exception, that exception is raised in
your local python session immediately (when using the Perspective
Broker client in multienginepb.py.  This behavior is not yet
implemented in the RemoteController class you get when importing
kernel.api.  But this will change soon) any other exceptions are
logged, but this behavior could be customized.  Right now we are not
passing back the engine id with the exception, but we know how to do
this and plan on implementing it.

> - Is there a limit on the size of data sent from the controller to an
>   engine?  When working on one machine (with more than one engine), is
>   data being sent around exactly as on a network (i.e. are there no
>   optimisations made for working on a local machine?)  Would a UNIX
>   socket be a viable alternative (to a socket) in this case?

Absolutely.  There are many limits that come into play:

1.  The RAM on each involved machine.

2.  The number of engines a controller is controlling.

3.  The network protocol being used.  If you  are using the
perspective broker protocols in the "saw" branch, the size limit for
this protocol is set in pbconfig.py.  Just a warning though, if you
exceed this limit on a pull/gather currently, things will hang.  On a
push you will get a sensible exception raised.  We know how to solve
the problem on pull/gather of too large objects, but haven't done that
yet.

On a local machine, if you are running things on localhost (127.0.0.1)
you won't be limited by the network pipe/latency.  I am not sure if
UNIX sockets would be better than that.

> - I'd like to code algorithms so they can be executed either with or
>   without using the new kernel.  How difficult would it be to
>   implement a dummy controller interface that can easily be included
>   with a package to avoid a dependency on ipython1?  I.e. (drastically
>   over-simplified):
>
>   def getIDs():
>       return [0]
>
>   def push(nodes,**vars):
>       for var,value in vars.iteritems():
>           # set self._cache[var] = value
>
>   def execute(node,cmd):
>       # In cached namespace, execute cmd
>
>   etc.

Absolutely, this is something that Fernando has talked about a number
of times.  Fernando, let's add this to our list of things to work on
next week.

> Hope these questions make sense on some level.
>

Absolutely.  We will keep everyone posted as we work on these things.

> Regards.
> St?fan
>
> On Wed, Jan 24, 2007 at 01:34:19PM -0700, Brian Granger wrote:
> > Hello all,
> >
> > The purpose of this email is to update IPython developers on the state
> > of IPython1, which is currently focused on bringing interactive
> > parallel and distributed capabilities to IPython.  Over the past month
> > or so, we have done a good amount of refactoring of IPython1.  As of
> > now, the current development branch for IPython1 is "saw" rather than
> > "chainsaw."  Because people are still using chainsaw, we will maintain
> > it for a while (maybe another month), but the saw branch has a lot of
> > new things.  I should say, that we are not yet recommending that
> > averge users start to use "saw" yet.  There are a few more things we
> > need to do before it is a full replacement of the "chainsaw" branch.
> > But anyone following the development of IPython1 closely, definitely
> > should look at "saw."
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/ipython-dev
>


From stefan at sun.ac.za  Thu Jan 25 02:50:16 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Thu, 25 Jan 2007 09:50:16 +0200
Subject: [IPython-dev] Update on IPython recent developments
In-Reply-To: <6ce0ac130701241945t618463ebu5073c26a0f20d0f7@mail.gmail.com>
References: <6ce0ac130701241234l19235b20v3d20d22bbbf349e2@mail.gmail.com>
	<20070125002842.GA5772@mentat.za.net>
	<6ce0ac130701241945t618463ebu5073c26a0f20d0f7@mail.gmail.com>
Message-ID: <20070125075016.GB5772@mentat.za.net>

Hi Brian

On Wed, Jan 24, 2007 at 08:45:51PM -0700, Brian Granger wrote:
> > - Is there a limit on the size of data sent from the controller to an
> >   engine?  When working on one machine (with more than one engine), is
> >   data being sent around exactly as on a network (i.e. are there no
> >   optimisations made for working on a local machine?)  Would a UNIX
> >   socket be a viable alternative (to a socket) in this case?
> 
> Absolutely.  There are many limits that come into play:
> 
> 1.  The RAM on each involved machine.
> 
> 2.  The number of engines a controller is controlling.
> 
> 3.  The network protocol being used.  If you  are using the
> perspective broker protocols in the "saw" branch, the size limit for
> this protocol is set in pbconfig.py.  Just a warning though, if you
> exceed this limit on a pull/gather currently, things will hang.  On a
> push you will get a sensible exception raised.  We know how to solve
> the problem on pull/gather of too large objects, but haven't done that
> yet.

In order to solve the problem of sending big heaps of data, I was
thinking of breaking up the data, and providing it piece-wise to an
engine when free, i.e.

while len(data_queue) != 0:
    id = rc.sendToFree({'x':data_queue.pop()})
    rc.execute(id,'compute_longtime("x")',block=False)

which is slightly different from scatter, which immediately sends off
all the data to the different nodes.  But this method will make the
'gather'-operation harder to implement.  Clearly, I've got some more
thinking to do.

Thanks for taking the time to answer all my questions!

Regards
St?fan


From fperez.net at gmail.com  Thu Jan 25 04:31:04 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 25 Jan 2007 02:31:04 -0700
Subject: [IPython-dev] Update on IPython recent developments
In-Reply-To: <6ce0ac130701241945t618463ebu5073c26a0f20d0f7@mail.gmail.com>
References: <6ce0ac130701241234l19235b20v3d20d22bbbf349e2@mail.gmail.com>
	<20070125002842.GA5772@mentat.za.net>
	<6ce0ac130701241945t618463ebu5073c26a0f20d0f7@mail.gmail.com>
Message-ID: <db6b5ecc0701250131o3aa539a5m20b88c25cf9c34fb@mail.gmail.com>

On 1/24/07, Brian Granger <ellisonbg.net at gmail.com> wrote:
> > Thanks for the update, and for all the hard work you guys have put
> > into the new ipython kernel so far!  I'm especially excited about the
> > new test framework (they all pass here, woohoo!).
>
> Great, we are also very excited about passing tests.

And I just got the last piece of the testing system that I wanted
finished on my local tree.  I'll clean it up and commit tomorrow or
Friday (I need to sleep), but this will make it easier to generate
large, standalone doctest files (in addition to doctests inside
modules of code).

After years of embarrassment with ipython's testing disaster, we're
finally going to have something nice and robust. 'bout time...

Brian, thanks a lot for dealing with the other questions and posting
this updated info.  We'll add more to the wiki later...

Cheers,

f


From fperez.net at gmail.com  Thu Jan 25 15:06:14 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 25 Jan 2007 13:06:14 -0700
Subject: [IPython-dev] Update on IPython recent developments
In-Reply-To: <db6b5ecc0701250131o3aa539a5m20b88c25cf9c34fb@mail.gmail.com>
References: <6ce0ac130701241234l19235b20v3d20d22bbbf349e2@mail.gmail.com>
	<20070125002842.GA5772@mentat.za.net>
	<6ce0ac130701241945t618463ebu5073c26a0f20d0f7@mail.gmail.com>
	<db6b5ecc0701250131o3aa539a5m20b88c25cf9c34fb@mail.gmail.com>
Message-ID: <db6b5ecc0701251206n157bc1dv4a82b08efab0180e@mail.gmail.com>

On 1/25/07, Fernando Perez <fperez.net at gmail.com> wrote:
> On 1/24/07, Brian Granger <ellisonbg.net at gmail.com> wrote:
> > > Thanks for the update, and for all the hard work you guys have put
> > > into the new ipython kernel so far!  I'm especially excited about the
> > > new test framework (they all pass here, woohoo!).
> >
> > Great, we are also very excited about passing tests.
>
> And I just got the last piece of the testing system that I wanted
> finished on my local tree.  I'll clean it up and commit tomorrow or
> Friday (I need to sleep), but this will make it easier to generate
> large, standalone doctest files (in addition to doctests inside
> modules of code).

Scratch that.  When I said 'last piece' above, it's because I'd given
up on the possibility of doctesting code that spawns subprocesses, due
to the fact that their stdout file descriptors are NOT the sys.stdout
of the original process.  This was stated in the 'limitations' section
here:

http://projects.scipy.org/ipython/ipython/browser/ipython/branches/saw/ipython1/test/ipdoctest.py

I now realized that there /is/ a solution to this problem using
pexpect, so before I commit the lot I'll solve this problem as well.
This will really leave us with a system where it will be /very easy/
to add doctests for pretty much any of ipython's functionality.  I'm
sure there will still  be a few dark corners, but I'm now determined
to reduce that count to zero.

Cheers,

f


From fperez.net at gmail.com  Fri Jan 26 04:47:55 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Fri, 26 Jan 2007 02:47:55 -0700
Subject: [IPython-dev] Update on IPython recent developments
In-Reply-To: <db6b5ecc0701251206n157bc1dv4a82b08efab0180e@mail.gmail.com>
References: <6ce0ac130701241234l19235b20v3d20d22bbbf349e2@mail.gmail.com>
	<20070125002842.GA5772@mentat.za.net>
	<6ce0ac130701241945t618463ebu5073c26a0f20d0f7@mail.gmail.com>
	<db6b5ecc0701250131o3aa539a5m20b88c25cf9c34fb@mail.gmail.com>
	<db6b5ecc0701251206n157bc1dv4a82b08efab0180e@mail.gmail.com>
Message-ID: <db6b5ecc0701260147p3cf7dde3g5ff534e25169159@mail.gmail.com>

On 1/25/07, Fernando Perez <fperez.net at gmail.com> wrote:

> Scratch that.  When I said 'last piece' above, it's because I'd given
> up on the possibility of doctesting code that spawns subprocesses, due
> to the fact that their stdout file descriptors are NOT the sys.stdout
> of the original process.  This was stated in the 'limitations' section
> here:
>
> http://projects.scipy.org/ipython/ipython/browser/ipython/branches/saw/ipython1/test/ipdoctest.py
>
> I now realized that there /is/ a solution to this problem using
> pexpect, so before I commit the lot I'll solve this problem as well.
> This will really leave us with a system where it will be /very easy/
> to add doctests for pretty much any of ipython's functionality.  I'm
> sure there will still  be a few dark corners, but I'm now determined
> to reduce that count to zero.

Done.  We now support 3 types of doctests:

1. Plain python ones (that's just what the doctest module does)
2. IPython-syntax ones, but which do NOT need direct access to the OS.
 Since these can be run in-process, just doing a syntax conversion
(ipython->python) is enough.

And as of today,

3. IPython examples that call aliases, system calls, use !cmd, etc.
These MUST be run in a separate process via pexpect, so we can access
the underlying file descriptors for output capture.  In this mode we
lose a few things, like the doctest sophisticated exception analysis,
but by and large this works great.  Doctests with exceptions or
fancier needs can always be coded in mode #2 above, so I don't see
this as any major limitation.

  These doctests simply need to be marked with

  # ipython-doctest: EXTERNAL


The code needs much cleanup, which I'll do as soon as I catch a
breather and will then commit it.

It's worth noting that unfortunately none of this works under Win32,
because there is no equivalent of pexpect there.  The author seems to
be working on a win32 port, but nothing exists yet.  Since pexpect is
a critical component of this, the test suite will simply not work
under win32.  Patches to at least induce more graceful failure will be
welcome (I have no clue how badly the current code will explode).

Linux, BSD, OSX and related should all be fine (if not, I'll consider
that a bug).

Cheers,

f