[IPython-dev] PyMAD/IPython which list updated

[Frédéric Mantegazza]

Hello,

Here is an updated version of our which list for IPython. I attached both 
text (reStructured) and html versions.

Cheers,

-- 
   Frédéric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20050318/f5319e84/attachment.html>
-------------- next part --------------
======================================================================
This document describes the interactions needed by PyMAD with IPython.
======================================================================

:Authors: 
    Frédéric Mantegazza, 

:Version: $Rev: 1291 $

1) Introduction
---------------

The philosophy of **PyMAD** is to give to the user a high-level set of tools to
drive a spectrometer. The idea is to directly make some internal python
objects available to the user, and let him combine them.

So, The final user just calls some methods of a few high-level objects to drive
the spectrometer. This as the advantage to make all the python scripting stuff
available, to build some new high level tools.

As we also need a system which can be used from several places, we use a
client/server framework, with **Pyro**. **Pyro** is a distributed objects
server. It just make some remote python objects available to a client as if they
where local.

2) User interaction
-------------------

To avoid the need for the final user to write python scripts and run them to do
something, we need a simple working environment which gives the possibility to
interactively use the server objects. That's where **IPython** solves lots of
problems !

**IPython** is an enhanced python shell. It let the user runs python code, but
has many powerfull features :

- history, even between sessions
- colorized and customizable traceback
- code completion
- magic commands
- much more...
 
**IPython** is on the client-side of **PyMAD**. In fact, there is a special
client which connects to the remote objects of **PyMAD**, launch an embbeded
**IPython** instance, and make the remote objects available to the user, in the
global namespace of **IPython**. This way, the user can drive the spectrometer
through the methods of these objects, or can build complex scripts to do complex
actions.

**PyMAD** also use the magic commands to make a simple command interpreter.
The magic functions use TPG (Toy Parser Generator), a easy-to-use parser based
on regexp. These MAD-like commands are for users which don't know about python,
but also to make shortcuts, to avoid the need to write several lines of normal
python code to do some complex but repetitive tasks.

One important point is that **PyMAD** can understand both syntax, which can be
combined. Most of the time, simple commands will be used, but python code can
be more powerfull to do expert measures (with automatic feedback interaction
according to the results), or to prototype a new complex command.

3) **IPython** needs
--------------------

In order to give users all these powerfull features, **PyMAD** needs to
interacts very closely with **IPython**. In the actual state of IPython we patch
some internal classes, by rebinding methods to custom ones. This is not very
clean, and can lead to problems if internal structures of new ipyton releases
change.

So, here is the main **PyMAD** interactions needed:

1. Catch custom *PyMADError* exceptions (now done with rebinding
   IPython.iplib.InteractiveShell.runcode() method), **with the possibility to
   get the inital text code involved in the exception**. For the moment, in the
   runcode() method, we only get the code object, from which it is impossible to
   retreive the text. Here is the code used::

    def runcode(self, code_obj):
        """Execute a code object.
    
        When an exception occurs, self.showtraceback() is called to display a
        traceback."""
        log = Logger()
        message = MessageLogger()
    
        # Set our own excepthook in case the user code tries to call it
        # directly, so that the IPython crash handler doesn't get triggered
        old_excepthook,sys.excepthook = sys.excepthook, self.excepthook
        try:
            try:
                exec code_obj in self.locals
            finally:
                # Reset our crash handler in place
                sys.excepthook = old_excepthook
    
        except SystemExit, message:
            if str(message)=='IPythonExit':
                raise
            else:
                self.resetbuffer()
                self.showtraceback()
                warn( __builtin__.exit,level=1)
                
        # We just add this few lines
        #except AttributeError, message:
            #print message
            # TODO: Use the same syntax (color) as IPython
            
        except PyMADError, exc:
            message.error(exc.standardStr())
            #log.exception("Console execution error")
       
        except Pyro.errors.ConnectionClosedError:
            message.critical("Pyro connexion closed")
            log.exception("Console execution error")
            # TODO: get the object and call rebindURI()
    
        except:
            self.showtraceback()
        else:
            if code.softspace(sys.stdout, 0):
                print

2. Add some new matchers for completion. As **PyMAD** uses remote objects,
   completion only shows the client Pyro proxy. So we added a new matcher by
   adding a IPython.iplib.MagicCompleter.proxy_matches() method, and insert this
   matcher in ipshell.IP.Completer.matchers list. The new matcher get the object
   (from the text param), call a special method on this object which returns all
   available attributes (in fact, only these we want to show to the user). Give
   the possibility to return all matchers, or only the no None first. Here is
   the code used::

    def proxy_matches(self, text, state):
        """ Get the attribute of a remove Pyro object.
        """
        log = Logger('client')
    
        # Another option, seems to work great. Catches things like ''.<tab>
        m = re.match(r"(\S+(\.\w+)*)\.(\w*)$", text)
    
        if not m:
            return []
        expr, attr = m.group(1, 3)
        matches = []
        try:
            object = eval(expr, self.namespace)
            if isinstance(object, Pyro.core.DynamicProxyWithAttrs):
                words = object.getAvailableAttributes()
                #if hasattr(object,'__class__'):
                    #words.append('__class__')
                    #words = words + get_class_members(object.__class__)
                matches = []
                n = len(attr)
                if words:
                    for word in words:
                        if word[:n] == attr and word != "__builtins__":
                            matches.append("%s.%s" % (expr, word))
    
        except NameError:
            pass
            
        except Pyro.errors.ConnectionClosedError:
            log.error("Connexion closed")
            object.adapter.rebindURI() # Should be moved to runcode()
            matches = [""]
    
        return matches
    
    ipshell.IP.Completer.proxy_matches = new.instancemethod(proxy_matches,
                                                            ipshell.IP.Completer)
    ipshell.IP.Completer.matchers.insert(0, ipshell.IP.Completer.proxy_matches)

3. In the same way as matchers, get the docstring from the remote object instead
   of the client one when using 'object?' syntaxe. This could be done on the
   same idea: calling a special method on the object, method returning the doc
   of our remote object).

4. New exception handler. Here, the idea is to be able to present different kind
   of exceptions in different ways. Some will only print a simple message, some
   others will print the entire traceback (maybe a modified traceback).

5. Prevent objects from beeing deleted by *del* keyword [1]_.

6. Prompt modification at run-time [2]_.
 
7. Access to the command-line interpreter, to have **IPython** interprets code
   has if it was entered through keyboard (ie make difference between magic
   commands and normal python code).

.. [1] Can be done with pre-filters. Have to be tested, but should work.
.. [2] Can be done with prompt_specials_color dict. Have to be tested,
       but should work.