[IPython-dev] iPipe and Hotwire Shell

[Walter Dörwald]

Colin Walters wrote:

> [originally sent with wrong From:]
> 
> Hi,
> 
> Recently I came across IPipe
> (http://ipython.scipy.org/ipython/ipython/wiki/UsingIPipe), and
> noticed a lot of similarity with the HotwirePipe language:
> http://code.google.com/p/hotwire-shell/wiki/HotwireArchitecture
> 
> Looking at the ipipe.py source, we actually have a number of things we
> could share.  First, for Hotwire we had to invent our own extended
> types such as File
> (http://hotwire-shell.googlecode.com/svn/trunk/hotwire/sysdep/fs.py),
> because Python sadly doesn't include one.  For example, on Windows you
> want to know whether or not the file is hidden, on modern Linux
> systems you want to use GnomeVFS to get extended file metadata like
> MIME type and file thumbnail, etc.  Basically we want something a lot
> closer to Java's File object.  It looks like ipipe defines a lot of
> its own classes like this.

Currently ipipe's ifile class is a thin wrapper around Jason Orendorff's
path class (actually it's a subclass). The only thing ifile does is add 
the properties that will we used when displaying the output and 
implementing the method __xattrs__() which tells ipipe which attributes 
to display. The basic file functionality (getting file metadata, listing 
directories etc.) is inherited directly from path.path.

> 
> Secondly, iPipe has some operators like "iwalk" which are close to
> Hotwire builtins
> (http://hotwire-shell.googlecode.com/svn/trunk/hotwire/builtins/walk.py).
>  But, sharing these things leads to a the first big distinction
> between Hotwire and IPipe:
> 
> Hotwire is closer to a "system" shell built on Python, and IPipe due
> to its IPython base is more a Python toplevel where you can call out
> to the system if you want.  Practically speaking, because I'm more of
> an OS developer, I find myself wanting to call out to system binaries
> more often than not.  I could use IPython and type "!make", "!sudo rpm
> ...", etc., but it's much nicer to just type "make" and "sudo rpm
> ...".  I do prefer Hotwire's approach of just taking over the system
> namespace with improved versions, rather than having two worlds
> separated by "!".

ipipe was implemented as pure Python expressions, because at the time I 
was more familiar with Python than with IPython's commandline 
preprocessing capabilities (and still am).

> So, the main difference that is reflected here is that IPython's
> operators are really *just* Python generators, while Hotwire builtins
> are structured to take Unix-style options and string arguments.  But
> the goal is that the Hotwire builtins are just a thin convenience
> wrapper around a real API, so we could have Hotwire builtins just call
> out to a shared implementation.

This sounds indeed interesting. I've looked around the hotwire webpages 
and a few other differences I see are the following (correct me, if I'm 
wrong):

According to 
http://code.google.com/p/hotwire-shell/wiki/HotwireArchitecture hotwire 
uses threading, ipipe uses simple iterators/generator.

hotwire propably does its own command line parsing, ipipe tables just 
implement the  __or__/__ror__ magic methods. But thats a superficial 
difference, ipipe could switch to command line parsing, or hotwire might 
add the required magic.

hotwire has one output method, the rendering infrastructure where each 
class can register a specific renderer for instances of the class. ipipe 
has three "render infrastructures": idump (basically this does just a 
print), ibrowse (curses based) and igrid (wx based), but each class can 
only specify which attributes (or methods) should be output, then for 
each attribute value (or method return value) an extended version of 
repr() (that supports colored output) kicks in.

> There's also some code in Hotwire like the SysBuiltin for executing
> subprocesses that would be good to share.  I see IPipe just does
> os.popen4(self.cmd).  I remember when I thought it was that
> easy...before I discovered the horrors of Unix pseudoterminals just so
> you can force subprocesses to be line buffered, and the pitfalls of
> processing through /bin/sh.

Actually I was trying to get an asynchronous system command that outputs 
the result as a generator yielding lines, but I could never get this to 
work.

> In the longer term, I was recently thinking about trying to rebase
> Hotwire around some of the IPython infrastructure; so the HotwirePipe
> input language would be compiled to something like IPipe, which is
> itself compiled to Python.

IPipe isn't "compiled to Python", it just *is* Python, so you don't have 
to generate an ipipe expression, you could simply instantiate the ipipe 
objects directly.

> Then if you switch the Hotwire input
> language to Python, we'd use IPython to evaluate rather than our own
> custom stuff.
> 
> Your thoughts appreciated!  Personally I enjoy PyGTK ninjitsu a lot
> more than I like maintaining File APIs and Python evaluation hacks, so
> I'm hoping we can share some of that =)

Unfortunately I'm on OS X, so I can't really test hotwire, so I'm 
limited to reading source and looking at screenshots.

As Fernando stated, we can't use any of the GPL licensed parts of 
hotwire, but if I've read COPYING/COPYING.API/COPYING.UI correctly, that 
only affects the GUI parts.

Looking forward to your comments!

Servus,
    Walter