From fperez at colorado.edu Thu Apr 1 04:48:43 2004 From: fperez at colorado.edu (Fernando Perez) Date: Thu, 01 Apr 2004 02:48:43 -0700 Subject: [IPython-dev] Commits to CVS, fixes, and shell access (RFC) Message-ID: <406BE57B.1080707@colorado.edu> Hi all, I'm trying to catch up a bit on ipython matters. I just committed some bugfixes to CVS for various problems (among others, Solaris crashes caused by the curses module and crashes when users closed sys.stdin/out/err). I also made some 'improvements' to the shell access mechanism, spurred by the recent discussions on the matter. I'll try to say more about that discussion in a couple of days, but I'd appreciate some feedback on the changes I made. Recapping, people have always wanted easy access to the shell, with a way to capture the result for further python use. I added @sc as a magic to do this in January, and it's flexible with options. But at the time, the '!!' syntax was also liked, and I realized that a very common usage mode is simply to want to loop over shell output in a line-oriented manner (as most unix tools are designed). So I decided today to add (it's in CVS for you to play with) the following: - @sx, a new magic to execute in the shell any command. This one, in contrast to @sc, doesn't require a variable name. Instead it _returns_ the output, split as a list on '\n', to the normal ipython namespace. Since ipython remembers all output as _N, Out[N], and the last three as _,__ and ___, I realized that this might be a very useful thing to have, leveraging the existing output storage mechanism. - !!cmd is a new syntax for '@sx cmd'. It makes it very easy to get a command to the shell remembering and typing as little as possible. Examples: In [5]: !ls IPython-0.5.0.tar.gz pyamazonpres.tgz In [6]: sx ls Out[6]: ['IPython-0.5.0.tar.gz', 'pyamazonpres.tgz'] In [7]: !!ls # identical to [6] above Out[7]: ['IPython-0.5.0.tar.gz', 'pyamazonpres.tgz'] In [8]: sc a=ls # nothing is printed, results in a In [9]: print a IPython-0.5.0.tar.gz pyamazonpres.tgz In [10]: sc -l a=ls # results split as a list, then stored in a In [11]: print a ['IPython-0.5.0.tar.gz', 'pyamazonpres.tgz'] In summary, you can now in ipython access the shell via (in addition to any aliases you may have defined): - !cmd -> execute cmd, return None (output of cmd is printed) - !!cmd, @sx cmd -> execute cmd and return output of cmd, split on '\n', as a python list. - @sc [options] var=cmd -> execute cmd and store its output in var. Returns None. Option -l causes a split on '\n' before storing into var, and option -v causes the contents to be printed. @sc is the one with the most flexibility for shell capture, while also requiring more typing and remembering options. I see myself mostly using !cmd as today for quick shell access, and !!cmd when I want to pick up for example some filenames for further manipulation through python. !!cmd is a natural enough extension of !cmd that I think I can remember it easily. I'd appreciate feedback on these changes, esp. on whether they enhance ipython's functionality for shell access without detracting from its main python nature. I'll try to address more radical (but optional) enhancements which others have proposed in another message in a few days. Also, any reports of breakage (or successful improvements) from the recent CVS commits would be very welcome. Cheers, f. From pythondev-dang at lazytwinacres.net Thu Apr 1 11:26:39 2004 From: pythondev-dang at lazytwinacres.net (pythondev-dang) Date: Thu, 01 Apr 2004 21:26:39 +0500 Subject: [IPython-dev] [IPython-user] Commits to CVS, fixes, and shell access (RFC) Message-ID: <20040401162639.14508.qmail@server265.com> This sounds great--similar to what I did for my "grep" shell access, but generalized. (I didn't look into storing the results in "hidden" variables, but that looks like a good idea also.) I'm having troubles accessing the CVS repository from here, so I won't be able to give usage feedback until after this weekend at the soonest. I've a question though. My grep shell extension would capture stdout and stderr separately, and wouldn't return stderr, but just printed it. It would then return whatever the command *had* sent to stdout (or None if nothing). What does @sc do for this? Thanks, --dang -------Original Message------- > From: Fernando Perez > Subject: [IPython-user] Commits to CVS, fixes, and shell access (RFC) > Sent: 01 Apr 2004 14:48:43 > > Hi all, > > I'm trying to catch up a bit on ipython matters.??I just committed some > bugfixes to CVS for various problems (among others, Solaris crashes caused by > the curses module and crashes when users closed sys.stdin/out/err). > > I also made some 'improvements' to the shell access mechanism, spurred by the > recent discussions on the matter.?? I'll try to say more about that discussion > in a couple of days, but I'd appreciate some feedback on the changes I made. > > Recapping, people have always wanted easy access to the shell, with a way to > capture the result for further python use.??I added @sc as a magic to do this > in January, and it's flexible with options.??But at the time, the '!!' syntax > was also liked, and I realized that a very common usage mode is simply to want > to loop over shell output in a line-oriented manner (as most unix tools are > designed). > > So I decided today to add (it's in CVS for you to play with) the following: > > - @sx, a new magic to execute in the shell any command.??This one, in contrast > to @sc, doesn't require a variable name.??Instead it _returns_ the output, > split as a list on '\n', to the normal ipython namespace.??Since ipython > remembers all output as _N, Out[N], and the last three as _,__ and ___, I > realized that this might be a very useful thing to have, leveraging the > existing output storage mechanism. > > - !!cmd is a new syntax for '@sx cmd'.??It makes it very easy to get a command > to the shell remembering and typing as little as possible. > > Examples: > > In [5]: !ls > IPython-0.5.0.tar.gz??pyamazonpres.tgz > > In [6]: sx ls > Out[6]: ['IPython-0.5.0.tar.gz', 'pyamazonpres.tgz'] > > In [7]: !!ls??# identical to [6] above > Out[7]: ['IPython-0.5.0.tar.gz', 'pyamazonpres.tgz'] > > In [8]: sc a=ls??# nothing is printed, results in a > > In [9]: print a > IPython-0.5.0.tar.gz > pyamazonpres.tgz > > In [10]: sc -l a=ls??# results split as a list, then stored in a > > In [11]: print a > ['IPython-0.5.0.tar.gz', 'pyamazonpres.tgz'] > > > In summary, you can now in ipython access the shell via (in addition to any > aliases you may have defined): > > - !cmd -> execute cmd, return None (output of cmd is printed) > > - !!cmd, @sx cmd -> execute cmd and return output of cmd, split on '\n', as a > python list. > > - @sc [options] var=cmd -> execute cmd and store its output in var.??Returns > None.??Option -l causes a split on '\n' before storing into var, and option -v > causes the contents to be printed. > > @sc is the one with the most flexibility for shell capture, while also > requiring more typing and remembering options. > > > I see myself mostly using !cmd as today for quick shell access, and !!cmd when > I want to pick up for example some filenames for further manipulation through > python.??!!cmd is a natural enough extension of !cmd that I think I can > remember it easily. > > I'd appreciate feedback on these changes, esp. on whether they enhance > ipython's functionality for shell access without detracting from its main > python nature.??I'll try to address more radical (but optional) enhancements > which others have proposed in another message in a few days. > > Also, any reports of breakage (or successful improvements) from the recent CVS > commits would be very welcome. > > Cheers, > > f. > > _______________________________________________ > IPython-user mailing list > IPython-user at scipy.net > http://scipy.net/mailman/listinfo/ipython-user -------Original Message------- From Fernando.Perez at colorado.edu Thu Apr 1 15:13:20 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Thu, 01 Apr 2004 13:13:20 -0700 Subject: [IPython-dev] Re: [IPython-user] Commits to CVS, fixes, and shell access (RFC) In-Reply-To: <20040401162639.14508.qmail@server265.com> References: <20040401162639.14508.qmail@server265.com> Message-ID: <406C77E0.4060103@colorado.edu> pythondev-dang wrote: > This sounds great--similar to what I did for my "grep" shell access, > but generalized. (I didn't look into storing the results in "hidden" > variables, but that looks like a good idea also.) Indeed, your grep (with the caveats below) would now just be one more case of: !!grep regexp files For example: !!grep assert *.py would work, since all wildcards propagate down directly to the shell. > I'm having troubles accessing the CVS repository from here, so I > won't be able to give usage feedback until after this weekend > at the soonest. > > I've a question though. My grep shell extension would capture > stdout and stderr separately, and wouldn't return stderr, but > just printed it. It would then return whatever the command > *had* sent to stdout (or None if nothing). What does @sc do > for this? I was lazy and just used commands.getoutput(), which lumps them together. But what you are doing is the right thing, IMO. I'll recode it later and recommit to CVS, so that both @sc and @sx function as follows: - stdout is captured, either to the named var for @sc or to the normal ipython return value for @sx. - stderr, if present, is just printed to the screen. How does this sound? Thanks for the feedback. Cheers, f. From fperez at colorado.edu Fri Apr 2 01:37:42 2004 From: fperez at colorado.edu (Fernando Perez) Date: Thu, 01 Apr 2004 23:37:42 -0700 Subject: [IPython-dev] Re: [IPython-user] Commits to CVS, fixes, and shell access (RFC) In-Reply-To: <20040401162639.14508.qmail@server265.com> References: <20040401162639.14508.qmail@server265.com> Message-ID: <406D0A36.2070904@colorado.edu> pythondev-dang wrote: > This sounds great--similar to what I did for my "grep" shell access, > but generalized. (I didn't look into storing the results in "hidden" > variables, but that looks like a good idea also.) These variables aren't really that 'hidden': In[N] and Out[N] are always there for you, and hopefully the names of the prompts hint at their existence. The magic stuff is that _N==Out[N], and _iN==In[N], and _,__ and ___ hold always Out[-1], Out[-2] and Out[-3]. When using ipython for scientific work, this kind of caching is incredibly useful. > I've a question though. My grep shell extension would capture > stdout and stderr separately, and wouldn't return stderr, but > just printed it. It would then return whatever the command > *had* sent to stdout (or None if nothing). What does @sc do > for this? Done. I've uploaded changes so that now stderr is separately handled: In [5]: d lrwxrwxrwx 1 fperez 3 Apr 1 21:49 bar -> foo ### broken link -rw-r--r-- 1 fperez 124 Oct 6 13:25 die.py -rwxr-xr-x 1 fperez 9988 Jul 7 2003 divc* -rwxr-xr-x 1 fperez 11671 Jul 7 2003 divf* In [6]: !!grep ff * grep: bar: No such file or directory Out[6]: ['Binary file divc matches', 'Binary file divf matches'] Note stderr message printed to console before the output is returned. Thanks again for the feedback. Hopefully this will provide some relief for those interested in ipython having better system integration. Cheers, f From a.schmolck at gmx.net Thu Apr 29 15:53:02 2004 From: a.schmolck at gmx.net (Alexander Schmolck) Date: Thu, 29 Apr 2004 20:53:02 +0100 Subject: [IPython-dev] minor ipython.el news Message-ID: Hi, After promising to look into the problems with CVS python-mode and my original patch, I've lapsed into long silence -- sorry about that but I've first discovered that I couldn't quickly determine the problem and then I unfortunately fell ill for several weeks and have been really busy ever since (so unless someone else has the time to work out how the new python.el screws up C- -, don't expect a new patch any time soon; the original one I submitted unfortunately didn't receive any attention and since the old python.el works OK for me, I don't really have some spare hours to sacrifice for writing a patch that might be ignored again; also other than breaking py-up-exception etc. there seem to be no problems in applying the patch to the CVS python-mode.el). I have however recently discovered in an email exchange with a ipytho/emacs user that the version of ipython.el in ipython-emacs.tgz is older than the one I use myself (I've made some slight improvements to the traceback regexp and added some dirtrack stuff, which might actually be redundant). Anyway, here is the newest version: -------------- next part -------------- A non-text attachment was scrubbed... Name: ipython.el Type: application/emacs-lisp Size: 13242 bytes Desc: not available URL: -------------- next part -------------- BTW, I think it would be nice if ipython's traceback output would be a bit more uniform, i.e. always match this (or some other emacs-regexp) pattern: \(^[^\t ].+?\.py\).*\n +[0-9]+[^\00]*?\n-+> \([0-9]+\) + Syntax errors for example currently produce output that doesn't match the abovee, so it's not possible to automatically jump to the right line in the source code (and emacs regexp limitations + the way traceback parsing is hardcoded in python-mode.el make it impossible to handle them seperately). 'as P.S: There has been some effort in supplying shell like functionality to ipython of late (not a bad idea in principle, I think, the fact that python doesn't have syntax extension facilities is severely limiting here) -- one thing that really often annoys me is the behavior of @run (which not only does not perform any shell like globbing, but also doesn't seem to allow actually passing parameters with spaces in them since there is no quoting or escaping mechanism as far as I'm aware). Are there any plans to add globbing and quoting or is there already an easy way to get the equivalent of > ./script.py --title 'some title' using @run?