[IPython-dev] refactoring magics

[Greg Wilson]

Hi,
Matt Davis and I have written a small magic that allows people to enter 
a regular expression and a few lines of text, and displays the matches 
in that text.  For example, if a user enters:

     %%regex ab+
     There are about three matches
     in all of this text: xab and xabbbbbyz are two of them

then the 'ab' in 'about', the 'ab' in 'xab', and the 'abbbbb' in 
'xabbbbbyz' are highlighted in the output.  (see 
https://github.com/gvwilson/regexmagic for the magic).  We'd like to use 
this to teach regular expressions in Software Carpentry in place of 
http://regexpal.com, but there's a problem.  Right now, this regular 
expression:

     \d{4}

doesn't work, because IPython interprets '{4}' as "expand the macro 
named '4'", which results in an empty string.  We can get around it by 
doubling up the parens and writing it as:

     \d{{4}}

but our main reason for creating this tool in the first place is to 
reduce the number of things we need to explain away when teaching novices.

An alternative is to refactor the code that runs magics so that they 
have both the original text (without macro expansion) and either the 
expanded text or a way to get it.  The key pieces of code are 
run_line_magic and run_cell_magic in IPython/core/interactiveshell.py.  
It's easy enough to change these to use the raw line instead of the 
expanded line, but I'd welcome suggestions on how to allow magic authors 
to get the expansions. Options I see are:

1. Leave it completely in the magic writer's hands.  I don't like this, 
because it requires magic writers to know that they need to go down 
exactly three stack frames in order to do expansion properly. (Right 
now, both run_line_magic and run_cell_magic need to go exactly two 
frames, no more and no less, to expand variables properly.)

2. Do the expansion in run_line_magic and run_cell_magic and pass in 
both the raw line and the expanded line.  This is the smallest change to 
interactiveshell.py, but requires changes to the signatures of magic 
functions (they need an extra parameter).

3. Provide an API function that users can call inside their magic that 
takes the raw line and returns the expanded version if they want it.  
The problem is, this function would then have to somehow know how far up 
the stack to go to get to the right frame (unless we def'd it in the 
calling code and passed it into the magic, in which case we might as 
well use option #2).

I'd welcome suggestions on which route people think is best, or whether 
there's a better way that I'm missing.
Thanks,
Greg