[IPython-dev] Musings: syntax for high-level expression of parallel (and other) execution control

[Fernando Perez]

On Sat, Sep 5, 2009 at 7:51 AM, Darren Dale <dsdale24 at gmail.com> wrote:
> I slept on this last night, and this morning when I reread your
> original post I had to laugh at myself for taking so long and such a
> winding route to understanding and processing all the points you
> raised. So thanks for being patient with me. You guys can continue
> discussing Einstein, I'll sit here and scratch my stomach.

Sorry for laughing, but after scipy I stayed in SoCal for a few days
with the family, and visited the San Diego zoo and wild animal park,
and saw lots of primates. So the image of you scratching your stomach
while more evolved behavior takes place next door is rather humorous
;)

But don't beat yourself up: while there's nothing complex about what
we're talking about, it is a slightly unusual usage of the language,
so it's natural to do a double-take with it.  I have the benefit of
having worried about this problem for a long time, but it took me
*many* tries to understand how to fit the pieces together.  And I had
the advantage of lots of help along the way:

- the first 'click' was a conversation with Eric Jones at Berkeley in
late 2007, where he pointed out really how 'with' could be used for
execution management, which they are doing a lot of with Enthought's
context library (BlockCanvas, I think?)

- in March 2008, William Stein implemented for Sage @interact at the
sprint at Enthought, using the 'call and consume' approach to the
decorated function.  On the flight back from that, I implemented for
ipython the trick using 'with', which worked but was so nasty that I
never really pursued it.

- in September 2008 at Scipy'08 I had a long talk about the problem
with Alex Martelli on whether extending the context manager protocol
with a __execute__ method to control the actual execution of the code
would be feasible.  This conversation was very enlightening, even
though it made it fairly clear that the 'with' approach was probably
doomed in the long run.  Alex pointed out very clearly a few of the
key issues regarding scoping that helped me a lot.

- then at SciPy'09 I had a talk with Peter Norvig again about the same
problem, so I got the whole thing back in my head.

- and finally, John Siracusa's review at Ars Technica about Apple's
work with anonymous blocks and Grand Central Dispatch make the whole
thing click.

As you can see, if you're slow for taking a day to put it together,
there's simply no hope for me: it took me almost 2 years, and I needed
the help of some of the very brightest people in the python world to
push me along.  They are the ones who did all the thinking and deserve
the credit, I was just thick enough never to understand the ideas
until now!

> Do you think there is the possibility of building on this mechanism? I
> mean, you have a working demonstration using features that are already
> built into the language. Do you think it would be possible/worth
> looking into adding syntax to the python language that provided
> anonymous code blocks without having to define functions and decorate
> them (in the general sense, using either @deco or deco(foo))? I'm
> thinking of something similar to the "with" syntax that creates a
> scope and allows you to manipulate it. Is there any chance that the
> GIL issue could be addressed using such a mechanism?

I think there is, but we should first explore  it  more deeply.  The
reason I'm happy is that I see lots of potential here, and if we find
solid, important uses, it will be *much* easier then to make a case to
Guido and python-dev for a core language change in the future.  Guido
is careful with syntax (fortunately!), but he does listen if a tried
and tested usage is presented, where the cost of new syntax is offset
by genuine benefits.  But fortunately we can get almost everything we
want today, even if with ugly syntax.

I say almost because with Python 2.x there is at least one real
annoyance: the inability to rebind non-local (but not global) names in
an inner scope.  This was fixed with the 'nonlocal' keyword in 3.0,
but for 2.x the following won't work:

def execute(func):
    return func()

def simple(n):
    s = 0.0

    @execute
    def block():
        for i in range(n):
            s += i**2

    return s

because you get an unbound local error:

In [13]: run simple

[...]

/home/fperez/research/code/contexts/simple.py in block()
     15     def block():
     16         for i in range(n):
---> 17             s += i**2
     18
     19     return s

UnboundLocalError: local variable 's' referenced before assignment
WARNING: Failure executing file: <simple.py>

In Python 3, this works great:

In [17]: !python3.1 simple.py
285.0

with the only change being the addition of

nonlocal s

to the block() definition.

I guess this will be a motivation to move to 3.x if this idea turns
out to be really useful...

In any case, we'll need to see if real-world uses of this trick really
pay off, develop some good libraries of decorators that pre-package
good functionality, and then we can consider syntax extensions to the
language itself.  But for now, I think it is still a little premature
to look in that direction, until experience shows us better what all
the various problems and patterns of use need to be.

Regards,

f