[IPython-dev] Musings: syntax for high-level expression of parallel (and other) execution control

[Brian Granger]

Hi,

Sorry I missed this one.  This is a very nice idea!  With respect to using
decorators like this,
I like the "you can't do that....oh, wait, yes you can" feel of it.  Very
creative.

I don't have time to read through the entire thread, but I did skim most of
it and have some
comments.

One of the big issues we have run into with the parallel computing stuff in
IPython,
is that it is very tough to avoid typing code in strings.  Then you send the
string to
a different execution context and it does exec in its namespace.  Here is an
example:

mec.push(dict(a=10))
mec.execute('b = 2*a')

Fernando's original work on using "with" for stuff like this was to try to
get something
that allowed you to just type your code:

a=10
with remote_engine:
  b = 2*a

While this is much nicer, you do need some code to grab the value of a from
the
top-level and push it over to the execution context.  I guess you could also
pass it to the context though like remote_engine(dict(a=10))

With Fernando's new idea, this example would read:

@remote(dict(a=10)
def foo():
    b = 2*a

While I wish we could get rid of the 2 line header (@remote...def foo), this
is a pretty
nice way of abstracting this.  No code in strings, and a simple way of
passing in variables.
My only complaint is that is a bit unexpected that this actually declares
and *calls* the function!
But renaming remote to call_remote or something would help.

I am going to start working on the parallel stuff in about 2 weeks, and I
will revisit this then.  It shouldn't be
too difficult to implement some nice things using this pattern.

Cheers,

Brian





On Fri, Sep 4, 2009 at 1:31 AM, Fernando Perez <fperez.net at gmail.com> wrote:

> Hi all,
>
> I know I should have been hard at work on continuing the branch review
> work I have in an open tab of Brian's push, but I couldn't resist.
> Please bear with me, this is  a bit technical but, I hope, very
> interesting in the long run for us...
>
> This part of Ars Technica's excellent review of Snow Leopard:
>
> http://arstechnica.com/apple/reviews/2009/08/mac-os-x-10-6.ars/13
>
> shows how Apple tackled the problem of providing civilized primitives
> to express parallellism in applications and a mechanism to make useful
> decisions based on this information.   The idea is to combine a
> kernel-level dispatcher (GCD), a beautiful extension to the C language
> (yes, they extended C!) in the form of anonymous code blocks, and an
> API to inform GCD of your parallelism breakup easily, so GCD can use
> your intentions at runtime efficiently.  It's really a remarkably
> simple, yet effective (IMHO) combination of tools to tackle a thorny
> problem.
>
> In any case, what does all this have to do with us?  For a long time
> we've wondered about how to provide the easiest, simplest APIs that
> appear natural to the user, that are easy to convert into serial
> execution mode trivially (with a simple global switch for debugging,
> NOT changing any actual code everywhere), and that can permit
> execution via ipython.  A while ago I hacked something via 'with' and
> context  managers, that was so horrible and brittle (it involved stack
> manipulations, manual source introspection and exception injection)
> that I realized that could never really fly for production work.
>
> But this article on GCD got me trying my 'with' approach again, and I
> realized that syntactically it felt quite nice, I could write python
> versions of the code examples in that review, yet the whole 'with'
> mess killed it for me.  And then it hit me that decorators could be
> abused just a little bit to get the same job done [1]!  While this may
> be somewhat of an abuse, it does NOT involve source introspection or
> stack manipulations, so in principle it's 100% kosher, robust python.
> A little weird the first time you see it, but bear with me.
>
> The code below shows an implementation of a simple for  loop directly
> and via a decorator.  Both versions do the same thing, but the point
> is that by providing such decorators, we can *trivially* provide a
> GCD-style API for users to express their parallelism and have
> execution chunks handled by ipython remotely.
>
> It's obvious that such decorators can also be used to dispatch code to
> Cython, to a GPU,  to a CorePy-based optimizer, to a profiler, etc.  I
> think this could be a useful idea in more than one context, and it
> certainly feels to me like one of the missing API/usability pieces
> we've struggled with for the ipython distributed machinery.
>
> Cheers,
>
> f
>
> [1] What clicked in my head was tying the 'with' mess to how the Sage
> notebook uses the @interact decorator to immediately call the
> decorated function rather than decorating it and returning it.  This
> immediate-consumption (ab)use of a decorator is what I'm using.
>
> ### CODE example
>
> # Consider a simple pair of 'loop body' and 'loop summary' functions:
> def do_work(data, i):
>    return data[i]/2
>
> def summarize(results, count):
>    return sum(results[:count])
>
> # And some 'dataset' (here just a list of 10 numbers
> count = 10
> data = [3.0*j for j in range(count) ]
>
> # That we'll process.  This is our processing loop, implemented as a
> regular
> # serial function that preallocates storage and then goes to work.
> def loop_serial():
>    results = [None]*count
>
>    for i in range(count):
>        results[i] = do_work(data, i)
>
>    return summarize(results, count)
>
> # The same thing can be done with a decorator:
> def for_each(iterable):
>    """This decorator-based loop does a normal serial run.
>    But in principle it could be doing the dispatch remotely, or into a
> thread
>    pool, etc.
>    """
>    def call(func):
>        map(func, iterable)
>
>    return call
>
> # This is the actual code of the decorator-based loop:
> def loop_deco():
>    results = [None]*count
>
>    @for_each(range(count))
>    def loop(i):
>        results[i] = do_work(data, i)
>
>    return summarize(results, count)
>
> # Test
> assert loop_serial() == loop_deco()
> print 'OK'
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20090908/4523dcb5/attachment.html>