[IPython-dev] Musings: syntax for high-level expression of parallel (and other) execution control

[Fernando Perez]

Hi Darren,

On Fri, Sep 4, 2009 at 5:05 AM, Darren Dale <dsdale24 at gmail.com> wrote:
> On Fri, Sep 4, 2009 at 4:31 AM, Fernando Perez<fperez.net at gmail.com> wrote:
>> # This is the actual code of the decorator-based loop:
>> def loop_deco():
>>    results = [None]*count
>>
>>    @for_each(range(count))
>>    def loop(i):
>>        results[i] = do_work(data, i)
>>
>>    return summarize(results, count)
>
> If I have understood correctly, this example passes around a lot more
> data than is necessary. for_each passes the whole data list to
> do_work, which picks off the bit its looking for. I guess this doesn't
> seem like an intuitive interface to me, plus its expensive. What is
> being parallelized is:
>
> results = [None]*len(data)
> for i, d in enumerate(data):
>    results[i] = do_work(d)
>
> Why not use a simple function instead of a decorator:
>
> results = for_each(data, do_work)
>
> ?

Indeed, in this case 'data' is passed around, but I wrote it that way
only to keep it as close as possible to the syntax of the original
example I started from in the Ars review of Snow Leopard and the Grand
Central Dispatch API.  In practice, you'd obviously implement things
differently.

But my main point was not about the parallelization of a loop, but
rather about the basic idea of using a decorator to swap the execution
context of a bit of code for another one, be it a thread, a remote
ipython engine, a GPU, a tracing utility, a profiler, a cython JIT
engine or anything else.  Perhaps I chose my example a little poorly
to get that point across, sorry if that was the case.  It would be
good to come up with more obviously useful and unambiguous examples of
this, I'd love it if we generate some interesting discussion here.
I'll continue playing with this idea in my copious spare time, until
Brian's patience with my lack of code review in the last few days runs
out ;)

Does that help clarify my intent?  Thanks for the interest and feedback!

Cheers,

f