[Python-ideas] Decorators on loops

[Chris Angelico]

On Thu, Jan 9, 2014 at 1:23 AM, Enric Tejedor <enric.tejedor at bsc.es> wrote:
> Yes, perhaps when a loop had a decorator, the loop body could be
> encapsulated and compiled as a function (similar to the "loop" function
> I wrote), and that function object would be received by the decorator,
> along with an iterable object that represents the iteration space. All
> this would be hidden from the programmer, who would only decorate a
> regular loop.
>

The biggest problem with that kind of magic is scoping. Look at this:

def func():
    best = 0
    for x in range(10):
        val = long_computation(x)
        if val > best: best = val
    return best

(Granted, this can be done with builtins, but let's keep the example simple.)

If the body of the loop becomes a new function, there needs to be a
nonlocal directive to make sure 'best' references the outer one:

def func():
    best = 0
    @parallelize(range(10))
    def body(x):
        nonlocal best
        val = long_computation(x)
        if val > best: best = val
    return best

This syntax would work, but it'll raise UnboundLocalError without the
nonlocal declaration. Since Python tags non-local variables (as
opposed to C-like languages, which tag local variables), there's no
easy way to just add another scope and have it function invisibly. Any
bit of magic that creates a local scope is going to cause problems in
any but the simplest cases. Far better to force people to be explicit
about it, and then the rules are clearer.

Note that the parallelize decorator I use here would be a little
unusual, in that it has to actually call the function (and in fact
call it multiple times), and its return value is ignored. This would
work, but it might confuse people, so you'd want to name it something
that explains what's happening. It wouldn't be hard to write, in this
form, though - it'd basically just pass the iterable to
multiprocessing.Pool().map().

However, the example I give here wouldn't work (at least, I don't
think it would) with multiprocessing, because external variable scopes
would be duplicated, not shared, between processes. So once again,
you'd have to write your code with parallelization in mind, rather
than simply stick a decorator on a loop and have it fork out across
processes.

ChrisA