Docorator Disected

[Bengt Richter]

On Mon, 04 Apr 2005 02:13:29 GMT, Ron_Adam <radam2 at tampabay.rr.com> wrote:

>On Sun, 03 Apr 2005 23:59:51 +0200, "Martin v. Löwis"
><martin at v.loewis.de> wrote:
>
>>Ron_Adam wrote:
>>> This would be the same without the nesting:
>>> 
>>> def foo(xx):
>>>     global x
>>>     x = xx
>>>     return fee
>>> 
>>> def fee(y):
>>>     global x
>>>     return y*x
>>> 
>>> z = foo(2)(6)
>>
>>Actually, it wouldn't.
>
>Ok, yes, besides the globals, but I figured that part is obvious so I
>didn't feel I needed to mention it.  The function call works the same
>even though they are not nested functions.

I am afraid that is wrong. But be happy, this may be the key to what ISTM
is missing in your concept of python functions ;-)

What you don't seem to grok is what the def statement really does.
When you compile a def statement, you code, but it's not principally the code
that runs when you call the function that is being defined.

When you compile a def statement, you get code that MAKES the function being defined,
from precompiled pieces (one of which is a code object representing the defined function)
and optionally code for evaluating default argument expressions. This becomes part of the
code generated by compiling the def statement. (Default argument value expressions are
the simplest examples).

When you work interactively as above, the interactive loop both compiles and executes
chunks as you go, so both your def foo and def fee would be compiled AND executed.

If you put def fee inside the body of foo, you get fee def-code inside the body of foo,
but it doesn't get executed, because foo hasn't been called yet, though the foo def
has been compiled and executed interactively (and thus the name foo is bound to
the finished foo function, ready to be called or otherwise used.

So, no, it does not "work the same." In fact, it is a misconception to talk about
a nested fee as if it existed ready to call in the same way as foo. It doesn't
exist that way until the fee def is EXECUTED, producing the callable fee.

It's something like the distinction between relocatable object files in C
representing a function and machine code  representing the function in a dll.
The execution of def is like a little make operation that links the function
pieces (and in the case of Python may dynamically generate some of the pieces
with arbitrarily complex code, including decorators etc).

def is hard to grok at first, if you're coming from languages that don't
do it Python's way.

We can use dis to see the above clearly:

Let's see what the difference in code is for a simple function
with the body

    fee = 'fee string'
    return fee

and the body

    def fee():
        return 'nested fee result'
    return fee

It's going to be returning whatever fee is either way, so what we need
to compare is how fee gets bound to something:

 >>> import dis
 >>> def foo():
 ...     fee = 'fee string'
 ...     return fee
 ...
 >>> dis.dis(foo)
   2           0 LOAD_CONST               1 ('fee string')
               3 STORE_FAST               0 (fee)

   3           6 LOAD_FAST                0 (fee)
               9 RETURN_VALUE

Now we'll look for what replaces
   2           0 LOAD_CONST               1 ('fee string')
               3 STORE_FAST               0 (fee)
when we do a simple nested function and return that:


 >>> def foo():
 ...     def fee():
 ...         return 'nested fee result'
 ...     return fee
 ...
 >>> dis.dis(foo)
   2           0 LOAD_CONST               1 (<code object fee at 02EF21E0, file "<stdin>", line 2>)
               3 MAKE_FUNCTION            0
               6 STORE_FAST               0 (fee)

   4           9 LOAD_FAST                0 (fee)
              12 RETURN_VALUE

Note the MAKE_FUNCTION. That is dynamically going to take its argument(s) -- in this case
what the single LOAD_CONST put on the stack -- and leave the now ready-to-call function
on the stack (as a reference, not the whole thing of course). Now we have the function
reference instead of a reference to the string 'fee string' in the first example on the
stack, and the next step is STORE_FAST to bind the local name fee to whatever was on the stack.
In both cases, what happens next is to return (by reference) whatever fee is bound to.

So fee doesn't exist as such until the three bytecodes (LOAD_CONST, MAKE_FUNCTION, STORE_FAST)
in this example have been executed. To show that MAKE_FUNCTION is in general doing more
than moving a constant, give fee an argument with a default value expression. It could be
arbitrarily complicated, but we'll keep it simple:

 >>> def foo():
 ...     def fee(x, y=2*globalfun()):
 ...         return x, y
 ...     return fee
 ...
 >>> dis.dis(foo)
   2           0 LOAD_CONST               1 (2)
               3 LOAD_GLOBAL              0 (globalfun)
               6 CALL_FUNCTION            0
               9 BINARY_MULTIPLY
              10 LOAD_CONST               2 (<code object fee at 02EF21A0, file "<stdin>", line 2>)
              13 MAKE_FUNCTION            1
              16 STORE_FAST               0 (fee)

   4          19 LOAD_FAST                0 (fee)
              22 RETURN_VALUE

Compare to what came before MAKE_FUNCTION in the previous example. The def fee ... compiled into
code to compute the argument default value expression right there. That is not part of the fee
code, that is part of the fee-making code.

If the nested function makes use of its nested environment, there is extra work in setting up
the closure variables, so MAKE_FUNCTION is replaced my MAKE_CLOSURE, but either way the creation
of fee doesn't happen until the code compiled from the def source is executed, and if the def
code is nested, it doesn't get executed until its enclosing function is called.

Notice that def foo (compiled AND executed interactively above) didn't complain about globalfun,
which is just a name I picked. That's because foo hasn't been called yet, and the def-code to
generate fee has not been executed yet. But when we do:

 >>> foo()
 Traceback (most recent call last):
   File "<stdin>", line 1, in ?
   File "<stdin>", line 2, in foo
 NameError: global name 'globalfun' is not defined

That's not from fee, that's from trying to create a fee default argument
as part of creating fee dynamically.

Now supplying globalfun, we can make it succeed:

 >>> import time
 >>> def globalfun(): return '[%s]'%time.ctime()
 ...
 >>> foo()
 <function fee at 0x02EF802C>
 >>> foo()(111, 222)
 (111, 222)
 >>> foo()(333)
 (333, '[Mon Apr 04 22:31:23 2005][Mon Apr 04 22:31:23 2005]')
 >>> foo()(333)
 (333, '[Mon Apr 04 22:31:37 2005][Mon Apr 04 22:31:37 2005]')

Note that the times changed, because we execute foo() and therefore def fee
multiple times but if we capture the fee output as f1 and f2, the times
are captured in fee's default arg according to when def fee was executed
and then that belongs to fee and won't change when fee is called.

 >>> foo()(111, 222)
 (111, 222)
That generated a new fee with a new default time, but didn't use the default

 >>> foo()(333)
 (333, '[Mon Apr 04 22:32:48 2005][Mon Apr 04 22:32:48 2005]')
That used the default. Now we will make instances of fee and
bind them to f1 and f2, pausing a little between so the times
should be different.

 >>> f1 = foo()
(paused 8 seconds apparently)
 >>> f2 = foo()

Now calling f1 or f2 is calling different fees with different,
(but constant because we are not calling foo again) defaults:

 >>> f1(1)
 (1, '[Mon Apr 04 22:32:55 2005][Mon Apr 04 22:32:55 2005]')
 >>> f1(1)
 (1, '[Mon Apr 04 22:32:55 2005][Mon Apr 04 22:32:55 2005]')

And different, but repeated:
 >>> f2(1)
 (1, '[Mon Apr 04 22:33:03 2005][Mon Apr 04 22:33:03 2005]')
 >>> f2(1)
 (1, '[Mon Apr 04 22:33:03 2005][Mon Apr 04 22:33:03 2005]')

Now get foo to make another fee and call it without even storing it:
 >>> foo()(1)
 (1, '[Mon Apr 04 22:33:40 2005][Mon Apr 04 22:33:40 2005]')


[...]
>Today I believe I have the correct view as I've said this morning. I
>could be wrong yet again. I hope not though I might have to give up
>programming. :/
Don't give up. It would be boring if it were all instantly clear.
The view is better after an enjoyable hike, and some of the flowers
along the way may turn out prettier than whatever the vista at the
top may be ;-)

For this part of the trail, just grok that def is executable,
not just the thing def's execution produces ;-)
>
>It's interesting that I have had several others tell me they had
>trouble with this too.
>
>So it is my opinion that decorators are a little too implicit.  I
>think there should be a way to make them easier to use while achieving
>the same objective and use. 
Maybe the above will help make functions and decorators a little easier
to understand.

HTH

Regards,
Bengt Richter