pre-PEP: Simple Thunks

[Bengt Richter]

On Fri, 15 Apr 2005 16:44:58 -0700, Brian Sabbey <sabbey at u.washington.edu> wrote:

>Here is a first draft of a PEP for thunks.  Please let me know what you 
>think. If there is a positive response, I will create a real PEP.
>
>I made a patch that implements thunks as described here. It is available 
>at:
>     http://staff.washington.edu/sabbey/py_do
>
>Good background on thunks can be found in ref. [1].

UIAM most of that pre-dates decorators. What is the relation of thunks
to decorators and/or how might they interact?

>
>Simple Thunks
>-------------
>
>Thunks are, as far as this PEP is concerned, anonymous functions that 
>blend into their environment. They can be used in ways similar to code 
>blocks in Ruby or Smalltalk. One specific use of thunks is as a way to 
>abstract acquire/release code. Another use is as a complement to 
>generators.

"blend into their environment" is not very precise ;-)
If you are talking about the code executing in the local namespace
as if part of a suite instead of apparently defined in a separate function,
I think I would prefer a different syntax ;-)

>
>A Set of Examples
>=================
>
>Thunk statements contain a new keyword, 'do', as in the example below. The 
>body of the thunk is the suite in the 'do' statement; it gets passed to 
>the function appearing next to 'do'. The thunk gets inserted as the first 
>argument to the function, reminiscent of the way 'self' is inserted as the 
>first argument to methods.
>
>def f(thunk):
>    before()
>    thunk()
>    after()
>
>do f():
>    stuff()
>
>The above code has the same effect as:
>
>before()
>stuff()
>after()
Meaning "do" forces the body of f to be exec'd in do's local space? What if there
are assignments in f? I don't think you mean that would get executed in do's local space,
that's what the thunk call is presumably supposed to do...

But let's get on to better examples, because this is probably confusing some, and I think there
are better ways to spell most use cases than we're seeing here so far ;-)

I want to explore using the thunk-accepting function as a decorator, and defining an anonymous
callable suite for it to "decorate" instead of using the do x,y in deco: or do f(27, 28): format.

To define an anonymous callable suite (aka thunk), I suggest the syntax for
    do x,y in deco:
        suite
should be
    @deco
    (x, y):        # like def foo(x, y):  without the def and foo
        suite

BTW, just dropping the def makes for a named thunk (aka callable suite), e.g.
    foo(x, y):
        suite
which you could call like
    foo(10, 4)
with the local-where-suite-was-define effect of
    x = 10
    y = 4
    suite

BTW, a callable local suite also makes case switching by calling through locals()[xsuitename]()
able to rebind local variables. Also, since a name is visible in an enclosing scope, it could
conceivably provide a mechanism for rebinding there. E.g.,

    def outer():
        xsuite(arg):
           x = arg
        def inner():
           xsuite(5)
        x = 2
        print x # => 2
        inner()
        print x # => 5

But it would be tricky if outer returned inner as a closure.
Or if it returned xsuite, for that matter. Probably simplest to limit
callable suites to the scope where they're defined.

>
>Other arguments to 'f' get placed after the thunk:
>
>def f(thunk, a, b):
>     # a == 27, b == 28
>     before()
>     thunk()
>     after()
>
>do f(27, 28):
>     stuff()
I'm not sure how you intend this to work. Above you implied (ISTM ;-)
that the entire body of f would effectively be executed locally. But is that
true? What if after after() in f, there were a last statment hi='from last statement of f'
Would hi be bound at this point in the flow (i.e., after d f(27, 28): stuff() )?

I'm thinking you didn't really mean that. IOW, by magic at the time of calling thunk from the
ordinary function f, thunk would be discovered to be what I call an executable suite, whose
body is the suite of your do statement.

In that case, f iself should not be a callable suite, since its body is _not_ supposed to be called locally,
and other than the fact that before and after got called, it was not quite exact to say it was _equivalent_ to

before()
stuff() # the do suite
after()

In that case, my version would just not have a do, instead defining the do suite
as a temp executable suite, e.g., if instead


we make an asignment in the suite, to make it clear it's not just a calling thing, e.g.,

 do f(27, 28):
      x = stuff()

then my version with explict name callable suite would be

 def f(thunk, a, b):
      # a == 27, b == 28
      before()
      thunk()
      after()

 set_x():
     x = stuff() # to make it plain it's not just a calling thing

 f(set_x, 27, 28)
 # x is now visible here as local binding

but a suitable decorator and an anonymous callable suite (thunk defined my way ;-) would make this

 @f(27, 28)
 (): x = stuff()


>
>Thunks can also accept arguments:
>
>def f(thunk):
>    thunk(6,7)
>
>do x,y in f():
>    # x==6, y==7
>    stuff(x,y)

IMO
    @f
    (x, y): stuff(x, y)   # like def foo(x, y): stuff(x, y)

is clearer, once you get used to the missing def foo format

>
>The return value can be captured
>
This is just a fallout of f's being an ordinary function right?
IOW, f doesn't really know thunk is not an ordinary callable too?
I imagine that is for the CALL_FUNCTION byte code implementation to discover?

>def f(thunk):
>    thunk()
>    return 8
>
>do t=f():
>    # t not bound yet
>    stuff()
>
>print t
>==> 8
That can't be done very well with a decorator, but you could pass an
explicit named callable suite, e.g.,

     thunk(): stuff()
     t = f(thunk)
>
>Thunks blend into their environment
ISTM this needs earlier emphasis ;-)

>
>def f(thunk):
>    thunk(6,7)
>
>a = 20
>do x,y in f():
>    a = 54
>print a,x,y
>
>==> 54,6,7

IMO that's more readable as

    def f(thunk):
        thunk(6, 7)
    @f
    (x, y):         # think def foo(x, y): with "def foo" missing to make it a thunk
        a = 54
    print a,x,y

IMO we need some real use cases, or we'll never be able to decide what's really useful.
>
>Thunks can return values.  Since using 'return' would leave it unclear 
>whether it is the thunk or the surrounding function that is returning, a 
>different keyword should be used. By analogy with 'for' and 'while' loops, 
>the 'continue' keyword is used for this purpose:
Gak ;-/
>
>def f(thunk):
>    before()
>    t = thunk()
>    # t == 11
>    after()
>
>do f():
>    continue 11

I wouldn't think return would be a problem if the compiler generated a
RETURN_CS_VALUE instead of RETURN_VALUE when it saw the end of
the callable suite (hence _CS_) (or thunk ;-)
Then it's up to f what to do with the result. It might pass it to after() sometimes.

>
>Exceptions raised in the thunk pass through the thunk's caller's frame 
>before returning to the frame in which the thunk is defined:
But it should be possible to have try/excepts within the thunk, IWT?
>
>def catch_everything(thunk):
>     try:
>         thunk()
>     except:
>         pass    # SomeException gets caught here
>
>try:
>     do catch_everything():
>        raise SomeException
>except:
>     pass        # SomeException doesn't get caught here because it was 
>already caught
>
>Because thunks blend into their environment, a thunk cannot be used after 
>its surrounding 'do' statement has finished:
>
>thunk_saver = None
>def f(thunk):
>     global thunk_saver
>     thunk_saver = thunk
>
>do f():
>     pass
>
>thunk_saver() # exception, thunk has expired
Why? IWT the above line would be equivalent to executing the suite (pass) in its place.
What happens if you defined

def f(thunk):
    def inner(it):
        it()
    inner(thunk)

do f():
    x = 123

Of course, I'd spell it
    @f
    (): x = 123

Is there a rule against that (passing thunk on to inner)?

>
>'break' and 'return' should probably not be allowed in thunks.  One could 
>use exceptions to simulate these, but it would be surprising to have 
>exceptions occur in what would otherwise be a non-exceptional situation. 
>One would have to use try/finally blocks in all code that calls thunks 
>just to deal with normal situations.  For example, using code like
>
>def f(thunk):
>     thunk()
>     prevent_core_meltdown()
>
>with code like
>
>do f():
>     p = 1
>return p
>
>would have a different effect than using it with
>
>do f():
>     return 1
>
>This behavior is potentially a cause of bugs since these two examples 
>might seem identical at first glance.
I think less so with decorator and anonymous callable suite format

    @f
    (): return 1  # as in def foo(): return 1  -- mnemonically removing "def foo"

>
>The thunk evaluates in the same frame as the function in which it was 
>defined. This frame is accessible:
>
>def f(thunk):
>     frame = thunk.tk_frame
      # no connection with tkinter, right? Maybe thunk._frame would also say be careful ;-)  
      assert sys._getframe(1) is frame # ?? when does that fail, if it can?
>
>do f():
>     pass
>
>Motivation
>==========
>
>Thunks can be used to solve most of the problems addressed by PEP 310 [2] 
>and PEP 288 [3].
>
>PEP 310 deals with the abstraction of acquire/release code. Such code is 
>needed when one needs to acquire a resource before its use and release it 
>after.  This often requires boilerplate, it is easy to get wrong, and 
>there is no visual indication that the before and after parts of the code 
>are related.  Thunks solve these problems by allowing the acquire/release 
>code to be written in a single, re-usable function.
>
>def acquire_release(thunk):
>    f = acquire()
>    try:
>       thunk(f)
>    finally:
>       f.release()
>
>do t in acquire_release():
>    print t
>
That could be done as a callable suite and decorator
   @acquire_release
   (t): print t   # like def foo(t): print t except that it's a thunk (or anonymous callable suite ;-)

BTW, since this callable suite definition is not named, there is no name binding
involved, so @acquire_release as a decorator doesn't have to return anything.

>More generally, thunks can be used whenever there is a repeated need for 
>the same code to appear before and after other code. For example,
>
>do WaitCursor():
>      compute_for_a_long_time()
>
>is more organized, easier to read and less bug-prone than the code
>
>DoWaitCursor(1)
>compute_for_a_long_time()
>DoWaitCursor(-1)

That would reduce to

    @WaitCursor
    (): compute_for_a_long_time()
>
>PEP 288 tries to overcome some of the limitations of generators.  One 
>limitation is that a 'yield' is not allowed in the 'try' block of a 
>'try'/'finally' statement.
>
>def get_items():
>     f = acquire()
>     try:
>         for i in f:
>             yield i   # syntax error
>     finally:
>         f.release()
>
>for i in get_items():
>     print i
>
>This code is not allowed because execution might never return after the 
>'yield' statement and therefore there is no way to ensure that the 
>'finally' block is executed.  A prohibition on such yields lessens the 
>suitability of generators as a way to produce items from a resource that 
>needs to be closed.  Of course, the generator could be wrapped in a class 
>that closes the resource, but this is a complication one would like to 
>avoid, and does not ensure that the resource will be released in a timely 
>manner.  Thunks do not have this limitation because the thunk-accepting 
>function is in control-- execution cannot break out of the 'do' statement 
>without first passing through the thunk-accepting function.
>
>def get_items(thunk):    # <-- "thunk-accepting function"
>     f = acquire()
>     try:
>         for i in f:
>             thunk(i)      # A-OK
>     finally:
>         f.release()
>
>do i in get_items():
>    print i

    @get_items
    (i): print i

But no yields in the thunk either, that would presumably not be A-OK ;-)
>
>Even though thunks can be used in some ways that generators cannot, they 
>are not nearly a replacement for generators.  Importantly, one has no 
>analogue of the 'next' method of generators when using thunks:
>
>def f():
>     yield 89
>     yield 91
>
>g = f()
>g.next()  # == 89
>g.next()  # == 91
>
>[1] see the "Extended Function syntax" thread, 
>http://mail.python.org/pipermail/python-dev/2003-February/
>[2] http://www.python.org/peps/pep-0310.html
>[3] http://www.python.org/peps/pep-0288.html

It's interesting, but I think I'd like to explore the decorator/callable suite version in some real
use cases. IMO the easy analogy with def foo(args): ... for specifying the thunk call parameters,
and the already established decorator call mechanism make this attractive. Also, if you allow named
thunks (I don't quite understand why they should "expire" if they can be bound to something. The
"thunk-accepting function" does not appear to "know" that the thunk reference will turn out to
be a real thunk as opposed to any other callable, so at that point it's just a reference,
and should be passable anywhere -- except maybe out of its defining scope, which would necessitate
generating a peculiar closure.

For a named callable suite, the decorator would have to return the callable, to preserve the binding,
or change it to something else useful. But without names, I could envisage a stack of decorators like
(not really a stack, since they are independent, and the first just uses the thunk to store a switch
class instance and a bound method to accumulate the cases ;-)

    @make_switch
    (switch, case): pass
    @case(1,2,3,5)
    (v): print 'small prime: %s'% v
    @case(*'abc')
    (v): print 'early alpha: %r'% v
    @case()
    (v): print 'default case value: %r'%v

and then being able to call
    switch('b')
and see early alpha: 'b'

Not that this example demonstrates the local rebinding capability of thunks, which was the whole
purpose of this kind of switch definition ;-/
Just substitute some interesting suites that bind something, in the place of the prints ;-)

Regards,
Bengt Richter