A friendlier, sugarier lambda -- a proposal for Ruby-like blocks in python

[brenocon at gmail.com]

[repost -- fixed formatting]
Hi all --

Compared to the Python I know and love, Ruby isn't quite the same.
However, it has at least one terrific feature: "blocks".  Whereas in
Python a "block" is just several lines of locally-scoped-together code,
in Ruby a "block" defines a closure (anonymous function).  To avoid
confusion let's call them Ruby block-closures.  I see them as just a
syntax for defining closures and passing them into method calls.  I
think something analogous could be added to Python in a very simple
manner that would make closures much more readable and usable, and nail
some use cases nicely.

To define a new closure and pass it into a function call, there are two
current methods: inline 'def' and 'lambda'.  Consider the following
Twisted-ish code:

    deferred = fetchPage('http://python.org')
    def _showResponse(response)
        print "fancy formatting: %s" % response.text
    deferred.addCallback(_showResponse)

Lots of Twisted code has to be written backwards like this.
Theoretically, it might be nice to use lambda right in the
addCallback() call, like:

    deferred.addCallback(lambda r:  print("fancy formatting %s"
%r.text) )

But this is awkward since the lambda is constrained to be one line; you
can't come back later and add much to the callback's code.
Furthermore, this example isn't even legal, because 'print' isn't a
function, but a statement -- lambda is further constrained to only
contain an expression.

Many have complained about this crippled-ness of lambda, but it
actually makes some sense.  Since Python uses colons and indentation to
define blocks of code, it would be awkward to close a multiline lambda.
 The best I could think of would look like

    deferred.addCallback(lambda r:
        print("fancy formatting %s" % r.text)
    )

    ^
    |

That trailing paranthesis is WAY un-Pythonic.  We don't close code
blocks like that!  And in general, declaring big multiline anonymous
functions in the middle of a list of normal variable arguments is weird
-- it just doesn't fit. It's perfectly legal to pass in 4 closures,
interspersed with number and string arguments.  Imagine defining all of
those inline with 'lambda' expressions!  And what about nesting?  And
then there's the term "lambda", while a great homage to Lisp and
computation theory, just isn't the friendliest programming
vocab term.

(from my limited understanding,) Ruby block-closures assume a specific
use case: You want to pass exactly one multiline, defined-right-there
closure to a method when calling it.  Therefore, you should get to
define the closure *immediately following* the method call.  I suggest
a Python version with a keyword 'using' (or 'with'?) that takes the
block of code as a closure, and passes it to the method call as the
*last argument*.  The above example becomes:

    deferred.addCallback() using response:
        print "fancy formatting %s" % response.text

and in general, the following two code snippets are equivalent:

    def _f(x,y):
        [do stuff with x and y]
    function_with_callback(a,b,c, _f)

    function_with_callback(a,b,c) using x,y:
        [do stuff with x and y]
    next_statement()

... where function_with_callback wants a 2-arg function as its last
argument.  It gets to call _f, or equivalently the defined-right-there
closure/ruby-block, however it wants to -- wait for an I/O operation to
finish, whatever. I'm not so hot about the fact that it looks like
addCallback() should be completed before the 'using' keyword kicks in,
but this is the simplest I could think of.

This syntax does not let you define a new function and store it as a
local variable.  Python already has inline 'def' for that (that is, you
can do a 'def' in any block of code you want, and it stays local to
that scope.)  It does not, strictly speaking, let you do anything new
-- as Guido has stated, you could ditch lambda and achieve the
equivalent by declaring the
little callback function as an inline 'def', like in the first deferred
example here.

This only optimizes for the case of defining a closure only for the
purpose of passing it in as an argument to a method.  However, this
must be the only use for anonymous functions in Python, since we
already have inline 'def'. I've always felt that passing in a lambda in
lisp and python, or the equivalent anonymous function(x,y) {} in
javascript, or anonymous classes in java, was always awkward.  You have
to think about defining a new anonymous
function *within* your method call and such, and then you have to close
parantheses after it -- it's just weird, I dunno.

This proposal could also handle some of the use cases mentioned in PEP
310 and 340.  If you design your callback-taking functions to have only
one callback and have it as the last argument, you can trivially write
lock acquition (omit 'using' for a no-arg block-closure):

    def protect(lock, f):
        lock.acquire()
        f()
        lock.release()

    protect(myLock):
        [do stuff that needs myLock to be acquired]

Of course, the definition of protect() might have try/finally wrapped
around the f() call.  (Interestingly, this starts looking like a way to
define new control-like structures.  I haven't thought through the
implications.)

ActiveRecord, Rails' object-relational mapper, does almost exactly this
for database transactions, and I have found it quite nice:

    # User is a sqlobject/sqlalchemy/django/whatever ORM class;
    # User.transaction is a class method executing its passed-in
    #closure within
    #  the user table's START TRANSACTION and STOP TRANSACTION.

    user1, user2 = getTwoUsers()
    User.transaction() using user1, user2:
        someRaceConditionProneStuff(user1, user2)
        moreRaceConditionProneStuff(user1, user2)

There might be some sort of overlap with PEP 343 and the 'with'
statement, but I'm not sure exactly.  Sorry I'm late to the game and
commenting on last year's PEP's, but I've only started reading them.
Note that PEP's 343 and 340 are very focused on resource management --
but I think that letting one define code blocks as closures could make
resource handling routines be easily written in Python.  Furthermore,
tons more stuff -- like Deferreds and such -- could be added.  (Ruby
uses block-closures to do really basic constructs such as foreach
iteration.  Python does a fine job with "for x in L" and list and
generator comprehensions... enough so that map/lambda is obsolete!  I'm
just trying to see if there are use cases in Python for
block-closures.)

I've been trying to search for similar proposals but have come up dry.
Anything like this out here?  I hear Ruby blocks are heavily inspired
by Smalltalk; anyone know more?

Is it feasible to assume the primary use of closures is as part of an
argument list, and such argument lists will want only one argument that
is a closure?  Does doing so avoid any big annoyances of functional
programming?

Is this completely, totally incompatible with the current state of
Python and previous deliberations :) ?  e.g. I haven't thought much
about how this would interact with yield and generators.

But really, I'm just idly curious -- does anyone think this might be
useful?

Take care,
Brendan