A friendlier, sugarier lambda -- a proposal for Ruby-like blocks in python

[brenocon at gmail.com]

Hi all --

Compared to the Python I know and love, Ruby isn't quite the same.
However, it has at least one terrific feature: "blocks".  Whereas in
Python a
"block" is just several lines of locally-scoped-together code, in Ruby
a
"block" defines a closure (anonymous function).  To avoid confusion
let's call
them Ruby block-closures.  I see them as just a syntax for defining
closures
and passing them into method calls.  I think something analogous could
be added
to Python in a very simple manner that would make closures much more
readable
and usable, and nail some use cases nicely.

To define a new closure and pass it into a function call, there are two
current
methods: inline 'def' and 'lambda'.  Consider the following Twisted-ish
code:

    deferred = fetchPage('http://python.org')
    def _showResponse(response)
        print "fancy formatting: %s" % response.text
    deferred.addCallback(_showResponse)

Lots of Twisted code has to be written backwards like this.
Theoretically, it
might be nice to use lambda right in the addCallback() call, like:

    deferred.addCallback(lambda r:  print("fancy formatting %s"
%r.text) )

But this is awkward since the lambda is constrained to be one line; you
can't
come back later and add much to the callback's code.  Furthermore, this
example
isn't even legal, because 'print' isn't a function, but a statement --
lambda
is further constrained to only contain an expression.

Many have complained about this crippled-ness of lambda, but it
actually makes
some sense.  Since Python uses colons and indentation to define blocks
of code,
it would be awkward to close a multiline lambda.  The best I could
think of
would look like

    deferred.addCallback(lambda r:
        print("fancy formatting %s" % r.text)
    )

    ^
    |

That trailing paranthesis is WAY un-Pythonic.  We don't close code
blocks like
that!  And in general, declaring big multiline anonymous functions in
the
middle of a list of normal variable arguments is weird -- it just
doesn't fit.
It's perfectly legal to pass in 4 closures, interspersed with number
and string
arguments.  Imagine defining all of those inline with 'lambda'
expressions!
And what about nesting?  And then there's the term "lambda", while a
great
homage to Lisp and computation theory, just isn't the friendliest
programming
vocab term.

(from my limited understanding,) Ruby block-closures assume a specific
use
case: You want to pass exactly one multiline, defined-right-there
closure to a
method when calling it.  Therefore, you should get to define the
closure
*immediately following* the method call.  I suggest a Python version
with a
keyword 'using' (or 'with'?) that takes the block of code as a closure,
and
passes it to the method call as the *last argument*.  The above example
becomes:

    deferred.addCallback() using response:
        print "fancy formatting %s" % response.text

and in general, the following two code snippets are equivalent:


    def _f(x,y):
        [do stuff with x and y]
    function_with_callback(a,b,c, _f)

    function_with_callback(a,b,c) using x,y:
        [do stuff with x and y]
    next_statement()

... where function_with_callback wants a 2-arg function as its last
argument.
It gets to call _f, or equivalently the defined-right-there
closure/ruby-block,
however it wants to -- wait for an I/O operation to finish, whatever.
I'm not
so hot about the fact that it looks like addCallback() should be
completed
before the 'using' keyword kicks in, but this is the simplest I could
think of.

This syntax does not let you define a new function and store it as a
local
variable.  Python already has inline 'def' for that (that is, you can
do a
'def' in any block of code you want, and it stays local to that scope.)
 It
does not, strictly speaking, let you do anything new -- as Guido has
stated,
you could ditch lambda and achieve the equivalent by declaring the
little
callback function as an inline 'def', like in the first deferred
example here.

This only optimizes for the case of defining a closure only for the
purpose of
passing it in as an argument to a method.  However, this must be the
only use
for anonymous functions in Python, since we already have inline 'def'.
I've
always felt that passing in a lambda in lisp and python, or the
equivalent
anonymous function(x,y) {} in javascript, or anonymous classes in java,
was
always awkward.  You have to think about defining a new anonymous
function
*within* your method call and such, and then you have to close
parantheses
after it -- it's just weird, I dunno.

This proposal could also handle some of the use cases mentioned in PEP
310 and
340.  If you design your callback-taking functions to have only one
callback
and have it as the last argument, you can trivially write lock
acquition (omit
'using' for a no-arg block-closure):

    def protect(lock, f):
        lock.acquire()
        f()
        lock.release()

    protect(myLock):
        [do stuff that needs myLock to be acquired]

Of course, the definition of protect() might have try/finally wrapped
around
the f() call.  (Interestingly, this starts looking like a way to define
new
control-like structures.  I haven't thought through the implications.)

ActiveRecord, Rails' object-relational mapper, does almost exactly this
for
database transactions, and I have found it quite nice:

    # User is a sqlobject/sqlalchemy/django/whatever ORM class;
    # User.transaction is a class method executing its passed-in
closure within
    #  the user table's START TRANSACTION and STOP TRANSACTION.

    user1, user2 = getTwoUsers()
    User.transaction() using user1, user2:
        someRaceConditionProneStuff(user1, user2)
        moreRaceConditionProneStuff(user1, user2)

There might be some sort of overlap with PEP 343 and the 'with'
statement, but
I'm not sure exactly.  Sorry I'm late to the game and commenting on
last year's
PEP's, but I've only started reading them.  Note that PEP's 343 and 340
are
very focused on resource management -- but I think that letting one
define code
blocks as closures could make resource handling routines be easily
written in
Python.  Furthermore, tons more stuff -- like Deferreds and such --
could be
added.  (Ruby uses block-closures to do really basic constructs such as
foreach
iteration.  Python does a fine job with "for x in L" and list and
generator
comprehensions... enough so that map/lambda is obsolete!  I'm just
trying to
see if there are use cases in Python for block-closures.)

I've been trying to search for similar proposals but have come up dry.
Anything like this out there?  I hear Ruby blocks are heavily inspired
by
Smalltalk; anyone know more?

Is it feasible to assume the primary use of closures is as part of an
argument
list, and such argument lists will want only one argument that is a
closure?
Does doing so avoid any big annoyances of functional programming?

Is this completely, totally incompatible with the current state of
Python and
previous deliberations :) ?  e.g. I haven't thought much about how this
would
interact with yield and generators.

But really, I'm just idly curious -- does anyone think this might be
useful?


Take care,
Brendan