closures and dynamic binding

[Paul Boddie]

On 29 Sep, 05:56, Terry Reedy <tjre... at udel.edu> wrote:
>
> As I understand it, partly from postings here years ago...
>
> Lexical: The namespace scope of 'n' in inner is determined by where
> inner is located in the code -- where is is compiled.  This is Python
> (and nearly all modern languages).  Even without closures, the global
> scope of a function is the module it is defined in.

This is how I understand it, too. The confusing part involves the
definition of any inner function and how any "external" names are
bound or defined. As we've seen...

  def f(x):
    def g():
      return x
    x += 1 # added for illustration
    return g

...it might look at first glance like the function known as g (within
f) should return the initial value of x (as known within f), since
that was the value x had when g was defined. The following is an
example execution trace based on that mental model:

fn = f(1)
-> def f(1):
->   def g(): # g defined with x as 1
->     return x # would be 1
->   x += 1 # x becomes 2
->   return g
fn()
-> def g():
->   return x # would still be 1

However, as we know, this isn't the case in real Python since g isn't
initialised with the value of x at the time of its definition - it
instead maintains access to the namespace providing x. We must
therefore revise the example:

fn = f(1)
-> def f(1):
->   def g(): # g refers to x within f(1)
->     return x # would be the current value of x within f(1)
->   x += 1 # x becomes 2
->   return g
fn()
-> def g(): # g refers to x within f(1)
->   return x # would be the current value of x within f(1), which is
2

This is the dynamic aspect of closures: values aren't used to
initialise inner functions; names are looked up from their origin.

> Dynamic: The namespace scope of 'n' in inner, how it is looked up, is
> determined by where inner is called from. This is what you seemed to be
> suggesting -- look up 'n' based on the scope it is *used* in.

Indeed. Dynamic scoping is confusing in that one has to set up an
appropriate "environment" for the closure to access so that references
to names can be meaningfully satisfied; obviously, this creates all
sorts of encapsulation problems. The confusing aspect of lexical
scoping, however, especially if non-local names can be changed, is
that the effects of closures are potentially distant - one doesn't
always invoke inner functions in the place where they were defined, as
we see above - and such effects may thus happen within "abandoned
namespaces" - that is, namespaces which can no longer be revisited and
used in their original context. So, in the above, once f has been
invoked, the namespace for that invocation effectively lives on, but
that namespace is not a general one for f - if we invoke f again, we
get another namespace as one should reasonably expect.

A somewhat separate issue is illustrated by the modification of x
within f. Although for most statements, we would expect the value of x
to evolve following from a top-to-bottom traversal of the code within
a unit, function definition statements do not behave like, say, "for",
"if" or "while" statements. Now although this should be obvious at the
module global level, I feel that it can be easy to overlook within a
function where one normally expects to find plain old control-flow
constructs, expressions, assignments and so on. It is pertinent to
note, with respect to the original inquiry, that lambda functions are
subject to the same caveats, and I believe that lexical scoping was
introduced precisely to make lambda functions less cumbersome to
employ, eliminating the need to explicitly initialise them using the
"identity" default parameters trick, but obviously introducing the
consequences and potential misunderstandings described above.

Paul