puzzled by name binding in local function

[Terry Reedy]

Code examples are Python 3

On 2/5/2013 10:18 AM, Ulrich Eckhardt wrote:

> Below you will find example code distilled from a set of unit tests,
> usable with Python 2 or 3. I'm using a loop over a list of parameters to
> generate tests with different permutations of parameters. Instead of
> calling util() with values 0-4 as I would expect, each call uses the
> same parameter 4. What I found out is that the name 'i' is resolved when
> Foo.test_1 is called

Names* in Python code are resolved when the code is executed.
Function bodies are executed when the function is called.
Ergo, names in function bodies are resolved when the function is called.
This is sometimes called late binding.

* This may exclude keyword names.

Late binding of global names within functions is why the following can 
work instead of raising NameError.

 >>> def f(): print(x)

 >>> x = 3
 >>> f()
3

Only the most recent binding of x, at the time of the call matters, as 
long as there is one. Does the following really surprise you?

 >>> x = 0
 >>> def f(): print(x)

 >>> x = 3
 >>> f()
3

What do you expect this to print?

 >>> x = 1
 >>> def f1(): print(x)

 >>> x = 2
 >>> def f2(): print(x)

 >>> x = 3
 >>> f1(), f2()

Rolling the repeated code into a loop does not magically change the 
behavior of def statements.

for i in range(1, 3):
   exec('''\
x = {0}
def f{0}(): print(x)'''.format(i))

x = 3
print((f1(), f2()))

This gives *exactly* the same output.

So does this:

from textwrap import dedent

for i in range(1, 3):
   exec(dedent('''
     x = {0}
     def f{0}():
       print(x)
     '''.format(i)))

x = 3
print((f1(), f2()))


Python does not do text substitution unless you explicit ask it too, as 
I did above.

Late binding is also why functions (and methods, such as .__init__) can 
call functions (methods) whose definitions follow later in the code, so 
don't change that this change ;-).

> and not substituted inside the for-loop,

> Now, I'm still not sure how to best solve this problem:
>   * Spell out all permutations is a no-go.
>   * Testing the different iterations inside a single test, is
> inconvenient because I want to know which permutation exactly fails and

A good test framework should give specifics as to the failure. The 
unittest assertxxx methods do this. In fact, emitting specific messages 
is one reason there are so many methods.

The real 'problem' with multiple tests within a test function is that 
the first failure ends that group of tests. But this is only a problem 
during development when there *are* failures. And it is possible to 
write a test function to run all tests and collect multiple error 
messages before 'failing' the test.

> which others don't. Further, I want to be able to run just that one
> because the tests take time.

Whether multiple tests are buried within one function or many, running 
just one of them will require some editing.

>   * Further, I could generate local test() functions using the current
> value of 'i' as default for a parameter, which is then used in the call
> to self.util(), but that code is just as non-obviously-to-me correct as
> the current code is non-obviously-to-me wrong.

LOL. You know the easiest and correct solution, but reject it because it 
is not 'obvious' - though it was obvious enough for you to see it.

If one understands that function definition are executable statements 
and that their execution is not magically changed by putting them inside 
loops, the problem with your code should be obvious. It creates 5 
*identical* functions objects. So it should not be surprising that they 
behave identically.

 > I'd prefer something more stable.

The fact that default arg expressions are evaluated when the function is 
defined is quite stable. Ain't gonna change.

> Any other suggestions?

Revise your obvious meter ;-).

> # example code
> from __future__ import print_function
> import unittest
>
> class Foo(unittest.TestCase):
>      def util(self, param):
>          print('util({}, {})'.format(self, param))
>
> for i in range(5):

>      def test(self):
>          self.util(param=i)

Executing this n times produces n identical functions. The easy fix is

def test(self, j = i): self.util(param = j)

>      setattr(Foo, 'test_{}'.format(i), test)

Another fix that should work: adapt my code above and use exec within a 
loop within the class statement itself (and delete setattr).

     for i in range(5):
         exec(dedent('''
             def test_{0}(self):
                 self.util(param={0})
             '''.format(i)))

> unittest.main()

-- 
Terry Jan Reedy