From brett at python.org  Mon Apr  8 13:55:11 2019
From: brett at python.org (Brett Cannon)
Date: Mon, 8 Apr 2019 10:55:11 -0700
Subject: [Python-porting] co_names behavior change
In-Reply-To: <DM6PR08MB4778CA585E56907028DB79F7CA5E0@DM6PR08MB4778.namprd08.prod.outlook.com>
References: <DM6PR08MB4778CA585E56907028DB79F7CA5E0@DM6PR08MB4778.namprd08.prod.outlook.com>
Message-ID: <CAP1=2W47W6dU4aqEXTLm5FPwugth=dFQp6ECyr4uQ1oc7DBgCA@mail.gmail.com>

I'm not aware of a specific change, but the What's New documentation tries
to capture critical changes like this that would break someone's code.

On Mon, Apr 8, 2019 at 10:31 AM Tom Ekberg <tekberg at uw.edu> wrote:

> I'm using the roundup program which has been ported from Python 2.7 to
> Python 3.x. My group uses this program internally to track progress and
> document issues. Roundup is written in Python and has a templating language
> (TAL) that allows one to define a Python expression. This is implemented by
> constructing a function:
>
>
>             d = {}
>             exec('def f():\n return %s\n' % expr.strip(), d)
>             self._f = d['f']
>
> where expr contains the Python expression. To reference variables defined
> in the templating language it constructs a list of variable names using the
> names in self.__code__.co_names:
>
>
>         self._f_varnames = vnames = []
>         for vname in self._f.__code__.co_names:
>             if vname[0] not in '$_':
>
> This works fine for Python 2.7. However for Python 3.5, the value of
> self._f.__code__.co_names does not always contain all names referenced in
> the expression. When the function is executed it generates an exception
> saying that a variable being referenced is undefined. Some expression work
> correctly, some do not. I have documented an example that fails in the
> roundup bug tracking database:
> https://issues.roundup-tracker.org/issue2551026. I came up with a roundup
> patch that implements variation that works for both Python 2.7 and Python
> 3.5.
>
>
> My question to this group:
>
>
> Do you have knowledge of the different contents of __code__.co_names
> between Python 2.7 and Python 3.5? I'm expecting that this is a difference
> that is documented somewhere.
>
>
> I have attached an example that illustrates this problem.
>
>
> Tom Ekberg
> Senior Computer Specialist, Lab Medicine
> 4th Floor, Pat Steel Building
> Department of Laboratory Medicine
> Work: (206) 520-4856
> Email: tekberg at uw.edu
> _______________________________________________
> Python-porting mailing list
> Python-porting at python.org
> https://mail.python.org/mailman/listinfo/python-porting
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-porting/attachments/20190408/a9d8d9fe/attachment-0001.html>

From tekberg at uw.edu  Mon Apr  8 17:20:05 2019
From: tekberg at uw.edu (Tom Ekberg)
Date: Mon, 8 Apr 2019 21:20:05 +0000
Subject: [Python-porting] co_names behavior change
In-Reply-To: <23723.46660.278362.453184@cochabamba.vanoostrum.org>
References: <DM6PR08MB4778CA585E56907028DB79F7CA5E0@DM6PR08MB4778.namprd08.prod.outlook.com>,
 <23723.46660.278362.453184@cochabamba.vanoostrum.org>
Message-ID: <DM6PR08MB477882A0DFD248358EF64BA5CA2C0@DM6PR08MB4778.namprd08.prod.outlook.com>

Being an old-time compiler person, I ended up using the symtable module and recursively called child.get_identifiers() to get all of the names. Attached is the diff (ignore the last 2 parts of the diff). It works with python 2.7 and 3.5.


Tom Ekberg
Senior Computer Specialist, Lab Medicine
4th Floor, Pat Steel Building
Department of Laboratory Medicine
Work: (206) 520-4856
Email: tekberg at uw.edu


________________________________
From: Piet van Oostrum <piet-l at vanoostrum.org>
Sent: Monday, April 8, 2019 1:59 PM
To: Tom Ekberg
Cc: Python-porting at python.org
Subject: Re: [Python-porting] co_names behavior change

You can get the variable names of the embedded comprehensions with:

[lc.co_names for lc in f.__code__.co_consts if  type(lc) is types.CodeType]

where f is the function defined by the exec (f = d['f'])

But to take nested comprehensions into account you would have to write a recursive function:

from types import CodeType

def varnames(codeobj):
    names = [lc.co_names for lc in codeobj.co_consts if isinstance(lc, CodeType)]
    names += [varnames(lc) for lc in codeobj.co_consts if isinstance(lc, CodeType)]
    # flatten the list
    return [item for sublist in names for item in sublist]

print(varnames(f.__code__))

--
Piet van Oostrum <piet-l at vanoostrum.org>
WWW: http://piet.vanoostrum.org/
piet . van oostrum ? Mijn Blog<http://piet.vanoostrum.org/>
piet.vanoostrum.org
Op 22 december hebben we mijn verjaardag gevierd met een barbequeue in restaurant `Los Jardines? (De Tuinen), hetzelfde restaurant waar we een week geleden het kerstdiner van het MEMI hadden. Het was de eerste keer dat mijn verjaardag in de zomer was.


PGP key: [8DAE142BE17999C4]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-porting/attachments/20190408/d1a6eda4/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PythonExpr.py.diff
Type: application/octet-stream
Size: 2259 bytes
Desc: PythonExpr.py.diff
URL: <http://mail.python.org/pipermail/python-porting/attachments/20190408/d1a6eda4/attachment.obj>

From piet-l at vanoostrum.org  Mon Apr  8 15:46:26 2019
From: piet-l at vanoostrum.org (Piet van Oostrum)
Date: Mon, 8 Apr 2019 21:46:26 +0200
Subject: [Python-porting] co_names behavior change
In-Reply-To: <DM6PR08MB4778CA585E56907028DB79F7CA5E0@DM6PR08MB4778.namprd08.prod.outlook.com>
References: <DM6PR08MB4778CA585E56907028DB79F7CA5E0@DM6PR08MB4778.namprd08.prod.outlook.com>
Message-ID: <23723.42258.603637.898700@cochabamba.vanoostrum.org>

Tom Ekberg wrote:
 > 
 > My question to this group:
 > 
 > Do you have knowledge of the different contents of __code__.co_names between Python 2.7 and Python
 > 3.5? I'm expecting that this is a difference that is documented somewhere.
 > 

The difference is that in Python 3 a (list) comprehension is compiled into an anonymous function, to prevent the leaking of the control variable (x in this case). The base from which the list is constructed (i.e. the expression after 'in') is given as the argument to this function, so any variable in that part appears in co_names. But the rest of the comprehension is part of that anonymous function, so in this case the names 'realname' and 'user_realnames' are not local to your function 'f', but of the anonymous function.

In Python 2, the list comprehension was compiled inline.
-- 
Piet van Oostrum <piet-l at vanoostrum.org>
WWW: http://piet.vanoostrum.org/
PGP key: [8DAE142BE17999C4]


From piet-l at vanoostrum.org  Mon Apr  8 16:59:48 2019
From: piet-l at vanoostrum.org (Piet van Oostrum)
Date: Mon, 8 Apr 2019 22:59:48 +0200
Subject: [Python-porting] co_names behavior change
In-Reply-To: <DM6PR08MB4778CA585E56907028DB79F7CA5E0@DM6PR08MB4778.namprd08.prod.outlook.com>
References: <DM6PR08MB4778CA585E56907028DB79F7CA5E0@DM6PR08MB4778.namprd08.prod.outlook.com>
Message-ID: <23723.46660.278362.453184@cochabamba.vanoostrum.org>

You can get the variable names of the embedded comprehensions with:

[lc.co_names for lc in f.__code__.co_consts if  type(lc) is types.CodeType]

where f is the function defined by the exec (f = d['f'])

But to take nested comprehensions into account you would have to write a recursive function:

from types import CodeType

def varnames(codeobj):
    names = [lc.co_names for lc in codeobj.co_consts if isinstance(lc, CodeType)]
    names += [varnames(lc) for lc in codeobj.co_consts if isinstance(lc, CodeType)]
    # flatten the list
    return [item for sublist in names for item in sublist]

print(varnames(f.__code__))

-- 
Piet van Oostrum <piet-l at vanoostrum.org>
WWW: http://piet.vanoostrum.org/
PGP key: [8DAE142BE17999C4]