[code-quality] PyLint catching comprehension binding leaks (was: Warn on list comprehension variables used outside list comprehensions?)

[Ian Cordasco]

On Wed, Nov 26, 2014 at 10:51 AM, Keith Derrick <keith.derrick at lge.com> wrote:
> I couldn't find anything in their bug list about this, so thought I'd check.
>
> Just ran pylint at the command line with pylint 1.4 on this file
> (docstrings and import just to suppress miscellaneous complaints)
>
>>      1    ''' Test file for pylint '''
>>      2    from __future__ import print_function
>>      3
>>      4    def method1():
>>      5        ''' Let x bleed from comprehension '''
>>      6        [x for x in range(3)]
>>      7        print(x)
>>      8
>>      9    def method2():
>>     10        ''' reuse/hide x from local scope in comprehension '''
>>     11        y = 10
>>     12        [y for y in range(3)]
>>     13        print(y)
>>     14
>
> For python 2.7.6
>> ************* Module bug
>> W:  6, 4: Expression "[x for x in range(3)]" is assigned to nothing
>> (expression-not-assigned)
>> W:  7,10: Using possibly undefined loop variable 'x'
>> (undefined-loop-variable)
>> C: 11, 4: Invalid variable name "y" (invalid-name)
>> W: 12, 4: Expression "[y for y in range(3)]" is assigned to nothing
>> (expression-not-assigned)
> For python 3.4.0
>> ************* Module bug
>> W:  6, 4: Expression "[x for x in range(3)]" is assigned to nothing
>> (expression-not-assigned)
>> E:  7,10: Undefined variable 'x' (undefined-variable)
>> C: 11, 4: Invalid variable name "y" (invalid-name)
>> W: 12, 4: Expression "[y for y in range(3)]" is assigned to nothing
>> (expression-not-assigned)
>
> So, it warns about the first case in py2 and gives an error in py3 which
> is expected.
>
> But it seems to be confused by the second case, flagging "y" as an
> invalid variable-name at the assignment point for both versions.
>
> Keith Derrick | Principal Engineer, Connected Platform | Engineering
> LG Silicon Valley Lab | 5150 Gt America Parkway, Santa Clara, CA 95054
> Office: 408.610-5746 | Mobile: 831.383.9567 |  LG.com
>

At this point, I'd recommend repling to the retitled message to catch
the PyLint maintainer's focus.

> On 11/26/2014 07:50 AM, Ian Cordasco wrote:
>> On Wed, Nov 26, 2014 at 9:27 AM, Keith Derrick <keith.derrick at lge.com> wrote:
>>> Definitely seems like something that should be warned about.
>>>
>>> For python2 it ought to be an error (which can always be turned off) for
>>> either code snippet.
>> PyFlakes doesn't allow you to ignore errors. Flake8 does so this is a
>> moot point.
>>
>>> For python3, I'd say the two line snippet should flag an error - yes, it
>>> will fail at run-time, but pydev in eclipse runs pylint and saves me a
>>> lot of grief by spotting this type of thing before I ever save the file
>>> (though it doesn't catch either of these, presumably because they are
>>> valid python).
>> If PyLint doesn't catch this I wonder if it's either never been raised
>> or been decided to be a bad idea. Could someone search their bugs?
>>
>>> The three line snippet should probably give a warning ("Are you sure you
>>> mean this?")
>>>
>>> The question is, how do we know which version of Python the code under
>>> inspection is targeted for, and hence how to treat this case.
>>>
>>> Some thoughts on that:
>>>
>>> * Anyone serious enough to be using pyflakes, pylint, or other static
>>> analysis tools is also probably using virtualenv.
>>> * pyflakes is therefore most likely running under the same python
>>> version for which the code is targeted.
>>> * code which is meant to run under multiple python versions is probably
>>> tested under something like tox - so again, pyflakes is run under the
>>> targeted python version (certainly my tox.ini runs it in every target
>>> environment)
>>>
>>> There's a lot of "probably"'s in there, but I don't think they are
>>> unreasonable assumptions.
>>>
>>> So, could  pyflakes reasonably assume that the interpreter under which
>>> it is running is also the intended target version for the code under
>>> inspection? At least for the major version.
>> It already works this way. If you use pyflakes on Python 2 but are
>> writing code only valid in Python 3 you will have a bad time. You have
>> to install pyflakes on the version of Python you want it to lint for
>> since pyflakes works only with the in-built ast module of Python.
>>
>>> This could then be overridden by a suitable command line option to
>>> specify the target version(s)
>> No. This isn't how PyFlakes is intended to work and this is extraneous.
>>
>>> Also, in cases like this, possibly a "This will behave differently in
>>> Python 2 and 3" warning is a worth while alternative.
>> Again, PyFlakes is very different from PyLint. We don't issue warnings
>> like this.
>>
>>> Keith Derrick | Principal Engineer, Connected Platform | Engineering
>>> LG Silicon Valley Lab | 5150 Gt America Parkway, Santa Clara, CA 95054
>>> Office: 408.610-5746 | Mobile: 831.383.9567 |  LG.com
>>>
>>> On 11/26/2014 06:57 AM, Skip Montanaro wrote:
>>>>> That's not the point of this check. The point of this check is that on
>>>>> Python 2, the binding to x in the comprehension bleeds outside of the
>>>>> comprehension scope
>>>> Got it.
>>>>
>>>> Still, this code:
>>>>
>>>>>       x = 10
>>>>>       [x for x in range(3)]
>>>>>       print(x + 1)
>>>> will run differently in Python 2 than Python3, so even if that was a
>>>> conscious choice by the author, a --py3k flag should cause a message
>>>> for this code. This is even worse than the simpler
>>>>
>>>>>       [x for x in range(3)]
>>>>>       print(x + 1)
>>>> because at least that will raise a NameError when run in Python 3. The
>>>> three-line construct will still run, though produce different output.
>>>>
>>>> Skip
>>>> _______________________________________________
>>>> code-quality mailing list
>>>> code-quality at python.org
>>>> https://mail.python.org/mailman/listinfo/code-quality
>>> _______________________________________________
>>> code-quality mailing list
>>> code-quality at python.org
>>> https://mail.python.org/mailman/listinfo/code-quality