From cortex at worlddomination.be Fri Nov 14 13:05:53 2014 From: cortex at worlddomination.be (Laurent Peuch) Date: Fri, 14 Nov 2014 13:05:53 +0100 Subject: [code-quality] RedBaron, a bottom-up refactoring lib/tool for python Message-ID: <20141114120553.GA2174@griboiedov> Hello everyone, Someone has suggested me to talk about the project I'm working on right now on this mailing list because this has a lot of chances to interest you. This tool is an answer to a frustration that I've had while trying to build tools for python and projects I was working on. While there is already good capacities in python to analyse code (ast.py, astroid (while it wasn't out at that time)), the "write code that modify source code" was really missing (in my opinion and my knowledge of the existing tools). I wanted a very intuitive and easy to use library that allows me to query and modify my source code only in the place I wanted to modify it without touching the rest of the code. So I've built what can be describe as "the BeautifulSoup of python source code". To do so, I've built what can be called "a lossless AST" for python (designed to be used by humans), an AST that satisfy this equation: source_code == ast_to_source(source_to_ast(source_code)) It produces json-serializable python data structures (because data structures are easier to use and don't hide anything from you). And now the part that should interest you more: on top on that AST, I've built an high level "query and modification" library that wraps those AST nodes into objects. I've put a lot of efforts in making this library intuitive and very easy to use while removing you the burden of having to deal with low level details. This "BeautifulSoup of the python source code" is called Redbaron. It looks like this: from redbaron import RedBaron # simple API # pass string red = RedBaron("some_value = 42") # get string back red.dumps() Queries are like BeautifulSoup: red.find("int", value=4) red.find_all("def", name="stuff") (You can pass lambda/regex/special syntaxe for globs/regex etc... to queries, they should be powerful enough for the vast majorities of your needs). Nodes modification is very simple: just pass source code stored in string and "voil?": red = RedBaron("some_value = 42") red[0].value = "1 + 1" # some_value = 1 + 1 red = RedBaron("def stuff():\n plop") red[0].value = "some_code" # def stuff():\n some_code # notice that the input is correctly formatting, indented and it # also takes care of not breaking the next node indentation # works too with decorators and where you expect it to works (It is possible to pass it ast datastructure or RedBaron objects to). And I've made an abstraction on top of "list of things" so you don't have to take care about when you need to put a separator (for eg: a "," in a list): red = RedBaron("[1, 2, 3]") red[0].append("plop") # [1, 2, 3, plop] # want to add a new django app to INSTALLED_APPS? just do: red.find("assignment", target=lambda x: x.dumps() == "INSTALLED_APPLICATIONS").value.append("'another_app'") # notice that the formatting of the list is detected # want to add "@profile" to every function of the root level for # line_profiler? red('def', recursive=False).map(lambda x: x.decorators.insert(0, '@profile')) # and remove them red("decorator", lambda x: x.dumps() == "@decorator").map(lambda x: x.parent.parent.decorators.remove(x)) # convert every "print a" to "logger.debug(a) red('print', value=lambda x: len(x) == 1).map(lambda x: x.replace('logger.debug(%s)' % x.value.dumps()) # and print a, b, c to logger.debug("%s %s %s" % (a, b, c)) red('print', value=lambda x: len(x) == 1).map(lambda x: x.replace('logger.debug("%s" % (%s))' % (" ".join('%s' * len(x.value))) Both library and fully tested (more than 2000 tests in total), fully *documented* (with lots of examples) and under freesoftware licences. I consider RedBaron to be in alpha stage, it is already very stable but a significant number of edge cases are probably not handled yet. Important point: RedBaron is not and will not do static analysis, I'm probably going to integrate (or integrate RedBaron into) a tool that already do that like astroid or rope. Links: * RedBaron tutorial: https://redbaron.readthedocs.org/en/latest/tuto.html * RedBaron documentation: https://redbaron.readthedocs.org * RedBaron source code: https://github.com/psycojoker/redbaron * Baron (the AST) source code: https://github.com/psycojoker/baron * Baron documentation: https://baron.readthedocs.org I hope that I have trigger your interest and I'm very interested by your feedback, Have a nice day and thanks for your time, PS: I've only been aware of the capacities of lib2to3 since 2 months and was very unhappy to discover it so late (I've spent months or googling before deciding to start this project), I'll probably swap my parser with lib2to3 one in the future. -- Laurent Peuch -- Bram From davidhalter88 at gmail.com Sat Nov 15 16:49:27 2014 From: davidhalter88 at gmail.com (Dave Halter) Date: Sat, 15 Nov 2014 16:49:27 +0100 Subject: [code-quality] RedBaron, a bottom-up refactoring lib/tool for python In-Reply-To: <20141114120553.GA2174@griboiedov> References: <20141114120553.GA2174@griboiedov> Message-ID: Hi Laurent Great to see somebody finally tackling refactoring. I'm answering, because I think we're working on the same issue. But we have finished two different parts: You have finished a refactoring implementation and I have finished the static analysis part. I'm the author of Jedi. https://github.com/davidhalter/jedi/ I'm currently working on the integration of the lib2to3 parser into Jedi. This would make refactoring really easy (I'm about 50% done with the parser). It's also well tested and offers a few other advantages. In a perfect world, we could now combine our projects :-) I will look in detail at Red Baron on Monday. ~ Dave PS: If you want to use a tool for static analysis, please use either Jedi or astroid, I don't think rope still makes sense, because the project is inactive. 2014-11-14 13:05 GMT+01:00 Laurent Peuch : > Hello everyone, > > Someone has suggested me to talk about the project I'm working on > right now on this mailing list because this has a lot of chances to > interest you. > > This tool is an answer to a frustration that I've had while trying to > build tools for python and projects I was working on. While there is > already good capacities in python to analyse code (ast.py, astroid > (while it wasn't out at that time)), the "write code that modify > source code" was really missing (in my opinion and my knowledge of the > existing tools). > > I wanted a very intuitive and easy to use library that allows me to > query and modify my source code only in the place I wanted to modify > it without touching the rest of the code. So I've built what can be > describe as "the BeautifulSoup of python source code". > > To do so, I've built what can be called "a lossless AST" for python > (designed to be used by humans), an AST that satisfy this equation: > > source_code == ast_to_source(source_to_ast(source_code)) > > It produces json-serializable python data structures (because data > structures are easier to use and don't hide anything from you). > > And now the part that should interest you more: on top on that AST, > I've built an high level "query and modification" library that wraps > those AST nodes into objects. I've put a lot of efforts in making this > library intuitive and very easy to use while removing you the burden > of having to deal with low level details. This "BeautifulSoup of the > python source code" is called Redbaron. > > It looks like this: > > from redbaron import RedBaron > > # simple API > > # pass string > red = RedBaron("some_value = 42") > > # get string back > red.dumps() > > Queries are like BeautifulSoup: > > red.find("int", value=4) > red.find_all("def", name="stuff") > > (You can pass lambda/regex/special syntaxe for globs/regex etc... to > queries, they should be powerful enough for the vast majorities of > your needs). > > Nodes modification is very simple: just pass source code stored in > string and "voil?": > > red = RedBaron("some_value = 42") > red[0].value = "1 + 1" # some_value = 1 + 1 > > red = RedBaron("def stuff():\n plop") > red[0].value = "some_code" # def stuff():\n some_code > > # notice that the input is correctly formatting, indented and it > # also takes care of not breaking the next node indentation > # works too with decorators and where you expect it to works > > (It is possible to pass it ast datastructure or RedBaron objects > to). > > And I've made an abstraction on top of "list of things" so you don't > have to take care about when you need to put a separator (for eg: a > "," in a list): > > red = RedBaron("[1, 2, 3]") > red[0].append("plop") # [1, 2, 3, plop] > > # want to add a new django app to INSTALLED_APPS? just do: > red.find("assignment", target=lambda x: x.dumps() == > "INSTALLED_APPLICATIONS").value.append("'another_app'") > # notice that the formatting of the list is detected > > # want to add "@profile" to every function of the root level for > # line_profiler? > red('def', recursive=False).map(lambda x: x.decorators.insert(0, > '@profile')) > > # and remove them > red("decorator", lambda x: x.dumps() == "@decorator").map(lambda x: > x.parent.parent.decorators.remove(x)) > > # convert every "print a" to "logger.debug(a) > red('print', value=lambda x: len(x) == 1).map(lambda x: > x.replace('logger.debug(%s)' % x.value.dumps()) > > # and print a, b, c to logger.debug("%s %s %s" % (a, b, c)) > red('print', value=lambda x: len(x) == 1).map(lambda x: > x.replace('logger.debug("%s" % (%s))' % (" ".join('%s' * len(x.value))) > > Both library and fully tested (more than 2000 tests in total), fully > *documented* (with lots of examples) and under freesoftware licences. > I consider RedBaron to be in alpha stage, it is already very stable > but a significant number of edge cases are probably not handled yet. > > Important point: RedBaron is not and will not do static analysis, > I'm probably going to integrate (or integrate RedBaron into) a tool > that already do that like astroid or rope. > > Links: > > * RedBaron tutorial: https://redbaron.readthedocs.org/en/latest/tuto.html > * RedBaron documentation: https://redbaron.readthedocs.org > * RedBaron source code: https://github.com/psycojoker/redbaron > > * Baron (the AST) source code: https://github.com/psycojoker/baron > * Baron documentation: https://baron.readthedocs.org > > I hope that I have trigger your interest and I'm very > interested by your feedback, > > Have a nice day and thanks for your time, > > PS: I've only been aware of the capacities of lib2to3 since 2 months > and was very unhappy to discover it so late (I've spent months or > googling before deciding to start this project), I'll probably swap my > parser with lib2to3 one in the future. > > -- > > Laurent Peuch -- Bram > _______________________________________________ > code-quality mailing list > code-quality at python.org > https://mail.python.org/mailman/listinfo/code-quality > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sylvain.thenault at logilab.fr Mon Nov 17 10:01:51 2014 From: sylvain.thenault at logilab.fr (Sylvain =?utf-8?B?VGjDqW5hdWx0?=) Date: Mon, 17 Nov 2014 10:01:51 +0100 Subject: [code-quality] RedBaron, a bottom-up refactoring lib/tool for python In-Reply-To: References: <20141114120553.GA2174@griboiedov> Message-ID: <20141117090151.GA3267@logilab.fr> On 15 novembre 16:49, Dave Halter wrote: > Hi Laurent Hi Laurent, David, > Great to see somebody finally tackling refactoring. indeed! > I'm answering, because I think we're working on the same issue. But we have > finished two different parts: You have finished a refactoring > implementation and I have finished the static analysis part. I'm the author > of Jedi. https://github.com/davidhalter/jedi/ Could I ask what do you mean by static analysis in the context of a completion library? > I'm currently working on the integration of the lib2to3 parser into Jedi. > This would make refactoring really easy (I'm about 50% done with the > parser). It's also well tested and offers a few other advantages. > > In a perfect world, we could now combine our projects :-) I will look in > detail at Red Baron on Monday. David, we've been talking about this during the latest EuroPython, and I've talked with Laurent yesterday at the Capitole du Libre in Toulouse: IMO we could start by extracting from lib2to3 "the" parser that could be used by every tools like ours (refactoring, completion, static analysis...). It would be: * loss-less (comments, indents...) * accurate (eg from/to line numbers) * fast * version agnostic within a reasonable frame (eg 2.7 -> 3.4?) I guess almost every one on this list would be interested in such a parser, even if most would have to do a second pass on the generated tree to get more "business oriented" tree for their own project. Whatever, we (pylint guys) would be greatly interested. -- Sylvain Th?nault, LOGILAB, Paris (01.45.32.03.12) - Toulouse (05.62.17.16.42) Formations Python, Debian, M?th. Agiles: http://www.logilab.fr/formations D?veloppement logiciel sur mesure: http://www.logilab.fr/services CubicWeb, the semantic web framework: http://www.cubicweb.org From schettino72 at gmail.com Mon Nov 17 10:58:02 2014 From: schettino72 at gmail.com (Eduardo Schettino) Date: Mon, 17 Nov 2014 17:58:02 +0800 Subject: [code-quality] RedBaron, a bottom-up refactoring lib/tool for python In-Reply-To: <20141117090151.GA3267@logilab.fr> References: <20141114120553.GA2174@griboiedov> <20141117090151.GA3267@logilab.fr> Message-ID: Hi guys, I also have done some work in this area [1]. Well, I didnt get into reactoring yet... Anyway, I thought you might be interested. My approach was to convert python code to XML (and back to python) This allow any generic XML tool to be used to manipulate the XML-code, and then convert back to python without the need to create specific tools. The implementation uses python's AST tree and the tokenize module to retrieve information that is thrown away by the AST. This way I didnt need to implement my own python parser. It ended up being much more work than I expected but I still believe thats better than maitaining another parser. [1] http://pythonhosted.org/pyRegurgitator/#py2xml-experimental https://github.com/schettino72/pyRegurgitator cheers, Eduardo On Mon, Nov 17, 2014 at 5:01 PM, Sylvain Th?nault < sylvain.thenault at logilab.fr> wrote: > On 15 novembre 16:49, Dave Halter wrote: > > Hi Laurent > > Hi Laurent, David, > > > Great to see somebody finally tackling refactoring. > > indeed! > > > I'm answering, because I think we're working on the same issue. But we > have > > finished two different parts: You have finished a refactoring > > implementation and I have finished the static analysis part. I'm the > author > > of Jedi. https://github.com/davidhalter/jedi/ > > Could I ask what do you mean by static analysis in the context of a > completion > library? > > > I'm currently working on the integration of the lib2to3 parser into Jedi. > > This would make refactoring really easy (I'm about 50% done with the > > parser). It's also well tested and offers a few other advantages. > > > > In a perfect world, we could now combine our projects :-) I will look in > > detail at Red Baron on Monday. > > David, we've been talking about this during the latest EuroPython, and I've > talked with Laurent yesterday at the Capitole du Libre in Toulouse: IMO we > could > start by extracting from lib2to3 "the" parser that could be used by every > tools > like ours (refactoring, completion, static analysis...). It would be: > * loss-less (comments, indents...) > * accurate (eg from/to line numbers) > * fast > * version agnostic within a reasonable frame (eg 2.7 -> 3.4?) > > I guess almost every one on this list would be interested in such a > parser, even > if most would have to do a second pass on the generated tree to get more > "business oriented" tree for their own project. Whatever, we (pylint guys) > would > be greatly interested. > > -- > Sylvain Th?nault, LOGILAB, Paris (01.45.32.03.12) - Toulouse > (05.62.17.16.42) > Formations Python, Debian, M?th. Agiles: http://www.logilab.fr/formations > D?veloppement logiciel sur mesure: http://www.logilab.fr/services > CubicWeb, the semantic web framework: http://www.cubicweb.org > _______________________________________________ > code-quality mailing list > code-quality at python.org > https://mail.python.org/mailman/listinfo/code-quality > -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidhalter88 at gmail.com Mon Nov 17 12:27:31 2014 From: davidhalter88 at gmail.com (Dave Halter) Date: Mon, 17 Nov 2014 12:27:31 +0100 Subject: [code-quality] RedBaron, a bottom-up refactoring lib/tool for python In-Reply-To: <20141117090151.GA3267@logilab.fr> References: <20141114120553.GA2174@griboiedov> <20141117090151.GA3267@logilab.fr> Message-ID: Sylvain, 2014-11-17 10:01 GMT+01:00 Sylvain Th?nault : > On 15 novembre 16:49, Dave Halter wrote: > > Hi Laurent > > Hi Laurent, David, > > > Great to see somebody finally tackling refactoring. > > indeed! > > > I'm answering, because I think we're working on the same issue. But we > have > > finished two different parts: You have finished a refactoring > > implementation and I have finished the static analysis part. I'm the > author > > of Jedi. https://github.com/davidhalter/jedi/ > > Could I ask what do you mean by static analysis in the context of a > completion > library? > Not the "linting" part. The name resolution/tuple assignment/function execution part. I assumed he meant that when he talked about using rope or astroid for static analysis. Jedi's static analysis part is not much further than what I showed at EuroPython, because I'm currently rewriting the parser stuff. > > > I'm currently working on the integration of the lib2to3 parser into Jedi. > > This would make refactoring really easy (I'm about 50% done with the > > parser). It's also well tested and offers a few other advantages. > > > > In a perfect world, we could now combine our projects :-) I will look in > > detail at Red Baron on Monday. > > David, we've been talking about this during the latest EuroPython, and I've > talked with Laurent yesterday at the Capitole du Libre in Toulouse: IMO we > could > start by extracting from lib2to3 "the" parser that could be used by every > tools > like ours (refactoring, completion, static analysis...). It would be: > * loss-less (comments, indents...) > * accurate (eg from/to line numbers) > * fast > * version agnostic within a reasonable frame (eg 2.7 -> 3.4?) > Yes, that's what I'm trying to create now (based on lib2to3). However my biggest problem is that I have to rewrite my evaluation engine as well, because it was depending on the old parser. I have two additional constraints: - decent error recovery - memory efficient The "fast" part is something I'm very eager to implement. I have done this before with my old parser. My approach is to re-parse only the parts of the file that have changed. My idea of "version agnostic" is to have the parser and the evaluation engine try to adhere to one version. My goal is to give Jedi a Python version number and Jedi would work according to that. This would make linting really cool, because you can run the linter for different Python versions. I would recommend you to wait until I have finished my parser (1-2 months) or can at least report back. You can then either take the parser and fork it or take the evaluation engine as well. ~ Dave > > I guess almost every one on this list would be interested in such a > parser, even > if most would have to do a second pass on the generated tree to get more > "business oriented" tree for their own project. Whatever, we (pylint guys) > would > be greatly interested. > > -- > Sylvain Th?nault, LOGILAB, Paris (01.45.32.03.12) - Toulouse > (05.62.17.16.42) > Formations Python, Debian, M?th. Agiles: http://www.logilab.fr/formations > D?veloppement logiciel sur mesure: http://www.logilab.fr/services > CubicWeb, the semantic web framework: http://www.cubicweb.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cortex at worlddomination.be Mon Nov 17 21:00:11 2014 From: cortex at worlddomination.be (Laurent Peuch) Date: Mon, 17 Nov 2014 21:00:11 +0100 Subject: [code-quality] RedBaron, a bottom-up refactoring lib/tool for python In-Reply-To: References: <20141114120553.GA2174@griboiedov> <20141117090151.GA3267@logilab.fr> Message-ID: <20141117200011.GA11340@griboiedov> Hello everyone and thanks for your answers :) > > > I'm currently working on the integration of the lib2to3 parser into Jedi. > > > This would make refactoring really easy (I'm about 50% done with the > > > parser). It's also well tested and offers a few other advantages. > > > > > > In a perfect world, we could now combine our projects :-) I will look in > > > detail at Red Baron on Monday. > > > > David, we've been talking about this during the latest EuroPython, and I've > > talked with Laurent yesterday at the Capitole du Libre in Toulouse: IMO we > > could > > start by extracting from lib2to3 "the" parser that could be used by every > > tools > > like ours (refactoring, completion, static analysis...). It would be: > > * loss-less (comments, indents...) > > * accurate (eg from/to line numbers) > > * fast > > * version agnostic within a reasonable frame (eg 2.7 -> 3.4?) > > > > Yes, that's what I'm trying to create now (based on lib2to3). However my > biggest problem is that I have to rewrite my evaluation engine as well, > because it was depending on the old parser. I have two additional > constraints: > > - decent error recovery > - memory efficient > > The "fast" part is something I'm very eager to implement. I have done this > before with my old parser. My approach is to re-parse only the parts of the > file that have changed. > My idea of "version agnostic" is to have the parser and the evaluation > engine try to adhere to one version. My goal is to give Jedi a Python > version number and Jedi would work according to that. This would make > linting really cool, because you can run the linter for different Python > versions. > > I would recommend you to wait until I have finished my parser (1-2 months) > or can at least report back. You can then either take the parser and fork > it or take the evaluation engine as well. Well, wouldn't it be a good idea if instead of us waiting 1-2 months that you finish your parser then seeing if it also fits our needs we discuss a bit of what each one of us needs and see if we could find an agreement on what could be a common ST for all of us? I can totally understand that discussing this right now might not be the most appealing idea but maybe think a bit about it of the benefit that we could get from a common AST for all of us: * only one code base to maintain instead of everyone doing his own parser on his side * we could share our efforts on making it as good as possible * more time spend on doing the actual tools than the backend * a de facto reference for every python developer that wants to join the field of tooling: more people -> more tools -> better python in general I really think that we should at least try to see if it's possible, having this kind of tool would really benefit us and the python community in general. What do you think? -- Laurent Peuch -- Bram From cortex at worlddomination.be Mon Nov 17 21:07:58 2014 From: cortex at worlddomination.be (Laurent Peuch) Date: Mon, 17 Nov 2014 21:07:58 +0100 Subject: [code-quality] RedBaron, a bottom-up refactoring lib/tool for python In-Reply-To: References: <20141114120553.GA2174@griboiedov> <20141117090151.GA3267@logilab.fr> Message-ID: <20141117200758.GB11340@griboiedov> Hello Eduardo, > My approach was to convert python code to XML (and back to python) > This allow any generic XML tool to be used to manipulate the XML-code, > and then convert back to python without the need to create specific tools. Funny, you aren't the first one I've found who has taken this approach. This guys is converting lib2to3 output to xml and throwing lxml on it https://github.com/bukzor/RefactorLib I don't know what it's worth, he doesn't seem to have done any advertisement for it and the code is not very big. > The implementation uses python's AST tree and the tokenize module > to retrieve information that is thrown away by the AST. > This way I didnt need to implement my own python parser. > It ended up being much more work than I expected but I still believe thats > better than maitaining another parser. Oh, I understand you so much, I didn't knew how much that would took me to parse python, if I knew I probably would have never done this project. Cheers, -- Laurent Peuch -- Bram From stefan.bucur at gmail.com Mon Nov 17 18:18:03 2014 From: stefan.bucur at gmail.com (Stefan Bucur) Date: Mon, 17 Nov 2014 17:18:03 +0000 Subject: [code-quality] Static checker for common Python programming errors References: Message-ID: I'm developing a Python static analysis tool that flags common programming errors in Python programs. The tool is meant to complement other tools like Pylint (which perform checks at lexical and AST level) by going deeper with the code analysis and keeping track of the possible control flow paths in the program (path-sensitive analysis). For instance, a path-sensitive analysis detects that the following snippet of code would raise an AttributeError exception: if object is None: # If the True branch is taken, we know the object is None object.doSomething() # ... so this statement would always fail I wanted first to tap into people's experience and get a sense of what common pitfalls in the language & its standard library such a static checker should look for. Just as an example of what I mean, here [1] is a list of static checks for the C++ language, as part of the Clang static analyzer project. My preliminary list of Python checks is quite rudimentary, but maybe could serve as a discussion starter: * Proper Unicode handling (for 2.x) - encode() is not called on str object - decode() is not called on unicode object * Check for integer division by zero * Check for None object dereferences Thanks a lot, Stefan Bucur [1] http://clang-analyzer.llvm.org/available_checks.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Tue Nov 18 06:20:16 2014 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 18 Nov 2014 16:20:16 +1100 Subject: [code-quality] Static checker for common Python programming errors References: Message-ID: <85zjbp13vz.fsf@benfinney.id.au> Stefan Bucur writes: > For instance, a path-sensitive analysis detects that the following > snippet of code would raise an AttributeError exception: > > if object is None: # If the True branch is taken, we know the object is None > object.doSomething() # ... so this statement would always fail Be careful with this. The above example would behave as you say; but it is a special case because re-binding ?None? is a SyntaxError. For most other built-ins, and for most other names, the above type of check cannot be done with static analysis. > I wanted first to tap into people's experience and get a sense of what > common pitfalls in the language & its standard library such a static > checker should look for. Prior art to investigate includes PyLint and PyFlakes. You probably already know, but it bears saying explicitly. > My preliminary list of Python checks is quite rudimentary [?] Obvious ones: * Mutable-value argument defaults (should instead use the ?None? or a custom sentinel). I think you can only detect this when the default is specified as some literal value of a built-in type. * Comparing to None by equality (should be comparing by identity). * A single value in parens (author might mistakenly believe it's a tuple). Heck, any of the Idiomatic Python warnings that you can detect . -- \ ?[The RIAA] have the patience to keep stomping. They're playing | `\ whack-a-mole with an infinite supply of tokens.? ?kennon, | _o__) http://kuro5hin.org/ | Ben Finney From pcmanticore at gmail.com Tue Nov 18 13:25:07 2014 From: pcmanticore at gmail.com (Claudiu Popa) Date: Tue, 18 Nov 2014 14:25:07 +0200 Subject: [code-quality] Static checker for common Python programming errors In-Reply-To: References: Message-ID: On Mon, Nov 17, 2014 at 7:18 PM, Stefan Bucur wrote: > I'm developing a Python static analysis tool that flags common programming > errors in Python programs. The tool is meant to complement other tools like > Pylint (which perform checks at lexical and AST level) by going deeper with > the code analysis and keeping track of the possible control flow paths in > the program (path-sensitive analysis). > Hey, Stefan. That's great! It's really cool to see more people interested in static analysis with Python. Control flow analysis is missing in Pylint and due to this, we have a couple of false positives. For instance, the following code will emit a raising-bad-type warning, even though exc is later reassigned (but not deduced by the inference engine): exc = None for i in range(10): try: 1 / 0 break except Exception as e: exc = e else: raise exc I was planning to tackle this issue, by adding a rudimentary flow statement analysis to Pylint's inference engine, but oriented on high level paths. From davidhalter88 at gmail.com Tue Nov 18 14:00:43 2014 From: davidhalter88 at gmail.com (Dave Halter) Date: Tue, 18 Nov 2014 14:00:43 +0100 Subject: [code-quality] RedBaron, a bottom-up refactoring lib/tool for python In-Reply-To: <20141117200011.GA11340@griboiedov> References: <20141114120553.GA2174@griboiedov> <20141117090151.GA3267@logilab.fr> <20141117200011.GA11340@griboiedov> Message-ID: 2014-11-17 21:00 GMT+01:00 Laurent Peuch : > Hello everyone and thanks for your answers :) > > > > > I'm currently working on the integration of the lib2to3 parser into > Jedi. > > > > This would make refactoring really easy (I'm about 50% done with the > > > > parser). It's also well tested and offers a few other advantages. > > > > > > > > In a perfect world, we could now combine our projects :-) I will > look in > > > > detail at Red Baron on Monday. > > > > > > David, we've been talking about this during the latest EuroPython, and > I've > > > talked with Laurent yesterday at the Capitole du Libre in Toulouse: > IMO we > > > could > > > start by extracting from lib2to3 "the" parser that could be used by > every > > > tools > > > like ours (refactoring, completion, static analysis...). It would be: > > > * loss-less (comments, indents...) > > > * accurate (eg from/to line numbers) > > > * fast > > > * version agnostic within a reasonable frame (eg 2.7 -> 3.4?) > > > > > > > Yes, that's what I'm trying to create now (based on lib2to3). However my > > biggest problem is that I have to rewrite my evaluation engine as well, > > because it was depending on the old parser. I have two additional > > constraints: > > > > - decent error recovery > > - memory efficient > > > > The "fast" part is something I'm very eager to implement. I have done > this > > before with my old parser. My approach is to re-parse only the parts of > the > > file that have changed. > > My idea of "version agnostic" is to have the parser and the evaluation > > engine try to adhere to one version. My goal is to give Jedi a Python > > version number and Jedi would work according to that. This would make > > linting really cool, because you can run the linter for different Python > > versions. > > > > I would recommend you to wait until I have finished my parser (1-2 > months) > > or can at least report back. You can then either take the parser and fork > > it or take the evaluation engine as well. > > Well, wouldn't it be a good idea if instead of us waiting 1-2 months > that you finish your parser then seeing if it also fits our needs we > discuss a bit of what each one of us needs and see if we could find an > agreement on what could be a common ST for all of us? > > I can totally understand that discussing this right now might not be > the most appealing idea but maybe think a bit about it of the benefit > that we could get from a common AST for all of us: > > * only one code base to maintain instead of everyone doing his > own parser on his side > * we could share our efforts on making it as good as possible > * more time spend on doing the actual tools than the backend > * a de facto reference for every python developer that wants to join > the field of tooling: more people -> more tools -> better python in > general > > I really think that we should at least try to see if it's possible, > having this kind of tool would really benefit us and the python > community in general. > > What do you think? > I totally agree. However the Jedi's parser right now (parser branch) is in a very confusing state. There's still a lot of old code lingering around that doesn't help a lot in understanding it. I'm going to clean it until the end of this week and then we can talk about it. The full 1-2 months would include Jedi being fast again + passing all tests. So I don't think we need to wait for that. I'm working full time on this project so I'm progressing quite quickly. ~ Dave > > -- > > Laurent Peuch -- Bram > -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidhalter88 at gmail.com Tue Nov 18 14:01:45 2014 From: davidhalter88 at gmail.com (Dave Halter) Date: Tue, 18 Nov 2014 14:01:45 +0100 Subject: [code-quality] Static checker for common Python programming errors In-Reply-To: References: Message-ID: Hi Stefan I'm playing with this as well in Jedi. I'm pretty far with flow analysis and AttributeErrors. (This includes everything you mention above except integer division by zero). Would be easy to implement in Jedi, though. I just have different priorities, at the moment. If you have some time on your hands you can watch my EuroPython talk about this: https://www.youtube.com/watch?v=DfVHSw0iOsk I'm also glad to skype (gaukler_) if you're interested. Jedi is not well known for doing static analysis. But it's my goal to change this now. ~ Dave 2014-11-17 18:18 GMT+01:00 Stefan Bucur : > I'm developing a Python static analysis tool that flags common programming > errors in Python programs. The tool is meant to complement other tools like > Pylint (which perform checks at lexical and AST level) by going deeper with > the code analysis and keeping track of the possible control flow paths in > the program (path-sensitive analysis). > > For instance, a path-sensitive analysis detects that the following snippet > of code would raise an AttributeError exception: > > if object is None: # If the True branch is taken, we know the object is > None > object.doSomething() # ... so this statement would always fail > > I wanted first to tap into people's experience and get a sense of what > common pitfalls in the language & its standard library such a static > checker should look for. Just as an example of what I mean, here [1] is a > list of static checks for the C++ language, as part of the Clang static > analyzer project. > > My preliminary list of Python checks is quite rudimentary, but maybe could > serve as a discussion starter: > > * Proper Unicode handling (for 2.x) > - encode() is not called on str object > - decode() is not called on unicode object > * Check for integer division by zero > * Check for None object dereferences > > Thanks a lot, > Stefan Bucur > > [1] http://clang-analyzer.llvm.org/available_checks.html > > > _______________________________________________ > code-quality mailing list > code-quality at python.org > https://mail.python.org/mailman/listinfo/code-quality > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve+python at pearwood.info Tue Nov 18 13:31:10 2014 From: steve+python at pearwood.info (Steven D'Aprano) Date: Tue, 18 Nov 2014 23:31:10 +1100 Subject: [code-quality] Static checker for common Python programming errors In-Reply-To: References: Message-ID: <20141118123110.GA15212@ando.pearwood.info> On Mon, Nov 17, 2014 at 05:18:03PM +0000, Stefan Bucur wrote: > I wanted first to tap into people's experience and get a sense of what > common pitfalls in the language & its standard library such a static > checker should look for. [...] > * Proper Unicode handling (for 2.x) > - encode() is not called on str object > - decode() is not called on unicode object > * Check for integer division by zero > * Check for None object dereferences This is a very interesting question, and quite hard to answer too. I presume that there will be a way to disable the check for individual lines of code, because these are not always wrong. * Check for `is` comparisons against arbitrary objects, especially int and str literals. * Equality comparisons against None (should use `is`). * Calling str.find() and str.rfind() without checking for a return result of -1. * Calling re.match() and similar without checking for a result of None. * alist = alist.sort() and similar. * Calling print for its return value, e.g.: print("template %s") % obj * Mutable default values in functions and methods. * Modifying a sequence or iterator while iterating over it, e.g. for i, x in enumerate(alist): if condition(x): del alist[i] I've done all of these :-( -- Steven From shlomme at gmail.com Thu Nov 20 22:19:17 2014 From: shlomme at gmail.com (Torsten Marek) Date: Thu, 20 Nov 2014 22:19:17 +0100 Subject: [code-quality] [ANN] Astroid 1.3 released Message-ID: Hello! I'm happy to announce the availability of astroid 1.3. Functional changes in this release are mostly bugfixes and small improvements, the large change is that support for Python versions < 2.7 has been dropped, which allows us to support Python 2.7 and 3.3+ from a single codebase, without relying on 2to3. In addition to that, the distribution itself has been restructured to make `python setup.py develop` work correctly. Other important changes: * Various speed improvements. Patch by Alex Munroe. * Add pytest brain plugin. Patch by Robbie Coomber. * Only C extensions from trusted sources (the standard library) are loaded into the examining Python process to build an AST from the live module. * Path names on case-insensitive filesystems are now properly handled. This fixes the stdlib detection code on Windows. * Metaclass-generating functions like six.with_metaclass are now supported via some explicit detection code. * Fix a maximum recursion error occured during the inference, where statements with the same name weren't filtered properly. Closes pylint issue #295. A new release of pylint will follow shortly. Happy hacking! // Torsten -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip.montanaro at gmail.com Fri Nov 21 15:32:55 2014 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Fri, 21 Nov 2014 08:32:55 -0600 Subject: [code-quality] Fwd: The doc demo have some problems In-Reply-To: <2014112115561728848213@gmail.com> References: <201411211521097974749@gmail.com> <2014112115561728848213@gmail.com> Message-ID: The appended email crossed the transom at webmaster at python.org today. Apologies for the hot mess Gmail makes of forwarded emails. I wonder if it makes sense to warn about misspelled dunder names. I tried pylinting (1.1.0, sorry, it's what I have here at work) and flake8ing (2.0) this: class Foo(object): def __init_(self): self.x = 0 def _add_(self, other): return self.x + other.x Pylint complained about attribute x not being defined in __init__, but that's a rather indirect message. It failed to say anything about _add_. Flake8 said nothing. Is this a common enough problem that these sort of tools should warn about mistakes in the leading/trailing underscore count of otherwise dunder attributes? Skip ---------- Forwarded message ---------- From: protestmailself Date: Fri, Nov 21, 2014 at 1:56 AM Subject: Re: The doc demo have some problems To: webmaster Hi: I make a mistake using _init_ not the __init__ ------------------------------ protestmailself *From:* protestmailself *Date:* 2014-11-21 15:30 *To:* webmaster *Subject:* The doc demo have some problems Hi: I using the code at location: [ https://docs.python.org/2/tutorial/classes.html] using python 2.7.8 class Complex: def _init_(self, realpart, imagpart): self.r = realpart self.i = imagpart x = Complex(3.0, -4.5) x.r, x.i I got the error: TypeError: this constructor takes no arguments i think this is a mistake as the demo not define a constructor, but use a constructor with two arguments. Change to this can be passed. class Complex: r = 0 i = 0 def _init_(self, realpart, imagpart): self.r = realpart self.i = imagpart x = Complex() x._init_(3.0, -4.5) x.r, x.i Best regards ------------------------------ protestmailself -------------- next part -------------- An HTML attachment was scrubbed... URL: From amulhern at redhat.com Fri Nov 21 18:58:13 2014 From: amulhern at redhat.com (Anne Mulhern) Date: Fri, 21 Nov 2014 12:58:13 -0500 (EST) Subject: [code-quality] Advice regarding caching values as a side-effect of visiting classes In-Reply-To: <1114454924.2638246.1416591897742.JavaMail.zimbra@redhat.com> Message-ID: <812774972.2645807.1416592693466.JavaMail.zimbra@redhat.com> Hi! I've implemented a very simple analysis of the kind described in this enhancement request: https://bitbucket.org/logilab/pylint/issue/201/pointless-attribute-override-checker. The analysis is neither sound nor complete, but has proven useful for removing of bits of dead code. It can be made more sound by altering the algorithm so that it accumulates data by visiting classes, and subsequently processes that accumulated data, preferably after all classes have been visited, in order to determine messages to report. Is there some infrastructure in pylint that can accomodate an analysis that works that way? If there is, is there a working example that uses it? Thanks for any help, - mulhern From amulhern at redhat.com Fri Nov 21 18:32:02 2014 From: amulhern at redhat.com (Anne Mulhern) Date: Fri, 21 Nov 2014 12:32:02 -0500 (EST) Subject: [code-quality] How to detect unused PyLint declarations In-Reply-To: References: Message-ID: <911904694.2633444.1416591122917.JavaMail.zimbra@redhat.com> ----- Original Message ----- > From: "Kay Hayen" > To: code-quality at python.org > Sent: Wednesday, October 29, 2014 10:00:15 AM > Subject: [code-quality] How to detect unused PyLint declarations > > > Hello, > > I have coding rules that require me to annotate exceptions to rules > for PyLint, but occasionally it happens that I find PyLint rules disabled > that would no longer be necessary. > > Is there a way or script, or anything to detect these automatically? I > was thinking of writing something that removes PyLint disablers one > by one, and checks if that doesn't generate PyLint warnings, and > warn about those. Didn't do it so far, but I feel tempted to do this > now. > > However, to PyLint, this might be way more easier to implement, and > maybe it was done. I cannot find anything in the manpage though. > > Yours, > Kay > > _______________________________________________ > code-quality mailing list > code-quality at python.org > https://mail.python.org/mailman/listinfo/code-quality > This is something I am interested in as well. It looks like the I0021: Useless suppression of %s warning should report those cases. - mulhern From shlomme at gmail.com Fri Nov 21 23:12:06 2014 From: shlomme at gmail.com (Torsten Marek) Date: Fri, 21 Nov 2014 23:12:06 +0100 Subject: [code-quality] How to detect unused PyLint declarations In-Reply-To: <911904694.2633444.1416591122917.JavaMail.zimbra@redhat.com> References: <911904694.2633444.1416591122917.JavaMail.zimbra@redhat.com> Message-ID: Yes, I0021 is useless-suppression. There's also suppressed-message, to show what kinds of messages have been suppressed. Looking at that every once in a while is interesting as well. Especially for badly understood messages, there is a tendency in developers to just paper over the warnings instead of fixing the underlying issue. // Torsten 2014-11-21 18:32 GMT+01:00 Anne Mulhern : > > > > > ----- Original Message ----- > > From: "Kay Hayen" > > To: code-quality at python.org > > Sent: Wednesday, October 29, 2014 10:00:15 AM > > Subject: [code-quality] How to detect unused PyLint declarations > > > > > > Hello, > > > > I have coding rules that require me to annotate exceptions to rules > > for PyLint, but occasionally it happens that I find PyLint rules disabled > > that would no longer be necessary. > > > > Is there a way or script, or anything to detect these automatically? I > > was thinking of writing something that removes PyLint disablers one > > by one, and checks if that doesn't generate PyLint warnings, and > > warn about those. Didn't do it so far, but I feel tempted to do this > > now. > > > > However, to PyLint, this might be way more easier to implement, and > > maybe it was done. I cannot find anything in the manpage though. > > > > Yours, > > Kay > > > > _______________________________________________ > > code-quality mailing list > > code-quality at python.org > > https://mail.python.org/mailman/listinfo/code-quality > > > > This is something I am interested in as well. > > It looks like the I0021: Useless suppression of %s warning > should report those cases. > > - mulhern > _______________________________________________ > code-quality mailing list > code-quality at python.org > https://mail.python.org/mailman/listinfo/code-quality > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shlomme at gmail.com Fri Nov 21 23:32:15 2014 From: shlomme at gmail.com (Torsten Marek) Date: Fri, 21 Nov 2014 23:32:15 +0100 Subject: [code-quality] Advice regarding caching values as a side-effect of visiting classes In-Reply-To: <812774972.2645807.1416592693466.JavaMail.zimbra@redhat.com> References: <1114454924.2638246.1416591897742.JavaMail.zimbra@redhat.com> <812774972.2645807.1416592693466.JavaMail.zimbra@redhat.com> Message-ID: Hi Anne, you can write a checker that simply stores all classes it encounters and then walks the inheritance tree to find all violations in `close`. This checker method is called after all modules specified on the commandline have been checked, but before reports are generated, so emitting warnings at this point is possible. // Torsten 2014-11-21 18:58 GMT+01:00 Anne Mulhern : > Hi! > > I've implemented a very simple analysis of the kind described in this > enhancement > request: > https://bitbucket.org/logilab/pylint/issue/201/pointless-attribute-override-checker > . > The analysis is neither sound nor complete, but has proven useful for > removing of bits of > dead code. > > It can be made more sound by altering the algorithm so that it accumulates > data by visiting classes, > and subsequently processes that accumulated data, preferably after all > classes have been visited, > in order to determine messages to report. > > Is there some infrastructure in pylint that can accomodate an analysis > that works that way? > If there is, is there a working example that uses it? > > Thanks for any help, > > - mulhern > > _______________________________________________ > code-quality mailing list > code-quality at python.org > https://mail.python.org/mailman/listinfo/code-quality > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kay.hayen at gmail.com Sat Nov 22 09:46:10 2014 From: kay.hayen at gmail.com (Kay Hayen) Date: Sat, 22 Nov 2014 09:46:10 +0100 Subject: [code-quality] RedBaron, a bottom-up refactoring lib/tool for python In-Reply-To: <20141114120553.GA2174@griboiedov> References: <20141114120553.GA2174@griboiedov> Message-ID: Hello Laurent, for my coding conventions, which are a bit unusual, to me e.g. this is the correct function call, if every parameter is passed by keyword argument, and the "=" signs are aligned on the maximum level. I had looked into lib2to3, and found it terribly complex to work with visitors, to pick up enough state information, so I dropped my idea to make an automatic source code formatter for my preferred style. I think Redbaron might be much better suited to achieve that, it seems. However, in the past years, when I did these things, I have come to expose this sort of tree to the users in a way, so that they can make xpath queries. Can you make your tree lxml walkable, and basically allow xml transformations with its API. That way tests on the syntax tree could be expressed more easily. I am, in Nuitka, my Python compiler, exposing an XML dump of the final internal tree, and use tests with xpath queries on it, to e.g. identify print statements of non-constants, to declare these errors. While I don't do enough of these, to me, that is the natural way now of selecting data from a tree. My function calls to modify, and the values to use in there, would be pretty natural xpath queries, I suppose. I would then iterate in a for loop, determine the maximum identifier length of keyword arguments, update the spacing before the equals sign, in the node tree, with more xpath queries to find children. Performance might be a red herring there, but xpath queries that make me avoid touching nodes already properly formatted, would probably also be faster. Most likely performance is not an issue at all. But xpath can be run time compiled in principle. Let me know what you think of that. Yours, Kay -------------- next part -------------- An HTML attachment was scrubbed... URL: From kay.hayen at gmail.com Sat Nov 22 10:02:46 2014 From: kay.hayen at gmail.com (Kay Hayen) Date: Sat, 22 Nov 2014 10:02:46 +0100 Subject: [code-quality] How to detect unused PyLint declarations In-Reply-To: References: <911904694.2633444.1416591122917.JavaMail.zimbra@redhat.com> Message-ID: Hello there, I have deployed that, in my check-with-pylint on the factory git branch, I am now for the first time ever, in Nuitka, fully pylint clean with PyLint1.3. What that message above does, is to make PyLint very version dependent. There are new messages to that version, and there are fixed false alarms and new false alarms. But there is no way around that, right? As a tool for distributed development, that's not ideal. Say I would want to make these checks part of commit hooks, I would require people to have 1.3 somewhere, when they already have 1.4, or still 1.2, you get the idea. Or do I still get to implement the "delta of PyLint warnings didn't worsen" check for that commit hook, even though I am supposedly now PyLint clean. Seems there is no real way around that? Yours, Kay 2014-11-21 23:12 GMT+01:00 Torsten Marek : > Yes, I0021 is useless-suppression. > > There's also suppressed-message, to show what kinds of messages have been > suppressed. Looking at that every once in a while is interesting as well. > Especially for badly understood messages, there is a tendency in developers > to just paper over the warnings instead of fixing the underlying issue. > > // Torsten > > 2014-11-21 18:32 GMT+01:00 Anne Mulhern : > >> >> >> >> >> ----- Original Message ----- >> > From: "Kay Hayen" >> > To: code-quality at python.org >> > Sent: Wednesday, October 29, 2014 10:00:15 AM >> > Subject: [code-quality] How to detect unused PyLint declarations >> > >> > >> > Hello, >> > >> > I have coding rules that require me to annotate exceptions to rules >> > for PyLint, but occasionally it happens that I find PyLint rules >> disabled >> > that would no longer be necessary. >> > >> > Is there a way or script, or anything to detect these automatically? I >> > was thinking of writing something that removes PyLint disablers one >> > by one, and checks if that doesn't generate PyLint warnings, and >> > warn about those. Didn't do it so far, but I feel tempted to do this >> > now. >> > >> > However, to PyLint, this might be way more easier to implement, and >> > maybe it was done. I cannot find anything in the manpage though. >> > >> > Yours, >> > Kay >> > >> > _______________________________________________ >> > code-quality mailing list >> > code-quality at python.org >> > https://mail.python.org/mailman/listinfo/code-quality >> > >> >> This is something I am interested in as well. >> >> It looks like the I0021: Useless suppression of %s warning >> should report those cases. >> >> - mulhern >> _______________________________________________ >> code-quality mailing list >> code-quality at python.org >> https://mail.python.org/mailman/listinfo/code-quality >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ned at nedbatchelder.com Sat Nov 22 14:20:19 2014 From: ned at nedbatchelder.com (Ned Batchelder) Date: Sat, 22 Nov 2014 08:20:19 -0500 Subject: [code-quality] How to detect unused PyLint declarations In-Reply-To: References: <911904694.2633444.1416591122917.JavaMail.zimbra@redhat.com> Message-ID: <54708D93.5030501@nedbatchelder.com> On 11/22/14 4:02 AM, Kay Hayen wrote: > > Hello there, > > I have deployed that, in my check-with-pylint on the factory git > branch, I am now > for the first time ever, in Nuitka, fully pylint clean with PyLint1.3. > > What that message above does, is to make PyLint very version > dependent. There are new messages to that version, and there are fixed > false alarms and new false alarms. But there is no way around that, right? > > As a tool for distributed development, that's not ideal. Say I would > want to make these checks part of commit hooks, I would require people > to have 1.3 somewhere, when they already have 1.4, or still 1.2, you > get the idea. Wouldn't you pin the version of PyLint in your requirements.txt file? > > Or do I still get to implement the "delta of PyLint warnings didn't > worsen" check for that commit hook, even though I am supposedly now > PyLint clean. Seems there is no real way around that? BTW, diff-cover is a tool that can give you coverage, pep8, and pylint measurements just for your changes, rather than for the entire source tree that results from your changes. It's a good way to focus developers on improving quality metrics. --Ned. > > Yours, > Kay > > > 2014-11-21 23:12 GMT+01:00 Torsten Marek >: > > Yes, I0021 is useless-suppression. > > There's also suppressed-message, to show what kinds of messages > have been suppressed. Looking at that every once in a while is > interesting as well. Especially for badly understood messages, > there is a tendency in developers to just paper over the warnings > instead of fixing the underlying issue. > > // Torsten > > 2014-11-21 18:32 GMT+01:00 Anne Mulhern >: > > > > > > ----- Original Message ----- > > From: "Kay Hayen" > > > To: code-quality at python.org > > Sent: Wednesday, October 29, 2014 10:00:15 AM > > Subject: [code-quality] How to detect unused PyLint declarations > > > > > > Hello, > > > > I have coding rules that require me to annotate exceptions > to rules > > for PyLint, but occasionally it happens that I find PyLint > rules disabled > > that would no longer be necessary. > > > > Is there a way or script, or anything to detect these > automatically? I > > was thinking of writing something that removes PyLint > disablers one > > by one, and checks if that doesn't generate PyLint warnings, and > > warn about those. Didn't do it so far, but I feel tempted to > do this > > now. > > > > However, to PyLint, this might be way more easier to > implement, and > > maybe it was done. I cannot find anything in the manpage though. > > > > Yours, > > Kay > > > > _______________________________________________ > > code-quality mailing list > > code-quality at python.org > > https://mail.python.org/mailman/listinfo/code-quality > > > > This is something I am interested in as well. > > It looks like the I0021: Useless suppression of %s warning > should report those cases. > > - mulhern > _______________________________________________ > code-quality mailing list > code-quality at python.org > https://mail.python.org/mailman/listinfo/code-quality > > > > > > _______________________________________________ > code-quality mailing list > code-quality at python.org > https://mail.python.org/mailman/listinfo/code-quality -------------- next part -------------- An HTML attachment was scrubbed... URL: From cortex at worlddomination.be Sun Nov 23 01:39:38 2014 From: cortex at worlddomination.be (Laurent Peuch) Date: Sun, 23 Nov 2014 01:39:38 +0100 Subject: [code-quality] RedBaron, a bottom-up refactoring lib/tool for python In-Reply-To: References: <20141114120553.GA2174@griboiedov> Message-ID: <20141123003938.GA30679@griboiedov> Hello Kay, > for my coding conventions, which are a bit unusual, to me e.g. this is the > correct function call, if every parameter is passed by keyword argument, > and the "=" signs are aligned on the maximum level. I had looked into > lib2to3, and found it terribly complex to work with visitors, to pick up > enough state information, so I dropped my idea to make an automatic source > code formatter for my preferred style. > > I think Redbaron might be much better suited to achieve that, it seems. Yep, this is totally an use case for it. Actually you can go a level lower and directly do that with baron by writing a custom "dumper". As an example: I'm doing this very approach right now for pyfmt (which is a pretty formatter for python code, like gofmt basically) (WARNING: project is not ready yet). If you need inspiration on how to do that, you can find the code here: Old (but easier to understand but less flexible) approach: https://github.com/Psycojoker/pyfmt/blob/cf97f25138e866fb70fed7414c46801c92cbbd6e/pyfmt.py New approach I'm working on: https://github.com/Psycojoker/pyfmt/blob/5299a3f2233264ced5158a29dd1cb1d1586bfd81/pyfmt.py#L136-566 The new approach is using this data structure (which describes the order and content of every node): https://github.com/Psycojoker/baron/blob/master/baron/render.py#L114 Documentation: https://baron.readthedocs.org/en/latest/technical.html > [...] > Performance might be a red herring there, but xpath queries that make me > avoid touching nodes already properly formatted, would probably also be > faster. Most likely performance is not an issue at all. But xpath can be > run time compiled in principle. > > Let me know what you think of that. Since Baron (the AST) is lossless, as long as you do a lossless convertion between Baron and anything else you can do wathever you want. I actually wanted to do that with xml (as a "joke" to be honest with you, I don't have a very good appreciation of xml while I understand the power of some of its tools) to demonstration this property, I was not expecting people to actually request it. So, my technical opinion on the subject: * yes this is totally possible and quite easy to do, be sure to use this data structure if you want to do it yourself https://github.com/Psycojoker/baron/blob/master/baron/render.py#L114 (documentation: https://baron.readthedocs.org/en/latest/technical.html) * yes, there is quite a lot of chances that performance will take an hit * I will eventually do it (a baron <-> xml convertor lib), but this isn't very high in my todolist * as a practical solution, you will end up with a very low level data structure (just like baron) especially for modifications. Basically, you will face every low level details that I have abstracted for you in RedBaron. If you want to follow this path, I really recommend you to read RedBaron documentation first before doing so to evaluate if the price to pay is worth it. Have a nice weekend, -- Laurent Peuch -- Bram From pcmanticore at gmail.com Sun Nov 23 10:45:38 2014 From: pcmanticore at gmail.com (Claudiu Popa) Date: Sun, 23 Nov 2014 11:45:38 +0200 Subject: [code-quality] [ANN] Pylint 1.4 released Message-ID: Hello! On behalf of the Pylint development team, I'm happy to announce that Pylint 1.4 has been released. This release has a lot of improvements over the last one. One of the main differences is that support for Python versions < 2.7 has been droped, which allows us to support Python 2.7 and 3.3+ from a single codebase. Other important changes: * A Python 3 porting checker, activated by the new flag '--py3k'. This mode will disable all other checkers and will emit warnings and errors for constructs which are invalid or removed in Python 3. * New options for controlling the loading of C extensions. By default, only C extensions from the stdlib will be loaded into the active Python interpreter for inspection, because they can run arbitrary code on import. The option `--extension-pkg-whitelist` can be used to specify modules or packages that are safe to load. * An experimental support for using multiple workers to process files and packages, activated with the new --jobs flag. * A new spelling checker (disabled by default). * New warnings: boolean-datetime, logging-format-interpolation, inherit-non-class, mixed-line-endings * A lot of other small improvements and bug fixes. If you find any bugs, don't hesitate to open a new issue on our issue tracker. Enjoy! From kay.hayen at gmail.com Sun Nov 23 18:08:26 2014 From: kay.hayen at gmail.com (Kay Hayen) Date: Sun, 23 Nov 2014 18:08:26 +0100 Subject: [code-quality] RedBaron, a bottom-up refactoring lib/tool for python In-Reply-To: <20141123003938.GA30679@griboiedov> References: <20141114120553.GA2174@griboiedov> <20141123003938.GA30679@griboiedov> Message-ID: Hello Laurent, you wrote: > I think Redbaron might be much better suited to achieve that, it seems. > > Yep, this is totally an use case for it. Actually you can go a level > lower and directly do that with baron by writing a custom "dumper". > Before reading this, I already managed to pretty much immediately get this to work. I checked your pointers now, and I think they wouldn't do what I want to do, because I don't want to change layout, and when I want, I would be highly context sensitive. f( a = 1 bb = 2 ) and f(a = 1, bb = 2) are both OK to me. But in one case, always one space is used, and in other cases, vertical alignment is done. Many more things, where I would be context sensitive. I like e.g. contractions to be written like this: a = [ some_operation(a) for a in range(b) if a != 2 ] unless in cases where I don't do it, typically because there is no merit to reviewing it. So, I am more like patching, or at least detecting cases where the existing white space layout is inconsistent. > > Performance might be a red herring there, but xpath queries that make me > > avoid touching nodes already properly formatted, would probably also be > > faster. Most likely performance is not an issue at all. But xpath can be > > run time compiled in principle. > > > > Let me know what you think of that. > > Since Baron (the AST) is lossless, as long as you do a lossless > convertion between Baron and anything else you can do wathever you > want. I actually wanted to do that with xml (as a "joke" to be honest > with you, I don't have a very good appreciation of xml while I > understand the power of some of its tools) to demonstration this > property, I was not expecting people to actually request it. > I appreciate the power of the xpath query language. It's really good at saying find me nodes with attributes of that kind, and children with attributes of another kind, and in a certain range. It doesn't technically have to written as XML ever. But as a query API it's fantastic. I have employed it with decoded binary data that systems output, even wireshark captures, and with internal data recorded. Given a good tree, that's the thing to have for me. It's much like what SQL is to databases. A standard for querying. I am not an XML fanboy otherwise. :-) * I will eventually do it (a baron <-> xml convertor lib), but this > isn't very high in my todolist > This is mainly an idea for you. I am OK with working with the Python based API. I am mailing you with strangeness as I encountered it, but it's absolutely workable, and for a Python programmer, not a big deal. > * as a practical solution, you will end up with a very low level data > structure (just like baron) especially for modifications. Basically, > you will face every low level details that I have abstracted for you > in RedBaron. If you want to follow this path, I really recommend you > to read RedBaron documentation first before doing so to evaluate if > the price to pay is worth it. > I am not sure, why the XML tree would be all that different from what "node.help()" does. Surely the first, second, third, etc. formatting will simply become children, few people will look at. Yours, Kay -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidhalter88 at gmail.com Mon Nov 24 02:38:51 2014 From: davidhalter88 at gmail.com (Dave Halter) Date: Mon, 24 Nov 2014 02:38:51 +0100 Subject: [code-quality] RedBaron, a bottom-up refactoring lib/tool for python In-Reply-To: <20141117200011.GA11340@griboiedov> References: <20141114120553.GA2174@griboiedov> <20141117090151.GA3267@logilab.fr> <20141117200011.GA11340@griboiedov> Message-ID: 2014-11-17 21:00 GMT+01:00 Laurent Peuch : > Hello everyone and thanks for your answers :) > > > > > I'm currently working on the integration of the lib2to3 parser into > Jedi. > > > > This would make refactoring really easy (I'm about 50% done with the > > > > parser). It's also well tested and offers a few other advantages. > > > > > > > > In a perfect world, we could now combine our projects :-) I will > look in > > > > detail at Red Baron on Monday. > > > > > > David, we've been talking about this during the latest EuroPython, and > I've > > > talked with Laurent yesterday at the Capitole du Libre in Toulouse: > IMO we > > > could > > > start by extracting from lib2to3 "the" parser that could be used by > every > > > tools > > > like ours (refactoring, completion, static analysis...). It would be: > > > * loss-less (comments, indents...) > > > * accurate (eg from/to line numbers) > > > * fast > > > * version agnostic within a reasonable frame (eg 2.7 -> 3.4?) > > > > > > > Yes, that's what I'm trying to create now (based on lib2to3). However my > > biggest problem is that I have to rewrite my evaluation engine as well, > > because it was depending on the old parser. I have two additional > > constraints: > > > > - decent error recovery > > - memory efficient > > > > The "fast" part is something I'm very eager to implement. I have done > this > > before with my old parser. My approach is to re-parse only the parts of > the > > file that have changed. > > My idea of "version agnostic" is to have the parser and the evaluation > > engine try to adhere to one version. My goal is to give Jedi a Python > > version number and Jedi would work according to that. This would make > > linting really cool, because you can run the linter for different Python > > versions. > > > > I would recommend you to wait until I have finished my parser (1-2 > months) > > or can at least report back. You can then either take the parser and fork > > it or take the evaluation engine as well. > > Well, wouldn't it be a good idea if instead of us waiting 1-2 months > that you finish your parser then seeing if it also fits our needs we > discuss a bit of what each one of us needs and see if we could find an > agreement on what could be a common ST for all of us? > All right. I have cleaned it up a little bit: https://github.com/davidhalter/jedi/tree/db76bbccc58729426cb39a6373e986139ea62d56/jedi/parser This is the latest parser branch. The parser itself is still working pretty well. There's still a lot of old code lurking around, that I'm not deleting yet (so I don't forget what I still need to add). Few notes about the files: - __init__.py contains the handlers. More about that later. - pgen2 contains a pretty much unchanged lib2to3 parser. - tree.py contains the whole business logic. It's all about searching the tree. I'm pretty open to add more helpers. I'm just not needing more at the moment. - fast.py will contain a faster version of the parser (not working right now). I'm going to do this by caching parts of the file. - tokenize.py is Jedi's "old" tokenizer. I will probably replace pgen2's tokenizer with this one to improve error recovery. - grammar.txt etc. are the files with the Python grammar. - user_context.py is Jedi related, will be partially rewritten and helps with understanding messy code that the parser doesn't understand. It's my goal to support multiple grammars at the same time. It should be possible to parse 2.7 while still being able to parse 3.4 in the same process (thread-safe). The same goal applies to Jedi: I want the user be able to choose the Python version for the evaluation of code as well. Therefore the parser has a `grammar` argument: Parser(grammar, source, module_path=None, tokenizer=None). There's a few design decisions that I took: - Like lib2to3 I am creating nodes only if they have more than one child. This is very important for performance reasons. There's an exception though: ExprStmt is always created (I might remove this "feature" again). - As you can see in tree.py (and also in jedi.parser.Parser._ast_mapping: There's classes for nodes like functions, classes, params and others, while there are no classes for `xor_expr`, `and_expr` and so on. This has been a very good solution for Jedi. It makes business logic possible for the classes where we need it, but at the same time doesn't bloat tree.py. The children attribute is available anyway. I might also add a type attribute (class attribute) to all the classes. Glad to hear any feedback. Also really happy to reverse design decision or change some things fundamentally. Just don't complain about the "messiness" too much, that will get better :-) ~ Dave PS: `Simple` is an old Jedi class. I will rename it to `BaseNode` later. > I can totally understand that discussing this right now might not be > the most appealing idea but maybe think a bit about it of the benefit > that we could get from a common AST for all of us: > > * only one code base to maintain instead of everyone doing his > own parser on his side > * we could share our efforts on making it as good as possible > * more time spend on doing the actual tools than the backend > * a de facto reference for every python developer that wants to join > the field of tooling: more people -> more tools -> better python in > general > > I really think that we should at least try to see if it's possible, > having this kind of tool would really benefit us and the python > community in general. > > What do you think? > > -- > > Laurent Peuch -- Bram > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan.bucur at gmail.com Tue Nov 25 11:50:16 2014 From: stefan.bucur at gmail.com (Stefan Bucur) Date: Tue, 25 Nov 2014 10:50:16 +0000 Subject: [code-quality] Static checker for common Python programming errors References: Message-ID: On Tue Nov 18 2014 at 1:25:07 PM Claudiu Popa wrote: > On Mon, Nov 17, 2014 at 7:18 PM, Stefan Bucur > wrote: > > I'm developing a Python static analysis tool that flags common > programming > > errors in Python programs. The tool is meant to complement other tools > like > > Pylint (which perform checks at lexical and AST level) by going deeper > with > > the code analysis and keeping track of the possible control flow paths in > > the program (path-sensitive analysis). > > > > > Hey, Stefan. > > That's great! It's really cool to see more people interested in static > analysis with Python. > Control flow analysis is missing in Pylint and due to this, we have a > couple of false positives. > For instance, the following code will emit a raising-bad-type warning, > even though exc is later > reassigned (but not deduced by the inference engine): > > exc = None > for i in range(10): > try: > 1 / 0 > break > except Exception as e: > exc = e > else: > raise exc > This is very interesting (and a good reminder of the existence of the for/else construct, which I should probably use more often :) ). It would actually be nice to have a suite of such snippets, to be able to benchmark analysis tools more uniformly. Stefan -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan.bucur at gmail.com Tue Nov 25 12:29:33 2014 From: stefan.bucur at gmail.com (Stefan Bucur) Date: Tue, 25 Nov 2014 11:29:33 +0000 Subject: [code-quality] Static checker for common Python programming errors References: Message-ID: On Tue Nov 18 2014 at 2:01:46 PM Dave Halter wrote: > Hi Stefan > > I'm playing with this as well in Jedi. I'm pretty far with flow analysis > and AttributeErrors. (This includes everything you mention above except > integer division by zero). Would be easy to implement in Jedi, though. I > just have different priorities, at the moment. > > If you have some time on your hands you can watch my EuroPython talk about > this: https://www.youtube.com/watch?v=DfVHSw0iOsk I'm also glad to skype > (gaukler_) if you're interested. Jedi is not well known for doing static > analysis. But it's my goal to change this now. > Thanks Dave for the pointer. I watched your talk and had a look at Jedi's code ? this is quite nice and would indeed be great to better expose the static analysis potential of the framework. What I'm trying to achieve is something a bit different. My goal is to reuse as much as possible the "implicit specs" of the interpreter itself (CPython), as opposed to implementing them again in my analysis tool. We already have an execution engine that uses the interpreter to automatically explore multiple paths through a piece of Python code. You can read here the academic paper, with case studies for Python and Lua: http://dslab.epfl.ch/pubs/chef.pdf I'm now working on extending this technique to other types of analyses and I'm trying to determine the most relevant types of analyses and checks for Python. So far, I conclude that type inference would serve multiple purposes and is on the top of my list. Stefan > 2014-11-17 18:18 GMT+01:00 Stefan Bucur : > >> I'm developing a Python static analysis tool that flags common >> programming errors in Python programs. The tool is meant to complement >> other tools like Pylint (which perform checks at lexical and AST level) by >> going deeper with the code analysis and keeping track of the possible >> control flow paths in the program (path-sensitive analysis). >> >> For instance, a path-sensitive analysis detects that the following >> snippet of code would raise an AttributeError exception: >> >> if object is None: # If the True branch is taken, we know the object is >> None >> object.doSomething() # ... so this statement would always fail >> >> I wanted first to tap into people's experience and get a sense of what >> common pitfalls in the language & its standard library such a static >> checker should look for. Just as an example of what I mean, here [1] is a >> list of static checks for the C++ language, as part of the Clang static >> analyzer project. >> >> My preliminary list of Python checks is quite rudimentary, but maybe >> could serve as a discussion starter: >> >> * Proper Unicode handling (for 2.x) >> - encode() is not called on str object >> - decode() is not called on unicode object >> * Check for integer division by zero >> * Check for None object dereferences >> >> Thanks a lot, >> Stefan Bucur >> >> [1] http://clang-analyzer.llvm.org/available_checks.html >> >> >> _______________________________________________ >> code-quality mailing list >> code-quality at python.org >> https://mail.python.org/mailman/listinfo/code-quality >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidhalter88 at gmail.com Tue Nov 25 14:27:46 2014 From: davidhalter88 at gmail.com (Dave Halter) Date: Tue, 25 Nov 2014 14:27:46 +0100 Subject: [code-quality] Static checker for common Python programming errors In-Reply-To: References: Message-ID: 2014-11-25 12:29 GMT+01:00 Stefan Bucur : > > > On Tue Nov 18 2014 at 2:01:46 PM Dave Halter > wrote: > >> Hi Stefan >> >> I'm playing with this as well in Jedi. I'm pretty far with flow analysis >> and AttributeErrors. (This includes everything you mention above except >> integer division by zero). Would be easy to implement in Jedi, though. I >> just have different priorities, at the moment. >> >> If you have some time on your hands you can watch my EuroPython talk >> about this: https://www.youtube.com/watch?v=DfVHSw0iOsk I'm also glad to >> skype (gaukler_) if you're interested. Jedi is not well known for doing >> static analysis. But it's my goal to change this now. >> > > Thanks Dave for the pointer. I watched your talk and had a look at Jedi's > code ? this is quite nice and would indeed be great to better expose the > static analysis potential of the framework. > > What I'm trying to achieve is something a bit different. My goal is to > reuse as much as possible the "implicit specs" of the interpreter itself > (CPython), as opposed to implementing them again in my analysis tool. > > We already have an execution engine that uses the interpreter to > automatically explore multiple paths through a piece of Python code. You > can read here the academic paper, with case studies for Python and Lua: > http://dslab.epfl.ch/pubs/chef.pdf > Stefan, I haven't read the full paper (but most of it). Interesting stuff! Are you doing any "argument propagation" or are you simply looking at a function without argument knowledge? How do you does your output look like? Can you give us a real world example (maybe simplejson)? Would be really interesting. BTW: I remember asking the same question that you asked here in the beginning: "What are your most typical mistakes that you would like a static analysis tool pick up?" https://github.com/davidhalter/jedi/issues/408 The first maybe 10 answers might interest you. ~ Dave > I'm now working on extending this technique to other types of analyses and > I'm trying to determine the most relevant types of analyses and checks for > Python. So far, I conclude that type inference would serve multiple > purposes and is on the top of my list. > > Stefan > > >> 2014-11-17 18:18 GMT+01:00 Stefan Bucur : >> >>> I'm developing a Python static analysis tool that flags common >>> programming errors in Python programs. The tool is meant to complement >>> other tools like Pylint (which perform checks at lexical and AST level) by >>> going deeper with the code analysis and keeping track of the possible >>> control flow paths in the program (path-sensitive analysis). >>> >>> For instance, a path-sensitive analysis detects that the following >>> snippet of code would raise an AttributeError exception: >>> >>> if object is None: # If the True branch is taken, we know the object is >>> None >>> object.doSomething() # ... so this statement would always fail >>> >>> I wanted first to tap into people's experience and get a sense of what >>> common pitfalls in the language & its standard library such a static >>> checker should look for. Just as an example of what I mean, here [1] is a >>> list of static checks for the C++ language, as part of the Clang static >>> analyzer project. >>> >>> My preliminary list of Python checks is quite rudimentary, but maybe >>> could serve as a discussion starter: >>> >>> * Proper Unicode handling (for 2.x) >>> - encode() is not called on str object >>> - decode() is not called on unicode object >>> * Check for integer division by zero >>> * Check for None object dereferences >>> >>> Thanks a lot, >>> Stefan Bucur >>> >>> [1] http://clang-analyzer.llvm.org/available_checks.html >>> >>> >>> _______________________________________________ >>> code-quality mailing list >>> code-quality at python.org >>> https://mail.python.org/mailman/listinfo/code-quality >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan.bucur at gmail.com Tue Nov 25 15:20:24 2014 From: stefan.bucur at gmail.com (Stefan Bucur) Date: Tue, 25 Nov 2014 14:20:24 +0000 Subject: [code-quality] Static checker for common Python programming errors References: Message-ID: On Tue Nov 25 2014 at 2:27:46 PM Dave Halter wrote: > 2014-11-25 12:29 GMT+01:00 Stefan Bucur : > >> >> >> On Tue Nov 18 2014 at 2:01:46 PM Dave Halter >> wrote: >> >>> Hi Stefan >>> >>> I'm playing with this as well in Jedi. I'm pretty far with flow analysis >>> and AttributeErrors. (This includes everything you mention above except >>> integer division by zero). Would be easy to implement in Jedi, though. I >>> just have different priorities, at the moment. >>> >>> If you have some time on your hands you can watch my EuroPython talk >>> about this: https://www.youtube.com/watch?v=DfVHSw0iOsk I'm also glad >>> to skype (gaukler_) if you're interested. Jedi is not well known for doing >>> static analysis. But it's my goal to change this now. >>> >> >> Thanks Dave for the pointer. I watched your talk and had a look at Jedi's >> code ? this is quite nice and would indeed be great to better expose the >> static analysis potential of the framework. >> >> What I'm trying to achieve is something a bit different. My goal is to >> reuse as much as possible the "implicit specs" of the interpreter itself >> (CPython), as opposed to implementing them again in my analysis tool. >> >> We already have an execution engine that uses the interpreter to >> automatically explore multiple paths through a piece of Python code. You >> can read here the academic paper, with case studies for Python and Lua: >> http://dslab.epfl.ch/pubs/chef.pdf >> > > Stefan, I haven't read the full paper (but most of it). Interesting stuff! > Are you doing any "argument propagation" or are you simply looking at a > function without argument knowledge? > There is no explicit handling of the Python semantics in Chef (our tool)?this is the major advantage of the technique. The Python interpreter simply runs the code under test inside the symbolic x86 VM. An example may help clarify this (see below). > > How do you does your output look like? > The output is a set of test cases that comprise an automatically-generated test suite for the program under test. These test cases capture all the execution paths (including corner cases & buggy cases) discovered automatically during execution. > Can you give us a real world example (maybe simplejson)? Would be really > interesting. > Say we want to test the code in the simplejson package. Traditionally, one would write a suite of unit tests that exercise a predetermined set of input-output pairs. For instance, you'd have to think of what inputs would best cover the package behavior, i.e., valid JSON, invalid JSON, empty strings, slightly malformed JSON, and so on. This is quite tedious and one may very well miss obscure corner cases. However, with Chef, we write instead so called "symbolic tests", which use Chef's API. Here is a simple example for simplejson, say *simplejson_test.py:* import importlib import simplejson import sys *from chef import light* class SimpleJSONTest(*light.SymbolicTest*): def setUp(self): pass def runTest(self): simplejson.loads(*self.getString("input", '\x00'*15)*) if __name__ == "__main__": *light.runFromArgs(SimpleJSONTest, arg_list=sys.argv)* This piece of code does several things: * It encapsulates the test functionality in a test class that derives from light.SymbolicTest (as opposed to Python's own TestCase). * Instead of defining a particular JSON string to pass to simplejson, it asks the framework to construct one, according to some specs. In this case, it asks for a string that will be referred to as "input", of 15 characters, with a default value of null characters everywhere. * When the script is executed, the framework instantiates the symbolic test and runs it. You can run simplejson_test.py in two modes: 1) In symbolic mode, the test runs inside the symbolic virtual machine. The call to SimpleJSONTest.getString(...) returns a special "symbolic string", which taints the returned variable and causes the execution of the interpreter to "fork" (akin to a process fork) everytime a branch depending on the symbolic string is encountered during execution. This works everywhere inside the interpreter -- either for Python-level branches, or for native branches inside C extension modules -- because the symbolic VM runs at x86 level. Every time the execution forks, the framework generates a new "test case" -- a concrete assignment to the symbolic string that would cause the interpreter to execute precisely the newly discovered path (e.g., the '{ "key": 1}' string). 2) In replay mode, all the test cases generated in symbolic mode can be replayed via the same simplejson_test.py file. In this mode, the SimpleJSONTest.getString(...) call returns one of the concrete input assignments generated and checks that the replayed output matches the output observed in symbolic mode (this is to weed out nondeterminism). > > BTW: I remember asking the same question that you asked here in the > beginning: "What are your most typical mistakes that you would like a > static analysis tool pick up?" > https://github.com/davidhalter/jedi/issues/408 The first maybe 10 answers > might interest you. > Oh, this is excellent! Thanks a lot! Cheers, Stefan > >> I'm now working on extending this technique to other types of analyses >> and I'm trying to determine the most relevant types of analyses and checks >> for Python. So far, I conclude that type inference would serve multiple >> purposes and is on the top of my list. >> >> Stefan >> >> >>> 2014-11-17 18:18 GMT+01:00 Stefan Bucur : >>> >>>> I'm developing a Python static analysis tool that flags common >>>> programming errors in Python programs. The tool is meant to complement >>>> other tools like Pylint (which perform checks at lexical and AST level) by >>>> going deeper with the code analysis and keeping track of the possible >>>> control flow paths in the program (path-sensitive analysis). >>>> >>>> For instance, a path-sensitive analysis detects that the following >>>> snippet of code would raise an AttributeError exception: >>>> >>>> if object is None: # If the True branch is taken, we know the object is >>>> None >>>> object.doSomething() # ... so this statement would always fail >>>> >>>> I wanted first to tap into people's experience and get a sense of what >>>> common pitfalls in the language & its standard library such a static >>>> checker should look for. Just as an example of what I mean, here [1] is a >>>> list of static checks for the C++ language, as part of the Clang static >>>> analyzer project. >>>> >>>> My preliminary list of Python checks is quite rudimentary, but maybe >>>> could serve as a discussion starter: >>>> >>>> * Proper Unicode handling (for 2.x) >>>> - encode() is not called on str object >>>> - decode() is not called on unicode object >>>> * Check for integer division by zero >>>> * Check for None object dereferences >>>> >>>> Thanks a lot, >>>> Stefan Bucur >>>> >>>> [1] http://clang-analyzer.llvm.org/available_checks.html >>>> >>>> >>>> _______________________________________________ >>>> code-quality mailing list >>>> code-quality at python.org >>>> https://mail.python.org/mailman/listinfo/code-quality >>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip.montanaro at gmail.com Tue Nov 25 21:54:03 2014 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Tue, 25 Nov 2014 14:54:03 -0600 Subject: [code-quality] Problems running pylint 1.4.0 Message-ID: We have a rather peculiar software distribution system here at work. I downloaded, packaged, and installed astroid, logilab-common, and pylint. When I run pylint now (1.1.0 worked just fine), I get this error: Traceback (most recent call last): File "/opt/local/bin/pylint", line 6, in from pkg_resources import load_entry_point File "/home/skipm/.local/lib/python2.7/site-packages/pkg_resources.py", line 2805, in working_set.require(__requires__) File "/home/skipm/.local/lib/python2.7/site-packages/pkg_resources.py", line 696, in require needed = self.resolve(parse_requirements(requirements)) File "/home/skipm/.local/lib/python2.7/site-packages/pkg_resources.py", line 594, in resolve raise DistributionNotFound(req) pkg_resources.DistributionNotFound: pylint==1.4.0 I see the (I think) necessary pylint bits in our site-packages directory: % ls -ld /opt/local/lib/python2.7/site-packages/pylint* drwxr-xr-x 6 root root 4096 Nov 25 14:43 /opt/local/lib/python2.7/site-packages/pylint drwxr-xr-x 2 root root 4096 Nov 25 14:43 /opt/local/lib/python2.7/site-packages/pylint-1.4.0-py2.7.egg-info and they seem to have reasonable contents: % ls -lL /opt/local/lib/python2.7/site-packages/pylint total 324 -rw-rw-r-- 1 root root 1458 Nov 15 12:18 __init__.py -rw-rw-r-- 1 root root 1838 Nov 25 14:42 __init__.pyc -rw-rw-r-- 1 root root 56 Nov 15 12:18 __main__.py -rw-rw-r-- 1 root root 226 Nov 25 14:42 __main__.pyc -rw-rw-r-- 1 root root 2925 Nov 23 03:03 __pkginfo__.py -rw-rw-r-- 1 root root 2439 Nov 25 14:42 __pkginfo__.pyc drwxr-xr-x 2 root root 4096 Nov 25 14:43 checkers -rw-rw-r-- 1 root root 5782 Nov 21 05:23 config.py -rw-rw-r-- 1 root root 5407 Nov 25 14:42 config.pyc -rw-rw-r-- 1 root root 6445 Nov 15 12:18 epylint.py -rw-rw-r-- 1 root root 5848 Nov 25 14:42 epylint.pyc -rw-rw-r-- 1 root root 19401 Nov 21 05:23 gui.py -rw-rw-r-- 1 root root 18419 Nov 25 14:42 gui.pyc -rw-rw-r-- 1 root root 2828 Nov 15 12:18 interfaces.py -rw-rw-r-- 1 root root 4143 Nov 25 14:42 interfaces.pyc -rw-rw-r-- 1 root root 57390 Nov 23 02:46 lint.py -rw-rw-r-- 1 root root 49003 Nov 25 14:42 lint.pyc drwxr-xr-x 2 root root 4096 Nov 25 14:43 pyreverse drwxr-xr-x 2 root root 4096 Nov 25 14:43 reporters drwxr-xr-x 7 root root 4096 Nov 25 14:43 test -rw-rw-r-- 1 root root 13371 Nov 21 05:23 testutils.py -rw-rw-r-- 1 root root 36111 Nov 21 05:23 utils.py -rw-rw-r-- 1 root root 34047 Nov 25 14:42 utils.pyc blade% ls -lL /opt/local/lib/python2.7/site-packages/pylint-1.4.0-py2.7.egg-info/ total 44 -rw-rw-r-- 1 root root 3227 Nov 25 14:42 PKG-INFO -rw-rw-r-- 1 root root 22242 Nov 25 14:42 SOURCES.txt -rw-rw-r-- 1 root root 1 Nov 25 14:42 dependency_links.txt -rw-rw-r-- 1 root root 172 Nov 25 14:42 entry_points.txt -rw-rw-r-- 1 root root 45 Nov 25 14:42 requires.txt -rw-rw-r-- 1 root root 7 Nov 25 14:42 top_level.txt The whole distutils/setuptools/pkg_resources stuff has always eluded my understanding. Can someone give me some suggestions about what's missing? Thanks, Skip Montanaro -------------- next part -------------- An HTML attachment was scrubbed... URL: From joaquinsargiotto at gmail.com Wed Nov 26 03:18:30 2014 From: joaquinsargiotto at gmail.com (Joaquin Sargiotto) Date: Tue, 25 Nov 2014 23:18:30 -0300 Subject: [code-quality] Problems running pylint 1.4.0 In-Reply-To: References: Message-ID: I had a similar issue and the problem was that somehow I had two setuptools versions installed... and the older version was overwriting the new... Hope that helps -- Joaquin Sargiotto El nov 25, 2014 5:54 p.m., "Skip Montanaro" escribi?: > We have a rather peculiar software distribution system here at work. I > downloaded, packaged, and installed astroid, logilab-common, and pylint. > When I run pylint now (1.1.0 worked just fine), I get this error: > > Traceback (most recent call last): > File "/opt/local/bin/pylint", line 6, in > from pkg_resources import load_entry_point > File "/home/skipm/.local/lib/python2.7/site-packages/pkg_resources.py", > line 2805, in > working_set.require(__requires__) > File "/home/skipm/.local/lib/python2.7/site-packages/pkg_resources.py", > line 696, in require > needed = self.resolve(parse_requirements(requirements)) > File "/home/skipm/.local/lib/python2.7/site-packages/pkg_resources.py", > line 594, in resolve > raise DistributionNotFound(req) > pkg_resources.DistributionNotFound: pylint==1.4.0 > > I see the (I think) necessary pylint bits in our site-packages directory: > > % ls -ld /opt/local/lib/python2.7/site-packages/pylint* > drwxr-xr-x 6 root root 4096 Nov 25 14:43 > /opt/local/lib/python2.7/site-packages/pylint > drwxr-xr-x 2 root root 4096 Nov 25 14:43 > /opt/local/lib/python2.7/site-packages/pylint-1.4.0-py2.7.egg-info > > and they seem to have reasonable contents: > > % ls -lL /opt/local/lib/python2.7/site-packages/pylint > total 324 > -rw-rw-r-- 1 root root 1458 Nov 15 12:18 __init__.py > -rw-rw-r-- 1 root root 1838 Nov 25 14:42 __init__.pyc > -rw-rw-r-- 1 root root 56 Nov 15 12:18 __main__.py > -rw-rw-r-- 1 root root 226 Nov 25 14:42 __main__.pyc > -rw-rw-r-- 1 root root 2925 Nov 23 03:03 __pkginfo__.py > -rw-rw-r-- 1 root root 2439 Nov 25 14:42 __pkginfo__.pyc > drwxr-xr-x 2 root root 4096 Nov 25 14:43 checkers > -rw-rw-r-- 1 root root 5782 Nov 21 05:23 config.py > -rw-rw-r-- 1 root root 5407 Nov 25 14:42 config.pyc > -rw-rw-r-- 1 root root 6445 Nov 15 12:18 epylint.py > -rw-rw-r-- 1 root root 5848 Nov 25 14:42 epylint.pyc > -rw-rw-r-- 1 root root 19401 Nov 21 05:23 gui.py > -rw-rw-r-- 1 root root 18419 Nov 25 14:42 gui.pyc > -rw-rw-r-- 1 root root 2828 Nov 15 12:18 interfaces.py > -rw-rw-r-- 1 root root 4143 Nov 25 14:42 interfaces.pyc > -rw-rw-r-- 1 root root 57390 Nov 23 02:46 lint.py > -rw-rw-r-- 1 root root 49003 Nov 25 14:42 lint.pyc > drwxr-xr-x 2 root root 4096 Nov 25 14:43 pyreverse > drwxr-xr-x 2 root root 4096 Nov 25 14:43 reporters > drwxr-xr-x 7 root root 4096 Nov 25 14:43 test > -rw-rw-r-- 1 root root 13371 Nov 21 05:23 testutils.py > -rw-rw-r-- 1 root root 36111 Nov 21 05:23 utils.py > -rw-rw-r-- 1 root root 34047 Nov 25 14:42 utils.pyc > blade% ls -lL > /opt/local/lib/python2.7/site-packages/pylint-1.4.0-py2.7.egg-info/ > total 44 > -rw-rw-r-- 1 root root 3227 Nov 25 14:42 PKG-INFO > -rw-rw-r-- 1 root root 22242 Nov 25 14:42 SOURCES.txt > -rw-rw-r-- 1 root root 1 Nov 25 14:42 dependency_links.txt > -rw-rw-r-- 1 root root 172 Nov 25 14:42 entry_points.txt > -rw-rw-r-- 1 root root 45 Nov 25 14:42 requires.txt > -rw-rw-r-- 1 root root 7 Nov 25 14:42 top_level.txt > > The whole distutils/setuptools/pkg_resources stuff has always eluded my > understanding. Can someone give me some suggestions about what's missing? > > Thanks, > > Skip Montanaro > > > _______________________________________________ > code-quality mailing list > code-quality at python.org > https://mail.python.org/mailman/listinfo/code-quality > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marius at gedmin.as Wed Nov 26 11:02:08 2014 From: marius at gedmin.as (Marius Gedminas) Date: Wed, 26 Nov 2014 12:02:08 +0200 Subject: [code-quality] Problems running pylint 1.4.0 In-Reply-To: References: Message-ID: <20141126100208.GA18345@platonas> On Tue, Nov 25, 2014 at 02:54:03PM -0600, Skip Montanaro wrote: > We have a rather peculiar software distribution system here at work. I > downloaded, packaged, and installed astroid, logilab-common, and pylint. > When I run pylint now (1.1.0 worked just fine), I get this error: > > Traceback (most recent call last): > File "/opt/local/bin/pylint", line 6, in > from pkg_resources import load_entry_point What's in the #! line of /opt/local/bin/pylint? > File "/home/skipm/.local/lib/python2.7/site-packages/pkg_resources.py", > line 2805, in > working_set.require(__requires__) > File "/home/skipm/.local/lib/python2.7/site-packages/pkg_resources.py", > line 696, in require > needed = self.resolve(parse_requirements(requirements)) > File "/home/skipm/.local/lib/python2.7/site-packages/pkg_resources.py", > line 594, in resolve > raise DistributionNotFound(req) > pkg_resources.DistributionNotFound: pylint==1.4.0 > > I see the (I think) necessary pylint bits in our site-packages directory: > > % ls -ld /opt/local/lib/python2.7/site-packages/pylint* > drwxr-xr-x 6 root root 4096 Nov 25 14:43 > /opt/local/lib/python2.7/site-packages/pylint > drwxr-xr-x 2 root root 4096 Nov 25 14:43 > /opt/local/lib/python2.7/site-packages/pylint-1.4.0-py2.7.egg-info But does your Python look in /opt/local/lib/python2.7/site-packages? What does sys.path look like if you run the Python interpreter named in the #! line of the pylint script? > The whole distutils/setuptools/pkg_resources stuff has always eluded my > understanding. Can someone give me some suggestions about what's missing? Marius Gedminas -- > I'm not sure what makes a given poem 'modern'... Well, one characteristic sometimes found is... to put it politely... creative typography. Very creative typography. -- Henry Spencer on the chores of typesetting modern poetry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 173 bytes Desc: Digital signature URL: From skip.montanaro at gmail.com Wed Nov 26 14:39:40 2014 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Wed, 26 Nov 2014 07:39:40 -0600 Subject: [code-quality] Problems running pylint 1.4.0 In-Reply-To: <20141126100208.GA18345@platonas> References: <20141126100208.GA18345@platonas> Message-ID: On Wed, Nov 26, 2014 at 4:02 AM, Marius Gedminas wrote: > What's in the #! line of /opt/local/bin/pylint? Ack! I'll bet that's the problem. Our production Python (/opt/local/bin/python) was compiled without debug symbols. Desperate at one point to step through the interpreter's C code, I built my own, and installed it in my directory space. I forgot about that when packaging pylint. It was found before /opt/local/bin/python. Let me see if fixing that solves the problem. My guess is that it will. Skip -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip.montanaro at gmail.com Wed Nov 26 14:50:05 2014 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Wed, 26 Nov 2014 07:50:05 -0600 Subject: [code-quality] Problems running pylint 1.4.0 In-Reply-To: References: <20141126100208.GA18345@platonas> Message-ID: On Wed, Nov 26, 2014 at 7:39 AM, Skip Montanaro wrote: > Let me see if fixing that solves the problem. My guess is that it will. That was it! Thanks for the excellent hint, Marius. Skip -------------- next part -------------- An HTML attachment was scrubbed... URL: From code-quality.wting at xoxy.net Wed Nov 26 06:41:53 2014 From: code-quality.wting at xoxy.net (code-quality.wting at xoxy.net) Date: Tue, 25 Nov 2014 21:41:53 -0800 Subject: [code-quality] Warn on list comprehension variables used outside list comprehensions? Message-ID: Python binds list comprehension variables to the local scope which has caused some subtle bugs. Is it possible to add a warning for this in pyflakes? I haven't implemented it yet, but here are the example tests: def test_listCompVariableUsedOutsideListComp(self): """ Test that a variable defined in a list comprehension is not used outside of the list comprehension. """ self.flakes(''' [x for x in range(3)] print x ''', m.VariableUsedOutsideListComp) self.flakes(''' [x for x in range(3)] [x for _ in range(3)] ''', m.VariableUsedOutsideListComp) def test_listCompVariableAllowReuse(self): """ Test that list comprehension variables are allowed to be reused if redefined. """ self.flakes(''' [x for x in range(3)] [x for x in range(3)]''') - William From paul at mad-scientist.net Wed Nov 26 08:03:05 2014 From: paul at mad-scientist.net (Paul Smith) Date: Wed, 26 Nov 2014 02:03:05 -0500 Subject: [code-quality] Lots of E1101 in standard modules after upgrading pylint to 1.4.0 Message-ID: <1416985385.23650.11.camel@homebase> Hi all; I just upgraded to a newer version of pylint and now I'm seeing many spurious E1101 errors. Before I was using: * Python 2.7.6 * Pylint 1.1.0 * Astroid 1.0.1 * logilab-common 0.61.0 Now I've upgraded to: * Python 2.7.8 * Pylint 1.4.0 * Astroid 1.3.2 * logilab-common 0.63.0 Everything about this new install appears to work fine, EXCEPT pylint. I'm seeing tons of strange E1101 error on standard modules, like time.sleep() and others (readline, etc.) For example: $ cat sl.py import time time.sleep(1) $ python sl.py $ pylint sl.py ************* Module sl C: 1, 0: Missing module docstring (missing-docstring) E: 2, 0: Module 'time' has no 'sleep' member (no-member) If I run this with my old setup (older python/pylint/etc.), it doesn't complain at all. Why am I seeing this invalid error? Thanks! From graffatcolmingov at gmail.com Wed Nov 26 15:30:34 2014 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Wed, 26 Nov 2014 08:30:34 -0600 Subject: [code-quality] Warn on list comprehension variables used outside list comprehensions? In-Reply-To: References: Message-ID: On Tue, Nov 25, 2014 at 11:41 PM, wrote: > Python binds list comprehension variables to the local scope which has > caused some subtle bugs. Is it possible to add a warning for this in > pyflakes? I haven't implemented it yet, but here are the example > tests: > > def test_listCompVariableUsedOutsideListComp(self): > """ > Test that a variable defined in a list comprehension is not used > outside of the list comprehension. > """ > self.flakes(''' > [x for x in range(3)] > print x > ''', m.VariableUsedOutsideListComp) > self.flakes(''' > [x for x in range(3)] > [x for _ in range(3)] > ''', m.VariableUsedOutsideListComp) > > > def test_listCompVariableAllowReuse(self): > """ > Test that list comprehension variables are allowed to be reused if > redefined. > """ > self.flakes(''' > [x for x in range(3)] > [x for x in range(3)]''') > > > - William The trick here would be ensuring this only applies to Python 2. Take for example the following on Python 3: [x for x in range(3)] print(x) You will get a NameError because x is undefined outside the list comprehension. Further, given that PyFlakes operates on the AST generated for the code, this shouldn't be very difficult, but I'm not sure how you were thinking of adding this. With that said, the ultimate decision would fall to Florent and the other maintainer(s) of PyFlakes. Cheers, Ian From paul at mad-scientist.net Wed Nov 26 15:41:58 2014 From: paul at mad-scientist.net (Paul Smith) Date: Wed, 26 Nov 2014 09:41:58 -0500 Subject: [code-quality] Lots of E1101 in standard modules after upgrading pylint to 1.4.0 In-Reply-To: <1416985385.23650.11.camel@homebase> References: <1416985385.23650.11.camel@homebase> Message-ID: <1417012918.23650.20.camel@homebase> On Wed, 2014-11-26 at 02:03 -0500, Paul Smith wrote: Forgot to say, I'm running on GNU/Linux Ubuntu GNOME 14.10 and I've built all these tools (including python) myself from source. > Hi all; I just upgraded to a newer version of pylint and now I'm seeing > many spurious E1101 errors. > > Before I was using: > * Python 2.7.6 > * Pylint 1.1.0 > * Astroid 1.0.1 > * logilab-common 0.61.0 > > Now I've upgraded to: > * Python 2.7.8 > * Pylint 1.4.0 > * Astroid 1.3.2 > * logilab-common 0.63.0 > > Everything about this new install appears to work fine, EXCEPT pylint. > > I'm seeing tons of strange E1101 error on standard modules, like > time.sleep() and others (readline, etc.) For example: > > $ cat sl.py > import time > time.sleep(1) > > $ python sl.py > > > $ pylint sl.py > ************* Module sl > C: 1, 0: Missing module docstring (missing-docstring) > E: 2, 0: Module 'time' has no 'sleep' member (no-member) > > If I run this with my old setup (older python/pylint/etc.), it doesn't > complain at all. Why am I seeing this invalid error? By "at all" I mean it doesn't complain about time.sleep(); of course I still get the missing docstring message in the old version :-). Looking at this it seems like all the members which are loaded from a shared library (e.g., lib/python2.7/lib-dynload/time.so etc.) have this problem: I see it with datetime, readline, some socket stuff, etc. Did I break something with my installation of python or one of the packages? How does pylint normally discover module members when the implementation is in C rather than python? From skip.montanaro at gmail.com Wed Nov 26 15:44:35 2014 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Wed, 26 Nov 2014 08:44:35 -0600 Subject: [code-quality] Warn on list comprehension variables used outside list comprehensions? In-Reply-To: References: Message-ID: On Wed, Nov 26, 2014 at 8:30 AM, Ian Cordasco wrote: > > The trick here would be ensuring this only applies to Python 2. Take > for example the following on Python 3: > > [x for x in range(3)] > print(x) > > You will get a NameError because x is undefined outside the list > comprehension. Note that pylint just grew a --py3k flag, which is used when scanning Python 2.x code looking for potential problems porting to Python 3.x. If pyflakes grew a similar flag, William's proposed check could be enabled only when --py3k was given. Skip From graffatcolmingov at gmail.com Wed Nov 26 15:52:09 2014 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Wed, 26 Nov 2014 08:52:09 -0600 Subject: [code-quality] Warn on list comprehension variables used outside list comprehensions? In-Reply-To: References: Message-ID: On Wed, Nov 26, 2014 at 8:44 AM, Skip Montanaro wrote: > On Wed, Nov 26, 2014 at 8:30 AM, Ian Cordasco > wrote: >> >> The trick here would be ensuring this only applies to Python 2. Take >> for example the following on Python 3: >> >> [x for x in range(3)] >> print(x) >> >> You will get a NameError because x is undefined outside the list >> comprehension. > > Note that pylint just grew a --py3k flag, which is used when scanning > Python 2.x code looking for potential problems porting to Python 3.x. > If pyflakes grew a similar flag, William's proposed check could be > enabled only when --py3k was given. > > Skip That's not the point of this check. The point of this check is that on Python 2, the binding to x in the comprehension bleeds outside of the comprehension scope (into the scope of the print statement) whereas that doesn't happen on Python 3. William is proposing adding a warning/error about using bindings created in a comprehension outside of the scope because it can cause problems. The following on Python 2 will surprise some people: x = 10 [x for x in range(3)] print(x + 1) On Python 2 that will print 3. On Python 3 that will print 11. The idea is to warn people of the fact that the binding will bleed but to only do it on Python 2. This has no basis in Python 3. From skip.montanaro at gmail.com Wed Nov 26 15:57:24 2014 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Wed, 26 Nov 2014 08:57:24 -0600 Subject: [code-quality] Warn on list comprehension variables used outside list comprehensions? In-Reply-To: References: Message-ID: > That's not the point of this check. The point of this check is that on > Python 2, the binding to x in the comprehension bleeds outside of the > comprehension scope Got it. Still, this code: > x = 10 > [x for x in range(3)] > print(x + 1) will run differently in Python 2 than Python3, so even if that was a conscious choice by the author, a --py3k flag should cause a message for this code. This is even worse than the simpler > [x for x in range(3)] > print(x + 1) because at least that will raise a NameError when run in Python 3. The three-line construct will still run, though produce different output. Skip From keith.derrick at lge.com Wed Nov 26 16:27:28 2014 From: keith.derrick at lge.com (Keith Derrick) Date: Wed, 26 Nov 2014 10:27:28 -0500 Subject: [code-quality] Warn on list comprehension variables used outside list comprehensions? In-Reply-To: References: Message-ID: <5475F160.3070903@lge.com> Definitely seems like something that should be warned about. For python2 it ought to be an error (which can always be turned off) for either code snippet. For python3, I'd say the two line snippet should flag an error - yes, it will fail at run-time, but pydev in eclipse runs pylint and saves me a lot of grief by spotting this type of thing before I ever save the file (though it doesn't catch either of these, presumably because they are valid python). The three line snippet should probably give a warning ("Are you sure you mean this?") The question is, how do we know which version of Python the code under inspection is targeted for, and hence how to treat this case. Some thoughts on that: * Anyone serious enough to be using pyflakes, pylint, or other static analysis tools is also probably using virtualenv. * pyflakes is therefore most likely running under the same python version for which the code is targeted. * code which is meant to run under multiple python versions is probably tested under something like tox - so again, pyflakes is run under the targeted python version (certainly my tox.ini runs it in every target environment) There's a lot of "probably"'s in there, but I don't think they are unreasonable assumptions. So, could pyflakes reasonably assume that the interpreter under which it is running is also the intended target version for the code under inspection? At least for the major version. This could then be overridden by a suitable command line option to specify the target version(s) Also, in cases like this, possibly a "This will behave differently in Python 2 and 3" warning is a worth while alternative. Keith Derrick | Principal Engineer, Connected Platform | Engineering LG Silicon Valley Lab | 5150 Gt America Parkway, Santa Clara, CA 95054 Office: 408.610-5746 | Mobile: 831.383.9567 | LG.com On 11/26/2014 06:57 AM, Skip Montanaro wrote: >> That's not the point of this check. The point of this check is that on >> Python 2, the binding to x in the comprehension bleeds outside of the >> comprehension scope > Got it. > > Still, this code: > >> x = 10 >> [x for x in range(3)] >> print(x + 1) > will run differently in Python 2 than Python3, so even if that was a > conscious choice by the author, a --py3k flag should cause a message > for this code. This is even worse than the simpler > >> [x for x in range(3)] >> print(x + 1) > because at least that will raise a NameError when run in Python 3. The > three-line construct will still run, though produce different output. > > Skip > _______________________________________________ > code-quality mailing list > code-quality at python.org > https://mail.python.org/mailman/listinfo/code-quality From graffatcolmingov at gmail.com Wed Nov 26 16:50:59 2014 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Wed, 26 Nov 2014 09:50:59 -0600 Subject: [code-quality] Warn on list comprehension variables used outside list comprehensions? In-Reply-To: <5475F160.3070903@lge.com> References: <5475F160.3070903@lge.com> Message-ID: On Wed, Nov 26, 2014 at 9:27 AM, Keith Derrick wrote: > Definitely seems like something that should be warned about. > > For python2 it ought to be an error (which can always be turned off) for > either code snippet. PyFlakes doesn't allow you to ignore errors. Flake8 does so this is a moot point. > For python3, I'd say the two line snippet should flag an error - yes, it > will fail at run-time, but pydev in eclipse runs pylint and saves me a > lot of grief by spotting this type of thing before I ever save the file > (though it doesn't catch either of these, presumably because they are > valid python). If PyLint doesn't catch this I wonder if it's either never been raised or been decided to be a bad idea. Could someone search their bugs? > The three line snippet should probably give a warning ("Are you sure you > mean this?") > > The question is, how do we know which version of Python the code under > inspection is targeted for, and hence how to treat this case. > > Some thoughts on that: > > * Anyone serious enough to be using pyflakes, pylint, or other static > analysis tools is also probably using virtualenv. > * pyflakes is therefore most likely running under the same python > version for which the code is targeted. > * code which is meant to run under multiple python versions is probably > tested under something like tox - so again, pyflakes is run under the > targeted python version (certainly my tox.ini runs it in every target > environment) > > There's a lot of "probably"'s in there, but I don't think they are > unreasonable assumptions. > > So, could pyflakes reasonably assume that the interpreter under which > it is running is also the intended target version for the code under > inspection? At least for the major version. It already works this way. If you use pyflakes on Python 2 but are writing code only valid in Python 3 you will have a bad time. You have to install pyflakes on the version of Python you want it to lint for since pyflakes works only with the in-built ast module of Python. > This could then be overridden by a suitable command line option to > specify the target version(s) No. This isn't how PyFlakes is intended to work and this is extraneous. > Also, in cases like this, possibly a "This will behave differently in > Python 2 and 3" warning is a worth while alternative. Again, PyFlakes is very different from PyLint. We don't issue warnings like this. > Keith Derrick | Principal Engineer, Connected Platform | Engineering > LG Silicon Valley Lab | 5150 Gt America Parkway, Santa Clara, CA 95054 > Office: 408.610-5746 | Mobile: 831.383.9567 | LG.com > > On 11/26/2014 06:57 AM, Skip Montanaro wrote: >>> That's not the point of this check. The point of this check is that on >>> Python 2, the binding to x in the comprehension bleeds outside of the >>> comprehension scope >> Got it. >> >> Still, this code: >> >>> x = 10 >>> [x for x in range(3)] >>> print(x + 1) >> will run differently in Python 2 than Python3, so even if that was a >> conscious choice by the author, a --py3k flag should cause a message >> for this code. This is even worse than the simpler >> >>> [x for x in range(3)] >>> print(x + 1) >> because at least that will raise a NameError when run in Python 3. The >> three-line construct will still run, though produce different output. >> >> Skip >> _______________________________________________ >> code-quality mailing list >> code-quality at python.org >> https://mail.python.org/mailman/listinfo/code-quality > _______________________________________________ > code-quality mailing list > code-quality at python.org > https://mail.python.org/mailman/listinfo/code-quality From keith.derrick at lge.com Wed Nov 26 17:51:00 2014 From: keith.derrick at lge.com (Keith Derrick) Date: Wed, 26 Nov 2014 11:51:00 -0500 Subject: [code-quality] Warn on list comprehension variables used outside list comprehensions? In-Reply-To: References: <5475F160.3070903@lge.com> Message-ID: <547604F4.80806@lge.com> I couldn't find anything in their bug list about this, so thought I'd check. Just ran pylint at the command line with pylint 1.4 on this file (docstrings and import just to suppress miscellaneous complaints) > 1 ''' Test file for pylint ''' > 2 from __future__ import print_function > 3 > 4 def method1(): > 5 ''' Let x bleed from comprehension ''' > 6 [x for x in range(3)] > 7 print(x) > 8 > 9 def method2(): > 10 ''' reuse/hide x from local scope in comprehension ''' > 11 y = 10 > 12 [y for y in range(3)] > 13 print(y) > 14 For python 2.7.6 > ************* Module bug > W: 6, 4: Expression "[x for x in range(3)]" is assigned to nothing > (expression-not-assigned) > W: 7,10: Using possibly undefined loop variable 'x' > (undefined-loop-variable) > C: 11, 4: Invalid variable name "y" (invalid-name) > W: 12, 4: Expression "[y for y in range(3)]" is assigned to nothing > (expression-not-assigned) For python 3.4.0 > ************* Module bug > W: 6, 4: Expression "[x for x in range(3)]" is assigned to nothing > (expression-not-assigned) > E: 7,10: Undefined variable 'x' (undefined-variable) > C: 11, 4: Invalid variable name "y" (invalid-name) > W: 12, 4: Expression "[y for y in range(3)]" is assigned to nothing > (expression-not-assigned) So, it warns about the first case in py2 and gives an error in py3 which is expected. But it seems to be confused by the second case, flagging "y" as an invalid variable-name at the assignment point for both versions. Keith Derrick | Principal Engineer, Connected Platform | Engineering LG Silicon Valley Lab | 5150 Gt America Parkway, Santa Clara, CA 95054 Office: 408.610-5746 | Mobile: 831.383.9567 | LG.com On 11/26/2014 07:50 AM, Ian Cordasco wrote: > On Wed, Nov 26, 2014 at 9:27 AM, Keith Derrick wrote: >> Definitely seems like something that should be warned about. >> >> For python2 it ought to be an error (which can always be turned off) for >> either code snippet. > PyFlakes doesn't allow you to ignore errors. Flake8 does so this is a > moot point. > >> For python3, I'd say the two line snippet should flag an error - yes, it >> will fail at run-time, but pydev in eclipse runs pylint and saves me a >> lot of grief by spotting this type of thing before I ever save the file >> (though it doesn't catch either of these, presumably because they are >> valid python). > If PyLint doesn't catch this I wonder if it's either never been raised > or been decided to be a bad idea. Could someone search their bugs? > >> The three line snippet should probably give a warning ("Are you sure you >> mean this?") >> >> The question is, how do we know which version of Python the code under >> inspection is targeted for, and hence how to treat this case. >> >> Some thoughts on that: >> >> * Anyone serious enough to be using pyflakes, pylint, or other static >> analysis tools is also probably using virtualenv. >> * pyflakes is therefore most likely running under the same python >> version for which the code is targeted. >> * code which is meant to run under multiple python versions is probably >> tested under something like tox - so again, pyflakes is run under the >> targeted python version (certainly my tox.ini runs it in every target >> environment) >> >> There's a lot of "probably"'s in there, but I don't think they are >> unreasonable assumptions. >> >> So, could pyflakes reasonably assume that the interpreter under which >> it is running is also the intended target version for the code under >> inspection? At least for the major version. > It already works this way. If you use pyflakes on Python 2 but are > writing code only valid in Python 3 you will have a bad time. You have > to install pyflakes on the version of Python you want it to lint for > since pyflakes works only with the in-built ast module of Python. > >> This could then be overridden by a suitable command line option to >> specify the target version(s) > No. This isn't how PyFlakes is intended to work and this is extraneous. > >> Also, in cases like this, possibly a "This will behave differently in >> Python 2 and 3" warning is a worth while alternative. > Again, PyFlakes is very different from PyLint. We don't issue warnings > like this. > >> Keith Derrick | Principal Engineer, Connected Platform | Engineering >> LG Silicon Valley Lab | 5150 Gt America Parkway, Santa Clara, CA 95054 >> Office: 408.610-5746 | Mobile: 831.383.9567 | LG.com >> >> On 11/26/2014 06:57 AM, Skip Montanaro wrote: >>>> That's not the point of this check. The point of this check is that on >>>> Python 2, the binding to x in the comprehension bleeds outside of the >>>> comprehension scope >>> Got it. >>> >>> Still, this code: >>> >>>> x = 10 >>>> [x for x in range(3)] >>>> print(x + 1) >>> will run differently in Python 2 than Python3, so even if that was a >>> conscious choice by the author, a --py3k flag should cause a message >>> for this code. This is even worse than the simpler >>> >>>> [x for x in range(3)] >>>> print(x + 1) >>> because at least that will raise a NameError when run in Python 3. The >>> three-line construct will still run, though produce different output. >>> >>> Skip >>> _______________________________________________ >>> code-quality mailing list >>> code-quality at python.org >>> https://mail.python.org/mailman/listinfo/code-quality >> _______________________________________________ >> code-quality mailing list >> code-quality at python.org >> https://mail.python.org/mailman/listinfo/code-quality From graffatcolmingov at gmail.com Wed Nov 26 17:57:54 2014 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Wed, 26 Nov 2014 10:57:54 -0600 Subject: [code-quality] PyLint catching comprehension binding leaks (was: Warn on list comprehension variables used outside list comprehensions?) Message-ID: On Wed, Nov 26, 2014 at 10:51 AM, Keith Derrick wrote: > I couldn't find anything in their bug list about this, so thought I'd check. > > Just ran pylint at the command line with pylint 1.4 on this file > (docstrings and import just to suppress miscellaneous complaints) > >> 1 ''' Test file for pylint ''' >> 2 from __future__ import print_function >> 3 >> 4 def method1(): >> 5 ''' Let x bleed from comprehension ''' >> 6 [x for x in range(3)] >> 7 print(x) >> 8 >> 9 def method2(): >> 10 ''' reuse/hide x from local scope in comprehension ''' >> 11 y = 10 >> 12 [y for y in range(3)] >> 13 print(y) >> 14 > > For python 2.7.6 >> ************* Module bug >> W: 6, 4: Expression "[x for x in range(3)]" is assigned to nothing >> (expression-not-assigned) >> W: 7,10: Using possibly undefined loop variable 'x' >> (undefined-loop-variable) >> C: 11, 4: Invalid variable name "y" (invalid-name) >> W: 12, 4: Expression "[y for y in range(3)]" is assigned to nothing >> (expression-not-assigned) > For python 3.4.0 >> ************* Module bug >> W: 6, 4: Expression "[x for x in range(3)]" is assigned to nothing >> (expression-not-assigned) >> E: 7,10: Undefined variable 'x' (undefined-variable) >> C: 11, 4: Invalid variable name "y" (invalid-name) >> W: 12, 4: Expression "[y for y in range(3)]" is assigned to nothing >> (expression-not-assigned) > > So, it warns about the first case in py2 and gives an error in py3 which > is expected. > > But it seems to be confused by the second case, flagging "y" as an > invalid variable-name at the assignment point for both versions. > > Keith Derrick | Principal Engineer, Connected Platform | Engineering > LG Silicon Valley Lab | 5150 Gt America Parkway, Santa Clara, CA 95054 > Office: 408.610-5746 | Mobile: 831.383.9567 | LG.com > At this point, I'd recommend repling to the retitled message to catch the PyLint maintainer's focus. > On 11/26/2014 07:50 AM, Ian Cordasco wrote: >> On Wed, Nov 26, 2014 at 9:27 AM, Keith Derrick wrote: >>> Definitely seems like something that should be warned about. >>> >>> For python2 it ought to be an error (which can always be turned off) for >>> either code snippet. >> PyFlakes doesn't allow you to ignore errors. Flake8 does so this is a >> moot point. >> >>> For python3, I'd say the two line snippet should flag an error - yes, it >>> will fail at run-time, but pydev in eclipse runs pylint and saves me a >>> lot of grief by spotting this type of thing before I ever save the file >>> (though it doesn't catch either of these, presumably because they are >>> valid python). >> If PyLint doesn't catch this I wonder if it's either never been raised >> or been decided to be a bad idea. Could someone search their bugs? >> >>> The three line snippet should probably give a warning ("Are you sure you >>> mean this?") >>> >>> The question is, how do we know which version of Python the code under >>> inspection is targeted for, and hence how to treat this case. >>> >>> Some thoughts on that: >>> >>> * Anyone serious enough to be using pyflakes, pylint, or other static >>> analysis tools is also probably using virtualenv. >>> * pyflakes is therefore most likely running under the same python >>> version for which the code is targeted. >>> * code which is meant to run under multiple python versions is probably >>> tested under something like tox - so again, pyflakes is run under the >>> targeted python version (certainly my tox.ini runs it in every target >>> environment) >>> >>> There's a lot of "probably"'s in there, but I don't think they are >>> unreasonable assumptions. >>> >>> So, could pyflakes reasonably assume that the interpreter under which >>> it is running is also the intended target version for the code under >>> inspection? At least for the major version. >> It already works this way. If you use pyflakes on Python 2 but are >> writing code only valid in Python 3 you will have a bad time. You have >> to install pyflakes on the version of Python you want it to lint for >> since pyflakes works only with the in-built ast module of Python. >> >>> This could then be overridden by a suitable command line option to >>> specify the target version(s) >> No. This isn't how PyFlakes is intended to work and this is extraneous. >> >>> Also, in cases like this, possibly a "This will behave differently in >>> Python 2 and 3" warning is a worth while alternative. >> Again, PyFlakes is very different from PyLint. We don't issue warnings >> like this. >> >>> Keith Derrick | Principal Engineer, Connected Platform | Engineering >>> LG Silicon Valley Lab | 5150 Gt America Parkway, Santa Clara, CA 95054 >>> Office: 408.610-5746 | Mobile: 831.383.9567 | LG.com >>> >>> On 11/26/2014 06:57 AM, Skip Montanaro wrote: >>>>> That's not the point of this check. The point of this check is that on >>>>> Python 2, the binding to x in the comprehension bleeds outside of the >>>>> comprehension scope >>>> Got it. >>>> >>>> Still, this code: >>>> >>>>> x = 10 >>>>> [x for x in range(3)] >>>>> print(x + 1) >>>> will run differently in Python 2 than Python3, so even if that was a >>>> conscious choice by the author, a --py3k flag should cause a message >>>> for this code. This is even worse than the simpler >>>> >>>>> [x for x in range(3)] >>>>> print(x + 1) >>>> because at least that will raise a NameError when run in Python 3. The >>>> three-line construct will still run, though produce different output. >>>> >>>> Skip >>>> _______________________________________________ >>>> code-quality mailing list >>>> code-quality at python.org >>>> https://mail.python.org/mailman/listinfo/code-quality >>> _______________________________________________ >>> code-quality mailing list >>> code-quality at python.org >>> https://mail.python.org/mailman/listinfo/code-quality From skip.montanaro at gmail.com Wed Nov 26 17:59:01 2014 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Wed, 26 Nov 2014 10:59:01 -0600 Subject: [code-quality] Warn on list comprehension variables used outside list comprehensions? In-Reply-To: <547604F4.80806@lge.com> References: <5475F160.3070903@lge.com> <547604F4.80806@lge.com> Message-ID: On Wed, Nov 26, 2014 at 10:51 AM, Keith Derrick wrote: > But it seems to be confused by the second case, flagging "y" as an > invalid variable-name at the assignment point for both versions. > I think that's just because pylint, by default, doesn't like variable names with fewer than three characters. I think single-character variable names have their uses, and suppress that warning in my pylintrc file. Skip -------------- next part -------------- An HTML attachment was scrubbed... URL: From indigo at bitglue.com Wed Nov 26 18:27:25 2014 From: indigo at bitglue.com (Phil Frost) Date: Wed, 26 Nov 2014 12:27:25 -0500 Subject: [code-quality] Warn on list comprehension variables used outside list comprehensions? In-Reply-To: References: Message-ID: On Wed, Nov 26, 2014 at 12:41 AM, wrote: > Python binds list comprehension variables to the local scope which has > caused some subtle bugs. Is it possible to add a warning for this in > pyflakes? > A guiding principle of pyflakes's design is that it doesn't have warnings. If it complains about anything, it is because it's almost certainly useless or broken code. Using a list comprehension variable outside of the comprehension is perfectly valid according to python2: >>> [x for x in range(3)] [0, 1, 2] >>> x 2 but an error in python3, which has different scoping rules: >>> [x for x in range(3)] [0, 1, 2] >>> x Traceback (most recent call last): File "", line 1, in NameError: name 'x' is not defined So in the case of python3, it's a bug that it doesn't emit a warning for this code. For python2, pyflakes is working as designed. There's an issue, though. Pyflakes doesn't know if it's checking python2 or python3 code. It uses the AST of whatever python is running pyflakes, and if you look around checker.py[1] you can see there are some conditionals on PY2, PY32 and PY33. I think though all of these are to accommodate differences in the AST, but not to alter the checking semantics. Given that parsing python2 with a python3 AST is already going to be problematic for a lot of other reasons (like `print` vs `print()`), maybe the thing to do is say that pyflakes always checks input against the semantics of the version of python running pyflakes. Then implement some tests, and change how list comprehensions are handled[2] to be conditional on the version. I think everyone using pyflakes for python2 and python3 concurrently either has it installed twice or invokes it with the interpreter corresponding to what they want to check, ie `python2 -m pyflakes` or `python3 -m pyflakes`, so I can't think of any reason this wouldn't work, and then pyflakes can continue not having any configuration file or options. Pull requests welcome. [1]: https://github.com/pyflakes/pyflakes/blob/master/pyflakes/checker.py [2]: https://github.com/pyflakes/pyflakes/blob/master/pyflakes/checker.py#L657 -------------- next part -------------- An HTML attachment was scrubbed... URL: From shlomme at gmail.com Wed Nov 26 22:00:57 2014 From: shlomme at gmail.com (Torsten Marek) Date: Wed, 26 Nov 2014 22:00:57 +0100 Subject: [code-quality] Lots of E1101 in standard modules after upgrading pylint to 1.4.0 In-Reply-To: <1417012918.23650.20.camel@homebase> References: <1416985385.23650.11.camel@homebase> <1417012918.23650.20.camel@homebase> Message-ID: Hi, this might be a bug in astroid, I'll have a look at it later this week. // Torsten 2014-11-26 15:41 GMT+01:00 Paul Smith : > On Wed, 2014-11-26 at 02:03 -0500, Paul Smith wrote: > > Forgot to say, I'm running on GNU/Linux Ubuntu GNOME 14.10 and I've > built all these tools (including python) myself from source. > > > Hi all; I just upgraded to a newer version of pylint and now I'm seeing > > many spurious E1101 errors. > > > > Before I was using: > > * Python 2.7.6 > > * Pylint 1.1.0 > > * Astroid 1.0.1 > > * logilab-common 0.61.0 > > > > Now I've upgraded to: > > * Python 2.7.8 > > * Pylint 1.4.0 > > * Astroid 1.3.2 > > * logilab-common 0.63.0 > > > > Everything about this new install appears to work fine, EXCEPT pylint. > > > > I'm seeing tons of strange E1101 error on standard modules, like > > time.sleep() and others (readline, etc.) For example: > > > > $ cat sl.py > > import time > > time.sleep(1) > > > > $ python sl.py > > > > > > $ pylint sl.py > > ************* Module sl > > C: 1, 0: Missing module docstring (missing-docstring) > > E: 2, 0: Module 'time' has no 'sleep' member (no-member) > > > > If I run this with my old setup (older python/pylint/etc.), it doesn't > > complain at all. Why am I seeing this invalid error? > > By "at all" I mean it doesn't complain about time.sleep(); of course I > still get the missing docstring message in the old version :-). > > Looking at this it seems like all the members which are loaded from a > shared library (e.g., lib/python2.7/lib-dynload/time.so etc.) have this > problem: I see it with datetime, readline, some socket stuff, etc. > > Did I break something with my installation of python or one of the > packages? How does pylint normally discover module members when the > implementation is in C rather than python? > > _______________________________________________ > code-quality mailing list > code-quality at python.org > https://mail.python.org/mailman/listinfo/code-quality > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shlomme at gmail.com Wed Nov 26 22:06:11 2014 From: shlomme at gmail.com (Torsten Marek) Date: Wed, 26 Nov 2014 22:06:11 +0100 Subject: [code-quality] Lots of E1101 in standard modules after upgrading pylint to 1.4.0 In-Reply-To: References: <1416985385.23650.11.camel@homebase> <1417012918.23650.20.camel@homebase> Message-ID: Hm, this might be a problem with standard library detection on with self-built Python installations. Can you give me the output of: >>> import sys >>> sys.path >>> from astroid import modutils >>> modutils.STD_LIB_DIRS >>> modutils.is_standard_module('time') Thanks! // Torsten 2014-11-26 22:00 GMT+01:00 Torsten Marek : > Hi, > > this might be a bug in astroid, I'll have a look at it later this week. > > // Torsten > > 2014-11-26 15:41 GMT+01:00 Paul Smith : > >> On Wed, 2014-11-26 at 02:03 -0500, Paul Smith wrote: >> >> Forgot to say, I'm running on GNU/Linux Ubuntu GNOME 14.10 and I've >> built all these tools (including python) myself from source. >> >> > Hi all; I just upgraded to a newer version of pylint and now I'm seeing >> > many spurious E1101 errors. >> > >> > Before I was using: >> > * Python 2.7.6 >> > * Pylint 1.1.0 >> > * Astroid 1.0.1 >> > * logilab-common 0.61.0 >> > >> > Now I've upgraded to: >> > * Python 2.7.8 >> > * Pylint 1.4.0 >> > * Astroid 1.3.2 >> > * logilab-common 0.63.0 >> > >> > Everything about this new install appears to work fine, EXCEPT pylint. >> > >> > I'm seeing tons of strange E1101 error on standard modules, like >> > time.sleep() and others (readline, etc.) For example: >> > >> > $ cat sl.py >> > import time >> > time.sleep(1) >> > >> > $ python sl.py >> > >> > >> > $ pylint sl.py >> > ************* Module sl >> > C: 1, 0: Missing module docstring (missing-docstring) >> > E: 2, 0: Module 'time' has no 'sleep' member (no-member) >> > >> > If I run this with my old setup (older python/pylint/etc.), it doesn't >> > complain at all. Why am I seeing this invalid error? >> >> By "at all" I mean it doesn't complain about time.sleep(); of course I >> still get the missing docstring message in the old version :-). >> >> Looking at this it seems like all the members which are loaded from a >> shared library (e.g., lib/python2.7/lib-dynload/time.so etc.) have this >> problem: I see it with datetime, readline, some socket stuff, etc. >> >> Did I break something with my installation of python or one of the >> packages? How does pylint normally discover module members when the >> implementation is in C rather than python? >> >> _______________________________________________ >> code-quality mailing list >> code-quality at python.org >> https://mail.python.org/mailman/listinfo/code-quality >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul at mad-scientist.net Wed Nov 26 23:04:57 2014 From: paul at mad-scientist.net (Paul Smith) Date: Wed, 26 Nov 2014 17:04:57 -0500 Subject: [code-quality] Lots of E1101 in standard modules after upgrading pylint to 1.4.0 In-Reply-To: References: <1416985385.23650.11.camel@homebase> <1417012918.23650.20.camel@homebase> Message-ID: <1417039497.23650.62.camel@homebase> On Wed, 2014-11-26 at 22:06 +0100, Torsten Marek wrote: > Hm, this might be a problem with standard library detection on with > self-built Python installations. > Can you give me the output of: > >>> import sys > >>> sys.path > >>> from astroid import modutils > >>> modutils.STD_LIB_DIRS > >>> modutils.is_standard_module('time') See below. I had a few minutes to play with this and I discovered that the version of Python and the version of logilab-common don't matter. If I keep Python 2.7.8 and logilab-common 0.63.0 but down-rev pylint to 1.1.0 and astroid to 1.0.1, then things start to work again. However if I upgrade EITHER (or both) of pylint (to 1.4.0) or astroid (to 1.3.2) then I see the bad behavior. That seems odd to me, but that's what I see. Here's the output you asked for from the failing install (this uses pylint 1.1.0 and astroid 1.3.2): Python 2.7.8 (default, Nov 25 2014, 22:45:58) [GCC 4.9.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.path ['', '/opt/python/lib/python2.7/site-packages/setuptools-7.0-py2.7.egg', '/opt/python/lib/python2.7/site-packages/docopt-0.6.2-py2.7.egg', '/opt/python/lib/python2.7/site-packages/paramiko-1.15.1-py2.7.egg', '/opt/python/lib/python2.7/site-packages/ecdsa-0.11-py2.7.egg', '/opt/python/lib/python2.7/site-packages/scp-0.8.0-py2.7.egg', '/opt/python/lib/python2.7/site-packages/boto-2.34.0-py2.7.egg', '/opt/python/lib/python2.7/site-packages/logilab_common-0.63.0-py2.7.egg', '/opt/python/lib/python2.7/site-packages/six-1.8.0-py2.7.egg', '/opt/python/lib/python2.7/site-packages/astroid-1.3.2-py2.7.egg', '/opt/python/lib/python2.7/site-packages/pylint-1.1.0-py2.7.egg', '/opt/python/lib/python2.7/site-packages/requests-2.4.3-py2.7.egg', '/opt/python/lib/python2.7/site-packages/selenium-2.44.0-py2.7.egg', '/opt/python/lib/python2.7/site-packages/oauthlib-0.7.2-py2.7.egg', '/opt/python/lib/python2.7/site-packages/requests_oauthlib-0.4.2-py2.7.egg', '/opt/python/lib/python2.7/site-packages/jira_python-0.16-py2.7.egg', '/opt/python/lib/python27.zip', '/opt/python/lib/python2.7', '/opt/python/lib/python2.7/plat-linux2', '/opt/python/lib/python2.7/lib-tk', '/opt/python/lib/python2.7/lib-old', '/opt/python/x86_64-linux/lib/python2.7/lib-dynload', '/opt/python/lib/python2.7/site-packages', '/opt/python/x86_64-linux/lib/python2.7/site-packages'] >>> from astroid import modutils >>> modutils.STD_LIB_DIRS ['/opt/python/lib/python2.7', '/opt/python/lib/python2.7'] >>> modutils.is_standard_module('time') False Hm. Why are both paths in STD_LIB_DIRS the same? Should one of them be the execprefix path? Aha! If I reset STD_LIB_DIRS like this: >>> modutils.STD_LIB_DIRS = ['/opt/python/lib/python2.7', '/opt/python/x86_64-linux/lib/python2.7'] Then I get: >>> modutils.is_standard_module('time') True So the problem appears to be that when I install Python I'm separating prefix and exec_prefix, and astroid is not grokking that: >>> sys.prefix '/opt/python' >>> sys.exec_prefix '/opt/python/x86_64-linux' > 2014-11-26 15:41 GMT+01:00 Paul Smith > : > On Wed, 2014-11-26 at 02:03 -0500, Paul Smith wrote: > > Forgot to say, I'm running on GNU/Linux Ubuntu GNOME > 14.10 and I've > built all these tools (including python) myself from > source. > > > Hi all; I just upgraded to a newer version of pylint > and now I'm seeing > > many spurious E1101 errors. > > > > Before I was using: > > * Python 2.7.6 > > * Pylint 1.1.0 > > * Astroid 1.0.1 > > * logilab-common 0.61.0 > > > > Now I've upgraded to: > > * Python 2.7.8 > > * Pylint 1.4.0 > > * Astroid 1.3.2 > > * logilab-common 0.63.0 > > > > Everything about this new install appears to work > fine, EXCEPT pylint. > > > > I'm seeing tons of strange E1101 error on standard > modules, like > > time.sleep() and others (readline, etc.) For > example: > > > > $ cat sl.py > > import time > > time.sleep(1) > > > > $ python sl.py > > > > > > $ pylint sl.py > > ************* Module sl > > C: 1, 0: Missing module docstring > (missing-docstring) > > E: 2, 0: Module 'time' has no 'sleep' member > (no-member) > > > > If I run this with my old setup (older > python/pylint/etc.), it doesn't > > complain at all. Why am I seeing this invalid > error? > > > By "at all" I mean it doesn't complain about > time.sleep(); of course I > still get the missing docstring message in the old > version :-). > > Looking at this it seems like all the members which > are loaded from a > shared library (e.g., > lib/python2.7/lib-dynload/time.so etc.) have this > problem: I see it with datetime, readline, some socket > stuff, etc. > > Did I break something with my installation of python > or one of the > packages? How does pylint normally discover module > members when the > implementation is in C rather than python? > > _______________________________________________ > code-quality mailing list > code-quality at python.org > https://mail.python.org/mailman/listinfo/code-quality > > > > > From paul at mad-scientist.net Wed Nov 26 23:14:57 2014 From: paul at mad-scientist.net (Paul Smith) Date: Wed, 26 Nov 2014 17:14:57 -0500 Subject: [code-quality] Lots of E1101 in standard modules after upgrading pylint to 1.4.0 In-Reply-To: <1417039497.23650.62.camel@homebase> References: <1416985385.23650.11.camel@homebase> <1417012918.23650.20.camel@homebase> <1417039497.23650.62.camel@homebase> Message-ID: <1417040097.23650.64.camel@homebase> On Wed, 2014-11-26 at 17:04 -0500, Paul Smith wrote: > If I reset STD_LIB_DIRS like this: > > >>> modutils.STD_LIB_DIRS = ['/opt/python/lib/python2.7', > '/opt/python/x86_64-linux/lib/python2.7'] > > Then I get: > > >>> modutils.is_standard_module('time') > True OK, if I apply the patch below to astroid it seems to fix the problem... not sure if this is correct or not. --- a/astroid/modutils.py 2014-11-26 17:07:17.110110766 -0500 +++ b/astroid/modutils.py 2014-11-26 17:07:47.846275548 -0500 @@ -66,7 +66,7 @@ # off the detection logic for standard library modules, thus the # workaround. STD_LIB_DIRS = [ - get_python_lib(standard_lib=True, prefix=sys.prefix), + get_python_lib(standard_lib=True, prefix=sys.exec_prefix), get_python_lib(standard_lib=True)] if os.name == 'nt': STD_LIB_DIRS.append(os.path.join(sys.prefix, 'dlls')) From paul at mad-scientist.net Wed Nov 26 23:27:46 2014 From: paul at mad-scientist.net (Paul Smith) Date: Wed, 26 Nov 2014 17:27:46 -0500 Subject: [code-quality] Lots of E1101 in standard modules after upgrading pylint to 1.4.0 In-Reply-To: <1417040097.23650.64.camel@homebase> References: <1416985385.23650.11.camel@homebase> <1417012918.23650.20.camel@homebase> <1417039497.23650.62.camel@homebase> <1417040097.23650.64.camel@homebase> Message-ID: <1417040866.23650.66.camel@homebase> On Wed, 2014-11-26 at 17:14 -0500, Paul Smith wrote: > if os.name == 'nt': > STD_LIB_DIRS.append(os.path.join(sys.prefix, 'dlls')) Also, I don't use this on Windows but this looks odd to me: wouldn't the "dlls" directory appear underneath exec_prefix, not prefix? From pcmanticore at gmail.com Thu Nov 27 11:15:29 2014 From: pcmanticore at gmail.com (Claudiu Popa) Date: Thu, 27 Nov 2014 12:15:29 +0200 Subject: [code-quality] PyLint catching comprehension binding leaks (was: Warn on list comprehension variables used outside list comprehensions?) In-Reply-To: References: Message-ID: On Wed, Nov 26, 2014 at 6:57 PM, Ian Cordasco wrote: > On Wed, Nov 26, 2014 at 10:51 AM, Keith Derrick wrote: >> I couldn't find anything in their bug list about this, so thought I'd check. >> >> Just ran pylint at the command line with pylint 1.4 on this file >> (docstrings and import just to suppress miscellaneous complaints) >> >>> 1 ''' Test file for pylint ''' >>> 2 from __future__ import print_function >>> 3 >>> 4 def method1(): >>> 5 ''' Let x bleed from comprehension ''' >>> 6 [x for x in range(3)] >>> 7 print(x) >>> 8 >>> 9 def method2(): >>> 10 ''' reuse/hide x from local scope in comprehension ''' >>> 11 y = 10 >>> 12 [y for y in range(3)] >>> 13 print(y) >>> 14 >> >> For python 2.7.6 >>> ************* Module bug >>> W: 6, 4: Expression "[x for x in range(3)]" is assigned to nothing >>> (expression-not-assigned) >>> W: 7,10: Using possibly undefined loop variable 'x' >>> (undefined-loop-variable) >>> C: 11, 4: Invalid variable name "y" (invalid-name) >>> W: 12, 4: Expression "[y for y in range(3)]" is assigned to nothing >>> (expression-not-assigned) >> For python 3.4.0 >>> ************* Module bug >>> W: 6, 4: Expression "[x for x in range(3)]" is assigned to nothing >>> (expression-not-assigned) >>> E: 7,10: Undefined variable 'x' (undefined-variable) >>> C: 11, 4: Invalid variable name "y" (invalid-name) >>> W: 12, 4: Expression "[y for y in range(3)]" is assigned to nothing >>> (expression-not-assigned) >> >> So, it warns about the first case in py2 and gives an error in py3 which >> is expected. >> >> But it seems to be confused by the second case, flagging "y" as an >> invalid variable-name at the assignment point for both versions. >> >> Keith Derrick | Principal Engineer, Connected Platform | Engineering >> LG Silicon Valley Lab | 5150 Gt America Parkway, Santa Clara, CA 95054 >> Office: 408.610-5746 | Mobile: 831.383.9567 | LG.com >> > > At this point, I'd recommend repling to the retitled message to catch > the PyLint maintainer's focus. We don't warn for this leak right now, but we are planning to add this in 1.5, under the --py3k checker. We just didn't have enough time to add this feature for this release. (See https://bitbucket.org/logilab/pylint/issue/376 for the remaining work related to the --py3k checker) From ned at nedbatchelder.com Thu Nov 27 16:19:28 2014 From: ned at nedbatchelder.com (Ned Batchelder) Date: Thu, 27 Nov 2014 10:19:28 -0500 Subject: [code-quality] [ANN] Pylint 1.4 released In-Reply-To: References: Message-ID: On 11/23/14 4:45 AM, Claudiu Popa wrote: > Hello! > > On behalf of the Pylint development team, I'm happy to announce that > Pylint 1.4 has been released. > > This release has a lot of improvements over the last one. One of the > main differences is that support for Python versions < 2.7 has been > droped, which allows us to support Python 2.7 and 3.3+ from a single > codebase. > > > Other important changes: BTW: it would be great to include a link to the full list of changes, wherever they might be. > > * A new spelling checker (disabled by default). > > If you find any bugs, don't hesitate to open a new issue on our issue tracker. I found a problem with the new spell feature, but the issue tracker (http://www.bytebucket.org/logilab/pylint/issues) seems broken: everything I tried ended at a 403 CSRF validation failure page. Is there another way to report problems? -- Ned Batchelder, http://nedbatchelder.com From pcmanticore at gmail.com Thu Nov 27 17:58:43 2014 From: pcmanticore at gmail.com (Claudiu Popa) Date: Thu, 27 Nov 2014 18:58:43 +0200 Subject: [code-quality] [Pylint-dev] [ANN] Pylint 1.4 released In-Reply-To: References: Message-ID: On Thu, Nov 27, 2014 at 5:19 PM, Ned Batchelder wrote: > On 11/23/14 4:45 AM, Claudiu Popa wrote: >> >> Hello! >> >> On behalf of the Pylint development team, I'm happy to announce that >> Pylint 1.4 has been released. >> >> This release has a lot of improvements over the last one. One of the >> main differences is that support for Python versions < 2.7 has been >> droped, which allows us to support Python 2.7 and 3.3+ from a single >> codebase. >> >> >> Other important changes: > > > BTW: it would be great to include a link to the full list of changes, > wherever they might be. Indeed. We don't have right now a full list of changes, except the changelog for 1.4: https://bitbucket.org/logilab/pylint/src/ede9e9ebc2557f768e129f29aeb365608ae96ea2/ChangeLog?at=default#cl-4 > >> >> * A new spelling checker (disabled by default). >> >> If you find any bugs, don't hesitate to open a new issue on our issue >> tracker. > > > I found a problem with the new spell feature, but the issue tracker > (http://www.bytebucket.org/logilab/pylint/issues) seems broken: everything I > tried ended at a 403 CSRF validation failure page. > > Is there another way to report problems? > Hm, isn't it bitbucket.org? Never encountered bytebucket.org until now. From julien.cristau at logilab.fr Thu Nov 27 17:59:34 2014 From: julien.cristau at logilab.fr (Julien Cristau) Date: Thu, 27 Nov 2014 17:59:34 +0100 Subject: [code-quality] [ANN] Pylint 1.4 released In-Reply-To: References: Message-ID: <20141127165934.GA7237@crater2.logilab.fr> On Thu, Nov 27, 2014 at 10:19:28 -0500, Ned Batchelder wrote: > I found a problem with the new spell feature, but the issue tracker > (http://www.bytebucket.org/logilab/pylint/issues) seems broken: > everything I tried ended at a 403 CSRF validation failure page. > Not sure where that url came from, afaik the tracker is https://bitbucket.org/logilab/pylint/issues Cheers, Julien -- Julien Cristau Logilab http://www.logilab.fr/ Informatique scientifique & gestion de connaissances From shlomme at gmail.com Thu Nov 27 19:35:24 2014 From: shlomme at gmail.com (Torsten Marek) Date: Thu, 27 Nov 2014 19:35:24 +0100 Subject: [code-quality] Lots of E1101 in standard modules after upgrading pylint to 1.4.0 In-Reply-To: <1417040866.23650.66.camel@homebase> References: <1416985385.23650.11.camel@homebase> <1417012918.23650.20.camel@homebase> <1417039497.23650.62.camel@homebase> <1417040097.23650.64.camel@homebase> <1417040866.23650.66.camel@homebase> Message-ID: Yeah, that looks like the problem. Let me play around with this a little bit, there are some subtleties with virtualenvs. We'll have an official fix soon. // Torsten 2014-11-26 23:27 GMT+01:00 Paul Smith : > On Wed, 2014-11-26 at 17:14 -0500, Paul Smith wrote: > > if os.name == 'nt': > > STD_LIB_DIRS.append(os.path.join(sys.prefix, 'dlls')) > > Also, I don't use this on Windows but this looks odd to me: wouldn't the > "dlls" directory appear underneath exec_prefix, not prefix? > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shlomme at gmail.com Thu Nov 27 22:05:00 2014 From: shlomme at gmail.com (Torsten Marek) Date: Thu, 27 Nov 2014 22:05:00 +0100 Subject: [code-quality] Lots of E1101 in standard modules after upgrading pylint to 1.4.0 In-Reply-To: References: <1416985385.23650.11.camel@homebase> <1417012918.23650.20.camel@homebase> <1417039497.23650.62.camel@homebase> <1417040097.23650.64.camel@homebase> <1417040866.23650.66.camel@homebase> Message-ID: Fixed at head. Until there's a new release, you can also use the hidden pylint option --unsafe-load-any-extension=y // Torsten 2014-11-27 19:35 GMT+01:00 Torsten Marek : > Yeah, that looks like the problem. Let me play around with this a little > bit, there are some subtleties with virtualenvs. We'll have an official fix > soon. > > // Torsten > > 2014-11-26 23:27 GMT+01:00 Paul Smith : > >> On Wed, 2014-11-26 at 17:14 -0500, Paul Smith wrote: >> > if os.name == 'nt': >> > STD_LIB_DIRS.append(os.path.join(sys.prefix, 'dlls')) >> >> Also, I don't use this on Windows but this looks odd to me: wouldn't the >> "dlls" directory appear underneath exec_prefix, not prefix? >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From godfryd at gmail.com Thu Nov 27 21:43:42 2014 From: godfryd at gmail.com (Michal Nowikowski) Date: Thu, 27 Nov 2014 21:43:42 +0100 Subject: [code-quality] [Pylint-dev] [ANN] Pylint 1.4 released In-Reply-To: References: <20141127165934.GA7237@crater2.logilab.fr> Message-ID: Hello, Just submitted pull request with a fix: https://bitbucket.org/logilab/pylint/pull-request/205/fixed-reading-list-of-ignored-words-for/diff Regards, Godfryd On Thu, Nov 27, 2014 at 7:00 PM, Ned Batchelder wrote: > On 11/27/14 11:59 AM, Julien Cristau wrote: > >> On Thu, Nov 27, 2014 at 10:19:28 -0500, Ned Batchelder wrote: >> >> I found a problem with the new spell feature, but the issue tracker >>> (http://www.bytebucket.org/logilab/pylint/issues) seems broken: >>> everything I tried ended at a 403 CSRF validation failure page. >>> >>> Not sure where that url came from, afaik the tracker is >> https://bitbucket.org/logilab/pylint/issues >> >> > Hmm, not sure where bytebucket came from either! Sorry for the noise... > > Cheers, >> Julien >> >> > > -- > Ned Batchelder, http://nedbatchelder.com > > _______________________________________________ > Pylint-dev mailing list > Pylint-dev at lists.logilab.org > http://lists.logilab.org/mailman/listinfo/pylint-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul at mad-scientist.net Fri Nov 28 21:57:28 2014 From: paul at mad-scientist.net (Paul Smith) Date: Fri, 28 Nov 2014 15:57:28 -0500 Subject: [code-quality] pylint can't deduce the type of an object Message-ID: <1417208248.3562.23.camel@homebase> Hi all; as mentioned in a previous post, I am looking into updating my curring pylint (1.0.1) to newer pylint 1.4.0. I've got one significant problem left and I'm not sure how to resolve it: in the new pylint I'm seeing a number of errors like this: mymod.py:227: [E1101(no-member), MyMod.__str__] Instance of 'int' has no 'get_val' member The code in question is something like: self.someobj.get_val() The problem appears to be that pylint cannot tell the type of the object stored in "someobj", so it assumes "int" and gives an error. The value of self.someobj is passed in via the constructor, and so this module really _doesn't_ know the type. I checked what pylint 1.0.1 is doing and it seems to be ignoring that value altogether, so it doesn't give the above incorrect error but it also doesn't complain if, for example, I change the call to "self.someobj.getval()" (or any other non-existent method). Can I do something to get pylint to understand this type? If it cannot figure it out automatically, is there some way I can hint or annotate the type of this member so pylint will understand what it is? Is there a way to disable the "assume an int" processing and ignore unknown members, like the old pylint did, without disabling E1101 altogether? Pylint is run automatically as part of our static analysis tests so I can't upgrade until I can get the new version to run clean. From pcmanticore at gmail.com Fri Nov 28 22:09:03 2014 From: pcmanticore at gmail.com (Claudiu Popa) Date: Fri, 28 Nov 2014 23:09:03 +0200 Subject: [code-quality] pylint can't deduce the type of an object In-Reply-To: <1417208248.3562.23.camel@homebase> References: <1417208248.3562.23.camel@homebase> Message-ID: On Fri, Nov 28, 2014 at 10:57 PM, Paul Smith wrote: > Hi all; as mentioned in a previous post, I am looking into updating my > curring pylint (1.0.1) to newer pylint 1.4.0. > > I've got one significant problem left and I'm not sure how to resolve > it: in the new pylint I'm seeing a number of errors like this: > > mymod.py:227: [E1101(no-member), MyMod.__str__] Instance of 'int' has no 'get_val' member Could you give me a small sample to reproduce this problem? If it's not an opened issue with this error, you could open one on our bug tracker, so it won't be lost in mails. From paul at mad-scientist.net Fri Nov 28 22:18:23 2014 From: paul at mad-scientist.net (Paul Smith) Date: Fri, 28 Nov 2014 16:18:23 -0500 Subject: [code-quality] pylint can't deduce the type of an object In-Reply-To: References: <1417208248.3562.23.camel@homebase> Message-ID: <1417209503.3562.28.camel@homebase> On Fri, 2014-11-28 at 23:09 +0200, Claudiu Popa wrote: > On Fri, Nov 28, 2014 at 10:57 PM, Paul Smith wrote: > > Hi all; as mentioned in a previous post, I am looking into updating my > > curring pylint (1.0.1) to newer pylint 1.4.0. > > > > I've got one significant problem left and I'm not sure how to resolve > > it: in the new pylint I'm seeing a number of errors like this: > > > > mymod.py:227: [E1101(no-member), MyMod.__str__] Instance of 'int' has no 'get_val' member > > Could you give me a small sample to reproduce this problem? > If it's not an opened issue with this error, you could open one on our > bug tracker, so it won't be lost in mails. Well, it's a really complex environment but I'll see if I can come up with a simpler repro case. I just figured that this would be something that's a known issue: since python is dynamically typed it must be the case that a member can contain an object where pylint (being a static checker) cannot determine its type. Are you saying that Python should NOT be assuming the type as int here? From paul at mad-scientist.net Sat Nov 29 18:43:10 2014 From: paul at mad-scientist.net (Paul Smith) Date: Sat, 29 Nov 2014 12:43:10 -0500 Subject: [code-quality] pylint can't deduce the type of an object In-Reply-To: References: <1417208248.3562.23.camel@homebase> Message-ID: <1417282990.3562.30.camel@homebase> On Fri, 2014-11-28 at 23:09 +0200, Claudiu Popa wrote: > On Fri, Nov 28, 2014 at 10:57 PM, Paul Smith wrote: > > Hi all; as mentioned in a previous post, I am looking into updating my > > curring pylint (1.0.1) to newer pylint 1.4.0. > > > > I've got one significant problem left and I'm not sure how to resolve > > it: in the new pylint I'm seeing a number of errors like this: > > > > mymod.py:227: [E1101(no-member), MyMod.__str__] Instance of 'int' has no 'get_val' member > > Could you give me a small sample to reproduce this problem? > If it's not an opened issue with this error, you could open one on our > bug tracker, so it won't be lost in mails. I constructed a small(-ish) test case and created an issue in the tracker: https://bitbucket.org/logilab/pylint/issue/400/pylint-assumes-an-incorrect-member-type As mentioned there the problem seems to be that I'm creating an object on a separate thread, asynchronously, based on some content I'm reading over a network socket and then when I try to use the object on the main thread pylint isn't able to deduce the type.