From noamraph at gmail.com  Thu Nov  5 05:21:01 2015
From: noamraph at gmail.com (Noam Yorav-Raphael)
Date: Thu, 5 Nov 2015 12:21:01 +0200
Subject: [code-quality] Make pylint faster by caching modules that passed
 without warnings
Message-ID: <CAO8o=D6zR8NRodQAWzdXccjn9JSD-c0htM_k50JSFf7KTiXAMA@mail.gmail.com>

Hi,

Take a look at cached_pylint.py at
https://gist.github.com/noamraph/c933b32deb8304ac7ccd

It records module dependencies, and when a module passes with no warnings,
it records it together with the sha1 of the contents of each of its
dependencies, so when run again it could pass without checking it.

Dependencies are recorded by using infer and by using ImportsChecker, so if
module A imports something from module B that was actually defined in
module C, both B and C will be recorded as dependencies of A.

We have a medium sized code repository that took two minutes for each
check. Now pylint only checks the modified modules, and it takes two
seconds.

I tested it with pylint 1.4.3.

Hope this helps someone,

Noam
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20151105/15d607f9/attachment.html>

From ned at nedbatchelder.com  Thu Nov  5 06:24:14 2015
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Thu, 5 Nov 2015 06:24:14 -0500
Subject: [code-quality] Make pylint faster by caching modules that
 passed without warnings
In-Reply-To: <CAO8o=D6zR8NRodQAWzdXccjn9JSD-c0htM_k50JSFf7KTiXAMA@mail.gmail.com>
References: <CAO8o=D6zR8NRodQAWzdXccjn9JSD-c0htM_k50JSFf7KTiXAMA@mail.gmail.com>
Message-ID: <563B3C5E.2030909@nedbatchelder.com>

On 11/5/15 5:21 AM, Noam Yorav-Raphael wrote:
> Hi,
>
> Take a look at cached_pylint.py at 
> https://gist.github.com/noamraph/c933b32deb8304ac7ccd
>
> It records module dependencies, and when a module passes with no 
> warnings, it records it together with the sha1 of the contents of each 
> of its dependencies, so when run again it could pass without checking it.
>
> Dependencies are recorded by using infer and by using ImportsChecker, 
> so if module A imports something from module B that was actually 
> defined in module C, both B and C will be recorded as dependencies of A.
>
> We have a medium sized code repository that took two minutes for each 
> check. Now pylint only checks the modified modules, and it takes two 
> seconds.
>
This sounds like a great feature.  Have you considered contributing it 
as a pull request into pylint?

--Ned.

> I tested it with pylint 1.4.3.
>
> Hope this helps someone,
>
> Noam

From ned at nedbatchelder.com  Thu Nov  5 10:55:19 2015
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Thu, 5 Nov 2015 10:55:19 -0500
Subject: [code-quality] Make pylint faster by caching modules that
 passed without warnings
In-Reply-To: <563B3C5E.2030909@nedbatchelder.com>
References: <CAO8o=D6zR8NRodQAWzdXccjn9JSD-c0htM_k50JSFf7KTiXAMA@mail.gmail.com>
 <563B3C5E.2030909@nedbatchelder.com>
Message-ID: <563B7BE7.3030002@nedbatchelder.com>

I tried this on my large project (Open edX), and it generally worked 
great, but was one crash:

   File "../more/cached_pylint.py", line 252, in <module>
     sys.exit(main())
   File "../more/cached_pylint.py", line 248, in main
     run = CachedRun(sys.argv[1:], exit=False)
   File 
"/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pylint/lint.py", 
line 1332, in __init__
     linter.check(args)
   File "../more/cached_pylint.py", line 137, in check
     PyLinter.check(self, files_or_modules)
   File 
"/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pylint/lint.py", 
line 747, in check
     self._do_check(files_or_modules)
   File 
"/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pylint/lint.py", 
line 869, in _do_check
     self.check_astroid_module(ast_node, walker, rawcheckers, tokencheckers)
   File 
"/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pylint/lint.py", 
line 946, in check_astroid_module
     walker.walk(ast_node)
   File 
"/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pylint/utils.py", 
line 874, in walk
     self.walk(child)
   File 
"/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pylint/utils.py", 
line 871, in walk
     cb(astroid)
   File 
"/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pylint/checkers/variables.py", 
line 904, in visit_from
     self._check_module_attrs(node, module, name.split('.'))
   File 
"/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/pylint/checkers/variables.py", 
line 974, in _check_module_attrs
     module = next(module.getattr(name)[0].infer())
StopIteration

Any clue what might be causing that?

--Ned.

On 11/5/15 6:24 AM, Ned Batchelder wrote:
> On 11/5/15 5:21 AM, Noam Yorav-Raphael wrote:
>> Hi,
>>
>> Take a look at cached_pylint.py at 
>> https://gist.github.com/noamraph/c933b32deb8304ac7ccd
>>
>> It records module dependencies, and when a module passes with no 
>> warnings, it records it together with the sha1 of the contents of 
>> each of its dependencies, so when run again it could pass without 
>> checking it.
>>
>> Dependencies are recorded by using infer and by using ImportsChecker, 
>> so if module A imports something from module B that was actually 
>> defined in module C, both B and C will be recorded as dependencies of A.
>>
>> We have a medium sized code repository that took two minutes for each 
>> check. Now pylint only checks the modified modules, and it takes two 
>> seconds.
>>
> This sounds like a great feature.  Have you considered contributing it 
> as a pull request into pylint?
>
> --Ned.
>
>> I tested it with pylint 1.4.3.
>>
>> Hope this helps someone,
>>
>> Noam


From noamraph at gmail.com  Sat Nov  7 16:04:49 2015
From: noamraph at gmail.com (Noam Yorav-Raphael)
Date: Sat, 7 Nov 2015 23:04:49 +0200
Subject: [code-quality] Make pylint faster by caching modules that
 passed without warnings
In-Reply-To: <CAO8o=D6zR8NRodQAWzdXccjn9JSD-c0htM_k50JSFf7KTiXAMA@mail.gmail.com>
References: <CAO8o=D6zR8NRodQAWzdXccjn9JSD-c0htM_k50JSFf7KTiXAMA@mail.gmail.com>
Message-ID: <CAO8o=D45pNcGXn0FZ8dc+JyAOF_Mvmq39rQYqC0jreK8q+XJ9w@mail.gmail.com>

(I'm sorry, I accidentally had mail delivery turned off, so I reply to
myself)

> This sounds like a great feature.  Have you considered contributing it
> as a pull request into pylint?

I will be glad to - you can use the code for whatever you like - but I
think that a member of the team should review my code and say what he
thinks and how it should be integrated into the main code.

> I tried this on my large project (Open edX), and it generally worked
> great, but was one crash:

I updated the gist so now I hope it would work - can you check?

Cheers,
Noam



On Thu, Nov 5, 2015 at 12:21 PM, Noam Yorav-Raphael <noamraph at gmail.com>
wrote:

> Hi,
>
> Take a look at cached_pylint.py at
> https://gist.github.com/noamraph/c933b32deb8304ac7ccd
>
> It records module dependencies, and when a module passes with no warnings,
> it records it together with the sha1 of the contents of each of its
> dependencies, so when run again it could pass without checking it.
>
> Dependencies are recorded by using infer and by using ImportsChecker, so
> if module A imports something from module B that was actually defined in
> module C, both B and C will be recorded as dependencies of A.
>
> We have a medium sized code repository that took two minutes for each
> check. Now pylint only checks the modified modules, and it takes two
> seconds.
>
> I tested it with pylint 1.4.3.
>
> Hope this helps someone,
>
> Noam
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20151107/7e93355e/attachment.html>

From ned at nedbatchelder.com  Sun Nov  8 06:51:36 2015
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Sun, 8 Nov 2015 06:51:36 -0500
Subject: [code-quality] Make pylint faster by caching modules that
 passed without warnings
In-Reply-To: <CAO8o=D45pNcGXn0FZ8dc+JyAOF_Mvmq39rQYqC0jreK8q+XJ9w@mail.gmail.com>
References: <CAO8o=D6zR8NRodQAWzdXccjn9JSD-c0htM_k50JSFf7KTiXAMA@mail.gmail.com>
 <CAO8o=D45pNcGXn0FZ8dc+JyAOF_Mvmq39rQYqC0jreK8q+XJ9w@mail.gmail.com>
Message-ID: <563F3748.9000206@nedbatchelder.com>

On 11/7/15 4:04 PM, Noam Yorav-Raphael wrote:
> (I'm sorry, I accidentally had mail delivery turned off, so I reply to 
> myself)
>
> > This sounds like a great feature.  Have you considered contributing it
> > as a pull request into pylint?
>
> I will be glad to - you can use the code for whatever you like - but I 
> think that a member of the team should review my code and say what he 
> thinks and how it should be integrated into the main code.
>
> > I tried this on my large project (Open edX), and it generally worked
> > great, but was one crash:
>
> I updated the gist so now I hope it would work - can you check?
Thanks, the crash is fixed.  On our code, which has a ton of violations 
that we are slowly working through, it's about the same speed as regular 
pylint though.  I think we need to shift the balance so that more files 
have no violations before the caching pays off.

Thanks, this is very interesting, and I hope it's pursued as a pylint 
feature.

--Ned.
>
> Cheers,
> Noam
>
>
>
> On Thu, Nov 5, 2015 at 12:21 PM, Noam Yorav-Raphael 
> <noamraph at gmail.com <mailto:noamraph at gmail.com>> wrote:
>
>     Hi,
>
>     Take a look at cached_pylint.py at
>     https://gist.github.com/noamraph/c933b32deb8304ac7ccd
>
>     It records module dependencies, and when a module passes with no
>     warnings, it records it together with the sha1 of the contents of
>     each of its dependencies, so when run again it could pass without
>     checking it.
>
>     Dependencies are recorded by using infer and by using
>     ImportsChecker, so if module A imports something from module B
>     that was actually defined in module C, both B and C will be
>     recorded as dependencies of A.
>
>     We have a medium sized code repository that took two minutes for
>     each check. Now pylint only checks the modified modules, and it
>     takes two seconds.
>
>     I tested it with pylint 1.4.3.
>
>     Hope this helps someone,
>
>     Noam
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20151108/ccd189b9/attachment.html>

From sylvain.thenault at logilab.fr  Mon Nov 23 12:02:45 2015
From: sylvain.thenault at logilab.fr (Sylvain =?utf-8?B?VGjDqW5hdWx0?=)
Date: Mon, 23 Nov 2015 18:02:45 +0100
Subject: [code-quality] some arguable pylint PR
Message-ID: <20151123170245.GC22844@logilab.fr>

Hi there,

we (Logilab) have been recently working on pylint for a customer, and provided
at this occasion a couple of pull requests. I would like your opinion about some
of them, not really technically but rather if you think the functionnality
should go in pylint or not.

The first one is about a message for 'else: if <cond>' that could be turned into
'elif <cond>:

https://bitbucket.org/logilab/pylint/pull-requests/300/check-for-use-of-else-if-i

The other one is about a message about avoiding unnecessary usage of 'not'
identifier. This is explained in
https://bitbucket.org/logilab/pylint/issues/670/warning-upon-unneeded-negations
and there is a discussion about whether we should add a special case for class
implementin '__neq__'. See
https://bitbucket.org/logilab/pylint/pull-requests/293/check-for-nots-too-many-in-comparison/activity#

You might also be interested about this one :
https://bitbucket.org/logilab/pylint/pull-requests/299/check-if-the-type-of-a-variable-is/diff
which checks for assigment of several types to a single identifier.

Depending on the community (you) opinion, I'll integrate those in pylint or in a
separated checker we've done for our customer.

Regards
-- 
Sylvain Th?nault, LOGILAB, Paris (01.45.32.03.12) - Toulouse (05.62.17.16.42)
Formations Python, Debian, M?th. Agiles: http://www.logilab.fr/formations
D?veloppement logiciel sur mesure:       http://www.logilab.fr/services
CubicWeb, the semantic web framework:    http://www.cubicweb.org

From pcmanticore at gmail.com  Mon Nov 30 18:42:46 2015
From: pcmanticore at gmail.com (Claudiu Popa)
Date: Tue, 1 Dec 2015 01:42:46 +0200
Subject: [code-quality] Pylint 1.5.0 / Astroid 1.4.1 released
Message-ID: <CAMy=CLpp1SfTpkpktFyFVg0NwUZrq3LgXd75ZNqCyp7kDgjFAg@mail.gmail.com>

Hello,


I'm happy to announce you the release of Pylint 1.5.0,
respectively Astroid 1.4.1.

It's been over a year since the last major release
and the amount of changes that were brought into pylint
in this time is humongous, with over 30 new checks
and tons of bug fixes.

I would like to use this occasion for thanking for their contributions
the new committers who joined pylint's team in the past months,
ceridwen and Dmitry, as well as thanking the
contributors that made this release possible.

Here are some of the major changes of this release:

- We finally support Python 3.5 in both projects.

- We stopped depending on logilab-common, which
  means that users of the popular testing framework
  pytest will finally stop being confused by having
  two executables with the same name doing different
  things.

- Almost 30 new checks you might enjoy not having in
  your project. These checks fall into multiple categories,
  regarding maintainability, readability, recommendations
  for improving the code, as well as new type-checking
  capabilities. They are:

    'wrong-import-order', 'ungrouped-imports', 'wrong-import-position',
    'unneeded-not', 'simplifiable-if-condition', 'too-many-boolean-expressions'
    'too-many-nested-blocks', 'multiple-imports', 'duplicate-except',
    'using-constant-test', 'confusing-with-statement', 'singleton-comparison',
    'misplaced-comparison-constant', 'consider-using-enumerate',
    'nonlocal-and-global', 'continue-in-finally', 'misplaced-bare-raise',
    'nonlocal-without-binding', 'yield-inside-async-function',
    'too-many-star-expressions', 'invalid-star-assignment-target',
    'import-star-module-level, 'star-needs-assignment',
    'unexpected-special-method-signature', 'repeated-keyword'.

    Some new type checks for finding violations of the type system
    are 'unsubscriptable-object', 'unsupported-membership-test',
    'not-an-iterable', 'not-context-manager', 'not-async-context-manager',
    'duplicate-bases' and 'inconsistent-mro'.


- We also added a new 'extensions' component, which contains optional
  checkers that needs to be activated explicitly.

  These includes 'extensions.check_docs', which verifies a bunch of
  properties of the docstrings, such as checking that all function,
  method and constructor parameters are mentioned
  in the params and types part of the docstring. Also, it checks that
  there are no naming inconsistencies between the signature and
  the documentation, i.e. also report documented parameters that are missing
  in the signature. This is important to find cases where parameters are
  renamed only in the code, not in the documentation.

  Activate this checker with:

      --load-plugins=pylint.extensions.check_docs

Most of the work was put into astroid though, which got more capable in this
period of time. Unfortunately, more than half of the work that went into it was
postponed for astroid 1.6 and astroid 2.0, since it wasn't deemed stable
enough for this release.
New features worth mentioning are:

  - Python 3.5 support

  - some of the nodes were renamed in order to be more similar to
    builtin's ast module, such as Class to ClassDef, Function to
    FunctionDef Getattr to Attribute etc, the old names being slated
    for removal in astroid 2.0.
    This affects the plugins as well, since it means that the visit
    methods should use the new names instead (so visit_classdef for class).
    Old names are still supported in visit methods for a while, until
    pylint 2.0. Activating the warnings when running should result in
    spurious PendingDeprecationWarnings when using the old node names.

  - add proper grammatical names for `infered` and `ass_type` methods,
    namely `inferred` and `assign_type`.

    The old methods will raise PendingDeprecationWarning, being slated
    for removal in astroid 2.0.

  - we added a new convenience API, `astroid.parse`, which can be used
    to retrieve an astroid AST from a source code string, similar to how
    ast.parse can be used to obtain a Python AST from a source string.

  - There's a new separate step for transforms.

    Until now, the transforms were applied at the same time the tree was
    being built. This was problematic if the transform functions were
    using inference, since the inference was executed on a partially
    constructed tree, which led to failures when post-building
    information was needed.

  - Better support for understanding builtins.

    We're understanding a bunch of new builtins, but unfortunately
    most of them weren't released, since they weren't stable
    enough for now, due to some inherent problems that astroid
    has. The most important that's released though is the understanding
    of super, which means that we can now detect problems of sort:

    class A(B, C, D):
      def __init__(self):
          super(A, self).call_missing_method()

You can read the complete changelog here
https://bitbucket.org/logilab/astroid/raw/4f45b6fc4c23b80b4f7154add1b9d9cee3a54297/ChangeLog
and https://bitbucket.org/logilab/pylint/raw/2871c4d7084478b5db02f384e4e5167641c698f2/ChangeLog

Now with every release there's going to be new false positives or regressions
or things that could be improved. Don't hesitate to open an issue or to
send a PR if you notice something's amiss and we'll try to have new
bug fix releases as soon as possible (you might have noticed that's
astroid 1.4.1..)

As a final note, we're planning to switch to GitHub soon, which will mean an
improved CI for us, resulting in less maintenance pain.


Thank you for reading all this and enjoy pylinting!


Claudiu