[code-quality] Need help with strange test failure for pylint-django

[James Cooke]

Update for those interested, the solution is here with explanation: https://github.com/PyCQA/pylint-django/pull/183/commits/538546044ac5a9785dc85369394d2de8c78349fc

Cheers,

James



On Tue, 24 Jul 2018, at 10:23 PM, Alexander Todorov wrote:
> Hi folks,
> I need help debugging a strange test failure in pylint-django that started 
> happening after the upgrade to pylint/astroid 2.0.
> 
> For all I can tell we've already adjusted the code and this failure is 
> happening 
> randomly. pylint-django's test used uses the 
> test_functional.LintModuleTest 
> class from pylint (I know it isn't public interface) and executes 
> against input 
> files comparing the linter results with expected.
> 
> The error that we get is:
> 
> E           Failed: Wrong results for file "func_noerror_foreignkeys":
> E
> E           Unexpected in testdata:
> E             27: no-member
> 
> The strange thing is that on one test job in Travis this passed with Python 3.4 
> and Django 2.0 (see job #447.3):
> https://travis-ci.org/PyCQA/pylint-django/builds/407129036
> 
> and on the next commit this failed on Python 3.4/Django 2.0 (see #448.3):
> https://travis-ci.org/PyCQA/pylint-django/builds/407800899
> 
> however it passed on 3.5 + Django 2.0.
> 
> The two test jobs use the same code (minor differences in commits due to 
> squash/conflicts).
> 
> When the linter is executed manually against the offending file it produces a 
> 10/10 score.
> 
> 
> If I modify the test suite like so:
> 
> --- a/pylint_django/tests/test_func.py
> +++ b/pylint_django/tests/test_func.py
> @@ -43,6 +43,8 @@ def get_tests(input_dir='input', sort=False):
> 
>       suite = []
>       for fname in os.listdir(input_dir):
> +        if fname.find('func_noerror_foreign') == -1:
> +            continue
>           if fname != '__init__.py' and fname.endswith('.py'):
>               suite.append(test_functional.FunctionalTestFile(input_dir, fname))
> 
> then I get only a subset of the tests executed (including the offending 
> one) and 
> this time it passes.
> 
> 
> Anyone seen such erratic behaviour ?  I think this is a flaky test but I 
> have no 
> idea how to approach debugging that.
> 
> 
> For the record this is blocking us from releasing a new version of 
> pylint-django 
> so any help is appreciated.
> 
> --
> Alex
> 
> _______________________________________________
> code-quality mailing list
> code-quality at python.org
> https://mail.python.org/mailman/listinfo/code-quality