From ahirnish at arista.com  Wed Jul  1 11:40:30 2015
From: ahirnish at arista.com (Ahirnish Pareek)
Date: Wed, 1 Jul 2015 15:10:30 +0530
Subject: [code-quality] Adding python function arguments related
 customized rule in pylint
In-Reply-To: <CAMy=CLpJScuStNn31nnQvKwSCd5zxOqDk_qw2YUsOE0uymXY2A@mail.gmail.com>
References: <CAFEivchXZfLdZmdYpDCGSen+jnpPnCBUiZ2T_9veEaf0zVt2iw@mail.gmail.com>
 <CAMy=CLpJScuStNn31nnQvKwSCd5zxOqDk_qw2YUsOE0uymXY2A@mail.gmail.com>
Message-ID: <CAFEivci36n2vfnOTL6V0iGTt8n4qKRaxEsVp1d0h1LfwdL5T4w@mail.gmail.com>

Hi all,

I was going through the *checkers/typecheck.py* for reference but could not
understand it concretely.

I am basically unable to find how to retrieve arguments from Astng Function
node.

I am using visit_function() inside a checker class to visit all function
nodes but not able to get list of arguments(and varargs and kwargs too).

Also, is there a function which returns <type function> if we pass Astng
Function node to it?

Thanks in advance!


On Mon, Jun 29, 2015 at 11:46 PM, Claudiu Popa <pcmanticore at gmail.com>
wrote:

> On Mon, Jun 29, 2015 at 12:32 PM, Ahirnish Pareek <ahirnish at arista.com>
> wrote:
> > Hi all,
> >
> > I want to add a customized check to my pylintrc file to check and report
> > cases whenever I define a function in a python file like this:
> >
> > def foo( var1, name1=value, *args ):
> >
> > The problem that I want to deal with is that we should not have keyword
> > arguments( name1 ) before positional arguments( *args ). Although the
> > function can still be called by passing name1 as positional argument but
> > there would be no way to not specify name1 but specify *args.
> >
> > How can I add this check to pylintrc file?
> >
> > Thanks in advance!
> >
> >
>
> Hi,
>
> If you already have the checker, you can add it in the
> load-plugins entry from the pylintrc file, as in
> `load-plugins=qualified plugin path`.
> We don't have this check builtin right now, but it's something
> that could be useful, so if you could send it upstream, that would be
> really
> awesome.
>
>
> Thanks,
>
> Claudiu
>



-- 
Regards,
Ahirnish
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20150701/e2df9c68/attachment.html>

From pcmanticore at gmail.com  Wed Jul  1 12:01:26 2015
From: pcmanticore at gmail.com (Claudiu Popa)
Date: Wed, 1 Jul 2015 13:01:26 +0300
Subject: [code-quality] Adding python function arguments related
 customized rule in pylint
In-Reply-To: <CAFEivci36n2vfnOTL6V0iGTt8n4qKRaxEsVp1d0h1LfwdL5T4w@mail.gmail.com>
References: <CAFEivchXZfLdZmdYpDCGSen+jnpPnCBUiZ2T_9veEaf0zVt2iw@mail.gmail.com>
 <CAMy=CLpJScuStNn31nnQvKwSCd5zxOqDk_qw2YUsOE0uymXY2A@mail.gmail.com>
 <CAFEivci36n2vfnOTL6V0iGTt8n4qKRaxEsVp1d0h1LfwdL5T4w@mail.gmail.com>
Message-ID: <CAMy=CLrXNR00aXPyx0nr+4esGF0-Ey_74OPD41juuTnxkYVkHQ@mail.gmail.com>

Hi,

On Wed, Jul 1, 2015 at 12:40 PM, Ahirnish Pareek <ahirnish at arista.com> wrote:
> Hi all,
>
> I was going through the checkers/typecheck.py for reference but could not
> understand it concretely.
>
> I am basically unable to find how to retrieve arguments from Astng Function
> node.
>
> I am using visit_function() inside a checker class to visit all function
> nodes but not able to get list of arguments(and varargs and kwargs too).
>


You should access node.args (an Arguments) node, as seen below:

from astroid.test_utils import extract_node
n = extract_node('''
def test(a, *args, b=None, **kwargs):
   pass
''')
print(n.args.args, n.args.kwonlyargs, n.args.vararg, n.args.kwarg)


> Also, is there a function which returns <type function> if we pass Astng
> Function node to it?

Do you mean the equivalent of type(function) from Python?
You can try the new astroid.helpers.object_type from soon-to-be astroid 1.4:
https://bitbucket.org/logilab/astroid/src/3d3cb4d45368bc80b0e80d24d758d884a1ab4d17/astroid/helpers.py?at=default#cl-81


/Claudiu

From ahirnish at arista.com  Wed Jul  1 13:32:12 2015
From: ahirnish at arista.com (Ahirnish Pareek)
Date: Wed, 1 Jul 2015 17:02:12 +0530
Subject: [code-quality] Adding python function arguments related
 customized rule in pylint
In-Reply-To: <CAMy=CLrXNR00aXPyx0nr+4esGF0-Ey_74OPD41juuTnxkYVkHQ@mail.gmail.com>
References: <CAFEivchXZfLdZmdYpDCGSen+jnpPnCBUiZ2T_9veEaf0zVt2iw@mail.gmail.com>
 <CAMy=CLpJScuStNn31nnQvKwSCd5zxOqDk_qw2YUsOE0uymXY2A@mail.gmail.com>
 <CAFEivci36n2vfnOTL6V0iGTt8n4qKRaxEsVp1d0h1LfwdL5T4w@mail.gmail.com>
 <CAMy=CLrXNR00aXPyx0nr+4esGF0-Ey_74OPD41juuTnxkYVkHQ@mail.gmail.com>
Message-ID: <CAFEivcgCHz69NXCH2p_kkdNODRCP7rnPooO5rcMKpPfZmor6yA@mail.gmail.com>

Thanks Claudiu.

I was looking for this only.

Yes, I meant type(function) from Python. My organization doesnt have
asteroid in its python's site-packages. We are still importing from
logilab.astng module. Thanks for the help!


On Wed, Jul 1, 2015 at 3:31 PM, Claudiu Popa <pcmanticore at gmail.com> wrote:

> Hi,
>
> On Wed, Jul 1, 2015 at 12:40 PM, Ahirnish Pareek <ahirnish at arista.com>
> wrote:
> > Hi all,
> >
> > I was going through the checkers/typecheck.py for reference but could not
> > understand it concretely.
> >
> > I am basically unable to find how to retrieve arguments from Astng
> Function
> > node.
> >
> > I am using visit_function() inside a checker class to visit all function
> > nodes but not able to get list of arguments(and varargs and kwargs too).
> >
>
>
> You should access node.args (an Arguments) node, as seen below:
>
> from astroid.test_utils import extract_node
> n = extract_node('''
> def test(a, *args, b=None, **kwargs):
>    pass
> ''')
> print(n.args.args, n.args.kwonlyargs, n.args.vararg, n.args.kwarg)
>
>
> > Also, is there a function which returns <type function> if we pass Astng
> > Function node to it?
>
> Do you mean the equivalent of type(function) from Python?
> You can try the new astroid.helpers.object_type from soon-to-be astroid
> 1.4:
>
> https://bitbucket.org/logilab/astroid/src/3d3cb4d45368bc80b0e80d24d758d884a1ab4d17/astroid/helpers.py?at=default#cl-81
>
>
> /Claudiu
>



-- 
Regards,
Ahirnish
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20150701/4d768378/attachment.html>

From pcmanticore at gmail.com  Wed Jul  1 13:35:50 2015
From: pcmanticore at gmail.com (Claudiu Popa)
Date: Wed, 1 Jul 2015 14:35:50 +0300
Subject: [code-quality] Adding python function arguments related
 customized rule in pylint
In-Reply-To: <CAFEivcgCHz69NXCH2p_kkdNODRCP7rnPooO5rcMKpPfZmor6yA@mail.gmail.com>
References: <CAFEivchXZfLdZmdYpDCGSen+jnpPnCBUiZ2T_9veEaf0zVt2iw@mail.gmail.com>
 <CAMy=CLpJScuStNn31nnQvKwSCd5zxOqDk_qw2YUsOE0uymXY2A@mail.gmail.com>
 <CAFEivci36n2vfnOTL6V0iGTt8n4qKRaxEsVp1d0h1LfwdL5T4w@mail.gmail.com>
 <CAMy=CLrXNR00aXPyx0nr+4esGF0-Ey_74OPD41juuTnxkYVkHQ@mail.gmail.com>
 <CAFEivcgCHz69NXCH2p_kkdNODRCP7rnPooO5rcMKpPfZmor6yA@mail.gmail.com>
Message-ID: <CAMy=CLrJkG5A7YqemOaQ4=HFk0VBxY3=9JTu7PAhq0Q_S_69oQ@mail.gmail.com>

On Wed, Jul 1, 2015 at 2:32 PM, Ahirnish Pareek <ahirnish at arista.com> wrote:
> Thanks Claudiu.
>
> I was looking for this only.
>
> Yes, I meant type(function) from Python. My organization doesnt have
> asteroid in its python's site-packages. We are still importing from
> logilab.astng module. Thanks for the help!
>


Well, astroid is the old logilab.astng, but much better. ;-)

From ahirnish at arista.com  Thu Jul  2 08:45:08 2015
From: ahirnish at arista.com (Ahirnish Pareek)
Date: Thu, 2 Jul 2015 12:15:08 +0530
Subject: [code-quality] Adding python function arguments related
 customized rule in pylint
In-Reply-To: <CAMy=CLrJkG5A7YqemOaQ4=HFk0VBxY3=9JTu7PAhq0Q_S_69oQ@mail.gmail.com>
References: <CAFEivchXZfLdZmdYpDCGSen+jnpPnCBUiZ2T_9veEaf0zVt2iw@mail.gmail.com>
 <CAMy=CLpJScuStNn31nnQvKwSCd5zxOqDk_qw2YUsOE0uymXY2A@mail.gmail.com>
 <CAFEivci36n2vfnOTL6V0iGTt8n4qKRaxEsVp1d0h1LfwdL5T4w@mail.gmail.com>
 <CAMy=CLrXNR00aXPyx0nr+4esGF0-Ey_74OPD41juuTnxkYVkHQ@mail.gmail.com>
 <CAFEivcgCHz69NXCH2p_kkdNODRCP7rnPooO5rcMKpPfZmor6yA@mail.gmail.com>
 <CAMy=CLrJkG5A7YqemOaQ4=HFk0VBxY3=9JTu7PAhq0Q_S_69oQ@mail.gmail.com>
Message-ID: <CAFEivchnDSoLDpwkA4vAku6T2mQtT1HvdZX-CfM-fh_3By9DXw@mail.gmail.com>

Hi all,

I was wondering if there's a pattern to calculate Pylint's message-id
numbers? I know that first alphabet indicates if its a warning, error,
convention, refactor or fatal but how do we come up with 4-digit number
after that?

Like - E1605. E = Error but what is 1605?

Thanks.

On Wed, Jul 1, 2015 at 5:05 PM, Claudiu Popa <pcmanticore at gmail.com> wrote:

> On Wed, Jul 1, 2015 at 2:32 PM, Ahirnish Pareek <ahirnish at arista.com>
> wrote:
> > Thanks Claudiu.
> >
> > I was looking for this only.
> >
> > Yes, I meant type(function) from Python. My organization doesnt have
> > asteroid in its python's site-packages. We are still importing from
> > logilab.astng module. Thanks for the help!
> >
>
>
> Well, astroid is the old logilab.astng, but much better. ;-)
>



-- 
Regards,
Ahirnish
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20150702/d019cd8f/attachment.html>

From pcmanticore at gmail.com  Fri Jul  3 11:32:14 2015
From: pcmanticore at gmail.com (Claudiu Popa)
Date: Fri, 3 Jul 2015 12:32:14 +0300
Subject: [code-quality] Adding python function arguments related
 customized rule in pylint
In-Reply-To: <CAFEivchnDSoLDpwkA4vAku6T2mQtT1HvdZX-CfM-fh_3By9DXw@mail.gmail.com>
References: <CAFEivchXZfLdZmdYpDCGSen+jnpPnCBUiZ2T_9veEaf0zVt2iw@mail.gmail.com>
 <CAMy=CLpJScuStNn31nnQvKwSCd5zxOqDk_qw2YUsOE0uymXY2A@mail.gmail.com>
 <CAFEivci36n2vfnOTL6V0iGTt8n4qKRaxEsVp1d0h1LfwdL5T4w@mail.gmail.com>
 <CAMy=CLrXNR00aXPyx0nr+4esGF0-Ey_74OPD41juuTnxkYVkHQ@mail.gmail.com>
 <CAFEivcgCHz69NXCH2p_kkdNODRCP7rnPooO5rcMKpPfZmor6yA@mail.gmail.com>
 <CAMy=CLrJkG5A7YqemOaQ4=HFk0VBxY3=9JTu7PAhq0Q_S_69oQ@mail.gmail.com>
 <CAFEivchnDSoLDpwkA4vAku6T2mQtT1HvdZX-CfM-fh_3By9DXw@mail.gmail.com>
Message-ID: <CAMy=CLrijytqX=roxD8yOwNh0f7H2mLLxbm5G=4UYbo_ie5-TA@mail.gmail.com>

On Thu, Jul 2, 2015 at 9:45 AM, Ahirnish Pareek <ahirnish at arista.com> wrote:
> Hi all,
>
> I was wondering if there's a pattern to calculate Pylint's message-id
> numbers? I know that first alphabet indicates if its a warning, error,
> convention, refactor or fatal but how do we come up with 4-digit number
> after that?
>
> Like - E1605. E = Error but what is 1605?
>
> Thanks.


Base id of standard checkers (used in msg and report ids):
01: base
02: classes
03: format
04: import
05: misc
06: variables
07: exceptions
08: similar
09: design_analysis
10: newstyle
11: typecheck
12: logging
13: string_format
14: string_constant
15: stdlib
16: python3

So for E1605, that would be from the python 3 checker. Since it's pretty
hard to remember all this, the message ids will be deprecated at some
point, in favor of using symbolic names.

From wwu at dropbox.com  Tue Jul  7 11:05:39 2015
From: wwu at dropbox.com (Willy Wu)
Date: Tue, 7 Jul 2015 12:05:39 +0300
Subject: [code-quality] Searching multiple setup.cfg files for flake8
	section?
Message-ID: <CAB_0my92-VR0QVUyoKbTDH0zW9GP229YCX-HiA7S2tNuE+YM3A@mail.gmail.com>

Hey all,

We use flake8 at Dropbox and are pretty happy with it.  But one small issue
is the way that flake8 configuration gets discovered.  We use the
per-project configuration via setup.cfg files which are checked into source
control, but flake8 only considers the first setup.cfg in the directory
path (regardless of whether it contains a flake8 section or not).  This
forces us to duplicate our [flake8] configuration across all of our various
setup.cfgs.

Said in crude ascii form, this is a simplified version of our project
layout:
- repo root
  |- setup.cfg (default values)
  \- subservice #1
    \- setup.cfg (which has non-flake8 overrides for subservice #1)
  \- subservice #2
    \- setup.cfg (which similarly contains non-flake8 overrides for
subservice #2)

But before I go off trying to patch flake8 code, is there relevant
background on why you guys don't try to discover the first setup.cfg with a
flake8 section, but instead just consider the first encountered setup.cfg?
We've seen teams accidentally break flake8 checking whenever they add
setup.cfg files for non-flake8 projects, and the "fix" is to copy/paste the
flake8 section into those new files.

Or alternatively, should we be doing something different with our project
layout as a workaround?  It's entirely possible that we should be
configuring things differently overall...?

Thanks!

Willy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20150707/f3d76420/attachment.html>

From graffatcolmingov at gmail.com  Tue Jul  7 15:36:52 2015
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Tue, 7 Jul 2015 08:36:52 -0500
Subject: [code-quality] Searching multiple setup.cfg files for flake8
	section?
In-Reply-To: <CAB_0my92-VR0QVUyoKbTDH0zW9GP229YCX-HiA7S2tNuE+YM3A@mail.gmail.com>
References: <CAB_0my92-VR0QVUyoKbTDH0zW9GP229YCX-HiA7S2tNuE+YM3A@mail.gmail.com>
Message-ID: <CAN-Kwu3m5CqKxspYq6kXfG0tkMN247XDWAFsHdj0jJRL2SKErw@mail.gmail.com>

Hey Willy,

Responses in-line:

On Tue, Jul 7, 2015 at 4:05 AM, Willy Wu via code-quality
<code-quality at python.org> wrote:
> Hey all,
>
> We use flake8 at Dropbox and are pretty happy with it.  But one small issue
> is the way that flake8 configuration gets discovered.  We use the
> per-project configuration via setup.cfg files which are checked into source
> control, but flake8 only considers the first setup.cfg in the directory path
> (regardless of whether it contains a flake8 section or not).  This forces us
> to duplicate our [flake8] configuration across all of our various
> setup.cfgs.
>
> Said in crude ascii form, this is a simplified version of our project
> layout:
> - repo root
>   |- setup.cfg (default values)
>   \- subservice #1
>     \- setup.cfg (which has non-flake8 overrides for subservice #1)
>   \- subservice #2
>     \- setup.cfg (which similarly contains non-flake8 overrides for
> subservice #2)

So... to be clear... you're putting all of your Flake8 configuration
in "repo root/setup.cfg" but you have other setup.cfg files in
sub-directories. How are you running Flake8 (or perhaps, where are you
running Flake8) such that it picks up the setup.cfg files in the
sub-directories? Am I correct in assuming that you are running it in
that sub-directory? If not, can you give me the command-invocation
that triggers this?

> But before I go off trying to patch flake8 code, is there relevant
> background on why you guys don't try to discover the first setup.cfg with a
> flake8 section, but instead just consider the first encountered setup.cfg?
> We've seen teams accidentally break flake8 checking whenever they add
> setup.cfg files for non-flake8 projects, and the "fix" is to copy/paste the
> flake8 section into those new files.

The easiest answer is simplicity. Flake8 piggy-backs off of pep8 when
it comes to finding config files and consuming them.

I think there were a couple of people who had requested that we search
through all parent directories until we could find no parent directory
to search for configuration files, but I hope you can understand why
that is not feasible in the slightest. I believe the two places we
search by default are:

- Current working directory
- User's home directory

Traversing sub-directories is a little bit less offensive, but still
not optimal. There is the option to specify the config file
explicitly, but that doesn't help our users that much unless they have
some (honestly bizarre) setup where they're using something like
autoenv + shell aliases for flake8, e.g., autoenv could pick up the
right config file path when changing into a directory and `flake8` is
aliased to `/path/to/flake8 --config=$ENVVARWITHPATH`

That's definitely a bit over-the-top and not exactly user-friendly.

> Or alternatively, should we be doing something different with our project
> layout as a workaround?  It's entirely possible that we should be
> configuring things differently overall...?

That's tough to say honestly. I don't really feel like it's my place
(or anyone else's place) to tell you how to structure your project
layouts. That decision should be 100% in your court.

I'd still like to know a bit more about exactly how you're invoking
flake8 in this case. I don't have a clear enough understanding of the
problem to really be very helpful.

Cheers,
Ian

From ned at nedbatchelder.com  Tue Jul  7 15:40:35 2015
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Tue, 07 Jul 2015 09:40:35 -0400
Subject: [code-quality] Searching multiple setup.cfg files for flake8
 section?
In-Reply-To: <CAB_0my92-VR0QVUyoKbTDH0zW9GP229YCX-HiA7S2tNuE+YM3A@mail.gmail.com>
References: <CAB_0my92-VR0QVUyoKbTDH0zW9GP229YCX-HiA7S2tNuE+YM3A@mail.gmail.com>
Message-ID: <559BD6D3.2070202@nedbatchelder.com>



On 7/7/15 5:05 AM, Willy Wu via code-quality wrote:
> Hey all,
>
> We use flake8 at Dropbox and are pretty happy with it.  But one small 
> issue is the way that flake8 configuration gets discovered.  We use 
> the per-project configuration via setup.cfg files which are checked 
> into source control, but flake8 only considers the first setup.cfg in 
> the directory path (regardless of whether it contains a flake8 section 
> or not).  This forces us to duplicate our [flake8] configuration 
> across all of our various setup.cfgs.
>
> Said in crude ascii form, this is a simplified version of our project 
> layout:
> - repo root
>   |- setup.cfg (default values)
>   \- subservice #1
>     \- setup.cfg (which has non-flake8 overrides for subservice #1)
>   \- subservice #2
>     \- setup.cfg (which similarly contains non-flake8 overrides for 
> subservice #2)
>
> But before I go off trying to patch flake8 code, is there relevant 
> background on why you guys don't try to discover the first setup.cfg 
> with a flake8 section, but instead just consider the first encountered 
> setup.cfg?  We've seen teams accidentally break flake8 checking 
> whenever they add setup.cfg files for non-flake8 projects, and the 
> "fix" is to copy/paste the flake8 section into those new files.
>
> Or alternatively, should we be doing something different with our 
> project layout as a workaround?  It's entirely possible that we should 
> be configuring things differently overall...?

I'm interested in this question for coverage.py.  Does setuptools or pip 
examine all of the setup.cfg files in your repo?  I wouldn't have 
guessed it would do that, I thought it would simply read the first one 
it finds.  I've never seen a project laid out like this, so it's all new 
to me.

--Ned.
>
> Thanks!
>
> Willy
>
>
> _______________________________________________
> code-quality mailing list
> code-quality at python.org
> https://mail.python.org/mailman/listinfo/code-quality

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20150707/9629e035/attachment.html>

From ianlee1521 at gmail.com  Tue Jul  7 15:48:07 2015
From: ianlee1521 at gmail.com (Ian Lee)
Date: Tue, 7 Jul 2015 06:48:07 -0700
Subject: [code-quality] Searching multiple setup.cfg files for flake8
	section?
In-Reply-To: <559BD6D3.2070202@nedbatchelder.com>
References: <CAB_0my92-VR0QVUyoKbTDH0zW9GP229YCX-HiA7S2tNuE+YM3A@mail.gmail.com>
 <559BD6D3.2070202@nedbatchelder.com>
Message-ID: <CA+GyjMk6wqLrhbZF36-fNgPhiY9scv2+Mwrsgy_8D=UPFmqoaA@mail.gmail.com>

On Tuesday, July 7, 2015, Ned Batchelder <ned at nedbatchelder.com> wrote:

>
>
> On 7/7/15 5:05 AM, Willy Wu via code-quality wrote:
>
> Hey all,
>
>  We use flake8 at Dropbox and are pretty happy with it.  But one small
> issue is the way that flake8 configuration gets discovered.  We use the
> per-project configuration via setup.cfg files which are checked into source
> control, but flake8 only considers the first setup.cfg in the directory
> path (regardless of whether it contains a flake8 section or not).  This
> forces us to duplicate our [flake8] configuration across all of our various
> setup.cfgs.
>
>  Said in crude ascii form, this is a simplified version of our project
> layout:
> - repo root
>   |- setup.cfg (default values)
>   \- subservice #1
>     \- setup.cfg (which has non-flake8 overrides for subservice #1)
>   \- subservice #2
>     \- setup.cfg (which similarly contains non-flake8 overrides for
> subservice #2)
>
>  But before I go off trying to patch flake8 code, is there relevant
> background on why you guys don't try to discover the first setup.cfg with a
> flake8 section, but instead just consider the first encountered setup.cfg?
> We've seen teams accidentally break flake8 checking whenever they add
> setup.cfg files for non-flake8 projects, and the "fix" is to copy/paste the
> flake8 section into those new files.
>
>  Or alternatively, should we be doing something different with our
> project layout as a workaround?  It's entirely possible that we should be
> configuring things differently overall...?
>
>
> I'm interested in this question for coverage.py.
>

This has been the bulk of the issue with solving this problem in pep8 as
well. Namely answering the question of which setup.cfg files to read, when
to stop (after reading the first with a particular section, or merge all
found?) and also whether per user or per project settings should take
precedence (additionally whether a config file specified with --config
should be the only configuration read or whether it should be merged with
per user and per project config, or be the only configuration used
(similarly for individual command line options).

I also agree with Ian's email asking for more information about your
specific use.


> Does setuptools or pip examine all of the setup.cfg files in your repo?  I
> wouldn't have guessed it would do that, I thought it would simply read the
> first one it finds.  I've never seen a project laid out like this, so it's
> all new to me.
>
> --Ned.
>
>
>  Thanks!
>
>  Willy
>
>
> _______________________________________________
> code-quality mailing listcode-quality at python.org <javascript:_e(%7B%7D,'cvml','code-quality at python.org');>https://mail.python.org/mailman/listinfo/code-quality
>
>
>
> _______________________________________________
> code-quality mailing list
> code-quality at python.org <javascript:;>
> https://mail.python.org/mailman/listinfo/code-quality
>
>

-- 

~ Ian Lee | IanLee1521 at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20150707/0909dc6e/attachment-0001.html>

From graffatcolmingov at gmail.com  Tue Jul  7 16:58:55 2015
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Tue, 7 Jul 2015 09:58:55 -0500
Subject: [code-quality] Searching multiple setup.cfg files for flake8
	section?
In-Reply-To: <CA+GyjMk6wqLrhbZF36-fNgPhiY9scv2+Mwrsgy_8D=UPFmqoaA@mail.gmail.com>
References: <CAB_0my92-VR0QVUyoKbTDH0zW9GP229YCX-HiA7S2tNuE+YM3A@mail.gmail.com>
 <559BD6D3.2070202@nedbatchelder.com>
 <CA+GyjMk6wqLrhbZF36-fNgPhiY9scv2+Mwrsgy_8D=UPFmqoaA@mail.gmail.com>
Message-ID: <CAN-Kwu25S0uws9fxugXK5So1q9wDJm+m6qD86Vc_nOBOPGSj=Q@mail.gmail.com>

On Tue, Jul 7, 2015 at 8:48 AM, Ian Lee <ianlee1521 at gmail.com> wrote:
>
>
> On Tuesday, July 7, 2015, Ned Batchelder <ned at nedbatchelder.com> wrote:
>>
>>
>>
>> On 7/7/15 5:05 AM, Willy Wu via code-quality wrote:
>>
>> Hey all,
>>
>> We use flake8 at Dropbox and are pretty happy with it.  But one small
>> issue is the way that flake8 configuration gets discovered.  We use the
>> per-project configuration via setup.cfg files which are checked into source
>> control, but flake8 only considers the first setup.cfg in the directory path
>> (regardless of whether it contains a flake8 section or not).  This forces us
>> to duplicate our [flake8] configuration across all of our various
>> setup.cfgs.
>>
>> Said in crude ascii form, this is a simplified version of our project
>> layout:
>> - repo root
>>   |- setup.cfg (default values)
>>   \- subservice #1
>>     \- setup.cfg (which has non-flake8 overrides for subservice #1)
>>   \- subservice #2
>>     \- setup.cfg (which similarly contains non-flake8 overrides for
>> subservice #2)
>>
>> But before I go off trying to patch flake8 code, is there relevant
>> background on why you guys don't try to discover the first setup.cfg with a
>> flake8 section, but instead just consider the first encountered setup.cfg?
>> We've seen teams accidentally break flake8 checking whenever they add
>> setup.cfg files for non-flake8 projects, and the "fix" is to copy/paste the
>> flake8 section into those new files.
>>
>> Or alternatively, should we be doing something different with our project
>> layout as a workaround?  It's entirely possible that we should be
>> configuring things differently overall...?
>>
>>
>> I'm interested in this question for coverage.py.
>
>
> This has been the bulk of the issue with solving this problem in pep8 as
> well. Namely answering the question of which setup.cfg files to read, when
> to stop (after reading the first with a particular section, or merge all
> found?) and also whether per user or per project settings should take
> precedence (additionally whether a config file specified with --config
> should be the only configuration read or whether it should be merged with
> per user and per project config, or be the only configuration used
> (similarly for individual command line options).
>
> I also agree with Ian's email asking for more information about your
> specific use.


So I'll answer Ned's question and this email at once. To Ned's
question of "Does setuptools or pip use all of the setup.cfg files it
can find?" (paraphrased) No. To the best of my knowledge, it uses the
one in the current directory only if it is there.

To expound on Ian's point, (not confusing at all that there are two
Ian's working on complimentary projects responding to the same thread)
Flake8 has a cap on pep8's version specifically because in 1.6.0 pep8
tried to solve some user confusion over this exact topic.

This broke config discovery and such with flake8 which caused a myriad
of bug reports which lead to the cap (as well as Flake8's strategy to
cap acceptable versions pre-emptively which lead to the thread with
the macports redistributor).

---

Now for something a little off-topic:

In my honest opinion, my mental model for per-user global configs
interacting with per-project configs is this:

- Read per-user global config
- Read per-project global config (notice the singular noun here)
- Merge the per-project into the per-user so that per-project takes
precedence. Reasoning: if I have, for my personal development
environment, a rule ignored that I don't like but I'm contributing to
a project that doesn't ignore it, pep8/flake8 should raise a warning
locally so that the CI jobs for that project do not have to tell me.

I think that's how most people expect that interaction to work. The
part where people's models differ is when having one or both of the
above AND specifying --config.

In this case, I think we should look at how we behave in similar situations:

If my user global config has one list of ignored errors and warnings,
and the project config has another, and I specify --ignore=... on the
command-line, then --ignore on the command-line wins and that's what
everyone expects. It doesn't follow very naturally, but as such, I'd
expect --config to be the only source of truth in that case. Why?
Because if you don't specify it explicitly, you want Flake8 to search
for other config files. When you specify it explicitly, you're
overriding that behaviour, just like when you specify --ignore
explicitly.

Here comes the fun and most convoluted part, let's say you have one or
both of (per-user global config, per-project global config) and (for
example) --ignore passed on the cli and --config passed on the CLI.
How does that win? Well, let's think of it like this:

--config wins over (per-user, per-project) configs (assuming we agree
with my perspective above)
--ignore wins over any configuration

So in that case, --ignore is respected over any configuration for
ignore in the config file passed in with --config, but the rest of it
is respected.

Everyone who is confused and has a headache now, raise your hand. *raises hand*

For what it's worth, I don't want this in a new pep8 release just yet.
And if it goes into one, it should (in my opinion) go into a pep8
2.0.0 release. This is a huge change that breaks the way pep8 works.
As such, other projects that either consume it or build upon it would
need time to properly prepare for this change. It should also be
documented extensively.

From wwu at dropbox.com  Tue Jul  7 19:56:27 2015
From: wwu at dropbox.com (Willy Wu)
Date: Tue, 7 Jul 2015 20:56:27 +0300
Subject: [code-quality] Searching multiple setup.cfg files for flake8
	section?
In-Reply-To: <CAN-Kwu3m5CqKxspYq6kXfG0tkMN247XDWAFsHdj0jJRL2SKErw@mail.gmail.com>
References: <CAB_0my92-VR0QVUyoKbTDH0zW9GP229YCX-HiA7S2tNuE+YM3A@mail.gmail.com>
 <CAN-Kwu3m5CqKxspYq6kXfG0tkMN247XDWAFsHdj0jJRL2SKErw@mail.gmail.com>
Message-ID: <CAB_0my9-F1Jcxf07CvRp3riPnAV76wznC35ERUW2vKT_+1+LKA@mail.gmail.com>

Hey Ian, answers inline:

On Tue, Jul 7, 2015 at 4:36 PM, Ian Cordasco <graffatcolmingov at gmail.com>
wrote:

> Hey Willy,
>
> Responses in-line:
>
> On Tue, Jul 7, 2015 at 4:05 AM, Willy Wu via code-quality
> <code-quality at python.org> wrote:
> > Hey all,
> >
> > We use flake8 at Dropbox and are pretty happy with it.  But one small
> issue
> > is the way that flake8 configuration gets discovered.  We use the
> > per-project configuration via setup.cfg files which are checked into
> source
> > control, but flake8 only considers the first setup.cfg in the directory
> path
> > (regardless of whether it contains a flake8 section or not).  This
> forces us
> > to duplicate our [flake8] configuration across all of our various
> > setup.cfgs.
> >
> > Said in crude ascii form, this is a simplified version of our project
> > layout:
> > - repo root
> >   |- setup.cfg (default values)
> >   \- subservice #1
> >     \- setup.cfg (which has non-flake8 overrides for subservice #1)
> >   \- subservice #2
> >     \- setup.cfg (which similarly contains non-flake8 overrides for
> > subservice #2)
>
> So... to be clear... you're putting all of your Flake8 configuration
> in "repo root/setup.cfg" but you have other setup.cfg files in
> sub-directories. How are you running Flake8 (or perhaps, where are you
> running Flake8) such that it picks up the setup.cfg files in the
> sub-directories? Am I correct in assuming that you are running it in
> that sub-directory? If not, can you give me the command-invocation
> that triggers this?


Flake8 actually gets run automatically by our lint tools during code diffs,
but ultimately the command that runs is:
$ flake8 /Users/name/{repo_root}/{path_to_file}

But the current working directory during the command that kicks off flake8
is {repo_root}, so I was hoping that flake8 would search anywhere between
the file's directory and the current working directory when trying to
discover various setup.cfgs to consider.

I think the source of our confusion is that we use py.test as our unit test
runner, and they only consider setup.cfg a match if it contains a [pytest]
section, so there's kind of a tension between how you guys and pytest are
resolving configs.

Let me know what else you need?


>
> > But before I go off trying to patch flake8 code, is there relevant
> > background on why you guys don't try to discover the first setup.cfg
> with a
> > flake8 section, but instead just consider the first encountered
> setup.cfg?
> > We've seen teams accidentally break flake8 checking whenever they add
> > setup.cfg files for non-flake8 projects, and the "fix" is to copy/paste
> the
> > flake8 section into those new files.
>
> The easiest answer is simplicity. Flake8 piggy-backs off of pep8 when
> it comes to finding config files and consuming them.
>
> I think there were a couple of people who had requested that we search
> through all parent directories until we could find no parent directory
> to search for configuration files, but I hope you can understand why
> that is not feasible in the slightest. I believe the two places we
> search by default are:
>
> - Current working directory
> - User's home directory
>
> Traversing sub-directories is a little bit less offensive, but still
> not optimal. There is the option to specify the config file
> explicitly, but that doesn't help our users that much unless they have
> some (honestly bizarre) setup where they're using something like
> autoenv + shell aliases for flake8, e.g., autoenv could pick up the
> right config file path when changing into a directory and `flake8` is
> aliased to `/path/to/flake8 --config=$ENVVARWITHPATH`
>
> That's definitely a bit over-the-top and not exactly user-friendly.
>
> > Or alternatively, should we be doing something different with our project
> > layout as a workaround?  It's entirely possible that we should be
> > configuring things differently overall...?
>
> That's tough to say honestly. I don't really feel like it's my place
> (or anyone else's place) to tell you how to structure your project
> layouts. That decision should be 100% in your court.
>
> I'd still like to know a bit more about exactly how you're invoking
> flake8 in this case. I don't have a clear enough understanding of the
> problem to really be very helpful.
>
> Cheers,
> Ian
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20150707/e3aefbbf/attachment.html>

From graffatcolmingov at gmail.com  Tue Jul  7 20:16:14 2015
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Tue, 7 Jul 2015 13:16:14 -0500
Subject: [code-quality] Searching multiple setup.cfg files for flake8
	section?
In-Reply-To: <CAB_0my9-F1Jcxf07CvRp3riPnAV76wznC35ERUW2vKT_+1+LKA@mail.gmail.com>
References: <CAB_0my92-VR0QVUyoKbTDH0zW9GP229YCX-HiA7S2tNuE+YM3A@mail.gmail.com>
 <CAN-Kwu3m5CqKxspYq6kXfG0tkMN247XDWAFsHdj0jJRL2SKErw@mail.gmail.com>
 <CAB_0my9-F1Jcxf07CvRp3riPnAV76wznC35ERUW2vKT_+1+LKA@mail.gmail.com>
Message-ID: <CAN-Kwu1Rsfj6N3bn-75p9DCJh8EC_JyHv9z=WfitHSRYhgHZmA@mail.gmail.com>

On Tue, Jul 7, 2015 at 12:56 PM, Willy Wu <wwu at dropbox.com> wrote:
> Hey Ian, answers inline:
>
> On Tue, Jul 7, 2015 at 4:36 PM, Ian Cordasco <graffatcolmingov at gmail.com>
> wrote:
>>
>> Hey Willy,
>>
>> Responses in-line:
>>
>> On Tue, Jul 7, 2015 at 4:05 AM, Willy Wu via code-quality
>> <code-quality at python.org> wrote:
>> > Hey all,
>> >
>> > We use flake8 at Dropbox and are pretty happy with it.  But one small
>> > issue
>> > is the way that flake8 configuration gets discovered.  We use the
>> > per-project configuration via setup.cfg files which are checked into
>> > source
>> > control, but flake8 only considers the first setup.cfg in the directory
>> > path
>> > (regardless of whether it contains a flake8 section or not).  This
>> > forces us
>> > to duplicate our [flake8] configuration across all of our various
>> > setup.cfgs.
>> >
>> > Said in crude ascii form, this is a simplified version of our project
>> > layout:
>> > - repo root
>> >   |- setup.cfg (default values)
>> >   \- subservice #1
>> >     \- setup.cfg (which has non-flake8 overrides for subservice #1)
>> >   \- subservice #2
>> >     \- setup.cfg (which similarly contains non-flake8 overrides for
>> > subservice #2)
>>
>> So... to be clear... you're putting all of your Flake8 configuration
>> in "repo root/setup.cfg" but you have other setup.cfg files in
>> sub-directories. How are you running Flake8 (or perhaps, where are you
>> running Flake8) such that it picks up the setup.cfg files in the
>> sub-directories? Am I correct in assuming that you are running it in
>> that sub-directory? If not, can you give me the command-invocation
>> that triggers this?
>
> Flake8 actually gets run automatically by our lint tools during code diffs,
> but ultimately the command that runs is:
> $ flake8 /Users/name/{repo_root}/{path_to_file}
>
> But the current working directory during the command that kicks off flake8
> is {repo_root}, so I was hoping that flake8 would search anywhere between
> the file's directory and the current working directory when trying to
> discover various setup.cfgs to consider.

Yeah I'm not sure that's desirable behaviour. We're then checking

$HOME (and variations of that based on operating system, and best
practice on that operating system)
$CWD
all paths between $CWD and $(dirname $file).

The problem is that only works for invocation of a single file. Given
your project layout, you could never just run

$ flake8 .

Flake8 would pick one configuration file over the other (probably
based on alphabetical order). As you can probably see from later
messages in this thread, our configuration discovery/selection is
already a mess and very very complicated (and looking as if it will
already get worse). We can't introduce that behaviour simply because
you're only running it against a single file. That would be worse than
a nightmare.

> I think the source of our confusion is that we use py.test as our unit test
> runner, and they only consider setup.cfg a match if it contains a [pytest]
> section, so there's kind of a tension between how you guys and pytest are
> resolving configs.

Except it's much more in line with how tox resolves configs. We can
find points and counter-points but saying X should behave like Y is
not a discussion worth having unless Y is a canonical implementation
of something (and there really is no canon in this kind of
configuration discovery/selection for Python tooling that looks in
several places).

> Let me know what else you need?

Which setup.cfg has the Flake8 config? Is it

- {repo_root}/setup.cfg
- {repo_root}/{subservice_x}/setup.cfg
- Something else?

Also, are each of the subservices actually their own packages or are
you simply using setup.cfg to hold config for tooling? If they are
separate packages there are different practices that could be followed
(separate repositories, etc.) but, again, that's not my place.

If your CI is so inflexible that you have to run flake8 from
repo_root, is it also so inflexible that you can't provide a path to
the setup.cfg?

Also, do the flake8 settings in each sub-service differ? If not, why
can't they all be kept in the single setup.cfg in repo_root? I would
imagine you'd want to enforce quality over an entire repository, not
selectively over subparts of it. Other large projects have
per-repository configs, not per-subproject configs. Wikimedia,
OpenStack, and a few others all do it per project. Some have
sub-packages that are in the repository and released from there, but
they all observe the same flake8 settings.

That said, I wonder if we stop searching for a config file when we've
found one of {setup.cfg, tox.ini, .flake8} even if it doesn't have a
flake8 section. That seems like a bug in pep8's config file discovery
and parsing. That still wouldn't help you though. =/

From wwu at dropbox.com  Tue Jul  7 21:11:25 2015
From: wwu at dropbox.com (Willy Wu)
Date: Tue, 7 Jul 2015 22:11:25 +0300
Subject: [code-quality] Searching multiple setup.cfg files for flake8
	section?
In-Reply-To: <CAN-Kwu1Rsfj6N3bn-75p9DCJh8EC_JyHv9z=WfitHSRYhgHZmA@mail.gmail.com>
References: <CAB_0my92-VR0QVUyoKbTDH0zW9GP229YCX-HiA7S2tNuE+YM3A@mail.gmail.com>
 <CAN-Kwu3m5CqKxspYq6kXfG0tkMN247XDWAFsHdj0jJRL2SKErw@mail.gmail.com>
 <CAB_0my9-F1Jcxf07CvRp3riPnAV76wznC35ERUW2vKT_+1+LKA@mail.gmail.com>
 <CAN-Kwu1Rsfj6N3bn-75p9DCJh8EC_JyHv9z=WfitHSRYhgHZmA@mail.gmail.com>
Message-ID: <CAB_0my_Pk1exau6mm-5eu7dSNJ=HWvu=w7t1RrQZt9W6Y0KU6w@mail.gmail.com>

On Tue, Jul 7, 2015 at 9:16 PM, Ian Cordasco <graffatcolmingov at gmail.com>
wrote:

> On Tue, Jul 7, 2015 at 12:56 PM, Willy Wu <wwu at dropbox.com> wrote:
> > Hey Ian, answers inline:
> >
> > On Tue, Jul 7, 2015 at 4:36 PM, Ian Cordasco <graffatcolmingov at gmail.com
> >
> > wrote:
> >>
> >> Hey Willy,
> >>
> >> Responses in-line:
> >>
> >> On Tue, Jul 7, 2015 at 4:05 AM, Willy Wu via code-quality
> >> <code-quality at python.org> wrote:
> >> > Hey all,
> >> >
> >> > We use flake8 at Dropbox and are pretty happy with it.  But one small
> >> > issue
> >> > is the way that flake8 configuration gets discovered.  We use the
> >> > per-project configuration via setup.cfg files which are checked into
> >> > source
> >> > control, but flake8 only considers the first setup.cfg in the
> directory
> >> > path
> >> > (regardless of whether it contains a flake8 section or not).  This
> >> > forces us
> >> > to duplicate our [flake8] configuration across all of our various
> >> > setup.cfgs.
> >> >
> >> > Said in crude ascii form, this is a simplified version of our project
> >> > layout:
> >> > - repo root
> >> >   |- setup.cfg (default values)
> >> >   \- subservice #1
> >> >     \- setup.cfg (which has non-flake8 overrides for subservice #1)
> >> >   \- subservice #2
> >> >     \- setup.cfg (which similarly contains non-flake8 overrides for
> >> > subservice #2)
> >>
> >> So... to be clear... you're putting all of your Flake8 configuration
> >> in "repo root/setup.cfg" but you have other setup.cfg files in
> >> sub-directories. How are you running Flake8 (or perhaps, where are you
> >> running Flake8) such that it picks up the setup.cfg files in the
> >> sub-directories? Am I correct in assuming that you are running it in
> >> that sub-directory? If not, can you give me the command-invocation
> >> that triggers this?
> >
> > Flake8 actually gets run automatically by our lint tools during code
> diffs,
> > but ultimately the command that runs is:
> > $ flake8 /Users/name/{repo_root}/{path_to_file}
> >
> > But the current working directory during the command that kicks off
> flake8
> > is {repo_root}, so I was hoping that flake8 would search anywhere between
> > the file's directory and the current working directory when trying to
> > discover various setup.cfgs to consider.
>
> Yeah I'm not sure that's desirable behaviour. We're then checking
>
> $HOME (and variations of that based on operating system, and best
> practice on that operating system)
> $CWD
> all paths between $CWD and $(dirname $file).
>
> The problem is that only works for invocation of a single file. Given
> your project layout, you could never just run
>
> $ flake8 .
>
> Flake8 would pick one configuration file over the other (probably
> based on alphabetical order). As you can probably see from later
> messages in this thread, our configuration discovery/selection is
> already a mess and very very complicated (and looking as if it will
> already get worse). We can't introduce that behaviour simply because
> you're only running it against a single file. That would be worse than
> a nightmare.
>

So our flake8 might be a simpler case, where our lint tool invokes flake8
on every file in the diff individually rather than a bunch of files at
once.  Am I right to guess that if you do "flake8 .", then flake8 picks a
single per-project config file to apply across all files?


>
> > I think the source of our confusion is that we use py.test as our unit
> test
> > runner, and they only consider setup.cfg a match if it contains a
> [pytest]
> > section, so there's kind of a tension between how you guys and pytest are
> > resolving configs.
>
> Except it's much more in line with how tox resolves configs. We can
> find points and counter-points but saying X should behave like Y is
> not a discussion worth having unless Y is a canonical implementation
> of something (and there really is no canon in this kind of
> configuration discovery/selection for Python tooling that looks in
> several places).
>
> > Let me know what else you need?
>
> Which setup.cfg has the Flake8 config? Is it
>
> - {repo_root}/setup.cfg
> - {repo_root}/{subservice_x}/setup.cfg
> - Something else?
>

{repo_root}/setup.cfg is originally the file w/ flake8 configs, and it's
where we were hoping to consolidate toward.


> Also, are each of the subservices actually their own packages or are
> you simply using setup.cfg to hold config for tooling? If they are
> separate packages there are different practices that could be followed
> (separate repositories, etc.) but, again, that's not my place.
>

Each of the subservices is a separate service that gets deployed to its own
cluster in prod.  We actually used to do separate repos for separate
services, but lately we wanted to do away with the overhead of lots of
little repos so we've been merging repos.  This issue with multiple
setup.cfgs is the fallout of the repo merging.  :)


> If your CI is so inflexible that you have to run flake8 from
> repo_root, is it also so inflexible that you can't provide a path to
> the setup.cfg?
>
> Also, do the flake8 settings in each sub-service differ? If not, why
> can't they all be kept in the single setup.cfg in repo_root? I would
> imagine you'd want to enforce quality over an entire repository, not
> selectively over subparts of it. Other large projects have
> per-repository configs, not per-subproject configs. Wikimedia,
> OpenStack, and a few others all do it per project. Some have
> sub-packages that are in the repository and released from there, but
> they all observe the same flake8 settings.
>

The flake8 settings in the subservices generally don't differ, which is why
we were hoping to consolidate their declarations everywhere.  It's just
that the various subservices are overriding service-specific settings to
different values in their respective setup.cfgs.

Did you mention that flake8 can support a --config parameter that overrides
per-project config discovery?  Since one source of the issue is that
multiple different projects are all using the same setup.cfg file, a
reasonable workaround might be to invoke flake8 with
--config={repo_root}/flake8.cfg which would ideally have the [flake8]
section that we're looking for.


> That said, I wonder if we stop searching for a config file when we've
> found one of {setup.cfg, tox.ini, .flake8} even if it doesn't have a
> flake8 section. That seems like a bug in pep8's config file discovery
> and parsing. That still wouldn't help you though. =/
>

Yeah, that's exactly the problem - that we stop searching, even if the
discovered file doesn't have a flake8 section.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20150707/a0cab23b/attachment.html>

From graffatcolmingov at gmail.com  Wed Jul  8 03:18:03 2015
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Tue, 7 Jul 2015 20:18:03 -0500
Subject: [code-quality] Searching multiple setup.cfg files for flake8
	section?
In-Reply-To: <CAB_0my_Pk1exau6mm-5eu7dSNJ=HWvu=w7t1RrQZt9W6Y0KU6w@mail.gmail.com>
References: <CAB_0my92-VR0QVUyoKbTDH0zW9GP229YCX-HiA7S2tNuE+YM3A@mail.gmail.com>
 <CAN-Kwu3m5CqKxspYq6kXfG0tkMN247XDWAFsHdj0jJRL2SKErw@mail.gmail.com>
 <CAB_0my9-F1Jcxf07CvRp3riPnAV76wznC35ERUW2vKT_+1+LKA@mail.gmail.com>
 <CAN-Kwu1Rsfj6N3bn-75p9DCJh8EC_JyHv9z=WfitHSRYhgHZmA@mail.gmail.com>
 <CAB_0my_Pk1exau6mm-5eu7dSNJ=HWvu=w7t1RrQZt9W6Y0KU6w@mail.gmail.com>
Message-ID: <CAN-Kwu0_KVF+Ted9-Q4ugbEdoPPO+Wrb2nhbYExbmodeDxk68Q@mail.gmail.com>

On Tue, Jul 7, 2015 at 2:11 PM, Willy Wu <wwu at dropbox.com> wrote:
> So our flake8 might be a simpler case, where our lint tool invokes flake8 on
> every file in the diff individually rather than a bunch of files at once.
> Am I right to guess that if you do "flake8 .", then flake8 picks a single
> per-project config file to apply across all files?

Like I said, we do not change how flake8 behaves whether its running
against a lot of files or whether it's running against a single file.

When running flake8, it looks in the current working directory for a
config file to apply to all files that it then discovers.

>> > I think the source of our confusion is that we use py.test as our unit
>> > test
>> > runner, and they only consider setup.cfg a match if it contains a
>> > [pytest]
>> > section, so there's kind of a tension between how you guys and pytest
>> > are
>> > resolving configs.
>>
>> Except it's much more in line with how tox resolves configs. We can
>> find points and counter-points but saying X should behave like Y is
>> not a discussion worth having unless Y is a canonical implementation
>> of something (and there really is no canon in this kind of
>> configuration discovery/selection for Python tooling that looks in
>> several places).
>>
>> > Let me know what else you need?
>>
>> Which setup.cfg has the Flake8 config? Is it
>>
>> - {repo_root}/setup.cfg
>> - {repo_root}/{subservice_x}/setup.cfg
>> - Something else?
>
>
> {repo_root}/setup.cfg is originally the file w/ flake8 configs, and it's
> where we were hoping to consolidate toward.

And you're saying that doesn't work right now? I'm confused. That
should Just Work? already. What won't work is shoving the config into
the setup.cfg files in the subservices.

>> Also, are each of the subservices actually their own packages or are
>> you simply using setup.cfg to hold config for tooling? If they are
>> separate packages there are different practices that could be followed
>> (separate repositories, etc.) but, again, that's not my place.
>
> Each of the subservices is a separate service that gets deployed to its own
> cluster in prod.  We actually used to do separate repos for separate
> services, but lately we wanted to do away with the overhead of lots of
> little repos so we've been merging repos.  This issue with multiple
> setup.cfgs is the fallout of the repo merging.  :)
>
>>
>> If your CI is so inflexible that you have to run flake8 from
>> repo_root, is it also so inflexible that you can't provide a path to
>> the setup.cfg?
>>
>> Also, do the flake8 settings in each sub-service differ? If not, why
>> can't they all be kept in the single setup.cfg in repo_root? I would
>> imagine you'd want to enforce quality over an entire repository, not
>> selectively over subparts of it. Other large projects have
>> per-repository configs, not per-subproject configs. Wikimedia,
>> OpenStack, and a few others all do it per project. Some have
>> sub-packages that are in the repository and released from there, but
>> they all observe the same flake8 settings.
>
> The flake8 settings in the subservices generally don't differ, which is why
> we were hoping to consolidate their declarations everywhere.  It's just that
> the various subservices are overriding service-specific settings to
> different values in their respective setup.cfgs.

So if you can put all of them into a config file in repo_root, this
should work. Did you try it and did it not work? (That config file can
be setup.cfg, tox.ini, or .flake8).

> Did you mention that flake8 can support a --config parameter that overrides
> per-project config discovery?  Since one source of the issue is that
> multiple different projects are all using the same setup.cfg file, a
> reasonable workaround might be to invoke flake8 with
> --config={repo_root}/flake8.cfg which would ideally have the [flake8]
> section that we're looking for.

I forget exactly how pep8 applies the settings when you use --config,
but yes that is an option you have.

>> That said, I wonder if we stop searching for a config file when we've
>> found one of {setup.cfg, tox.ini, .flake8} even if it doesn't have a
>> flake8 section. That seems like a bug in pep8's config file discovery
>> and parsing. That still wouldn't help you though. =/
>
> Yeah, that's exactly the problem - that we stop searching, even if the
> discovered file doesn't have a flake8 section.

I'm specifically talking about it being a bug if you have both
setup.cfg and tox.ini and one has a [flake8] section while the other
doesn't. If it finds/searches the one that doesn't first and then
quits, that's a bug.

From ben+python at benfinney.id.au  Wed Jul  8 03:47:49 2015
From: ben+python at benfinney.id.au (Ben Finney)
Date: Wed, 08 Jul 2015 11:47:49 +1000
Subject: [code-quality] Include content of other files in a ConfigFile input
	(was: Searching multiple setup.cfg files for flake8 section?)
References: <CAB_0my92-VR0QVUyoKbTDH0zW9GP229YCX-HiA7S2tNuE+YM3A@mail.gmail.com>
 <CAN-Kwu3m5CqKxspYq6kXfG0tkMN247XDWAFsHdj0jJRL2SKErw@mail.gmail.com>
 <CAB_0my9-F1Jcxf07CvRp3riPnAV76wznC35ERUW2vKT_+1+LKA@mail.gmail.com>
 <CAN-Kwu1Rsfj6N3bn-75p9DCJh8EC_JyHv9z=WfitHSRYhgHZmA@mail.gmail.com>
 <CAB_0my_Pk1exau6mm-5eu7dSNJ=HWvu=w7t1RrQZt9W6Y0KU6w@mail.gmail.com>
 <CAN-Kwu0_KVF+Ted9-Q4ugbEdoPPO+Wrb2nhbYExbmodeDxk68Q@mail.gmail.com>
Message-ID: <85r3ojxvwa.fsf_-_@benfinney.id.au>

Ian Cordasco <graffatcolmingov at gmail.com>
writes:

> On Tue, Jul 7, 2015 at 2:11 PM, Willy Wu <wwu at dropbox.com> wrote:
> > So our flake8 might be a simpler case, where our lint tool invokes
> > flake8 on every file in the diff individually rather than a bunch of
> > files at once. Am I right to guess that if you do "flake8 .", then
> > flake8 picks a single per-project config file to apply across all
> > files?
>
> Like I said, we do not change how flake8 behaves whether its running
> against a lot of files or whether it's running against a single file.

That is correct behaviour, in my view.

The handling of multiple files for configuration should not be special
to each tool that reads those files. Instead, it should be part of the
specification for the configuration format.

> When running flake8, it looks in the current working directory for a
> config file to apply to all files that it then discovers.

This discussion speaks, to me, of the need for an ?include these other
files? directive in the INI format.

The conventional syntax doesn't have such a thing. So different
extensions to the format have provided it in different (and probably
incompatible) ways.

As one example, Mercurial has a ?%include? directive
<URL:https://selenic.com/hg/help/config>.

Do we know of a convention that should be followed for including
multiple configuration files into a single stream of ConfigFile input?

-- 
 \                              ?Holy rising hemlines, Batman!? ?Robin |
  `\                                                                   |
_o__)                                                                  |
Ben Finney


From carl.crowder at gmail.com  Wed Jul  8 09:06:45 2015
From: carl.crowder at gmail.com (Carl Crowder)
Date: Wed, 8 Jul 2015 09:06:45 +0200
Subject: [code-quality] Include content of other files in a ConfigFile
 input (was: Searching multiple setup.cfg files for flake8 section?)
In-Reply-To: <85r3ojxvwa.fsf_-_@benfinney.id.au>
References: <CAB_0my92-VR0QVUyoKbTDH0zW9GP229YCX-HiA7S2tNuE+YM3A@mail.gmail.com>
 <CAN-Kwu3m5CqKxspYq6kXfG0tkMN247XDWAFsHdj0jJRL2SKErw@mail.gmail.com>
 <CAB_0my9-F1Jcxf07CvRp3riPnAV76wznC35ERUW2vKT_+1+LKA@mail.gmail.com>
 <CAN-Kwu1Rsfj6N3bn-75p9DCJh8EC_JyHv9z=WfitHSRYhgHZmA@mail.gmail.com>
 <CAB_0my_Pk1exau6mm-5eu7dSNJ=HWvu=w7t1RrQZt9W6Y0KU6w@mail.gmail.com>
 <CAN-Kwu0_KVF+Ted9-Q4ugbEdoPPO+Wrb2nhbYExbmodeDxk68Q@mail.gmail.com>
 <85r3ojxvwa.fsf_-_@benfinney.id.au>
Message-ID: <CAK2BMQtmGhoPL_-43jG+BayFTNVrGO3734f18XVqRi21p-C13w@mail.gmail.com>

I added this to prospector by allowing inheritance, so that your
custom directives could inherit from premade ones that come as part of
the library or 'project-wide' ones that you make but want to tweak on
a per-codebase level:

http://prospector.readthedocs.org/en/master/profiles.html#inheritance

I don't know how many people are using this in the wild though :)

On 8 July 2015 at 03:47, Ben Finney <ben+python at benfinney.id.au> wrote:
> Ian Cordasco <graffatcolmingov at gmail.com>
> writes:
>
>> On Tue, Jul 7, 2015 at 2:11 PM, Willy Wu <wwu at dropbox.com> wrote:
>> > So our flake8 might be a simpler case, where our lint tool invokes
>> > flake8 on every file in the diff individually rather than a bunch of
>> > files at once. Am I right to guess that if you do "flake8 .", then
>> > flake8 picks a single per-project config file to apply across all
>> > files?
>>
>> Like I said, we do not change how flake8 behaves whether its running
>> against a lot of files or whether it's running against a single file.
>
> That is correct behaviour, in my view.
>
> The handling of multiple files for configuration should not be special
> to each tool that reads those files. Instead, it should be part of the
> specification for the configuration format.
>
>> When running flake8, it looks in the current working directory for a
>> config file to apply to all files that it then discovers.
>
> This discussion speaks, to me, of the need for an ?include these other
> files? directive in the INI format.
>
> The conventional syntax doesn't have such a thing. So different
> extensions to the format have provided it in different (and probably
> incompatible) ways.
>
> As one example, Mercurial has a ?%include? directive
> <URL:https://selenic.com/hg/help/config>.
>
> Do we know of a convention that should be followed for including
> multiple configuration files into a single stream of ConfigFile input?
>
> --
>  \                              ?Holy rising hemlines, Batman!? ?Robin |
>   `\                                                                   |
> _o__)                                                                  |
> Ben Finney
>
> _______________________________________________
> code-quality mailing list
> code-quality at python.org
> https://mail.python.org/mailman/listinfo/code-quality

From wwu at dropbox.com  Wed Jul  8 17:15:55 2015
From: wwu at dropbox.com (Willy Wu)
Date: Wed, 8 Jul 2015 18:15:55 +0300
Subject: [code-quality] Searching multiple setup.cfg files for flake8
	section?
In-Reply-To: <CAN-Kwu0_KVF+Ted9-Q4ugbEdoPPO+Wrb2nhbYExbmodeDxk68Q@mail.gmail.com>
References: <CAB_0my92-VR0QVUyoKbTDH0zW9GP229YCX-HiA7S2tNuE+YM3A@mail.gmail.com>
 <CAN-Kwu3m5CqKxspYq6kXfG0tkMN247XDWAFsHdj0jJRL2SKErw@mail.gmail.com>
 <CAB_0my9-F1Jcxf07CvRp3riPnAV76wznC35ERUW2vKT_+1+LKA@mail.gmail.com>
 <CAN-Kwu1Rsfj6N3bn-75p9DCJh8EC_JyHv9z=WfitHSRYhgHZmA@mail.gmail.com>
 <CAB_0my_Pk1exau6mm-5eu7dSNJ=HWvu=w7t1RrQZt9W6Y0KU6w@mail.gmail.com>
 <CAN-Kwu0_KVF+Ted9-Q4ugbEdoPPO+Wrb2nhbYExbmodeDxk68Q@mail.gmail.com>
Message-ID: <CAB_0my-te=OKQ3dgtFWiTypBMdHs05kz-2ow2vPECnFx1K5n+A@mail.gmail.com>

On Wed, Jul 8, 2015 at 4:18 AM, Ian Cordasco <graffatcolmingov at gmail.com>
wrote:

> On Tue, Jul 7, 2015 at 2:11 PM, Willy Wu <wwu at dropbox.com> wrote:
> > So our flake8 might be a simpler case, where our lint tool invokes
> flake8 on
> > every file in the diff individually rather than a bunch of files at once.
> > Am I right to guess that if you do "flake8 .", then flake8 picks a single
> > per-project config file to apply across all files?
>
> Like I said, we do not change how flake8 behaves whether its running
> against a lot of files or whether it's running against a single file.
>
> When running flake8, it looks in the current working directory for a
> config file to apply to all files that it then discovers.
>
> >> > I think the source of our confusion is that we use py.test as our unit
> >> > test
> >> > runner, and they only consider setup.cfg a match if it contains a
> >> > [pytest]
> >> > section, so there's kind of a tension between how you guys and pytest
> >> > are
> >> > resolving configs.
> >>
> >> Except it's much more in line with how tox resolves configs. We can
> >> find points and counter-points but saying X should behave like Y is
> >> not a discussion worth having unless Y is a canonical implementation
> >> of something (and there really is no canon in this kind of
> >> configuration discovery/selection for Python tooling that looks in
> >> several places).
> >>
> >> > Let me know what else you need?
> >>
> >> Which setup.cfg has the Flake8 config? Is it
> >>
> >> - {repo_root}/setup.cfg
> >> - {repo_root}/{subservice_x}/setup.cfg
> >> - Something else?
> >
> >
> > {repo_root}/setup.cfg is originally the file w/ flake8 configs, and it's
> > where we were hoping to consolidate toward.
>
> And you're saying that doesn't work right now? I'm confused. That
> should Just Work? already. What won't work is shoving the config into
> the setup.cfg files in the subservices.
>

I made a gist with working shell code, it'll be easier for everyone if I
gave an example.  :)  It looks like the behavior for "flake8 ." vs "flake8
{path}" is different, which is probably why we're both kinda confused.

https://gist.github.com/willywu/f18211d1977a7189402b

Notice that if the root's setup.cfg is the only file w/ flake8 configs,
then `flake8 subproject1/src/myfile.py` doesn't use the root configs and
gives an error.  But when I put the flake8 configs inside
of subproject1/setup.cfg, then we get the behavior that I was hoping for.

But with that said, I think a totally viable workaround is to run `flake8
subproject1/src/myfile.py --config=setup.cfg` which also does the behavior
that I'm hoping for.

Soo...seems to be that if you invoke flake8 with a path to a file, flake8
will stop at the first discovered setup.cfg (my original issue).  But
that's atleast how pep8/tox works, even if it's not how py.test works.  You
guys should update the docs on
http://flake8.readthedocs.org/en/latest/config.html to mention the --config
param!  Because why would someone run `flake8 --help` instead of reading
the web docs...

>> Also, are each of the subservices actually their own packages or are
> >> you simply using setup.cfg to hold config for tooling? If they are
> >> separate packages there are different practices that could be followed
> >> (separate repositories, etc.) but, again, that's not my place.
> >
> > Each of the subservices is a separate service that gets deployed to its
> own
> > cluster in prod.  We actually used to do separate repos for separate
> > services, but lately we wanted to do away with the overhead of lots of
> > little repos so we've been merging repos.  This issue with multiple
> > setup.cfgs is the fallout of the repo merging.  :)
> >
> >>
> >> If your CI is so inflexible that you have to run flake8 from
> >> repo_root, is it also so inflexible that you can't provide a path to
> >> the setup.cfg?
> >>
> >> Also, do the flake8 settings in each sub-service differ? If not, why
> >> can't they all be kept in the single setup.cfg in repo_root? I would
> >> imagine you'd want to enforce quality over an entire repository, not
> >> selectively over subparts of it. Other large projects have
> >> per-repository configs, not per-subproject configs. Wikimedia,
> >> OpenStack, and a few others all do it per project. Some have
> >> sub-packages that are in the repository and released from there, but
> >> they all observe the same flake8 settings.
> >
> > The flake8 settings in the subservices generally don't differ, which is
> why
> > we were hoping to consolidate their declarations everywhere.  It's just
> that
> > the various subservices are overriding service-specific settings to
> > different values in their respective setup.cfgs.
>
> So if you can put all of them into a config file in repo_root, this
> should work. Did you try it and did it not work? (That config file can
> be setup.cfg, tox.ini, or .flake8).
>
> > Did you mention that flake8 can support a --config parameter that
> overrides
> > per-project config discovery?  Since one source of the issue is that
> > multiple different projects are all using the same setup.cfg file, a
> > reasonable workaround might be to invoke flake8 with
> > --config={repo_root}/flake8.cfg which would ideally have the [flake8]
> > section that we're looking for.
>
> I forget exactly how pep8 applies the settings when you use --config,
> but yes that is an option you have.
>
> >> That said, I wonder if we stop searching for a config file when we've
> >> found one of {setup.cfg, tox.ini, .flake8} even if it doesn't have a
> >> flake8 section. That seems like a bug in pep8's config file discovery
> >> and parsing. That still wouldn't help you though. =/
> >
> > Yeah, that's exactly the problem - that we stop searching, even if the
> > discovered file doesn't have a flake8 section.
>
> I'm specifically talking about it being a bug if you have both
> setup.cfg and tox.ini and one has a [flake8] section while the other
> doesn't. If it finds/searches the one that doesn't first and then
> quits, that's a bug.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20150708/84cfdacb/attachment.html>

From graffatcolmingov at gmail.com  Wed Jul  8 23:27:49 2015
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Wed, 8 Jul 2015 16:27:49 -0500
Subject: [code-quality] Searching multiple setup.cfg files for flake8
	section?
In-Reply-To: <CAB_0my-te=OKQ3dgtFWiTypBMdHs05kz-2ow2vPECnFx1K5n+A@mail.gmail.com>
References: <CAB_0my92-VR0QVUyoKbTDH0zW9GP229YCX-HiA7S2tNuE+YM3A@mail.gmail.com>
 <CAN-Kwu3m5CqKxspYq6kXfG0tkMN247XDWAFsHdj0jJRL2SKErw@mail.gmail.com>
 <CAB_0my9-F1Jcxf07CvRp3riPnAV76wznC35ERUW2vKT_+1+LKA@mail.gmail.com>
 <CAN-Kwu1Rsfj6N3bn-75p9DCJh8EC_JyHv9z=WfitHSRYhgHZmA@mail.gmail.com>
 <CAB_0my_Pk1exau6mm-5eu7dSNJ=HWvu=w7t1RrQZt9W6Y0KU6w@mail.gmail.com>
 <CAN-Kwu0_KVF+Ted9-Q4ugbEdoPPO+Wrb2nhbYExbmodeDxk68Q@mail.gmail.com>
 <CAB_0my-te=OKQ3dgtFWiTypBMdHs05kz-2ow2vPECnFx1K5n+A@mail.gmail.com>
Message-ID: <CAN-Kwu3+pZ=e9u-oij9URdC+5jKpizZx=8mBTKSUkpNNKuS2Zg@mail.gmail.com>

On Wed, Jul 8, 2015 at 10:15 AM, Willy Wu <wwu at dropbox.com> wrote:
>
>
> On Wed, Jul 8, 2015 at 4:18 AM, Ian Cordasco <graffatcolmingov at gmail.com>
> wrote:
>>
>> On Tue, Jul 7, 2015 at 2:11 PM, Willy Wu <wwu at dropbox.com> wrote:
>> > So our flake8 might be a simpler case, where our lint tool invokes
>> > flake8 on
>> > every file in the diff individually rather than a bunch of files at
>> > once.
>> > Am I right to guess that if you do "flake8 .", then flake8 picks a
>> > single
>> > per-project config file to apply across all files?
>>
>> Like I said, we do not change how flake8 behaves whether its running
>> against a lot of files or whether it's running against a single file.
>>
>> When running flake8, it looks in the current working directory for a
>> config file to apply to all files that it then discovers.
>>
>> >> > I think the source of our confusion is that we use py.test as our
>> >> > unit
>> >> > test
>> >> > runner, and they only consider setup.cfg a match if it contains a
>> >> > [pytest]
>> >> > section, so there's kind of a tension between how you guys and pytest
>> >> > are
>> >> > resolving configs.
>> >>
>> >> Except it's much more in line with how tox resolves configs. We can
>> >> find points and counter-points but saying X should behave like Y is
>> >> not a discussion worth having unless Y is a canonical implementation
>> >> of something (and there really is no canon in this kind of
>> >> configuration discovery/selection for Python tooling that looks in
>> >> several places).
>> >>
>> >> > Let me know what else you need?
>> >>
>> >> Which setup.cfg has the Flake8 config? Is it
>> >>
>> >> - {repo_root}/setup.cfg
>> >> - {repo_root}/{subservice_x}/setup.cfg
>> >> - Something else?
>> >
>> >
>> > {repo_root}/setup.cfg is originally the file w/ flake8 configs, and it's
>> > where we were hoping to consolidate toward.
>>
>> And you're saying that doesn't work right now? I'm confused. That
>> should Just Work? already. What won't work is shoving the config into
>> the setup.cfg files in the subservices.
>
>
> I made a gist with working shell code, it'll be easier for everyone if I
> gave an example.  :)  It looks like the behavior for "flake8 ." vs "flake8
> {path}" is different, which is probably why we're both kinda confused.
>
> https://gist.github.com/willywu/f18211d1977a7189402b
>
> Notice that if the root's setup.cfg is the only file w/ flake8 configs, then
> `flake8 subproject1/src/myfile.py` doesn't use the root configs and gives an
> error.  But when I put the flake8 configs inside of subproject1/setup.cfg,
> then we get the behavior that I was hoping for.
>
> But with that said, I think a totally viable workaround is to run `flake8
> subproject1/src/myfile.py --config=setup.cfg` which also does the behavior
> that I'm hoping for.
>
> Soo...seems to be that if you invoke flake8 with a path to a file, flake8
> will stop at the first discovered setup.cfg (my original issue).  But that's
> atleast how pep8/tox works, even if it's not how py.test works.  You guys
> should update the docs on
> http://flake8.readthedocs.org/en/latest/config.html to mention the --config
> param!  Because why would someone run `flake8 --help` instead of reading the
> web docs...

This is awesome. I was about to start trying permutations of this to
really grok what you were encountering/try to reproduce it. This gives
me a clear set of steps.

So flake8 takes full advantage of pep8's config handling. (Which you
can skim here: https://github.com/jcrocholl/pep8/blob/a6e423cd4e972b19dc1d2c3d6db6269c2c445151/pep8.py#L1985)

If I had read things closer I would have seen
https://github.com/jcrocholl/pep8/blob/a6e423cd4e972b19dc1d2c3d6db6269c2c445151/pep8.py#L1998
but I was confused and not reading the code clearly. Sorry about that,
this confusion is probably my fault.

At this point, I'm not sure we can fix that, although I agree that it
isn't ideal. We should probably attempt to read the config file into a
separate config parser, check if it has a relevant section and /then/
read it into our actual config object and break. That said, the config
parsing part of pep8 seems to be very fragile (as is evidenced by pep8
1.6.0, 1.6.1, and 1.6.2). If Ian (Lee) is up for it, I'd love to work
with him to carefully fix this and other config discovery/parsing bugs
that we know about in pep8 for a pep8 2.0 release. I'd also like to
see at least one more 1.x release if not one more 1.6.x release that
flake8 users can consume with the reverted changes from the current
1.6.x branch.

I'll file a bug about `--config` not being covered in our web docs,
but I think I might just remove that stuff altogether. It's too easy
for it to get out of date and extensions can add extra flags that will
effectively make that output wrong for users who do not check "flake8
--help".

--
Ian

From ianlee1521 at gmail.com  Thu Jul  9 06:35:44 2015
From: ianlee1521 at gmail.com (Ian Lee)
Date: Wed, 8 Jul 2015 21:35:44 -0700
Subject: [code-quality] Searching multiple setup.cfg files for flake8
	section?
In-Reply-To: <CAN-Kwu3+pZ=e9u-oij9URdC+5jKpizZx=8mBTKSUkpNNKuS2Zg@mail.gmail.com>
References: <CAB_0my92-VR0QVUyoKbTDH0zW9GP229YCX-HiA7S2tNuE+YM3A@mail.gmail.com>
 <CAN-Kwu3m5CqKxspYq6kXfG0tkMN247XDWAFsHdj0jJRL2SKErw@mail.gmail.com>
 <CAB_0my9-F1Jcxf07CvRp3riPnAV76wznC35ERUW2vKT_+1+LKA@mail.gmail.com>
 <CAN-Kwu1Rsfj6N3bn-75p9DCJh8EC_JyHv9z=WfitHSRYhgHZmA@mail.gmail.com>
 <CAB_0my_Pk1exau6mm-5eu7dSNJ=HWvu=w7t1RrQZt9W6Y0KU6w@mail.gmail.com>
 <CAN-Kwu0_KVF+Ted9-Q4ugbEdoPPO+Wrb2nhbYExbmodeDxk68Q@mail.gmail.com>
 <CAB_0my-te=OKQ3dgtFWiTypBMdHs05kz-2ow2vPECnFx1K5n+A@mail.gmail.com>
 <CAN-Kwu3+pZ=e9u-oij9URdC+5jKpizZx=8mBTKSUkpNNKuS2Zg@mail.gmail.com>
Message-ID: <CA+GyjM=fg+TE2TyinRgTHoQQa1sjuSFtnPJYkZbVB9g_Rq6=PA@mail.gmail.com>

On Wed, Jul 8, 2015 at 2:27 PM, Ian Cordasco <graffatcolmingov at gmail.com>
wrote:

> On Wed, Jul 8, 2015 at 10:15 AM, Willy Wu <wwu at dropbox.com> wrote:
> >
> >
> > On Wed, Jul 8, 2015 at 4:18 AM, Ian Cordasco <graffatcolmingov at gmail.com
> >
> > wrote:
> >>
> >> On Tue, Jul 7, 2015 at 2:11 PM, Willy Wu <wwu at dropbox.com> wrote:
> >> > So our flake8 might be a simpler case, where our lint tool invokes
> >> > flake8 on
> >> > every file in the diff individually rather than a bunch of files at
> >> > once.
> >> > Am I right to guess that if you do "flake8 .", then flake8 picks a
> >> > single
> >> > per-project config file to apply across all files?
> >>
> >> Like I said, we do not change how flake8 behaves whether its running
> >> against a lot of files or whether it's running against a single file.
> >>
> >> When running flake8, it looks in the current working directory for a
> >> config file to apply to all files that it then discovers.
> >>
> >> >> > I think the source of our confusion is that we use py.test as our
> >> >> > unit
> >> >> > test
> >> >> > runner, and they only consider setup.cfg a match if it contains a
> >> >> > [pytest]
> >> >> > section, so there's kind of a tension between how you guys and
> pytest
> >> >> > are
> >> >> > resolving configs.
> >> >>
> >> >> Except it's much more in line with how tox resolves configs. We can
> >> >> find points and counter-points but saying X should behave like Y is
> >> >> not a discussion worth having unless Y is a canonical implementation
> >> >> of something (and there really is no canon in this kind of
> >> >> configuration discovery/selection for Python tooling that looks in
> >> >> several places).
> >> >>
> >> >> > Let me know what else you need?
> >> >>
> >> >> Which setup.cfg has the Flake8 config? Is it
> >> >>
> >> >> - {repo_root}/setup.cfg
> >> >> - {repo_root}/{subservice_x}/setup.cfg
> >> >> - Something else?
> >> >
> >> >
> >> > {repo_root}/setup.cfg is originally the file w/ flake8 configs, and
> it's
> >> > where we were hoping to consolidate toward.
> >>
> >> And you're saying that doesn't work right now? I'm confused. That
> >> should Just Work? already. What won't work is shoving the config into
> >> the setup.cfg files in the subservices.
> >
> >
> > I made a gist with working shell code, it'll be easier for everyone if I
> > gave an example.  :)  It looks like the behavior for "flake8 ." vs
> "flake8
> > {path}" is different, which is probably why we're both kinda confused.
> >
> > https://gist.github.com/willywu/f18211d1977a7189402b
> >
> > Notice that if the root's setup.cfg is the only file w/ flake8 configs,
> then
> > `flake8 subproject1/src/myfile.py` doesn't use the root configs and
> gives an
> > error.  But when I put the flake8 configs inside of
> subproject1/setup.cfg,
> > then we get the behavior that I was hoping for.
> >
> > But with that said, I think a totally viable workaround is to run `flake8
> > subproject1/src/myfile.py --config=setup.cfg` which also does the
> behavior
> > that I'm hoping for.
> >
> > Soo...seems to be that if you invoke flake8 with a path to a file, flake8
> > will stop at the first discovered setup.cfg (my original issue).  But
> that's
> > atleast how pep8/tox works, even if it's not how py.test works.  You guys
> > should update the docs on
> > http://flake8.readthedocs.org/en/latest/config.html to mention the
> --config
> > param!  Because why would someone run `flake8 --help` instead of reading
> the
> > web docs...
>
> This is awesome. I was about to start trying permutations of this to
> really grok what you were encountering/try to reproduce it. This gives
> me a clear set of steps.
>
> So flake8 takes full advantage of pep8's config handling. (Which you
> can skim here:
> https://github.com/jcrocholl/pep8/blob/a6e423cd4e972b19dc1d2c3d6db6269c2c445151/pep8.py#L1985
> )
>
> If I had read things closer I would have seen
>
> https://github.com/jcrocholl/pep8/blob/a6e423cd4e972b19dc1d2c3d6db6269c2c445151/pep8.py#L1998
> but I was confused and not reading the code clearly. Sorry about that,
> this confusion is probably my fault.
>

Another consideration that was proposed once was that instead of breaking
when a config was found here in line 1998, that all config files (with
appropriate flake8 / pep8 sections) would get merged together.

Another gotcha that I encountered while trying to sort this out once before
was that there are three different names that the project config file can
have: 'setup.cfg', 'tox.ini', or '.pep8' (see:
https://github.com/jcrocholl/pep8/blob/a6e423cd4e972b19dc1d2c3d6db6269c2c445151/pep8.py#L80).
And I believe flake8 adds a fourth, '.flake8'. It should also be noted that
the '.pep8' has been deprecated for a year plus, and we could probably
remove it as part of the 2.0 version.

It seems like there ought to be just one way to handle the reading and
usage of config files as a more generic level in Python that could then be
used by pep8, flake8, tox, setuptools, py.test, etc, etc.  (One man's
opinion). The drawback of course is that then adds extra dependencies.


>
> At this point, I'm not sure we can fix that, although I agree that it
> isn't ideal. We should probably attempt to read the config file into a
> separate config parser, check if it has a relevant section and /then/
> read it into our actual config object and break. That said, the config
> parsing part of pep8 seems to be very fragile (as is evidenced by pep8
> 1.6.0, 1.6.1, and 1.6.2). If Ian (Lee) is up for it, I'd love to work
> with him to carefully fix this and other config discovery/parsing bugs
> that we know about in pep8 for a pep8 2.0 release. I'd also like to
> see at least one more 1.x release if not one more 1.6.x release that
> flake8 users can consume with the reverted changes from the current
> 1.6.x branch.
>

Absolutely could use some help on this. I am definitely on board with
getting this fixed and released in the 1.6.x line. If you wanted to start a
branch for this I'll try to make it a priority of mine to get it merged.
Unfortunately work travel has kept me away from contributing here the past
several months, and I apologize for that absence.


> I'll file a bug about `--config` not being covered in our web docs,
> but I think I might just remove that stuff altogether. It's too easy
> for it to get out of date and extensions can add extra flags that will
> effectively make that output wrong for users who do not check "flake8
> --help".
>
> --
> Ian
> _______________________________________________
> code-quality mailing list
> code-quality at python.org
> https://mail.python.org/mailman/listinfo/code-quality
>



-- 

~ Ian Lee | IanLee1521 at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20150708/39fe5c3f/attachment.html>

From juanleon.lahoz at gmail.com  Thu Jul  9 11:32:00 2015
From: juanleon.lahoz at gmail.com (JuanLeon Lahoz)
Date: Thu, 9 Jul 2015 11:32:00 +0200
Subject: [code-quality] Detecting circular package dependencies with pylint
	(or similar tool)
Message-ID: <CANYPdHB-uaRiG+URKn4ftYivpZTOjvL7EbyM454PFHamP09YFQ@mail.gmail.com>

Hi,

Is there any way to make pylint report circular package dependencies with
an exit code != 0?

Assume code has packages A and B.  A.a1 imports B.b1, and B.b2 imports
A.a2.  While this may work, imho this is a bad smell in design that I want
to detect as early as possible (same as any other bad smells that pylint
detects).

I know that pylint can write import dependency graphs, but I could not find
a trivial way to use them a a check (trivial == no coding required).

Any option that I am missing?

Thanks
juanleon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20150709/0923ae42/attachment.html>

From graffatcolmingov at gmail.com  Sat Jul 11 22:19:26 2015
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Sat, 11 Jul 2015 15:19:26 -0500
Subject: [code-quality] PEP8's repository has moved to the PyCQA
	organization on GitHub
Message-ID: <CAN-Kwu2cw_1y8fYDDSZ7JQnmCdqaXapd2k1jqvJU7gSmKb390w@mail.gmail.com>

Hey everyone,

I just wanted to let all of you know that late last night Johann
Rocholl, creator of pep8, has moved the project to the PyCQA
organization on GitHub. The new URL for the project is
https://github.com/PyCQA/pep8. All old issues and pull requests are
preserved as part of the process.

For more information, please read
http://www.coglib.com/~icordasc/blog/2015/07/moving-pep8-to-the-pycqa.html

Cheers,
Ian Cordasco
Core developer of Flake8, requests, and a few other projects

From pcmanticore at gmail.com  Thu Jul 16 12:47:18 2015
From: pcmanticore at gmail.com (Claudiu Popa)
Date: Thu, 16 Jul 2015 13:47:18 +0300
Subject: [code-quality] Detecting circular package dependencies with
 pylint (or similar tool)
In-Reply-To: <CANYPdHB-uaRiG+URKn4ftYivpZTOjvL7EbyM454PFHamP09YFQ@mail.gmail.com>
References: <CANYPdHB-uaRiG+URKn4ftYivpZTOjvL7EbyM454PFHamP09YFQ@mail.gmail.com>
Message-ID: <CAMy=CLqwwDwB0HusiggMLMuRd+Gcj8+AEzv_dFOobY8Td-aNPw@mail.gmail.com>

On Thu, Jul 9, 2015 at 12:32 PM, JuanLeon Lahoz
<juanleon.lahoz at gmail.com> wrote:
> Hi,
>
> Is there any way to make pylint report circular package dependencies with an
> exit code != 0?
>
> Assume code has packages A and B.  A.a1 imports B.b1, and B.b2 imports A.a2.
> While this may work, imho this is a bad smell in design that I want to
> detect as early as possible (same as any other bad smells that pylint
> detects).
>
> I know that pylint can write import dependency graphs, but I could not find
> a trivial way to use them a a check (trivial == no coding required).
>
> Any option that I am missing?
>
> Thanks
> juanleon
>
> _______________________________________________
> code-quality mailing list
> code-quality at python.org
> https://mail.python.org/mailman/listinfo/code-quality
>


Hi Juan,

I'm sorry for the delayed response. Currently it's not possible,
since the circular import detection is pretty simple and it works only
for imports between the analyzed files, so it doesn't go for each other
import in your file and tries to see if that import has circular dependency
on other file. But improvements are welcomed and you could create
at least a bitbucket issue in order to track this improvement.

Thanks,
Claudiu

From stefan.hocko at gmail.com  Sat Jul 25 10:57:45 2015
From: stefan.hocko at gmail.com (=?UTF-8?Q?=C5=A0tefan_Hocko?=)
Date: Sat, 25 Jul 2015 10:57:45 +0200
Subject: [code-quality] .pylint.d has 170MB, how to clean?
Message-ID: <CAOE234ZWQWB6sQML7YMpb_2OXJXt2NK3Y1WucRUy5skOnTiHVw@mail.gmail.com>

Hello everybody,

I found out that my .pylint.d directory has grown to 170MB, as I am working
on a 1GB workspace this is a lot. I haven't found a way to clean it in the
FAQ or manual. Can I just delete the contents without damaging PyLint?

Thanks for answers,

Best Regards,
Stefan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20150725/b7e450f3/attachment.html>

From pcmanticore at gmail.com  Sat Jul 25 12:46:16 2015
From: pcmanticore at gmail.com (Claudiu Popa)
Date: Sat, 25 Jul 2015 13:46:16 +0300
Subject: [code-quality] .pylint.d has 170MB, how to clean?
In-Reply-To: <CAOE234ZWQWB6sQML7YMpb_2OXJXt2NK3Y1WucRUy5skOnTiHVw@mail.gmail.com>
References: <CAOE234ZWQWB6sQML7YMpb_2OXJXt2NK3Y1WucRUy5skOnTiHVw@mail.gmail.com>
Message-ID: <CAMy=CLr9tMD3xgVz2sPvSL+djg4pviE5jpm6dQJWf5-PM2gh8w@mail.gmail.com>

On Sat, Jul 25, 2015 at 11:57 AM, ?tefan Hocko <stefan.hocko at gmail.com> wrote:
> Hello everybody,
>
> I found out that my .pylint.d directory has grown to 170MB, as I am working
> on a 1GB workspace this is a lot. I haven't found a way to clean it in the
> FAQ or manual. Can I just delete the contents without damaging PyLint?
>
> Thanks for answers,
>
> Best Regards,
> Stefan
>
> _______________________________________________
> code-quality mailing list
> code-quality at python.org
> https://mail.python.org/mailman/listinfo/code-quality
>

Yeah, you could just delete it and pylint will work normally.
That folder is used only for caching stats.

/Claudiu

From rshepard at appl-ecosys.com  Sat Jul 25 18:36:15 2015
From: rshepard at appl-ecosys.com (Rich Shepard)
Date: Sat, 25 Jul 2015 09:36:15 -0700 (PDT)
Subject: [code-quality] Questions: Flake8-2.4.1
Message-ID: <alpine.LNX.2.11.1507250935400.18533@localhost>

   Just installed flake8-2.4.1. Checking a file generates this error:

pwschemasl.py:698:39: E225 missing whitespace around operator

on this line:

     rpt_filename=Column(BLOB, nullable=False)

where position 39 is 'F'. So, I add the space:

     rpt_filename=Column(BLOB, nullable= False)

and now flake8 shows this error:

pwschemasl.py:698:40: E251 unexpected spaces around keyword / parameter
equals

   This is an infinite loop that affects many lines of code in the file. The
lines throw a PEP8 compliance error both with and without a space following
the '=' sign.

   Is there something I've done incorrectly that I don't see?

Puzzled,

Rich

From graffatcolmingov at gmail.com  Sat Jul 25 22:52:15 2015
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Sat, 25 Jul 2015 15:52:15 -0500
Subject: [code-quality] Questions: Flake8-2.4.1
In-Reply-To: <alpine.LNX.2.11.1507250935400.18533@localhost>
References: <alpine.LNX.2.11.1507250935400.18533@localhost>
Message-ID: <CAN-Kwu3n6wtyq15GvsDc99EvK_hPqogC5bFBz+yazC6W4Ty7gA@mail.gmail.com>

On Sat, Jul 25, 2015 at 11:36 AM, Rich Shepard <rshepard at appl-ecosys.com> wrote:
>   Just installed flake8-2.4.1. Checking a file generates this error:
>
> pwschemasl.py:698:39: E225 missing whitespace around operator
>
> on this line:
>
>     rpt_filename=Column(BLOB, nullable=False)
>
> where position 39 is 'F'. So, I add the space:
>
>     rpt_filename=Column(BLOB, nullable= False)
>
> and now flake8 shows this error:
>
> pwschemasl.py:698:40: E251 unexpected spaces around keyword / parameter
> equals
>
>   This is an infinite loop that affects many lines of code in the file. The
> lines throw a PEP8 compliance error both with and without a space following
> the '=' sign.
>
>   Is there something I've done incorrectly that I don't see?
>
> Puzzled,
>
> Rich


Hey Rich,

Could you provide a bit more context for the code? A single line and
an error message aren't very useful for me to reproduce with.

Further, could you provide the output from

$ flake8 --version

Thanks!
Ian

From rshepard at appl-ecosys.com  Sun Jul 26 02:00:33 2015
From: rshepard at appl-ecosys.com (Rich Shepard)
Date: Sat, 25 Jul 2015 17:00:33 -0700 (PDT)
Subject: [code-quality] Questions: Flake8-2.4.1
In-Reply-To: <CAN-Kwu3n6wtyq15GvsDc99EvK_hPqogC5bFBz+yazC6W4Ty7gA@mail.gmail.com>
References: <alpine.LNX.2.11.1507250935400.18533@localhost>
 <CAN-Kwu3n6wtyq15GvsDc99EvK_hPqogC5bFBz+yazC6W4Ty7gA@mail.gmail.com>
Message-ID: <alpine.LNX.2.11.1507251646440.30374@localhost>

On Sat, 25 Jul 2015, Ian Cordasco wrote:

> Could you provide a bit more context for the code? A single line and an
> error message aren't very useful for me to reproduce with.

Ian,

   This is interesting. I shut down emacs for a couple of hours. Just invoked
it and loaded the source file. Running flake8 on it is now consistent. When
before it complained about white space, then the lack of it, now it
complains about the lack of white space and likes when I add it. Strange.

   Sorry to interupt your weekend. Seems like a flake8 fluke that a rest
resolved.

> Further, could you provide the output from
> $ flake8 --version

flake8 --version
2.4.1 (pep8: 1.5.7, pyflakes: 0.8.1, mccabe: 0.3.1) CPython 2.7.5 on Linux

Regards,

Rich

From graffatcolmingov at gmail.com  Sun Jul 26 02:46:26 2015
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Sat, 25 Jul 2015 19:46:26 -0500
Subject: [code-quality] Questions: Flake8-2.4.1
In-Reply-To: <alpine.LNX.2.11.1507251646440.30374@localhost>
References: <alpine.LNX.2.11.1507250935400.18533@localhost>
 <CAN-Kwu3n6wtyq15GvsDc99EvK_hPqogC5bFBz+yazC6W4Ty7gA@mail.gmail.com>
 <alpine.LNX.2.11.1507251646440.30374@localhost>
Message-ID: <CAN-Kwu36-Wfg+9yUhTSCtfJRr6Msv0uDd-zL8GyOusudjxcWYg@mail.gmail.com>

On Sat, Jul 25, 2015 at 7:00 PM, Rich Shepard <rshepard at appl-ecosys.com> wrote:
> On Sat, 25 Jul 2015, Ian Cordasco wrote:
>
>> Could you provide a bit more context for the code? A single line and an
>> error message aren't very useful for me to reproduce with.
>
>
> Ian,
>
>   This is interesting. I shut down emacs for a couple of hours. Just invoked
> it and loaded the source file. Running flake8 on it is now consistent. When
> before it complained about white space, then the lack of it, now it
> complains about the lack of white space and likes when I add it. Strange.

I'm not sure what you mean but you seem content with the resolution so
I won't press my luck.

>   Sorry to interupt your weekend. Seems like a flake8 fluke that a rest
> resolved.

No worries. You didn't cause me any trouble. I check my email every
day of the week. :-D

>> Further, could you provide the output from
>> $ flake8 --version
>
>
> flake8 --version
> 2.4.1 (pep8: 1.5.7, pyflakes: 0.8.1, mccabe: 0.3.1) CPython 2.7.5 on Linux
>
> Regards,
>
>
> Rich
> _______________________________________________
> code-quality mailing list
> code-quality at python.org
> https://mail.python.org/mailman/listinfo/code-quality

From songofacandy at gmail.com  Sun Jul 26 07:29:56 2015
From: songofacandy at gmail.com (INADA Naoki)
Date: Sun, 26 Jul 2015 14:29:56 +0900
Subject: [code-quality] Questions: Flake8-2.4.1
In-Reply-To: <alpine.LNX.2.11.1507250935400.18533@localhost>
References: <alpine.LNX.2.11.1507250935400.18533@localhost>
Message-ID: <CAEfz+TxOyNTPAQJWYR0j-Jz_DDpHehV9Nztez=XyCF-jz_JsgQ@mail.gmail.com>

assignment `=` should be have spaces.
keyword argument `=` should not have spaces (except case of function
annotation in definition).

So

On Sun, Jul 26, 2015 at 1:36 AM, Rich Shepard <rshepard at appl-ecosys.com>
wrote:

>   Just installed flake8-2.4.1. Checking a file generates this error:
>
> pwschemasl.py:698:39: E225 missing whitespace around operator
>
> on this line:
>
>     rpt_filename=Column(BLOB, nullable=False)
>
> where position 39 is 'F'. So, I add the space:
>
>     rpt_filename=Column(BLOB, nullable= False)
>
>
?This should be:

rpt_filename = Column(BLOB, nullable=False)

?



> and now flake8 shows this error:
>
> pwschemasl.py:698:40: E251 unexpected spaces around keyword / parameter
> equals
>
>   This is an infinite loop that affects many lines of code in the file. The
> lines throw a PEP8 compliance error both with and without a space following
> the '=' sign.
>
>   Is there something I've done incorrectly that I don't see?
>
> Puzzled,
>
> Rich
> _______________________________________________
> code-quality mailing list
> code-quality at python.org
> https://mail.python.org/mailman/listinfo/code-quality
>



-- 
INADA Naoki  <songofacandy at gmail.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20150726/14923bf2/attachment-0001.html>

From rshepard at appl-ecosys.com  Sun Jul 26 17:58:22 2015
From: rshepard at appl-ecosys.com (Rich Shepard)
Date: Sun, 26 Jul 2015 08:58:22 -0700 (PDT)
Subject: [code-quality] Not Seeing PEP8 Formatting Errors
Message-ID: <alpine.LNX.2.11.1507260849570.4769@localhost>

   I've numbered the lines in this code fragment:

83   category = Column(Unicode(16), value = 'Environmental', nullable = False,
84                     CheckConstraint(
84       "category IN('Air', 'Environmental', 'Health', 'Safety', 'Occupancy',
86       'Water'"))

    flake8 has these errors:

84:39: E201 whitespace after '('
86:9: E128 continuation line under-indented for visual indent

   I've read PEP8 and found several fora threads related to E128 and E201 but
no matter how I reformat that statement I cannot remove these (or other)
errors when flake8 runs on this module.

   When I continue line 84 the '(' and emacs breaks it near column 78 I get a
different error about closing brackets not being correctly aligned. But, no
matter where I place the final bracket it shows an error.

   This module has many error conditions simular to the above and I want to
understand how these long lines should be formatted so flake8 no longer
complains regardless of how I change the lines.

Thanks in advance,

Rich



From rshepard at appl-ecosys.com  Sun Jul 26 18:02:10 2015
From: rshepard at appl-ecosys.com (Rich Shepard)
Date: Sun, 26 Jul 2015 09:02:10 -0700 (PDT)
Subject: [code-quality] Not Seeing PEP8 Formatting Errors
In-Reply-To: <alpine.LNX.2.11.1507260849570.4769@localhost>
References: <alpine.LNX.2.11.1507260849570.4769@localhost>
Message-ID: <alpine.LNX.2.11.1507260901460.4769@localhost>


On Sun, 26 Jul 2015, Rich Shepard wrote:

> 84                     CheckConstraint(
> 84       "category IN('Air', 'Environmental', 'Health', 'Safety',

   Typo: Should be 84 and 85. Time for more coffee.

Rich

From graffatcolmingov at gmail.com  Sun Jul 26 18:37:22 2015
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Sun, 26 Jul 2015 11:37:22 -0500
Subject: [code-quality] Not Seeing PEP8 Formatting Errors
In-Reply-To: <alpine.LNX.2.11.1507260849570.4769@localhost>
References: <alpine.LNX.2.11.1507260849570.4769@localhost>
Message-ID: <CAN-Kwu0YKrXnvt-dGwMqiUu0HBcmNPXgxKCxUZBhpFrLpZ-3nA@mail.gmail.com>

On Sun, Jul 26, 2015 at 10:58 AM, Rich Shepard <rshepard at appl-ecosys.com> wrote:
>   I've numbered the lines in this code fragment:
>
> 83   category = Column(Unicode(16), value = 'Environmental', nullable =
> False,
> 84                     CheckConstraint(
> 84       "category IN('Air', 'Environmental', 'Health', 'Safety',
> 'Occupancy',
> 86       'Water'"))

So that would be formatted like so:

https://gist.github.com/sigmavirus24/67268ec32d614dffbcd1

I used a gist because it uses pre-formatted (fixed-width font) text so
that it's much clearer than email where depending on your reader it
can either be fixed with, or variable width, etc.

That said, your "CheckConstraint" parameter looks odd to me but I
copied it and made it appropriately multi-line as you would expect it
to work. (Taking advantage of the interpreter's native concatenation
of adjacent string literals.)

>    flake8 has these errors:
>
> 84:39: E201 whitespace after '('
> 86:9: E128 continuation line under-indented for visual indent
>
>   I've read PEP8 and found several fora threads related to E128 and E201 but
> no matter how I reformat that statement I cannot remove these (or other)
> errors when flake8 runs on this module.
>
>   When I continue line 84 the '(' and emacs breaks it near column 78 I get a
> different error about closing brackets not being correctly aligned. But, no
> matter where I place the final bracket it shows an error.
>
>   This module has many error conditions simular to the above and I want to
> understand how these long lines should be formatted so flake8 no longer
> complains regardless of how I change the lines.
>
> Thanks in advance,
>
> Rich
>
>
> _______________________________________________
> code-quality mailing list
> code-quality at python.org
> https://mail.python.org/mailman/listinfo/code-quality

From graffatcolmingov at gmail.com  Sun Jul 26 18:40:45 2015
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Sun, 26 Jul 2015 11:40:45 -0500
Subject: [code-quality] Not Seeing PEP8 Formatting Errors
In-Reply-To: <CAN-Kwu0YKrXnvt-dGwMqiUu0HBcmNPXgxKCxUZBhpFrLpZ-3nA@mail.gmail.com>
References: <alpine.LNX.2.11.1507260849570.4769@localhost>
 <CAN-Kwu0YKrXnvt-dGwMqiUu0HBcmNPXgxKCxUZBhpFrLpZ-3nA@mail.gmail.com>
Message-ID: <CAN-Kwu0VGgCCazBvvBO409h7krjRD+z5WYOW4qeNBzxki4PtXg@mail.gmail.com>

On Sun, Jul 26, 2015 at 11:37 AM, Ian Cordasco
<graffatcolmingov at gmail.com> wrote:
> On Sun, Jul 26, 2015 at 10:58 AM, Rich Shepard <rshepard at appl-ecosys.com> wrote:
>>   I've numbered the lines in this code fragment:
>>
>> 83   category = Column(Unicode(16), value = 'Environmental', nullable =
>> False,
>> 84                     CheckConstraint(
>> 84       "category IN('Air', 'Environmental', 'Health', 'Safety',
>> 'Occupancy',
>> 86       'Water'"))
>
> So that would be formatted like so:
>
> https://gist.github.com/sigmavirus24/67268ec32d614dffbcd1
>
> I used a gist because it uses pre-formatted (fixed-width font) text so
> that it's much clearer than email where depending on your reader it
> can either be fixed with, or variable width, etc.
>
> That said, your "CheckConstraint" parameter looks odd to me but I
> copied it and made it appropriately multi-line as you would expect it
> to work. (Taking advantage of the interpreter's native concatenation
> of adjacent string literals.)

Also keep in mind, that there's more than one way to satisfy PEP8.

I updated that gist with a few other ways that *should* satisfy pep8.

From rshepard at appl-ecosys.com  Sun Jul 26 19:39:36 2015
From: rshepard at appl-ecosys.com (Rich Shepard)
Date: Sun, 26 Jul 2015 10:39:36 -0700 (PDT)
Subject: [code-quality] Not Seeing PEP8 Formatting Errors
In-Reply-To: <CAN-Kwu0VGgCCazBvvBO409h7krjRD+z5WYOW4qeNBzxki4PtXg@mail.gmail.com>
References: <alpine.LNX.2.11.1507260849570.4769@localhost>
 <CAN-Kwu0YKrXnvt-dGwMqiUu0HBcmNPXgxKCxUZBhpFrLpZ-3nA@mail.gmail.com>
 <CAN-Kwu0VGgCCazBvvBO409h7krjRD+z5WYOW4qeNBzxki4PtXg@mail.gmail.com>
Message-ID: <alpine.LNX.2.11.1507261031570.4769@localhost>

On Sun, 26 Jul 2015, Ian Cordasco wrote:

>> So that would be formatted like so:
>> https://gist.github.com/sigmavirus24/67268ec32d614dffbcd1

   Ex2.py has this formatting:

category = Column(
     Unicode(16), value='Environmental', nullable=False,
     CheckConstraint(
         "category IN('Air', 'Environmental', 'Health', 'Safety', 'Occupancy',"
         "'Water')"
     )
)

which is how I reformatted my code (now starting on line 85):

category = Column(
     Unicode(16), value = 'Environmental', nullable = False,
     CheckConstraint(
         "category IN('Air', 'Environmental', 'Health', 'Safety', 'Occupancy',"
         "'Water')"
     )
)

and this is what flake8 shows:

85:25: E201 whitespace after '('
87:13: E128 continuation line under-indented for visual indent

   flake8 seems to have issues also with how emacs enters a space. In lines
like this one

__tablename__ = 'permits'

it tells me there is a missing white space aound the operator. I delete the
both spaces and re-enter them and that error goes away.

Rich


From graffatcolmingov at gmail.com  Sun Jul 26 19:53:10 2015
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Sun, 26 Jul 2015 12:53:10 -0500
Subject: [code-quality] Not Seeing PEP8 Formatting Errors
In-Reply-To: <alpine.LNX.2.11.1507261031570.4769@localhost>
References: <alpine.LNX.2.11.1507260849570.4769@localhost>
 <CAN-Kwu0YKrXnvt-dGwMqiUu0HBcmNPXgxKCxUZBhpFrLpZ-3nA@mail.gmail.com>
 <CAN-Kwu0VGgCCazBvvBO409h7krjRD+z5WYOW4qeNBzxki4PtXg@mail.gmail.com>
 <alpine.LNX.2.11.1507261031570.4769@localhost>
Message-ID: <CAN-Kwu3oZJa2obzs7aFi4XOWyBFjuV9KqFcvPSgRQyDd-gSz0Q@mail.gmail.com>

On Sun, Jul 26, 2015 at 12:39 PM, Rich Shepard <rshepard at appl-ecosys.com> wrote:
> On Sun, 26 Jul 2015, Ian Cordasco wrote:
>
>>> So that would be formatted like so:
>>> https://gist.github.com/sigmavirus24/67268ec32d614dffbcd1
>
>
>   Ex2.py has this formatting:
>
> category = Column(
>     Unicode(16), value='Environmental', nullable=False,
>     CheckConstraint(
>         "category IN('Air', 'Environmental', 'Health', 'Safety',
> 'Occupancy',"
>         "'Water')"
>     )
> )
>
> which is how I reformatted my code (now starting on line 85):
>
> category = Column(
>     Unicode(16), value = 'Environmental', nullable = False,
>     CheckConstraint(
>         "category IN('Air', 'Environmental', 'Health', 'Safety',
> 'Occupancy',"
>         "'Water')"
>     )
> )

Note that pep8 disallows spaces around keyword arguments, you have

    value = 'Environmental'

When pep8 wants

    value='Environmental'

Spaces around an = is for assignment, not for keyword arguments.

> and this is what flake8 shows:
>
> 85:25: E201 whitespace after '('
> 87:13: E128 continuation line under-indented for visual indent

This doesn't seem right. Can you share a file that causes this error?
Literally save the file and either email it to me privately or post it
somewhere where I can download this and run it locally?

>   flake8 seems to have issues also with how emacs enters a space. In lines
> like this one
>
> __tablename__ = 'permits'
>
> it tells me there is a missing white space aound the operator. I delete the
> both spaces and re-enter them and that error goes away.

Is Flake8 running as part of Emacs with something like Flycheck?

From rshepard at appl-ecosys.com  Sun Jul 26 20:03:05 2015
From: rshepard at appl-ecosys.com (Rich Shepard)
Date: Sun, 26 Jul 2015 11:03:05 -0700 (PDT)
Subject: [code-quality] Not Seeing PEP8 Formatting Errors
In-Reply-To: <CAN-Kwu3oZJa2obzs7aFi4XOWyBFjuV9KqFcvPSgRQyDd-gSz0Q@mail.gmail.com>
References: <alpine.LNX.2.11.1507260849570.4769@localhost>
 <CAN-Kwu0YKrXnvt-dGwMqiUu0HBcmNPXgxKCxUZBhpFrLpZ-3nA@mail.gmail.com>
 <CAN-Kwu0VGgCCazBvvBO409h7krjRD+z5WYOW4qeNBzxki4PtXg@mail.gmail.com>
 <alpine.LNX.2.11.1507261031570.4769@localhost>
 <CAN-Kwu3oZJa2obzs7aFi4XOWyBFjuV9KqFcvPSgRQyDd-gSz0Q@mail.gmail.com>
Message-ID: <alpine.LNX.2.11.1507261101110.4769@localhost>

On Sun, 26 Jul 2015, Ian Cordasco wrote:

> Note that pep8 disallows spaces around keyword arguments, you have
>    value = 'Environmental'
> When pep8 wants
>    value='Environmental'
> Spaces around an = is for assignment, not for keyword arguments.

Ian,

   Mea culpa! Thought I had fixed all these.

> This doesn't seem right. Can you share a file that causes this error?
> Literally save the file and either email it to me privately or post it
> somewhere where I can download this and run it locally?

   I'll send you the file and a copy of the flake8 output. There must be
something simple I'm just not seeing despite looking and looking.

> Is Flake8 running as part of Emacs with something like Flycheck?

   No. I open another buffer and run the shell in it.

Rich

From noamraph at gmail.com  Tue Jul 28 10:12:35 2015
From: noamraph at gmail.com (Noam Yorav-Raphael)
Date: Tue, 28 Jul 2015 11:12:35 +0300
Subject: [code-quality] Is there a way to speedup pylint by caching results
	between runs?
Message-ID: <CAO8o=D4goa24i2SmdYKsCQCTMTxTcscDBP4VcL8BeF+3AdibUQ@mail.gmail.com>

Hi,

We're using pylint as a pre-commit hook to check our code. However, it
takes about a minute for each run, and it's quite frustrating.

I guess that most of the work is redundant, as most files do not change
between runs. How hard would it be to add the ability to cache results to
speed-up linting? I may be able to spend a day or two working on this, if
there's a relatively simple way to do this.

Is it possible?

Thanks,
Noam
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20150728/b7d0ce71/attachment.html>

From mike at pythonlibrary.org  Thu Jul 30 20:00:30 2015
From: mike at pythonlibrary.org (Mike Driscoll)
Date: Thu, 30 Jul 2015 13:00:30 -0500
Subject: [code-quality] flake8 documentation out of date
Message-ID: <CAO4gEAuNd6gQ1rnKEG_cELQAVEuNR=kx9AHgqMrXtiPt50SM1Q@mail.gmail.com>

Hi,

I noticed that the documentation for flake8 on Read The Docs doesn't appear
to be up-to-date:

http://flake8.readthedocs.org/en/latest/config.html

This page doesn't mention where to put the config file on Windows whereas
the file that page is based on does (on Gitlab anyway):

https://gitlab.com/pycqa/flake8/blob/master/docs/config.rst

-----------------
Mike Driscoll

Blog:   http://blog.pythonlibrary.org
Book: https://gumroad.com/l/bppWr
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/code-quality/attachments/20150730/852f7339/attachment.html>

From pcmanticore at gmail.com  Fri Jul 31 09:30:50 2015
From: pcmanticore at gmail.com (Claudiu Popa)
Date: Fri, 31 Jul 2015 10:30:50 +0300
Subject: [code-quality] Is there a way to speedup pylint by caching
 results between runs?
In-Reply-To: <CAO8o=D4goa24i2SmdYKsCQCTMTxTcscDBP4VcL8BeF+3AdibUQ@mail.gmail.com>
References: <CAO8o=D4goa24i2SmdYKsCQCTMTxTcscDBP4VcL8BeF+3AdibUQ@mail.gmail.com>
Message-ID: <CAMy=CLqu5kRPBT5TFydPQdm1Rz4-VbF7C7a9qKmi6JyJ4Wpdyg@mail.gmail.com>

On Tue, Jul 28, 2015 at 11:12 AM, Noam Yorav-Raphael <noamraph at gmail.com> wrote:
> Hi,
>
> We're using pylint as a pre-commit hook to check our code. However, it takes
> about a minute for each run, and it's quite frustrating.
>
> I guess that most of the work is redundant, as most files do not change
> between runs. How hard would it be to add the ability to cache results to
> speed-up linting? I may be able to spend a day or two working on this, if
> there's a relatively simple way to do this.
>
> Is it possible?
>
> Thanks,
> Noam
>
> _______________________________________________
> code-quality mailing list
> code-quality at python.org
> https://mail.python.org/mailman/listinfo/code-quality
>

Hi Noam,

I guess that it won't be too hard if the unchanged files
will not be analyzed, but if a file has a couple of lines changed
that completely complicates the situation, since you'll have
to do some sort of diff approach over the AST, which might not
be trivial.

For the latter case, you might find diff-cover
(https://github.com/edx/diff-cover)
useful.


/Claudiu