[Distutils] Getting more momentum for pip

[Donald Stufft]

> On Mar 5, 2015, at 11:38 AM, Marc Abramowitz <msabramo at gmail.com> wrote:
> 
> Thoughts?

So I agree with Paul in that part of the reason why some of the PRs languish is that they are adding new features that none of us feel strongly about, so nobody wants to push the button to say “yes we’re going to support this”. Being conservative with features is generally a good thing for a (mostly) volunteer project because we do have a limited amount of manpower and more features increases the surface area we have to take care of. We also tend to have a problem saying no to features and tend to just let them languish instead of making a decision one way or the other when we don’t feel strongly.

Another issue we have is that often we get requests (bug fixes or otherwise) which break tests or which themselves do not have tests written. We can’t merge these as is but it’s not unusual for the original author to not update their PR and leave it languishing with broken/no tests. It’s hard to want to close these as “abandoned” because there’s code that may or may not be working if someone takes the time to go through and finish it, but until someone feels like doing that the PR is not able to be merged.

Yet another issue is that pip’s test suite is not particularly very good. We’re missing a lot of coverage and we don’t have *any* CI running on platforms other than Ubuntu. This means that merging things is somewhat “dangerous” because it’s easy to break things without noticing unless you pull down the change and manually test things you can try. Even then that’s not good enough unless you can test it on other platforms as well. I’m sure Paul can fill in the blank on how often the test suite simply doesn’t run on Windows because of some POSIX assumption snuck in somewhere.

Another issue (that sort of ties in with the test one) is pip’s code base is simply not very good. Things are not well encapsulated and it’s non obvious what’s going to change anytime you change something. There was recently an issue where changing the order we checked for things to be uninstalled in broke an assumption that a lot of people were relying on.

Obviously the biggest limiting factor is simple manpower, and most of the above issues deal with things that make our use of use of what manpower we have more inefficient. I think that the best way to increase the momentum is to explore ways to make our use of the manpower we have more effective and reduce the waste.

I don’t think that an artificial limit on the number of issues or pull requests is a good path forward. Since we’re more or less entirely volunteer besides myself, I feel like an artificial limit on numbers will just mean that things don’t happen. I feel like if someone doesn’t want to close issues for whatever reasons, trying to force them to do that will just mean they are more likely to choose to spend their time elsewhere.

I also worry that anything which autocloses PRs (after a period of time, or if there wasn’t an associated issue, or whatever) is somewhat hostile to potential contributors and I feel like having an open PR isn’t that big of a deal in general.

I think the most effective way forward is that we need to work on fixing the things that needlessly suck up the pip core team’s time.

Things that would be huge improvements in this area:

* Refactoring pip to better encapsulate and separate concerns, creating boundaries between different parts[1]
* Improve the test suite by covering cases that aren’t being covered, moving functional tests to unit tests
* Creating a high level test suite that runs setuptools, pip, virtualenv, etc all together against real projects.
* Help with getting CI for other systems setup, particularly for Windows. This may take the shape of helping Travis, or it may be setting up another service or our own CI system.

Other things that would help are:

* People doing in-depth reviews of the current PRs that are there and suggesting changes or pointing out issues, etc.
* People triaging issues (unfortunately this one isn’t super easy with GitHub Issues since you have to be a committer to change these things).
* People going through and reviewing old issues and PRs to try and figure out if the situations that caused them to be opened originally still apply or if that problem has been fixed or if the code has changed significantly enough that it’s likely to not longer exist.

These sorts of things would make it *much* easier to merge new things because there would be less risk and less things involved in actually going through and figuring out if any particular merge is a good idea or not. I also think that people willing to put in the work to do things like this would be good candidates for becoming core developers themselves, which would also help by increasing the number of people we have able to review and commit.

[1] http://pyvideo.org/video/1670/boundaries

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150305/2fc33b93/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150305/2fc33b93/attachment-0001.sig>