[python-committers] Statistics: growth of core dev number vs growth of the code size/complexity

[Ezio Melotti]

On Thu, Dec 7, 2017 at 8:43 PM, Brett Cannon <brett at python.org> wrote:
>
>
> On Wed, 6 Dec 2017 at 15:17 Victor Stinner <victor.stinner at gmail.com> wrote:
>>
>> Hi,
>>
>> I wrote a quick & dirty parser to compute statistics on *new* CPython
>> core developer per year using the following page as data:
>> https://devguide.python.org/developers/
>>
>> 2007: 15
>> 2008: 19
>> 2009: 11
>> 2010: 20
>> 2011: 12
>> 2012: 9
>> 2013: 4
>> 2014: 10
>> 2015: 2
>> 2016: 5
>> 2017: 2
>>
>> Compare these numbers to Stéphane Wirtel's statistics on pull requests:
>>    https://speakerdeck.com/matrixise/cpython-loves-your-pull-requests
>>
>> => Number of active core developerson on GitHub pull requests: 27
>> (stats from February 2017 to October 2017)
>> (I'm not sure of the meaning of this number, it's the number of core
>> developer who authored pull requests, I don't think that it counts
>> core developers who only made reviews.)
>>
>> If you look at the size of the source code, it's still growing
>> constanly since 1990:
>> https://www.openhub.net/p/python/
>>
>> 2007: around 783k lines
>> 2010: around 683k lines
>> 2013: around 800k lines
>> 2015: around 875k lines
>> 2017: around 973k lines
>>
>> The number of bugs is also constanly growing. Statistics on bugs since
>> 2011:
>> https://bugs.python.org/issue?@template=stats
>>
>> 2011: around 2500 open issues
>> 2013: around 4000 open issues
>> 2015: around 5000 open issues
>> 2017: around 6200 open issues
>
>
> Do realize that open issues is a really misleading statistic as they include
> enhancement requests which we historically never close unless there's zero
> chance we will accept such a change.
>

Here is a breakdown:
2443 behavior
2124 enhancements
224 crash
170 compile error
125 resource usage
104 performance
44 security


>>
>>
>> The size of the CPython project is constantly growing as its
>> complexity (technical debt? what is this? :-)), but the growth of core
>> developers is slowing down.
>
>
> Well, you added code to speed up Unicode encoding/decoding, right? So it's
> just adding stuff to keep things performant as well as new things. It's just
> what happens when you're willing to improve things.
>
>>
>>
>> I do consider that we need more people to handle the growing number of
>> issues and pull requests, so the question is now how to find and
>> "hire" (sorry, promote) them ;-)
>>
>> Maybe we have a problem with mentoring. Maybe the CPython code base
>> became too hard to train newcomers? Maybe we are too conservative? I
>> don't know.
>
>
> I think it's partially a fact that Python's popularity has increased the
> pool size of contributors, so lots of people grabbing individual things.
> This leads to less of a chance to make sustained contributions. E.g. when I
> became a core dev it was because I was able to grab a new issue to work on
> that was easy at a very regular cadence, but I don't know if I could rectify
> that at this point.
>

I think it also has to do with the maturity of the project.

When I started I remember I wanted to fix some issue and when I ran
the whole test suite I had at least a dozen failing on my machine, so
I went and fix those as well.  But to fix those I needed to add more
stuff to test.support and to regrtest, and so I did.  Then those
changes needed to be documented, but the documentation was missing.
So I started improving the documentation and even after all that was
done there were still plenty of low hanging fruits on the bug tracker
or other issues to work on.  Then Python 3 came, and there was more
work to be done.

Nowadays the situation is much better, Python is more stable and
mature, and what's left is more difficult, obscure, or controversial.
There are still new modules and features being added and ISTM that
most of the new core devs are working on those (e.g.
asyncio/typing/etc), but otherwise finding new easy issues is becoming
increasingly more difficult.

Best Regards,
Ezio Melotti