[python-committers] My cavalier and aggressive manner, API change and bugs introduced for basically zero benefit

[Nick Coghlan]

On 21 January 2017 at 09:02, Lukasz Langa <lukasz at langa.pl> wrote:
>> On Jan 20, 2017, at 9:26 AM, Yury Selivanov <yselivanov.ml at gmail.com> wrote:
>> I think that we need to become less conservative about
>> development of CPython internals.  At this point it's impossible
>> to make CPython any faster without invasive refactorings, and
>> I think it's OK to trust our core developers to make them.
>
> I agree with the sentiment but I'm worried about the implications. Your suggested approach only makes sense if we have a good safety net in place to catch regressions early. Unfortunately, apart from regrtest, buildbots, and dumb luck, we don't. Betas have very limited adoption, too limited to be useful in my experience. I tried and failed to introduce 3.6 during beta phase at work because there was too much risk it would be picked up by unsuspecting users. Other users seem to be in the same camp. Consequently, most x.y.0 releases are relatively lower quality.

The Red Hat python-maint folks (Peter Viktorin et al) similarly tried
to time things so there was a window to provide feedback from Fedora
Rawhide on a beta release prior to the final 3.6.0 release, but it
unfortunately ended up not being practical, especially with the lack
of clarity around when the 3.6 ABI would actually be frozen.

That uncertainty around the release ABI stability also impacted the
ability of the manylinux folks to include any of the 3.6 pre-releases
in their reference build environment.

To further complicate matters, the user visible syntax and semantics
of various subsystems have also been changing at a rate not really
seen within a single release series since the 2.3/4/5 era more than a
decade ago, where we saw extended slicing support in the builtin
types, decorators, with statements, and more.

I think both of these shifts (faster moving internals, increased rate
of change in public interfaces) is a symptom of the fact that while we
don't have anyone being paid specifically to focus on stability and
general maintenance, we *do* have more active folks with paid or
personal time to spend on particular problems related to dealing with
large code bases and software deployments (reducing CPU usage, memory
management, asynchronous flow control, improving static analysis
support, security), improving integration with particular deployment
platforms (mainly Windows and Linux, but also mobile devices), and
iterating on low level areas of personal interest (e.g. the import
system, metaclasses, timezones).

In that environment, we need to be careful to remember that we're
collectively stewards of the overall language ecosystem in addition to
the reference implementation.

That means it isn't just CPython we need to worry about - yes, it's
beneficial to the ecosystem as a whole to have significant low level
improvements being made to the reference interpreter (since it's still
far and away the most popular implementation), but we're the *most*
well-resourced of the Python interpreter implementations, and even we
have a grand total of zero people paid to work full-time on monitoring
and facilitating the core development process (our commercial
redistributors appear to be OK with that state of affairs, which
implies that their customers are either also fine with it, or else
aren't looking too closely at what they're getting for their money).

Other implementations are often still attempting to catch up to where
CPython was a year or more ago, before they can even start thinking
about where we're going to be a year or more from now.

So if we're hitting review bottlenecks for low level changes, then
"I'll commit the change anyway" probably isn't the best path for us to
be taking - it's likely to be more beneficial in the long run to take
a step back and ask *why* there isn't anyone else with the time and
inclination to review the change, and whether or not there might be a
way for us to help address the underlying sustainability problems (or
advocate internally for our respective employers to start doing so in
a more systematic way).

Cheers,
Nick.

P.S. The simplest illustration I know of for those underlying
sustainability problems is the "up and to the right" nature of the
graph of open issues: http://bugs.python.org/issue?@template=stats

The move to GitHub and the introduction of automated pre-merge CI
testing will hopefully help streamline the patch review process for
simple fixes, but it's far from clear at this point whether or not
that will be enough to get the issue metrics moving in a healthier
direction.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia