[Python-Dev] Community buildbots and Python release quality metrics

[Guido van Rossum]

Too verbose, Glyph. :-)

It needs to be decided case-by-case. The beta is called beta because,
well, it may break stuff and we may want to fix it. But there are also
likely many frameworks that use undefined behavior. I'm not
particularly impressed by statistics like "all tests are red" -- this
may all be caused by a single issue. For example, I'd much rather read
an explanation about *why* Django cannot even be imported than a
blanket complaint that this is a disgrace. So why is it?

--Guido

On Thu, Jun 26, 2008 at 8:18 AM,  <glyph at divmod.com> wrote:
> Today on planetpython.org, Doug Hellman announced the June issue of Python
> magazine.  The cover story this month is about Pybots, "the fantastic
> automation system that has been put in place to make sure new releases of
> Python software are as robust and stable as possible".
>
> Last week, there was a "beta" release of Python which, according to the
> community buildbots, cannot run any existing python software.  Normally I'd
> be complaining here about Twisted, but in fact Twisted is doing relatively
> well right now; only 80 failing tests.  Django apparently cannot even be
> imported.
>
> The community buildbots have been in a broken state for months now[1]. I've
> been restraining myself from whinging about this, but now that it's getting
> close to release, it's time to get these into shape, or it's time to get rid
> of them.
>
> There are a few separate issues here.  One is, to what extent should Python
> actually be compatible between releases?  There's not much point in saying
> "this release is compatible with python 2.5" if all python 2.5 programs need
> to be modified in order to run on python 2.6.  Of course, it's quite likely
> that there are some 3rd-party Python programs that continue to work, but
> python.org provides links to 2 projects that don't and none that do.
>
> Another way to phrase this question is, "whose responsibility is it to make
> Python 2.5 programs run on Python 2.6"?  Or, "what happens when the core
> team finds out that a change they have made has broken some python software
> 'in the wild'"?
>
> Here are a couple of ways to answer these questions:
>
>  1) It's the python core team's responsibility.  There should be Python
> buildbots for lots of different projects and they should all be green before
> the code can be considered [one of: allowed in trunk, alpha, beta, RC].
>  2) It's the python core team's responsibility as long as the major projects
> are using public APIs.  The code should not be considered [alpha, beta, RC]
> if there are any known breakages in public APIs.
>  3) It's only the python core team's responsibility to notify projects of
> incompatible changes of which they are aware, which may occur in public or
> private APIs.  Major projects with breakages will be given a grace period
> before a [beta, RC, final] to release a forward-compatible version.
>  4) The status quo: every new Python release can (and will, at least
> speaking in terms of 2.6) break lots of popular software, with no warning
> aside from the beta period.  There are no guarantees.
>
> For obvious reasons, I'd prefer #1, but any of the above is better than #4.
>  I don't think #4 is an intentional choice, either; I think it's the result
> of there being no clear consequence for community buildbots failing.  Or,
> for that matter, of core buildbots failing.  Investigating the history is
> tricky (I see a bunch of emails from Barry here, but I'm not sure which is
> the final state) but I believe the beta went out with a bunch of buildbots
> offline or failing.
>
> A related but independent issue is: should the community buildbots continue
> to exist?  An automated test is only as good as the consequences of its
> failure.  As it stands, the community buildbots seem to be nothing but an
> embarrassment.  I don't mean this to slam Grig, or the Python team - I mean,
> literally, if there is no consequence of their failure then the only use I
> can see for the data the buildbots currently provide (i.e. months and months
> of failures) is to embarrass the Python core developers.  python.org's
> purpose should not be to provide negative press for Python ;), so if this
> continues to be the case, they probably shouldn't be linked.  This isn't
> just an issue with changes to Python breaking stuff; the pybots'
> configuration is apparently not well maintained, since the buildbots which
> say "2.5" aren't passing either, and that's not a problem with the code,
> it's just that the slaves are actually running against trunk at HEAD.
>
> Ultimately these tools only exist to ensure the quality of Python releases.
>  The really critical question here is, what does it mean to have a Python
> release that is high-quality?  There are some obvious things: it shouldn't
> crash, it should conform to its own documentation. Personally I think "it
> passes all of its own tests" and "it runs existing Python code" are
> important metrics too, but the most important thing I'm trying to get across
> here is that there should be a clear understanding of which goals the
> release / QA / buildbot / test process is trying to accomplish.  For me,
> personally, it really needs to be clear when I can say "you guys screwed up
> and broke stuff", and when I just have to suck it up and deal with the
> consequences of a new version of Python in Twisted.
>
> It's definitely bad for all of us if the result is that new releases of
> Python just break everything.  Users don't care where the responsibility
> lies, they just know that stuff doesn't work, so we should decide on a
> process which allows Twisted (and other popular projects, like Django,
> Plone, pytz, PyGTK, pylons, ...) to (mostly) run when new versions of Python
> are released.
>
> [1]: http://mail.python.org/pipermail/python-dev/2008-May/079212.html
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)