[Python-Dev] Community buildbots (was Re: User's complaints)

[glyph at divmod.com]

On Fri, 14 Jul 2006 06:46:55 -0500, skip at pobox.com wrote:
>
>    Neal> How often is the python build broken or otherwise unusable?
>
>Not very often.

I have to agree.  The effort I'm talking about is not in fixing large numbers
of problems, but simply gearing up to properly test to see if there *are*
problems.  Keep in mind though: just because the problems are small or easy
to fix doesn't mean they're not severe.  One tiny bug can prevent a program
from even starting up.

>Admittedly, I'm not as sophisticated a user as Fredrik or Glyph, but I
>suspect that my usage of the language isn't all that different from most
>Python developers out there.

A huge percentage of Python developers are working with Zope, which means that
although *their* code might not be terribly "sophisticated", it is
nevertheless sitting on top of a rather large and intricate pile of implicit
dependencies on interpreter behavior.

>    Neal> Is part of your point that these developers only care about
>    Neal> something called "release" and they won't start testing before
>    Neal> then?  If that's the case why don't we start making
>    Neal> (semi-)automated alpha releases every month?

>How would that be any easier than a user setting up a read-only repository
>and svn-up-ing it once a month then using that as the default interpreter on
>that person's development machine?  I maintain interpreters for 2.3, 2.4 and
>bleeding edge at the moment.  If I need to it's fairly trivial (a symlink
>change) to fall back to the latest stable release.

>Glyph, would that sort of scheme work for you?

No.

I really think you're underestimating the sort of effort required to upgrade
Python.

First of all, I do have a job :) and it'd be very hard to make the case to an
investor that it was worth tracking down every potential bug I had to
determine whether it was a problem because I was working with an unreleased
version of Python.  This is the same reason I don't use beta versions of the
kernel or libc for development.

For that matter, I've had to avoid posting to this mailing list because even
*this* is stretching my already overburdened schedule :).

Secondly, I have to work with a few extension modules.  To name a few:
  * ctypes
  * PyPAM
  * pysqlite2
  * zope.interface (C optimizations)
  * xapian
  * pylucene
  * dspam
  * PyOpenSSL
  * PIL
  * PyCrypto
  * pyexpat
  * pygtk
  * pyvte

Recompiling all of these is a project that takes a day.  PyLucene, in fact,
I've never managed to build myself, and I can only run because there happen
to be debian packages which work (with some fiddling) on Ubuntu.  There's no
interactive warning during 'svn up' that it's time to recompile everything,
either, so I don't even know when the ABIs are going to have changed.

Even if everything works perfectly, and were perfectly automated, the compile 
itself would take a few hours.

I also test work on Windows on occasion and recompiling these things _there_
is the work of a week and a half, not to mention it requires that I be sitting
at the one machine where I have my Microsoft™ Developer™ Tools™ installed.

I made the buildbot recommendation specifically because it would centralize
the not inconsiderable effort of integrating these numerous dependencies
(presuming that I could submit a Divmod buildbot as well as a Twisted one).