[Python-Dev] buildbot vs. Windows

[Tim Peters]

[Martin v. Löwis]
> So what is your recipe:

I don't have one.  I personally always use -uall and -r, and then run
the tests 8 times, w/ and w/o -O, under debug and release builds, and
w/ and w/o deleting .py[co] files first.  Because that last one almost
never finds problems anymore, perhaps it would be good to stop
bothering with it routinely (it really doesn't have potential to find
a problem unless someone has been mucking with the marshaling of code
objects, right?).

> Add -r to all buildbots?

Sure.  -r adds variety to testing at no cost (the same number of tests
run, in the same pamount of time, with or without -r).

> Only to those which have an 'a' in their name?

Sorry, no idea what that means.

> Only to every third build? Duplicating the number of builders?

For -r, no.  I'd always use -r (and always do anyway).

> Same question for --with-pydebug. Combining this with -r would multiply
> the number of builders by 4 already.

I would much rather see a debug-build run than the current "with and
without deleting .py[co] files first" variant.  If the latter were
dropped and the former were added, and -r were used all the time, the
number of recipes wouldn't change.  Testing time would increase, by
the time to _do_ a debug build, and by the extra time a debug build
test run requires.

We should test with and without -O too, although that's another that
rarely finds a problem.

> I'm not keen on deciding this for myself. Somebody else please decide
> for me.

I don't know how hard it is to teach the system how to do something
"not so often", and I expect that's an important unknown since I
imagine that vastly increasing test time would discourage people from
volunteering buildbot slaves.

Since the most fruitful variations (IME) for finding code errors are
using -r and running a debug build too, I'd change the current
run-all-the-time recipes to:

- Stop doing the second "without deleting .py[co]" run.
- Do one run with a release build.
- Do one run with a debug build.
- Use -uall -r for both.

If we know how to get something done "occasionally", then about once a
week it would be prudent to also:

- Try the "with and without deleting .py[co] files first" business.
- Try with and without -O,

Those last two choices cover 8 distinct modes, when paired with each
other and with the "release versus debug build" choice.