[issue40754] Test installers before releasing (ModuleNotFoundErrors)

[Ned Deily]

Ned Deily <nad at python.org> added the comment:

> So it seems that only Linux has "installed" builbot tests. I suppose the next step is adding some macOS and Windows ones.

Having more buildbots test from "installed" locations, rather than the build directory, is good.

But that's not the whole issue here. Those "installed" tests are only testing that a Python runs from an installed location on the same system it was built.  The Windows and macOS installers use various packaging methods to provide complete Python installations, including binaries, that can be installed on other systems using various OS-native install methods familiar to users of each platform. And the binaries are generally built in such a way to support running on multiple release versions of its operating system. For example, the current Python installers for macOS from python.org are usable on macOS 10.9 through 10.15. To produce the various installer packages, there are other layers of tooling involved, say, on top of "make install" in the Unixy build process for macOS; something similar is true for Windows releases.

FWIW, every macOS installer is test installed on a vanilla system, the whole standard test suite is run, and an IDLE smoke test is run to ensure there are no noticeable regressions before handing the build artifacts off to the release manager. Every installer is tested on at least one supported macOS version and usually more than one.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue40754>
_______________________________________