[Python-Dev] cpython (3.2): Issue #11956: Skip test_import.test_unwritable_directory on FreeBSD when run as

[Cameron Simpson]

On 07Oct2011 12:46, Andrew Bennetts <andrew at bemusement.org> wrote:
| On Fri, Oct 07, 2011 at 08:27:01AM +1100, Cameron Simpson wrote:
| […]
| > | >> running buildbot tests as root does not reflect the experience of
| > | >> non-root users. It seems some tests need to be run both ways just for
| > | >> correctness testing.
| > | 
| > | (except I'd say "all", not "some")
| > 
| > No. Terry is right and you are ... not. Most tests need no special
| > privileges - they're testing language/library semantics that do not
| > depend on the system facilities much, and when they do they should work
| > for unprivileged users.
| 
| You could also say that most tests that work on Linux work on FreeBSD
| too, so when they work on Linux they should work for FreeBSD too… so why
| bother running tests on FreeBSD at all?  The reason is because the
| assumptions behind that “should” are wrong frequently enough to make it
| worth running tests in both environments.

For thoroughness, yes.

| Like Glyph, I think that “running as root” is sufficiently different
| environment to “running as typical user” (in terms of how POSIX-like
| systems behave w.r.t. to things like permissions checks) to make it
| worthwhile to regularly run the whole test suite as root.

Hmm. Glyph seemed to be arguing both ways - that everything should be
tested as root, and also that root is not special. I have unease over the
former and disagreement over the latter.

| > HOWEVER, the whole suite should not be _tested_ as root because the code
| > being testing is by definition untrusted.
| 
| No, that just means you shouldn't trust *root*.  Which is where a VM is
| a very useful tool.  You can have the “as root” environment for your
| tests without the need to have anything important trust it.
| 
| > Root _is_ special, within the host and with scope to misbehave beyond
| > the host.
| > 1: The permission system does _not_ behave the same for root as for
| >    other users.
| 
| Those are arguments *for* running tests as root!

I think they're arguments for running _specific_ tests as root. File I/O
based tests primarily I would suppose, though my first instinct would be
to constrain even these to permission related tests.

| > 2: Root _can_ corrupt things anywhere in the system (within the VM, of
| >    course, but the builtbot is a subset of it). A normal unprivileged user
| 
| This appears to be a key error in your logic.  There's no fundamental
| reason why “tests run as root inside a VM” must necessarily imply
| “buildbot process is run inside that same VM and is therefore vulnerable
| to code in that test run.”

Indeed, I had no considered that the tests might be run in a special VM
distinct from the build environment. In that setup most of my concerns
are moot.

| It may be more convenient to deploy it that way, but I'm sure it's
| possible to have a buildslave configured to e.g. start a pristine VM
| (from a snapshot with all the necessary build dependencies installed) and
| via SSH copy the the source into it, build it, run it, and report the
| results.  The VM could be fully isolated from the real network and
| filesystem etc if you like.
| 
| Given that it is certainly possible to run tests as root about as
| securely as running them without root, do you still feel it is not worth
| running the tests as root?

Not in the style you describe above. To clarify: I agree with your
suggestion.

| > The prudent way to run the buildbots, especially if they cycle (refetch
| > newer codebase, rebuilt, retest) instead of (scrub VM, reinstall,
| > install built system, etc) is:
| > 
| >   - a user to fetch source and dispatch builds
| >   - possibly a distinct user to run the builds
| >   - definitely a distinct user to run the test suite
| 
| If we're talking prudence, then surely s/user/VM/ is even better :)

Yes. And a distinct VM instance for the root and non-root tests, too.

Cheers,
-- 
Cameron Simpson <cs at zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

all coders are created equal; that they are endowed with certain
unalienable rights, of these are beer, net connectivity, and the
pursuit of bugfixes...  - Gregory R Block