[Python-Dev] Allowing to run certain regression tests in subprocesses

[R. David Murray]

On Sat, 03 Aug 2013 19:04:21 -0700, Eli Bendersky <eliben at gmail.com> wrote:
> On Sat, Aug 3, 2013 at 6:57 PM, R. David Murray <rdmurray at bitdance.com> wrote:
> > On Sat, 03 Aug 2013 16:47:37 -0700, Eli Bendersky <eliben at gmail.com> wrote:
> >> On Sat, Aug 3, 2013 at 4:36 PM, Eli Bendersky <eliben at gmail.com> wrote:
> >> > Hi All,
> >> >
> >> > Today the issue of cross-test global env dependencies showed its ugly
> >> > head again for me. I recall a previous discussion
> >> > (http://mail.python.org/pipermail/python-dev/2013-January/123409.html)
> >> > but there were many more over the years.
> >> >
> >> > The core problem is that some tests modify the global env
> >> > (particularly importing modules) and this sometimes has adverse
> >> > effects on other tests, because test.regrtest runs all tests in a
> >> > single process. In the discussion linked above, the particular culprit
> >> > test__all__ was judged as a candidate to be moved to a subprocess.
> >> >
> >> > I want to propose adding a capability to our test harness to run
> >> > specific tests in subprocesses. Each test will have some simple way of
> >> > asking to be run in a subprocess, and regrtest will concur (even when
> >> > running -j1). test__all__ can go there, and it can help solve other
> >> > problems.
> >> >
> >> > My particular case is trying to write a test for
> >> > http://bugs.python.org/issue14988 - wherein I have to simulate a
> >> > situation of non-existent pyexpat. It's not hard to write a test for
> >> > it, but when run in tandem with other tests (where C extensions loaded
> >> > pyexpat) it becomes seemingly impossible to set up. This should not be
> >> > the case - there's nothing wrong with wanting to simulate this case,
> >> > and there's nothing wrong in Python and the stdlib - it's purely an
> >> > artifact of the way our regression suite works.
> >> >
> >> > Thoughts?
> >> >
> >> > Eli
> >>
> >> FWIW the problem is also discussed here:
> >> http://bugs.python.org/issue1674555, w.r.t. test_site
> >
> > Can't you just launch a subprocess from the test itself using script_helpers?
> >
> 
> I can, but such launching will be necessarily duplicated across all
> tests that need this functionality (test_site, test___all__, etc).
> Since regrtest already has functionality for launching whole
> test-suites in subprocesses, it makes sense to reuse it, no?

In the case of test_site and test___all___ we are talking about running
the entire test file in a subprocess.  It sounds like you are only
talking about running one individual test function in a subprocess,
for which using script_helpers seems the more natural solution.

--David