[Python-Dev] Allowing to run certain regression tests in subprocesses

[Eli Bendersky]

On Sat, Aug 3, 2013 at 7:08 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> On 4 Aug 2013 12:03, "Eli Bendersky" <eliben at gmail.com> wrote:
>>
>> On Sat, Aug 3, 2013 at 6:59 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> >
>> > On 4 Aug 2013 09:43, "Eli Bendersky" <eliben at gmail.com> wrote:
>> >>
>> >> Hi All,
>> >>
>> >> Today the issue of cross-test global env dependencies showed its ugly
>> >> head again for me. I recall a previous discussion
>> >> (http://mail.python.org/pipermail/python-dev/2013-January/123409.html)
>> >> but there were many more over the years.
>> >>
>> >> The core problem is that some tests modify the global env
>> >> (particularly importing modules) and this sometimes has adverse
>> >> effects on other tests, because test.regrtest runs all tests in a
>> >> single process. In the discussion linked above, the particular culprit
>> >> test__all__ was judged as a candidate to be moved to a subprocess.
>> >>
>> >> I want to propose adding a capability to our test harness to run
>> >> specific tests in subprocesses. Each test will have some simple way of
>> >> asking to be run in a subprocess, and regrtest will concur (even when
>> >> running -j1). test__all__ can go there, and it can help solve other
>> >> problems.
>> >>
>> >> My particular case is trying to write a test for
>> >> http://bugs.python.org/issue14988 - wherein I have to simulate a
>> >> situation of non-existent pyexpat. It's not hard to write a test for
>> >> it, but when run in tandem with other tests (where C extensions loaded
>> >> pyexpat) it becomes seemingly impossible to set up. This should not be
>> >> the case - there's nothing wrong with wanting to simulate this case,
>> >> and there's nothing wrong in Python and the stdlib - it's purely an
>> >> artifact of the way our regression suite works.
>> >
>> > I'm not actively opposed to the suggested idea, but is there a specific
>> > reason "test.support.import_fresh_module" doesn't work for this test?
>>
>> I'm not an expert on this topic, but I believe there's a problem
>> unloading code that was loaded by C extensions. import_fresh_module is
>> thus powerless here (which also appears to be the case empirically).
>
> Sure, it's just unusual to have a case where "importing is blocked by adding
> None to sys.modules" differs from "not actually available", so I'd like to
> understand the situation better.

I must admit I'm confused by the behavior of import_fresh_module too.

Snippet #1 raises the expected ImportError:

sys.modules['pyexpat'] = None
import _elementtree

However, snippet #2 succeeds importing:

ET = import_fresh_module('_elementtree', blocked=['pyexpat'])
print(ET)

I believe this happens because import_fresh_module does an import of
the 'name' it's given before even looking at the blocked list. Then,
it assigns None to sys.modules for the blocked names and re-imports
the module. So in essence, this is somewhat equivalent to snippet #3:

modname = '_elementtree'
__import__(modname)
del sys.modules[modname]
for m in sys.modules:
    if modname == m or m.startswith(modname + '.'):
        del sys.modules[m]
sys.modules['pyexpat'] = None
ET = importlib.import_module(modname)
print(ET)

Which also succeeds.

I fear I'm not familiar enough with the logic of importing to
understand what's going on, but it has been my impression that this
problem is occasionally encountered with import_fresh_module and C
code that imports stuff (the import of pyexpat is done by C code in
this case).

CC'ing Brett.

Eli