[Python-checkins] [3.7] bpo-38546: Backport multiprocessing tests fixes from master (GH-19689)
Victor Stinner
webhook-mailer at python.org
Thu Apr 23 18:44:13 EDT 2020
https://github.com/python/cpython/commit/fd32a0e2ee445dd7156323d216627112e66b0a69
commit: fd32a0e2ee445dd7156323d216627112e66b0a69
branch: 3.7
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2020-04-24T00:44:08+02:00
summary:
[3.7] bpo-38546: Backport multiprocessing tests fixes from master (GH-19689)
* bpo-37421: multiprocessing tests call _run_finalizers() (GH-14527)
multiprocessing tests now call explicitly _run_finalizers() to remove
immediately temporary directories created by
multiprocessing.util.get_temp_dir().
(cherry picked from commit 039fb49c185570ab7b02f13fbdc51c859cfd831e)
Co-authored-by: Victor Stinner <vstinner at redhat.com>
(cherry picked from commit 632cb36084dc9d13f1cdb31a0e7e3ba80745a51a)
* bpo-37421: multiprocessing tests now stop ForkServer (GH-14601)
multiprocessing tests now stop the ForkServer instance if it's
running: close the "alive" file descriptor to ask the server to stop
and then remove its UNIX address.
(cherry picked from commit 8fbeb14312b4c1320d31ad86e69749515879d1c3)
Co-authored-by: Victor Stinner <vstinner at redhat.com>
(cherry picked from commit 229f6e85f8b4d57a2e742e0d3fc361c5bd15f1cb)
* bpo-38546: multiprocessing tests stop the resource tracker (GH-17641) (GH-17647)
Multiprocessing and concurrent.futures tests now stop the resource
tracker process when tests complete.
Add ResourceTracker._stop() method to
multiprocessing.resource_tracker.
Add _cleanup_tests() helper function to multiprocessing.util: share
code between multiprocessing and concurrent.futures tests.
(cherry picked from commit 9707e8e22d80ca97bf7a9812816701cecde6d226)
(cherry picked from commit 35acb3597208e10a101140474adec86859d57f61)
* Remove NEWS about resource tracker
Python 3.7 multiprocessing does not have resource tracker.
Co-authored-by: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
files:
A Misc/NEWS.d/next/Tests/2019-07-01-19-57-26.bpo-37421.NFH1f0.rst
A Misc/NEWS.d/next/Tests/2019-07-05-14-47-55.bpo-37421.n8o2to.rst
M Lib/multiprocessing/forkserver.py
M Lib/multiprocessing/util.py
M Lib/test/_test_multiprocessing.py
M Lib/test/test_concurrent_futures.py
diff --git a/Lib/multiprocessing/forkserver.py b/Lib/multiprocessing/forkserver.py
index ffd1735848c45..ee5f76780bd0c 100644
--- a/Lib/multiprocessing/forkserver.py
+++ b/Lib/multiprocessing/forkserver.py
@@ -39,6 +39,25 @@ def __init__(self):
self._lock = threading.Lock()
self._preload_modules = ['__main__']
+ def _stop(self):
+ # Method used by unit tests to stop the server
+ with self._lock:
+ self._stop_unlocked()
+
+ def _stop_unlocked(self):
+ if self._forkserver_pid is None:
+ return
+
+ # close the "alive" file descriptor asks the server to stop
+ os.close(self._forkserver_alive_fd)
+ self._forkserver_alive_fd = None
+
+ os.waitpid(self._forkserver_pid, 0)
+ self._forkserver_pid = None
+
+ os.unlink(self._forkserver_address)
+ self._forkserver_address = None
+
def set_forkserver_preload(self, modules_names):
'''Set list of module names to try to load in forkserver process.'''
if not all(type(mod) is str for mod in self._preload_modules):
diff --git a/Lib/multiprocessing/util.py b/Lib/multiprocessing/util.py
index f5331abe821fb..327fe42d6efdb 100644
--- a/Lib/multiprocessing/util.py
+++ b/Lib/multiprocessing/util.py
@@ -456,3 +456,24 @@ def spawnv_passfds(path, args, passfds):
finally:
os.close(errpipe_read)
os.close(errpipe_write)
+
+
+def _cleanup_tests():
+ """Cleanup multiprocessing resources when multiprocessing tests
+ completed."""
+
+ from test import support
+
+ # cleanup multiprocessing
+ process._cleanup()
+
+ # Stop the ForkServer process if it's running
+ from multiprocessing import forkserver
+ forkserver._forkserver._stop()
+
+ # bpo-37421: Explicitly call _run_finalizers() to remove immediately
+ # temporary directories created by multiprocessing.util.get_temp_dir().
+ _run_finalizers()
+ support.gc_collect()
+
+ support.reap_children()
diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py
index a889ad3ae3918..23b9835a53775 100644
--- a/Lib/test/_test_multiprocessing.py
+++ b/Lib/test/_test_multiprocessing.py
@@ -5059,8 +5059,8 @@ def tearDownModule():
# Sleep 500 ms to give time to child processes to complete.
if need_sleep:
time.sleep(0.5)
- multiprocessing.process._cleanup()
- test.support.gc_collect()
+
+ multiprocessing.util._cleanup_tests()
remote_globs['setUpModule'] = setUpModule
remote_globs['tearDownModule'] = tearDownModule
diff --git a/Lib/test/test_concurrent_futures.py b/Lib/test/test_concurrent_futures.py
index b42670e16a6c9..a261bfeb4d373 100644
--- a/Lib/test/test_concurrent_futures.py
+++ b/Lib/test/test_concurrent_futures.py
@@ -26,6 +26,7 @@
BrokenExecutor)
from concurrent.futures.process import BrokenProcessPool
from multiprocessing import get_context
+import multiprocessing.util
def create_future(state=PENDING, exception=None, result=None):
@@ -1253,6 +1254,7 @@ def test_main():
test.support.run_unittest(__name__)
finally:
test.support.reap_children()
+ multiprocessing.util._cleanup_tests()
if __name__ == "__main__":
test_main()
diff --git a/Misc/NEWS.d/next/Tests/2019-07-01-19-57-26.bpo-37421.NFH1f0.rst b/Misc/NEWS.d/next/Tests/2019-07-01-19-57-26.bpo-37421.NFH1f0.rst
new file mode 100644
index 0000000000000..c379b504ba8a4
--- /dev/null
+++ b/Misc/NEWS.d/next/Tests/2019-07-01-19-57-26.bpo-37421.NFH1f0.rst
@@ -0,0 +1,2 @@
+multiprocessing tests now explicitly call ``_run_finalizers()`` to
+immediately remove temporary directories created by tests.
diff --git a/Misc/NEWS.d/next/Tests/2019-07-05-14-47-55.bpo-37421.n8o2to.rst b/Misc/NEWS.d/next/Tests/2019-07-05-14-47-55.bpo-37421.n8o2to.rst
new file mode 100644
index 0000000000000..136faa22d47a6
--- /dev/null
+++ b/Misc/NEWS.d/next/Tests/2019-07-05-14-47-55.bpo-37421.n8o2to.rst
@@ -0,0 +1,3 @@
+multiprocessing tests now stop the ForkServer instance if it's running: close
+the "alive" file descriptor to ask the server to stop and then remove its UNIX
+address.
More information about the Python-checkins
mailing list