[Python-checkins] gh-84559: Remove the new multiprocessing warning, too disruptive. (#101551)
gpshead
webhook-mailer at python.org
Fri Feb 3 18:20:57 EST 2023
https://github.com/python/cpython/commit/d4c410f0f922683f38c9d435923939d037fbd8c2
commit: d4c410f0f922683f38c9d435923939d037fbd8c2
branch: main
author: Gregory P. Smith <greg at krypto.org>
committer: gpshead <greg at krypto.org>
date: 2023-02-03T15:20:46-08:00
summary:
gh-84559: Remove the new multiprocessing warning, too disruptive. (#101551)
This reverts the core of #100618 while leaving relevant documentation
improvements and minor refactorings in place.
files:
D Lib/test/test_multiprocessing_defaults.py
D Misc/NEWS.d/next/Library/2023-01-01-01-19-33.gh-issue-84559.zEjsEJ.rst
M Doc/library/concurrent.futures.rst
M Doc/library/multiprocessing.rst
M Doc/whatsnew/3.12.rst
M Lib/concurrent/futures/process.py
M Lib/multiprocessing/context.py
M Lib/test/_test_multiprocessing.py
M Lib/test/test_concurrent_futures.py
M Lib/test/test_logging.py
M Lib/test/test_re.py
diff --git a/Doc/library/concurrent.futures.rst b/Doc/library/concurrent.futures.rst
index 10cffdaee0bb..c543c849585b 100644
--- a/Doc/library/concurrent.futures.rst
+++ b/Doc/library/concurrent.futures.rst
@@ -281,18 +281,18 @@ to a :class:`ProcessPoolExecutor` will result in deadlock.
Added the *initializer* and *initargs* arguments.
+ .. note::
+ The default :mod:`multiprocessing` start method
+ (see :ref:`multiprocessing-start-methods`) will change away from
+ *fork* in Python 3.14. Code that requires *fork* be used for their
+ :class:`ProcessPoolExecutor` should explicitly specify that by
+ passing a ``mp_context=multiprocessing.get_context("fork")``
+ parameter.
+
.. versionchanged:: 3.11
The *max_tasks_per_child* argument was added to allow users to
control the lifetime of workers in the pool.
- .. versionchanged:: 3.12
- The implicit use of the :mod:`multiprocessing` *fork* start method as a
- platform default (see :ref:`multiprocessing-start-methods`) now raises a
- :exc:`DeprecationWarning`. The default will change in Python 3.14.
- Code that requires *fork* should explicitly specify that when creating
- their :class:`ProcessPoolExecutor` by passing a
- ``mp_context=multiprocessing.get_context('fork')`` parameter.
-
.. _processpoolexecutor-example:
ProcessPoolExecutor Example
diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst
index c60b229ae2d0..0ec47bb956a9 100644
--- a/Doc/library/multiprocessing.rst
+++ b/Doc/library/multiprocessing.rst
@@ -126,6 +126,11 @@ to start a process. These *start methods* are
Available on POSIX systems. Currently the default on POSIX except macOS.
+ .. note::
+ The default start method will change away from *fork* in Python 3.14.
+ Code that requires *fork* should explicitly specify that via
+ :func:`get_context` or :func:`set_start_method`.
+
*forkserver*
When the program starts and selects the *forkserver* start method,
a server process is spawned. From then on, whenever a new process
@@ -138,11 +143,6 @@ to start a process. These *start methods* are
Available on POSIX platforms which support passing file descriptors
over Unix pipes such as Linux.
-.. versionchanged:: 3.12
- Implicit use of the *fork* start method as the default now raises a
- :exc:`DeprecationWarning`. Code that requires it should explicitly
- specify *fork* via :func:`get_context` or :func:`set_start_method`.
- The default will change away from *fork* in 3.14.
.. versionchanged:: 3.8
@@ -1107,6 +1107,7 @@ Miscellaneous
launched (before creating a :class:`Pool` or starting a :class:`Process`).
Only meaningful when using the ``'forkserver'`` start method.
+ See :ref:`multiprocessing-start-methods`.
.. versionadded:: 3.4
diff --git a/Doc/whatsnew/3.12.rst b/Doc/whatsnew/3.12.rst
index e675fada339a..0c5a70b64574 100644
--- a/Doc/whatsnew/3.12.rst
+++ b/Doc/whatsnew/3.12.rst
@@ -440,12 +440,6 @@ Deprecated
warning at compile time. This field will be removed in Python 3.14.
(Contributed by Ramvikrams and Kumar Aditya in :gh:`101193`. PEP by Ken Jin.)
-* Use of the implicit default ``'fork'`` start method for
- :mod:`multiprocessing` and :class:`concurrent.futures.ProcessPoolExecutor`
- now emits a :exc:`DeprecationWarning` on Linux and other non-macOS POSIX
- systems. Avoid this by explicitly specifying a start method.
- See :ref:`multiprocessing-start-methods`.
-
Pending Removal in Python 3.13
------------------------------
@@ -510,9 +504,13 @@ Pending Removal in Python 3.14
* Testing the truth value of an :class:`xml.etree.ElementTree.Element`
is deprecated and will raise an exception in Python 3.14.
-* The default :mod:`multiprocessing` start method will change to one of either
- ``'forkserver'`` or ``'spawn'`` on all platforms for which ``'fork'`` remains
- the default per :gh:`84559`.
+* The default :mod:`multiprocessing` start method will change to a safer one on
+ Linux, BSDs, and other non-macOS POSIX platforms where ``'fork'`` is currently
+ the default (:gh:`84559`). Adding a runtime warning about this was deemed too
+ disruptive as the majority of code is not expected to care. Use the
+ :func:`~multiprocessing.get_context` or
+ :func:`~multiprocessing.set_start_method` APIs to explicitly specify when
+ your code *requires* ``'fork'``. See :ref:`multiprocessing-start-methods`.
Pending Removal in Future Versions
----------------------------------
diff --git a/Lib/concurrent/futures/process.py b/Lib/concurrent/futures/process.py
index 257dd02fbc6c..bee162430a6f 100644
--- a/Lib/concurrent/futures/process.py
+++ b/Lib/concurrent/futures/process.py
@@ -57,7 +57,6 @@
import itertools
import sys
from traceback import format_exception
-import warnings
_threads_wakeups = weakref.WeakKeyDictionary()
@@ -651,22 +650,6 @@ def __init__(self, max_workers=None, mp_context=None,
mp_context = mp.get_context("spawn")
else:
mp_context = mp.get_context()
- if (mp_context.get_start_method() == "fork" and
- mp_context == mp.context._default_context._default_context):
- warnings.warn(
- "The default multiprocessing start method will change "
- "away from 'fork' in Python >= 3.14, per GH-84559. "
- "ProcessPoolExecutor uses multiprocessing. "
- "If your application requires the 'fork' multiprocessing "
- "start method, explicitly specify that by passing a "
- "mp_context= parameter. "
- "The safest start method is 'spawn'.",
- category=mp.context.DefaultForkDeprecationWarning,
- stacklevel=2,
- )
- # Avoid the equivalent warning from multiprocessing itself via
- # a non-default fork context.
- mp_context = mp.get_context("fork")
self._mp_context = mp_context
# https://github.com/python/cpython/issues/90622
diff --git a/Lib/multiprocessing/context.py b/Lib/multiprocessing/context.py
index 010a920540e8..de8a264829df 100644
--- a/Lib/multiprocessing/context.py
+++ b/Lib/multiprocessing/context.py
@@ -23,9 +23,6 @@ class TimeoutError(ProcessError):
class AuthenticationError(ProcessError):
pass
-class DefaultForkDeprecationWarning(DeprecationWarning):
- pass
-
#
# Base type for contexts. Bound methods of an instance of this type are included in __all__ of __init__.py
#
@@ -284,23 +281,6 @@ def _Popen(process_obj):
from .popen_fork import Popen
return Popen(process_obj)
- _warn_package_prefixes = (os.path.dirname(__file__),)
-
- class _DeprecatedForkProcess(ForkProcess):
- @classmethod
- def _Popen(cls, process_obj):
- import warnings
- warnings.warn(
- "The default multiprocessing start method will change "
- "away from 'fork' in Python >= 3.14, per GH-84559. "
- "Use multiprocessing.get_context(X) or .set_start_method(X) to "
- "explicitly specify it when your application requires 'fork'. "
- "The safest start method is 'spawn'.",
- category=DefaultForkDeprecationWarning,
- skip_file_prefixes=_warn_package_prefixes,
- )
- return super()._Popen(process_obj)
-
class SpawnProcess(process.BaseProcess):
_start_method = 'spawn'
@staticmethod
@@ -324,9 +304,6 @@ class ForkContext(BaseContext):
_name = 'fork'
Process = ForkProcess
- class _DefaultForkContext(ForkContext):
- Process = _DeprecatedForkProcess
-
class SpawnContext(BaseContext):
_name = 'spawn'
Process = SpawnProcess
@@ -342,16 +319,13 @@ def _check_available(self):
'fork': ForkContext(),
'spawn': SpawnContext(),
'forkserver': ForkServerContext(),
- # Remove None and _DefaultForkContext() when changing the default
- # in 3.14 for https://github.com/python/cpython/issues/84559.
- None: _DefaultForkContext(),
}
if sys.platform == 'darwin':
# bpo-33725: running arbitrary code after fork() is no longer reliable
# on macOS since macOS 10.14 (Mojave). Use spawn by default instead.
_default_context = DefaultContext(_concrete_contexts['spawn'])
else:
- _default_context = DefaultContext(_concrete_contexts[None])
+ _default_context = DefaultContext(_concrete_contexts['fork'])
else:
diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py
index e4a60a4d6746..9a2db24b4bd5 100644
--- a/Lib/test/_test_multiprocessing.py
+++ b/Lib/test/_test_multiprocessing.py
@@ -4098,10 +4098,9 @@ def test_shared_memory_SharedMemoryServer_ignores_sigint(self):
def test_shared_memory_SharedMemoryManager_reuses_resource_tracker(self):
# bpo-36867: test that a SharedMemoryManager uses the
# same resource_tracker process as its parent.
- cmd = f'''if 1:
+ cmd = '''if 1:
from multiprocessing.managers import SharedMemoryManager
- from multiprocessing import set_start_method
- set_start_method({multiprocessing.get_start_method()!r})
+
smm = SharedMemoryManager()
smm.start()
diff --git a/Lib/test/test_concurrent_futures.py b/Lib/test/test_concurrent_futures.py
index 4493cd312528..b3520ae3994e 100644
--- a/Lib/test/test_concurrent_futures.py
+++ b/Lib/test/test_concurrent_futures.py
@@ -18,7 +18,6 @@
import threading
import time
import unittest
-import warnings
import weakref
from pickle import PicklingError
@@ -572,24 +571,6 @@ def test_shutdown_no_wait(self):
assert all([r == abs(v) for r, v in zip(res, range(-5, 5))])
- at unittest.skipIf(mp.get_all_start_methods()[0] != "fork", "non-fork default.")
-class ProcessPoolExecutorDefaultForkWarning(unittest.TestCase):
- def test_fork_default_warns(self):
- with self.assertWarns(mp.context.DefaultForkDeprecationWarning):
- with futures.ProcessPoolExecutor(2):
- pass
-
- def test_explicit_fork_does_not_warn(self):
- with warnings.catch_warnings(record=True) as ws:
- warnings.simplefilter("ignore")
- warnings.filterwarnings(
- 'always', category=mp.context.DefaultForkDeprecationWarning)
- ctx = mp.get_context("fork") # Non-default fork context.
- with futures.ProcessPoolExecutor(2, mp_context=ctx):
- pass
- self.assertEqual(len(ws), 0, msg=[str(x) for x in ws])
-
-
create_executor_tests(ProcessPoolShutdownTest,
executor_mixins=(ProcessPoolForkMixin,
ProcessPoolForkserverMixin,
diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py
index 8a12d570f26f..072056d37221 100644
--- a/Lib/test/test_logging.py
+++ b/Lib/test/test_logging.py
@@ -4759,9 +4759,8 @@ def test_multiprocessing(self):
# In other processes, processName is correct when multiprocessing in imported,
# but it is (incorrectly) defaulted to 'MainProcess' otherwise (bpo-38762).
import multiprocessing
- mp = multiprocessing.get_context('spawn')
- parent_conn, child_conn = mp.Pipe()
- p = mp.Process(
+ parent_conn, child_conn = multiprocessing.Pipe()
+ p = multiprocessing.Process(
target=self._extract_logrecord_process_name,
args=(2, LOG_MULTI_PROCESSING, child_conn,)
)
diff --git a/Lib/test/test_multiprocessing_defaults.py b/Lib/test/test_multiprocessing_defaults.py
deleted file mode 100644
index 7ea872fef20d..000000000000
--- a/Lib/test/test_multiprocessing_defaults.py
+++ /dev/null
@@ -1,85 +0,0 @@
-"""Test default behavior of multiprocessing."""
-
-from inspect import currentframe, getframeinfo
-import multiprocessing
-from multiprocessing.context import DefaultForkDeprecationWarning
-import sys
-from test.support import import_helper, threading_helper
-import unittest
-import warnings
-
-# Skip tests if _multiprocessing wasn't built.
-import_helper.import_module('_multiprocessing')
-
-
-def do_nothing():
- pass
-
-
-# Process has the same API as Thread so this helper works.
-join_process = threading_helper.join_thread
-
-
-class DefaultWarningsTest(unittest.TestCase):
-
- @unittest.skipIf(sys.platform in ('win32', 'darwin'),
- 'The default is not "fork" on Windows or macOS.')
- def setUp(self):
- self.assertEqual(multiprocessing.get_start_method(), 'fork')
- self.assertIsInstance(multiprocessing.get_context(),
- multiprocessing.context._DefaultForkContext)
-
- def test_default_fork_start_method_warning_process(self):
- with warnings.catch_warnings(record=True) as ws:
- warnings.simplefilter('ignore')
- warnings.filterwarnings('always', category=DefaultForkDeprecationWarning)
- process = multiprocessing.Process(target=do_nothing)
- process.start() # warning should point here.
- join_process(process)
- self.assertIsInstance(ws[0].message, DefaultForkDeprecationWarning)
- self.assertIn(__file__, ws[0].filename)
- self.assertEqual(getframeinfo(currentframe()).lineno-4, ws[0].lineno)
- self.assertIn("'fork'", str(ws[0].message))
- self.assertIn("get_context", str(ws[0].message))
- self.assertEqual(len(ws), 1, msg=[str(x) for x in ws])
-
- def test_default_fork_start_method_warning_pool(self):
- with warnings.catch_warnings(record=True) as ws:
- warnings.simplefilter('ignore')
- warnings.filterwarnings('always', category=DefaultForkDeprecationWarning)
- pool = multiprocessing.Pool(1) # warning should point here.
- pool.terminate()
- pool.join()
- self.assertIsInstance(ws[0].message, DefaultForkDeprecationWarning)
- self.assertIn(__file__, ws[0].filename)
- self.assertEqual(getframeinfo(currentframe()).lineno-5, ws[0].lineno)
- self.assertIn("'fork'", str(ws[0].message))
- self.assertIn("get_context", str(ws[0].message))
- self.assertEqual(len(ws), 1, msg=[str(x) for x in ws])
-
- def test_default_fork_start_method_warning_manager(self):
- with warnings.catch_warnings(record=True) as ws:
- warnings.simplefilter('ignore')
- warnings.filterwarnings('always', category=DefaultForkDeprecationWarning)
- manager = multiprocessing.Manager() # warning should point here.
- manager.shutdown()
- self.assertIsInstance(ws[0].message, DefaultForkDeprecationWarning)
- self.assertIn(__file__, ws[0].filename)
- self.assertEqual(getframeinfo(currentframe()).lineno-4, ws[0].lineno)
- self.assertIn("'fork'", str(ws[0].message))
- self.assertIn("get_context", str(ws[0].message))
- self.assertEqual(len(ws), 1, msg=[str(x) for x in ws])
-
- def test_no_mp_warning_when_using_explicit_fork_context(self):
- with warnings.catch_warnings(record=True) as ws:
- warnings.simplefilter('ignore')
- warnings.filterwarnings('always', category=DefaultForkDeprecationWarning)
- fork_mp = multiprocessing.get_context('fork')
- pool = fork_mp.Pool(1)
- pool.terminate()
- pool.join()
- self.assertEqual(len(ws), 0, msg=[str(x) for x in ws])
-
-
-if __name__ == '__main__':
- unittest.main()
diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py
index eacb1a7c82a5..11628a236ade 100644
--- a/Lib/test/test_re.py
+++ b/Lib/test/test_re.py
@@ -2431,8 +2431,7 @@ def test_regression_gh94675(self):
input_js = '''a(function() {
///////////////////////////////////////////////////////////////////
});'''
- mp = multiprocessing.get_context('spawn')
- p = mp.Process(target=pattern.sub, args=('', input_js))
+ p = multiprocessing.Process(target=pattern.sub, args=('', input_js))
p.start()
p.join(SHORT_TIMEOUT)
try:
diff --git a/Misc/NEWS.d/next/Library/2023-01-01-01-19-33.gh-issue-84559.zEjsEJ.rst b/Misc/NEWS.d/next/Library/2023-01-01-01-19-33.gh-issue-84559.zEjsEJ.rst
deleted file mode 100644
index 3793e0f1fddb..000000000000
--- a/Misc/NEWS.d/next/Library/2023-01-01-01-19-33.gh-issue-84559.zEjsEJ.rst
+++ /dev/null
@@ -1,11 +0,0 @@
-The :mod:`multiprocessing` module and
-:class:`concurrent.futures.ProcessPoolExecutor` will emit a
-:exc:`DeprecationWarning` on Linux and other non-macOS POSIX systems when
-the default multiprocessing start method of ``'fork'`` is used implicitly
-rather than being explicitly specified through a
-:func:`multiprocessing.get_context` context.
-
-This is in preparation for default start method to change in Python 3.14 to
-a default that is safe for multithreaded applications.
-
-Windows and macOS are unaffected as their default start method is ``spawn``.
More information about the Python-checkins
mailing list