[Python-checkins] gh-107305: Update the C-API Docs for PEP 684 (gh-107324)
ericsnowcurrently
webhook-mailer at python.org
Fri Jul 28 15:46:59 EDT 2023
https://github.com/python/cpython/commit/c0b81c4b5438a3565fadd9d6f5bc69f989a3fdee
commit: c0b81c4b5438a3565fadd9d6f5bc69f989a3fdee
branch: main
author: Eric Snow <ericsnowcurrently at gmail.com>
committer: ericsnowcurrently <ericsnowcurrently at gmail.com>
date: 2023-07-28T13:46:55-06:00
summary:
gh-107305: Update the C-API Docs for PEP 684 (gh-107324)
files:
A Misc/NEWS.d/next/Documentation/2023-07-26-16-33-04.gh-issue-107305.qB2LS4.rst
M Doc/c-api/init.rst
diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
index a447eefc55e44..60912a1ecaede 100644
--- a/Doc/c-api/init.rst
+++ b/Doc/c-api/init.rst
@@ -1226,7 +1226,95 @@ You can switch between sub-interpreters using the :c:func:`PyThreadState_Swap`
function. You can create and destroy them using the following functions:
-.. c:function:: PyThreadState* Py_NewInterpreter()
+.. c:type:: PyInterpreterConfig
+
+ Structure containing most parameters to configure a sub-interpreter.
+ Its values are used only in :c:func:`Py_NewInterpreterFromConfig` and
+ never modified by the runtime.
+
+ .. versionadded:: 3.12
+
+ Structure fields:
+
+ .. c:member:: int use_main_obmalloc
+
+ If this is ``0`` then the sub-interpreter will use its own
+ "object" allocator state.
+ Otherwise it will use (share) the main interpreter's.
+
+ If this is ``0`` then
+ :c:member:`~PyInterpreterConfig.check_multi_interp_extensions`
+ must be ``1`` (non-zero).
+ If this is ``1`` then :c:member:`~PyInterpreterConfig.gil`
+ must not be :c:macro:`PyInterpreterConfig_OWN_GIL`.
+
+ .. c:member:: int allow_fork
+
+ If this is ``0`` then the runtime will not support forking the
+ process in any thread where the sub-interpreter is currently active.
+ Otherwise fork is unrestricted.
+
+ Note that the :mod:`subprocess` module still works
+ when fork is disallowed.
+
+ .. c:member:: int allow_exec
+
+ If this is ``0`` then the runtime will not support replacing the
+ current process via exec (e.g. :func:`os.execv`) in any thread
+ where the sub-interpreter is currently active.
+ Otherwise exec is unrestricted.
+
+ Note that the :mod:`subprocess` module still works
+ when exec is disallowed.
+
+ .. c:member:: int allow_threads
+
+ If this is ``0`` then the sub-interpreter's :mod:`threading` module
+ won't create threads.
+ Otherwise threads are allowed.
+
+ .. c:member:: int allow_daemon_threads
+
+ If this is ``0`` then the sub-interpreter's :mod:`threading` module
+ won't create daemon threads.
+ Otherwise daemon threads are allowed (as long as
+ :c:member:`~PyInterpreterConfig.allow_threads` is non-zero).
+
+ .. c:member:: int check_multi_interp_extensions
+
+ If this is ``0`` then all extension modules may be imported,
+ including legacy (single-phase init) modules,
+ in any thread where the sub-interpreter is currently active.
+ Otherwise only multi-phase init extension modules
+ (see :pep:`489`) may be imported.
+
+ This must be ``1`` (non-zero) if
+ :c:member:`~PyInterpreterConfig.use_main_obmalloc` is ``0``.
+
+ .. c:member:: int gil
+
+ This determines the operation of the GIL for the sub-interpreter.
+ It may be one of the following:
+
+ .. c:namespace:: NULL
+
+ .. c:macro:: PyInterpreterConfig_DEFAULT_GIL
+
+ Use the default selection (:c:macro:`PyInterpreterConfig_SHARED_GIL`).
+
+ .. c:macro:: PyInterpreterConfig_SHARED_GIL
+
+ Use (share) the main interpreter's GIL.
+
+ .. c:macro:: PyInterpreterConfig_OWN_GIL
+
+ Use the sub-interpreter's own GIL.
+
+ If this is :c:macro:`PyInterpreterConfig_OWN_GIL` then
+ :c:member:`PyInterpreterConfig.use_main_obmalloc` must be ``0``.
+
+
+.. c:function:: PyStatus Py_NewInterpreterFromConfig(PyThreadState **tstate_p, const PyInterpreterConfig *config)
.. index::
pair: module; builtins
@@ -1246,16 +1334,47 @@ function. You can create and destroy them using the following functions:
``sys.stdout`` and ``sys.stderr`` (however these refer to the same underlying
file descriptors).
- The return value points to the first thread state created in the new
+ The given *config* controls the options with which the interpreter
+ is initialized.
+
+ Upon success, *tstate_p* will be set to the first thread state
+ created in the new
sub-interpreter. This thread state is made in the current thread state.
Note that no actual thread is created; see the discussion of thread states
- below. If creation of the new interpreter is unsuccessful, ``NULL`` is
- returned; no exception is set since the exception state is stored in the
- current thread state and there may not be a current thread state. (Like all
- other Python/C API functions, the global interpreter lock must be held before
- calling this function and is still held when it returns; however, unlike most
- other Python/C API functions, there needn't be a current thread state on
- entry.)
+ below. If creation of the new interpreter is unsuccessful,
+ *tstate_p* is set to ``NULL``;
+ no exception is set since the exception state is stored in the
+ current thread state and there may not be a current thread state.
+
+ Like all other Python/C API functions, the global interpreter lock
+ must be held before calling this function and is still held when it
+ returns. Likewise a current thread state must be set on entry. On
+ success, the returned thread state will be set as current. If the
+ sub-interpreter is created with its own GIL then the GIL of the
+ calling interpreter will be released. When the function returns,
+ the new interpreter's GIL will be held by the current thread and
+ the previously interpreter's GIL will remain released here.
+
+ .. versionadded:: 3.12
+
+ Sub-interpreters are most effective when isolated from each other,
+ with certain functionality restricted::
+
+ PyInterpreterConfig config = {
+ .use_main_obmalloc = 0,
+ .allow_fork = 0,
+ .allow_exec = 0,
+ .allow_threads = 1,
+ .allow_daemon_threads = 0,
+ .check_multi_interp_extensions = 1,
+ .gil = PyInterpreterConfig_OWN_GIL,
+ };
+ PyThreadState *tstate = Py_NewInterpreterFromConfig(&config);
+
+ Note that the config is used only briefly and does not get modified.
+ During initialization the config's values are converted into various
+ :c:type:`PyInterpreterState` values. A read-only copy of the config
+ may be stored internally on the :c:type:`PyInterpreterState`.
.. index::
single: Py_FinalizeEx()
@@ -1290,19 +1409,79 @@ function. You can create and destroy them using the following functions:
.. index:: single: close() (in module os)
+.. c:function:: PyThreadState* Py_NewInterpreter(void)
+
+ .. index::
+ pair: module; builtins
+ pair: module; __main__
+ pair: module; sys
+ single: stdout (in module sys)
+ single: stderr (in module sys)
+ single: stdin (in module sys)
+
+ Create a new sub-interpreter. This is essentially just a wrapper
+ around :c:func:`Py_NewInterpreterFromConfig` with a config that
+ preserves the existing behavior. The result is an unisolated
+ sub-interpreter that shares the main interpreter's GIL, allows
+ fork/exec, allows daemon threads, and allows single-phase init
+ modules.
+
+
.. c:function:: void Py_EndInterpreter(PyThreadState *tstate)
.. index:: single: Py_FinalizeEx()
- Destroy the (sub-)interpreter represented by the given thread state. The given
- thread state must be the current thread state. See the discussion of thread
- states below. When the call returns, the current thread state is ``NULL``. All
- thread states associated with this interpreter are destroyed. (The global
- interpreter lock must be held before calling this function and is still held
- when it returns.) :c:func:`Py_FinalizeEx` will destroy all sub-interpreters that
+ Destroy the (sub-)interpreter represented by the given thread state.
+ The given thread state must be the current thread state. See the
+ discussion of thread states below. When the call returns,
+ the current thread state is ``NULL``. All thread states associated
+ with this interpreter are destroyed. The global interpreter lock
+ used by the target interpreter must be held before calling this
+ function. No GIL is held when it returns.
+
+ :c:func:`Py_FinalizeEx` will destroy all sub-interpreters that
haven't been explicitly destroyed at that point.
+A Per-Interpreter GIL
+---------------------
+
+Using :c:func:`Py_NewInterpreterFromConfig` you can create
+a sub-interpreter that is completely isolated from other interpreters,
+including having its own GIL. The most important benefit of this
+isolation is that such an interpreter can execute Python code without
+being blocked by other interpreters or blocking any others. Thus a
+single Python process can truly take advantage of multiple CPU cores
+when running Python code. The isolation also encourages a different
+approach to concurrency than that of just using threads.
+(See :pep:`554`.)
+
+Using an isolated interpreter requires vigilance in preserving that
+isolation. That especially means not sharing any objects or mutable
+state without guarantees about thread-safety. Even objects that are
+otherwise immutable (e.g. ``None``, ``(1, 5)``) can't normally be shared
+because of the refcount. One simple but less-efficient approach around
+this is to use a global lock around all use of some state (or object).
+Alternately, effectively immutable objects (like integers or strings)
+can be made safe in spite of their refcounts by making them "immortal".
+In fact, this has been done for the builtin singletons, small integers,
+and a number of other builtin objects.
+
+If you preserve isolation then you will have access to proper multi-core
+computing without the complications that come with free-threading.
+Failure to preserve isolation will expose you to the full consequences
+of free-threading, including races and hard-to-debug crashes.
+
+Aside from that, one of the main challenges of using multiple isolated
+interpreters is how to communicate between them safely (not break
+isolation) and efficiently. The runtime and stdlib do not provide
+any standard approach to this yet. A future stdlib module would help
+mitigate the effort of preserving isolation and expose effective tools
+for communicating (and sharing) data between interpreters.
+
+.. versionadded:: 3.12
+
+
Bugs and caveats
----------------
diff --git a/Misc/NEWS.d/next/Documentation/2023-07-26-16-33-04.gh-issue-107305.qB2LS4.rst b/Misc/NEWS.d/next/Documentation/2023-07-26-16-33-04.gh-issue-107305.qB2LS4.rst
new file mode 100644
index 0000000000000..038f9e68a5422
--- /dev/null
+++ b/Misc/NEWS.d/next/Documentation/2023-07-26-16-33-04.gh-issue-107305.qB2LS4.rst
@@ -0,0 +1,3 @@
+Add documentation for :c:type:`PyInterpreterConfig` and
+:c:func:`Py_NewInterpreterFromConfig`. Also clarify some of the nearby docs
+relative to per-interpreter GIL.
More information about the Python-checkins
mailing list