[Python-checkins] gh-103509: PEP 697 -- Limited C API for Extending Opaque Types (GH-103511)
encukou
webhook-mailer at python.org
Thu May 4 03:57:29 EDT 2023
https://github.com/python/cpython/commit/cd9a56c2b0e14f56f2e83dd4db43c5c69a74b232
commit: cd9a56c2b0e14f56f2e83dd4db43c5c69a74b232
branch: main
author: Petr Viktorin <encukou at gmail.com>
committer: encukou <encukou at gmail.com>
date: 2023-05-04T09:56:53+02:00
summary:
gh-103509: PEP 697 -- Limited C API for Extending Opaque Types (GH-103511)
Co-authored-by: Oleg Iarygin <oleg at arhadthedev.net>
Co-authored-by: Erlend E. Aasland <erlend.aasland at protonmail.com>
files:
A Misc/NEWS.d/next/C API/2023-04-13-16-54-00.gh-issue-103509.A26Qu8.rst
A Modules/_testcapi/heaptype_relative.c
M Doc/c-api/object.rst
M Doc/c-api/structures.rst
M Doc/c-api/type.rst
M Doc/c-api/typeobj.rst
M Doc/data/stable_abi.dat
M Doc/whatsnew/3.12.rst
M Include/cpython/object.h
M Include/descrobject.h
M Include/internal/pycore_object.h
M Include/object.h
M Include/pyport.h
M Lib/test/test_capi/test_misc.py
M Lib/test/test_stable_abi_ctypes.py
M Misc/stable_abi.toml
M Modules/Setup.stdlib.in
M Modules/_testcapi/heaptype.c
M Modules/_testcapi/parts.h
M Modules/_testcapimodule.c
M Objects/descrobject.c
M Objects/typeobject.c
M PC/pyconfig.h
M PC/python3dll.c
M PCbuild/_testcapi.vcxproj
M PCbuild/_testcapi.vcxproj.filters
M Python/structmember.c
M configure
M configure.ac
M pyconfig.h.in
diff --git a/Doc/c-api/object.rst b/Doc/c-api/object.rst
index 0a12bb9e8c54..a0c3194ab0fb 100644
--- a/Doc/c-api/object.rst
+++ b/Doc/c-api/object.rst
@@ -395,3 +395,42 @@ Object Protocol
returns ``NULL`` if the object cannot be iterated.
.. versionadded:: 3.10
+
+.. c:function:: void *PyObject_GetTypeData(PyObject *o, PyTypeObject *cls)
+
+ Get a pointer to subclass-specific data reserved for *cls*.
+
+ The object *o* must be an instance of *cls*, and *cls* must have been
+ created using negative :c:member:`PyType_Spec.basicsize`.
+ Python does not check this.
+
+ On error, set an exception and return ``NULL``.
+
+ .. versionadded:: 3.12
+
+.. c:function:: Py_ssize_t PyType_GetTypeDataSize(PyTypeObject *cls)
+
+ Return the size of the instance memory space reserved for *cls*, i.e. the size of the
+ memory :c:func:`PyObject_GetTypeData` returns.
+
+ This may be larger than requested using :c:member:`-PyType_Spec.basicsize <PyType_Spec.basicsize>`;
+ it is safe to use this larger size (e.g. with :c:func:`!memset`).
+
+ The type *cls* **must** have been created using
+ negative :c:member:`PyType_Spec.basicsize`.
+ Python does not check this.
+
+ On error, set an exception and return a negative value.
+
+ .. versionadded:: 3.12
+
+.. c:function:: void *PyObject_GetItemData(PyObject *o)
+
+ Get a pointer to per-item data for a class with
+ :const:`Py_TPFLAGS_ITEMS_AT_END`.
+
+ On error, set an exception and return ``NULL``.
+ :py:exc:`TypeError` is raised if *o* does not have
+ :const:`Py_TPFLAGS_ITEMS_AT_END` set.
+
+ .. versionadded:: 3.12
diff --git a/Doc/c-api/structures.rst b/Doc/c-api/structures.rst
index 9618a0cf6769..338db6378d24 100644
--- a/Doc/c-api/structures.rst
+++ b/Doc/c-api/structures.rst
@@ -486,6 +486,22 @@ The following flags can be used with :c:member:`PyMemberDef.flags`:
Emit an ``object.__getattr__`` :ref:`audit event <audit-events>`
before reading.
+.. c:macro:: Py_RELATIVE_OFFSET
+
+ Indicates that the :c:member:`~PyMemberDef.offset` of this ``PyMemberDef``
+ entry indicates an offset from the subclass-specific data, rather than
+ from ``PyObject``.
+
+ Can only be used as part of :c:member:`Py_tp_members <PyTypeObject.tp_members>`
+ :c:type:`slot <PyTypeSlot>` when creating a class using negative
+ :c:member:`~PyTypeDef.basicsize`.
+ It is mandatory in that case.
+
+ This flag is only used in :c:type:`PyTypeSlot`.
+ When setting :c:member:`~PyTypeObject.tp_members` during
+ class creation, Python clears it and sets
+ :c:member:`PyMemberDef.offset` to the offset from the ``PyObject`` struct.
+
.. index::
single: READ_RESTRICTED
single: WRITE_RESTRICTED
diff --git a/Doc/c-api/type.rst b/Doc/c-api/type.rst
index 9fd40e1008c4..c21fd9255d28 100644
--- a/Doc/c-api/type.rst
+++ b/Doc/c-api/type.rst
@@ -353,25 +353,57 @@ The following functions and structs are used to create
Structure defining a type's behavior.
- .. c:member:: const char* PyType_Spec.name
+ .. c:member:: const char* name
Name of the type, used to set :c:member:`PyTypeObject.tp_name`.
- .. c:member:: int PyType_Spec.basicsize
- .. c:member:: int PyType_Spec.itemsize
+ .. c:member:: int basicsize
- Size of the instance in bytes, used to set
- :c:member:`PyTypeObject.tp_basicsize` and
- :c:member:`PyTypeObject.tp_itemsize`.
+ If positive, specifies the size of the instance in bytes.
+ It is used to set :c:member:`PyTypeObject.tp_basicsize`.
- .. c:member:: int PyType_Spec.flags
+ If zero, specifies that :c:member:`~PyTypeObject.tp_basicsize`
+ should be inherited.
+
+ If negative, the absolute value specifies how much space instances of the
+ class need *in addition* to the superclass.
+ Use :c:func:`PyObject_GetTypeData` to get a pointer to subclass-specific
+ memory reserved this way.
+
+ .. versionchanged:: 3.12
+
+ Previously, this field could not be negative.
+
+ .. c:member:: int itemsize
+
+ Size of one element of a variable-size type, in bytes.
+ Used to set :c:member:`PyTypeObject.tp_itemsize`.
+ See ``tp_itemsize`` documentation for caveats.
+
+ If zero, :c:member:`~PyTypeObject.tp_itemsize` is inherited.
+ Extending arbitrary variable-sized classes is dangerous,
+ since some types use a fixed offset for variable-sized memory,
+ which can then overlap fixed-sized memory used by a subclass.
+ To help prevent mistakes, inheriting ``itemsize`` is only possible
+ in the following situations:
+
+ - The base is not variable-sized (its
+ :c:member:`~PyTypeObject.tp_itemsize`).
+ - The requested :c:member:`PyType_Spec.basicsize` is positive,
+ suggesting that the memory layout of the base class is known.
+ - The requested :c:member:`PyType_Spec.basicsize` is zero,
+ suggesting that the subclass does not access the instance's memory
+ directly.
+ - With the :const:`Py_TPFLAGS_ITEMS_AT_END` flag.
+
+ .. c:member:: unsigned int flags
Type flags, used to set :c:member:`PyTypeObject.tp_flags`.
If the ``Py_TPFLAGS_HEAPTYPE`` flag is not set,
:c:func:`PyType_FromSpecWithBases` sets it automatically.
- .. c:member:: PyType_Slot *PyType_Spec.slots
+ .. c:member:: PyType_Slot *slots
Array of :c:type:`PyType_Slot` structures.
Terminated by the special slot value ``{0, NULL}``.
diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst
index e963b90628aa..e13db3fb2211 100644
--- a/Doc/c-api/typeobj.rst
+++ b/Doc/c-api/typeobj.rst
@@ -1171,6 +1171,26 @@ and :c:type:`PyType_Type` effectively act as defaults.)
:c:member:`~PyTypeObject.tp_weaklistoffset` field is set in a superclass.
+ .. data:: Py_TPFLAGS_ITEMS_AT_END
+
+ Only usable with variable-size types, i.e. ones with non-zero
+ :c:member:`~PyObject.tp_itemsize`.
+
+ Indicates that the variable-sized portion of an instance of this type is
+ at the end of the instance's memory area, at an offset of
+ :c:expr:`Py_TYPE(obj)->tp_basicsize` (which may be different in each
+ subclass).
+
+ When setting this flag, be sure that all superclasses either
+ use this memory layout, or are not variable-sized.
+ Python does not check this.
+
+ .. versionadded:: 3.12
+
+ **Inheritance:**
+
+ This flag is inherited.
+
.. XXX Document more flags here?
diff --git a/Doc/data/stable_abi.dat b/Doc/data/stable_abi.dat
index 4cc06d22baaa..f112d268129f 100644
--- a/Doc/data/stable_abi.dat
+++ b/Doc/data/stable_abi.dat
@@ -521,6 +521,7 @@ function,PyObject_GetAttrString,3.2,,
function,PyObject_GetBuffer,3.11,,
function,PyObject_GetItem,3.2,,
function,PyObject_GetIter,3.2,,
+function,PyObject_GetTypeData,3.12,,
function,PyObject_HasAttr,3.2,,
function,PyObject_HasAttrString,3.2,,
function,PyObject_Hash,3.2,,
@@ -675,6 +676,7 @@ function,PyType_GetModuleState,3.10,,
function,PyType_GetName,3.11,,
function,PyType_GetQualName,3.11,,
function,PyType_GetSlot,3.4,,
+function,PyType_GetTypeDataSize,3.12,,
function,PyType_IsSubtype,3.2,,
function,PyType_Modified,3.2,,
function,PyType_Ready,3.2,,
diff --git a/Doc/whatsnew/3.12.rst b/Doc/whatsnew/3.12.rst
index 3dfd787e3df8..edbf92146755 100644
--- a/Doc/whatsnew/3.12.rst
+++ b/Doc/whatsnew/3.12.rst
@@ -1159,6 +1159,21 @@ New Features
(Contributed by Petr Viktorin in :gh:`101101`.)
+* :pep:`697`: Added API for extending types whose instance memory layout is
+ opaque:
+
+ - :c:member:`PyType_Spec.basicsize` can be zero or negative to specify
+ inheriting or extending the base class size.
+ - :c:func:`PyObject_GetTypeData` and :c:func:`PyType_GetTypeDataSize`
+ added to allow access to subclass-specific instance data.
+ - :const:`Py_TPFLAGS_ITEMS_AT_END` and :c:func:`PyObject_GetItemData`
+ added to allow safely extending certain variable-sized types, including
+ :c:var:`PyType_Type`.
+ - :c:macro:`Py_RELATIVE_OFFSET` added to allow defining
+ :c:type:`members <PyMemberDef>` in terms of a subclass-specific struct.
+
+ (Contributed by Petr Viktorin in :gh:`103509`.)
+
* Added the new limited C API function :c:func:`PyType_FromMetaclass`,
which generalizes the existing :c:func:`PyType_FromModuleAndSpec` using
an additional metaclass argument.
diff --git a/Include/cpython/object.h b/Include/cpython/object.h
index ce4d13cd9c28..d8eff691039d 100644
--- a/Include/cpython/object.h
+++ b/Include/cpython/object.h
@@ -553,6 +553,7 @@ Py_DEPRECATED(3.11) typedef int UsingDeprecatedTrashcanMacro;
Py_TRASHCAN_END; \
} while(0);
+PyAPI_FUNC(void *) PyObject_GetItemData(PyObject *obj);
PyAPI_FUNC(int) _PyObject_VisitManagedDict(PyObject *obj, visitproc visit, void *arg);
PyAPI_FUNC(void) _PyObject_ClearManagedDict(PyObject *obj);
diff --git a/Include/descrobject.h b/Include/descrobject.h
index 0a420b865dfd..fd66d17b497a 100644
--- a/Include/descrobject.h
+++ b/Include/descrobject.h
@@ -83,6 +83,7 @@ struct PyMemberDef {
#define Py_READONLY 1
#define Py_AUDIT_READ 2 // Added in 3.10, harmless no-op before that
#define _Py_WRITE_RESTRICTED 4 // Deprecated, no-op. Do not reuse the value.
+#define Py_RELATIVE_OFFSET 8
PyAPI_FUNC(PyObject *) PyMember_GetOne(const char *, PyMemberDef *);
PyAPI_FUNC(int) PyMember_SetOne(char *, PyMemberDef *, PyObject *);
diff --git a/Include/internal/pycore_object.h b/Include/internal/pycore_object.h
index 91853ad0525b..b9e700ea280c 100644
--- a/Include/internal/pycore_object.h
+++ b/Include/internal/pycore_object.h
@@ -389,11 +389,6 @@ extern PyObject ** _PyObject_ComputedDictPointer(PyObject *);
extern void _PyObject_FreeInstanceAttributes(PyObject *obj);
extern int _PyObject_IsInstanceDictEmpty(PyObject *);
-// Access macro to the members which are floating "behind" the object
-static inline PyMemberDef* _PyHeapType_GET_MEMBERS(PyHeapTypeObject *etype) {
- return (PyMemberDef*)((char*)etype + Py_TYPE(etype)->tp_basicsize);
-}
-
PyAPI_FUNC(PyObject *) _PyObject_LookupSpecial(PyObject *, PyObject *);
/* C function call trampolines to mitigate bad function pointer casts.
diff --git a/Include/object.h b/Include/object.h
index 66c3df0d7f78..81aeb2d8bd5a 100644
--- a/Include/object.h
+++ b/Include/object.h
@@ -355,6 +355,8 @@ PyAPI_FUNC(PyObject *) PyType_GetQualName(PyTypeObject *);
#endif
#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x030C0000
PyAPI_FUNC(PyObject *) PyType_FromMetaclass(PyTypeObject*, PyObject*, PyType_Spec*, PyObject*);
+PyAPI_FUNC(void *) PyObject_GetTypeData(PyObject *obj, PyTypeObject *cls);
+PyAPI_FUNC(Py_ssize_t) PyType_GetTypeDataSize(PyTypeObject *cls);
#endif
/* Generic type check */
@@ -521,6 +523,9 @@ given type object has a specified feature.
// subject itself (rather than a mapped attribute on it):
#define _Py_TPFLAGS_MATCH_SELF (1UL << 22)
+/* Items (ob_size*tp_itemsize) are found at the end of an instance's memory */
+#define Py_TPFLAGS_ITEMS_AT_END (1UL << 23)
+
/* These flags are used to determine if a type is a subclass. */
#define Py_TPFLAGS_LONG_SUBCLASS (1UL << 24)
#define Py_TPFLAGS_LIST_SUBCLASS (1UL << 25)
diff --git a/Include/pyport.h b/Include/pyport.h
index bd0ba6d0681b..d7c6ae64f2bf 100644
--- a/Include/pyport.h
+++ b/Include/pyport.h
@@ -765,4 +765,15 @@ extern char * _getpty(int *, int, mode_t, int);
#undef __bool__
#endif
+// Make sure we have maximum alignment, even if the current compiler
+// does not support max_align_t. Note that:
+// - Autoconf reports alignment of unknown types to 0.
+// - 'long double' has maximum alignment on *most* platforms,
+// looks like the best we can do for pre-C11 compilers.
+// - The value is tested, see test_alignof_max_align_t
+#if !defined(ALIGNOF_MAX_ALIGN_T) || ALIGNOF_MAX_ALIGN_T == 0
+# undef ALIGNOF_MAX_ALIGN_T
+# define ALIGNOF_MAX_ALIGN_T _Alignof(long double)
+#endif
+
#endif /* Py_PYPORT_H */
diff --git a/Lib/test/test_capi/test_misc.py b/Lib/test/test_capi/test_misc.py
index 1d426d0f8f82..22be3c081427 100644
--- a/Lib/test/test_capi/test_misc.py
+++ b/Lib/test/test_capi/test_misc.py
@@ -16,11 +16,13 @@
import unittest
import warnings
import weakref
+import operator
from test import support
from test.support import MISSING_C_DOCSTRINGS
from test.support import import_helper
from test.support import threading_helper
from test.support import warnings_helper
+from test.support import requires_limited_api
from test.support.script_helper import assert_python_failure, assert_python_ok, run_python_until_end
try:
import _posixsubprocess
@@ -770,7 +772,6 @@ def meth(self):
MutableBase.meth = lambda self: 'changed'
self.assertEqual(instance.meth(), 'changed')
-
def test_pynumber_tobase(self):
from _testcapi import pynumber_tobase
small_number = 123
@@ -1072,6 +1073,161 @@ class Data(_testcapi.ObjExtraData):
self.assertIsNone(d.extra)
+ at requires_limited_api
+class TestHeapTypeRelative(unittest.TestCase):
+ """Test API for extending opaque types (PEP 697)"""
+
+ @requires_limited_api
+ def test_heaptype_relative_sizes(self):
+ # Test subclassing using "relative" basicsize, see PEP 697
+ def check(extra_base_size, extra_size):
+ Base, Sub, instance, data_ptr, data_offset, data_size = (
+ _testcapi.make_sized_heaptypes(
+ extra_base_size, -extra_size))
+
+ # no alignment shenanigans when inheriting directly
+ if extra_size == 0:
+ self.assertEqual(Base.__basicsize__, Sub.__basicsize__)
+ self.assertEqual(data_size, 0)
+
+ else:
+ # The following offsets should be in increasing order:
+ offsets = [
+ (0, 'start of object'),
+ (Base.__basicsize__, 'end of base data'),
+ (data_offset, 'subclass data'),
+ (data_offset + extra_size, 'end of requested subcls data'),
+ (data_offset + data_size, 'end of reserved subcls data'),
+ (Sub.__basicsize__, 'end of object'),
+ ]
+ ordered_offsets = sorted(offsets, key=operator.itemgetter(0))
+ self.assertEqual(
+ offsets, ordered_offsets,
+ msg=f'Offsets not in expected order, got: {ordered_offsets}')
+
+ # end of reserved subcls data == end of object
+ self.assertEqual(Sub.__basicsize__, data_offset + data_size)
+
+ # we don't reserve (requested + alignment) or more data
+ self.assertLess(data_size - extra_size,
+ _testcapi.ALIGNOF_MAX_ALIGN_T)
+
+ # The offsets/sizes we calculated should be aligned.
+ self.assertEqual(data_offset % _testcapi.ALIGNOF_MAX_ALIGN_T, 0)
+ self.assertEqual(data_size % _testcapi.ALIGNOF_MAX_ALIGN_T, 0)
+
+ sizes = sorted({0, 1, 2, 3, 4, 7, 8, 123,
+ object.__basicsize__,
+ object.__basicsize__-1,
+ object.__basicsize__+1})
+ for extra_base_size in sizes:
+ for extra_size in sizes:
+ args = dict(extra_base_size=extra_base_size,
+ extra_size=extra_size)
+ with self.subTest(**args):
+ check(**args)
+
+ def test_HeapCCollection(self):
+ """Make sure HeapCCollection works properly by itself"""
+ collection = _testcapi.HeapCCollection(1, 2, 3)
+ self.assertEqual(list(collection), [1, 2, 3])
+
+ def test_heaptype_inherit_itemsize(self):
+ """Test HeapCCollection subclasses work properly"""
+ sizes = sorted({0, 1, 2, 3, 4, 7, 8, 123,
+ object.__basicsize__,
+ object.__basicsize__-1,
+ object.__basicsize__+1})
+ for extra_size in sizes:
+ with self.subTest(extra_size=extra_size):
+ Sub = _testcapi.subclass_var_heaptype(
+ _testcapi.HeapCCollection, -extra_size, 0, 0)
+ collection = Sub(1, 2, 3)
+ collection.set_data_to_3s()
+
+ self.assertEqual(list(collection), [1, 2, 3])
+ mem = collection.get_data()
+ self.assertGreaterEqual(len(mem), extra_size)
+ self.assertTrue(set(mem) <= {3}, f'got {mem!r}')
+
+ def test_heaptype_invalid_inheritance(self):
+ with self.assertRaises(SystemError,
+ msg="Cannot extend variable-size class without "
+ + "Py_TPFLAGS_ITEMS_AT_END"):
+ _testcapi.subclass_heaptype(int, -8, 0)
+
+ def test_heaptype_relative_members(self):
+ """Test HeapCCollection subclasses work properly"""
+ sizes = sorted({0, 1, 2, 3, 4, 7, 8, 123,
+ object.__basicsize__,
+ object.__basicsize__-1,
+ object.__basicsize__+1})
+ for extra_base_size in sizes:
+ for extra_size in sizes:
+ for offset in sizes:
+ with self.subTest(extra_base_size=extra_base_size, extra_size=extra_size, offset=offset):
+ if offset < extra_size:
+ Sub = _testcapi.make_heaptype_with_member(
+ extra_base_size, -extra_size, offset, True)
+ Base = Sub.mro()[1]
+ instance = Sub()
+ self.assertEqual(instance.memb, instance.get_memb())
+ instance.set_memb(13)
+ self.assertEqual(instance.memb, instance.get_memb())
+ self.assertEqual(instance.get_memb(), 13)
+ instance.memb = 14
+ self.assertEqual(instance.memb, instance.get_memb())
+ self.assertEqual(instance.get_memb(), 14)
+ self.assertGreaterEqual(instance.get_memb_offset(), Base.__basicsize__)
+ self.assertLess(instance.get_memb_offset(), Sub.__basicsize__)
+ with self.assertRaises(SystemError):
+ instance.get_memb_relative()
+ with self.assertRaises(SystemError):
+ instance.set_memb_relative(0)
+ else:
+ with self.assertRaises(SystemError):
+ Sub = _testcapi.make_heaptype_with_member(
+ extra_base_size, -extra_size, offset, True)
+ with self.assertRaises(SystemError):
+ Sub = _testcapi.make_heaptype_with_member(
+ extra_base_size, extra_size, offset, True)
+ with self.subTest(extra_base_size=extra_base_size, extra_size=extra_size):
+ with self.assertRaises(SystemError):
+ Sub = _testcapi.make_heaptype_with_member(
+ extra_base_size, -extra_size, -1, True)
+
+ def test_heaptype_relative_members_errors(self):
+ with self.assertRaisesRegex(
+ SystemError,
+ r"With Py_RELATIVE_OFFSET, basicsize must be negative"):
+ _testcapi.make_heaptype_with_member(0, 1234, 0, True)
+ with self.assertRaisesRegex(
+ SystemError, r"Member offset out of range \(0\.\.-basicsize\)"):
+ _testcapi.make_heaptype_with_member(0, -8, 1234, True)
+ with self.assertRaisesRegex(
+ SystemError, r"Member offset out of range \(0\.\.-basicsize\)"):
+ _testcapi.make_heaptype_with_member(0, -8, -1, True)
+
+ Sub = _testcapi.make_heaptype_with_member(0, -8, 0, True)
+ instance = Sub()
+ with self.assertRaisesRegex(
+ SystemError, r"PyMember_GetOne used with Py_RELATIVE_OFFSET"):
+ instance.get_memb_relative()
+ with self.assertRaisesRegex(
+ SystemError, r"PyMember_SetOne used with Py_RELATIVE_OFFSET"):
+ instance.set_memb_relative(0)
+
+ def test_pyobject_getitemdata_error(self):
+ """Test PyObject_GetItemData fails on unsupported types"""
+ with self.assertRaises(TypeError):
+ # None is not variable-length
+ _testcapi.pyobject_getitemdata(None)
+ with self.assertRaises(TypeError):
+ # int is variable-length, but doesn't have the
+ # Py_TPFLAGS_ITEMS_AT_END layout (and flag)
+ _testcapi.pyobject_getitemdata(0)
+
+
class TestPendingCalls(unittest.TestCase):
def pendingcalls_submit(self, l, n):
diff --git a/Lib/test/test_stable_abi_ctypes.py b/Lib/test/test_stable_abi_ctypes.py
index 2feaaf8603b8..4ca39d85e546 100644
--- a/Lib/test/test_stable_abi_ctypes.py
+++ b/Lib/test/test_stable_abi_ctypes.py
@@ -529,6 +529,7 @@ def test_windows_feature_macros(self):
"PyObject_GetBuffer",
"PyObject_GetItem",
"PyObject_GetIter",
+ "PyObject_GetTypeData",
"PyObject_HasAttr",
"PyObject_HasAttrString",
"PyObject_Hash",
@@ -679,6 +680,7 @@ def test_windows_feature_macros(self):
"PyType_GetName",
"PyType_GetQualName",
"PyType_GetSlot",
+ "PyType_GetTypeDataSize",
"PyType_IsSubtype",
"PyType_Modified",
"PyType_Ready",
diff --git a/Misc/NEWS.d/next/C API/2023-04-13-16-54-00.gh-issue-103509.A26Qu8.rst b/Misc/NEWS.d/next/C API/2023-04-13-16-54-00.gh-issue-103509.A26Qu8.rst
new file mode 100644
index 000000000000..af630c3aafa9
--- /dev/null
+++ b/Misc/NEWS.d/next/C API/2023-04-13-16-54-00.gh-issue-103509.A26Qu8.rst
@@ -0,0 +1,5 @@
+Added C API for extending types whose instance memory layout is opaque:
+:c:member:`PyType_Spec.basicsize` can now be zero or negative,
+:c:func:`PyObject_GetTypeData` can be used to get subclass-specific data,
+and :c:macro:`Py_TPFLAGS_ITEMS_AT_END` can be used to safely extend
+variable-size objects. See :pep:`697` for details.
diff --git a/Misc/stable_abi.toml b/Misc/stable_abi.toml
index 23baeeeae791..48299e9b35ff 100644
--- a/Misc/stable_abi.toml
+++ b/Misc/stable_abi.toml
@@ -2397,3 +2397,12 @@
added = '3.12' # Before 3.12, available in "structmember.h" w/o Py_ prefix
[const.Py_AUDIT_READ]
added = '3.12' # Before 3.12, available in "structmember.h"
+
+[function.PyObject_GetTypeData]
+ added = '3.12'
+[function.PyType_GetTypeDataSize]
+ added = '3.12'
+[const.Py_RELATIVE_OFFSET]
+ added = '3.12'
+[const.Py_TPFLAGS_ITEMS_AT_END]
+ added = '3.12'
diff --git a/Modules/Setup.stdlib.in b/Modules/Setup.stdlib.in
index a90c1e96ef02..6b4833419537 100644
--- a/Modules/Setup.stdlib.in
+++ b/Modules/Setup.stdlib.in
@@ -169,7 +169,7 @@
@MODULE__XXTESTFUZZ_TRUE at _xxtestfuzz _xxtestfuzz/_xxtestfuzz.c _xxtestfuzz/fuzzer.c
@MODULE__TESTBUFFER_TRUE at _testbuffer _testbuffer.c
@MODULE__TESTINTERNALCAPI_TRUE at _testinternalcapi _testinternalcapi.c
- at MODULE__TESTCAPI_TRUE@_testcapi _testcapimodule.c _testcapi/vectorcall.c _testcapi/vectorcall_limited.c _testcapi/heaptype.c _testcapi/unicode.c _testcapi/getargs.c _testcapi/pytime.c _testcapi/datetime.c _testcapi/docstring.c _testcapi/mem.c _testcapi/watchers.c _testcapi/long.c _testcapi/float.c _testcapi/structmember.c _testcapi/exceptions.c _testcapi/code.c _testcapi/pyos.c _testcapi/immortal.c
+ at MODULE__TESTCAPI_TRUE@_testcapi _testcapimodule.c _testcapi/vectorcall.c _testcapi/vectorcall_limited.c _testcapi/heaptype.c _testcapi/unicode.c _testcapi/getargs.c _testcapi/pytime.c _testcapi/datetime.c _testcapi/docstring.c _testcapi/mem.c _testcapi/watchers.c _testcapi/long.c _testcapi/float.c _testcapi/structmember.c _testcapi/exceptions.c _testcapi/code.c _testcapi/pyos.c _testcapi/immortal.c _testcapi/heaptype_relative.c
@MODULE__TESTCLINIC_TRUE at _testclinic _testclinic.c
# Some testing modules MUST be built as shared libraries.
diff --git a/Modules/_testcapi/heaptype.c b/Modules/_testcapi/heaptype.c
index 6384fbc485fb..3488e35922c5 100644
--- a/Modules/_testcapi/heaptype.c
+++ b/Modules/_testcapi/heaptype.c
@@ -371,7 +371,6 @@ create_type_from_repeated_slots(PyObject *self, PyObject *variant_obj)
}
-
static PyObject *
make_immutable_type_with_base(PyObject *self, PyObject *base)
{
@@ -399,6 +398,17 @@ make_type_with_base(PyObject *self, PyObject *base)
}
+static PyObject *
+pyobject_getitemdata(PyObject *self, PyObject *o)
+{
+ void *pointer = PyObject_GetItemData(o);
+ if (pointer == NULL) {
+ return NULL;
+ }
+ return PyLong_FromVoidPtr(pointer);
+}
+
+
static PyMethodDef TestMethods[] = {
{"pytype_fromspec_meta", pytype_fromspec_meta, METH_O},
{"test_type_from_ephemeral_spec", test_type_from_ephemeral_spec, METH_NOARGS},
@@ -411,6 +421,7 @@ static PyMethodDef TestMethods[] = {
METH_NOARGS},
{"make_immutable_type_with_base", make_immutable_type_with_base, METH_O},
{"make_type_with_base", make_type_with_base, METH_O},
+ {"pyobject_getitemdata", pyobject_getitemdata, METH_O},
{NULL},
};
@@ -987,6 +998,113 @@ static PyType_Spec HeapCTypeSetattr_spec = {
HeapCTypeSetattr_slots
};
+PyDoc_STRVAR(HeapCCollection_doc,
+"Tuple-like heap type that uses PyObject_GetItemData for items.");
+
+static PyObject*
+HeapCCollection_new(PyTypeObject *subtype, PyObject *args, PyObject *kwds)
+{
+ PyObject *self = NULL;
+ PyObject *result = NULL;
+
+ Py_ssize_t size = PyTuple_GET_SIZE(args);
+ self = subtype->tp_alloc(subtype, size);
+ if (!self) {
+ goto finally;
+ }
+ PyObject **data = PyObject_GetItemData(self);
+ if (!data) {
+ goto finally;
+ }
+
+ for (Py_ssize_t i = 0; i < size; i++) {
+ data[i] = Py_NewRef(PyTuple_GET_ITEM(args, i));
+ }
+
+ result = self;
+ self = NULL;
+ finally:
+ Py_XDECREF(self);
+ return result;
+}
+
+static Py_ssize_t
+HeapCCollection_length(PyVarObject *self)
+{
+ return Py_SIZE(self);
+}
+
+static PyObject*
+HeapCCollection_item(PyObject *self, Py_ssize_t i)
+{
+ if (i < 0 || i >= Py_SIZE(self)) {
+ return PyErr_Format(PyExc_IndexError, "index %zd out of range", i);
+ }
+ PyObject **data = PyObject_GetItemData(self);
+ if (!data) {
+ return NULL;
+ }
+ return Py_NewRef(data[i]);
+}
+
+static int
+HeapCCollection_traverse(PyObject *self, visitproc visit, void *arg)
+{
+ PyObject **data = PyObject_GetItemData(self);
+ if (!data) {
+ return -1;
+ }
+ for (Py_ssize_t i = 0; i < Py_SIZE(self); i++) {
+ Py_VISIT(data[i]);
+ }
+ return 0;
+}
+
+static int
+HeapCCollection_clear(PyObject *self)
+{
+ PyObject **data = PyObject_GetItemData(self);
+ if (!data) {
+ return -1;
+ }
+ Py_ssize_t size = Py_SIZE(self);
+ Py_SET_SIZE(self, 0);
+ for (Py_ssize_t i = 0; i < size; i++) {
+ Py_CLEAR(data[i]);
+ }
+ return 0;
+}
+
+static void
+HeapCCollection_dealloc(PyObject *self)
+{
+ PyTypeObject *tp = Py_TYPE(self);
+ HeapCCollection_clear(self);
+ PyObject_GC_UnTrack(self);
+ tp->tp_free(self);
+ Py_DECREF(tp);
+}
+
+static PyType_Slot HeapCCollection_slots[] = {
+ {Py_tp_new, HeapCCollection_new},
+ {Py_sq_length, HeapCCollection_length},
+ {Py_sq_item, HeapCCollection_item},
+ {Py_tp_traverse, HeapCCollection_traverse},
+ {Py_tp_clear, HeapCCollection_clear},
+ {Py_tp_dealloc, HeapCCollection_dealloc},
+ {Py_tp_doc, (void *)HeapCCollection_doc},
+ {0, 0},
+};
+
+static PyType_Spec HeapCCollection_spec = {
+ .name = "_testcapi.HeapCCollection",
+ .basicsize = sizeof(PyVarObject),
+ .itemsize = sizeof(PyObject*),
+ .flags = (Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE |
+ Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_ITEMS_AT_END),
+ .slots = HeapCCollection_slots,
+};
+
int
_PyTestCapi_Init_Heaptype(PyObject *m) {
_testcapimodule = PyModule_GetDef(m);
@@ -1110,5 +1228,16 @@ _PyTestCapi_Init_Heaptype(PyObject *m) {
}
PyModule_AddObject(m, "HeapCTypeMetaclassCustomNew", HeapCTypeMetaclassCustomNew);
+ PyObject *HeapCCollection = PyType_FromMetaclass(
+ NULL, m, &HeapCCollection_spec, NULL);
+ if (HeapCCollection == NULL) {
+ return -1;
+ }
+ int rc = PyModule_AddType(m, (PyTypeObject *)HeapCCollection);
+ Py_DECREF(HeapCCollection);
+ if (rc < 0) {
+ return -1;
+ }
+
return 0;
}
diff --git a/Modules/_testcapi/heaptype_relative.c b/Modules/_testcapi/heaptype_relative.c
new file mode 100644
index 000000000000..c247ca33b337
--- /dev/null
+++ b/Modules/_testcapi/heaptype_relative.c
@@ -0,0 +1,343 @@
+#define Py_LIMITED_API 0x030c0000 // 3.12
+#include "parts.h"
+#include <stddef.h> // max_align_t
+#include <string.h> // memset
+
+#ifdef LIMITED_API_AVAILABLE
+
+static PyType_Slot empty_slots[] = {
+ {0, NULL},
+};
+
+static PyObject *
+make_sized_heaptypes(PyObject *module, PyObject *args)
+{
+ PyObject *base = NULL;
+ PyObject *sub = NULL;
+ PyObject *instance = NULL;
+ PyObject *result = NULL;
+
+ int extra_base_size, basicsize;
+
+ int r = PyArg_ParseTuple(args, "ii", &extra_base_size, &basicsize);
+ if (!r) {
+ goto finally;
+ }
+
+ PyType_Spec base_spec = {
+ .name = "_testcapi.Base",
+ .basicsize = sizeof(PyObject) + extra_base_size,
+ .flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,
+ .slots = empty_slots,
+ };
+ PyType_Spec sub_spec = {
+ .name = "_testcapi.Sub",
+ .basicsize = basicsize,
+ .flags = Py_TPFLAGS_DEFAULT,
+ .slots = empty_slots,
+ };
+
+ base = PyType_FromMetaclass(NULL, module, &base_spec, NULL);
+ if (!base) {
+ goto finally;
+ }
+ sub = PyType_FromMetaclass(NULL, module, &sub_spec, base);
+ if (!sub) {
+ goto finally;
+ }
+ instance = PyObject_CallNoArgs(sub);
+ if (!instance) {
+ goto finally;
+ }
+ char *data_ptr = PyObject_GetTypeData(instance, (PyTypeObject *)sub);
+ if (!data_ptr) {
+ goto finally;
+ }
+ Py_ssize_t data_size = PyType_GetTypeDataSize((PyTypeObject *)sub);
+ if (data_size < 0) {
+ goto finally;
+ }
+
+ result = Py_BuildValue("OOOKnn", base, sub, instance,
+ (unsigned long long)data_ptr,
+ (Py_ssize_t)(data_ptr - (char*)instance),
+ data_size);
+ finally:
+ Py_XDECREF(base);
+ Py_XDECREF(sub);
+ Py_XDECREF(instance);
+ return result;
+}
+
+static PyObject *
+var_heaptype_set_data_to_3s(
+ PyObject *self, PyTypeObject *defining_class,
+ PyObject **args, Py_ssize_t nargs, PyObject *kwnames)
+{
+ void *data_ptr = PyObject_GetTypeData(self, defining_class);
+ if (!data_ptr) {
+ return NULL;
+ }
+ Py_ssize_t data_size = PyType_GetTypeDataSize(defining_class);
+ if (data_size < 0) {
+ return NULL;
+ }
+ memset(data_ptr, 3, data_size);
+ Py_RETURN_NONE;
+}
+
+static PyObject *
+var_heaptype_get_data(PyObject *self, PyTypeObject *defining_class,
+ PyObject **args, Py_ssize_t nargs, PyObject *kwnames)
+{
+ void *data_ptr = PyObject_GetTypeData(self, defining_class);
+ if (!data_ptr) {
+ return NULL;
+ }
+ Py_ssize_t data_size = PyType_GetTypeDataSize(defining_class);
+ if (data_size < 0) {
+ return NULL;
+ }
+ return PyBytes_FromStringAndSize(data_ptr, data_size);
+}
+
+static PyMethodDef var_heaptype_methods[] = {
+ {"set_data_to_3s", _PyCFunction_CAST(var_heaptype_set_data_to_3s),
+ METH_METHOD | METH_FASTCALL | METH_KEYWORDS},
+ {"get_data", _PyCFunction_CAST(var_heaptype_get_data),
+ METH_METHOD | METH_FASTCALL | METH_KEYWORDS},
+ {NULL},
+};
+
+static PyObject *
+subclass_var_heaptype(PyObject *module, PyObject *args)
+{
+ PyObject *result = NULL;
+
+ PyObject *base; // borrowed from args
+ int basicsize, itemsize;
+ long pfunc;
+
+ int r = PyArg_ParseTuple(args, "Oiil", &base, &basicsize, &itemsize, &pfunc);
+ if (!r) {
+ goto finally;
+ }
+
+ PyType_Slot slots[] = {
+ {Py_tp_methods, var_heaptype_methods},
+ {0, NULL},
+ };
+
+ PyType_Spec sub_spec = {
+ .name = "_testcapi.Sub",
+ .basicsize = basicsize,
+ .itemsize = itemsize,
+ .flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_ITEMS_AT_END,
+ .slots = slots,
+ };
+
+ result = PyType_FromMetaclass(NULL, module, &sub_spec, base);
+ finally:
+ return result;
+}
+
+static PyObject *
+subclass_heaptype(PyObject *module, PyObject *args)
+{
+ PyObject *result = NULL;
+
+ PyObject *base; // borrowed from args
+ int basicsize, itemsize;
+
+ int r = PyArg_ParseTuple(args, "Oii", &base, &basicsize, &itemsize);
+ if (!r) {
+ goto finally;
+ }
+
+ PyType_Slot slots[] = {
+ {Py_tp_methods, var_heaptype_methods},
+ {0, NULL},
+ };
+
+ PyType_Spec sub_spec = {
+ .name = "_testcapi.Sub",
+ .basicsize = basicsize,
+ .itemsize = itemsize,
+ .flags = Py_TPFLAGS_DEFAULT,
+ .slots = slots,
+ };
+
+ result = PyType_FromMetaclass(NULL, module, &sub_spec, base);
+ finally:
+ return result;
+}
+
+static PyMemberDef *
+heaptype_with_member_extract_and_check_memb(PyObject *self)
+{
+ PyMemberDef *def = PyType_GetSlot(Py_TYPE(self), Py_tp_members);
+ if (!def) {
+ if (!PyErr_Occurred()) {
+ PyErr_SetString(PyExc_ValueError, "tp_members is NULL");
+ }
+ return NULL;
+ }
+ if (!def[0].name) {
+ PyErr_SetString(PyExc_ValueError, "tp_members[0] is NULL");
+ return NULL;
+ }
+ if (def[1].name) {
+ PyErr_SetString(PyExc_ValueError, "tp_members[1] is not NULL");
+ return NULL;
+ }
+ if (strcmp(def[0].name, "memb")) {
+ PyErr_SetString(PyExc_ValueError, "tp_members[0] is not for `memb`");
+ return NULL;
+ }
+ if (def[0].flags) {
+ PyErr_SetString(PyExc_ValueError, "tp_members[0] has flags set");
+ return NULL;
+ }
+ return def;
+}
+
+static PyObject *
+heaptype_with_member_get_memb(PyObject *self, PyObject *Py_UNUSED(ignored))
+{
+ PyMemberDef *def = heaptype_with_member_extract_and_check_memb(self);
+ return PyMember_GetOne((const char *)self, def);
+}
+
+static PyObject *
+heaptype_with_member_set_memb(PyObject *self, PyObject *value)
+{
+ PyMemberDef *def = heaptype_with_member_extract_and_check_memb(self);
+ int r = PyMember_SetOne((char *)self, def, value);
+ if (r < 0) {
+ return NULL;
+ }
+ Py_RETURN_NONE;
+}
+
+static PyObject *
+get_memb_offset(PyObject *self, PyObject *Py_UNUSED(ignored))
+{
+ PyMemberDef *def = heaptype_with_member_extract_and_check_memb(self);
+ return PyLong_FromSsize_t(def->offset);
+}
+
+static PyObject *
+heaptype_with_member_get_memb_relative(PyObject *self, PyObject *Py_UNUSED(ignored))
+{
+ PyMemberDef def = {"memb", Py_T_BYTE, sizeof(PyObject), Py_RELATIVE_OFFSET};
+ return PyMember_GetOne((const char *)self, &def);
+}
+
+static PyObject *
+heaptype_with_member_set_memb_relative(PyObject *self, PyObject *value)
+{
+ PyMemberDef def = {"memb", Py_T_BYTE, sizeof(PyObject), Py_RELATIVE_OFFSET};
+ int r = PyMember_SetOne((char *)self, &def, value);
+ if (r < 0) {
+ return NULL;
+ }
+ Py_RETURN_NONE;
+}
+
+static PyMethodDef heaptype_with_member_methods[] = {
+ {"get_memb", heaptype_with_member_get_memb, METH_NOARGS},
+ {"set_memb", heaptype_with_member_set_memb, METH_O},
+ {"get_memb_offset", get_memb_offset, METH_NOARGS},
+ {"get_memb_relative", heaptype_with_member_get_memb_relative, METH_NOARGS},
+ {"set_memb_relative", heaptype_with_member_set_memb_relative, METH_O},
+ {NULL},
+};
+
+static PyObject *
+make_heaptype_with_member(PyObject *module, PyObject *args)
+{
+ PyObject *base = NULL;
+ PyObject *result = NULL;
+
+ int extra_base_size, basicsize, offset, add_flag;
+
+ int r = PyArg_ParseTuple(args, "iiip", &extra_base_size, &basicsize, &offset, &add_flag);
+ if (!r) {
+ goto finally;
+ }
+
+ PyType_Spec base_spec = {
+ .name = "_testcapi.Base",
+ .basicsize = sizeof(PyObject) + extra_base_size,
+ .flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,
+ .slots = empty_slots,
+ };
+ base = PyType_FromMetaclass(NULL, module, &base_spec, NULL);
+ if (!base) {
+ goto finally;
+ }
+
+ PyMemberDef members[] = {
+ {"memb", Py_T_BYTE, offset, add_flag ? Py_RELATIVE_OFFSET : 0},
+ {0},
+ };
+ PyType_Slot slots[] = {
+ {Py_tp_members, members},
+ {Py_tp_methods, heaptype_with_member_methods},
+ {0, NULL},
+ };
+
+ PyType_Spec sub_spec = {
+ .name = "_testcapi.Sub",
+ .basicsize = basicsize,
+ .flags = Py_TPFLAGS_DEFAULT,
+ .slots = slots,
+ };
+
+ result = PyType_FromMetaclass(NULL, module, &sub_spec, base);
+ finally:
+ Py_XDECREF(base);
+ return result;
+}
+
+
+static PyObject *
+test_alignof_max_align_t(PyObject *module, PyObject *Py_UNUSED(ignored))
+{
+ // We define ALIGNOF_MAX_ALIGN_T even if the compiler doesn't support
+ // max_align_t. Double-check that it's correct.
+ assert(ALIGNOF_MAX_ALIGN_T > 0);
+ assert(ALIGNOF_MAX_ALIGN_T >= _Alignof(long long));
+ assert(ALIGNOF_MAX_ALIGN_T >= _Alignof(long double));
+ assert(ALIGNOF_MAX_ALIGN_T >= _Alignof(void*));
+ assert(ALIGNOF_MAX_ALIGN_T >= _Alignof(void (*)(void)));
+
+ // Ensure it's a power of two
+ assert((ALIGNOF_MAX_ALIGN_T & (ALIGNOF_MAX_ALIGN_T - 1)) == 0);
+
+ Py_RETURN_NONE;
+}
+
+static PyMethodDef TestMethods[] = {
+ {"make_sized_heaptypes", make_sized_heaptypes, METH_VARARGS},
+ {"subclass_var_heaptype", subclass_var_heaptype, METH_VARARGS},
+ {"subclass_heaptype", subclass_heaptype, METH_VARARGS},
+ {"make_heaptype_with_member", make_heaptype_with_member, METH_VARARGS},
+ {"test_alignof_max_align_t", test_alignof_max_align_t, METH_NOARGS},
+ {NULL},
+};
+
+int
+_PyTestCapi_Init_HeaptypeRelative(PyObject *m) {
+ if (PyModule_AddFunctions(m, TestMethods) < 0) {
+ return -1;
+ }
+
+ if (PyModule_AddIntMacro(m, ALIGNOF_MAX_ALIGN_T) < 0) {
+ return -1;
+ }
+
+ return 0;
+}
+
+#endif // LIMITED_API_AVAILABLE
diff --git a/Modules/_testcapi/parts.h b/Modules/_testcapi/parts.h
index 4d2d6832a827..d75412d51160 100644
--- a/Modules/_testcapi/parts.h
+++ b/Modules/_testcapi/parts.h
@@ -43,6 +43,7 @@ int _PyTestCapi_Init_Immortal(PyObject *module);
#ifdef LIMITED_API_AVAILABLE
int _PyTestCapi_Init_VectorcallLimited(PyObject *module);
+int _PyTestCapi_Init_HeaptypeRelative(PyObject *module);
#endif // LIMITED_API_AVAILABLE
#endif
diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c
index 47e0ed9be8e7..1ecc44205808 100644
--- a/Modules/_testcapimodule.c
+++ b/Modules/_testcapimodule.c
@@ -4324,6 +4324,9 @@ PyInit__testcapi(void)
if (_PyTestCapi_Init_VectorcallLimited(m) < 0) {
return NULL;
}
+ if (_PyTestCapi_Init_HeaptypeRelative(m) < 0) {
+ return NULL;
+ }
#endif
PyState_AddModule(m, &_testcapimodule);
diff --git a/Objects/descrobject.c b/Objects/descrobject.c
index 334be75e8df9..17c0c85a06c4 100644
--- a/Objects/descrobject.c
+++ b/Objects/descrobject.c
@@ -978,6 +978,12 @@ PyDescr_NewMember(PyTypeObject *type, PyMemberDef *member)
{
PyMemberDescrObject *descr;
+ if (member->flags & Py_RELATIVE_OFFSET) {
+ PyErr_SetString(
+ PyExc_SystemError,
+ "PyDescr_NewMember used with Py_RELATIVE_OFFSET");
+ return NULL;
+ }
descr = (PyMemberDescrObject *)descr_new(&PyMemberDescr_Type,
type, member->name);
if (descr != NULL)
diff --git a/Objects/typeobject.c b/Objects/typeobject.c
index 4ced04b0bde9..171c76a59a55 100644
--- a/Objects/typeobject.c
+++ b/Objects/typeobject.c
@@ -18,6 +18,7 @@
#include "structmember.h" // PyMemberDef
#include <ctype.h>
+#include <stddef.h> // ptrdiff_t
/*[clinic input]
class type "PyTypeObject *" "&PyType_Type"
@@ -1686,6 +1687,12 @@ PyType_GenericNew(PyTypeObject *type, PyObject *args, PyObject *kwds)
/* Helpers for subtyping */
+static inline PyMemberDef *
+_PyHeapType_GET_MEMBERS(PyHeapTypeObject* type)
+{
+ return PyObject_GetItemData((PyObject *)type);
+}
+
static int
traverse_slots(PyTypeObject *type, PyObject *self, visitproc visit, void *arg)
{
@@ -3873,6 +3880,15 @@ static const PySlot_Offset pyslot_offsets[] = {
#include "typeslots.inc"
};
+/* Align up to the nearest multiple of alignof(max_align_t)
+ * (like _Py_ALIGN_UP, but for a size rather than pointer)
+ */
+static Py_ssize_t
+_align_up(Py_ssize_t size)
+{
+ return (size + ALIGNOF_MAX_ALIGN_T - 1) & ~(ALIGNOF_MAX_ALIGN_T - 1);
+}
+
/* Given a PyType_FromMetaclass `bases` argument (NULL, type, or tuple of
* types), return a tuple of types.
*/
@@ -4013,6 +4029,20 @@ _PyType_FromMetaclass_impl(
assert(memb->flags == READONLY);
vectorcalloffset = memb->offset;
}
+ if (memb->flags & Py_RELATIVE_OFFSET) {
+ if (spec->basicsize > 0) {
+ PyErr_SetString(
+ PyExc_SystemError,
+ "With Py_RELATIVE_OFFSET, basicsize must be negative.");
+ goto finally;
+ }
+ if (memb->offset < 0 || memb->offset >= -spec->basicsize) {
+ PyErr_SetString(
+ PyExc_SystemError,
+ "Member offset out of range (0..-basicsize)");
+ goto finally;
+ }
+ }
}
break;
case Py_tp_doc:
@@ -4154,6 +4184,32 @@ _PyType_FromMetaclass_impl(
// here we just check its work
assert(_PyType_HasFeature(base, Py_TPFLAGS_BASETYPE));
+ /* Calculate sizes */
+
+ Py_ssize_t basicsize = spec->basicsize;
+ Py_ssize_t type_data_offset = spec->basicsize;
+ if (basicsize == 0) {
+ /* Inherit */
+ basicsize = base->tp_basicsize;
+ }
+ else if (basicsize < 0) {
+ /* Extend */
+ type_data_offset = _align_up(base->tp_basicsize);
+ basicsize = type_data_offset + _align_up(-spec->basicsize);
+
+ /* Inheriting variable-sized types is limited */
+ if (base->tp_itemsize
+ && !((base->tp_flags | spec->flags) & Py_TPFLAGS_ITEMS_AT_END))
+ {
+ PyErr_SetString(
+ PyExc_SystemError,
+ "Cannot extend variable-size class without Py_TPFLAGS_ITEMS_AT_END.");
+ goto finally;
+ }
+ }
+
+ Py_ssize_t itemsize = spec->itemsize;
+
/* Allocate the new type
*
* Between here and PyType_Ready, we should limit:
@@ -4201,8 +4257,8 @@ _PyType_FromMetaclass_impl(
/* Copy the sizes */
- type->tp_basicsize = spec->basicsize;
- type->tp_itemsize = spec->itemsize;
+ type->tp_basicsize = basicsize;
+ type->tp_itemsize = itemsize;
/* Copy all the ordinary slots */
@@ -4219,6 +4275,16 @@ _PyType_FromMetaclass_impl(
size_t len = Py_TYPE(type)->tp_itemsize * nmembers;
memcpy(_PyHeapType_GET_MEMBERS(res), slot->pfunc, len);
type->tp_members = _PyHeapType_GET_MEMBERS(res);
+ PyMemberDef *memb;
+ Py_ssize_t i;
+ for (memb = _PyHeapType_GET_MEMBERS(res), i = nmembers;
+ i > 0; ++memb, --i)
+ {
+ if (memb->flags & Py_RELATIVE_OFFSET) {
+ memb->flags &= ~Py_RELATIVE_OFFSET;
+ memb->offset += type_data_offset;
+ }
+ }
}
break;
default:
@@ -4227,6 +4293,7 @@ _PyType_FromMetaclass_impl(
PySlot_Offset slotoffsets = pyslot_offsets[slot->slot];
short slot_offset = slotoffsets.slot_offset;
if (slotoffsets.subslot_offset == -1) {
+ /* Set a slot in the main PyTypeObject */
*(void**)((char*)res_start + slot_offset) = slot->pfunc;
}
else {
@@ -4461,6 +4528,34 @@ PyType_GetModuleByDef(PyTypeObject *type, PyModuleDef *def)
return NULL;
}
+void *
+PyObject_GetTypeData(PyObject *obj, PyTypeObject *cls)
+{
+ assert(PyObject_TypeCheck(obj, cls));
+ return (char *)obj + _align_up(cls->tp_base->tp_basicsize);
+}
+
+Py_ssize_t
+PyType_GetTypeDataSize(PyTypeObject *cls)
+{
+ ptrdiff_t result = cls->tp_basicsize - _align_up(cls->tp_base->tp_basicsize);
+ if (result < 0) {
+ return 0;
+ }
+ return result;
+}
+
+void *
+PyObject_GetItemData(PyObject *obj)
+{
+ if (!PyType_HasFeature(Py_TYPE(obj), Py_TPFLAGS_ITEMS_AT_END)) {
+ PyErr_Format(PyExc_TypeError,
+ "type '%s' does not have Py_TPFLAGS_ITEMS_AT_END",
+ Py_TYPE(obj)->tp_name);
+ return NULL;
+ }
+ return (char *)obj + Py_TYPE(obj)->tp_basicsize;
+}
/* Internal API to look for a name through the MRO, bypassing the method cache.
This returns a borrowed reference, and might set an exception.
@@ -5158,7 +5253,8 @@ PyTypeObject PyType_Type = {
0, /* tp_as_buffer */
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC |
Py_TPFLAGS_BASETYPE | Py_TPFLAGS_TYPE_SUBCLASS |
- Py_TPFLAGS_HAVE_VECTORCALL, /* tp_flags */
+ Py_TPFLAGS_HAVE_VECTORCALL |
+ Py_TPFLAGS_ITEMS_AT_END, /* tp_flags */
type_doc, /* tp_doc */
(traverseproc)type_traverse, /* tp_traverse */
(inquiry)type_clear, /* tp_clear */
@@ -6572,9 +6668,14 @@ inherit_special(PyTypeObject *type, PyTypeObject *base)
else if (PyType_IsSubtype(base, &PyDict_Type)) {
type->tp_flags |= Py_TPFLAGS_DICT_SUBCLASS;
}
+
+ /* Setup some inheritable flags */
if (PyType_HasFeature(base, _Py_TPFLAGS_MATCH_SELF)) {
type->tp_flags |= _Py_TPFLAGS_MATCH_SELF;
}
+ if (PyType_HasFeature(base, Py_TPFLAGS_ITEMS_AT_END)) {
+ type->tp_flags |= Py_TPFLAGS_ITEMS_AT_END;
+ }
}
static int
diff --git a/PC/pyconfig.h b/PC/pyconfig.h
index 8a3bf8968ce2..3415efe2dea1 100644
--- a/PC/pyconfig.h
+++ b/PC/pyconfig.h
@@ -330,6 +330,7 @@ Py_NO_ENABLE_SHARED to find out. Also support MS_NO_COREDLL for b/w compat */
# define SIZEOF_HKEY 8
# define SIZEOF_SIZE_T 8
# define ALIGNOF_SIZE_T 8
+# define ALIGNOF_MAX_ALIGN_T 8
/* configure.ac defines HAVE_LARGEFILE_SUPPORT iff
sizeof(off_t) > sizeof(long), and sizeof(long long) >= sizeof(off_t).
On Win64 the second condition is not true, but if fpos_t replaces off_t
@@ -351,6 +352,7 @@ Py_NO_ENABLE_SHARED to find out. Also support MS_NO_COREDLL for b/w compat */
# else
# define SIZEOF_TIME_T 4
# endif
+# define ALIGNOF_MAX_ALIGN_T 8
#endif
#ifdef _DEBUG
diff --git a/PC/python3dll.c b/PC/python3dll.c
index 706affa18351..7e848abccfd1 100755
--- a/PC/python3dll.c
+++ b/PC/python3dll.c
@@ -467,6 +467,7 @@ EXPORT_FUNC(PyObject_GetAttrString)
EXPORT_FUNC(PyObject_GetBuffer)
EXPORT_FUNC(PyObject_GetItem)
EXPORT_FUNC(PyObject_GetIter)
+EXPORT_FUNC(PyObject_GetTypeData)
EXPORT_FUNC(PyObject_HasAttr)
EXPORT_FUNC(PyObject_HasAttrString)
EXPORT_FUNC(PyObject_Hash)
@@ -618,6 +619,7 @@ EXPORT_FUNC(PyType_GetModuleState)
EXPORT_FUNC(PyType_GetName)
EXPORT_FUNC(PyType_GetQualName)
EXPORT_FUNC(PyType_GetSlot)
+EXPORT_FUNC(PyType_GetTypeDataSize)
EXPORT_FUNC(PyType_IsSubtype)
EXPORT_FUNC(PyType_Modified)
EXPORT_FUNC(PyType_Ready)
diff --git a/PCbuild/_testcapi.vcxproj b/PCbuild/_testcapi.vcxproj
index 21941247eb96..56448b6ee7d4 100644
--- a/PCbuild/_testcapi.vcxproj
+++ b/PCbuild/_testcapi.vcxproj
@@ -98,6 +98,7 @@
<ClCompile Include="..\Modules\_testcapi\vectorcall.c" />
<ClCompile Include="..\Modules\_testcapi\vectorcall_limited.c" />
<ClCompile Include="..\Modules\_testcapi\heaptype.c" />
+ <ClCompile Include="..\Modules\_testcapi\heaptype_relative.c" />
<ClCompile Include="..\Modules\_testcapi\unicode.c" />
<ClCompile Include="..\Modules\_testcapi\pytime.c" />
<ClCompile Include="..\Modules\_testcapi\datetime.c" />
diff --git a/PCbuild/_testcapi.vcxproj.filters b/PCbuild/_testcapi.vcxproj.filters
index 0e42e4982c21..297c9ce799be 100644
--- a/PCbuild/_testcapi.vcxproj.filters
+++ b/PCbuild/_testcapi.vcxproj.filters
@@ -24,6 +24,9 @@
<ClCompile Include="..\Modules\_testcapi\heaptype.c">
<Filter>Source Files</Filter>
</ClCompile>
+ <ClCompile Include="..\Modules\_testcapi\heaptype_relative.c">
+ <Filter>Source Files</Filter>
+ </ClCompile>
<ClCompile Include="..\Modules\_testcapi\unicode.c">
<Filter>Source Files</Filter>
</ClCompile>
diff --git a/Python/structmember.c b/Python/structmember.c
index 1b8be28dcf2e..19a75224a0f3 100644
--- a/Python/structmember.c
+++ b/Python/structmember.c
@@ -8,6 +8,12 @@ PyObject *
PyMember_GetOne(const char *obj_addr, PyMemberDef *l)
{
PyObject *v;
+ if (l->flags & Py_RELATIVE_OFFSET) {
+ PyErr_SetString(
+ PyExc_SystemError,
+ "PyMember_GetOne used with Py_RELATIVE_OFFSET");
+ return NULL;
+ }
const char* addr = obj_addr + l->offset;
switch (l->type) {
@@ -103,6 +109,12 @@ int
PyMember_SetOne(char *addr, PyMemberDef *l, PyObject *v)
{
PyObject *oldv;
+ if (l->flags & Py_RELATIVE_OFFSET) {
+ PyErr_SetString(
+ PyExc_SystemError,
+ "PyMember_SetOne used with Py_RELATIVE_OFFSET");
+ return -1;
+ }
addr += l->offset;
diff --git a/configure b/configure
index 8133d47f6135..b8fa9d66e735 100755
--- a/configure
+++ b/configure
@@ -10729,6 +10729,41 @@ cat >>confdefs.h <<_ACEOF
_ACEOF
+# The cast to long int works around a bug in the HP C Compiler,
+# see AC_CHECK_SIZEOF for more information.
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking alignment of max_align_t" >&5
+$as_echo_n "checking alignment of max_align_t... " >&6; }
+if ${ac_cv_alignof_max_align_t+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ if ac_fn_c_compute_int "$LINENO" "(long int) offsetof (ac__type_alignof_, y)" "ac_cv_alignof_max_align_t" "$ac_includes_default
+#ifndef offsetof
+# define offsetof(type, member) ((char *) &((type *) 0)->member - (char *) 0)
+#endif
+typedef struct { char x; max_align_t y; } ac__type_alignof_;"; then :
+
+else
+ if test "$ac_cv_type_max_align_t" = yes; then
+ { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
+$as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
+as_fn_error 77 "cannot compute alignment of max_align_t
+See \`config.log' for more details" "$LINENO" 5; }
+ else
+ ac_cv_alignof_max_align_t=0
+ fi
+fi
+
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_alignof_max_align_t" >&5
+$as_echo "$ac_cv_alignof_max_align_t" >&6; }
+
+
+
+cat >>confdefs.h <<_ACEOF
+#define ALIGNOF_MAX_ALIGN_T $ac_cv_alignof_max_align_t
+_ACEOF
+
+
{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for long double" >&5
diff --git a/configure.ac b/configure.ac
index 3f20d8980d8a..0940b93c25f7 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2921,6 +2921,7 @@ AC_CHECK_SIZEOF(size_t, 4)
AC_CHECK_ALIGNOF(size_t)
AC_CHECK_SIZEOF(pid_t, 4)
AC_CHECK_SIZEOF(uintptr_t)
+AC_CHECK_ALIGNOF(max_align_t)
AC_TYPE_LONG_DOUBLE
AC_CHECK_SIZEOF(long double, 16)
diff --git a/pyconfig.h.in b/pyconfig.h.in
index 236cee6588d4..2c22b27af65e 100644
--- a/pyconfig.h.in
+++ b/pyconfig.h.in
@@ -19,6 +19,9 @@
/* The normal alignment of `long', in bytes. */
#undef ALIGNOF_LONG
+/* The normal alignment of `max_align_t', in bytes. */
+#undef ALIGNOF_MAX_ALIGN_T
+
/* The normal alignment of `size_t', in bytes. */
#undef ALIGNOF_SIZE_T
More information about the Python-checkins
mailing list