[issue35134] Move !Py_LIMITED_API to Include/pycapi/

[STINNER Victor]

STINNER Victor <vstinner at redhat.com> added the comment:

> Do you want to keep only stable ABI v.3.2 and move both newer stable API and non-stable API to the pycapi/ subdirectory? Sorry, I don't found a sense in this.

The raw definition could be that Include/*.h is part of the stable ABI, and Include/pycapi/*.h are the definitions using Py_LIMITED_API and so can be stable or not stable depending on Py_LIMITED_API value :-)

To be honest, I'm not sure that I understand how "Py_LIMITED_API+0 >= 0x03050000" works and should be used.

I understand that you would prefer to leave PyObject_Calloc() in Include/objimpl.h. Honestly, I have no strong opinion on that. We can leave it there if you prefer.

--

Maybe the rule "move everything using Py_LIMITED_API to pycapi" is misleading.

My intent is that API in Include/*.h should not leak implementation details. It should be the starting point to design a new C API which does not leak any implementation detail:
http://pythoncapi.readthedocs.io/

It's easier with an example:


#define _PyObject_GC_TRACK(o) do { \
    PyGC_Head *g = _Py_AS_GC(o); \
    if (g->_gc_next != 0) { \
        Py_FatalError("GC object already tracked"); \
    } \
    assert((g->_gc_prev & _PyGC_PREV_MASK_COLLECTING) == 0); \
    ...

This macro is private: it starts with "_Py", so it doesn't belong to Include/*.h. Moreover, it access private fields like PyGC_Head._gc_prev.

>From my point of view, the ideal API would not access *any* structure field and PyGC_Header structure must not be used nor part of the C API.

--

After saying that, I looked again at my PR, and I still see private functions in objimpl.h. Example:

PyAPI_FUNC(PyObject *) _PyObject_New(PyTypeObject *);
PyAPI_FUNC(PyVarObject *) _PyObject_NewVar(PyTypeObject *, Py_ssize_t);

#define PyObject_New(type, typeobj) \
                ( (type *) _PyObject_New(typeobj) )
#define PyObject_NewVar(type, typeobj, n) \
                ( (type *) _PyObject_NewVar((typeobj), (n)) )

These functions are not excluded from Py_LIMITED_API. Since they are private, we are free to remove them whenever we want, so IMHO it's fine to exclude from Py_LIMITED_API right now if we want.

Another example:

static inline PyObject*
PyObject_INIT(PyObject *op, PyTypeObject *typeobj)
{
    assert(op != NULL);
    Py_TYPE(op) = typeobj;
    _Py_NewReference(op);
    return op;
}

It's a public function but it calls the private function _Py_NewReference(). So _Py_NewReference() must be part of Py_LIMITED_API somehow...

In release mode (if Py_TRACE_REFS is not defined), _Py_NewReference() is defined like that:

/* Without Py_TRACE_REFS, there's little enough to do that we expand code
   inline. */
static inline void _Py_NewReference(PyObject *op)
{
    if (_Py_tracemalloc_config.tracing) {
        _PyTraceMalloc_NewReference(op);
    }
    _Py_INC_TPALLOCS(op);
    _Py_INC_REFTOTAL;
    Py_REFCNT(op) = 1;
}

It does access to the private _Py_tracemalloc_config variable and private macros/functions _Py_INC_TPALLOCS(op) and _Py_INC_REFTOTAL.

We *can* always define _Py_NewReference() as a function call if Py_LIMITED_API is defined, but it would have an impact on performance.

Right now, I don't want to risk to introduce a performance slowdown.

I have a "Proof-of-concept" implementation of my proposed "new C API":
https://github.com/pythoncapi/cpython/

My implementation currently uses 3 defines:

* Py_NEWCAPI_NO_MACRO: replace macros with function calls PyTuple_GET_SIZE() becomes PyTuple_Size()
* Py_NEWCAPI_NO_STRUCT: must not use PyObject.ob_refcnt or any other field of Python object structures; structures should hide their fields: compilation error.
* Py_NEWCAPI: new C API without borrowed references, without macro, without struct

But this project is highly experimental and I don't want to make it upstream before we measured properly the impact on the performance, the API has been properly reviewed and discussed, and the overall project has been approved by core developers. For example, by writing a PEP :-)

--

In short, I'm not sure of what can or should be done right now for Include/pycapi/ :-)

I wrote the PR to open the discussion :-)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue35134>
_______________________________________