[Python-Dev] C API changes

[Armin Rigo]

Hi Hugo, hi all,

On Sun, 18 Nov 2018 at 22:53, Hugh Fisher <hugo.fisher at gmail.com> wrote:
> I suggest that for the language reference, use the license plate
> or registration analogy to introduce "handle" and after that use
> handle throughout. It's short, distinctive, and either will match
> up with what the programmer already knows or won't clash if
> or when they encounter handles elsewhere.

FWIW, a "handle" is typically something that users of an API store and
pass around, and which can be used to do all operations on some
object.  It is whatever a specific implementation needs to describe
references to an object.  In the CPython C API, this is ``PyObject*``.
I think that using "handle" for something more abstract is just going
to create confusion.

Also FWIW, my own 2 cents on the topic of changing the C API: let's
entirely drop ``PyObject *`` and instead use more opaque
handles---like a ``PyHandle`` that is defined as a pointer-sized C
type but is not actually directly a pointer.  The main difference this
would make is that the user of the API cannot dereference anything
from the opaque handle, nor directly compare handles with each other
to learn about object identity.  They would work exactly like Windows
handles or POSIX file descriptors.  These handles would be returned by
C API calls, and would need to be closed when no longer used.  Several
different handles may refer to the same object, which stays alive for
at least as long as there are open handles to it.  Doing it this way
would untangle the notion of objects from their actual implementation.
In CPython objects would internally use reference counting, a handle
is really just a PyObject pointer in disguise, and closing a handle
decreases the reference counter.  In PyPy we'd have a global table of
"open objects", and a handle would be an index in that table; closing
a handle means writing NULL into that table entry.  No emulated
reference counting needed: we simply use the existing GC to keep alive
objects that are referenced from one or more table entries.  The cost
is limited to a single indirection.

The C API would change a lot, so it's not reasonable to do that in the
CPython repo.  But it could be a third-party project, attempting to
define an API like this and implement it well on top of both CPython
and PyPy.  IMHO this might be a better idea than just changing the API
of functions defined long ago to make them more regular (e.g. stop
returning borrowed references); by now this would mostly mean creating
more work for the PyPy team to track and adapt to the changes, with no
real benefits.


A bientôt,

Armin.