[Python-Dev] Making PyInterpreterState an opaque type

[Eric Snow]

On Sat, Feb 16, 2019 at 3:16 AM Jeroen Demeyer <J.Demeyer at ugent.be> wrote:
> On 2019-02-16 00:37, Eric Snow wrote:
> > One thing that would help simplify changes
> > in this area is if PyInterpreterState were defined in
> > Include/internal.
>
> How would that help anything?

I'm talking just about changes in the runtime implementation.  A lot
of of the runtime-related API is defined in Include/internal.  Relying
on the public headers (i.e. Include/*) for internal runtime API can
complicate changes there.  I've run into this recently.  Moving more
internal API into the internal headers helps with that problem.
Having distinct header files for the internal API is a relatively new
thing (i.e. in the last year), which is why some of the internal API
is still defined in the public header files.

> I don't like the idea (in general, I'm not
> talking about PyInterpreterState specifically) that external modules
> should be second-class citizens compared to modules inside CPython.
>
> If you want to break the undocumented API, just break it. I don't mind.
> But I don't see why it's required to move the include to
> Include/internal for that.

Keep in mind that the "internal" (or "private") API is intended for
use exclusively in the runtime and in the builtin modules.
Historically our approach to keeping API private was to use underscore
prefixes and to leave them out of the documentation (along with
guarding with "#ifndef Py_LIMITED_API").  However, this has lead to
occasional confusion and breakage, and even to leaking things into the
stable ABI that shouldn't have been.  Lately we've been working on
making the distinction between internal and public API (and stable
ABI) more clear and less prone to accidental exposure.  Victor has
done a lot of work in this area.

So I'd like to understand your objection.  Is it with exposing some
things only under Py_BUILD_CORE (i.e. when building Python itself)?
Is it to having "private" C-API in general?  Is it just to having
separate include directories?

-eric