[Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

[Brett Cannon]

On Mon., Nov. 19, 2018, 14:04 Neil Schemenauer <nas-python at arctrix.com
wrote:

> On 2018-11-19, Antoine Pitrou wrote:
> > There are important use cases for the C API where it is desired to have
> > fast type-specific access to Python objects such as tuples, ints,
> > strings, etc.  This is relied upon by modules such as _json and _pickle,
> > and third-party extensions as well.
>
> Thank you for pointing this out.  The feedback from Stefan on what
> Cython would like (e.g. more access to functions that are currently
> "internal") is useful too.  Keeping our dreams tied to reality
> is important. ;-P
>
> It seems to me that we can't "have our cake and eat it too". I.e. on
> the one hand hide CPython implementation internals but on the other
> hand allow extensions that want to take advantage of those internals
> to provide the best performance.
>

No, but those are different APIs as well. E.g. no one is saying CPython has
to do away with any of its API. What I and some others have said is the
CPython API is too broad to be called "universal".


> Maybe we could have a multiple levels of API:
>
> A) maximum portability (Py_LIMITED_API)
>
> B) source portability (non-stable ABI, inlined functions)
>
> C) portability but poor performance on non-CPython VMs
>    (PySequence_Fast_ITEMS, borrowed refs, etc)
>

I don't know own how doable that is as e.g. borrowed refs are not pleasant
to simulate.


> D) non-portability, CPython specific (access to more internals like
>    Stefan was asking for).  The extension would have to be
>    re-implemented on each VM or provide a pure Python
>    alternative.


> I think it would be nice if the extension module could explicitly
> choose which level of API it wants to use.
>

Yes, and I thought we were working towards nesting our header files so you
very clearly opted into your level of compatibility.

In my head there's:
- bare minimum, cross-VM, gets you FFI
- CPython API for more performance that we're willing to maintain
- Everything open for e.g. CPython with no compatibility guarantees

Due note my first point isn't necessarily worrying about crazy performance
to start. I would assume an alternative VM would help make up for this with
a faster runtime where dropping into C is more about FFI than performance
(we all know PyPy, for instance, wished people just wrote more Python code).

Otherwise we're back to the idea of standardizing on some Cython solution
to help make perfect easier without tying oneself to the C API (like
Julia's FFI solution).


> It would be interesting to do a census on what extensions are out
> there.  If they mostly fall into wanting level "C" then I think this
> API overhaul is not going to work out too well.  Level C is mostly
> what we have now.  No point in putting the effort into A and B if no
> one will use them.


It won't until someone can show benefits for switching. This is very much a
chicken-and-egg problem.


_______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181119/556ccc4d/attachment.html>