[Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

[Nathaniel Smith]

On Tue, Nov 20, 2018 at 6:05 PM Glenn Linderman <v+python at g.nevcal.com> wrote:
>
> On 11/20/2018 2:17 PM, Victor Stinner wrote:
>> IMHO performance and hiding implementation details are exclusive. You
>> should either use the C API with impl. details for best performances,
>> or use a "limited" C API for best compatibility.
>
> The "limited" C API concept would seem to be quite sufficient for extensions that want to extend Python functionality to include new system calls, etc. (pywin32, pyMIDI, pySide, etc.) whereas the numpy and decimal might want best performance.

To make things more complicated: numpy and decimal are in a category
of modules where if you want them to perform well on JIT-based VMs,
then there's no possible C API that can achieve that. To get the
benefits of a JIT on code using numpy or decimal, the JIT has to be
able to see into their internals to do inlining etc., which means they
can't be written in C at all [1], at which point the C API becomes
irrelevant.

It's not clear to me how this affects any of the discussion in
CPython, since supporting JITs might not be part of the goal of a new
C API, and I'm not sure how many packages fall between the
numpy/decimal side and the pure-ffi side.

-n

[1] Well, there's also the option of teaching your Python JIT to
handle LLVM bitcode as a source language, which is the approach that
Graal is experimenting with. It seems completely wacky to me to hope
you could write a C API emulation layer like PyPy's cpyext, and
compile that + C extension code to LLVM bitcode, translate the LLVM
bitcode to JVM bytecode, inline the whole mess into your Python JIT,
and then fold everything away to produce something reasonable. But I
could be wrong, and Oracle is throwing a lot of money at Graal so I
guess we'll find out.

-- 
Nathaniel J. Smith -- https://vorpus.org