[Python-Dev] Ctypes and the stdlib (was Re: LZMA compression support in 3.3)

[Terry Reedy]

On 8/30/2011 1:05 PM, Guido van Rossum wrote:

>> I see. So there is potential for error there.
>
> To elaborate, with CPython it looks pretty solid, at least for
> functions and constants (does it do structs?). You must manually
> declare the name and signature of a function, and Pyrex/Cython emits C
> code that includes the header and calls the function with the
> appropriate types. If the signature you declare doesn't match what's
> in the .h file you'll get a compiler error when the C code is
> compiled. If (perhaps on some platforms) the function is really a
> macro, the macro in the .h file will be invoked and the right thing
> will happen. So far so good.
>
> The problem lies with the PyPy backend -- there it generates ctypes
> code, which means that the signature you declare to Cython/Pyrex must
> match the *linker* level API, not the C compiler level API. Thus, if
> in a system header a certain function is really a macro that invokes
> another function with a permuted or augmented argument list, you'd
> have to know what that macro does. I also don't see how this would
> work for #defined constants: where does Cython/Pyrex get their value?
> ctypes doesn't have their values.
>
> So, for PyPy, a solution based on Cython/Pyrex has many of the same
> downsides as one based on ctypes where it comes to complying with an
> API defined by a .h file.

Thank you for this elaboration. My earlier comment that ctypes seems to 
be hard to use was based on observation of posts to python-list 
presenting failed attempts (which have included somehow getting function 
signatures wrong) and a sense that ctypes was somehow bypassing the 
public compiler API to make a more direct access via some private api. 
You have explained and named that as the 'linker API', so I understand 
much better now.

Nothing like 'linker API' or 'signature' appears in the ctypes doc. All 
I could find about discovering specific function calling conventions is 
"To find out the correct calling convention you have to look into the C 
header file or the documentation for the function you want to call." 
Perhaps that should be elaborated to explain, as you did above, the need 
to trace macro definitions to find the actual calling convention and the 
need to be aware that macro definitions can change to accommodate 
implementation detail changes even as the surface calling conventions 
seems to remain the same.

-- 
Terry Jan Reedy