[Python-Dev] Use of Cython

[Stefan Behnel]

Ronald Oussoren via Python-Dev schrieb am 06.08.2018 um 15:25:
>> On 5 Aug 2018, at 18:14, Nick Coghlan wrote:
>> On 5 August 2018 at 18:06, Ronald Oussoren wrote:
>>> I’m not sure if I understand this, ctypes and cffi are used to access C APIs
>>> without writing C code including the CPython API (see for example
>>> <https://github.com/abarnert/superhackyinternals/blob/master/internals.py>).
>>>
>>> The code code below should be mostly equivalent to the Cython example posted
>>> earlier:
>>>
>>> import unittest
>>> import ctypes
>>> from ctypes import pythonapi
>>>
>>> class PyObject(ctypes.Structure):
>>>    _fields_ = (
>>>        ('ob_refcnt', ctypes.c_ssize_t),
>>>    )
>>>
>>> pythonapi.PyList_Append.argtypes = [ctypes.py_object, ctypes.py_object]
>>>
>>> def refcount(v):
>>>    return PyObject.from_address(id(v)).ob_refcnt
>>
>> The quoted code is what I was referring to in:
>> ====
>> ctypes & cffi likely wouldn't help as much in the case, since they
>> don't eliminate the need to come up with custom code for parts 3 & 4,
>> they just let you write that logic in Python rather than C.
>> ====
> 
> And earlier Nick wrote:
>> 1. The test case itself (what action to take, which assertions to make about it)
>> 2. The C code to make the API call you want to test
>> 3. The Python->C interface for the test case from 1 to pass test
>> values in to the code from 2
>> 4. The C->Python interface to get state of interest from 2 back to the
>> test case from 1
> 
> For all of Cython, ctypes and cffi you almost never have to write (2), and hence (3) and (4), but can write that code in Python.

Which then means that you have a mix of Python and C in many cases. I guess
that's what you meant with your next sentence:

> This is at the code of making it harder to know which bits of the CPython API are used in step (2), which makes it harder to review a testcase. 

Not sure I understand this correctly, but I think we're on the same page
here: writing test code in C is cumbersome, writing test code in a mix of C
and Python across different files is aweful. And making it difficult to
write or even just review test code simply means that people will either
waste their precious contribution time on it, or try to get around it.


> BTW. In other projects I use tests there almost all of the test code is in C, the unittest runner only calls a C function and uses the result of that function to deduce if the test passed or failed. This only works nicely for fairly simple tests (such as the example test in this thread), not for more complicated and interesting tests due to having to write more C code.

I totally agree with that. For less trivial tests, people will often want
to stear the test case at the C level, because some things are really
difficult to do from Python. Good luck making assertions about reference
counts when you're orchestrating the C-API through ctypes. And this is
where Cython shines – your code *always* ends up running in C, regardless
of how much of it is plain Python. But at any point, you can do pretty
arbitrary C things, all in the same function. And unittest can execute that
function directly for you, without having to write a Python wrapper or
separate test runner.

And for the really hard cases, you can resort to writing a literal C code
snippet in your Cython source file (as a string) and let Cython drop it
into the file it generates, e.g. to quickly define a macro, a little C
function, or an interface wrapper around a C macro that would otherwise be
difficult to test. That little feature removes the last reason for
resorting to a separate C file.

Stefan