[capi-sig] Problem with reference

[Stefan Behnel]

M.-A. Lemburg wrote:
> On 2009-03-03 10:38, Stefan Behnel wrote:
>> Hrvoje Niksic wrote:
>>> Stefan Behnel writes:
>>>> pycnweb wrote:
>>>>> I am now writing a python extension, but I confuse with the
>>>>> reference.
>>>> If you care more about working code than fighting confusion, Cython
>>>> might be worth a look.
>>> Stefan, please stop doing this.  This list is about helping people
>>> using Python/C, not Cython advocacy.
>>
>> Where is the difference?
>
> The list would be called cython-sig otherwise :-)
>
> There are many other alternatives to approaching Python extension writing
> in C/C++:
>
>  * use ctypes
>  * use Boost.Python
>  * use SWIG
>  * use sip
>  * use PyCXX
>  * use PyRex/Cython
>  * use Robin
>  * ...
>
> (there are lots more)

I know almost all of these at least by name. However, most of them are
wrapper generators, which clearly limits their field of applicability (and
rarely leads to good wrappers without major manual interaction). Cython is
not a wrapper generator. It's a programming language that sits right
between Python and C and generates code for the C-API of Python. That's
why its related to this mailing list and why I advocate it to people who
are clearly starting to write code against that API. It just makes their
life a lot easier without sacrificing anything.

It's similar to what the Jython developers say, "we write C so you don't
have to". If you ever need to make a direct call to Python's C-API in
Cython code, it's one cimport away. In all other cases, it's better to
concentrate on real logic and algorithmic optimisations in working code,
than on counting references, converting types, building dicts and writing
extension class structs. It's pretty close to Python's own spirit in that
regard. Plus, no matter how long you grow your code, Cython just doesn't
start to get in your way - which has always been the case for any wrapper
generator I've ever worked with.

I do see ctypes as a pure-Python-level alternative, but it's not related
to this list in any obvious way.


> However, this list is about the layer underneath all of those.
> It's about the Python C API and there are many good reasons
> to use it directly rather than using one of the above wrappers
> or wrapper generators.

I agree regarding the wrapper generators, but I do not agree regarding the
need to call Python's C-API. I know that from personal experience as the
author of lxml (which is written in Cython). lxml's code uses direct C-API
calls in quite a few places, but that's almost purely for historical
reasons. Today, most of that has become a premature performance
optimisation that only makes the code less obvious. Last year, I started
refactoring that back into Python/Cython code, because in 95% of the code,
there just isn't any reason left to use direct C-API calls. Plain Cython
code is at least as fast and often faster than hand-written code,
especially when it comes to interaction with the Python runtime.

Call me a burnt child, I hope this makes it understandable that I'm trying
to keep new C-API users from investing tons of time into their C code,
just to notice that it will finally run, say, 30% slower and be less
portable than what they could have had with 10-15% of the code size in
Cython. And since the factor of 10 in code size is very real, it leaves
you with 80-95% of the plain-C development time to benchmark and optimise
your code.

Stefan