[Python-Dev] Use of Cython

[Stefan Behnel]

Yury,

given that people are starting to quote enthusiastically the comments you
made below, let me set a couple of things straight.

Yury Selivanov schrieb am 07.08.2018 um 19:34:
> On Mon, Aug 6, 2018 at 11:49 AM Ronald Oussoren via Python-Dev wrote:
> 
>> I have no strong opinion on using Cython for tests or in the stdlib, other than that it is a fairly large dependency.  I do think that adding a “Cython-lite” tool the CPython distribution would be less ideal, creating and maintaining that tool would be a lot of work without clear benefits over just using Cython.
> 
> Speaking of which, Dropbox is working on a new compiler they call "mypyc".
> 
> mypyc will compile type-annotated Python code to an optimized C.

That's their plan. Saying that "it will" is a bit premature at this point.
The list of failed attempts at writing static Python compilers is rather
long, even if you only count those that compile the usual "easy subset" of
Python.

I wish them the best of luck and endurance, but they have a long way to go.


> The
> first goal is to compile mypy with it to make it faster, so I hope
> that the project will be completed.

That's not "the first goal". It's the /only/ goal. The only intention of
mypyc is to be able to compile and optimise enough of Python to speed up
the kind or style of code that mypy uses.


> Essentially, mypyc will be similar
> to Cython, but mypyc is a *subset of Python*, not a superset.

Which is bad, right? It means that there will be many things that simply
don't work, and that you need to change your code in order to make it
compile at all. Cython is way beyond that point by now. Even RPython will
probably continue to be way better than mypyc for quite a while, maybe
forever, who knows.


> Interfacing with C libraries can be easily achieved with cffi.

Except that it will be fairly slow. cffi is not designed for static
analysis but for runtime operations. You can obviously also use cffi from
Cython – but then, why would you, if you can get much faster code much more
easily without using cffi?

That being said, if someone wants to write a static cffi optimiser for
Cython, why not, I'd be happy to help with my advice. The cool thing is
that this can be improved gradually, because compiling the cffi code
probably already works out of the box. It's just not (much) faster than
when interpreted.


> Being a
> strict subset of Python means that mypyc code will execute just fine
> in PyPy.

So does normal (non-subset) Python code. You can run it in PyPy, have
CPython interpret it, or compile it with Cython if you want it to run
faster in CPython, all without having to limit yourself to a subset of
Python. Seriously, you make this sound like requiring users to rewrite
their code to make it compilable with mypyc was a good thing.


> They can even apply some optimizations to it eventually, as
> it has a strict and static type system.

In case "they" refers to PyPy here, then I remember the PyPy project
stating very clearly that they are not interested in PEP-484 typing because
it is completely irrelevant for their JIT. It's really best for them to
ignore it.

That's similar for Cython, simply because PEP-484 typing isn't designed for
optimisation at all, definitely not for C-level optimisation. Still, Cython
can make some use of PEP-484 typing, if you use it to define specific C
types. That allows normal execution in CPython, static analysis with
PEP-484 analyser tools (e.g. PyCharm or mypy), and efficient optimisation
by Cython. The best of all worlds. See the docs on how to do that, it's
been supported for about a year now (and has been around in a similar,
non-PEP-484 form for years before that PEP even existed).


> I'd be more willing to start using mypyc+cffi in CPython stdlib
> *eventually*, than Cython now.  Cython is a relatively complex and
> still poorly documented language.

You are free to improve the documentation or otherwise help us find and
discuss concrete problems with it. Calling Cython a "poorly documented
language" could easily feel offensive towards those who have put a lot of
work into the documentation, wiki, tutorials, trainings and what not that
help people use the language. Even stack overflow is getting better and
better in documenting Cython these days, even though responses over there
that describe work-arounds tend to get outdated fairly quickly.

Besides, don't forget that it's Python, so consider reading the Python
documentation first if something is unclear. And maybe some documentation
of C data types as well. (.5 wink)


> I'm speaking from experience after
> writing thousands of lines of Cython in uvloop & asyncpg.  In skillful
> hands Cython is amazing, but I'd be cautious to advertise and use it
> in CPython.

Why not? You didn't actually give any reasons for that.


> I'm also -1 on using Cython to test C API. While writing C tests is
> annoying (I wrote a fair share myself), their very purpose is to make
> third-party tools/extensions more stable. Using a third-party tool to
> test C API to track regressions that break third-party tools feels
> wrong.

I don't understand that argument. What's wrong about using a tool that
helps you get around writing boiler plate code? The actual testing does not
need to be done by Cython at all, you can write it any way you like.

Stefan