[Cython] [cython-users] Problems with "private" functions/methods in Cython 3.0a2

[Stefan Behnel]

Hi,

thanks for the very detailed report.

I'm CC-ing cython-devel, but let's keep the discussion for this on the
cython-users list.

The background here is Github issue 1382 and PR 3123

https://github.com/cython/cython/issues/1382

https://github.com/cython/cython/pull/3123

This was newly merged for alpha 2, and I consider it somewhat provisional.
It makes "private" Python names follow Python semantics, but if that poses
excessive breakage for existing code, then we'll have to find a pragmatic
way to deal with it.


'Toni Barth' via cython-users schrieb am 23.04.20 um 19:07:
> I just updated Cython from 3.0a1 to 3.0a2 in one of my projects and
> noticed that it seems to introduce issues with private
> functions/methods, means methods with leading __.
> 
> I noticed two cases which now fail in my project. I'm however unsure if
> thats supposed to happen or if its a problem with the new Cython build.
> 
> 
> Case 1: private function not declared
> 
> 
> crash.pxd:
> 
> 
> cpdef __foo()
> 
> cdef class Bar:
>   cpdef run(Bar self)
> 
> 
> crash.pyx:
> 
> 
> cpdef __foo():
>   print("foo")
> 
> cdef class Bar:
>   cpdef run(Bar self):
>     __foo()
> 
>     print("bar")
> 
> 
> Running this with the following command:
> 
> py -3 -c "import pyximport;pyximport.install();import
> crash;crash.Bar().run()"
> 
> 
> Result:
> 
> crash.c
> C:\Program Files (x86)\Microsoft Visual
> Studio\Shared\Python36_64\lib\site-packages\Cython\Compiler\Main.py:344:
> FutureWarning: Cython directive 'language_level' not set, using '3str'
> for now (Py3). This has changed from earlier releases! File:
> D:\crash\crash.pxd
>   tree = Parsing.p_module(s, pxd, full_module_name)
> 
> Error compiling Cython file:
> ------------------------------------------------------------
> ...
> cpdef __foo():
>   print("foo")
> 
> cdef class Bar:
>   cpdef run(Bar self):
>     __foo()
>    ^
> ------------------------------------------------------------
> 
> crash.pyx:6:4: undeclared name not builtin: __foo

At least this looks like a bug to me. Global module names should not be
impacted by this change.


> Error compiling Cython file:
> ------------------------------------------------------------
> ...
> cpdef __foo():
>   print("foo")
> 
> cdef class Bar:
>   cpdef run(Bar self):
>     __foo()
>    ^
> ------------------------------------------------------------
> 
> crash.pyx:6:4: '__foo' redeclared
> 
> Error compiling Cython file:
> ------------------------------------------------------------
> ...
> cpdef __foo()
>            ^
> ------------------------------------------------------------
> 
> crash.pxd:1:12: Previous declaration is here

Lovely :)



> Another problem, probably related, occurs when trying to override a
> private method in a derived class:
> 
> 
> Case 2: overriding private class methods
> 
> 
> base.pxd:
> 
> 
> cdef class Base:
>   cdef void __foo(Base self)
>   cpdef run(Base self)
> 
> 
> base.pyx:
> 
> 
> cdef class Base:
> 
>   cdef void __foo(Base self):
>     print("foo!")
>  
>   cpdef run(Base self):
>     self.__foo()
> 
> 
> extend.pxd:
> 
> 
> from base cimport Base
> 
> cdef class Extend(Base):
>   pass
> 
> 
> extend.pyx:
> 
> 
> cdef class Extend(Base):
> 
>   cdef void __foo(Extend self):
>     print("bar")
> 
> Running with the following command:
> 
> py -3 -c "import pyximport;pyximport.install();import
> extend;extend.Extend().run()"
> 
> 
> Results in the following output:
> 
> 
> extend.c
> C:\Program Files (x86)\Microsoft Visual
> Studio\Shared\Python36_64\lib\site-packages\Cython\Compiler\Main.py:344:
> FutureWarning: Cython directive 'language_level' not set, using '3str'
> for now (Py3). This has changed from earlier releases! File:
> D:\crash\extend.pxd
>   tree = Parsing.p_module(s, pxd, full_module_name)
> 
> Error compiling Cython file:
> ------------------------------------------------------------
> ...
> cdef class Extend(Base):
> 
>   cdef void __foo(Extend self):
>       ^
> ------------------------------------------------------------
> 
> extend.pyx:3:7: C method '_Extend__foo' not previously declared in
> definition part of extension type 'Extend'

This, OTOH, is intended. Private methods are not meant to be inherited and
overridden. That's what makes them "private" :) That's consistent with
Python semantics.

I think it's somewhat up for debate if this should apply in the same way to
cdef classes as for Python classes. But from a language design point of
view, having this kind of difference between the two seems a bad idea.

Do you feel like argueing against this? :)

Stefan