[Cython] [ENH] Compile-time code execution (macros / templates)

[Celelibi]

1) My suggestion would only run some standard python code during the
compilation. Running the cdef functions at compile-time doesn't have to
be supported. At least not for a first version.

Even if it gets supported, calling external functions (written in C)
doesn't have to be supported either. If only running the cdef functions
is supported, then it means cython has the full code and would "only"
have to implement the C semantics in python. And actually, not the whole
C semantics has to be supported. The language D has quite some
restrictions on the code that can run at compile-time. Especially when
it comes to pointers.

But first, running only python code seems quite doable.


2) That's something I didn't elaborate on in my initial message to keep
it short (kinda). But indeed, some compile-time functions just wouldn't
make sens at runtime. Especially, `twice` and `make_ptr_class` wouldn't
as far as I can tell, because returning a cdef function or class at run
time doesn't really make sens right now. (Maybe in the future if cython
support compiling closures to C?)

I guess some kind annotations would be necessary to mark a function (or
maybe even a class?) as "compile-time only". Of course, other functions
and classes might make sens to be used at compile-time and at run-time.


The only thing I'm not sure about, is how to determine what should get
evaluated at compile-time. A few places in the code should probably get
automatically arked for CTFE, like decorators on cdef functions.

IIRC, D uses "enum" to declare a compile-time constant. But this only
solves half the issue. What about top-level code evaluation, like the
`if os =="windows"` example?
Maybe this could be wrapped in a function evaluated at compile-time. And
thuse only function evaluation would be requred.
Something like this:


@cython.ctfe_only
def choice_thread_something(os):
	if os == "windows":
		cdef thread_something():
			# Windows behavior
	else:
		cdef thread_something():
			# POSIX behavior
	return thread_something

ctfe thread_something = choice_thread_something(os)


Where the "ctfe" keyword would play a similar role to "enum" in D and
force the evaluation of the right hand side during the compilation.

As a technical detail, the "cdef" lines that define the thread_something
functions could be remplaced during the compilation by some python code
like:
thread_something = CdefFunction(...)

And the calling line `ctfe thread_something = ...` would just insert the
returned object into the AST. All the remaining (non-CTFE) code would
be converted to code that build the AST.



I hope that makes it more clear.


Celelibi


Le Wed, Feb 24, 2021 at 10:20:39PM +0000, da-woods a écrit :
> A few things to point out:
> 1) Because Cython is mainly a is code translation tool it doesn't currently
> have an overview of how things would eventually be evaluated - it leaves a
> lot for the C compiler to sort out. Your suggestion would presumably add a
> dependency on a C compiler at the .pyx -> .c stage (currently the only
> dependencies are Python+Cython itself).
> 2) Presumably `twice` and `make_ptr_class` would also have to work at
> run-time? They are regular `def` functions after all? That seems like a big
> change to now have to dynamically create `cdef` functions and classes at
> both Cythonization-time and runtime. And also a Python-runtime
> representation of every C type (so that you can call `make_ptr_class` at
> runtime)
> 
> I totally get the appeal of being able to wrap template-types quickly and
> easily. However, to me this idea actually seems a lot harder than something
> which creates new syntax, because you don't distinguish these special
> functions and so they have to work universally as Python functions too.
> 
> In summary: if it worked it'd be very nice, but I personally don't have any
> idea how you'd implement it within how Cython current works.
> 
> David
> 
> 
> 
> On 24/02/2021 14:29, Celelibi wrote:
> > Le Tue, Feb 23, 2021 at 11:24:42AM -0500, Prakhar Goel a écrit :
> > >     I had similar ideas in the past (assuming I understand your proposal). I'd
> > >     love it if a decorator could link to some python function. This python
> > >     function would get something that represents the cdef function (with the
> > >     ast). It would then return a transformed ast that could be spliced into
> > >     the original program.
> > As I said, I'm not sure allowing AST manipulation would be that useful.
> > I mean, it's rarely ever needed in python. Why would it be needed with
> > cython?
> > However, it would surely make the code much less readable, since the
> > code that's executed is not the code that's written.
> > 
> > The use-cases I have in mind are more like these. Which are common
> > patterns in python.
> > 
> > Compile-time implementation choice:
> > 
> > if os == "Windows":
> > 	cdef thread_something():
> > 		# Handle Windows-specific behavior
> > else:
> > 	cdef thread_something():
> > 		# Assume POSIX behavior
> > 
> > 
> > Decorators on cdef functions:
> > def twice(f):
> > 	cdef fun(x):
> > 		return f(2*x)
> > 	return fun
> > 
> > @twice
> > cdef foo(x):
> > 	# Do something math-y
> > 
> > 
> > 
> > Creating cdef classes on demand:
> > 
> > def make_ptr_class(T):
> > 	cdef class C:
> > 		cdef T *ptr
> > 	return C
> > 
> > IntPtr = make_ptr_class(int)
> > FloatPtr = make_ptr_class(float)
> > 
> > My current use-case is actually exactly this one. Creating wrappers
> > classes for pointer types. Which I currently cannot make both generic
> > and type safe.
> > 
> > 
> > The python code run at compile-time would basically manipulate python
> > objects representing cdef functions, classes and C types.
> > 
> > Internally, the way it could be implemented is that cython could
> > generate a python program that would produce the AST. But it doesn't
> > mean the AST itself has to be visible.
> > 
> > Best regards,
> > Celelibi
> > _______________________________________________
> > cython-devel mailing list
> > cython-devel at python.org
> > https://mail.python.org/mailman/listinfo/cython-devel
> 
> 
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel