No macros in Python

[jepler at unpythonic.net]

I really don't want to be taken as merely trying to poke holes in your
presentation of how python-with-macros would look.  Instead, I'm trying to
hit on what seem to be the hard issues of "getting there", with cpython as
a starting point.

The biggest omission is how to allow 'suite's in macros.  For instance,
one recent request that might be well served by macros would be a way to
abstract
	lock.acquire()
	try: body
	finally: lock.release()
into a macro.  If your proposal is all there is to support macros, then I
have to write
	def body(): suite
	with_lock(lock, body)
but that can be implemented as a function.  I'd rather just write 'with
lock: suite'.  So some additional (non-function) syntax has to be introduced 
for this purpose.  It's here that things start to get hairy: your proposal
*could* be implemented by leaving the grammar unchanged, but changing the
compiler's behavior when the production that now means "function call" is
encountered.

It's at the point that this syntax must be added that things seem to get
ugly.  For instance, you might permit at most one 'suite' to be given to a
macro, and mark the beginning of a macro invocation with a statement that
starts with a symbol that can't now begin a statement, or with a new token.
For instance,
	macro with_lock lock: body
(Having a parser complicated enough to parse statements like
	with_lock lock: body
correctly, even if the parser knows that 'with_lock' is a macro, is
tough--it requires the kind of analysis that C requires to know the
difference between a variable identifer and a typedef identifierwhen it
starts on an expression like 'uint32_t i = 3;'.  Python's parser doesn't
have this kind of smarts now, as far as I know (with the slight exception
of the 'import ... as ..' statement-- "as" is not a reserved keyword))

So maybe you need to keep the function-call syntax, but find another way to
introduce a suite.  Well, how about making ': suite' an expression that is
equal to the function created by the statement
	def anonymous(): suite
then you could write
	with_lock(lock, :body)
or
	with_lock(lock, :
		executed_with_lock_held
	    , if_already_locked=:
		execute if lock is already held)
?  I'm pretty sure the compiler current suppresses creation of INDENT,
DEDENT, and NEWLINE tokens while inside grouping characters ('()', etc) so
this is another big change, this time to the tokenizer.  What about the
line ", :"?  That's NEWLINE DEDENT COMMA COLON, but with what will the
tokenizer match indentation to find the correct dedent?  Maybe the "," or
")" will have to be on the same column as the beginning of the macro, but
that looks even worse to me:
	with_lock(lock, :
		executed_with_lock_held
	, if_already_locked=:
		execute if lock is already held)

I think with_lock can still be written as a function, not a macro, by the
way, if an extension like this is adopted:
	def with_lock(lock, body, if_already_locked=lambda: None):
		if lock.acquire():
			try: return body()
			finally: lock.release()
		else: return if_already_locked()
(I think of a true macro as something that would actually affect the parse
tree of the function before it's byte-compiled, I guess, but I guess I
don't actually see this reflected in your original post)

If you don't want to fancy transformations of parse trees (and when I tried
it, it got ugly quick) then why not settle for
	def ternary(cond, lt, lf):
		if cond: return lt()
		else: return lf()
	x = ternary(f(), lambda: g(1), lambda: g(2))
as a substitute for
	a = f() ? g(1) : g(2)
?

I guess that in my mind, we don't need macros, but a nice way to create
anonymous multi-line functions (and possibly a way to modify variables
visible thanks to nested scopes).  If you have that, macros seem to simply
become a static tool for getting more performance-- and everybody knows
that nobody cares about performance in python.

Jeff