[Python-ideas] PEP 511: API for code transformers

[Nick Coghlan]

On 17 January 2016 at 02:22, Sjoerd Job Postmus <sjoerdjob at sjec.nl> wrote:
> On Sat, Jan 16, 2016 at 12:06:58PM +0100, Petr Viktorin wrote:
>> The "transformer" API can be used for syntax extensions as well, but the
>> registration needs to be different so the effects are localized. For
>> example it could be something like::
>>
>>     importlib.util.import_with_transformer(
>>         'mypackage.specialmodule', MyTransformer())
>>
>> or a special flag in packages::
>>
>>     __transformers_for_submodules__ = [MyTransformer()]
>>
>> or extendeding exec (which you actually might want to add to the PEP, to
>> make giving examples easier)::
>>
>>     exec("print('Hello World!')", transformers=[MyTransformer()])
>>
>> or making it easier to write an import hook with them, etc...
>
> So, you'd have to supply the transformer used before importing? That
> seems like a troublesome solution to me.

I think Sjoerd's confusion here is a strong argument in favour of
clearly and permanently distinguishing semantics preserving code
optimizers (which can be sensibly applied externally and/or globally),
and semantically significant code transformers (which also need to be
taken into account when *reading* the code, and hence should be
visible locally, at least at the module level, and often at the
function level).

Making that distinction means we can be clear that the transformation
case is already well served by import hooks that process alternate
filename extensions rather than standard Python source or bytecode
files, encoding cookie tricks (which are visible as a comment in the
module header), and function decorators that alter the semantics of
the functions they're applied to.

The case which *isn't* currently well served is transparently applying
a semantics preserving code optimiser like FAT Python - that's a
decision for the person *running* the code, rather than the person
writing it, so this PEP is about providing the hooks at the
interpreter level to let them do that. While we can't *prevent* people
from using these new hooks with semantically significant transformers,
we *can* make it clear that we think actually doing is a bad idea, as
it is likely to result in a tightly coupled hard to maintain code base
that can't even be read reliably without understand the transforms
that are being implicitly applied.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia