[Python-ideas] PEP 511: Add a check function to decide if a "language extension" code transformer should be used or not

Wed Jan 27 16:01:30 EST 2016

My thought about decorators is that they allow obvious scoping of changes
for the reader. Anything that becomes module scope or is implied based on
system state that is set in another module will make debugging and reading
much harder. Both lazy_python and codetransformer use bytecode
manipulation; however, it is a purely opt-in system where the transformed
function is decorated. This keeps the transformations in view while you are
reading the code that is affected by them. I would find debugging a project
much more difficult if I needed to remember that the order my modules were
imported matters a lot because they setup a bunch of state. I am not sure
why people want the module to be the smallest unit that is transformed when
really it is the code object that should be the smallest unit. This means
class bodies and functions. If we treat the module as the most atomic unit
then you wouldn't be able to use something like `asconstants`

This is a really great local optimzation when calling a function in a loop,
especially builtins that you know will most likely never change and you
don't want to change if they do. For example:

In [1]: from codetransformer.transformers.constants import asconstants

In [2]: @asconstants(a=1)
   ...: def f():
   ...:     return a
   ...:

In [3]: a = 5

In [4]: f()
Out[4]: 1

In [5]: @asconstants('pow')  # string means use the built in for this name
   ...: def g(ns):
   ...:     for n in ns:
   ...:         yield pow(n, 2)
   ...:

In [6]: list(g([1, 2, 3]))
Out[6]: [1, 4, 9]

In [7]: dis(g)
  3           0 SETUP_LOOP              28 (to 31)
              3 LOAD_FAST                0 (ns)
              6 GET_ITER
        >>    7 FOR_ITER                20 (to 30)
             10 STORE_FAST               1 (n)
             13 LOAD_CONST               0 (<built-in function pow>)
             16 LOAD_FAST                1 (n)
             19 LOAD_CONST               1 (2)
             22 CALL_FUNCTION            2 (2 positional, 0 keyword pair)
             25 YIELD_VALUE
             26 POP_TOP
             27 JUMP_ABSOLUTE            7
        >>   30 POP_BLOCK
        >>   31 LOAD_CONST               2 (None)
             34 RETURN_VALUE

This is a simple optimization that people emulate all the time with things
like `sum_ = sum` before the loop or `def g(ns, *, _sum=sum)`.  This cannot
be used at module scope because often you only think it is safe or worth it
to lock in the value for a small segment of code.
Hopefully this use case is being considered as I think this is a very
simple, non-semantics preserving case that is not very and also practical.

On Wed, Jan 27, 2016 at 3:20 PM, Brett Cannon <brett at python.org> wrote:

>
>
> On Wed, 27 Jan 2016 at 08:49 Andrew Barnert via Python-ideas <
> python-ideas at python.org> wrote:
>
>> On Jan 27, 2016, at 07:39, Victor Stinner <victor.stinner at gmail.com>
>> wrote:
>> >
>> > Hi,
>> >
>> > Thank you for all feedback on my PEP 511. It looks like the current
>> > blocker point is the unclear status of "language extensions": code
>> > tranformers which deliberately changes the Python semantics. I would
>> > like to discuss how we should register them. I think that the PEP 511
>> > must discuss "language extensions" even if it doesn't have to propose
>> > a solution to make their usage easier. It's an obvious usage of code
>> > transformers. If possible, I would like to find a compromise to
>> > support them, but make it explicit that they change the Python
>> > semantics.
>>
>> Is this really necessary?
>>
>> If someone is testing a language change locally, and just wants to use
>> your (original) API for his tests instead of the more complicated
>> alternative of building an import hook, it works fine. If he can't deploy
>> that way, that's fine.
>>
>> If someone builds a transformer that adds a feature in a way that makes
>> it a pure superset of Python, he should be fine with running it on all
>> files, so your API works fine. And if some files that didn't use any of the
>> new features get .pyc files that imply they did, so what?
>>
>> If someone builds a transformer that only runs on files with a different
>> extension, he already needs an import hook, so he might as well just call
>> his transformer from the input hook, same as he does today.
>>
>
> And the import hook is not that difficult. You can reuse everything from
> importlib without modification except for needing to override a single
> method in some loader to do your transformation (
> https://docs.python.org/3/library/importlib.html#importlib.abc.InspectLoader.source_to_code).
> Otherwise the only complication is instantiating the right classes and
> setting the path hook in `sys.path_hooks`.
>
>
>>
>> So... What case is served by this new, more complicated API that wasn't
>> already served by your original, simple one (remembering that import hooks
>> are already there as a fallback)?
>>
>
> As Victor pointed out, the discussion could end in "nothing changed, but
> we at least discussed it". I think both you and I currently agree that's
> the answer to his question. :)
>
> -Brett
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20160127/0260a551/attachment-0001.html>