[Python-ideas] Incorporating something like byteplay into the stdlib

[Victor Stinner]

Hi,

I understand that you have 3 major points:

(1) byteplay lags behind CPython, it's difficult to maintain it

(2) you want to integrate the code features of byteplay into the dis module

(3) you want to use a new API of the dis module in the PEP 511 for
bytecode transformers


For the point (1), it may be fixed by the point (2).

Otherwise, I'm not interested to add byteplay into CPython. I prefer
to not promote too much the usage of bytecode transformers, since the
bytecode is very low-level: it depends on the Python minor version and
is not portable accross implementations of Python. We want to be free
to modify deeply the bytecode in CPython. Yury explained that before
than me ;-)


IMHO the point (2) is just fine. Go ahead!


I'm opposed to the point (3) because it would couple too much the
exact implementation of bytecodes to the PEP 511 API. I tried to write
an API which can be implemented by all Python implementations, not
only CPython. See:
https://www.python.org/dev/peps/pep-0511/#other-python-implementations

IMHO a "black-box" bytecode transformer API is the best we can do to
support all Python implementations.

By the way, (3) requires to reimplement the dis module in C to
bootstrap Python. IMHO it will be boring to write the C code, and much
more annoying to maintain it. So quickly, we will refuse any
enhancement on the API. I don't think that you want that, since you
want to experiment new things, right?


It is already to build the API you described *on top of the PEP 511*.
Such API would probably be specific to CPython and to byteplay (or dis
module if you enhance it). Example:
---
import dis
import sys

# PEP 511 code transformer
class ByteplayRegistry:
    name = "bytecode"

    def __init__(self):
        self._transformers = []

    def register(self, name, transformer):
        # FIXME: update self.name?
        self._transformers.append(transformer)

    def disassemble(self, code):
        # FIXME: optional support for byteplay?
        return dis.dis(code)

    def assemble(self, asm, code):
        # FIXME: implement assembler
        return ...

    def code_transformer(self, code, context):
        # disassemble() and assemble() is only called once for all transformers
        asm = self.disassemble(code)
        for transformer in self._transformers:
            asm = transformer(asm, code, context)
        return self.assemble(asm, code)


def global_to_const(asm, code, context):
    # FIXME: implement optimization
    return asm


byteplay_registry = ByteplayRegistry()
byteplay_registry.register("global_to_const", global_to_const)

sys.set_code_transformers([byteplay_registry])
---


2016-02-12 20:16 GMT+01:00 Andrew Barnert <abarnert at yahoo.com>:
> Your API passes only the bytecode string, consts, lnotab, and global names. There is no way to write a bytecode processor that looks up locals with that interface, because you didn't pass the locals.

Oh, it looks like you are referring to an old version of the PEP 511
which passed 5 parameters to code_transformer(). The latest version
now pass a whole code object to the transformer, and the transformer
must return a new code object:
https://www.python.org/dev/peps/pep-0511/#code-transformer-method

I just saw that I forgot to update the example of bytecode
transformer. It's now fixed.
https://www.python.org/dev/peps/pep-0511/#bytecode-transformer

I also updated the implementation:
https://bugs.python.org/issue26145

Victor