Python and the need for speed

[Steve D'Aprano]

On Tue, 11 Apr 2017 07:11 pm, Chris Angelico wrote:

> (Picking up one small part of your email; for the most part, I believe
> we are in belligerent agreement.)

:-)


[...]
> Did literal_eval get changed in 3.6 to add support for this? Nope. It
> just hands the string of text to the compiler and says "give me an
> AST", and then does its whitelisted walk on that. That's a pretty huge
> slab of work that would have to be redone if compile() were removed
> from the language. 

Not necessarily. You are still focused on *one specific implementation*,
where literal_eval() is written in pure Python and calls the compile()
built-in function.

But that's not the only way to solve the problem. That's implementation, not
interface. Here's another implementation:

Suppose the Python compiler/interpreter included a parser that takes a
string and compiles it to an AST. Pretty much every modern language has
that, the days of BASIC interpreters that loop over lines of source code
executing them as they go are long gone. So we need something to parse
source to AST, just so the interpreter can interpret the source code.

It just needs to exist. It doesn't have to be a public part of the language.

So the interpreter relies on this internal "compile-to-AST". And
literal_eval() can do exactly the same thing, if it is part of the
interpreter. There's no reason why literal_eval() must be written in pure
Python. It can be implemented any way you like, anywhere you like.

Obviously this requires hooking literal_eval() into the interpreter as a
built-in command. Maybe its even a single op-code in the (Turbo)Python
virtual machine. We can make the op-codes as complex as we need! The
existing op-codes range from the incredibly simple:

POP_TOP

    Removes the top-of-stack (TOS) item.


to quite complex:


SETUP_WITH(delta)

    This opcode performs several operations before a with block 
    starts. First, it loads __exit__() from the context manager
    and pushes it onto the stack for later use by WITH_CLEANUP.
    Then, __enter__() is called, and a finally block pointing to
    delta is pushed. Finally, the result of calling the enter 
    method is pushed onto the stack. The next opcode will either
    ignore it (POP_TOP), or store it in (a) variable(s) 
    (STORE_FAST, STORE_NAME, or UNPACK_SEQUENCE).

https://docs.python.org/3/library/dis.html

so there is absolutely no reason why literal_eval() cannot be a built-in
function that uses the exact same parser as the interpreter, if that's
important to you.

Just because literal_eval() has access to the interpreter's parser, doesn't
mean that there has to be a compile() builtin that can handle arbitrary
source code.


> That's why a "no-runtime-compilation-allowed" Python has to either
> lose or manually reimplement this. I very much doubt that CPython
> upstream is interested in a patch to literal_eval to make it not use
> compile - it makes it slower, larger, and more work to maintain,
> merely in order to remove a dependency on something that's a core
> feature. 

Nobody is talking about CPython removing the compile() builtin.

But if TurboPython wants to remove the compile() builtin, AND it becomes
super-popular -- maybe Google and Facebook and Apple decide to throw away
Go, PHP and Swift and move exclusively to Python *wink* -- then perhaps
Python 4.5 or 5.2 will actually include a switch to disable compile(),
exec() and eval() and enable a bunch of optimizations. Anything is
possible, if enough people want it.

*If*.


-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.