Python and the need for speed

[Chris Angelico]

(Picking up one small part of your email; for the most part, I believe
we are in belligerent agreement.)

On Tue, Apr 11, 2017 at 6:19 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> Are you aware that literal_eval is *not* the same code that the Python
> interpreter uses for evaluating literals? They are kept in sync manually,
> or not kept in sync as the case may be.

Yes, I am; however, literal_eval starts by compiling to AST. (At
least, it does in current versions. I don't know about older
CPythons.) That means that, unless it gets completely rewritten, it
still gets *for free* all changes to the parsing of source to AST. For
example:

rosuav at sikorsky:~$ python3.5 -c "import ast;
print(ast.literal_eval('1_234_567'))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.5/ast.py", line 46, in literal_eval
    node_or_string = parse(node_or_string, mode='eval')
  File "/usr/lib/python3.5/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 1
    1_234_567
            ^
SyntaxError: unexpected EOF while parsing
rosuav at sikorsky:~$ python3.6 -c "import ast;
print(ast.literal_eval('1_234_567'))"
1234567

Did literal_eval get changed in 3.6 to add support for this? Nope. It
just hands the string of text to the compiler and says "give me an
AST", and then does its whitelisted walk on that. That's a pretty huge
slab of work that would have to be redone if compile() were removed
from the language. Yes, brand new language features (eg f-strings)
have to be considered and either accepted or rejected, and yes,
literal_eval does have some definitions of "literal" that aren't atoms
(eg complex numbers), necessitating some slightly more complicated
rules - but they're all derived from the AST.

That's why a "no-runtime-compilation-allowed" Python has to either
lose or manually reimplement this. I very much doubt that CPython
upstream is interested in a patch to literal_eval to make it not use
compile - it makes it slower, larger, and more work to maintain,
merely in order to remove a dependency on something that's a core
feature. Ergo TurboPython would need to be the one maintaining this
alternate literal_eval, adding significantly to the initial _and_
ongoing work of the project.

And that's why I said that removing compile() would mean that
ast.literal_eval() would go.

ChrisA