[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

[Nick Coghlan]

Nick Coghlan added the comment:

Hugo, Serhiy, and Victor: I think you're all agreeing with each other, but to make sure I'm understanding the point correctly:

1. ast.literal_eval() is currently safe from malicious code like "100000 ** 100000" or "1073741824 * 'a'" because it only traverses addition and subtraction nodes, so any such operations will just throw ValueError (As a point of interest: unary plus and minus are required to support positive and negative numeric literals, while binary addition and subtraction are required to support complex number literals. So the status quo isn't precisely the result of a conscious security decision, it's just a minimalist implementation of exactly what's necessary to support all of the builtin types, which also provides some highly desirable security properties when evaluating untrusted code)


2. an eager constant folding optimisation in the AST tier would happen *before* literal_eval filtered out the multiplication and exponentiation nodes, and hence would make literal_eval vulnerable to remote DOS attacks in cases where it is expected to be safe

However, that's not exactly how this patch works: if you pass "PyCF_ONLY_AST" as ast.parse does, it *doesn't* run the constant-folding step. Instead, the constant folding is run as an AST-to-AST transform during the AST-to-bytecode compilation step, *not* the initial source-to-AST step. (see http://bugs.python.org/issue11549#msg132361 )

This has a few nice properties:

- ast.literal_eval() remains safe
- source -> AST -> source transformation pipelines continue to preserve the original code structure
- externally generated AST structures still benefit from the AST optimisation pass
- we don't need a new flag to turn this optimisation pass off when generating the AST for a given piece of source code

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11549>
_______________________________________