Performance in exec environnements

[Steven D'Aprano]

Jean-Baptiste Braun wrote:

> Hi,
> 
> I'm working on auto-generated python code with the exec function. I've
> done some performance benches :

[snip timing results]

> Am I missing something or should I expect that result ? What does using
> exec imply that causes such a difference ?

exec'ing a string will be slower than directly executing code.

Firstly, your initial test:

    python -m timeit "1 + 1"

 is probably biased. Python has a keyhole optimizer which does constant
folding, so that is equivalent to:

    python -m timeit "2"

which does very little work. To get a more realistic timing, try this:

    python -m timeit "n = 1; n + 1"

which will defeat the peephole optimizer.

So you have been comparing:

    2

versus

    exec('1+1')


The first case just fetches a reference to a pre-existing int object, and
then deletes the reference. That's fast.

The second case:

- creates a new string '1+1'
- does a global lookup to find the built-in exec function
- passes the string to the function
- the function then parses that string and compiles it to byte-code
- runs the keyhole optimizer over it
- and finally executes the byte code for "2", same as above.

Only the last step is the same as your earlier test case.


In my experience, the difference between running a piece of code, and
running the same code as a string passed to exec, will be *at least* a
factor of 10 slowdown.

Sometimes you can speed things up by pre-compiling code with compile(), or
with little micro-optimization tricks such as:

def f():
    for x in seq:
        exec(x)  # Looks up the global exec many times.


def f():
    ex = exec  # Look up the global once.
    for x in seq:
        ex(x)  # fast local variable lookup


But that is a micro-optimization which might shave off a few microseconds
per loop, don't expect it to give big savings.

Bigger savings come from avoiding exec. Instead, try to use factory
functions, closures, etc. If you give an example of what you are trying to
generate with exec, we may be able to offer an alternative.



-- 
Steven