[Speed] Latest enhancements of perf 0.8.1 and performance 0.3.1

[Armin Rigo]

Hi Victor,

On 2 November 2016 at 11:50, Victor Stinner <victor.stinner at gmail.com> wrote:
> 2016-11-02 11:04 GMT+01:00 Armin Rigo <armin.rigo at gmail.com>:
>> On 19 October 2016 at 18:55, Victor Stinner <victor.stinner at gmail.com> wrote:
>>> 3) new --duplication option to perf timeit
>>
>> This is never a good idea on top of PyPy, so I wouldn't mind if using
>> this option on top of PyPy threw an error.
>
> Can you please elaborate?

Yes, exactly :-)  Consider a benchmark written like that:

    for i in range(lots):
         z = a + b
         z = a + b
         z = a + b
         z = a + b
         z = a + b

What you are really measuring by running PyPy on this is completely
different from what you *think* you are measuring---in this case,
mostly everything is optimized away.  If you try to make it actually
do something so that it's not optimized away, then the problem of
duplicating lines becomes of making the tracing JIT compiler not happy
at all.  If you duplicate the lines too many times, the loop body
becomes too long for the JIT compiler to swallow---never duplicates
1000 times, that's always too much!  But even if you duplicate only 10
times, then the more subtle problem is: assume that each line can
follow *two* control flow paths (even internally, e.g. because of some
condition done in RPython).  (It is likely the case, if you try to do
something non-trivially-optimizable-away.)  Then if you duplicate the
line 10 times, there are suddenly 2**10 control flow paths.  That
means the JIT will never be able to warm up completely.  Suddenly you
are measuring the JIT compiler's performance and not at all your
code's.

The --duplication option on PyPy is thus either useless or limited to
use cases where you definitely know there is only one code path ever
followed, and don't duplicate too much, and know for sure that
multiple repetitions of the same line won't cause cross-line
optimizations.  That's not possible to explain without going very
technical.


A bientôt,

Armin.