[Speed] Linux tip: use isolcpus to have (more) reliable benchmark

[Maciej Fijalkowski]

Hi.

Disabling ASLR means you get more repeatable benchmarks, of course,
but also means that on another identical machine (or a bit different
circumstances), you can get different results, hence you moved
statistical error to a more systematic one. I don't think that's a win

On Fri, Feb 12, 2016 at 8:42 AM, Patrascu, Alecsandru
<alecsandru.patrascu at intel.com> wrote:
> Hi,
>
> Some of the things we do here at Intel, in our Languages Performance Lab [1,2], is to disable ASLR as you get more reliable results. This can be achieved on Linux by running echo 0 > /proc/sys/kernel/randomize_va_space. Also, setting the CPU frequency at a fixed frequency, disabling Turbo Boost and Hyper Threading, also helps for benchmark stability.
>
> From my experience, the isolcpus feature is useful when you have a lot of cores on your machine because the kernel will have other cores on which it can schedule its work; furthermore, it is a best effort situation and it is not an absolute guarantee that the kernel will not use the cores specified if you have a lot of processes running (for example, if you benchmark on a machine with 2 physical cores and you isolate one of the cores, there is a big chance that the kernel will schedule processes on this core also, even it is for a small amount of time). Nevertheless, for machines with more physical cores, it can be good to have dedicated core(s) on which we do benchmarking.
>
> [1] http://languagesperformance.intel.com/
> [2] https://lists.01.org/pipermail/langperf/
>
> Thank you,
> Alecsandru
>
>> -----Original Message-----
>> From: Speed [mailto:speed-
>> bounces+alecsandru.patrascu=intel.com at python.org] On Behalf Of Victor
>> Stinner
>> Sent: Friday, February 12, 2016 12:54 AM
>> To: speed at python.org
>> Subject: [Speed] Linux tip: use isolcpus to have (more) reliable benchmark
>>
>> Hi,
>>
>> I'm sharing with you my notes (tricks) to get more reliable benchmarks on
>> Linux if your CPU have multiple cores:
>>
>> https://haypo-notes.readthedocs.org/microbenchmark.html#reliable-micro-
>> benchmarks
>>
>> FYI perf.py recently got a new --affinity= optional parameter. I plan to
>> send a patch to automatically use /sys/devices/system/cpu/isolated if it's
>> not empty.
>>
>> What are your "tricks" to get reliable benchmarks?
>>
>> Victor
>> _______________________________________________
>> Speed mailing list
>> Speed at python.org
>> https://mail.python.org/mailman/listinfo/speed
> _______________________________________________
> Speed mailing list
> Speed at python.org
> https://mail.python.org/mailman/listinfo/speed