[Numpy-discussion] numpy vbench-marking, compiler comparison

[Julian Taylor]

On 25.11.2013 02:32, Yaroslav Halchenko wrote:
> 
> On Tue, 15 Oct 2013, Nathaniel Smith wrote:
>> What do you have to lose?
> 
>>> btw -- fresh results are here http://yarikoptic.github.io/numpy-vbench/ .
> 
>>> I have tuned benchmarking so it now reflects the best performance across
>>> multiple executions of the whole battery, thus eliminating spurious
>>> variance if estimate is provided from a single point in time.  Eventually I
>>> expect many of those curves to become even "cleaner".
> 
>> On another note, what do you think of moving the vbench benchmarks
>> into the main numpy tree? We already require everyone who submits a
>> bug fix to add a test; there are a bunch of speed enhancements coming
>> in these days and it would be nice if we had some way to ask people to
>> submit a benchmark along with each one so that we know that the
>> enhancement stays enhanced...
> 
> On this positive note (it is boring to start a new thread, isn't it?) --
> would you be interested in me transfering numpy-vbench over to
> github.com/numpy ? 
> 
> as of today, plots on http://yarikoptic.github.io/numpy-vbench  should
> be updating 24x7 (just a loop, thus no time guarantee after you submit
> new changes).  
> 
> Besides benchmarking new benchmarks (your PRs  would still be very
> welcome,  so far it was just me and Julian T) and revisions, that
> process also goes through a random sample of existing previously
> benchmarked revisions and re-runs the benchmarks thus improving upon the
> ultimate 'min' timing performance.  So you can see already that many
> plots became much 'cleaner', although now there might be a bit of bias
> in estimates for recent revisions since they hadn't accumulated yet as
> many of 'independent runs' as older revisions.
> 

using the vbench I created a comparison of gcc and clang with different
options.
Cliffnotes:
* gcc -O2 performs 5-10% better than -O3 in most benchmarks, except in a
few select cases where the vectorizer does its magic
* gcc and clang are very close in performance, but the cases where a
compiler wins by a large margin its mostly gcc that wins

I have collected some interesting plots on this notebook:
http://nbviewer.ipython.org/7646615