[Numpy-discussion] RFC: is it worth giving a lightning talk at PyCon 2014 on numpy vbench-marking?

[Nathaniel Smith]

On Sun, Nov 24, 2013 at 5:32 PM, Yaroslav Halchenko
<lists at onerussian.com> wrote:
>
> On Tue, 15 Oct 2013, Nathaniel Smith wrote:
>> What do you have to lose?
>
>> > btw -- fresh results are here http://yarikoptic.github.io/numpy-vbench/ .
>
>> > I have tuned benchmarking so it now reflects the best performance across
>> > multiple executions of the whole battery, thus eliminating spurious
>> > variance if estimate is provided from a single point in time.  Eventually I
>> > expect many of those curves to become even "cleaner".
>
>> On another note, what do you think of moving the vbench benchmarks
>> into the main numpy tree? We already require everyone who submits a
>> bug fix to add a test; there are a bunch of speed enhancements coming
>> in these days and it would be nice if we had some way to ask people to
>> submit a benchmark along with each one so that we know that the
>> enhancement stays enhanced...
>
> On this positive note (it is boring to start a new thread, isn't it?) --
> would you be interested in me transfering numpy-vbench over to
> github.com/numpy ?

If you mean just moving the existing git repo under the numpy
organization, like github.com/numpy/numpy-vbench, then I'm not sure
how much difference it would make really. What seems like it'd be
really useful though would be if the code could move into the main
numpy tree, so that people could submit both benchmarks and
optimizations within a single PR.

> as of today, plots on http://yarikoptic.github.io/numpy-vbench  should
> be updating 24x7 (just a loop, thus no time guarantee after you submit
> new changes).
>
> Besides benchmarking new benchmarks (your PRs  would still be very
> welcome,  so far it was just me and Julian T) and revisions, that
> process also goes through a random sample of existing previously
> benchmarked revisions and re-runs the benchmarks thus improving upon the
> ultimate 'min' timing performance.  So you can see already that many
> plots became much 'cleaner', although now there might be a bit of bias
> in estimates for recent revisions since they hadn't accumulated yet as
> many of 'independent runs' as older revisions.

Cool!

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org