[Numpy-discussion] Adding a 2D with a 1D array...

[Dag Sverre Seljebotn]

Francesc Alted wrote:
> A Thursday 10 September 2009 13:45:10 Dag Sverre Seljebotn escrigué:
>  > Do you see any issues with this approach: Add a flag timeit to provide
> 
>  > two modes:
> 
>  >
> 
>  > a) Do an initial run which is always not included in timings (in fact,
> 
>  > as it gets "min" and not "mean", I think this is the current behaviour)
> 
> Yup, you are right, it is 'min'. In fact, this is why timeit normally 
> 'forgets' about data transmission times (with a 'mean' the effect is 
> very similar anyways).
> 
>  > b) Do something else between every run which should clear out the cache
> 
>  > (like, just do another big dummy calculation).
> 
> Yeah. In fact, you can simulate this behaviour by running two instances 
> of timeit: one with your code + big dummy calculation, and the other 
> with just the big dummy calculation. Subtract both numbers and you will 
> have a better guess for non-cached calculations.
> 
>  >
> 
>  > (Also a guard in timeit against CPU frequency scaling errors would be
> 
>  > great :-) Like simply outputting a warning if frequency scaling is
> 
>  > detected).
> 
> Sorry, I don't get this one.


I had some trouble getting reliable benchmarks on my own computer until 
I realised that the power-saving capabilities of my CPU down-throttled 
the clock speed when it was not in use. Thus if I did two calls to 
timeit right after one another, the second would always report lower 
runtime, because the first one started at a lower clock speed.

Changing a BIOS setting solved this, but it might be a gotcha which e.g. 
timeit and IPython could report (they could just inspect the CPU 
information and emit a warning -- or, do something to throttle up the 
CPU to full speed first).

-- 
Dag Sverre