[Tutor] Astonishing timing result

[Lie Ryan]

On Wed, 2008-06-25 at 12:56 -0400, Kent Johnson wrote:
> On Wed, Jun 25, 2008 at 12:05 PM, Lie Ryan <lie.1296 at gmail.com> wrote:
> 
> >    t_a = min(t_A, t_a)
> >    t_b = min(t_A, t_b)
> >    t_c = min(t_A, t_c)
> >    t_d = min(t_A, t_d)
> 
> What is this for? It should at least be t_B, t_C, t_D.

A common pitfall in benchmarking is averaging the benchmark result. That
is WRONG, FLAT WRONG. Why? Variations of how long a code is executed is
caused by the environment, not the code itself. The correct way to
benchmark is to use the one with the lowest time (i.e. min() function),
since the lowest one is the one that is least interfered by the
environment.

> > ## OUTPUT
> > # 1.02956604958
> > # 1.02956604958
> > # 1.02956604958
> > # 1.02956604958
> 
> It's *very easy* to write bogus timing tests, as this thread
> demonstrates. Some protections:
> - when comparing different implementations of a function, make sure
> each implementation returns the correct result by checking the return
> value. 

Since the purpose of the test is to benchmark the difference of where
the code is located, we should use a very simple function, that doesn't
even do much of anything, thus 'a = 1'. If that simple code is
substituted with anything else, I'm still confident that the result
won't be far off.

> You probably want to make this check outside the actual timing
> test.

Actually the timing is all equal because of the timer's resolution. I
don't have a high-precision timer on hand.

> - when your results don't make sense, suspect your tests.



> Kent
> Kent