[Python-checkins] r46146 - sandbox/trunk/rjsh-pybench/pybench.py

[M.-A. Lemburg]

Steve Holden wrote:
> M.-A. Lemburg wrote:
>> steve.holden wrote:
>>
>>> Author: steve.holden
>>> Date: Tue May 23 21:21:00 2006
>>> New Revision: 46146
>>>
>>> Modified:
>>>   sandbox/trunk/rjsh-pybench/pybench.py
>>> Log:
>>> Use the appropriate clock for the platform.
>>> Default the number of calibration runs to 0 (can set with -C)
>>
>>
>> Defaulting to 0 is a bad idea, since you basically turn
>> off calibration altogether.
>>
>> Note that calibration is *very* important so
>> that only the operation itself is timed, not the setup,
>> loop and other logic not related to the test run.
>>
> I understand the purpose of the calibration and agree that omitting
> calibration could falsely skew the operation times. I don't feel the
> operation times are a particularly valuable performance parameter anyway
> - nobody is going to say "ah, well I'll use dict lookup because it's
> faster than instance creation".

That's not the point: you want to measure and compare the operation,
not the setup time and loop mechanism. Both can have quite different
performance timings on different machines, since they usually
involve a lot more complex machinery than the simple operation
you are trying to time.

If you add a parameter to change the number of rounds the calibration
is run, that's fine. Switching it off completely will make comparisons
between platforms generate wrong results.

> It may not be appropriate to check back into the trunk a version with
> the default number of calibration runs at zero. However, repeated
> testing has shown that there is sufficient variability in calibration
> times to throw some results into confusion on some platforms. Comparing
> a run with calibration to a run without, for example, does not (on
> Windows, at least) yield the uniform speedup I should have expected.

In that case, you should try to raise the number of rounds
the calibration code is run. The default of 20 rounds may
not be enough to create an average on Windows.

It appears that time.clock() measures wall time instead of
just the process time on Windows, so it's likely that the
calibration run result depends a lot on what else is going on
on the system at the time you run pybench.

> I therefore think we should at least retain the option to ignore the
> operation times and omit calibration as it will give a more reproducible
> measure of absolute execution speed.

On the change of time.clock() to time.time(): the precision
of time.time() doesn't appear to be more accurate than
time.clock() on Windows. Is this documented somewhere ?

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 24 2006)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::