[Python-Dev] Python Benchmarks

[M.-A. Lemburg]

Fredrik Lundh wrote:
> M.-A. Lemburg wrote:
> 
>> Seriously, I've been using and running pybench for years
>> and even though tweaks to the interpreter do sometimes
>> result in speedups or slow-downs where you wouldn't expect
>> them (due to the interpreter using the Python objects),
>> they are reproducable and often enough have uncovered
>> that optimizations in one area may well result in slow-downs
>> in other areas.
> 
>  > Often enough the results are related to low-level features
>  > of the architecture you're using to run the code such as
>  > cache size, cache lines, number of registers in the CPU or
>  > on the FPU stack, etc. etc.
> 
> and that observation has never made you stop and think about whether 
> there might be some problem with the benchmarking approach you're using? 

The approach pybench is using is as follows:

* Run a calibration step which does the same as the actual
  test without the operation being tested (ie. call the
  function running the test, setup the for-loop, constant
  variables, etc.)

  The calibration step is run multiple times and is used
  to calculate an average test overhead time.

* Run the actual test which runs the operation multiple
  times.

  The test is then adjusted to make sure that the
  test overhead / test run ratio remains within
  reasonable bounds.

  If needed, the operation code is repeated verbatim in
  the for-loop, to decrease the ratio.

* Repeat the above for each test in the suite

* Repeat the suite N number of rounds

* Calculate the average run time of all test runs in all rounds.

>   after all, if a change to e.g. the try/except code slows things down 
> or speed things up, is it really reasonable to expect that the time it 
> takes to convert Unicode strings to uppercase should suddenly change due 
> to cache effects or a changing number of registers in the CPU?  real 
> hardware doesn't work that way...

Of course, but then changes to try-except logic can interfere
with the performance of setting up method calls. This is what
pybench then uncovers.

The only problem I see in the above approach is the way
calibration is done. The run-time of the calibration code
may be to small w/r to the resolution of the used timers.

Again, please provide the parameters you've used to run the
test case and the output. Things like warp factor, overhead,
etc. could hint to the problem you're seeing.

> is PyBench perhaps using the following approach:
> 
>      T = set of tests
>      for N in range(number of test runs):
>          for t in T:
>              t0 = get_process_time()
>              t()
>              t1 = get_process_time()
>              assign t1 - t0 to test t
>              print assigned time
> 
> where t1 - t0 is very short?

See above (or the code in pybench.py). t1-t0 is usually
around 20-50 seconds:

"""
        The tests must set .rounds to a value high enough to let the
        test run between 20-50 seconds. This is needed because
        clock()-timing only gives rather inaccurate values (on Linux,
        for example, it is accurate to a few hundreths of a
        second). If you don't want to wait that long, use a warp
        factor larger than 1.
"""

> that's not a very good idea, given how get_process_time tends to be 
> implemented on current-era systems (google for "jiffies")...  but it 
> definitely explains the bogus subtest results I'm seeing, and the "magic 
> hardware" behaviour you're seeing.

That's exactly the reason why tests run for a relatively long
time - to minimize these effects. Of course, using wall time
make this approach vulnerable to other effects such as current
load of the system, other processes having a higher priority
interfering with the timed process, etc.

For this reason, I'm currently looking for ways to measure the
process time on Windows.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jun 02 2006)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2006-07-03: EuroPython 2006, CERN, Switzerland              30 days left

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::