[Python-checkins] r46146 - sandbox/trunk/rjsh-pybench/pybench.py

[Steve Holden]

M.-A. Lemburg wrote:
> Steve Holden wrote:
[...]
>>The sandbox copy already uses the same code timeit.py uses to determine 
>>the clock to use. 
> 
> 
> Sorry, but that approach is just plain wrong:
> 
> if sys.platform == "win32":
>     # On Windows, the best timer is time.clock()
>     default_timer = time.clock
> else:
>     # On most other platforms the best timer is time.time()
>     default_timer = time.time
> 
It's a good job you pointed that out, since that would appear to mean 
that timeit.py is also wrong, since it uses exactly the same code.

> You never want to use a wall clock timer to measure
> the speed of code execution. There are simply too many
> other things going on in a multi-process system to make
> this a reliable approach.
> 
It's not a matter of what I *want* to use, it's a matter of what I *can* 
use. Easily, portably, and with reasonable repeatibility.

> On Unix time.clock() will give you process time which
> is far more reliable as it provides feedback on the
> time the process actually received from the kernel
> for execution.
> 
> Try it:
> 
> print time.clock(); time.sleep(10); print time.clock()
> 
> On Unix, you'll get the same results for both prints.
> On Windows, you get wall clock results, ie. a 10 second
> difference.
> 
So timeit.py isn't just wrong for one platform, it's doing exactly the 
reverse of what it should, and it's just my bad luck that you didn't 
spot *that* checkin?
> 
>>Uncle Timmy points out that the GetProcessTimes() 
>>isn't ubiquitous, so I don't think there's much point in trying to make 
>>use of it.
> 
> 
> According to MSDN it's only available on the WinNT branch
> of the Windows kernel. Since we're trying to optimize for
> a current system, it should be available on all systems
> where you'd normally be running pybench to check whether
> an optimization makes sense or not.
> 
So I can also ignore the comments about retaining 1.5.2 compatibility?
> 
>>The one change I still do want to make is to allow the use of some 
>>bogomips-like feature to provide scaled testing to ensure that the 
>>individual test times can more easily be made large enough.
> 
> 
> Isn't that what the warp factor already implements ?
> 
Yes, but not automatically.
> 
>>At that point I'll probably be looking to check it back into the trunk 
>>as version 1.4 - unless you have any further suggestions? Indications 
>>are that the timings do seem to be more reliable.
> 
> 
> I disagree on the clock approach you implemented and the
> change to use a 0 default for calibration runs, so those
> changes should not go into the trunk.
> 
I wasn't proposing to check it back in with 0 calibration.

There's obviously plenty of mileage in the discussion about how to do 
the timings. I'm just not sure we will come to any reliable conclusion.

regards
  Steve
-- 
Steve Holden       +44 150 684 7255  +1 800 494 3119
Holden Web LLC/Ltd          http://www.holdenweb.com
Love me, love my blog  http://holdenweb.blogspot.com
Recent Ramblings     http://del.icio.us/steve.holden