[Python-Dev] [RFC] PEP 418: Add monotonic time, performance counter and process time functions
M.-A. Lemburg
mal at egenix.com
Sun Apr 15 17:36:03 CEST 2012
Victor Stinner wrote:
> Hi,
>
> Here is a simplified version of the first draft of the PEP 418. The
> full version can be read online.
> http://www.python.org/dev/peps/pep-0418/
>
> The implementation of the PEP can be found in this issue:
> http://bugs.python.org/issue14428
>
> I post a simplified version for readability and to focus on changes
> introduced by the PEP. Removed sections: Existing Functions,
> Deprecated Function, Glossary, Hardware clocks, Operating system time
> functions, System Standby, Links.
Looks good.
I'd suggest to also include a tool or API to determine the
real resolution of a time function (as opposed to the advertised
one). See pybench's clockres.py helper as example. You often
find large differences between the advertised resolution and
the available one, e.g. while process timers often advertise
very good resolution, they are in fact often only updated
at very coarse rates.
E.g. compare the results of clockres.py on Linux:
Clock resolution of various timer implementations:
time.clock: 10000.000us
time.time: 0.954us
systimes.processtime: 999.000us
and FreeBSD:
Clock resolution of various timer implementations:
time.clock: 7812.500us
time.time: 1.907us
systimes.processtime: 1.000us
and Mac OS X:
Clock resolution of various timer implementations:
time.clock: 1.000us
time.time: 0.954us
systimes.processtime: 1.000us
Regarding changing pybench:
pybench has to stay backwards incompatible with
earlier releases to make it possible to compare timings.
You can add support for new timers to pybench, but please leave
the existing timers and defaults in place.
> ---
>
> PEP: 418
> Title: Add monotonic time, performance counter and process time functions
> Version: f2bb3f74298a
> Last-Modified: 2012-04-15 17:06:07 +0200 (Sun, 15 Apr 2012)
> Author: Cameron Simpson <cs at zip.com.au>, Jim Jewett
> <jimjjewett at gmail.com>, Victor Stinner <victor.stinner at gmail.com>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 26-March-2012
> Python-Version: 3.3
>
> Abstract
> ========
>
> This PEP proposes to add ``time.get_clock_info(name)``,
> ``time.monotonic()``, ``time.perf_counter()`` and
> ``time.process_time()`` functions to Python 3.3.
>
> Rationale
> =========
>
> If a program uses the system time to schedule events or to implement
> a timeout, it will not run events at the right moment or stop the
> timeout too early or too late when the system time is set manually or
> adjusted automatically by NTP. A monotonic clock should be used
> instead to not be affected by system time updates:
> ``time.monotonic()``.
>
> To measure the performance of a function, ``time.clock()`` can be used
> but it is very different on Windows and on Unix. On Windows,
> ``time.clock()`` includes time elapsed during sleep, whereas it does
> not on Unix. ``time.clock()`` precision is very good on Windows, but
> very bad on Unix. The new ``time.perf_counter()`` function should be
> used instead to always get the most precise performance counter with a
> portable behaviour (ex: include time spend during sleep).
>
> To measure CPU time, Python does not provide directly a portable
> function. ``time.clock()`` can be used on Unix, but it has a bad
> precision. ``resource.getrusage()`` can also be used on Unix, but it
> requires to get fields of a structure and compute the sum of time
> spent in kernel space and user space. The new ``time.process_time()``
> function acts as a portable counter that always measures CPU time
> (doesn't include time elapsed during sleep) and has the best available
> precision.
>
> Each operating system implements clocks and performance counters
> differently, and it is useful to know exactly which function is used
> and some properties of the clock like its resolution and its
> precision. The new ``time.get_clock_info()`` function gives access to
> all available information of each Python time function.
>
> New functions:
>
> * ``time.monotonic()``: timeout and scheduling, not affected by system
> clock updates
> * ``time.perf_counter()``: benchmarking, most precise clock for short
> period
> * ``time.process_time()``: profiling, CPU time of the process
>
> Users of new functions:
>
> * time.monotonic(): concurrent.futures, multiprocessing, queue, subprocess,
> telnet and threading modules to implement timeout
> * time.perf_counter(): trace and timeit modules, pybench program
> * time.process_time(): profile module
> * time.get_clock_info(): pybench program to display information about the
> timer like the precision or the resolution
>
> The ``time.clock()`` function is deprecated because it is not
> portable: it behaves differently depending on the operating system.
> ``time.perf_counter()`` or ``time.process_time()`` should be used
> instead, depending on your requirements. ``time.clock()`` is marked as
> deprecated but is not planned for removal.
>
>
> Python functions
> ================
>
> New Functions
> -------------
>
> time.get_clock_info(name)
> ^^^^^^^^^^^^^^^^^^^^^^^^^
>
> Get information on the specified clock. Supported clock names:
>
> * ``"clock"``: ``time.clock()``
> * ``"monotonic"``: ``time.monotonic()``
> * ``"perf_counter"``: ``time.perf_counter()``
> * ``"process_time"``: ``time.process_time()``
> * ``"time"``: ``time.time()``
>
> Return a dictionary with the following keys:
>
> * Mandatory keys:
>
> * ``"implementation"`` (str): name of the underlying operating system
> function. Examples: ``"QueryPerformanceCounter()"``,
> ``"clock_gettime(CLOCK_REALTIME)"``.
> * ``"resolution"`` (float): resolution in seconds of the clock.
> * ``"is_monotonic"`` (bool): True if the clock cannot go backward.
>
> * Optional keys:
>
> * ``"precision"`` (float): precision in seconds of the clock
> reported by the operating system.
> * ``"is_adjusted"`` (bool): True if the clock is adjusted (e.g. by a
> NTP daemon).
>
>
> time.monotonic()
> ^^^^^^^^^^^^^^^^
>
> Monotonic clock, i.e. cannot go backward. It is not affected by system
> clock updates. The reference point of the returned value is
> undefined, so that only the difference between the results of
> consecutive calls is valid and is a number of seconds.
>
> On Windows versions older than Vista, ``time.monotonic()`` detects
> ``GetTickCount()`` integer overflow (32 bits, roll-over after 49.7
> days): it increases a delta by 2\ :sup:`32` each time than an overflow
> is detected. The delta is stored in the process-local state and so
> the value of ``time.monotonic()`` may be different in two Python
> processes running for more than 49 days. On more recent versions of
> Windows and on other operating systems, ``time.monotonic()`` is
> system-wide.
>
> Availability: Windows, Mac OS X, Unix, Solaris. Not available on
> GNU/Hurd.
>
> Pseudo-code [#pseudo]_::
>
> if os.name == 'nt':
> # GetTickCount64() requires Windows Vista, Server 2008 or later
> if hasattr(time, '_GetTickCount64'):
> def monotonic():
> return _time.GetTickCount64() * 1e-3
> else:
> def monotonic():
> ticks = _time.GetTickCount()
> if ticks < monotonic.last:
> # Integer overflow detected
> monotonic.delta += 2**32
> monotonic.last = ticks
> return (ticks + monotonic.delta) * 1e-3
> monotonic.last = 0
> monotonic.delta = 0
>
> elif os.name == 'mac':
> def monotonic():
> if monotonic.factor is None:
> factor = _time.mach_timebase_info()
> monotonic.factor = timebase[0] / timebase[1]
> return _time.mach_absolute_time() * monotonic.factor
> monotonic.factor = None
>
> elif hasattr(time, "clock_gettime") and hasattr(time, "CLOCK_HIGHRES"):
> def monotonic():
> return time.clock_gettime(time.CLOCK_HIGHRES)
>
> elif hasattr(time, "clock_gettime") and hasattr(time, "CLOCK_MONOTONIC"):
> def monotonic():
> return time.clock_gettime(time.CLOCK_MONOTONIC)
>
>
> On Windows, ``QueryPerformanceCounter()`` is not used even though it
> has a better precision than ``GetTickCount()``. It is not reliable
> and has too many issues.
>
>
> time.perf_counter()
> ^^^^^^^^^^^^^^^^^^^
>
> Performance counter with the highest available precision to measure a
> duration. It does include time elapsed during sleep and is
> system-wide. The reference point of the returned value is undefined,
> so that only the difference between the results of consecutive calls
> is valid and is a number of seconds.
>
> Pseudo-code::
>
> def perf_counter():
> if perf_counter.use_performance_counter:
> if perf_counter.performance_frequency is None:
> try:
> perf_counter.performance_frequency =
> _time.QueryPerformanceFrequency()
> except OSError:
> # QueryPerformanceFrequency() fails if the installed
> # hardware does not support a high-resolution performance
> # counter
> perf_counter.use_performance_counter = False
> else:
> return _time.QueryPerformanceCounter() /
> perf_counter.performance_frequency
> else:
> return _time.QueryPerformanceCounter() /
> perf_counter.performance_frequency
> if perf_counter.use_monotonic:
> # The monotonic clock is preferred over the system time
> try:
> return time.monotonic()
> except OSError:
> perf_counter.use_monotonic = False
> return time.time()
> perf_counter.use_performance_counter = (os.name == 'nt')
> if perf_counter.use_performance_counter:
> perf_counter.performance_frequency = None
> perf_counter.use_monotonic = hasattr(time, 'monotonic')
>
>
> time.process_time()
> ^^^^^^^^^^^^^^^^^^^
>
> Sum of the system and user CPU time of the current process. It does
> not include time elapsed during sleep. It is process-wide by
> definition. The reference point of the returned value is undefined,
> so that only the difference between the results of consecutive calls
> is valid.
>
> It is available on all platforms.
>
> Pseudo-code [#pseudo]_::
>
> if os.name == 'nt':
> def process_time():
> handle = win32process.GetCurrentProcess()
> process_times = win32process.GetProcessTimes(handle)
> return (process_times['UserTime'] +
> process_times['KernelTime']) * 1e-7
> else:
> import os
> try:
> import resource
> except ImportError:
> has_resource = False
> else:
> has_resource = True
>
> def process_time():
> if process_time.use_process_cputime:
> try:
> return time.clock_gettime(time.CLOCK_PROCESS_CPUTIME_ID)
> except OSError:
> process_time.use_process_cputime = False
> if process_time.use_getrusage:
> try:
> usage = resource.getrusage(resource.RUSAGE_SELF)
> return usage[0] + usage[1]
> except OSError:
> process_time.use_getrusage = False
> if process_time.use_times:
> try:
> times = os.times()
> return times[0] + times[1]
> except OSError:
> process_time.use_getrusage = False
> return _time.clock()
> process_time.use_process_cputime = (
> hasattr(time, 'clock_gettime')
> and hasattr(time, 'CLOCK_PROCESS_CPUTIME_ID'))
> process_time.use_getrusage = has_resource
> # On OS/2, only the 5th field of os.times() is set, others are zeros
> process_time.use_times = (hasattr(os, 'times') and os.name != 'os2')
>
>
> Alternatives: API design
> ========================
>
> Other names for time.monotonic()
> --------------------------------
>
> * time.counter()
> * time.metronomic()
> * time.seconds()
> * time.steady(): "steady" is ambiguous: it means different things to
> different people. For example, on Linux, CLOCK_MONOTONIC is
> adjusted. If we uses the real time as the reference clock, we may
> say that CLOCK_MONOTONIC is steady. But CLOCK_MONOTONIC gets
> suspended on system suspend, whereas real time includes any time
> spent in suspend.
> * time.timeout_clock()
> * time.wallclock(): time.monotonic() is not the system time aka the
> "wall clock", but a monotonic clock with an unspecified starting
> point.
>
> The name "time.try_monotonic()" was also proposed for an older
> proposition of time.monotonic() which was falling back to the system
> time when no monotonic clock was available.
>
> Other names for time.perf_counter()
> -----------------------------------
>
> * time.hires()
> * time.highres()
> * time.timer()
>
> Only expose operating system clocks
> -----------------------------------
>
> To not have to define high-level clocks, which is a difficult task, a
> simpler approach is to only expose operating system clocks.
> time.clock_gettime() and related clock identifiers were already added
> to Python 3.3 for example.
>
>
> time.monotonic(): Fallback to system time
> -----------------------------------------
>
> If no monotonic clock is available, time.monotonic() falls back to the
> system time.
>
> Issues:
>
> * It is hard to define correctly such function in the documentation:
> is it monotonic? Is it steady? Is it adjusted?
> * Some user want to decide what to do when no monotonic clock is
> available: use another clock, display an error, or do something
> else?
>
> Different APIs were proposed to define such function.
>
> One function with a flag: time.monotonic(fallback=True)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
> * time.monotonic(fallback=True) falls back to the system time if no
> monotonic clock is available or if the monotonic clock failed.
> * time.monotonic(fallback=False) raises OSError if monotonic clock
> fails and NotImplementedError if the system does not provide a
> monotonic clock
>
> A keyword argument that gets passed as a constant in the caller is
> usually poor API.
>
> Raising NotImplementedError for a function is something uncommon in
> Python and should be avoided.
>
>
> One time.monotonic() function, no flag
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
> time.monotonic() returns (time: float, is_monotonic: bool).
>
> An alternative is to use a function attribute:
> time.monotonic.is_monotonic. The attribute value would be None before
> the first call to time.monotonic().
>
>
> Choosing the clock from a list of constraints
> ---------------------------------------------
>
> The PEP as proposed offers a few new clocks, but their guarentees
> are deliberately loose in order to offer useful clocks on different
> platforms. This inherently embeds policy in the calls, and the
> caller must thus choose a policy.
>
> The "choose a clock" approach suggests an additional API to let
> callers implement their own policy if necessary
> by making most platform clocks available and letting the caller pick
> amongst them.
> The PEP's suggested clocks are still expected to be available for the common
> simple use cases.
>
> To do this two facilities are needed:
> an enumeration of clocks, and metadata on the clocks to enable the user to
> evaluate their suitability.
>
> The primary interface is a function make simple choices easy:
> the caller can use ``time.get_clock(*flags)`` with some combination of flags.
> This include at least:
>
> * time.MONOTONIC: clock cannot go backward
> * time.STEADY: clock rate is steady
> * time.ADJUSTED: clock may be adjusted, for example by NTP
> * time.HIGHRES: clock with the highest precision
>
> It returns a clock object with a .now() method returning the current time.
> The clock object is annotated with metadata describing the clock feature set;
> its .flags field will contain at least all the requested flags.
>
> time.get_clock() returns None if no matching clock is found and so calls can
> be chained using the or operator. Example of a simple policy decision::
>
> T = get_clock(MONOTONIC) or get_clock(STEADY) or get_clock()
> t = T.now()
>
> The available clocks always at least include a wrapper for ``time.time()``,
> so a final call with no flags can always be used to obtain a working clock.
>
> Example of flags of system clocks:
>
> * QueryPerformanceCounter: MONOTONIC | HIGHRES
> * GetTickCount: MONOTONIC | STEADY
> * CLOCK_MONOTONIC: MONOTONIC | STEADY (or only MONOTONIC on Linux)
> * CLOCK_MONOTONIC_RAW: MONOTONIC | STEADY
> * gettimeofday(): (no flag)
>
> The clock objects contain other metadata including the clock flags
> with additional feature flags above those listed above, the name
> of the underlying OS facility, and clock precisions.
>
> ``time.get_clock()`` still chooses a single clock; an enumeration
> facility is also required.
> The most obvious method is to offer ``time.get_clocks()`` with the
> same signature as ``time.get_clock()``, but returning a sequence
> of all clocks matching the requested flags.
> Requesting no flags would thus enumerate all available clocks,
> allowing the caller to make an arbitrary choice amongst them based
> on their metadata.
>
> Example partial implementation:
> `clockutils.py <http://hg.python.org/peps/file/tip/pep-0418/clockutils.py>`_.
>
> Working around operating system bugs?
> -------------------------------------
>
> Should Python ensure manually that a monotonic clock is truly
> monotonic by computing the maximum with the clock value and the
> previous value?
>
> Since it's relatively straightforward to cache the last value returned
> using a static variable, it might be interesting to use this to make
> sure that the values returned are indeed monotonic.
>
> * Virtual machines provide less reliable clocks.
> * QueryPerformanceCounter() has known bugs (only one is not fixed yet)
>
> Python may only work around a specific known operating system bug:
> `KB274323`_ contains a code example to workaround the bug (use
> GetTickCount() to detect QueryPerformanceCounter() leap).
>
> Issues of a hacked monotonic function:
>
> * if the clock is accidentally set forward by an hour and then back
> again, you wouldn't have a useful clock for an hour
> * the cache is not shared between processes so different processes
> wouldn't see the same clock value
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/mal%40egenix.com
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, Apr 15 2012)
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________
2012-04-28: PythonCamp 2012, Cologne, Germany 13 days to go
::: Try our new mxODBC.Connect Python Database Interface for free ! ::::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
More information about the Python-Dev
mailing list