[Python-Dev] PEP 564: Add new time functions with nanosecond resolution

Sun Oct 22 05:40:04 EDT 2017

Hi Victor,

I made some small fixes to the PEP.

As far as I'm concerned, the PEP is ok and should be approved :-)

Regards

Antoine.

On Mon, 16 Oct 2017 12:42:30 +0200
Victor Stinner <victor.stinner at gmail.com> wrote:
> Hi,
> 
> While discussions on this PEP are not over on python-ideas, I proposed
> this PEP directly on python-dev since I consider that my PEP already
> summarizes current and past proposed alternatives.
> 
> python-ideas threads:
> 
> * Add time.time_ns(): system clock with nanosecond resolution
> * Why not picoseconds?
> 
> The PEP 564 will be shortly online at:
> https://www.python.org/dev/peps/pep-0564/
> 
> Victor
> 
> 
> PEP: 564
> Title: Add new time functions with nanosecond resolution
> Version: $Revision$
> Last-Modified: $Date$
> Author: Victor Stinner <victor.stinner at gmail.com>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 16-October-2017
> Python-Version: 3.7
> 
> 
> Abstract
> ========
> 
> Add five new functions to the ``time`` module: ``time_ns()``,
> ``perf_counter_ns()``, ``monotonic_ns()``, ``clock_gettime_ns()`` and
> ``clock_settime_ns()``. They are similar to the function without the
> ``_ns`` suffix, but have nanosecond resolution: use a number of
> nanoseconds as a Python int.
> 
> The best ``time.time_ns()`` resolution measured in Python is 3 times
> better then ``time.time()`` resolution on Linux and Windows.
> 
> 
> Rationale
> =========
> 
> Float type limited to 104 days
> ------------------------------
> 
> The clocks resolution of desktop and latop computers is getting closer
> to nanosecond resolution. More and more clocks have a frequency in MHz,
> up to GHz for the CPU TSC clock.
> 
> The Python ``time.time()`` function returns the current time as a
> floatting point number which is usually a 64-bit binary floatting number
> (in the IEEE 754 format).
> 
> The problem is that the float type starts to lose nanoseconds after 104
> days.  Conversion from nanoseconds (``int``) to seconds (``float``) and
> then back to nanoseconds (``int``) to check if conversions lose
> precision::
> 
>     # no precision loss
>     >>> x = 2 ** 52 + 1; int(float(x * 1e-9) * 1e9) - x  
>     0
>     # precision loss! (1 nanosecond)
>     >>> x = 2 ** 53 + 1; int(float(x * 1e-9) * 1e9) - x  
>     -1
>     >>> print(datetime.timedelta(seconds=2 ** 53 / 1e9))  
>     104 days, 5:59:59.254741
> 
> ``time.time()`` returns seconds elapsed since the UNIX epoch: January
> 1st, 1970. This function loses precision since May 1970 (47 years ago)::
> 
>     >>> import datetime
>     >>> unix_epoch = datetime.datetime(1970, 1, 1)
>     >>> print(unix_epoch + datetime.timedelta(seconds=2**53 / 1e9))  
>     1970-04-15 05:59:59.254741
> 
> 
> Previous rejected PEP
> ---------------------
> 
> Five years ago, the PEP 410 proposed a large and complex change in all
> Python functions returning time to support nanosecond resolution using
> the ``decimal.Decimal`` type.
> 
> The PEP was rejected for different reasons:
> 
> * The idea of adding a new optional parameter to change the result type
>   was rejected. It's an uncommon (and bad?) programming practice in
>   Python.
> 
> * It was not clear if hardware clocks really had a resolution of 1
>   nanosecond, especially at the Python level.
> 
> * The ``decimal.Decimal`` type is uncommon in Python and so requires
>   to adapt code to handle it.
> 
> 
> CPython enhancements of the last 5 years
> ----------------------------------------
> 
> Since the PEP 410 was rejected:
> 
> * The ``os.stat_result`` structure got 3 new fields for timestamps as
>   nanoseconds (Python ``int``): ``st_atime_ns``, ``st_ctime_ns``
>   and ``st_mtime_ns``.
> 
> * The PEP 418 was accepted, Python 3.3 got 3 new clocks:
>   ``time.monotonic()``, ``time.perf_counter()`` and
>   ``time.process_time()``.
> 
> * The CPython private "pytime" C API handling time now uses a new
>   ``_PyTime_t`` type: simple 64-bit signed integer (C ``int64_t``).
>   The ``_PyTime_t`` unit is an implementation detail and not part of the
>   API. The unit is currently ``1 nanosecond``.
> 
> Existing Python APIs using nanoseconds as int
> ---------------------------------------------
> 
> The ``os.stat_result`` structure has 3 fields for timestamps as
> nanoseconds (``int``): ``st_atime_ns``, ``st_ctime_ns`` and
> ``st_mtime_ns``.
> 
> The ``ns`` parameter of the ``os.utime()`` function accepts a
> ``(atime_ns: int, mtime_ns: int)`` tuple: nanoseconds.
> 
> 
> Changes
> =======
> 
> New functions
> -------------
> 
> This PEP adds five new functions to the ``time`` module:
> 
> * ``time.clock_gettime_ns(clock_id)``
> * ``time.clock_settime_ns(clock_id, time: int)``
> * ``time.perf_counter_ns()``
> * ``time.monotonic_ns()``
> * ``time.time_ns()``
> 
> These functions are similar to the version without the ``_ns`` suffix,
> but use nanoseconds as Python ``int``.
> 
> For example, ``time.monotonic_ns() == int(time.monotonic() * 1e9)`` if
> ``monotonic()`` value is small enough to not lose precision.
> 
> Unchanged functions
> -------------------
> 
> This PEP only proposed to add new functions getting or setting clocks
> with nanosecond resolution. Clocks are likely to lose precision,
> especially when their reference is the UNIX epoch.
> 
> Python has other functions handling time (get time, timeout, etc.), but
> no nanosecond variant is proposed for them since they are less likely to
> lose precision.
> 
> Example of unchanged functions:
> 
> * ``os`` module: ``sched_rr_get_interval()``, ``times()``, ``wait3()``
>   and ``wait4()``
> 
> * ``resource`` module: ``ru_utime`` and ``ru_stime`` fields of
>   ``getrusage()``
> 
> * ``signal`` module: ``getitimer()``, ``setitimer()``
> 
> * ``time`` module: ``clock_getres()``
> 
> Since the ``time.clock()`` function was deprecated in Python 3.3, no
> ``time.clock_ns()`` is added.
> 
> 
> Alternatives and discussion
> ===========================
> 
> Sub-nanosecond resolution
> -------------------------
> 
> ``time.time_ns()`` API is not "future-proof": if clocks resolutions
> increase, new Python functions may be needed.
> 
> In practive, the resolution of 1 nanosecond is currently enough for all
> structures used by all operating systems functions.
> 
> Hardware clock with a resolution better than 1 nanosecond already
> exists. For example, the frequency of a CPU TSC clock is the CPU base
> frequency: the resolution is around 0.3 ns for a CPU running at 3
> GHz. Users who have access to such hardware and really need
> sub-nanosecond resolution can easyly extend Python for their needs.
> Such rare use case don't justify to design the Python standard library
> to support sub-nanosecond resolution.
> 
> For the CPython implementation, nanosecond resolution is convenient: the
> standard and well supported ``int64_t`` type can be used to store time.
> It supports a time delta between -292 years and 292 years. Using the
> UNIX epoch as reference, this type supports time since year 1677 to year
> 2262::
> 
>     >>> 1970 - 2 ** 63 / (10 ** 9 * 3600 * 24 * 365.25)  
>     1677.728976954687
>     >>> 1970 + 2 ** 63 / (10 ** 9 * 3600 * 24 * 365.25)  
>     2262.271023045313
> 
> Different types
> ---------------
> 
> It was proposed to modify ``time.time()`` to use float type with better
> precision. The PEP 410 proposed to use ``decimal.Decimal``, but it was
> rejected. Apart ``decimal.Decimal``, no portable ``float`` type with
> better precision is currently available in Python. Changing the builtin
> Python ``float`` type is out of the scope of this PEP.
> 
> Other ideas of new types were proposed to support larger or arbitrary
> precision: fractions, structures or 2-tuple using integers,
> fixed-precision floating point number, etc.
> 
> See also the PEP 410 for a previous long discussion on other types.
> 
> Adding a new type requires more effort to support it, than reusing
> ``int``. The standard library, third party code and applications would
> have to be modified to support it.
> 
> The Python ``int`` type is well known, well supported, ease to
> manipulate, and supports all arithmetic operations like:
> ``dt = t2 - t1``.
> 
> Moreover, using nanoseconds as integer is not new in Python, it's
> already used for ``os.stat_result`` and
> ``os.utime(ns=(atime_ns, mtime_ns))``.
> 
> .. note::
>    If the Python ``float`` type becomes larger (ex: decimal128 or
>    float128), the ``time.time()`` precision will increase as well.
> 
> Different API
> -------------
> 
> The ``time.time(ns=False)`` API was proposed to avoid adding new
> functions. It's an uncommon (and bad?) programming practice in Python to
> change the result type depending on a parameter.
> 
> Different options were proposed to allow the user to choose the time
> resolution. If each Python module uses a different resolution, it can
> become difficult to handle different resolutions, instead of just
> seconds (``time.time()`` returning ``float``) and nanoseconds
> (``time.time_ns()`` returning ``int``). Moreover, as written above,
> there is no need for resolution better than 1 nanosecond in practive in
> the Python standard library.
> 
> 
> Annex: Clocks Resolution in Python
> ==================================
> 
> Script ot measure the smallest difference between two ``time.time()`` and
> ``time.time_ns()`` reads ignoring differences of zero::
> 
>     import math
>     import time
> 
>     LOOPS = 10 ** 6
> 
>     print("time.time_ns(): %s" % time.time_ns())
>     print("time.time(): %s" % time.time())
> 
>     min_dt = [abs(time.time_ns() - time.time_ns())
>               for _ in range(LOOPS)]
>     min_dt = min(filter(bool, min_dt))
>     print("min time_ns() delta: %s ns" % min_dt)
> 
>     min_dt = [abs(time.time() - time.time())
>               for _ in range(LOOPS)]
>     min_dt = min(filter(bool, min_dt))
>     print("min time() delta: %s ns" % math.ceil(min_dt * 1e9))
> 
> Results of time(), perf_counter() and monotonic().
> 
> Linux (kernel 4.12 on Fedora 26):
> 
> * time_ns(): **84 ns**
> * time(): **239 ns**
> * perf_counter_ns(): 84 ns
> * perf_counter(): 82 ns
> * monotonic_ns(): 84 ns
> * monotonic(): 81 ns
> 
> Windows 8.1:
> 
> * time_ns(): **318000 ns**
> * time(): **894070 ns**
> * perf_counter_ns(): 100 ns
> * perf_counter(): 100 ns
> * monotonic_ns(): 15000000 ns
> * monotonic(): 15000000 ns
> 
> The difference on ``time.time()`` is significant: **84 ns (2.8x better)
> vs 239 ns on Linux and 318 us (2.8x better) vs 894 us on Windows**. The
> difference (presion loss) will be larger next years since every day adds
> 864,00,000,000,000 nanoseconds to the system clock.
> 
> The difference on ``time.perf_counter()`` and ``time.monotonic clock()``
> is not visible in this quick script since the script runs less than 1
> minute, and the uptime of the computer used to run the script was
> smaller than 1 week. A significant difference should be seen with an
> uptime of 104 days or greater.
> 
> .. note::
>    Internally, Python starts ``monotonic()`` and ``perf_counter()``
>    clocks at zero on some platforms which indirectly reduce the
>    precision loss.
> 
> 
> 
> Copyright
> =========
> 
> This document has been placed in the public domain.