Notice: While Javascript is not essential for this website, your interaction with the content will be limited. Please turn Javascript on for the full experience.

PEP 563 -- Postponed Evaluation of Annotations

PEP:563
Title:Postponed Evaluation of Annotations
Author:Ɓukasz Langa <lukasz at langa.pl>
Discussions-To:Python-Dev <python-dev at python.org>
Status:Draft
Type:Standards Track
Created:8-Sep-2017
Python-Version:3.7
Post-History:
Resolution:

Abstract

PEP 3107 introduced syntax for function annotations, but the semantics were deliberately left undefined. PEP 484 introduced a standard meaning to annotations: type hints. PEP 526 defined variable annotations, explicitly tying them with the type hinting use case.

This PEP proposes changing function annotations and variable annotations so that they are no longer evaluated at function definition time. Instead, they are preserved in __annotations__ in string form.

This change is going to be introduced gradually, starting with a new __future__ import in Python 3.7.

Rationale and Goals

PEP 3107 added support for arbitrary annotations on parts of a function definition. Just like default values, annotations are evaluated at function definition time. This creates a number of issues for the type hinting use case:

  • forward references: when a type hint contains names that have not been defined yet, that definition needs to be expressed as a string literal;
  • type hints are executed at module import time, which is not computationally free.

Postponing the evaluation of annotations solves both problems.

Non-goals

Just like in PEP 484 and PEP 526, it should be emphasized that Python will remain a dynamically typed language, and the authors have no desire to ever make type hints mandatory, even by convention.

Annotations are still available for arbitrary use besides type checking. Using @typing.no_type_hints in this case is recommended to disambiguate the use case.

Implementation

In a future version of Python, function and variable annotations will no longer be evaluated at definition time. Instead, a string form will be preserved in the respective __annotations__ dictionary. Static type checkers will see no difference in behavior, whereas tools using annotations at runtime will have to perform postponed evaluation.

If an annotation was already a string, this string is preserved verbatim. In other cases, the string form is obtained from the AST during the compilation step, which means that the string form preserved might not preserve the exact formatting of the source.

Annotations need to be syntactically valid Python expressions, also when passed as literal strings (i.e. compile(literal, '', 'eval')). Annotations can only use names present in the module scope as postponed evaluation using local names is not reliable.

Note that as per PEP 526, local variable annotations are not evaluated at all since they are not accessible outside of the function's closure.

Enabling the future behavior in Python 3.7

The functionality described above can be enabled starting from Python 3.7 using the following special import:

from __future__ import annotations

Resolving Type Hints at Runtime

To resolve an annotation at runtime from its string form to the result of the enclosed expression, user code needs to evaluate the string.

For code that uses type hints, the typing.get_type_hints() function correctly evaluates expressions back from its string form. Note that all valid code currently using __annotations__ should already be doing that since a type annotation can be expressed as a string literal.

For code which uses annotations for other purposes, a regular eval(ann, globals, locals) call is enough to resolve the annotation. The trick here is to get the correct value for globals. Fortunately, in the case of functions, they hold a reference to globals in an attribute called __globals__. To get the correct module-level context to resolve class variables, use:

cls_globals = sys.modules[SomeClass.__module__].__dict__

Runtime annotation resolution and class decorators

Metaclasses and class decorators that need to resolve annotations for the current class will fail for annotations that use the name of the current class. Example:

def class_decorator(cls):
    annotations = get_type_hints(cls)  # raises NameError on 'C'
    print(f'Annotations for {cls}: {annotations}')
    return cls

@class_decorator
class C:
    singleton: 'C' = None

This was already true before this PEP. The class decorator acts on the class before it's assigned a name in the current definition scope.

The situation is made somewhat stricter when class-level variables are considered. Previously, when the string form wasn't used in annotations, a class decorator would be able to cover situations like:

@class_decorator
class Restaurant:
    class MenuOption(Enum):
        SPAM = 1
        EGGS = 2

    default_menu: List[MenuOption] = []

This is no longer possible.

Runtime annotation resolution and TYPE_CHECKING

Sometimes there's code that must be seen by a type checker but should not be executed. For such situations the typing module defines a constant, TYPE_CHECKING, that is considered True during type checking but False at runtime. Example:

import typing

if typing.TYPE_CHECKING:
    import expensive_mod

def a_func(arg: expensive_mod.SomeClass) -> None:
    a_var: expensive_mod.SomeClass = arg
    ...

This approach is also useful when handling import cycles.

Trying to resolve annotations of a_func at runtime using typing.get_type_hints() will fail since the name expensive_mod is not defined (TYPE_CHECKING variable being False at runtime). This was already true before this PEP.

Backwards Compatibility

This is a backwards incompatible change. Applications depending on arbitrary objects to be directly present in annotations will break if they are not using typing.get_type_hints() or eval().

Annotations that depend on locals at the time of the function/class definition are now invalid. Example:

def generate_class():
    some_local = datetime.datetime.now()
    class C:
        field: some_local = 1  # NOTE: INVALID ANNOTATION
        def method(self, arg: some_local.day) -> None:  # NOTE: INVALID ANNOTATION
            ...

Annotations using nested classes and their respective state are still valid, provided they use the fully qualified name. Example:

class C:
    field = 'c_field'
    def method(self, arg: C.field) -> None:  # this is OK
        ...

    class D:
        field2 = 'd_field'
        def method(self, arg: C.field -> C.D.field2:  # this is OK
            ...

In the presence of an annotation that cannot be resolved using the current module's globals, a NameError is raised at compile time.

Deprecation policy

In Python 3.7, a __future__ import is required to use the described functionality and a PendingDeprecationWarning is raised by the compiler in the presence of type annotations in modules without the __future__ import. In Python 3.8 the warning becomes a DeprecationWarning. In the next version this will become the default behavior.

Rejected Ideas

Keep the ability to use local state when defining annotations

With postponed evaluation, this is impossible for function locals. For classes, it would be possible to keep the ability to define annotations using the local scope. However, when using eval() to perform the postponed evaluation, we need to provide the correct globals and locals to the eval() call. In the face of nested classes, the routine to get the effective "globals" at definition time would have to look something like this:

def get_class_globals(cls):
    result = {}
    result.update(sys.modules[cls.__module__].__dict__)
    for child in cls.__qualname__.split('.'):
        result.update(result[child].__dict__)
    return result

This is brittle and doesn't even cover slots. Requiring the use of module-level names simplifies runtime evaluation and provides the "one obvious way" to read annotations. It's the equivalent of absolute imports.

Acknowledgements

This document could not be completed without valuable input, encouragement and advice from Guido van Rossum, Jukka Lehtosalo, and Ivan Levkivskyi.

Source: https://github.com/python/peps/blob/master/pep-0563.txt