[Python-Dev] PEP 561 implemented and minor clarification

[Ethan Smith]

Hello,

I've updated PEP 561 to clarify that any installed stub package should
supersede an installed inline package. In other words if there is:

/global/site-packages/pkg/
/user/site-packages/pkg-stubs/

Even if pkg in the global site packages is found first and marks that it
supports types, the stub package should supersede it.

I also point to mypy's docs on its implementation of the PEP (which can be
read about here:
https://mypy.readthedocs.io/en/latest/installed_packages.html). The
implementation has been merged into master and will be available in the
0.590 release.

As always, the PEP text is replicated below.

Cheers!
Ethan


==============

PEP: 561
Title: Distributing and Packaging Type Information
Author: Ethan Smith <ethan at ethanhs.me>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 09-Sep-2017
Python-Version: 3.7
Post-History: 10-Sep-2017, 12-Sep-2017, 06-Oct-2017, 26-Oct-2017


Abstract
========

PEP 484 introduced type hinting to Python, with goals of making typing
gradual and easy to adopt. Currently, typing information must be distributed
manually. This PEP provides a standardized means to leverage existing tooling
to package and distribute type information with minimal work and an ordering
for type checkers to resolve modules and collect this information for type
checking.


Rationale
=========

Currently, package authors wish to distribute code that has inline type
information. Additionally, maintainers would like to distribute stub files
to keep Python 2 compatibility while using newer annotation syntax. However,
there is no standard method to distribute packages with type information.
Also, if one wished to ship stub files privately the only method available
would be via setting ``MYPYPATH`` or the equivalent to manually point to
stubs. If the package can be released publicly, it can be added to
typeshed [1]_. However, this does not scale and becomes a burden on the
maintainers of typeshed. In addition, it ties bug fixes in stubs to releases
of the tool using typeshed.

PEP 484 has a brief section on distributing typing information. In this
section [2]_ the PEP recommends using ``shared/typehints/pythonX.Y/`` for
shipping stub files. However, manually adding a path to stub files for each
third party library does not scale. The simplest approach people have taken
is to add ``site-packages`` to their ``MYPYPATH``, but this causes type
checkers to fail on packages that are highly dynamic (e.g. sqlalchemy
and Django).


Definition of Terms
===================

The definition of "MAY", "MUST", and "SHOULD", and "SHOULD NOT" are
to be interpreted as described in RFC 2119.

"inline" - the types are part of the runtime code using PEP 526 and 3107
syntax.

"stubs" - files containing only type information, empty of runtime code.

"Distributions" are the packaged files which are used to publish and distribute
a release. [3]_

"Module" a file containing Python runtime code or stubbed type information.

"Package" a directory or directories that namespace Python modules.


Specification
=============

There are several motivations and methods of supporting typing in a package.
This PEP recognizes three (3) types of packages that users of typing wish to
create:

1. The package maintainer would like to add type information inline.

2. The package maintainer would like to add type information via stubs.

3. A third party or package maintainer would like to share stub files for
   a package, but the maintainer does not want to include them in the source
   of the package.

This PEP aims to support these scenarios and make them simple to add to
packaging and deployment.

The two major parts of this specification are the packaging specifications
and the resolution order for resolving module type information. The type
checking spec is meant to replace the ``shared/typehints/pythonX.Y/`` spec
of PEP 484 [2]_.

New third party stub libraries SHOULD distribute stubs via the third party
packaging methods proposed in this PEP in place of being added to typeshed.
Typeshed will remain in use, but if maintainers are found, third party stubs
in typeshed MAY be split into their own package.


Packaging Type Information
--------------------------

In order to make packaging and distributing type information as simple and
easy as possible, packaging and distribution is done through existing
frameworks.

Package maintainers who wish to support type checking of their code MUST add
a marker file named ``py.typed`` to their package supporting typing.
This marker applies
recursively: if a top-level package includes it, all its sub-packages
MUST support
type checking as well. To have this file installed with the package,
maintainers can use existing packaging options such as ``package_data`` in
distutils, shown below.

Distutils option example::

    setup(
        ...,
        package_data = {
            'foopkg': ['py.typed'],
        },
        ...,
        )

For namespace packages, the ``py.typed`` file should be in the submodules of
the namespace, to avoid conflicts and for clarity.

This PEP does not support distributing typing information as part of
module-only distributions. The code should be refactored into a package-based
distribution and indicate that the package supports typing as described
above.

Stub Only Packages
''''''''''''''''''

For package maintainers wishing to ship stub files containing all of their
type information, it is preferred that the ``*.pyi`` stubs are alongside the
corresponding ``*.py`` files. However, the stubs can also be put in a separate
package and distributed separately. Third parties can also find this method
useful if they wish to distribute stub files. The name of the stub package
MUST follow the scheme ``foopkg-stubs`` for type stubs for the package named
``foopkg``. Note that for stub only packages adding a py.typed marker is not
needed since the name ``*-stubs`` is enough to indicate it is a source of typing
information.

Third parties seeking to distribute stub files are encouraged to contact the
maintainer of the package about distribution alongside the package. If the
maintainer does not wish to maintain or package stub files or type information
inline, then a third party stub only package can be created.

In addition, stub-only distributions SHOULD indicate which version(s)
of the runtime package are supported by indicating the runtime distribution's
version(s) through normal dependency data. For example, the
stub package ``flyingcircus-stubs`` can indicate the versions of the
runtime ``flyingcircus`` distribution it supports through ``install_requires``
in distutils-based tools, or the equivalent in other packaging tools. Note that
in pip 9.0, if you update ``flyingcircus-stubs``, it will update
``flyingcircus``. In pip 9.0, you can use the
``--upgrade-strategy=only-if-needed`` flag. In pip 10.0 this is the default
behavior.


Type Checker Module Resolution Order
------------------------------------

The following is the order in which type checkers supporting this PEP SHOULD
resolve modules containing type information:

1. User code - the files the type checker is running on.

2. Stubs or Python source manually put in the beginning of the path. Type
   checkers SHOULD provide this to allow the user complete control of which
   stubs to use, and patch broken stubs/inline types from packages.

3. Stub packages - these packages SHOULD supersede any installed inline
   package. They can be found at ``foopkg-stubs`` for package ``foopkg``.

4. Inline packages - if there is nothing overriding the installed
   package, and it opts into type checking, inline types SHOULD be used.

5. Typeshed (if used) - Provides the stdlib types and several third party
   libraries.

Type checkers that check a different Python version than the version they run
on MUST find the type information in the ``site-packages``/``dist-packages``
of that Python version. This can be queried e.g.
``pythonX.Y -c 'import site; print(site.getsitepackages())'``. It is
also recommended
that the type checker allow for the user to point to a particular Python
binary, in case it is not in the path.

The normal resolution order of checking ``*.pyi`` before ``*.py``
will be maintained.


Implementation
==============

The proposed scheme of indicating support for typing is completely backwards
compatible, and requires no modification to package tooling. A sample package
with inline types is available [typed_pkg]_, as well as a sample package
checker [pkg_checker]_ which reads the metadata of installed packages and
reports on their status as either not typed, inline typed, or a stub package.

The mypy type checker has an implementation of PEP 561 searching which can be
read about in the mypy docs [4]_.


Acknowledgements
================

This PEP would not have been possible without the ideas, feedback, and support
of Ivan Levkivskyi, Jelle Zijlstra, Nick Coghlan, Daniel F Moisset, Nathaniel
Smith, and Guido van Rossum.


Version History
===============

* 2018-04-09

    * Add reference to mypy implementation
    * Clarify stub package priority.

* 2018-02-02

    * Change stub only package suffix to be -stubs not _stubs.
    * Note that py.typed is not needed for stub only packages.
    * Add note about pip and upgrading stub packages.

* 2017-11-12

    * Rewritten to use existing tooling only
    * No need to indicate kind of type information in metadata
    * Name of marker file changed from ``.typeinfo`` to ``py.typed``

* 2017-11-10

    * Specification re-written to use package metadata instead of distribution
      metadata.
    * Removed stub only packages and merged into third party packages spec.
    * Removed suggestion for typecheckers to consider checking runtime versions
    * Implementations updated to reflect PEP changes.

* 2017-10-26

    * Added implementation references.
    * Added acknowledgements and version history.

* 2017-10-06

    * Rewritten to use .distinfo/METADATA over a distutils specific command.
    * Clarify versioning of third party stub packages.

* 2017-09-11

    * Added information about current solutions and typeshed.
    * Clarify rationale.


References
==========
.. [1] Typeshed (https://github.com/python/typeshed)

.. [2] PEP 484, Storing and Distributing Stub Files
   (https://www.python.org/dev/peps/pep-0484/#storing-and-distributing-stub-files)

.. [3] PEP 426 definitions
   (https://www.python.org/dev/peps/pep-0426/)

.. [4] Example implementation in a type checker
   (https://mypy.readthedocs.io/en/latest/installed_packages.html)

.. [typed_pkg] Sample typed package
   (https://github.com/ethanhs/sample-typed-package)

.. [pkg_checker] Sample package checker
   (https://github.com/ethanhs/check_typedpkg)

Copyright
=========

This document has been placed in the public domain.




..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180412/3a4c1b62/attachment-0001.html>