[Python-Dev] PEP 561 rework
Ethan Smith
ethan at ethanhs.me
Sun Nov 12 14:21:33 EST 2017
On Sun, Nov 12, 2017 at 9:53 AM, Jelle Zijlstra <jelle.zijlstra at gmail.com>
wrote:
>
>
> 2017-11-12 3:40 GMT-08:00 Ethan Smith <ethan at ethanhs.me>:
>
>> Hello,
>>
>> I re-wrote my PEP to have typing opt-in be per-package rather than
>> per-distribution. This greatly simplifies things, and thanks to the
>> feedback and suggestions of Nick Coghlan, it is entirely compatible with
>> older packaging tooling.
>>
>> The main idea is there are two types of packages:
>> - types are packaged with runtime code (inline or stubs in the same
>> package)
>> - types are in a separate package (a third party or maintainer wants to
>> ship type information, but not with runtime code).
>>
>> The PEP is live on python.org: https://www.python.org/dev/peps/pep-0561/
>>
>> And as always, duplicated below.
>>
>> Cheers,
>>
>> Ethan Smith
>>
>> ---------------------------------------------------
>>
>> PEP: 561
>> Title: Distributing and Packaging Type Information
>> Author: Ethan Smith <ethan at ethanhs.me>
>> Status: Draft
>> Type: Standards Track
>> Content-Type: text/x-rst
>> Created: 09-Sep-2017
>> Python-Version: 3.7
>> Post-History: 10-Sep-2017, 12-Sep-2017, 06-Oct-2017, 26-Oct-2017
>>
>>
>> Abstract
>> ========
>>
>> PEP 484 introduced type hinting to Python, with goals of making typing
>> gradual and easy to adopt. Currently, typing information must be distributed
>> manually. This PEP provides a standardized means to leverage existing tooling
>> to package and distribute type information with minimal work and an ordering
>> for type checkers to resolve modules and collect this information for type
>> checking.
>>
>>
>> Rationale
>> =========
>>
>> Currently, package authors wish to distribute code that has inline type
>> information. Additionally, maintainers would like to distribute stub files
>> to keep Python 2 compatibility while using newer annotation syntax. However,
>> there is no standard method to distribute packages with type information.
>> Also, if one wished to ship stub files privately the only method available
>> would be via setting ``MYPYPATH`` or the equivalent to manually point to
>> stubs. If the package can be released publicly, it can be added to
>> typeshed [1]_. However, this does not scale and becomes a burden on the
>> maintainers of typeshed. In addition, it ties bug fixes in stubs to releases
>> of the tool using typeshed.
>>
>> PEP 484 has a brief section on distributing typing information. In this
>> section [2]_ the PEP recommends using ``shared/typehints/pythonX.Y/`` for
>> shipping stub files. However, manually adding a path to stub files for each
>> third party library does not scale. The simplest approach people have taken
>> is to add ``site-packages`` to their ``MYPYPATH``, but this causes type
>> checkers to fail on packages that are highly dynamic (e.g. sqlalchemy
>> and Django).
>>
>>
>> Definition of Terms
>> ===================
>>
>> The definition of "MAY", "MUST", and "SHOULD", and "SHOULD NOT" are
>> to be interpreted as described in RFC 2119.
>>
>> "inline" - the types are part of the runtime code using PEP 526 and 3107
>> syntax.
>>
>> "stubs" - files containing only type information, empty of runtime code.
>>
>> "Distributions" are the packaged files which are used to publish and distribute
>> a release. [3]_
>>
>> "Module" a file containing Python runtime code or stubbed type information.
>>
>> "Package" a directory or directories that namespace Python modules.
>>
>>
>> Specification
>> =============
>>
>> There are several motivations and methods of supporting typing in a package.
>> This PEP recognizes three (3) types of packages that users of typing wish to
>> create:
>>
>> 1. The package maintainer would like to add type information inline.
>>
>> 2. The package maintainer would like to add type information via stubs.
>>
>> 3. A third party or package maintainer would like to share stub files for
>> a package, but the maintainer does not want to include them in the source
>> of the package.
>>
>> This PEP aims to support these scenarios and make them simple to add to
>> packaging and deployment.
>>
>> The two major parts of this specification are the packaging specifications
>> and the resolution order for resolving module type information. The type
>> checking spec is meant to replace the ``shared/typehints/pythonX.Y/`` spec
>> of PEP 484 [2]_.
>>
>> New third party stub libraries SHOULD distribute stubs via the third party
>> packaging methods proposed in this PEP in place of being added to typeshed.
>> Typeshed will remain in use, but if maintainers are found, third party stubs
>> in typeshed MAY be split into their own package.
>>
>>
>> Packaging Type Information
>> --------------------------
>>
>> In order to make packaging and distributing type information as simple and
>> easy as possible, packaging and distribution is done through existing
>> frameworks.
>>
>> Package maintainers who wish to support type checking of their code MUST add
>> a ``py.typed`` file to their package supporting typing. This marker is
>> recursive, if a top-level package includes it, all sub-packages MUST support
>> type checking as well. To have this file installed with the package,
>> maintainers can use existing packaging options such as ``package_data`` in
>> distutils, shown below.
>>
>> Distutils option example::
>>
>> ...
>> package_data = {
>> 'pkg': ['py.typed'],
>> },
>> ...
>>
>> For namespace packages, the ``py.typed`` file should be in the submodules of
>> the namespace, to avoid conflicts and for clarity.
>>
>> Stub Only Packages
>> ''''''''''''''''''
>>
>> For package maintainers wishing to ship stub files containing all of their
>> type information, it is preferred that the ``*.pyi`` stubs are alongside the
>> corresponding ``*.py`` files. However, the stubs can also be put in a separate
>> package and distributed separately. Third parties can also find this method
>> useful if they wish to distribute stub files. The name of the stub package
>> MUST follow the scheme ``pkg_stubs`` for type stubs for the package named
>> ``pkg``. The normal resolution order of checking ``*.pyi`` before ``*.py``
>> will be maintained.
>>
>> This is very minor, but what do you think of using "pkg-stubs" instead of
> "pkg_stubs" (using a hyphen rather than an underscore)? This would make the
> name illegal to import as a normal Python package, which makes it clear
> that it's not a normal package. Also, there could be real packages named
> "_stubs".
>
I suppose this makes sense. I checked PyPI and as of a few weeks ago there
were no packages with the name pattern, but I like the idea of making it
explicitly non-runtime importable. I cannot think of any reason not to do
it, and the avoidance of confusion about the package being importable is a
benefit. I will make the change with my next round of edits.
> Third parties seeking to distribute stub files are encouraged to contact the
>> maintainer of the package about distribution alongside the package. If the
>> maintainer does not wish to maintain or package stub files or type information
>> inline, then a third party stub only package can be created.
>>
>> In addition, stub only distributions SHOULD indicate which version(s)
>> of the runtime package are supported by indicating the runtime distribution's
>> version(s) through normal dependency data. For example, if there was a
>> stub package ``flyingcircus_stubs``, it can indicate the versions of the
>> runtime ``Flyingcircus`` distribution supported through ``install_requires``
>> in distutils based tools, or the equivalent in other packaging tools.
>>
>>
>> Type Checker Module Resolution Order
>> ------------------------------------
>>
>> The following is the order that type checkers supporting this PEP SHOULD
>> resolve modules containing type information:
>>
>> 1. User code - the files the type checker is running on.
>>
>> 2. Stubs or Python source manually put in the beginning of the path. Type
>> checkers SHOULD provide this to allow the user complete control of which
>> stubs to use, and patch broken stubs/inline types from packages.
>>
>> 3. Stub packages - these packages can supersede the installed packages.
>> They can be found at ``pkg_stubs`` for package ``pkg``.
>>
>> 4. Inline packages - if there is nothing overriding the installed
>> package, and it opts into type checking, inline types SHOULD be used.
>>
>> 5. Typeshed (if used) - Provides the stdlib types and several third party
>> libraries.
>>
>> Type checkers that check a different Python version than the version they run
>> on MUST find the type information in the ``site-packages``/``dist-packages``
>> of that Python version. This can be queried e.g.
>> ``pythonX.Y -c 'import site; print(site.getsitepackages())'``. It is also recommended
>> that the type checker allow for the user to point to a particular Python
>> binary, in case it is not in the path.
>>
>>
>> Implementation
>> ==============
>>
>> The proposed scheme of indicating support for typing is completely backwards
>> compatible, and requires no modification to tooling. A sample package with
>> inline types is available [typed_pkg]_, as well as a sample package checker
>> [pkg_checker]_ which reads the metadata of installed packages and reports on
>> their status as either not typed, inline typed, or a stub package.
>>
>>
>> Acknowledgements
>> ================
>>
>> This PEP would not have been possible without the ideas, feedback, and support
>> of Ivan Levkivskyi, Jelle Zijlstra, Nick Coghlan, Daniel F Moisset, Nathaniel
>> Smith, and Guido van Rossum.
>>
>>
>> Version History
>> ===============
>>
>> * 2017-11-12
>>
>> * Rewritten to use existing tooling only
>> * No need to indicate kind of type information in metadata
>> * Name of marker file changed from ``.typeinfo`` to ``py.typed``
>>
>> * 2017-11-10
>>
>> * Specification re-written to use package metadata instead of distribution
>> metadata.
>> * Removed stub only packages and merged into third party packages spec.
>> * Removed suggestion for typecheckers to consider checking runtime versions
>> * Implementations updated to reflect PEP changes.
>>
>> * 2017-10-26
>>
>> * Added implementation references.
>> * Added acknowledgements and version history.
>>
>> * 2017-10-06
>>
>> * Rewritten to use .distinfo/METADATA over a distutils specific command.
>> * Clarify versioning of third party stub packages.
>>
>> * 2017-09-11
>>
>> * Added information about current solutions and typeshed.
>> * Clarify rationale.
>>
>>
>> References
>> ==========
>> .. [1] Typeshed (https://github.com/python/typeshed)
>>
>> .. [2] PEP 484, Storing and Distributing Stub Files
>> (https://www.python.org/dev/peps/pep-0484/#storing-and-distributing-stub-files)
>>
>> .. [3] PEP 426 definitions
>> (https://www.python.org/dev/peps/pep-0426/)
>>
>> .. [typed_pkg] Sample typed package
>> (https://github.com/ethanhs/sample-typed-package)
>>
>> .. [pkg_checker] Sample package checker
>> (https://github.com/ethanhs/check_typedpkg)
>>
>> Copyright
>> =========
>>
>> This document has been placed in the public domain.
>>
>>
>>
>> ..
>> Local Variables:
>> mode: indented-text
>> indent-tabs-mode: nil
>> sentence-end-double-space: t
>> fill-column: 70
>> coding: utf-8
>> End:
>>
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/jelle.
>> zijlstra%40gmail.com
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20171112/1fc86bef/attachment-0001.html>
More information about the Python-Dev
mailing list