[Python-checkins] [3.11] gh-98706: Sync with importlib_metadata 4.13.0. (GH-98875)
miss-islington
webhook-mailer at python.org
Sat Nov 5 09:55:10 EDT 2022
https://github.com/python/cpython/commit/07734a42dbea13900199334ff9cee4af0e2ce2fc
commit: 07734a42dbea13900199334ff9cee4af0e2ce2fc
branch: 3.11
author: Jason R. Coombs <jaraco at jaraco.com>
committer: miss-islington <31488909+miss-islington at users.noreply.github.com>
date: 2022-11-05T06:55:04-07:00
summary:
[3.11] gh-98706: Sync with importlib_metadata 4.13.0. (GH-98875)
These changes are already applied to main but have been selected from importlib_metadata 4.x for their bug fixes.
files:
A Misc/NEWS.d/next/Library/2022-10-30-12-22-24.gh-issue-98706.v1Kuy5.rst
M Doc/library/importlib.metadata.rst
M Lib/importlib/metadata/__init__.py
M Lib/test/test_importlib/fixtures.py
M Lib/test/test_importlib/test_main.py
M Lib/test/test_importlib/test_metadata_api.py
diff --git a/Doc/library/importlib.metadata.rst b/Doc/library/importlib.metadata.rst
index 107b7465c91b..57991779ccfd 100644
--- a/Doc/library/importlib.metadata.rst
+++ b/Doc/library/importlib.metadata.rst
@@ -13,29 +13,61 @@
**Source code:** :source:`Lib/importlib/metadata/__init__.py`
-``importlib.metadata`` is a library that provides for access to installed
-package metadata. Built in part on Python's import system, this library
+``importlib_metadata`` is a library that provides access to
+the metadata of an installed `Distribution Package <https://packaging.python.org/en/latest/glossary/#term-Distribution-Package>`_,
+such as its entry points
+or its top-level names (`Import Package <https://packaging.python.org/en/latest/glossary/#term-Import-Package>`_\s, modules, if any).
+Built in part on Python's import system, this library
intends to replace similar functionality in the `entry point
API`_ and `metadata API`_ of ``pkg_resources``. Along with
-:mod:`importlib.resources` (with new features backported to the
-`importlib_resources`_ package), this can eliminate the need to use the older
-and less efficient
+:mod:`importlib.resources`,
+this package can eliminate the need to use the older and less efficient
``pkg_resources`` package.
-By "installed package" we generally mean a third-party package installed into
-Python's ``site-packages`` directory via tools such as `pip
-<https://pypi.org/project/pip/>`_. Specifically,
-it means a package with either a discoverable ``dist-info`` or ``egg-info``
-directory, and metadata defined by :pep:`566` or its older specifications.
-By default, package metadata can live on the file system or in zip archives on
+``importlib_metadata`` operates on third-party *distribution packages*
+installed into Python's ``site-packages`` directory via tools such as
+`pip <https://pypi.org/project/pip/>`_.
+Specifically, it works with distributions with discoverable
+``dist-info`` or ``egg-info`` directories,
+and metadata defined by the `Core metadata specifications <https://packaging.python.org/en/latest/specifications/core-metadata/#core-metadata>`_.
+
+.. important::
+
+ These are *not* necessarily equivalent to or correspond 1:1 with
+ the top-level *import package* names
+ that can be imported inside Python code.
+ One *distribution package* can contain multiple *import packages*
+ (and single modules),
+ and one top-level *import package*
+ may map to multiple *distribution packages*
+ if it is a namespace package.
+ You can use :ref:`package_distributions() <package-distributions>`
+ to get a mapping between them.
+
+By default, distribution metadata can live on the file system
+or in zip archives on
:data:`sys.path`. Through an extension mechanism, the metadata can live almost
anywhere.
+.. seealso::
+
+ https://importlib-metadata.readthedocs.io/
+ The documentation for ``importlib_metadata``, which supplies a
+ backport of ``importlib.metadata``.
+ This includes an `API reference
+ <https://importlib-metadata.readthedocs.io/en/latest/api.html>`__
+ for this module's classes and functions,
+ as well as a `migration guide
+ <https://importlib-metadata.readthedocs.io/en/latest/migration.html>`__
+ for existing users of ``pkg_resources``.
+
+
Overview
========
-Let's say you wanted to get the version string for a package you've installed
+Let's say you wanted to get the version string for a
+`Distribution Package <https://packaging.python.org/en/latest/glossary/#term-Distribution-Package>`_ you've installed
using ``pip``. We start by creating a virtual environment and installing
something into it:
@@ -54,9 +86,9 @@ You can get the version string for ``wheel`` by running the following:
>>> version('wheel') # doctest: +SKIP
'0.32.3'
-You can also get the set of entry points keyed by group, such as
+You can also get a collection of entry points selectable by properties of the EntryPoint (typically 'group' or 'name'), such as
``console_scripts``, ``distutils.commands`` and others. Each group contains a
-sequence of :ref:`EntryPoint <entry-points>` objects.
+collection of :ref:`EntryPoint <entry-points>` objects.
You can get the :ref:`metadata for a distribution <metadata>`::
@@ -91,7 +123,7 @@ Query all entry points::
>>> eps = entry_points() # doctest: +SKIP
The ``entry_points()`` function returns an ``EntryPoints`` object,
-a sequence of all ``EntryPoint`` objects with ``names`` and ``groups``
+a collection of all ``EntryPoint`` objects with ``names`` and ``groups``
attributes for convenience::
>>> sorted(eps.groups) # doctest: +SKIP
@@ -156,7 +188,8 @@ interface to retrieve entry points by group.
Distribution metadata
---------------------
-Every distribution includes some metadata, which you can extract using the
+Every `Distribution Package <https://packaging.python.org/en/latest/glossary/#term-Distribution-Package>`_ includes some metadata,
+which you can extract using the
``metadata()`` function::
>>> wheel_metadata = metadata('wheel') # doctest: +SKIP
@@ -174,6 +207,13 @@ all the metadata in a JSON-compatible form per :PEP:`566`::
>>> wheel_metadata.json['requires_python']
'>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*'
+.. note::
+
+ The actual type of the object returned by ``metadata()`` is an
+ implementation detail and should be accessed only through the interface
+ described by the
+ `PackageMetadata protocol <https://importlib-metadata.readthedocs.io/en/latest/api.html#importlib_metadata.PackageMetadata>`_.
+
.. versionchanged:: 3.10
The ``Description`` is now included in the metadata when presented
through the payload. Line continuation characters have been removed.
@@ -187,7 +227,8 @@ all the metadata in a JSON-compatible form per :PEP:`566`::
Distribution versions
---------------------
-The ``version()`` function is the quickest way to get a distribution's version
+The ``version()`` function is the quickest way to get a
+`Distribution Package <https://packaging.python.org/en/latest/glossary/#term-Distribution-Package>`_'s version
number, as a string::
>>> version('wheel') # doctest: +SKIP
@@ -200,7 +241,8 @@ Distribution files
------------------
You can also get the full set of files contained within a distribution. The
-``files()`` function takes a distribution package name and returns all of the
+``files()`` function takes a `Distribution Package <https://packaging.python.org/en/latest/glossary/#term-Distribution-Package>`_ name
+and returns all of the
files installed by this distribution. Each file object returned is a
``PackagePath``, a :class:`pathlib.PurePath` derived object with additional ``dist``,
``size``, and ``hash`` properties as indicated by the metadata. For example::
@@ -245,19 +287,24 @@ distribution is not known to have the metadata present.
Distribution requirements
-------------------------
-To get the full set of requirements for a distribution, use the ``requires()``
+To get the full set of requirements for a `Distribution Package <https://packaging.python.org/en/latest/glossary/#term-Distribution-Package>`_,
+use the ``requires()``
function::
>>> requires('wheel') # doctest: +SKIP
["pytest (>=3.0.0) ; extra == 'test'", "pytest-cov ; extra == 'test'"]
-Package distributions
----------------------
+.. _package-distributions:
+.. _import-distribution-package-mapping:
-A convenience method to resolve the distribution or
-distributions (in the case of a namespace package) for top-level
-Python packages or modules::
+Mapping import to distribution packages
+---------------------------------------
+
+A convenience method to resolve the `Distribution Package <https://packaging.python.org/en/latest/glossary/#term-Distribution-Package>`_
+name (or names, in the case of a namespace package)
+that provide each importable top-level
+Python module or `Import Package <https://packaging.python.org/en/latest/glossary/#term-Import-Package>`_::
>>> packages_distributions()
{'importlib_metadata': ['importlib-metadata'], 'yaml': ['PyYAML'], 'jaraco': ['jaraco.classes', 'jaraco.functools'], ...}
@@ -271,7 +318,8 @@ Distributions
While the above API is the most common and convenient usage, you can get all
of that information from the ``Distribution`` class. A ``Distribution`` is an
-abstract object that represents the metadata for a Python package. You can
+abstract object that represents the metadata for
+a Python `Distribution Package <https://packaging.python.org/en/latest/glossary/#term-Distribution-Package>`_. You can
get the ``Distribution`` instance::
>>> from importlib.metadata import distribution # doctest: +SKIP
@@ -291,22 +339,36 @@ instance::
>>> dist.metadata['License'] # doctest: +SKIP
'MIT'
-The full set of available metadata is not described here. See :pep:`566`
-for additional details.
+The full set of available metadata is not described here.
+See the `Core metadata specifications <https://packaging.python.org/en/latest/specifications/core-metadata/#core-metadata>`_ for additional details.
+
+
+Distribution Discovery
+======================
+
+By default, this package provides built-in support for discovery of metadata
+for file system and zip file `Distribution Package <https://packaging.python.org/en/latest/glossary/#term-Distribution-Package>`_\s.
+This metadata finder search defaults to ``sys.path``, but varies slightly in how it interprets those values from how other import machinery does. In particular:
+
+- ``importlib.metadata`` does not honor :class:`bytes` objects on ``sys.path``.
+- ``importlib.metadata`` will incidentally honor :py:class:`pathlib.Path` objects on ``sys.path`` even though such values will be ignored for imports.
Extending the search algorithm
==============================
-Because package metadata is not available through :data:`sys.path` searches, or
-package loaders directly, the metadata for a package is found through import
-system :ref:`finders <finders-and-loaders>`. To find a distribution package's metadata,
+Because `Distribution Package <https://packaging.python.org/en/latest/glossary/#term-Distribution-Package>`_ metadata
+is not available through :data:`sys.path` searches, or
+package loaders directly,
+the metadata for a distribution is found through import
+system `finders`_. To find a distribution package's metadata,
``importlib.metadata`` queries the list of :term:`meta path finders <meta path finder>` on
:data:`sys.meta_path`.
-The default ``PathFinder`` for Python includes a hook that calls into
-``importlib.metadata.MetadataPathFinder`` for finding distributions
-loaded from typical file-system-based paths.
+By default ``importlib_metadata`` installs a finder for distribution packages
+found on the file system.
+This finder doesn't actually find any *distributions*,
+but it can find their metadata.
The abstract class :py:class:`importlib.abc.MetaPathFinder` defines the
interface expected of finders by Python's import system.
@@ -335,4 +397,4 @@ a custom finder, return instances of this derived ``Distribution`` in the
.. _`entry point API`: https://setuptools.readthedocs.io/en/latest/pkg_resources.html#entry-points
.. _`metadata API`: https://setuptools.readthedocs.io/en/latest/pkg_resources.html#metadata-api
-.. _`importlib_resources`: https://importlib-resources.readthedocs.io/en/latest/index.html
+.. _`finders`: https://docs.python.org/3/reference/import.html#finders-and-loaders
diff --git a/Lib/importlib/metadata/__init__.py b/Lib/importlib/metadata/__init__.py
index 9ceae8a0e04b..bbdbceebe759 100644
--- a/Lib/importlib/metadata/__init__.py
+++ b/Lib/importlib/metadata/__init__.py
@@ -184,6 +184,10 @@ class EntryPoint(DeprecatedTuple):
following the attr, and following any extras.
"""
+ name: str
+ value: str
+ group: str
+
dist: Optional['Distribution'] = None
def __init__(self, name, value, group):
@@ -543,7 +547,7 @@ def locate_file(self, path):
"""
@classmethod
- def from_name(cls, name):
+ def from_name(cls, name: str):
"""Return the Distribution for the given package name.
:param name: The name of the distribution package to search for.
@@ -551,13 +555,13 @@ def from_name(cls, name):
package, if found.
:raises PackageNotFoundError: When the named package's distribution
metadata cannot be found.
+ :raises ValueError: When an invalid value is supplied for name.
"""
- for resolver in cls._discover_resolvers():
- dists = resolver(DistributionFinder.Context(name=name))
- dist = next(iter(dists), None)
- if dist is not None:
- return dist
- else:
+ if not name:
+ raise ValueError("A distribution name is required.")
+ try:
+ return next(cls.discover(name=name))
+ except StopIteration:
raise PackageNotFoundError(name)
@classmethod
@@ -945,13 +949,26 @@ def _normalized_name(self):
normalized name from the file system path.
"""
stem = os.path.basename(str(self._path))
- return self._name_from_stem(stem) or super()._normalized_name
+ return (
+ pass_none(Prepared.normalize)(self._name_from_stem(stem))
+ or super()._normalized_name
+ )
- def _name_from_stem(self, stem):
- name, ext = os.path.splitext(stem)
+ @staticmethod
+ def _name_from_stem(stem):
+ """
+ >>> PathDistribution._name_from_stem('foo-3.0.egg-info')
+ 'foo'
+ >>> PathDistribution._name_from_stem('CherryPy-3.0.dist-info')
+ 'CherryPy'
+ >>> PathDistribution._name_from_stem('face.egg-info')
+ 'face'
+ >>> PathDistribution._name_from_stem('foo.bar')
+ """
+ filename, ext = os.path.splitext(stem)
if ext not in ('.dist-info', '.egg-info'):
return
- name, sep, rest = stem.partition('-')
+ name, sep, rest = filename.partition('-')
return name
@@ -991,6 +1008,15 @@ def version(distribution_name):
return distribution(distribution_name).version
+_unique = functools.partial(
+ unique_everseen,
+ key=operator.attrgetter('_normalized_name'),
+)
+"""
+Wrapper for ``distributions`` to return unique distributions by name.
+"""
+
+
def entry_points(**params) -> Union[EntryPoints, SelectableGroups]:
"""Return EntryPoint objects for all installed packages.
@@ -1008,10 +1034,8 @@ def entry_points(**params) -> Union[EntryPoints, SelectableGroups]:
:return: EntryPoints or SelectableGroups for all installed packages.
"""
- norm_name = operator.attrgetter('_normalized_name')
- unique = functools.partial(unique_everseen, key=norm_name)
eps = itertools.chain.from_iterable(
- dist.entry_points for dist in unique(distributions())
+ dist.entry_points for dist in _unique(distributions())
)
return SelectableGroups.load(eps).select(**params)
diff --git a/Lib/test/test_importlib/fixtures.py b/Lib/test/test_importlib/fixtures.py
index 803d3738d263..e7be77b3957c 100644
--- a/Lib/test/test_importlib/fixtures.py
+++ b/Lib/test/test_importlib/fixtures.py
@@ -5,6 +5,7 @@
import pathlib
import tempfile
import textwrap
+import functools
import contextlib
from test.support.os_helper import FS_NONASCII
@@ -296,3 +297,18 @@ def setUp(self):
# Add self.zip_name to the front of sys.path.
self.resources = contextlib.ExitStack()
self.addCleanup(self.resources.close)
+
+
+def parameterize(*args_set):
+ """Run test method with a series of parameters."""
+
+ def wrapper(func):
+ @functools.wraps(func)
+ def _inner(self):
+ for args in args_set:
+ with self.subTest(**args):
+ func(self, **args)
+
+ return _inner
+
+ return wrapper
diff --git a/Lib/test/test_importlib/test_main.py b/Lib/test/test_importlib/test_main.py
index c80a0e4a5a9f..d9d067c4b23d 100644
--- a/Lib/test/test_importlib/test_main.py
+++ b/Lib/test/test_importlib/test_main.py
@@ -1,7 +1,6 @@
import re
import json
import pickle
-import textwrap
import unittest
import warnings
import importlib.metadata
@@ -16,6 +15,7 @@
Distribution,
EntryPoint,
PackageNotFoundError,
+ _unique,
distributions,
entry_points,
metadata,
@@ -51,6 +51,14 @@ def test_package_not_found_mentions_metadata(self):
def test_new_style_classes(self):
self.assertIsInstance(Distribution, type)
+ @fixtures.parameterize(
+ dict(name=None),
+ dict(name=''),
+ )
+ def test_invalid_inputs_to_from_name(self, name):
+ with self.assertRaises(Exception):
+ Distribution.from_name(name)
+
class ImportTests(fixtures.DistInfoPkg, unittest.TestCase):
def test_import_nonexistent_module(self):
@@ -78,48 +86,50 @@ def test_resolve_without_attr(self):
class NameNormalizationTests(fixtures.OnSysPath, fixtures.SiteDir, unittest.TestCase):
@staticmethod
- def pkg_with_dashes(site_dir):
+ def make_pkg(name):
"""
- Create minimal metadata for a package with dashes
- in the name (and thus underscores in the filename).
+ Create minimal metadata for a dist-info package with
+ the indicated name on the file system.
"""
- metadata_dir = site_dir / 'my_pkg.dist-info'
- metadata_dir.mkdir()
- metadata = metadata_dir / 'METADATA'
- with metadata.open('w', encoding='utf-8') as strm:
- strm.write('Version: 1.0\n')
- return 'my-pkg'
+ return {
+ f'{name}.dist-info': {
+ 'METADATA': 'VERSION: 1.0\n',
+ },
+ }
def test_dashes_in_dist_name_found_as_underscores(self):
"""
For a package with a dash in the name, the dist-info metadata
uses underscores in the name. Ensure the metadata loads.
"""
- pkg_name = self.pkg_with_dashes(self.site_dir)
- assert version(pkg_name) == '1.0'
-
- @staticmethod
- def pkg_with_mixed_case(site_dir):
- """
- Create minimal metadata for a package with mixed case
- in the name.
- """
- metadata_dir = site_dir / 'CherryPy.dist-info'
- metadata_dir.mkdir()
- metadata = metadata_dir / 'METADATA'
- with metadata.open('w', encoding='utf-8') as strm:
- strm.write('Version: 1.0\n')
- return 'CherryPy'
+ fixtures.build_files(self.make_pkg('my_pkg'), self.site_dir)
+ assert version('my-pkg') == '1.0'
def test_dist_name_found_as_any_case(self):
"""
Ensure the metadata loads when queried with any case.
"""
- pkg_name = self.pkg_with_mixed_case(self.site_dir)
+ pkg_name = 'CherryPy'
+ fixtures.build_files(self.make_pkg(pkg_name), self.site_dir)
assert version(pkg_name) == '1.0'
assert version(pkg_name.lower()) == '1.0'
assert version(pkg_name.upper()) == '1.0'
+ def test_unique_distributions(self):
+ """
+ Two distributions varying only by non-normalized name on
+ the file system should resolve as the same.
+ """
+ fixtures.build_files(self.make_pkg('abc'), self.site_dir)
+ before = list(_unique(distributions()))
+
+ alt_site_dir = self.fixtures.enter_context(fixtures.tempdir())
+ self.fixtures.enter_context(self.add_sys_path(alt_site_dir))
+ fixtures.build_files(self.make_pkg('ABC'), alt_site_dir)
+ after = list(_unique(distributions()))
+
+ assert len(after) == len(before)
+
class NonASCIITests(fixtures.OnSysPath, fixtures.SiteDir, unittest.TestCase):
@staticmethod
@@ -128,11 +138,12 @@ def pkg_with_non_ascii_description(site_dir):
Create minimal metadata for a package with non-ASCII in
the description.
"""
- metadata_dir = site_dir / 'portend.dist-info'
- metadata_dir.mkdir()
- metadata = metadata_dir / 'METADATA'
- with metadata.open('w', encoding='utf-8') as fp:
- fp.write('Description: pôrˈtend')
+ contents = {
+ 'portend.dist-info': {
+ 'METADATA': 'Description: pôrˈtend',
+ },
+ }
+ fixtures.build_files(contents, site_dir)
return 'portend'
@staticmethod
@@ -141,19 +152,15 @@ def pkg_with_non_ascii_description_egg_info(site_dir):
Create minimal metadata for an egg-info package with
non-ASCII in the description.
"""
- metadata_dir = site_dir / 'portend.dist-info'
- metadata_dir.mkdir()
- metadata = metadata_dir / 'METADATA'
- with metadata.open('w', encoding='utf-8') as fp:
- fp.write(
- textwrap.dedent(
- """
+ contents = {
+ 'portend.dist-info': {
+ 'METADATA': """
Name: portend
- pôrˈtend
- """
- ).strip()
- )
+ pôrˈtend""",
+ },
+ }
+ fixtures.build_files(contents, site_dir)
return 'portend'
def test_metadata_loads(self):
diff --git a/Lib/test/test_importlib/test_metadata_api.py b/Lib/test/test_importlib/test_metadata_api.py
index c86bb4df19c5..69c78e9820c0 100644
--- a/Lib/test/test_importlib/test_metadata_api.py
+++ b/Lib/test/test_importlib/test_metadata_api.py
@@ -89,15 +89,15 @@ def test_entry_points_distribution(self):
self.assertIn(ep.dist.name, ('distinfo-pkg', 'egginfo-pkg'))
self.assertEqual(ep.dist.version, "1.0.0")
- def test_entry_points_unique_packages(self):
+ def test_entry_points_unique_packages_normalized(self):
"""
Entry points should only be exposed for the first package
- on sys.path with a given name.
+ on sys.path with a given name (even when normalized).
"""
alt_site_dir = self.fixtures.enter_context(fixtures.tempdir())
self.fixtures.enter_context(self.add_sys_path(alt_site_dir))
alt_pkg = {
- "distinfo_pkg-1.1.0.dist-info": {
+ "DistInfo_pkg-1.1.0.dist-info": {
"METADATA": """
Name: distinfo-pkg
Version: 1.1.0
diff --git a/Misc/NEWS.d/next/Library/2022-10-30-12-22-24.gh-issue-98706.v1Kuy5.rst b/Misc/NEWS.d/next/Library/2022-10-30-12-22-24.gh-issue-98706.v1Kuy5.rst
new file mode 100644
index 000000000000..9b5deb0372f2
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2022-10-30-12-22-24.gh-issue-98706.v1Kuy5.rst
@@ -0,0 +1,7 @@
+[3.11] Applied changes from importlib_metadata `4.11.4 through 4.13
+<https://importlib-metadata.readthedocs.io/en/latest/history.html#v4-13-0>`_,
+including compatibility and robustness fixes for ``Distribution`` objects
+without ``_normalized_name``, disallowing invalid inputs to
+``Distribution.from_name``, and refined behaviors in
+``PathDistribution._name_from_stem`` and
+``PathDistribution._normalized_name``.
More information about the Python-checkins
mailing list