Notice: While Javascript is not essential for this website, your interaction with the content will be limited. Please turn Javascript on for the full experience.

Editable installs for PEP 517 style build backends

PEP:660
Title:Editable installs for PEP 517 style build backends
Author:Daniel Holth <dholth at gmail.com>, Stéphane Bidoul <stephane.bidoul at gmail.com>
Sponsor:Paul Moore <p.f.moore at gmail.com>
Discussions-To:https://discuss.python.org/t/draft-pep-editable-installs-for-pep-517-style-build-backends/8510
Status:Draft
Type:Standards Track
Created:30-Mar-2021
Post-History:

Abstract

This document describes a PEP 517 style method for the installation of packages in editable mode.

Motivation

Python programmers want to be able to develop packages without having to install (i.e. copy) them into site-packages, for example, by working in a checkout of the source repository.

While this can be done by adding the relevant source directories to PYTHONPATH, setuptools provides the setup.py develop mechanism that makes the process easier, and also installs dependencies and entry points such as console scripts. pip exposes this mechanism via its pip install --editable option.

The installation of projects in such a way that the python code being imported remains in the source directory is known as the editable installation mode.

Now that PEP 517 provides a mechanism to create alternatives to setuptools, and decouple installation front ends from build backends, we need a new mechanism to install packages in editable mode.

Rationale

PEP 517 deferred "Editable installs", meaning non-setup.py distributions lacked that feature. The only way to retain editable installs for these distributions was to provide a compatible setup.py develop implementation. By definining an editable hook other build frontends gain parity with setup.py.

Terminology and goals

The editable installation mode implies that the source code of the project being installed is available in a local directory.

Once the project is installed in editable mode, users expect that changes to the project python code in the local source tree become effective without the need of a new installation step.

Some kind of changes, such as the addition or modification of entry points, or the addition of new dependencies, require a new installation step to become effective. These changes are typically made in build backend configuration files (such as pyproject.toml), so it is consistent with the general user expectation that python source code is imported from the source tree.

The modification of non-python source code such a C extension modules obviously require a compilation and/or installation step to become effective. The exact steps to perform will remain specific to the build backend used.

When a project is installed in editable mode, users expect the installation to behave identically as a regular installation. Depending on the way build backends implement this specification, some minor differences may be visible such as the presence of additional files that are in the source tree and would not be part of a regular install. Build backends are encouraged to document such potential differences.

The Mechanism

This PEP adds a single optional hook to the PEP 517 backend interface. The hook is used to build a wheel that, when installed, allows that distribution to be imported from its source folder.

build_wheel_for_editable

def build_wheel_for_editable(
    wheel_directory,
    scheme=scheme,
    config_settings=None):
    ...

scheme: a dictionary of installation categories { 'purelib': '/home/py/.../site-packages', 'platlib': '...'}. This makes it possible to use relative paths to the source code, which might help the interpreter find the package after the root path changes with chroot or similar.

Must build a .whl file, and place it in the specified wheel_directory. It must return the basename (not the full path) of the .whl file it creates, as a unicode string.

May do an in-place build of the distribution as a side effect so that any extension modules or other built artifacts are ready to be used.

The .whl file must comply with the Wheel binary file format specification (PEP 427). In particular it must contain a compliant .dist-info directory. Metadata must be identical as the one that would have been produced by build_wheel or prepare_metadata_for_build_wheel, except for Requires-Dist which may differ slightly as explained below.

Build-backends must produce wheels that have the same dependencies (Requires-Dist metadata) as wheels produced by the build_wheel hook, with the exception that they can add dependencies necessary for their editable mechanism to function at runtime (such as editables [1]).

The filename for the “editable” wheel needs to be PEP 427 compliant too. It does not need to use the same tags as build_wheel but it must be tagged as compatible with the system.

An “editable” wheel uses the wheel format not for distribution but as ephemeral communication between the build system and the front end. This avoids having the build backend install anything directly. This wheel must not be exposed to end users, nor cached, nor distributed.

get_requires_for_build_wheel_for_editable

def get_requires_for_build_wheel_for_editable(config_settings=None):
    ...

This hook MUST return an additional list of strings containing PEP 508 dependency specifications, above and beyond those specified in the pyproject.toml file, to be installed when calling the build_wheel_for_editable hooks.

If not defined, the default implementation is equivalent to return [].

What to put in the wheel

Build backends may use different techniques to achive the goals of an editable install. This section provides examples and is not normative.

  • Build backends may choose to place a .pth file at the root of the .whl file, containing the root directory of the source tree. This approach is simple but not very precise, although it may be considered good enough (especially when using the src layout) and is similar to what setup.py develop currently does.
  • The editables [1] library shows how to build proxy modules that provide a high quality editable installation. It accepts a list of modules to include, and hide. When imported, these proxy modules replace themselves with the code from the source tree. Path-based methods make all scripts under a path importable, often including the project's own setup.py and other scripts that would not be part of a normal installation. The proxy strategy can achieve a higher level of fidelity than path-based methods.

Frontend requirements

Frontends must install editable wheels in the same way as regular wheels. This also means uninstallation of editables does not require any special treatment.

Frontends must create a direct_url.json file in the .dist-info directory of the installed distribution, in compliance with PEP 610. The url value must be a file:// url pointing to the project directory (i.e. the directory containing pyproject.toml), and the dir_info value must be {'editable': true}.

Frontends must execute get_requires_for_build_wheel_for_editable hooks in an environment which contains the bootstrap requirements specified in the pyproject.toml file.

Frontends must execute the build_wheel_for_editable hook in an environment which contains the bootstrap requirements from pyproject.toml and those specified by the get_requires_for_build_wheel_for_editable hook.

Frontends must not rely on the prepare_metadata_for_build_wheel hook when installing in editable mode. They must use build_wheel_for_editable and inspect the resulting wheel.

Frontends must not expose the wheel obtained from build_wheel_for_editable to end users. The wheel must be discarded after installation and must not be cached nor distributed.

Rejected ideas

The ideas of having build backends append or modify the local version identifier to include the editable string has been rejected because it would not satisfy == version speicifier that include the local version identifier. In other workds pkg==1.0+local is not satisfied by version 1.0+local.editable.

Source: https://github.com/python/peps/blob/master/pep-0660.rst